Abstract
Contact-rich manipulation is ubiquitous in our day-to-day lives, encompassing a broad range of tasks including sweeping dust into a dustpan, wiping tables, erasing a whiteboard, and applying paint with a brush. A key challenge in performing these tasks lies in controlling the interactions between tools and their environments. For instance, when sweeping, it is crucial to ensure continuous contact between the bristles and the surface while directing the collected dust towards the dustpan.
In this work, we investigate learning language-conditioned, vision-based manipulation policies wherein the action representation is contact itself— predicting the lateral areas at which the robot’s end effector should meet an observable surface. Our approach, Contact-Aware and Language conditioned spatial Action MApping for contact-RIch manipulation (CALAMARI), exhibits several advantages as follows:
- benefiting from existing visual-language models for pretrained spatial features, grounding instructions to behaviors, and for sim2real transfer.
- factorizing perception and control over a natural boundary (i.e., contact) into two modules that synergize with each other, whereby action predictions can be aligned per pixel with image observations.
- low-level controllers can optimize motion trajectories that maintain contact while avoiding penetration.
Result: Sim2Real
A few contact goal (3) sequence task
“sweep dirt to dustpan”
“sweep dirt to dustpan” – compliant tool manipulation
Multiple contact goal (~20) task
“wipe up the dots”
Single contact goal task
“push the {red (train)/ blue (test)/ green (test)} button”
Video
- In 7th Conference on Robotic Learning (CoRL 2023), Atlanta (poster)
Authors
Citation
@inproceedings{wi2023calamari,
title={CALAMARI: Contact-Aware and Language conditioned spatial Action MApping for contact-RIch manipulation},
author={Wi, Youngsun and Van der Merwe, Mark and Florence, Pete and Zeng, Andy and Fazeli, Nima},
booktitle={7th Annual Conference on Robot Learning},
year={2023}
}