Extended to multi-object complex tasks: Pouring Water
To demonstrate the extensibility of our approach, we modify the teacher policy and generate data that detects both bottles and cups in the bounding box for the pouring task.
Learning a precise robotic grasping policy is crucial for embodied agents operating in complex real-world manipulation tasks. Despite significant advancements, most models still struggle with accurate spatial positioning of objects to be grasped. We first show that this spatial generalization challenge stems primarily from the extensive data requirements for adequate spatial understanding. However, collecting such data with real robots is prohibitively expensive, and relying on simulation data often leads to visual generalization gaps upon deployment. To overcome these challenges, we then focus on state-based policy generalization and present ManiBox, a novel bounding-box-guided manipulation method built on a simulation-based teacher-student framework. The teacher policy efficiently generates scalable simulation data using bounding boxes, which are proven to uniquely determine the objects' spatial positions. The student policy then utilizes these low-dimensional spatial states to enable zero-shot transfer to real robots. Through comprehensive evaluations in simulated and real-world environments, ManiBox demonstrates a marked improvement in spatial grasping generalization and adaptability to diverse objects and backgrounds. Further, our empirical study into scaling laws for policy performance indicates that spatial volume generalization scales positively with data volume. For a certain level of spatial volume, the success rate of grasping empirically follows Michaelis-Menten kinetics relative to data volume, showing a saturation effect as data increases.
To demonstrate the extensibility of our approach, we modify the teacher policy and generate data that detects both bottles and cups in the bounding box for the pouring task.
To demonstrate the extensibility of our approach, we use a detection model to recognize parts of irregular objects, which enables the grasping of the cup's handle.
ManiBox can also get objects from mid-air, from people's hands, and work in very cluttered environments.
Our setting involves the more challenging first-person robot manipulation rather than the conventional third-person perspective.
@article{tan2024manibox,
title={ManiBox: Enhancing Spatial Grasping Generalization via Scalable Simulation Data Generation},
author={Tan, Hengkai and Xu, Xuezhou and Ying, Chengyang and Mao, Xinyi and Liu, Songming and Zhang, Xingxing and Su, Hang and Zhu, Jun},
journal={arXiv preprint arXiv:2411.01850},
year={2024}
}