ManiBox: Enhancing Spatial Grasping Generalization via Scalable Simulation Data Generation

Learning a precise robotic grasping policy is crucial for embodied agents operating in complex real-world manipulation tasks. Despite significant advancements, most models still struggle with accurate spatial positioning of objects to be grasped. We first show that this spatial generalization challenge stems primarily from the extensive data requirements for adequate spatial understanding. However, collecting such data with real robots is prohibitively expensive, and relying on simulation data often leads to visual generalization gaps upon deployment. To overcome these challenges, we then focus on state-based policy generalization and present ManiBox, a novel bounding-box-guided manipulation method built on a simulation-based teacher-student framework. The teacher policy efficiently generates scalable simulation data using bounding boxes, which are proven to uniquely determine the objects' spatial positions. The student policy then utilizes these low-dimensional spatial states to enable zero-shot transfer to real robots. Through comprehensive evaluations in simulated and real-world environments, ManiBox demonstrates a marked improvement in spatial grasping generalization and adaptability to diverse objects and backgrounds. Further, our empirical study into scaling laws for policy performance indicates that spatial volume generalization scales with data volume in a power law. For a certain level of spatial volume, the success rate of grasping empirically follows Michaelis-Menten kinetics relative to data volume, showing a saturation effect as data increases.

Spatial Generalization

ManiBox can also get objects from mid-air, from people's hands, and work in very cluttered environments. (2x)

Extended to multi-object complex tasks: Pouring Water

To demonstrate the extensibility of our approach, we modify the teacher policy and generate data that detects both bottles and cups in the bounding box for the pouring task. (2x)

To demonstrate the extensibility of our approach, we use a detection model to recognize parts of irregular objects, which enables the grasping of the cup's handle. (2x)

Overview

Our setting involves the more challenging first-person robot manipulation rather than the conventional third-person perspective. The idea behind ManiBox lies in:

A scalable and automated method for generating action data is used to train the policy module, building the model's understanding of actions and alleviating the issue of data scarcity in the action modality. (i.e. policy which generates actions)
Internet-scale data is fully utilized to pretrain generalizable models in visual and textual modalities, providing essential guidance for task completion. (i.e. low-dimensional visual features like bounding boxes and large-scale pretrained visual models)

Spatial Scaling Laws -- Michaelis-Menten Curve

The success rate and the data volume show the Michaelis-Menten kinetic curve:

when the success rate is low, increasing the data volume can significantly improve the success rate;
after the success rate reaches 80%-90%, even if the data volume continues to increase, the success rate of the imitation learning policy tends to be saturated, and the increase is slow;
when the data volume tends to be infinite, the success rate tends to be 100%.

The relationship between the success rate and the data volume is expressed by the formula: success_rate = 100% * D / (K_m + D), where D is the data volume of imitation policy, K_m is the data volume required to achieve 50% success rate.

Power-law Relationship Between Spatial Volume and Data volume needed

The data volume required for spatial generalization is related to the spatial volume in an power-law relationship, i.e., more data significantly improves the ability to generalize over a larger spatial range.

For example, in our settings, if we want to generalize to x times the spatial volume, then the data volume needs to be expanded by about x^0.35 times.
In the setting in the paper, 34400cm^3 versus 1cm^3, the data volume required for spatial generalization of the former is 34400^0.35=38 times that of the latter.

Spatial Generalization, Object Generalization, and Background Generalization

BibTeX

If you find our work helpful, please cite us:

@article{tan2024manibox,
      title={ManiBox: Enhancing Spatial Grasping Generalization via Scalable Simulation Data Generation},
      author={Tan, Hengkai and Xu, Xuezhou and Ying, Chengyang and Mao, Xinyi and Liu, Songming and Zhang, Xingxing and Su, Hang and Zhu, Jun},
      journal={arXiv preprint arXiv:2411.01850},
      year={2024}
    }

Thank you!

ManiBox: Enhancing Spatial Grasping Generalization via Scalable Simulation Data Generation

Abstract

Spatial Generalization

Background Generalization

Object Generalization

Get objects from mid-air, from people's hands, and cluttered env.