12 papers with open-source code, published in 2024.
Karl Pertsch, Oliver Groth, Jonas Frey et al. · arXiv preprint · Oct 2024
π0 is a 3.5B-parameter VLA flow model from Physical Intelligence that achieves state-of-the-art general robot manipulation by mixing online RL with high-quality human demonstrations, available as an open-source PyTorch implementation.
Alexander Khazatsky, Karl Pertsch, Suraj Nair et al. · RSS 2024 · Jul 2024
A 350-hour dataset of diverse real-world robot manipulation across 22 robots in 71 scenes, designed to train scalable and generalist imitation learning policies.
Dibya Ghosh, Homer Walke, Karl Pertsch et al. · RSS 2024 · Jul 2024
Octo is a large open-source transformer-based generalist robot policy trained on 800k trajectories, supporting language-conditioned and goal-image-conditioned control across 9 robotic platforms with efficient fine-tuning on consumer GPUs.
Xuanlin Li, Kyle Hsu, Jiayuan Gu et al. · RSS 2024 · Jul 2024
A benchmark environment for evaluating real-world robot manipulation policies in simulation using real-world video prompts, enabling zero-shot sim-to-real transfer evaluation without physical hardware.
Remi Cadene, Simon Alibert, Alexander Soare et al. · NeurIPS 2024 Workshop · Jun 2024
Hugging Face's LeRobot is an open-source PyTorch framework providing pretrained models, datasets, and training scripts for imitation and reinforcement learning on real robots, lowering the entry barrier to robot learning.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti et al. · arXiv · Jun 2024
A 7B-parameter open-source Vision-Language-Action model pre-trained on 970k real-world robot demonstrations, achieving strong generalization across robots and tasks.
Yecheng Jason Ma, William Liang, Guanzhi Wang et al. · ICLR 2024 (Oral) · May 2024
Eureka uses GPT-4 to write reward functions for RL environments, achieving human-level reward design on 29 tasks and enabling zero-shot sim-to-real transfer on Shadow Hand dexterous manipulation.
Anthony Brohan, Noah Brown, Justice Carbajal et al. · ICRA 2024 · May 2024
Google DeepMind's VLA model combining a vision-language foundation model with robot action outputs, showing emergent generalization to novel objects, backgrounds, and semantic instructions far beyond training data.
Nikita Rudin, Fabian Jenelten, Timon Homberger et al. · ICRA 2024 · May 2024
Unitree Go2 uses cross-modal vision + proprioception transformers trained in Isaac Gym to achieve agile navigation over rough terrain with minimal sim-to-real gap.
Peiqi Liu, Yat Long Lo, Ted Xiao et al. · arXiv preprint · Mar 2024
OK-Robot uses off-the-shelf VLMs (CLIP, OWL-ViT) and LLMs (GPT-4) to perform open-ended object manipulation in unseen homes without any training, achieving 58% success on real-world pick-and-place tasks.
Zachary Kousik, Shanpu Fang, Rohan Thakker et al. · arXiv · 2024
An open-source 3D-printed humanoid robot platform designed for reinforcement-learning based locomotion research with affordable hardware.
Wang J., Xiao B., Wang J., Heider P., Li H., Zhang Y. · arXiv · 2024
An Isaac-Gym-based training framework for humanoid locomotion that enables zero-shot sim-to-real transfer on XBot-L/XBot-S robots via domain randomization.
