Sim-to-Real Transfer for Robotics Applications

🚀 New paper from our lab: Sym2Real: a data-efficient way to train adaptive robot controllers. With only ~10 trajectories (80 seconds!!!) in total (most from a low-fidelity and untuned sim + 2–3 real runs), we achieve robust real-world control of a palm-sized drone and a 1/10 racing car. No expert priors on dynamics or heavy sim tuning needed. 📹 The video below shows the entire 80-second demo video, from the beginning to flying. The key intuition is to capture the shared core physics in a simplified setting with concrete equations (in our case, differential equations!), then adapt with a lightweight residual from just a handful of real-world samples. This continues our line of work on robot self-models for resilient behaviors: - Visual self-modeling of full bodies (Science Robotics 2022) - Self-modeling animatronic face control (ICRA 2021, Science Robotics 2024) 📄 Read our preprint: https://lnkd.in/eZrb6i3d 🎥 Video: https://lnkd.in/e3dMUs3J 🔬 Code + data: https://lnkd.in/e24ayhJ2 Led by our amazing Easop Lee at the General Robotics Lab at Duke University!

Imagine training a robot dog to balance and walk on a yoga ball - entirely in a virtual environment - without any fine-tuning when moving to the real world. That’s precisely what’s been achieved with DrEureka, an AI agent that writes its own code to teach robots new skills in simulation and then seamlessly transfers them to physical hardware. Why is this a big deal? - Zero Shot Transfer: No additional tweaking is needed once the skill is learned in simulation. The robot can just do it in real life. - Complex Physics: A bouncy yoga ball is notoriously hard to simulate accurately. DrEureka navigates this challenge by exploring a large range of simulation settings to find what works best in reality. - Adaptability: The same approach can be extended to different terrains and tasks - like walking sideways on unstable surfaces! DrEureka builds on earlier work that taught a robotic hand to spin a pen through pure simulation. The implications go beyond fun demonstrations. Picture these capabilities in rescue operations or medical emergencies - robotic helpers balancing and navigating tricky terrain to deliver essential supplies or assist first responders. What do you think? Could we see agile, AI-driven robots supporting firefighters and medics in disaster zones? #innovation #technology #future #management #startups

Atlas the humanoid is now learning to assemble other robots autonomousy. The leading Massachusetts-based robotics firm Boston Dynamics just shared new demo footage to show the process its world-famous humanoid robot, Atlas, is making learning tasks autonomously. In the video, Atlas handles and sorts parts for the leading Massachusetts robotics firm’s other famous robot, the quadruped Spot, despite an engineer’s incessant trolling. It hints at a future where AI-powered robots like Atlas assemble other robots without human oversight. Boston Dynamics says the autonomous behavior in the demo is thanks to an end-to-end neural network it’s developing with the Toyota Research Institute (TRI). They’re building on the work TRI began back in 2023, integrating its so-called Large Behavior Model (LBM) so Atlas can respond to natural language prompts with autonomous robotic actions. According to Boston Dynamics, Atlas learns through human teleoperation data. Instead of learning one task at a time, the robot trains on many of them together. The company says this allows for generalization, meaning the robot can handle new situations without extra programming. Atlas practices in the real world and within high-fidelity simulations powered by NVIDIA’s digital twin technology. In simulation, Boston Dynamics says it can double or triple the speed it takes to learn tasks vs. real life. However, the transfer of the skills learned in simulation, a process called Sim2Real (Simulation to Reality), is never seamless. Boston Dynamics has built a hybrid training loop to make transitions from sim to real as painless as possible. The robot’s AI brain is anchored in real-world physics since human demonstrations serve as the foundation of the LBM. Digital twins replicate all 78 DoF of Atlas and simulate how each of its joints and motors work, down to tiny forces and micro movements. Within NVIDIA’s Omniverse, thousands of training iterations are done simultaneously. After simulation training, the physical robot runs through real-world evaluation tasks. Failures are logged, corrections are made via teleoperation, and those adjustments get back into the larger dataset. This creates what’s known as a data flywheel, where each failure teaches the AI model to make the next deployment smoother. Atlas, which began as a DARPA project in 2012, is getting its learn on as it prepares for its first real job for its parent company, Hyundai. According to reports, Atlas robots have spent much of 2025 training for imminent deployment at the South Korean automaker’s US manufacturing facilities.

🤖💼 Reinforcement Learning for Humanoid Robots (Project: Playing Atari using Deep Reinforcement Learning - https://lnkd.in/e6Rpf2ET) 🔹 Humanoid-Gym: Zero-Shot Sim2Real Transfer 🔄 Framework: Engineered on Nvidia Isaac Gym, Humanoid-Gym specializes in training humanoid robots. 🏃 Skills Training: Its focus extends to honing locomotion skills, vital for real-world navigation and interaction. 🌐 Zero-Shot Transfer: Uniquely emphasizing zero-shot transfer, Humanoid-Gym seamlessly transitions trained policies from simulation to reality without the need for additional fine-tuning. 🛠️ Sim-to-Sim Integration: Through its integration of sim-to-sim capabilities from Isaac Gym to Mujoco, the framework offers a robust platform for verifying trained policies across a spectrum of physical simulations, ensuring adaptability and generalization. 🤖 Real-World Validation: Rigorously tested with RobotEra's humanoid robots, including XBot-S and XBot-L, Humanoid-Gym demonstrates its efficacy and reliability in real-world environments, showcasing its capability for zero-shot sim-to-real transfer. 🌍 Resources: Explore comprehensive documentation, tutorials, and access the framework's source code at the project website: https://lnkd.in/eZnjDCZG GitHub repository: https://lnkd.in/eJpta--u Paper: https://lnkd.in/ePCeVKsY 🔹 Facilitating Sim-to-Real Transfers Humanoid-Gym's innovative methodology facilitates zero-shot transfer for humanoid robots, ensuring a seamless transition from simulation to reality. By closely aligning simulated dynamics with real-world performance, researchers can confidently validate training policies through sim-to-sim simulations, significantly enhancing the likelihood of successful sim-to-real transfers. This advancement marks a significant leap forward in the development of humanoid robotics, promising greater efficiency and efficacy in real-world applications.

How can we bridge the gap between simulation and reality in robotics? Developed by a team from UC Berkeley, Google DeepMind, and other leading institutions, MuJoCo Playground is a fully open-source framework revolutionizing robotic learning and deployment. This tool enables rapid simulation, training, and 𝘇𝗲𝗿𝗼-𝘀𝗵𝗼𝘁 𝘀𝗶𝗺-𝘁𝗼-𝗿𝗲𝗮𝗹 𝘁𝗿𝗮𝗻𝘀𝗳𝗲𝗿 across diverse robotic platforms. MuJoCo Playground supports quadrupeds, humanoids, dexterous hands, and robotic arms, train reinforcement learning policies in minutes on a single GPU, and streamline vision-based and state-based policy training with integrated batch rendering and a powerful physics engine. The framework’s real-world success is evidenced by its deployment on platforms like Unitree Go1, LEAP hand, and the Franka arm within 8 weeks. Its efficiency and simplicity empower researchers to focus on innovation. A simple 'pip install playground' will do! Congratulations to the team, Kevin Zakka, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Carmelo (Carlo) Sferrazza, Yuval Tassa, Pieter Abbeel and collaborators, for this game-changing contribution to robotics! 🔗 Check out their website here https://lnkd.in/g7mbZtXg for their paper, github, live demo, and even a google colab setup for an easy start! 💬 What do you think is the next big challenge for sim-to-real transfer in robotics? Let's discuss below! P.S. Excited to share an open source framework I've been experimenting with recently! #Robotics #AI #Simulation #MachineLearning #Engineering #Innovation #ReinforcementLearning 

From Isaac Sim to Isaac Lab – Building an Intelligent Robot Starting without hardware access, I built a full simulation and control pipeline using NVIDIA Isaac Sim and MoveIt with ROS2. In my first blog, I walk through setting up LeRobot using the API standalone method for simulation and planning: 🔗 Setting Up LeRobot in Isaac Sim https://lnkd.in/g9agNFsQ But I didn’t stop there. My goal was intelligent behavior — not just moving, but understanding what and when to pick up. I moved to Isaac Lab and trained a robot arm to pick up a cube using reinforcement learning. This approach turned the robot into a smart agent — learning just like a student being taught. 🔗 Training Robot Arm with Isaac Lab https://lnkd.in/gnpgJBte Now that Sim-to-Sim is complete, I’m preparing for deployment on real hardware. The final vision? A robot that can see, understand, and serve — even handing over a plate of fruits 🍎🍇 🧠 Simulation → Learning → Real-World Execution This is what AI-powered robotics is all about. Ninad Madhab Dustin Franklin Jigar Halani Sunil Patel Pramod KP #IsaacSim #IsaacLab #MoveIt #ROS2 #SimToReal #ReinforcementLearning #NVIDIAJetson #LeRobot #RoboticArm #Automation #RoboticsResearch #MediumBlog #TechJourney