Challenges of Robotic Manipulation

Yesterday, we explored Synthetic Interoception and how robots might gain self-awareness. Today, we shift focus to physical intelligence: how robots can achieve the touch and finesse of human hands. Rigid machines are precise but lack delicacy. Humans, on the other hand, easily manipulate fragile objects, thanks to our bodies' softness and sensitivity. Soft-body Tactile Dexterity Systems integrate soft, flexible materials with advanced tactile sensing, granting robots the ability to: ⭐ Adapt to Object Shapes: Conform to and securely grasp items of diverse forms. ⭐ Handle Fragile Items: Apply appropriate force to prevent damage. ⭐ Perform Complex Manipulations: Execute tasks requiring nuanced movements and adjustments. Robots can achieve a new level of dexterity by emulating the compliance and sensory feedback of human skin and muscles. 🤖 Caregiver: A soft-handed robot supports elderly individuals and handles personal items with gentle precision. 🤖 Harvester: A robot picks ripe tomatoes without bruising them in a greenhouse, using tactile sensing to gauge ripeness. 🤖 Surgical Assistant: In the OR, a robot holds tissues delicately with soft instruments, improving access and reducing trauma. These are some recent relevant research papers on the topic: 📚 Soft Robotic Hand with Tactile Palm-Finger Coordination (Nature Communications, 2025): https://lnkd.in/g_XRnGGa 📚 Bi-Touch: Bimanual Tactile Manipulation (arXiv, 2023): https://lnkd.in/gbJSpSDu 📚 GelSight EndoFlex Hand (arXiv, 2023): https://lnkd.in/g-JTUd2b These are some examples of translating research into real-world applications: 🚀 Figure AI: Their Helix system enables humanoid robots to perform complex tasks using natural language commands and real-time visual processing. https://lnkd.in/gj6_N3MN 🚀 Shadow Robot Company: Developers of the Shadow Dexterous Hand, a robotic hand that mimics the human hand's size and movement, featuring advanced tactile sensing for precise manipulation. https://lnkd.in/gbpmdMG4 🚀 Toyota Research Institute's Punyo: Introduced 'Punyo,' a soft robot with air-filled 'bubbles' providing compliance and tactile sensing, combining traditional robotic precision with soft robotics' adaptability. https://lnkd.in/gyedaK65 The journey toward widespread adoption is progressing: 1–3 years: Implementation in controlled environments like manufacturing and assembly lines, where repetitive tasks are structured. 4–6 years: Expansion into dynamic healthcare and domestic assistance settings requiring advanced adaptability and safety measures. Robots are poised to perform tasks with unprecedented dexterity and sensitivity by integrating soft materials and tactile sensing, bringing us closer to seamless human-robot collaboration. Next up: Cognitive World Modeling for Autonomous Agents.

Tesla’s pause on Optimus robots isn’t just a production hiccup. It’s a reminder that robotics hype always hits reality. And Elon Musk provides us with yet another case study in over-promising and under-delivering, as evidenced by his premature scaling, in an attempt to shore up an overextended stock valuation. --- 🚨 WHAT JUST HAPPENED Tesla has temporarily halted mass production of its Optimus humanoid robot. The reason? Persistent, predictable engineering challenges, especially in the hands and forearms. Reports cite: ⚙️ Motors overheating ✋ Weak grip strength 🔩 Joint failures In other words, the exact systems that make real-world dexterity possible. Tesla has reportedly stockpiled hundreds of incomplete robot bodies, now waiting on a fundamental redesign. This isn’t a delay. It’s a reset. Because human-like hands are among the most complex mechanical systems ever attempted: high-precision, thermally constrained, and brutally unforgiving at scale. They also represent 25% of the cost of a humanoid robot and 50% of the engineering challenge. --- 💡 KEY COMPETITORS The complexity of human-level grasping and manipulation remains the single greatest bottleneck in general-purpose robotics. But there are some making strides in this field: - Boston Dynamics' Atlas - The OG industrial humanoid - Figure AI - It's new Figure 03 bot impresses - Unitree Robotics - Its Dex5-1 end effector continues progress --- 🧠 THE BIGGER PICTURE We’ve seen this pattern before. Every industrial transition hits a “last mile” problem – from the carburetor in combustion engines to early PC storage drives. In robotics, we’ve largely solved locomotion (walking) and vision (AI perception). But the final frontier is dexterous, reliable manipulation (the hands). They’re the primary interface between robots and the infinite variability of the human world. Without hands, you don't have a useful general-purpose robot. Companies obsessed with form before function will keep hitting this wall. And success won’t be measured by how well a robot pretends to know Kung Fu – but by its Mean Time Between Failure (MTBF) on a factory floor. --- THE QUESTION If general-purpose robotic hands remain too costly or unreliable, will humanoid robots just stay expensive toys? #robotics #Tesla #optimus #humanoid #tron

Robotic AI’s reliance on vision is limiting its ability to interact with the physical world accurately. Vision systems dominate robotic AI because they’re cost-effective and can collect massive datasets. But this overemphasis on vision overlooks the critical role of force sensing—providing tactile data that vision simply can’t replicate. Without it, robots are limited to estimating force feedback from visuals, leading to inefficiencies in delicate tasks like assembly, gripping, or threading. As Edward Adelson, professor at Massachusetts Institute of Technology, explained in his TED Talk, “Force feedback allows robots to perform tactile tasks that vision alone cannot achieve—like folding a towel or threading a cable—by feeling their way through interactions, just as humans do.” Adelson’s work on GelSight technology highlights how tactile sensing can unlock superhuman precision for robots, enabling them to understand their environment through touch. The challenge? Force sensors are an added cost, generate less data, and are harder to integrate. But they offer essential benefits: • Reliability and Safety: For tasks where mistakes aren’t an option, force feedback provides the assurance vision alone cannot. • Deeper Learning: Force sensing enriches AI by adding layers of contact-based data for more robust decision-making. • Expanding Applications: From industrial automation to medical robotics, tactile data opens doors to tasks beyond vision’s reach. ATI Industrial Automation supports robotics through robust, precise robotic force sensors—helping to bring accuracy to robotic AI data collection. Edward Adelson’s TED Talk: https://lnkd.in/epeCvwqj #robotics

Just collecting manipulation data isn’t enough for robots - they need to be able to move around in the world, which has a whole different set of challenges from pure manipulation. And bringing navigation and manipulation together in a single framework is even more challenging. Enter HERMES, from Zhecheng Yuan and Tianming Wei. This is a four-stage process in which human videos are used to set up an RL sim-to-real training pipeline in order to overcome differences between robot and human kinematics, and used together with a navigation foundation model to move around in a variety of environments. To learn more, join us as Zhecheng Yuan and Tianming Wei tell us about how they built their system to perform mobile dexterous manipulation from human videos in a variety of environments. Watch Episode #45 of RoboPapers today, hosted by Michael Cho and Chris Paxton! Abstract: Leveraging human motion data to impart robots with versatile manipulation skills has emerged as a promising paradigm in robotic manipulation. Nevertheless, translating multi-source human hand motions into feasible robot behaviors remains challenging, particularly for robots equipped with multi-fingered dexterous hands characterized by complex, high-dimensional action spaces. Moreover, existing approaches often struggle to produce policies capable of adapting to diverse environmental conditions. In this paper, we introduce HERMES, a human-to-robot learning framework for mobile bimanual dexterous manipulation. First, HERMES formulates a unified reinforcement learning approach capable of seamlessly transforming heterogeneous human hand motions from multiple sources into physically plausible robotic behaviors. Subsequently, to mitigate the sim2real gap, we devise an end-to-end, depth image-based sim2real transfer method for improved generalization to real-world scenarios. Furthermore, to enable autonomous operation in varied and unstructured environments, we augment the navigation foundation model with a closed-loop Perspective-n-Point (PnP) localization mechanism, ensuring precise alignment of visual goals and effectively bridging autonomous navigation and dexterous manipulation. Extensive experimental results demonstrate that HERMES consistently exhibits generalizable behaviors across diverse, in-the-wild scenarios, successfully performing numerous complex mobile bimanual dexterous manipulation tasks Project Page: https://lnkd.in/e-aEbQzn ArXiV: https://lnkd.in/eemU6Pwa Watch/listen: Youtube: https://lnkd.in/erzbkYjz Substack: https://lnkd.in/e3ea76Q8

Humans use many different skills to handle objects. Fine motor skills, like picking up a coin, rely on careful finger movements. Gross motor skills, such as lifting or pushing something heavy, involve the whole body. In these larger tasks, the skin and muscles help by spreading out contact, adding stability, and reducing mistakes. For robots, learning these full-body, contact-rich skills is much harder. The number of possible contact points quickly becomes overwhelming, making it impossible to calculate each interaction directly. To address this, researchers used a method called example-guided reinforcement learning, which teaches a robot to copy skills from just one demonstration. The method was tested on Toyota Research Institute’s Punyo robot, a humanoid torso with soft, pressure-sensing skin. Training was done in simulation, but the learned skills transferred smoothly to the real robot thanks to domain randomization and Punyo’s flexible design. With this setup, the robot was able to manipulate objects such as water jugs and large boxes in ways similar to the example. The research team also showed that the robot could perform “blind” full-body manipulation—using only touch and body feedback, without knowing the exact position of the object. The results highlight the importance of compliant, flexible skin in helping humanoid robots perform whole-body manipulation successfully. Read the research here: https://lnkd.in/eRqK8DF5

It is difficult to get robots to be both precise and general. We just released a new technique for precise manipulation that achieves millimeter-level precision while being robust to large visual variations. The key is a careful combination of visuo-tactile learning and RL. The insight here is: vision and tactile are complementary. Vision is good at spatial, semantic cues, while touch excels at local contact feedback. ViTaL is a recipe to combine the two to enable precise control at >90% success rates even in unseen environments. For the full paper, videos and open-sourced code: https://lnkd.in/eAfhz8sE This work was led by Zifan Zhao & Raunaq Bhirangi, and a collaboration with Siddhant Haldar & Jinda Cui.

Humanoids can now jog on rough terrain and even pull off backflips. From the outside, it feels like a sci-fi dream inching closer. But when I spoke with Ken Goldberg (UC Berkeley, co-founder of Ambi Robotics), he laid out why we’re still a long way from humanoids showing up in everyday life. The roadblocks aren’t small: 1 - Manipulation. Robots still can’t grasp and control objects with human-level dexterity. Jobs that involve human dexterity (construction, repairs, electrician work) are too complex and nuanced for them. For humanoids, the challenge compounds: every limb is packed with motors, so even a knee adjustment can throw off hand precision. 2- Safety. Those slick demo videos don’t show what happens when a humanoid topples over. They’re inherently unstable, far heavier than humans, and can cause real harm if they fall or misjudge their strength. 3 - Cost. Even with progress in lowering component costs, we’re still talking upwards of $60k per robot (all for a machine that can handle only the simplest of tasks.) Ken's takeaway: "I'm very worried that the hype has gotten ahead of the reality." As he explained, when expectations get too far ahead, it can set the entire field back. “Overselling” leads to disillusionment, which risks another robotics winter. We got into this and more - from the limits of humanoids to the promise of logistics robots - in the latest episode of AI in the Real World: https://lnkd.in/gGk8rwnx