Touch-Dreaming pairs tactile sensing with whole-body teleoperation in a single humanoid policy
A multimodal policy called Humanoid Transformer with Touch Dreaming combines VR whole-body teleoperation, a reinforcement-learned lower-body controller, dexterous hand retargeting and distributed tactile sensing — folding balance, manipulation and touch into one learned system.
Humanoid control has historically been assembled from separate subsystems: one controller for balance, another for arms, a third for hands, stitched together by engineers. A single policy that spans locomotion, manipulation and tactile feedback is the architectural consolidation the field has been working toward, and it is what lets a robot use its whole body for one task.
Distributed touch is the underrated ingredient. Vision tells a robot where an object is; touch tells it whether the grasp is slipping, how much force to apply, when contact has been made. Most manipulation failures are contact failures, and they are invisible to cameras — which is why tactile data is the missing modality rather than a refinement.
VR teleoperation supplies the training signal. Having humans puppet the robot through tasks generates exactly the demonstration data these policies need, and it means every hour of remote operation is also an hour of data collection. That flywheel — teleoperate, learn, autonomise — is how the deployed fleets now scaling will pay for themselves twice.
IEEE Spectrum — Videos: physical AI robotics, robot hands and more → · NVIDIA — National Robotics Week — latest physical AI research and breakthroughs →