Hybrid training: mixing web-scale vision-language data with scarce robot data
Hybrid training for vision-language-action models addresses the field's structural imbalance directly. There is effectively unlimited image-text data and very little paired robot data, and the question is how to combine them without the abundant data drowning the scarce.
The naive approach fails in a predictable way. Train on both and the vast web corpus dominates the gradient, producing a model with excellent visual understanding and no useful motor policy. The mixing ratio is not a hyperparameter detail, it is the whole method.
What makes it tractable is that the two data types teach different things. Web data supplies perception and semantics; robot data supplies dynamics and control. A training scheme that routes each to the parameters it should shape is doing something more structured than blending.
It is the same shape as the trade running through the whole field: how much capacity goes to the body and how much to the task.
arXiv — Hybrid training for vision-language-action models → · arXiv — Grounded world model for semantically generalizable planning → · arXiv — Gaze2Act: gaze-conditioned vision-language-action policies for interactive robot manipulation →