Tokenizing the body

Between a language model and a physical arm sits a compression problem, and how you solve it decides how much of the model's capacity gets spent on the body instead of the task.

Vision-language-action models must turn continuous joint commands into discrete tokens a transformer can predict.

The naive version fails predictably

Bin every joint finely and sequence length explodes, consuming context that should hold the task. Bin coarsely and the motion is jerky and unusable. Action tokenization has been solved badly for most of the field's history, largely because it looked like a detail.

It is also where cross-embodiment transfer lives or dies. A tokenizer tied to a specific joint configuration produces a model that cannot move to another robot. One that captures motion primitives rather than raw angles has a chance.

The parallel to byte-pair encoding is close enough to be instructive. Tokenization was never the interesting part of language modelling and it quietly determined a great deal about what those models could represent efficiently.

The data imbalance underneath

There is effectively unlimited image-text data and very little paired robot data. Train on both naively and the web corpus dominates the gradient — excellent visual understanding, useless motor policy. The mixing ratio is the method, not a hyperparameter.

What the field is now asking

How to find where a policy fails before deploying it, using boundary-focused sampling rather than average-case testing.

That shift matters. A policy succeeding 95% of the time is not 95% safe — it is a system whose 5% needs characterising, because that is where hardware damage and injuries live. Average-case evaluation is nearly useless for anything with mass.

Serving systems, failure discovery, deployment conditions. The research questions have moved from capability ceilings to operating conditions, which is what a field looks like shortly before it becomes an industry.

arXiv — FAST: efficient action tokenization for vision-language-action models → · arXiv — ROBOGATE: adaptive failure discovery for safe robot policy deployment → · arXiv — Hybrid training for vision-language-action models →