The perception encoder is the quiet half of every multimodal model
A 30B language model paired with a ~1.8B ViT-G/14 image encoder. The encoder is 6% of the parameters and decides everything the model is capable of seeing.
Muse Glimmer pairs a 30B dense transformer with a roughly 1.8B ViT-G/14 perception encoder for image input — about 6% of total parameters devoted to turning pixels into something the language model can reason over.
That small component sets a hard ceiling. Whatever the encoder fails to represent is not recoverable downstream: if fine text, small objects or spatial relationships are lost at the encoding step, no amount of reasoning capacity in the language model reconstructs them. The transformer is reasoning over a summary it did not produce and cannot inspect.
It explains a failure pattern anyone using these models has hit — a model that discusses an image fluently and confidently while getting a specific detail wrong. That is rarely a reasoning failure. It is the encoder having discarded the detail, and the language model doing exactly what it should with what it received.
Encoder architecture and resolution get a fraction of the attention that parameter counts and context windows receive, despite determining what the system can perceive at all. Benchmark reporting compounds this: multimodal scores are attributed to the model as a whole, so an encoder limitation reads as a reasoning limitation.
The practical advice for anyone evaluating a multimodal model is to test perception separately from reasoning — ask it to transcribe and locate before asking it to interpret. If it cannot read the label, its analysis of the label is fiction. That distinction matters more as models take more modalities as input at once.
BentoML — Multimodal AI: A Guide to Open-Source Vision Language Models → · Roboflow — Best 76 Multimodal Vision Models →