Omni Models and the Collapse of Modality Boundaries
Three weeks after Gemini Omni and one month after Nemotron 3 Nano Omni, the modality stack has quietly folded into a single architecture. The interesting question is no longer whether unified models work — it is what happens to the specialist video, audio, and vision stacks built on the assumption that they wouldn't.
The pattern that emerged this spring is starting to look load-bearing. Google's Gemini Omni debuted at I/O on May 19 as a single architecture that ingests and emits text, images, audio, and video. NVIDIA shipped Nemotron 3 Nano Omni in late April with the same pitch at edge scale — 30B total, 3B active, vision and audio encoders welded to a Mamba-Transformer MoE backbone. Alibaba's Qwen3.5 Omni dropped in March. By the first week of June, every frontier lab and a meaningful chunk of the open-weight ecosystem has shipped an "omni" SKU. The branding is the tell: vendors stopped marketing modality count and started marketing modality unification.
That distinction matters more than the spec sheets suggest. The previous generation of "multimodal" models was almost always a language model with a vision encoder bolted on, sometimes an audio one, occasionally a video tokenizer behind a feature flag. Inference paths were separate, attention was sliced, and most of the cross-modal reasoning happened in a thin projection layer. Omni architectures change the shape of the cross-attention itself — modalities share the same token space and the same residual stream, which is why Nemotron's 9x throughput claim over comparable open omni models is plausible rather than benchmark theater. You only get that number when you stop paying the cost of three separate forward passes.
The commercial consequence is already showing up in the video-generation tier. Kling v3 leads the arena leaderboard at 2127, but its differentiator is the Omni variant — native lip-sync in five languages, multi-shot storyboards, audio and image and editing all in one architecture. Seedance 2.0 and Veo 3.1 made the same architectural bet earlier this year. The specialist text-to-video stack that dominated 2025 is being absorbed into general-purpose omni models on one side and squeezed by purpose-built unified video models on the other. Sora 2 was deprecated April 26 and shuts down September 24 — OpenAI exited the consumer video market entirely, which reads less like a strategic retreat and more like an acknowledgment that the specialist play was about to lose its moat.
Edge deployment is the other shoe. Nemotron 3 Nano Omni at 3B active parameters is small enough to run on a single workstation GPU, and the open weights plus open training data — Palantir, Foxconn, and Dell were named as launch adopters — make it the first credible omni model that doesn't require a hyperscaler in the loop. Gemini Omni Flash rolling into YouTube Shorts and YouTube Create at no cost is the other end of the same vector: omni capability as a consumer commodity, not a premium tier. Between those two release patterns, the assumption that multimodal reasoning would stay scarce is no longer operative.
The interesting downstream question is what happens to the application layer that was built on the old separation. Video editors that orchestrate a speech-to-text model, a vision model, and a generation model through a workflow graph are now competing with a single omni call that does the whole pipeline in one forward pass — with shared context across all modalities, which is the part the workflow graph never quite managed. The same logic applies to robotics stacks, accessibility tools, and the entire category of "AI agent" frameworks whose value proposition was largely about gluing modalities together.
None of this means the specialist models go away tomorrow. Kling still wins on motion quality and Veo still wins on cinematic color grading, and those gaps will persist for as long as the specialists can out-iterate the omni teams on a narrow axis. But the architectural direction is set: by the next product cycle, the default assumption will be that any frontier model handles every modality, and the burden of proof will shift to anyone shipping a single-modality system. The interesting work moves to what you build on top of that assumption, not under it.
Google — Introducing Gemini Omni → · NVIDIA — Nemotron 3 Nano Omni Powers Multimodal Agent Reasoning in a Single Efficient Open Model → · TechCrunch — Google's Gemini Omni turns images, audio, and text into video — and that's just the start →