Open-source multimodal LLMs stopped pretending video is just a pile of images with a transcript stapled on. Qwen3-Omni is the proof: a 30B-A3B open-weight checkpoint, mixture-of-experts, that takes text, image, audio. And video in and gives you text plus realtime speech out the other side. Its own team calls it a “natively end-to-end multilingual omni-modal foundation” model (GitHub).
That quote is the whole story in four words.
Natively. End-to-end.
It is not alone. VITA, Baichuan-Omni, Mini-Omni2, and InteractiveOmni all ship open releases spanning video, image, text, and audio together, and MiniCPM-V 2.6 covers image and video understanding at 8B parameters. For a developer, no single model is the news here.
The plumbing changed shape.
The Open-Source Multimodal LLM Lineup: 30B Down to 4B
Table first, commentary after. Only what the sources actually state.
| Model | Parameters | Input | Standout trait |
|—|—|—|—|
| Qwen3-Omni | 30B-A3B MoE | text, image, audio, video | text plus realtime speech out |
| VITA | count not stated in the sources cited here | video, image, text, audio | simultaneous omni-modal processing |
| Baichuan-Omni | count not stated in the sources cited here | image, video, audio, text | concurrent modal processing |
| Mini-Omni2 | count not stated in the sources cited here | vision, speech, text | auditory interruption mechanism |
| InteractiveOmni | 4B to 8B | audio-visual multi-turn | lightweight, speech generation |
| MiniCPM-V 2.6 | 8B (SigLip-400M + Qwen2-7B) | single image, multi-image, video | reported to outperform GPT-4V |
Those three blank-ish cells are not me being lazy.
The papers and readmes referenced here describe capability in loving detail and go quiet on parameter counts, which itself tells you what each release is competing on.
Qwen3-Omni anchors the top of the table. One roundup describes it as “the most capable open-weight omni-modal model,” swallowing text, image, audio. And video and producing text plus realtime speech from a single 30B-A3B mixture-of-experts checkpoint (ertas.ai). The MoE half of that label does quiet work: 3B active parameters inside a 30B-class model is what makes an omni checkpoint thinkable outside a hyperscaler’s cluster. In principle, anyway. I have not loaded this one onto a box myself yet. This piece comes from papers and readmes, not terminal output. And I would rather say that than fake a deployment story.
InteractiveOmni sits at the other end, targeting “audio-visual multi-turn interaction, ranging from 4B to 8B parameters,” explicitly aiming to lead lightweight omni-modal models with understanding and speech generation (arXiv). That 4B-to-8B band is where a solo builder should look first. Reasons below.
MiniCPM-V 2.6 earns its own paragraph by solving a narrower job well. Built on SigLip-400M and Qwen2-7B, 8 billion parameters total. And reported to “outperform GPT-4V on single image, multi-image and video understanding” (daily.dev). Treat the GPT-4V line as a single-source report, not settled truth. But notice the scope: image and video understanding, no audio anywhere. If your workload is screenshots, scanned docs, and screen recordings, you may not need an omni model at all.
Four “Firsts” at Once Should Make You Suspicious
I read the source papers back to back recently — four PDFs in one sitting, on a Tuesday night that had better things coming to it.
And a pattern jumped out.
Every one of these teams claims to be first.
VITA’s paper introduces itself as “the first-ever open-source Multimodal Large Language Model (MLLM) adept at simultaneous processing and analysis of Video, Image, Text.
And Audio modalities” (arXiv) — that one carries arXiv number 2408.05211, if you track papers by ID the way some of us track train times.
Baichuan-Omni’s release calls itself “the first high-performing open-source Multimodal Large Language Model” for concurrently processing image, video, audio, and text (GitHub).
Mini-Omni2 claims to be “the first open-source large multi-modal model with capabilities in vision, speech, text.
And an auditory interruption mechanism” (arXiv). Whoever decided every release needs “Omni” in the name has quietly wrecked GitHub search for the rest of us.
Four simultaneous firsts at roughly the same capability set. “First” is a scoping game, not a spec. Each claim is technically true inside its own carefully chosen slice. And none of them tells you which checkpoint to run on Monday.
So ignore the firsts and grade on three things: what the model weighs, whether the weights are genuinely downloadable under open terms.
And whether you can reproduce the training data. That third axis is where Mini-Omni2 stands alone in a way almost nobody talks about. Its authors state the datasets used are all publicly available and that “all generated data, along with the generated scripts (semantic interruption data), will be open-sourced” (arXiv). Downloadable weights are a gift. A downloadable data recipe is how a field actually moves.
How Open-Source Multimodal LLMs Collapse Your Stack
Old multimodal pipeline was a relay race. Transcription model turned audio into text. Frame sampler turned video into images. Vision model captioned the images. Text LLM reasoned over the pile. TTS service read the answer back aloud. Five components, five APIs, five places for latency to pile up, and one blurred handoff between each leg.
Omni models delete the handoffs.
Qwen3-Omni streams responses in both text and natural speech in real time (GitHub). Mini-Omni2 pushes further with what its authors call an auditory interruption mechanism: a visual-audio assistant giving real-time, end-to-end voice responses to user video and voice queries. And one you can interrupt by voice (arXiv). The interruption detail sounds small. It is not.
Barge-in behavior is the difference between a demo and something a person will tolerate on a support line.
Now the honest tradeoff, because consolidation cuts both ways.
When one checkpoint does everything, one checkpoint’s weakness poisons everything. If your stitched-together transcriber mishears a regional accent today, you swap that component and move on. The same mishearing inside an omni model is baked into the reasoning layer, and you cannot hot-swap it. You are trading integration risk for model risk. For most small teams the trade still wins.
But walk in knowing which kind of failure you just signed up for.
Why a One-Person Shop Cares
I run a one-person automation agency. The number that decides whether a project ships is not a benchmark score. It is integration count. Every seam between two models is code you write, monitor. And eventually debug at 11pm, and I have done enough of that debugging to hold grudges.
A model that takes video with audio in one end and pushes speech out the other removes seams. Removed seams are the cheapest engineering you will ever do.
Second reason is data gravity. Client call recordings, video intake, support screenshares — plenty of that material legally or contractually cannot take a round trip through someone else’s API. Open weights keep the bytes on hardware you control. That used to require a closed-vendor enterprise contract. Now InteractiveOmni’s 4B-to-8B range puts a genuine audio-visual, multi-turn model in territory a solo operator can realistically self-host and experiment with (arXiv). And Qwen3-Omni’s open terms put the full stack in reach when you need speech out (ertas.ai).
Concrete move for this month: find the single seam in your current pipeline where a transcription tool hands off to a language model, or where a vision model hands off to a reasoning model.
Price what that seam cost you in glue code, latency, and failure modes over the last quarter. Then test one open omni model against that seam alone, with ten real examples from your own data, before committing to anything. The winners of this shift are not the labs with the loudest “first” — they are the builders who noticed the glue layer became optional.
FAQ: Picking and Running an Open-Source Multimodal LLM
Which open-source multimodal LLM should I self-host first?
Start at the small end.
InteractiveOmni’s 4B-to-8B band is the realistic entry point for audio-visual, multi-turn work on hardware one person actually owns (arXiv). If your workload is image and video understanding with no audio in it, MiniCPM-V 2.6 at 8B is the tighter fit and the smaller job.
Qwen3-Omni or MiniCPM-V 2.6?
Different jobs, not rival jobs.
Qwen3-Omni is a 30B-A3B MoE that takes text, image, audio.
And video in and produces text plus realtime speech out (GitHub).
MiniCPM-V 2.6 is 8B (SigLip-400M + Qwen2-7B), image and video understanding only, reported to outperform GPT-4V on single image, multi-image and video understanding (daily.dev). Need speech out? Qwen3-Omni. Need image and video only, on modest hardware? MiniCPM-V 2.6.
What hardware do I need?
Straight answer: none of the sources cited here publish GPU requirements. So I cannot hand you a VRAM number without inventing it.
What the sources do support is directional. The MoE design behind 30B-A3B is what makes a 30B-class omni model thinkable outside a hyperscaler’s cluster. And the 4B-to-8B band is explicitly framed as lightweight. Size your own machine against the model card before you promise a client anything.
What are the licensing terms?
Varies by release, and “open” is doing heavy lifting in every one of these announcements. The sources here describe open weights and open terms without spelling out full license text (GitHub, ertas.ai). The clearest commitment in the whole set comes from Mini-Omni2’s authors: datasets publicly available, with “all generated data, along with the generated scripts (semantic interruption data), will be open-sourced” (arXiv). Read the actual license file in each repo before commercial use. A README is not a lawyer, and neither am I.
Do any of them publish the training data?
One does, on paper at least. Mini-Omni2 again — datasets publicly available, generated data and scripts committed to open-sourcing (arXiv). The rest describe capability, not recipe. If reproducibility matters more to you than a benchmark row, that single fact should reorder the whole table.
Mediascout builds exactly these pipelines for small teams.
If you want a second pair of eyes on which seam to collapse first, get in touch and we will walk your stack together, pick the one seam worth testing this month. And set up the ten-example trial before you spend a single dollar on hardware.
