Edge0 put a 145B-parameter model on a single consumer GPU. On November 13, 2026, the open-source project released edge0-145b and edge0-120b, latent mixture-of-experts models that, per its README, run on one NVIDIA GPU or an Apple Silicon Mac, on Linux or Windows. That is local AI infrastructure arriving at retail: no cluster, no meter, no procurement conversation.
Stack it against the rest of 2026 and the floor drops even lower. LocalAI bills itself as a free, open-source alternative to OpenAI and Claude, running on consumer-grade hardware with no GPU at all. A used RTX 3090 with 24 GB of memory sells for about $800 and, according to a 2026 GPU guide, is still the best price-per-VRAM consumer card for local LLM inference. The question stopped being whether local AI is practical and became which tier you should buy into. The rest of this post is the map, plus where I’d put my own money.
What Edge0 Actually Shipped
Edge0 describes itself as “an open-source streaming MoE inference framework for running large sparse models on consumer hardware.” The November 13 release makes that concrete: edge0-145b and edge0-120b, latent-MoE models derived from Qwen3.8-Flash-Next and, in the project’s words, “trained natively at 4 bits.”
Two details do the heavy lifting.
Mixture-of-experts means the model is sparse. So a 145B-parameter model does not compute like a dense one at inference; only part of the network fires per token. And trained natively at 4 bits means the model lives at that precision from the start instead of being squeezed down after training, which is how most local weights get small.
The interface story is deliberately boring, and that is the point. Edge0 says you use it “through the command line, a local API server, or the Edge0 Chat app.” The local API server is the one that matters if you build automation: your code talks to a machine you own instead of a hosted endpoint. And the request shape is the part you already know how to write.
The Hardware Floor Is a Used $800 Card
The 2026 consumer-GPU guide puts its bottom line up front: as of April 2026, the RTX 3090 (24 GB, roughly $800 used) remains the best price-per-VRAM consumer GPU for local LLM inference, while the RTX 5090 (32 GB GDDR7, roughly $2,000) is the only consumer card capable of running 70B models at Q5 quantization in a single slot.
A separate 2026 developer article reports 27B coding models running on a single RTX 3090 “at usable speeds.”
Read those together and you get the practical tiers. Twenty-four GB gets you serious coding models. Thirty-two GB GDDR7 gets a 70B model in one slot. A sparse MoE release claims 145B-class weights on a single NVIDIA GPU. Notice what the entry point is, though: a used card. The best value in local inference right now is secondhand silicon, which tells you exactly how far accessibility has moved.
One caution before you spend anything. “A single NVIDIA GPU” is doing a lot of work in Edge0’s sentence.
And the README does not pin a card or a memory floor. “Usable speeds” is the article’s phrase, not a benchmark. Test on hardware you already own before you put money down.
The Stack Settled, and the Zero-Dollar Tier Exists
Per that same developer article, local inference in 2026 sits on one foundation: llama.cpp, the foundational C/C++ inference engine, with Ollama as the developer default, LM Studio as the visual explorer, vLLM for production GPU serving. And Exo for distributed inference.
The starting command on existing hardware is one line: `ollama run llama3`.
For the no-GPU tier, LocalAI is the one to know.
It is self-hosted and local-first, calls itself a drop-in replacement for hosted AI services, runs on consumer-grade hardware. And states flatly: “No GPU required.” It handles GGUF, Transformers and diffusion models. And its feature list runs from text generation and MCP to audio, video, image generation and voice cloning, plus distributed, P2P and decentralized inference.
The release notes tell you who this is for.
July 2026’s LocalAI 4.6.0 shipped AMD ROCm fixes, predictable realtime pipeline warmup, conversation forking. And more reliable distributed model loading. That is maintenance language. Projects ship ROCm fixes and loader reliability because people depend on the thing daily, not for a launch-day demo.
“No GPU required” is the most underrated line in the entire local stack if you run a small operation. Most small-business machines are laptops and minis with integrated graphics, and that tier just got served. A drop-in replacement also means your migration is pointing existing code somewhere else, not rewriting it.
What I’d Do This Week
My readiness checklist, built only from what the sources above support:
– Zero budget, machine you already own: install LocalAI. Free, open source, no GPU required.
– Fastest start on existing hardware: run `ollama run llama3` and see what your machine actually does.
– Coding workhorse: a used RTX 3090 at roughly $800, 24 GB, with 27B coding models reported at usable speeds.
– One-slot 70B at Q5: the RTX 5090 at roughly $2,000, 32 GB GDDR7.
– Maximum single-card weights: Edge0’s edge0-145b or edge0-120b on a single NVIDIA GPU or an Apple Silicon Mac, via command line, local API server, or its chat app.
My contrarian read on all of it: model shopping is over as a differentiator.
The 2026 skill is infrastructure triage, knowing your memory ceiling, your engine. And your privacy constraints before you ever compare model names. Edge0 and LocalAI cost nothing in software, and the hardware is a one-time purchase. The recurring meter is the thing that quietly disappeared.
The entry fee for local AI infrastructure is now zero on the software side.
Start with `ollama run llama3` on what you own, find your memory ceiling. And only then spend money on a card.
If a 145B sparse model on one GPU and a no-GPU local server both shipped in 2026, your reason for not testing local inference expired with them.
I publish breakdowns like this weekly at ghost.mediascout.online.
