Non-autoregressive parallel decoding just cut decoding steps from 256 to 24 on MBPP, a 10.5× speedup reported by the dParallel paper. And it is the most interesting speed story in local LLM inference right now. The term covers methods that generate draft tokens “without causal dependencies among them,” per a recent speculative decoding survey, meaning the model proposes several tokens at once instead of writing one, waiting, then writing the next. Two camps are racing: prompt-based methods that squeeze up to 2.49× out of edge-scale models like MobileLlama. And diffusion language models reporting 8.5× to 10.5×. The catch, from the ParallelBench benchmark, is that parallel decoding can collapse quality on real-world tasks.
Here is the honest breakdown.
Why Token-By-Token Is The Wall
Your local model is slow for a structural reason, not a hardware-brand reason.
Standard autoregressive decoding writes one token, conditions the next token on it.
And repeats, so a long answer costs one sequential round of compute per token no matter how fast any single round is. On a laptop or a single desktop card, sequential steps are exactly the thing you pay for most.
Non-autoregressive methods break that chain. The survey defines them as speculative decoding algorithms where “the draft tokens are generated without causal dependencies among them,” and it names the most common implementations: Multi-Token Prediction (MTP, Gloeckle et al. 2024) and Medusa (Cai et al. 2024). The model guesses multiple positions at once, then something verifies the guesses. That verify step is where all the risk hides, and I will come back to it.
PPD: The Version Built For Edge-Scale Models
A 2025 paper proposes Hardware-Aware Parallel Prompt Decoding, abbreviated PPD. And it is the closest thing in this literature to a small-operator play (paper PDF). Instead of keeping a second, smaller draft model beside your real one, the authors tune prompts: PPD approximates outputs at future timesteps in parallel using multiple prompt tokens. And a single unified model handles both speculation and verification.
The authors report three things that matter to anyone running local models.
First, the method requires only runtime memory overhead. Because there is no separate draft model to fit in RAM you do not have. Second, evaluated across LLMs ranging from MobileLlama to Vicuna-13B, it demonstrated up to 2.49× speedup. Third, it stacks with what you may already use: combined with existing speculative decoding, the authors report up to 1.22× further speed improvement.
MobileLlama on that evaluation list is the tell. This paper opens by targeting edge-scale LLMs, not datacenter hardware. And the prompt-tuning design means one model does two jobs. Treat 2.49× as the authors’ best case on their benchmarks, not a floor you are guaranteed.
Even discounted, a single-model method with no draft model to host is the shape of thing that actually lands in runtimes you and I can install.
dParallel: Where The Headline Numbers Live
The spectacular numbers come from diffusion language models. dParallel, a learnable parallel decoding method for dLLMs, reports cutting decoding steps on LLaDA-8B-Instruct from 256 to 30 on GSM8K, an 8.5× speedup without performance degradation (paper).
On MBPP it cuts decoding steps from 256 to 24, a 10.5× speedup while maintaining accuracy.
The evaluation spans GSM8K, MATH, HumanEval. And MBPP across two open-source dLLMs: LLaDA, trained from scratch as a native dLLM. And Dream, initialized from an autoregressive LLM.
The authors frame diffusion models as having “inherent parallelism” that standard sampling leaves locked. And that framing is what makes the 10.5× plausible rather than magic.
These architectures were not built around strict left-to-right dependency. So asking them to revise many positions at once fights their nature less. The catch is that you are probably not serving traffic with a diffusion LLM today. So for now these are research-model numbers on math and code benchmarks, not a speedup you can route your chat workload through.
ParallelBench: The Fine Print Nobody Reprints
ParallelBench, a benchmark presented in the ICLR 2026 proceedings, exists to check this work. And it evaluates trade-offs rather than speed alone.
Its two headline findings:
– dLLMs under parallel decoding “can suffer dramatic quality degradation in real-world scenarios.”
– Current parallel decoding strategies “struggle to adapt their degree of parallelism based on task difficulty, thus failing to achieve meaningful speedup without compromising quality.”
That second finding is the whole ballgame.
A method that parallelizes hard reasoning the same way it parallelizes an easy fill-in either degrades quality on the hard task or dials back the parallelism and loses the speedup. Benchmark leaderboards reward the headline number; your workload is the thing that pays for it.
A follow-up paper, “Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding?” (arXiv), proposes NAP as a proof-of-concept fix. NAP curates training examples as “multiple independent reasoning trajectories” and couples them with a parallel-forced decoding strategy that encourages multi-token parallel updates, aiming to better align supervision with non-AR parallel decoding. Read that as confirmation the problem is not just the decoding loop; the training data itself resists being generated in parallel.
What This Means If You Run Local Models
For solo operators and small shops, I see three practical reads.
The PPD template, one model doing its own speculation and verification with prompt tuning, is the realistic near-term shape: modest, stackable gains without a second model in memory.
The diffusion numbers are real but live on research models. So file them under “watch,” not “deploy.” And any parallel decoding claim you evaluate should be judged on your hardest prompt with quality checked by eye, as ParallelBench found that speed without quality loss is precisely what current methods fail to deliver.
The decision rule I would use: a speedup that only holds on easy tasks is a benchmark artifact.
And a method that adapts its parallelism to task difficulty is the one worth waiting for.
The Takeaway
Non-autoregressive parallel decoding is moving from paper to plausible. The honest summary is that prompt-based single-model methods deliver modest, stackable gains (up to 2.49×, plus up to 1.22× when combined with existing speculative decoding, per the authors), while diffusion architectures post spectacular ones (up to 10.5×) that ParallelBench says degrade under real-world task difficulty. The methods that win will be the ones that adjust how much they parallelize based on what you asked them to do.
Until then, measure before you adopt.
Time your slowest local generation today. And the next time a runtime ships a parallel or speculative decode option, test it on your hardest real prompt and diff the output quality yourself. If this saved you reading three papers, follow along, since I will keep turning these inference papers into decisions you can act on.
