There is a quirk hiding in that fact. Because a decoding step is limited by moving weights rather than by compute, scoring several candidate tokens in one pass costs almost the same wall-clock time as scoring one (Leviathan et al., arXiv 2211.17192, Section 1; Chen et al., arXiv 2302.01318). The weights are already loaded. You may as well check more than one token while they are in hand.

Speculative decoding turns that quirk into a speed-up, and it does so with a property that is rare in this field: the text it produces is mathematically identical to what the original model would have written on its own. You get the speed without trading away a single token of quality. This piece walks through how that works, and then through EAGLE-3, the current best way to make it pay off.

The EAGLE-3 decode loop. A draft model proposes a tree of next tokens, the target model checks the whole tree at once, the longest agreed path is committed, one correction is added, and the cycle repeats. Original animation made for this piece.

The core idea: draft, then check

Speculative decoding runs two models. A small, fast draft model and the large target model you actually want the output of. One round goes like this (Leviathan, Section 2; Chen, Algorithm 2):

1. Draft. The small model proposes the next several tokens, one after another. This is cheap because the draft model is small.

2. Verify in one pass. The large target model runs once over the prompt plus all the drafted tokens. In that single pass it produces, at every drafted position at once, the distribution it would have used to pick that token itself.

3. Accept or correct. Walk the drafted tokens left to right and decide which to keep.

The third step is where the magic lives, so it is worth slowing down. At each position the draft proposed a token, and the target now tells us how likely it considered that same token. Call the target's probability for the token p and the draft's probability q. (A notation warning for anyone reading the source papers: the two foundational works swap these letters. I use p for the target and q for the draft, matching Leviathan and the EAGLE papers.)

The rule is: accept the drafted token outright if the target liked it at least as much as the draft did, and otherwise accept it with probability p / q. The first time a token is rejected, throw away the rest of the draft and resample a single replacement token from the leftover probability, the part of the target's distribution that the draft under-supplied. If every drafted token is accepted, you even get one bonus token for free from the pass you already paid for (Leviathan, Section 2.3 and Algorithm 1; Chen, "Modified Rejection Sampling").

So each target pass commits at least one token and at most all the drafted tokens plus one, never fewer tokens than plain decoding would and often several more.

Why it changes nothing about the output

Here is the part I find genuinely elegant, and the reason the title says "for free." That accept-or-correct rule is built so the probability of finally emitting any given token works out to exactly the target's own probability for it. The accepted mass plus the resampled correction sum back to p, for any draft model at all. A weak draft just gets rejected more often, which costs speed, never correctness (Leviathan, Appendix A.1; Chen, Theorem 1).

That is the whole trick. The draft model is only ever a guesser. The target model remains the sole authority on what gets written, because every token is either one the target endorsed or one resampled from the target's own distribution. A better draft makes it faster and a worse one slower, but the output you would have gotten from the target alone does not move. It holds for greedy decoding and for sampling with temperature, top-k, or nucleus, by adjusting the two distributions before the test.

A fair question is what "free" actually costs, because nothing is truly free. You pay in compute and memory: you run a second model, and you verify extra tokens that may be thrown away. That is affordable precisely because decoding was memory-bound to begin with, so the extra parallel checking is nearly free in wall-clock time. The honest caveat is that the win shrinks as you batch more requests together, because a busy server is no longer sitting idle waiting on memory. EAGLE-3's own serving numbers show this clearly: a reported 1.81x at batch size 2 but 1.38x at batch size 64 (arXiv 2503.01840, Table 3, on one server setup). The free lunch is real, but it is largest when you are latency-bound, which is most of the time for a single user.

The hard part is the draft

The speed-up only pays off when the draft is both cheap and right often enough that the target keeps accepting its tokens. Making a good draft is the real research problem, and a family of methods called multi-token prediction grew up around it.

The pattern starts with blockwise parallel decoding (Stern et al., arXiv 1811.03115), which introduced the predict-several, verify-in-one-pass, keep-the-longest-agreed-prefix loop that everything since has built on. Medusa (arXiv 2401.10774) bolts a few extra prediction heads onto the model, five of them, and checks many candidate continuations together using a trick called tree attention, reporting speed-ups of about 2.2x in its simpler form and 2.3 to 3.6x in its trained form. Meta's multi-token prediction (Gloeckle et al., arXiv 2404.19737) trains parallel heads on a shared trunk. DeepSeek-V3 (arXiv 2412.19437) keeps sequential prediction modules that can be repurposed as a drafter for roughly 1.8x faster generation. EAGLE is the line of work that has pushed draft quality furthest, and its third generation is the focus here.

From EAGLE-1 to EAGLE-3

EAGLE-1 (arXiv 2401.15077) made one good observation: instead of drafting at the level of tokens, draft at the level of the target's internal features, the hidden state just below the output. It feeds the just-sampled token in alongside that feature so the draft knows what was actually chosen, and trains a small draft head to predict the next feature.

EAGLE-2 (arXiv 2406.16858) kept that draft model but made the set of candidates it proposes smarter. Instead of a fixed shape, it grows a dynamic draft tree, using the draft's own confidence as a stand-in for how likely each branch is to be accepted, expanding the promising branches and reranking them.

EAGLE-3 (arXiv 2503.01840) is the current generation, and it makes three changes that are the heart of the "how it's done":

1. It predicts tokens directly instead of matching a feature vector. EAGLE-1 and 2 forced the draft to reproduce one of the target's feature vectors, and that constraint capped how much extra training data could help. Dropping it lets draft accuracy keep climbing as you train on more data, which the authors report as the headline scalability result (Section 3.1).

2. It fuses several layers of the target, not just the top one. The draft is fed a combination of low, middle, and high level hidden states from the target rather than a single layer, giving the small draft model a richer picture of what the target is thinking (Section 3.1).

3. It practices the way it will play. During training the draft is run through the same multi-step loop it uses at inference, feeding its own outputs back as the next step's input under a tree-shaped attention mask. The authors call this a "training-time test." It means the draft learns to recover from its own mistakes instead of only ever seeing perfect inputs (Section 3.2).

One decode round, end to end

This is what the animation above shows, step by step.

From the current text, the draft model grows a small tree of possible continuations rather than a single line. EAGLE-3 reports it can run a deeper tree than its predecessor, a depth of 8 versus EAGLE-2's depth of 6, while keeping the same number of nodes, because its better draft earns the extra depth (Section 4.1). The tree keeps roughly ten expansion nodes and on the order of fifty candidate tokens in total, sized to the target model (the authors use 60, 50, and 48 for 7B-or-8B, 13B, and 70B targets).

The whole tree is then flattened and scored by the target in a single forward pass. A special tree attention mask lets each candidate token see only its own ancestors and the original context, never its sibling branches, so one pass scores every root-to-leaf path as if they were independent (EAGLE-1, Sections 3.1 and 3.3; Medusa). The accept-or-correct rule from earlier is then applied down the branches. The single longest agreed path is committed, the rest collapse, and at the first rejected token one correction is resampled. Then the loop repeats from the new, longer context.

How many tokens you commit per target pass is the number that decides the speed-up. The authors call it the average acceptance length. Across the EAGLE generations on the same setup it rises from about 3.96 to 4.83 to 6.62 (Table 1), and on 8B-class targets EAGLE-3 reports averages of 6.23 for Llama-3.1-8B and 5.84 for DeepSeek-R1-Distill-8B (Table 1). More tokens per pass is more weights-load amortized, which is where the end-to-end speed-up comes from: up to 6.47x over plain decoding on one coding benchmark, and a reported 20 to 40 percent on top of EAGLE-2 (Sections 4.1 and the abstract). Those are the authors' numbers on their hardware, not mine.

One round, as a diagram. p is the target distribution, q is the draft. Each target pass commits at least one and at most all the drafted tokens plus one, and the emitted text is distribution-identical to the target model decoding on its own (Leviathan, Appendix A.1; Chen, Theorem 1).

Why this is worth knowing

Speculative decoding is one of the few places in this field where you do not have to choose between fast and good. The output is the target model's, token for token. Everything clever goes into the draft, where being wrong only costs a little speed. EAGLE-3's contribution is a draft that is right more often and keeps getting better with more training data, which is what carries the average acceptance length up and the latency down.

If you serve or run local models, this is a lever worth reaching for before you reach for a smaller model or a heavier quantization, because unlike those it does not change what comes out. The honest boundaries: the reported numbers are the authors' own on their hardware and will shift with your draft, your model, and your load; the gains are largest for single-stream, latency-bound use and smaller under heavy batching; and a draft that fits your target well matters more than any headline multiplier. None of that dents the core idea, which is the part I keep coming back to. You can make a large model generate several times faster and prove, on paper, that you changed nothing about what it says.

The animation and pipeline diagram are my own, made to illustrate the mechanism. Every number here is the cited authors' reported figure, not a measurement I made. If you want to go deeper, the sources below are the primary ones, and they are very readable.

Sources

• EAGLE-3. Li, Wei, Zhang, Zhang, "EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test." https://arxiv.org/abs/2503.01840

• EAGLE-2. "EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees." https://arxiv.org/abs/2406.16858

• EAGLE-1. "EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty." https://arxiv.org/abs/2401.15077

• Leviathan, Kalman, Matias, "Fast Inference from Transformers via Speculative Decoding" (ICML 2023). https://arxiv.org/abs/2211.17192

• Chen et al., "Accelerating Large Language Model Decoding with Speculative Sampling." https://arxiv.org/abs/2302.01318

• Stern, Shazeer, Uszkoreit, "Blockwise Parallel Decoding for Deep Autoregressive Models" (NeurIPS 2018). https://arxiv.org/abs/1811.03115

• Cai et al., "Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads." https://arxiv.org/abs/2401.10774

• Gloeckle et al., "Better and Faster Large Language Models via Multi-token Prediction." https://arxiv.org/abs/2404.19737

• DeepSeek-AI, "DeepSeek-V3 Technical Report." https://arxiv.org/abs/2412.19437

• Reference implementation: SafeAILab/EAGLE (Apache-2.0). https://github.com/SafeAILab/EAGLE

Almost every AI tool you use today runs somewhere else. You type a question, it travels to a giant datacenter, a rack of expensive chips answers it, and the reply comes back. That works, but it has costs. Your words leave your device. It needs a network connection. Someone pays for all that hardware and electricity.

There is another way, and it is getting better fast: run the model on the device you already own. Your laptop. Your phone. The benefits are obvious once you say them out loud. Your data stays with you. It works on a plane with no signal. There is no per-question bill. The catch has always been that these devices are small, and big AI models are hungry, so the question is what we can actually fit.

I spend my time on exactly that question. I run AI models on an ordinary, fan-less laptop and measure what they do. Recently a quiet design change has started to move the line on what fits, and I want to walk through it in plain terms, then show what it bought me on real hardware.

The problem: a good memory gets expensive

To hold a conversation, an AI model has to remember what was already said. The standard way it does this is to keep a note for every single word it has seen so far. Early in a chat that pile of notes is small. By the time you are deep into a long document or a long back-and-forth, the pile is huge.

That pile causes two problems, and both are worst on a small device. The model gets slower, because before writing each new word it re-reads the whole growing pile. And it eats memory, because the pile has to live in the device’s RAM, which is fixed and shared with everything else.

In a datacenter you hide both problems by throwing more memory at them. On a laptop you cannot. This is the real reason capable AI has lived in the cloud, and it is the part most explanations skip.

Figure 1. The standard approach keeps a note per word, so its memory grows with the conversation. The newer approach keeps one fixed-size summary and updates it.

The fix: keep a summary, not a transcript

The newer idea is simple to state. Instead of keeping every note, the model keeps one fixed-size summary and updates it as it reads. Because the summary never grows, the memory it uses stops depending on how long the conversation is, and so does the time to produce each new word. You get a flat cost instead of a rising one.

There is a real trade. You are squeezing everything that was said into a fixed amount of space, so the whole game becomes how cleverly the model updates that summary without losing what matters.

This family of “fixed-size summary” models started with one called Mamba, and it has been refined several times since. The newest refinement, the one in the latest open models, is called Gated DeltaNet. Its trick is a smarter way to edit the summary. Older versions could only fade the whole summary a bit at a time, like turning down the lights in a room. Gated DeltaNet can instead find one specific fact in the summary and overwrite just that, like correcting a single line in your notes and leaving the rest alone. It pairs that targeted edit with a “forget” dial that decides how much of the past to keep when the topic changes.

Figure 2. You do not need the math. The idea is the two moves above: a “forget” dial, and a targeted overwrite of one entry.

The researchers who introduced Gated DeltaNet (a 2025 paper from a team at NVIDIA and MIT) showed this combined trick beats the earlier fixed-summary designs across a range of tests. If you want the family tree, the figure below sketches how the approaches relate. You can skip it without losing the thread.

Figure 3. How the “fixed-size summary” family grew. The two recent branches make the summary either richer (Mamba-3) or smarter to edit (Gated DeltaNet).

Why the newest models mix the two

Here is the part I respect with my physics brain. The latest models did not bet everything on the cheap summary. A model is built from many internal stages, called layers, stacked one after another. A recent open model, Qwen3.5, mixes the two styles across those layers in a three-to-one ratio: three cheap fixed-summary layers for every one old-style “keep every note” layer. The cheap layers carry the memory savings across most of the model, and the occasional full-memory layer preserves the exact, look-anything-up recall that a summary is worst at. Mixing them beats going all-in on either one. It is a sensible compromise, and it is the design I took to the laptop.

What it actually did on a fan-less laptop

Theory is nice. I wanted numbers from real hardware, so I ran these models on a thin, fan-less Snapdragon laptop and measured them. The numbers below are from a mid-sized model, about four billion parameters, the size that fits a thin laptop comfortably; I also tried smaller and larger versions and the picture held. A few things stood out. Quick translations first: the “AI chip” below is the laptop’s NPU, a small dedicated chip for AI math that sits next to the regular processor (the CPU). “Tokens per second” is roughly words per second.

It runs on the laptop’s AI chip, and a close cousin does not. Every single operation in the Gated DeltaNet model ran on the AI chip, with nothing falling back to the slower general processor. That matters because an older fixed-summary design, Mamba, cannot do this on the same laptop. One of its core steps has no support on the AI chip, so it spills back onto the CPU. Of the efficient designs, this is the one the AI chip can actually run today.

It can hold a far longer conversation before running out of memory. Because most of its layers keep a fixed summary instead of a growing pile of notes, its memory grows about four and a half times more slowly than a traditional model of the same size. In practice, on this laptop’s spare memory, it can hold a conversation roughly four to five times longer before it runs out. That is very roughly 760,000 versus 170,000 word-sized chunks of text or tokens. Same machine, several times the headroom.....LET me put this into context for you a 760,000 tokens amounts to a running conversation between "You" and the model "AI" that spans roughly half Harry Potter series. That's a long convo or tons of text.

For long inputs, the AI chip pulls ahead. When you paste in a long document, the AI chip reads it at a steady rate while the regular processor slows down as the document grows. Past roughly six to seven thousand words the AI chip becomes the faster of the two for reading, though only modestly, by about ten to seventeen percent. It also stays cool and steady under sustained load, where the regular processor throttles and loses about ten percent of its speed. And you can run a second task alongside it and keep most of your speed: about 95 percent at short lengths, settling into the mid-80s to mid-90s as the conversation grows.

The honest catch. For a short, one-at-a-time reply, the regular processor is still faster. On this generation of AI chip, the model’s word-by-word writing speed on the chip is only about a third of the CPU’s, and that does not flip no matter the length. So the AI chip’s advantage here is not raw speed on a single quick answer. It is endurance: longer conversations, far more memory headroom, faster handling of long inputs, staying cool, and doing more than one thing at once. Anyone who tells you these efficient models are simply faster has not measured them on a small device.

To show the underlying effect on its own, I ran a pure summary-only model against a traditional one across a conversation that grew 64 times longer. These are not the Gated DeltaNet model from the tests above. They are a clean, summary-only design, shown here only to illustrate the family’s core behavior. The summary-only model held a steady speed of about 18 to 19 words per second and a flat memory footprint the whole way. The traditional model fell from 16 words per second to under 2, about ten times slower at the long end, and its memory kept climbing. One holds its cost flat while the other’s keeps rising.

Figure 4. The same laptop, the same growing conversation. The summary-only model (green) holds about 18 to 19 words per second and a flat memory footprint; the traditional model (orange) slows to under 2 words per second and its memory keeps rising. These are a pure summary-only model (about seven billion parameters) and a traditional one (about eight billion), used to show the family behavior, not the Gated DeltaNet model from the rest of the piece. My own measurements on a Snapdragon X Plus.

The takeaway

The future of AI is not only the cloud. The pieces are quietly falling into place to run genuinely capable models on the devices we already carry, privately and offline. The thing that stood in the way was never raw intelligence. It was memory: the cost of remembering a long conversation on a small machine. Design changes like Gated DeltaNet are how we get around that, by keeping a smart fixed-size summary instead of an ever-growing pile of notes.

So if you are ever choosing one of these models for a phone or a laptop, do not start with the leaderboard. Start with the shape of your constraint, and pick the one whose costs stay flat where yours would otherwise climb. On a small device, that is what actually decides whether the thing runs at all.

The diagrams are my own. The design details come from the cited research papers and model cards. The laptop measurements are from my own experiments on a Snapdragon X Plus, and to be clear, the regular processor still wins on short, single replies. These numbers are for this hardware, not a newer chip.

Sources

• Gated DeltaNet. Yang, Kautz & Hatamizadeh, "Gated Delta Networks: Improving Mamba2 with Delta Rule," ICLR 2025. https://arxiv.org/abs/2412.06464

• The delta rule. Yang, Wang, Zhang, Shen & Kim, "Parallelizing Linear Transformers with the Delta Rule over Sequence Length," NeurIPS 2024. https://arxiv.org/abs/2406.06484

• Mamba. Gu & Dao, "Mamba: Linear-Time Sequence Modeling with Selective State Spaces." https://arxiv.org/abs/2312.00752

• Mamba-2. Dao & Gu, "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality." https://arxiv.org/abs/2405.21060

• Mamba-3. Lahoti, Li, Bick, Kolter, Chen, Wang, Dao & Gu, "Mamba-3: Improved Sequence Modeling using State Space Principles," ICLR 2026. https://arxiv.org/abs/2603.15569

• State-space models on NPUs (XAMBA). https://arxiv.org/abs/2502.06924

• Qwen3.5 and Qwen3-Next. Model cards on Hugging Face: https://huggingface.co/Qwen and the Qwen3-Next blog: https://qwen.ai