Memory (weights + KV)

Model size dominates. 27B @ BF16 ≈ 54 GB. Quantization shrinks weights; KV-cache grows with context length.

Compute (decode loop)

Autoregressive: 1 token per forward pass. Each pass is a big matmul over all weights — bandwidth-bound on most hardware.

Attention IO

Naive attention materializes N×N scores → quadratic memory + HBM traffic. FlashAttention fixes this at the kernel level.

What it does

Post-training quantization that assigns different bit-widths per layer / per tensor. Important tensors stay 8/16-bit; unimportant ones drop to 1–2-bit. Pure PTQ — no QAT, no QAD, no training on the calibration set.

• Improved imatrix calibration tuned for agentic coding, chat, multilingual
• Per-model, per-layer quantization schemes (Gemma-3 ≠ Llama-4 ≠ Qwen)
• Ships as GGUF (llama.cpp, Ollama) and safetensors
• New metric Divergence-300@32: 32-token trajectory vs BF16 (vs Top-1 / KL-Div)
• Calibration & test sets fully separated → low overfit risk

Qwen3.8-27B, measured

Bar width ∝ disk size. Lower bars = runs on consumer HW.

Gotcha: sub-Q2_K_XL loops/empty replies in agent mode → set presence_penalty=1.5; 1-bit not for tool-calls.

Mechanism

A tiny draft head baked into the model (e.g. Qwen3.5/3.6 ship mtp.*, 1 transformer layer) predicts t+2, t+3 … cheaply. The main model verifies them in one extra forward pass and accepts the longest matching prefix.

• Lossless (greedy verify) — identical output to AR
• Trained jointly → draft matches main-model distribution
• No separate draft model; needs MTP-shipped checkpoint

Why it is "free tokens"

• Qwen3.5: ~70 → 131 tok/s (GB10); 1.5–1.6× on M4 Pro
• Acceptance ~80–88% (temp 0–1)
• Sweet spot: depth 1 on Metal; n_max 2–3 on CUDA
• Does not affect prefill (prompt is parallel anyway)

Gotcha: single-head MTP → acceptance decays geometrically with depth; llama.cpp default was 16 (catastrophic) → now 3. MoE wants n_max=2, dense wants 3.

Block-diffusion drafter

Unlike sequential drafters, DFlash2 proposes the whole draft block in one parallel pass via a small block-diffusion network. A path selector keeps top-16 candidates per slot; a two-tap conv keeps coherence. Adds ~2M + 16.5M params, ~1.3% latency.

• +21% acceptance length over DFlash (16–25% per case)
• Runs on SGLang, vLLM, llama.cpp (PR #27342), oMLX
• MLX build for Apple Silicon (Qwen3.8-27B)
• Depth/default block-size 5–8; small quality loss vs AR = 0

Measured speedup

Biggest gain when baseline is compute/VRAM-starved.

Gotcha: speculation is single-stream only — at concurrency ≥2 it can cost ~25% vs no-draft. Needs ~77GB VRAM for main+draft on A100.

Unified memory advantage

MLX (Apple ML Research) keeps tensors in shared memory — no CPU↔GPU copy. On 128 GB M-series, a 27B model sits fully resident and bandwidth-efficient, so quantized + MTP/DFlash overhead is amortized.

oMLX = optimized local MLX serving stack (e.g. Qwen3.5-4B-OptiQ on 127.0.0.1:8000).

Why it matters for these methods

• Native MTP in mlx-lm (#990): 1.5× Qwen3.6-27B on M4 Pro
• DFlash2 ships an oMLX build for Apple Silicon
• Lazy eval + dynamic graphs → no recompile per shape
• Skip MTP below ~4B (overhead > gain); 4B+ wins
• M1/M2 need --dtype float16 (no native BF16)

The kernel everything rides on

Exact (no approximation) attention that tiles Q/K/V into SRAM, does online softmax, and never writes the N×N score matrix to HBM. Memory becomes linear in sequence length; the decode matmul stays bandwidth-bound, not memory-bound.

Impact on the other 4 methods

• MTP/DFlash2 verify a batch of draft tokens — FA-2 makes that batched verify nearly free
• Long-context KV cache stays small enough to keep on-chip
• Underpins vLLM, SGLang, llama.cpp, MLX backends
• Up to 9× vs PyTorch attention; 225 TFLOPs/s GPT training