← Tap Left / Press Left Arrow | Tap Right / Press Right Arrow →
Inference Engineering · 2026

New LLM Optimization Methods

How to run frontier-scale models on a laptop, a single GPU, or Apple Silicon — fast and cheap. One method per page, with a visualization, then a synthesis for the Qwen3.8-Flash class.

5core methods
2×–10×typical speedup
1-bitto 4-bit viable
Qwen3.8target model

The Bottleneck: Where the Cost Lives

Memory (weights + KV)

Model size dominates. 27B @ BF16 ≈ 54 GB. Quantization shrinks weights; KV-cache grows with context length.

# RAM needed (decode) RAM ≈ params × bits/8 + seq × layers × 2 × d_model × 2 27B × 16b = 54 GB → 27B × 4b = 13.5 GB

Compute (decode loop)

Autoregressive: 1 token per forward pass. Each pass is a big matmul over all weights — bandwidth-bound on most hardware.

for t in range(N): # N sequential passes logits = model(x[:, -1:]) # full-weight matmul x = cat(x, sample(logits))

Attention IO

Naive attention materializes N×N scores → quadratic memory + HBM traffic. FlashAttention fixes this at the kernel level.

# standard S = QKᵀ # N×N, stored to HBM # flash tile→SRAM, online softmax, never materialize S

Three levers → three families: shrink weights, draft many tokens per pass, fuse the kernel.

1 · Unsloth Dynamic 3.0 — Smart Quantization PTQ

What it does

Post-training quantization that assigns different bit-widths per layer / per tensor. Important tensors stay 8/16-bit; unimportant ones drop to 1–2-bit. Pure PTQ — no QAT, no QAD, no training on the calibration set.

  • Improved imatrix calibration tuned for agentic coding, chat, multilingual
  • Per-model, per-layer quantization schemes (Gemma-3 ≠ Llama-4 ≠ Qwen)
  • Ships as GGUF (llama.cpp, Ollama) and safetensors
  • New metric Divergence-300@32: 32-token trajectory vs BF16 (vs Top-1 / KL-Div)
  • Calibration & test sets fully separated → low overfit risk

Qwen3.8-27B, measured

Quant Size Top-1 Note UD-IQ1_S 6.2 GB ~72% -89% size, 1-bit UD-Q2_K_XL 9.83 GB +8% best agent tier UD-Q4_0 ~13 GB higher MTP optional # >10% Top-1 better at same disk vs other providers
IQ1_S 6.2GB
Q2_K_XL 9.8GB
Q4_0 ~13GB
BF16 54GB

Bar width ∝ disk size. Lower bars = runs on consumer HW.

Gotcha: sub-Q2_K_XL loops/empty replies in agent mode → set presence_penalty=1.5; 1-bit not for tool-calls.

2 · Multi-Token Prediction (MTP) — Self-Speculative decode

Mechanism

A tiny draft head baked into the model (e.g. Qwen3.5/3.6 ship mtp.*, 1 transformer layer) predicts t+2, t+3 … cheaply. The main model verifies them in one extra forward pass and accepts the longest matching prefix.

t2, t3 = mtp_head(hidden_t, emb(t1)) # draft L = model([t1,t2,t3], kv=shared) # verify (1 pass) accept longest prefix where argmax(L) == draft
  • Lossless (greedy verify) — identical output to AR
  • Trained jointly → draft matches main-model distribution
  • No separate draft model; needs MTP-shipped checkpoint

Why it is "free tokens"

t₁
main
→ draft →
t₂ t₃
MTP head
→ 1 verify →
accept 2
2 tok/1 pass
  • Qwen3.5: ~70 → 131 tok/s (GB10); 1.5–1.6× on M4 Pro
  • Acceptance ~80–88% (temp 0–1)
  • Sweet spot: depth 1 on Metal; n_max 2–3 on CUDA
  • Does not affect prefill (prompt is parallel anyway)

Gotcha: single-head MTP → acceptance decays geometrically with depth; llama.cpp default was 16 (catastrophic) → now 3. MoE wants n_max=2, dense wants 3.

3 · DFlash2 — Parallel Speculative Decoding z-lab / Inco AI

Block-diffusion drafter

Unlike sequential drafters, DFlash2 proposes the whole draft block in one parallel pass via a small block-diffusion network. A path selector keeps top-16 candidates per slot; a two-tap conv keeps coherence. Adds ~2M + 16.5M params, ~1.3% latency.

  • +21% acceptance length over DFlash (16–25% per case)
  • Runs on SGLang, vLLM, llama.cpp (PR #27342), oMLX
  • MLX build for Apple Silicon (Qwen3.8-27B)
  • Depth/default block-size 5–8; small quality loss vs AR = 0

Measured speedup

Setup Base DFlash2 ×speed A100 (Qwen3.8-27B) 28.9 59.1 ~2.0× M5 Pro Q4_K_M 10.42 19.31 1.85× RTX 5050 (8GB, starved) 6.8 11.6 1.7×
baseline
DFlash2 ~2×

Biggest gain when baseline is compute/VRAM-starved.

Gotcha: speculation is single-stream only — at concurrency ≥2 it can cost ~25% vs no-draft. Needs ~77GB VRAM for main+draft on A100.

4 · oMLX / MLX — Apple-Silicon Native Runtime runtime

Unified memory advantage

MLX (Apple ML Research) keeps tensors in shared memory — no CPU↔GPU copy. On 128 GB M-series, a 27B model sits fully resident and bandwidth-efficient, so quantized + MTP/DFlash overhead is amortized.

pip install mlx-lm mlx_lm.generate --model Qwen3.8-27B-4bit --mtp mlx_lm.server --model Qwen3.8-27B-4bit --mtp

oMLX = optimized local MLX serving stack (e.g. Qwen3.5-4B-OptiQ on 127.0.0.1:8000).

Why it matters for these methods

  • Native MTP in mlx-lm (#990): 1.5× Qwen3.6-27B on M4 Pro
  • DFlash2 ships an oMLX build for Apple Silicon
  • Lazy eval + dynamic graphs → no recompile per shape
  • Skip MTP below ~4B (overhead > gain); 4B+ wins
  • M1/M2 need --dtype float16 (no native BF16)
weights
unified
→ no copy →
GPU+CPU
same RAM

5 · FlashAttention-2 — IO-Aware Kernel foundation

The kernel everything rides on

Exact (no approximation) attention that tiles Q/K/V into SRAM, does online softmax, and never writes the N×N score matrix to HBM. Memory becomes linear in sequence length; the decode matmul stays bandwidth-bound, not memory-bound.

FlashAttention : 2–4× faster, 10–20× less memory (linear) FlashAttention-2: +2× again → up to 73% A100 FLOPs · split Q across warps (not K/V) → less shared-mem traffic · parallelize over sequence length · supports head_dim ≤ 256, MQA/GQA

Impact on the other 4 methods

naive (quadratic)
FlashAttn
FlashAttn-2
  • MTP/DFlash2 verify a batch of draft tokens — FA-2 makes that batched verify nearly free
  • Long-context KV cache stays small enough to keep on-chip
  • Underpins vLLM, SGLang, llama.cpp, MLX backends
  • Up to 9× vs PyTorch attention; 225 TFLOPs/s GPT training

Method Map — Where Each Lever Hits

Shrink weights

Unsloth Dynamic 3.0 → fits 27B in 6–14 GB.

Draft tokens

MTP (self) + DFlash2 (parallel) → 1.5–2× decode.

Fuse kernel

FlashAttention-2 → linear memory, free batched verify.

Run natively

oMLX/MLX → unified memory removes copy cost.

Stack them: quantize → load in MLX → enable MTP/DFlash2 → rely on FlashAttn-2 under the hood.

Synthesis — Qwen3.8-Flash: Stack the Stack

The Qwen3.8-Flash class (next-gen small/fast variant) is the ideal target for the full stack. Recipe for a 27B-class model on a single Apple-Silicon or 24 GB GPU box:

  1. Quantize with Unsloth Dynamic 3.0 → UD-Q4_K_M (~13 GB) or UD-Q2_K_XL (~9.8 GB, best agent tier).
  2. Load via oMLX / mlx-lm (unified memory) — no CPU↔GPU copies.
  3. Enable MTP (--mtp); depth 1 on Metal, n_max 2–3 on CUDA → +1.5× decode.
  4. Add DFlash2 drafter when VRAM allows → up to +2×; pairs with MTP on llama.cpp.
  5. FlashAttention-2 is automatic in the backend — keeps long context cheap.
# Apple Silicon, 27B, ~2× throughput mlx_lm.server --model Qwen3.8-27B-4bit \ --mtp # CUDA / llama.cpp, chain drafters llama-server -m Qwen3.8-27B-UD-Q4_K_M.gguf \ --spec-type draft-mtp --spec-draft-n-max 3 \ --spec-draft dflash2-q8 # optional

Expected: BF16-class quality at 4-bit, 1.5–2× faster, runs on a laptop or single GPU.

Numbers from vendor/repro benchmarks (Unsloth, z-lab, mlx-lm, Dao et al.); real throughput is hardware- and prompt-dependent.

Validated Skills Library installed & scanned

All installed via hermes skills install, security-scanned, and verified (valid SKILL.md, enabled).

Web & Research

agent-reach · youtube-full · defuddle · resemble-detect

Build & Engineering

using-agent-skills · setup-matt-pocock-skills · make-interfaces-feel-better · humanizer

Agents & Ops

browser-harness · i-have-adhd · loopy · loop-library · skillclaw

# list active skills hermes skills list # 14 validated community/url skills now active

Command Cheat Sheet DESK · CLI · MSG · SHELL

Everyday commands — Desktop, CLI chat, messaging, shell. (Verified vs current Hermes source, Sep 2026.)

Sessions & Context

/new · /resume · /sessions · /title · /branch · /compress · /context · /status

Control Work

/queue · /steer · /bg · /btw · /agents · /stop

Goals, Loops & Plans

/goal · /subgoal · /heartbeat · /loop · /plan · /review · /refine

Models & Behavior

/model · /moa · /personality · /reasoning · /fast · /approvals · /yolo · /busy · /voice

Skills, Memory & Tools

/skills · /learn · /memory · /init · /tools · /browser

Automation & Coord

/cron · /suggestions · /blueprint · /kanban

Inspect, Recover & Fix

/retry · /undo · /save · /diff · /rollback · /usage · /debug · /help

Messaging / Gateway

/sethome · /topic · /commands · /approve · /deny · /pause · /platform · /restart

Terminal Essentials

hermes / hermes chat · -z "prompt" · model · status · doctor · gateway status · --safe-mode · update

# repeat-work modes /goal = work until objective met /loop = repeat w/ stop conditions /heartbeat = one recurring check /cron = durable schedule outside chat # type / + letters for autocomplete; /help for full list

Thank You & Discussion

Quantize smart · Draft in parallel · Fuse the kernel · Run native

← All presentations · New Era of Research Tools