---
layout: farshid_default
title: "New LLM Optimization Methods — Run Local & Fast"
permalink: /notes/slides/llm-optimization/
description: "Practical methods for running LLMs locally and fast: quantization, KV-cache strategies, attention variants, and measured throughput on consumer hardware."
---

<div class="presentation-panel">
  <div class="nav-hint">Tap the right side for the next slide · tap the left side to go back · swipe or arrow keys work too</div>
  <div class="reveal">
    <div class="slides">

      
      <section>
        <div style="padding:34px 40px; border-radius:18px">
          <div style="font-size:.6em; letter-spacing:.18em; color:#1a56db; text-transform:uppercase">Inference Engineering · 2026</div>
          <h1>New LLM Optimization Methods</h1>
          <p style="color:#2b3038; max-width:760px">How to run frontier-scale models on a laptop, a single GPU, or Apple Silicon — fast and cheap. One method per page, with a visualization, then a synthesis for the <b>Qwen3.8-Flash</b> class.</p>
          <div style="margin-top:18px">
            <div class="strip"><b>5</b><span>core methods</span></div>
            <div class="strip"><b>2×–10×</b><span>typical speedup</span></div>
            <div class="strip"><b>1-bit</b><span>to 4-bit viable</span></div>
            <div class="strip"><b>Qwen3.8</b><span>target model</span></div>
          </div>
        </div>
      </section>

      
      <section>
        <h2>The Bottleneck: Where the Cost Lives</h2>
        <div class="m-3">
          <div class="c">
            <h3 style="color:#1a56db">Memory (weights + KV)</h3>
            <p>Model size dominates. 27B @ BF16 ≈ 54 GB. Quantization shrinks weights; KV-cache grows with context length.</p>
            <div class="code-box">
<span class="cmt"># RAM needed (decode)</span>
RAM ≈ params × bits/8  +  seq × layers × 2 × d_model × 2
27B × 16b = 54 GB   →   27B × 4b = 13.5 GB
            </div>
          </div>
          <div class="c">
            <h3 style="color:#6d28d9">Compute (decode loop)</h3>
            <p>Autoregressive: 1 token per forward pass. Each pass is a big matmul over all weights — bandwidth-bound on most hardware.</p>
            <div class="code-box">
<span class="kw">for</span> t <span class="kw">in</span> range(N):        <span class="cmt"># N sequential passes</span>
    logits = model(x[:, -1:]) <span class="cmt"># full-weight matmul</span>
    x = cat(x, sample(logits))
            </div>
          </div>
          <div class="c">
            <h3 style="color:#1a56db">Attention IO</h3>
            <p>Naive attention materializes N×N scores → quadratic memory + HBM traffic. FlashAttention fixes this at the kernel level.</p>
            <div class="code-box">
<span class="cmt"># standard</span>
S = QKᵀ            <span class="cmt"># N×N, stored to HBM</span>
<span class="cmt"># flash</span>
tile→SRAM, online softmax, never materialize S
            </div>
          </div>
        </div>
        <p style="color:#6b7280; font-size:.6em">Three levers → three families: <b>shrink weights</b>, <b>draft many tokens per pass</b>, <b>fuse the kernel</b>.</p>
      </section>

      
      <section>
        <h2>1 · Unsloth Dynamic 3.0 — Smart Quantization <span class="tag p">PTQ</span></h2>
        <div class="m-2">
          <div class="c">
            <h3 style="color:#6d28d9">What it does</h3>
            <p>Post-training quantization that assigns <b>different bit-widths per layer / per tensor</b>. Important tensors stay 8/16-bit; unimportant ones drop to 1–2-bit. Pure PTQ — no QAT, no QAD, no training on the calibration set.</p>
            <ul style="font-size:.62em">
              <li>Improved <b>imatrix calibration</b> tuned for agentic coding, chat, multilingual</li>
              <li>Per-model, per-layer quantization schemes (Gemma-3 ≠ Llama-4 ≠ Qwen)</li>
              <li>Ships as GGUF (llama.cpp, Ollama) and safetensors</li>
              <li>New metric <b>Divergence-300@32</b>: 32-token trajectory vs BF16 (vs Top-1 / KL-Div)</li>
              <li>Calibration & test sets fully separated → low overfit risk</li>
            </ul>
          </div>
          <div class="c">
            <h3 style="color:#1a56db">Qwen3.8-27B, measured</h3>
            <div class="code-box">
Quant          Size      Top-1     Note
UD-IQ1_S       6.2 GB    ~72%      -89% size, 1-bit
UD-Q2_K_XL     9.83 GB   +8%       best agent tier
UD-Q4_0        ~13 GB    higher    MTP optional
<span class="cmt"># >10% Top-1 better at same disk vs other providers</span>
            </div>
            <div class="vis" style="font-size:.5em; margin-top:8px">
              <div class="bar"><span style="width:12%; background:#a78bfa">IQ1_S 6.2GB</span></div>
              <div class="bar"><span style="width:18%; background:#8b5cf6">Q2_K_XL 9.8GB</span></div>
              <div class="bar"><span style="width:24%; background:#93c5fd">Q4_0 ~13GB</span></div>
              <div class="bar"><span style="width:100%; background:#2b3038">BF16 54GB</span></div>
            </div>
            <p style="font-size:.5em; color:#6b7280">Bar width ∝ disk size. Lower bars = runs on consumer HW.</p>
            <p style="font-size:.52em; color:#92400e">Gotcha: sub-Q2_K_XL loops/empty replies in agent mode → set <code>presence_penalty=1.5</code>; 1-bit not for tool-calls.</p>
          </div>
        </div>
      </section>

      
      <section>
        <h2>2 · Multi-Token Prediction (MTP) — Self-Speculative <span class="tag">decode</span></h2>
        <div class="m-2">
          <div class="c">
            <h3 style="color:#1a56db">Mechanism</h3>
            <p>A tiny <b>draft head</b> baked into the model (e.g. Qwen3.5/3.6 ship <code>mtp.*</code>, 1 transformer layer) predicts t+2, t+3 … cheaply. The main model verifies them in <b>one</b> extra forward pass and accepts the longest matching prefix.</p>
            <div class="code-box">
t2, t3 = mtp_head(hidden_t, emb(t1))  <span class="cmt"># draft</span>
L      = model([t1,t2,t3], kv=shared) <span class="cmt"># verify (1 pass)</span>
accept longest prefix where argmax(L) == draft
            </div>
            <ul style="font-size:.58em">
              <li>Lossless (greedy verify) — identical output to AR</li>
              <li>Trained jointly → draft matches main-model distribution</li>
              <li>No separate draft model; needs MTP-shipped checkpoint</li>
            </ul>
          </div>
          <div class="c">
            <h3 style="color:#1a56db">Why it is "free tokens"</h3>
            <div class="flow">
              <div class="node">t₁<br><small>main</small></div>
              <div class="arrow">→ draft →</div>
              <div class="node ok">t₂ t₃<br><small>MTP head</small></div>
              <div class="arrow">→ 1 verify →</div>
              <div class="node ok">accept 2<br><small>2 tok/1 pass</small></div>
            </div>
            <ul style="font-size:.58em">
              <li>Qwen3.5: <b>~70 → 131 tok/s</b> (GB10); <b>1.5–1.6×</b> on M4 Pro</li>
              <li>Acceptance ~80–88% (temp 0–1)</li>
              <li>Sweet spot: depth <b>1</b> on Metal; n_max <b>2–3</b> on CUDA</li>
              <li>Does not affect prefill (prompt is parallel anyway)</li>
            </ul>
            <p style="font-size:.5em; color:#92400e">Gotcha: single-head MTP → acceptance decays geometrically with depth; llama.cpp default was 16 (catastrophic) → now 3. MoE wants n_max=2, dense wants 3.</p>
          </div>
        </div>
      </section>

      
      <section>
        <h2>3 · DFlash2 — Parallel Speculative Decoding <span class="tag p">z-lab / Inco AI</span></h2>
        <div class="m-2">
          <div class="c">
            <h3 style="color:#6d28d9">Block-diffusion drafter</h3>
            <p>Unlike sequential drafters, DFlash2 proposes the <b>whole draft block in one parallel pass</b> via a small block-diffusion network. A <b>path selector</b> keeps top-16 candidates per slot; a <b>two-tap conv</b> keeps coherence. Adds ~2M + 16.5M params, ~1.3% latency.</p>
            <ul style="font-size:.6em">
              <li>+21% acceptance length over DFlash (16–25% per case)</li>
              <li>Runs on SGLang, vLLM, llama.cpp (PR #27342), oMLX</li>
              <li>MLX build for Apple Silicon (Qwen3.8-27B)</li>
              <li>Depth/default block-size 5–8; small quality loss vs AR = 0</li>
            </ul>
          </div>
          <div class="c">
            <h3 style="color:#1a56db">Measured speedup</h3>
            <div class="code-box">
Setup                     Base    DFlash2    ×speed
A100 (Qwen3.8-27B)        28.9    59.1       ~2.0×
M5 Pro Q4_K_M            10.42   19.31      1.85×
RTX 5050 (8GB, starved)   6.8     11.6       1.7×
            </div>
            <div class="vis" style="font-size:.5em; margin-top:8px">
              <div class="bar"><span style="width:35%; background:#2b3038">baseline</span></div>
              <div class="bar"><span style="width:100%; background:#a78bfa">DFlash2 ~2×</span></div>
            </div>
            <p style="font-size:.5em; color:#6b7280">Biggest gain when baseline is compute/VRAM-starved.</p>
            <p style="font-size:.5em; color:#92400e">Gotcha: speculation is single-stream only — at concurrency ≥2 it can cost ~25% vs no-draft. Needs ~77GB VRAM for main+draft on A100.</p>
          </div>
        </div>
      </section>

      
      <section>
        <h2>4 · oMLX / MLX — Apple-Silicon Native Runtime <span class="tag">runtime</span></h2>
        <div class="m-2">
          <div class="c">
            <h3 style="color:#1a56db">Unified memory advantage</h3>
            <p>MLX (Apple ML Research) keeps tensors in <b>shared memory</b> — no CPU↔GPU copy. On 128 GB M-series, a 27B model sits fully resident and bandwidth-efficient, so quantized + MTP/DFlash overhead is amortized.</p>
            <div class="code-box">
<span class="cmd">pip install mlx-lm</span>
mlx_lm.generate --model Qwen3.8-27B-4bit --mtp
mlx_lm.server   --model Qwen3.8-27B-4bit --mtp
            </div>
            <p style="font-size:.58em">oMLX = optimized local MLX serving stack (e.g. Qwen3.5-4B-OptiQ on 127.0.0.1:8000).</p>
          </div>
          <div class="c">
            <h3 style="color:#1a56db">Why it matters for these methods</h3>
            <ul style="font-size:.6em">
              <li>Native <b>MTP</b> in mlx-lm (#990): 1.5× Qwen3.6-27B on M4 Pro</li>
              <li>DFlash2 ships an <b>oMLX build</b> for Apple Silicon</li>
              <li>Lazy eval + dynamic graphs → no recompile per shape</li>
              <li>Skip MTP below ~4B (overhead &gt; gain); 4B+ wins</li>
              <li>M1/M2 need <code>--dtype float16</code> (no native BF16)</li>
            </ul>
            <div class="flow">
              <div class="node">weights<br><small>unified</small></div>
              <div class="arrow">→ no copy →</div>
              <div class="node ok">GPU+CPU<br><small>same RAM</small></div>
            </div>
          </div>
        </div>
      </section>

      
      <section>
        <h2>5 · FlashAttention-2 — IO-Aware Kernel <span class="tag p">foundation</span></h2>
        <div class="m-2">
          <div class="c">
            <h3 style="color:#1a56db">The kernel everything rides on</h3>
            <p>Exact (no approximation) attention that tiles Q/K/V into SRAM, does online softmax, and <b>never writes the N×N score matrix to HBM</b>. Memory becomes <b>linear</b> in sequence length; the decode matmul stays bandwidth-bound, not memory-bound.</p>
            <div class="code-box">
FlashAttention : 2–4× faster, 10–20× less memory (linear)
FlashAttention-2: +2× again → up to 73% A100 FLOPs
  · split Q across warps (not K/V) → less shared-mem traffic
  · parallelize over sequence length
  · supports head_dim ≤ 256, MQA/GQA
            </div>
          </div>
          <div class="c">
            <h3 style="color:#1a56db">Impact on the other 4 methods</h3>
            <div class="vis" style="font-size:.5em">
              <div class="bar"><span style="width:100%; background:#2b3038">naive (quadratic)</span></div>
              <div class="bar"><span style="width:42%; background:#93c5fd">FlashAttn</span></div>
              <div class="bar"><span style="width:22%; background:#99e6f2">FlashAttn-2</span></div>
            </div>
            <ul style="font-size:.56em">
              <li>MTP/DFlash2 verify a <b>batch</b> of draft tokens — FA-2 makes that batched verify nearly free</li>
              <li>Long-context KV cache stays small enough to keep on-chip</li>
              <li>Underpins vLLM, SGLang, llama.cpp, MLX backends</li>
              <li>Up to 9× vs PyTorch attention; 225 TFLOPs/s GPT training</li>
            </ul>
          </div>
        </div>
      </section>

      
      <section>
        <h2>Method Map — Where Each Lever Hits</h2>
        <div class="m-3">
          <div class="card"><h3 style="color:#6d28d9">Shrink weights</h3><p style="font-size:.6em">Unsloth Dynamic 3.0 → fits 27B in 6–14 GB.</p></div>
          <div class="card"><h3 style="color:#1a56db">Draft tokens</h3><p style="font-size:.6em">MTP (self) + DFlash2 (parallel) → 1.5–2× decode.</p></div>
          <div class="card"><h3 style="color:#1a56db">Fuse kernel</h3><p style="font-size:.6em">FlashAttention-2 → linear memory, free batched verify.</p></div>
          <div class="card"><h3 style="color:#1a56db">Run natively</h3><p style="font-size:.6em">oMLX/MLX → unified memory removes copy cost.</p></div>
        </div>
        <p style="color:#6b7280; font-size:.58em">Stack them: <b>quantize → load in MLX → enable MTP/DFlash2 → rely on FlashAttn-2 under the hood</b>.</p>
      </section>

      
      <section>
        <h2>Synthesis — Qwen3.8-Flash: Stack the Stack</h2>
        <div style="border:1px solid #e6e8ec; border-radius:14px; padding:16px 20px; width:96%; max-width:1080px">
          <p style="font-size:.66em; color:#1f2329">The <b>Qwen3.8-Flash</b> class (next-gen small/fast variant) is the ideal target for the full stack. Recipe for a 27B-class model on a single Apple-Silicon or 24 GB GPU box:</p>
          <div class="m-2" style="margin-top:8px">
            <div class="c">
              <ol style="font-size:.58em; line-height:1.45">
                <li><b>Quantize</b> with Unsloth Dynamic 3.0 → <code>UD-Q4_K_M</code> (~13 GB) or <code>UD-Q2_K_XL</code> (~9.8 GB, best agent tier).</li>
                <li><b>Load</b> via oMLX / mlx-lm (unified memory) — no CPU↔GPU copies.</li>
                <li><b>Enable MTP</b> (<code>--mtp</code>); depth 1 on Metal, n_max 2–3 on CUDA → +1.5× decode.</li>
                <li><b>Add DFlash2</b> drafter when VRAM allows → up to +2×; pairs with MTP on llama.cpp.</li>
                <li><b>FlashAttention-2</b> is automatic in the backend — keeps long context cheap.</li>
              </ol>
            </div>
            <div class="c">
              <div class="code-box">
<span class="cmt"># Apple Silicon, 27B, ~2× throughput</span>
mlx_lm.server --model Qwen3.8-27B-4bit \
              --mtp

<span class="cmt"># CUDA / llama.cpp, chain drafters</span>
llama-server -m Qwen3.8-27B-UD-Q4_K_M.gguf \
  --spec-type draft-mtp --spec-draft-n-max 3 \
  --spec-draft dflash2-q8   <span class="cmt"># optional</span>
              </div>
              <p style="font-size:.5em; color:#6b7280">Expected: BF16-class quality at 4-bit, 1.5–2× faster, runs on a laptop or single GPU.</p>
            </div>
          </div>
        </div>
        <p style="font-size:.5em; color:#6b7280">Numbers from vendor/repro benchmarks (Unsloth, z-lab, mlx-lm, Dao et al.); real throughput is hardware- and prompt-dependent.</p>
      </section>

      
      <section>
        <h2>Validated Skills Library <span class="tag">installed &amp; scanned</span></h2>
        <p style="font-size:0.6em; color:#6b7280">All installed via <code>hermes skills install</code>, security-scanned, and verified (valid SKILL.md, enabled).</p>
        <div class="m-3">
          <div class="c"><h3 style="color:#1a56db">Web &amp; Research</h3><p style="font-size:0.6em">agent-reach · youtube-full · defuddle · resemble-detect</p></div>
          <div class="c"><h3 style="color:#6d28d9">Build &amp; Engineering</h3><p style="font-size:0.6em">using-agent-skills · setup-matt-pocock-skills · make-interfaces-feel-better · humanizer</p></div>
          <div class="c"><h3 style="color:#1a56db">Agents &amp; Ops</h3><p style="font-size:0.6em">browser-harness · i-have-adhd · loopy · loop-library · skillclaw</p></div>
        </div>
        <div class="code-box" style="font-size:0.56em; margin-top:10px">
<span class="cmt"># list active skills</span>
hermes skills list
<span class="cmt"># 14 validated community/url skills now active</span>
        </div>
      </section>

      
      <section>
        <h2>Command Cheat Sheet <span class="tag">DESK · CLI · MSG · SHELL</span></h2>
        <p style="font-size:0.5em; color:#6b7280">Everyday commands — Desktop, CLI chat, messaging, shell. (Verified vs current Hermes source, Sep 2026.)</p>
        <div class="m-3">
          <div class="c"><h3 style="color:#1a56db">Sessions &amp; Context</h3><p style="font-size:0.55em">/new · /resume · /sessions · /title · /branch · /compress · /context · /status</p></div>
          <div class="c"><h3 style="color:#1a56db">Control Work</h3><p style="font-size:0.55em">/queue · /steer · /bg · /btw · /agents · /stop</p></div>
          <div class="c"><h3 style="color:#1a56db">Goals, Loops &amp; Plans</h3><p style="font-size:0.55em">/goal · /subgoal · /heartbeat · /loop · /plan · /review · /refine</p></div>
          <div class="c"><h3 style="color:#6d28d9">Models &amp; Behavior</h3><p style="font-size:0.55em">/model · /moa · /personality · /reasoning · /fast · /approvals · /yolo · /busy · /voice</p></div>
          <div class="c"><h3 style="color:#6d28d9">Skills, Memory &amp; Tools</h3><p style="font-size:0.55em">/skills · /learn · /memory · /init · /tools · /browser</p></div>
          <div class="c"><h3 style="color:#6d28d9">Automation &amp; Coord</h3><p style="font-size:0.55em">/cron · /suggestions · /blueprint · /kanban</p></div>
          <div class="c"><h3 style="color:#1a56db">Inspect, Recover &amp; Fix</h3><p style="font-size:0.55em">/retry · /undo · /save · /diff · /rollback · /usage · /debug · /help</p></div>
          <div class="c"><h3 style="color:#be185d">Messaging / Gateway</h3><p style="font-size:0.55em">/sethome · /topic · /commands · /approve · /deny · /pause · /platform · /restart</p></div>
          <div class="c"><h3 style="color:#1a56db">Terminal Essentials</h3><p style="font-size:0.55em">hermes / hermes chat · -z "prompt" · model · status · doctor · gateway status · --safe-mode · update</p></div>
        </div>
        <div class="code-box" style="font-size:0.52em; margin-top:8px">
<span class="cmt"># repeat-work modes</span>
/goal  = work until objective met   /loop  = repeat w/ stop conditions
/heartbeat = one recurring check    /cron  = durable schedule outside chat
<span class="cmt"># type / + letters for autocomplete; /help for full list</span>
        </div>
      </section>

      
      <section>
        <h1>Thank You & Discussion</h1>
        <p style="color:#6b7280; font-size:.7em">Quantize smart · Draft in parallel · Fuse the kernel · Run native</p>
        <p style="font-size:.6em; margin-top:8px"><a href="/farshid/content/research-tools.md" style="color:#1a56db">← All presentations</a> · <a href="/farshid/content/presentation-research-tools.md" style="color:#6d28d9">New Era of Research Tools</a></p>
      </section>

    </div>
  </div>
</div>