last_modified_at: 2026-09-05

Local LLM Optimization on Apple Silicon — Run the fastest small reasoning model inside Hermes Agent at 64K context on Apple M3 (16GB) using MLX. — https://www.pirahansiah.com/notes/docs/llm/local-llm-optimization/

Local LLM Optimization on Apple Silicon (M3, 16GB)

Target machine (verified live): Apple M3, 16GB, macOS 27. Goal: run the fastest small reasoning model inside Hermes Agent with 64K context, using the latest Apple-native acceleration (CPU + GPU + NPU via MLX).

Last updated: 2026-09-05.

Current live state (this machine, Sep 5 2026)

• Hermes → oMLX (port 8000). Default model Qwen3.5-4B-OptiQ-4bit (since Sep 5 2026); Qwen3.5-0.8B-OptiQ-4bit also served (fallback / small-quick). Both MLX safetensors, 64K context via the q35 custom provider.
• Ollama is up but unused by Hermes: ollama serve runs on :11434 but Hermes’s base_url points at oMLX on :8000.
• Hermes gateway runs as a headless daemon (launchd ai.hermes.gateway). Telegram bot works with no terminal open.
• oMLX auto-starts via launchd: com.farshid.omlx-serve.plist (RunAtLoad + KeepAlive) holds :8000 across reboots/logout.

Bottom line (measured on this M3)

Key insight: Qwen3/3.5 are reasoning models. Out of the box they burn the whole prompt budget on thinking and return an empty final message if you don’t give them room or suppress thinking. oMLX makes thinking suppression trivial via chat_template_kwargs.

What “all processors” means on Apple Silicon

• MLX (mlx-lm): schedules on CPU + GPU + ANE (Neural Engine).
• llama.cpp with Metal: CPU + GPU only, no NPU.
• oMLX is built on mlx-lm → full MLX device coverage (CPU + GPU + NPU).

If you specifically need NPU: use the MLX/oMLX path. llama.cpp cannot touch the Neural Engine.

Install oMLX (macOS app; precompiled kernels included)

# Recommended: official DMG (ships precompiled MLX custom kernels, no Xcode build)
curl -L -o /tmp/oMLX.dmg \
  "https://github.com/jundot/omlx/releases/download/v0.6.4/oMLX-0.6.4-macos26-27.dmg"
hdiutil attach /tmp/oMLX.dmg -nobrowse
cp -R /Volumes/oMLX/oMLX.app /Applications/
hdiutil detach /Volumes/oMLX

CLI entry points:

/Applications/oMLX.app/Contents/MacOS/omlx-cli        # CLI
/Applications/oMLX.app/Contents/Resources              # bundled Python + MLX

Serve a model with oMLX (64K context, continuous batching, paged KV cache)

omlx-cli start
# Or run the CLI server directly:
/Applications/oMLX.app/Contents/MacOS/omlx-cli serve \
  --model-dir ~/.omlx/models \
  --port 8000 --host 127.0.0.1 \
  --log-level info

oMLX enables the optimization stack by default: continuous batching, tiered/paged KV cache (hot RAM + cold SSD), LRU multi-model memory + memory guard, prefix sharing + Copy-on-Write blocks.

Suppress thinking (why many Qwen3.5 results come back empty)

Per-request, in the OpenAI-compatible body:

{ "model": "Qwen3.5-0.8B-OptiQ-4bit",
  "messages": [...],
  "max_tokens": 256,
  "chat_template_kwargs": { "enable_thinking": false } }

Hermes integration

Edit ~/.hermes/config.yaml:

model:
  default: Qwen3.5-4B-OptiQ-4bit
  provider: custom
  base_url: http://127.0.0.1:8000/v1   # was http://127.0.0.1:11434/v1 (Ollama)

Then restart the gateway: hermes gateway restart. Verify: hermes status → Model Qwen3.5-4B-OptiQ-4bit, Provider “Custom endpoint”.

Alternate path — llama.cpp (Metal) full optimization

llama-server -m /path/Qwen3.5-0.8B-Q4_K_M.gguf \
  -c 65536 -ngl 99 -fa on -ctk q8_0 -ctv q8_0 \
  -b 1024 -ub 1024 --cache-reuse 256 -t 8 \
  --alias qwen3.5-0.8b-q4 \
  --port 8080 --host 127.0.0.1

• -ngl 99 = all layers on GPU (Metal device MTL0: Apple M3, 12GB)
• -fa on = flash attention
• -ctk q8_0 -ctv q8_0 = quantized KV cache → biggest long-context memory win

Persistence — launchd daemon (survives reboot / logout, no terminal needed)

~/Library/LaunchAgents/com.farshid.omlx-serve.plist — RunAtLoad + KeepAlive:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.farshid.omlx-serve</string>
    <key>ProgramArguments</key>
    <array>
        <string>/opt/homebrew/bin/omlx</string>
        <string>serve</string>
        <string>--model-dir</string>
        <string>/Users/farshid/.omlx/models</string>
        <string>--host</string>
        <string>127.0.0.1</string>
        <string>--port</string>
        <string>8000</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <true/>
</dict>
</plist>

Install/load:

launchctl load -w ~/Library/LaunchAgents/com.farshid.omlx-serve.plist
launchctl list | grep omlx
curl -s http://127.0.0.1:8000/v1/models

RAM math (16GB unified)

• 0.8B MLX OptiQ-4bit ≈ 0.7 GB weights + KV (oMLX server ~1.2 GB RSS)
• 4B MLX OptiQ-4bit ≈ 3 GB (KV grows; keep 64K but watch total)
• llama.cpp 1.7B Q8 ≈ 1.8–2.2 GB at 64K with quantized KV

Never let context default to a model’s full 262K on 16GB (pre-set KV eats RAM you don’t have).

Add a new model to oMLX (and the GGUF exception)

oMLX has no add/pull subcommand — it discovers models from subdirectories of ~/.omlx/models (default --model-dir). Each subdirectory must be a valid MLX safetensors model: config.json + *.safetensors (+ optional chat_template.jinja).

Hard rule: oMLX eats MLX safetensors ONLY. A .gguf is not ingestible by oMLX. If the source is a GGUF (e.g. the K2-Horizon GGUF), serve it with the llama.cpp fork on a separate port instead (see the K2-Horizon local page).

Add an MLX safetensors model:

MODEL_ID="Qwen3.5-4B-OptiQ-4bit"   # dir name = the id Hermes references
DEST="$HOME/.omlx/models/$MODEL_ID"
mkdir -p "$DEST"
APP_PY=/Applications/oMLX.app/Contents/Resources/Python/cpython-3.11/bin/python3
PYTHONHOME="$(dirname "$(dirname "$APP_PY")")" PYTHONPATH="$(dirname "$APP_PY")" \
  "$APP_PY" - "$MODEL_REPO" "$DEST" <<'PY'
import sys, os
from huggingface_hub import snapshot_download
repo, dest = sys.argv[1], sys.argv[2]
os.makedirs(dest, exist_ok=True)
snapshot_download(repo, local_dir=dest,
                  allow_patterns=["*.safetensors","*.json","*.txt","tokenizer.*"])
print("downloaded to", dest)
PY
curl -s http://127.0.0.1:8000/v1/models | python3 -m json.tool

Then set Hermes model: $MODEL_ID, base_url: http://127.0.0.1:8000/v1 (see Hermes integration above). If it is a reasoning model, also pass enable_thinking per §”Suppress thinking”.

Gotchas when adding models:

• Format mismatch = #1 failure. config.json + *.safetensors for oMLX; .gguf for llama.cpp.
• Hermes enforces a 64K context floor for the default model. If the server reports < 65536 (--ctx-size / n_ctx_slot), Hermes refuses it. Set --ctx-size 65536 or higher.
• One port, one process. Only one server may hold :8000 (oMLX) or :8080 (llama.cpp). Kill any hand-started instance before launchctl load-ing the plist.

Download the files

All three source files are available for download — the full guide, the prompt/keyword bank, and the auto-detect startup script.