Michael-AI · RTX 3090 Ti · local model routing

Local model tests, turned into a routing decision.

The useful answer is not just “which model scored highest?” It is “which model fits each job without running out of context, VRAM, or strict-output reliability?”

Latest evals: 2026‑06‑08 · QAT+MTP 2-lane speed lane
🏆 Current local runtime

Gemma 4 26B QAT+MTP · 2×128k live

The active local endpoint is now the QAT lane with a QAT-specific MTP drafter: two concurrent 128k slots, vision enabled, and verified draft-token acceptance. It is a 2-lane speed configuration, not yet a 3-lane default.

2×128klive slots
MTP n=1draft depth
~20.5GBVRAM used live
+11.6%2-lane sweep gain

Latest full-suite contenders

Scores use the latest 93-case suite unless noted. The quality layer is intentionally hard.
Live on :8080

Gemma 4 26B QAT+MTP

Best current two-lane speed setup. QAT target plus QAT-specific MTP drafter, served as 2 concurrent 128k slots with mmproj vision enabled.

63/93
QAT score
2×128k
live ctx
20.5GB
live vram
Agent
8/9
Research
9/10
Coding
0/7
Short quality

Gemma 4 26B Q5

Standing worker and best short-context quality baseline, but 32k context is cramped for real long-context state.

64/93
score
32k
ctx
21.9GB
vram
Quality
6/18
Speed
64 t/s
Headroom
~0.9GB
Cheap long lane

Gemma 4 12B QAT

Very roomy 262k extractor candidate. Good enough to keep, but 26B QAT now beats it as the main long-context pick.

63/93
score
262k
ctx
10.0GB
vram
Speed
50 t/s
Headroom
12.8GB
Quality
4/18

Decision ledger

Current choices plus older candidates that were ruled out or kept for narrow use.
Model
Context
Score
Speed
Status
Why
Gemma 4 26B QAT+MTP n=1
live two-lane speed route
2×128k
60/93*
+11.6%
Live 2-lane
MTP is active and accepted draft tokens; improves 2-lane aggregate throughput, but not 3-lane.
Gemma 4 26B QAT UD-Q4_K_XL
no-draft baseline
262k
63/93
71.5 t/s
3-lane baseline
Best context/speed/VRAM compromise; preferred if running three lanes because MTP overhead ties/loses there.
Gemma 4 26B Q5_K_M
standing worker
32k
64/93
64.3 t/s
Keep baseline
Best short-context quality, but tiny context and almost no VRAM headroom.
Gemma 4 12B QAT
262k
63/93
50.4 t/s
Cheap extractor
Very roomy; now secondary to 26B QAT for long-context use.
Gemma 4 12B Q8
262k
64/93
33.2 t/s
Comparison only
Slight score lift, much slower and more VRAM than 12B QAT.
Gemma 4 31B QAT
32k
64/93
24.7 t/s
Not promoted
Deterministic gain, but slow and hard quality still bad.
Qwen3.6 27B Q4_K_M
32k
56/93
23.5 t/s
No route
Fenced diffs; weak hard judgment/planning/debugging.
Qwen3.6 35B A3B
32–65k
60/93*
65 t/s*
Research only
VRAM-tight; thinking/template issues; not strict-tool safe.
Kimi-Linear 48B A3B IQ3_XS
16k
52/93
64.2 t/s
Prune / alternate only
Interesting architecture, but weak eval and default KV caveat.
Qwen3-Coder 30B A3B AWQ
24k
older
vLLM
Strict specialist
Best strict/tool specialist; switching tax and context make it non-default.
Carnice V2 27B Q4_K_M
16k
bounded
38–39 t/s
Niche JSON
Passed small strict envelope checks, but slower and not proven on full suite.
GLM AWQ/GPTQ / GLM-4.7
varies
rejected
—
Deprioritized
Older local GLM tests poor; newer GLM-4.7 too large for 24GB workflow.
* QAT+MTP score is the 32k full-suite compatibility run; the lane-sweep result is about throughput. Qwen3.6 35B summarizes multiple runs: latest 35B MoE score was 60/93 at 32k; older ik_llama 65k testing produced useful output but still had strict-output issues.
Live serving note

What changed on :8080

  1. The live route is now QAT+MTP at -np 2. The drafter uses --spec-draft-n-max 1, which means one speculative token of draft depth, not one serving lane.
  2. Two real 128k lanes require -c 262144 -np 2. llama.cpp splits total context across slots; the endpoint reports ~131k context per slot.
  3. Vision still works with explicit mmproj. The MTP standing worker advertises completion plus multimodal capability.
Concurrency read

What the lane sweep says

  1. 2 lanes benefit from MTP. The lane sweep showed about +11.6% aggregate throughput with MTP n=1/n=2.
  2. 3 lanes do not benefit yet. Baseline no-draft at -np 3 tied or beat MTP because draft overhead consumed the gain.
  3. To unlock 3 lanes, test a smaller drafter. More n_max tuning is not the likely lever.
Takeaway

What matters

  1. Context beat tiny quality gains. Q5’s 32k window is the practical bottleneck, even though it remains the best short-context baseline.
  2. 26B QAT+MTP is the current 2-lane speed lane. Same QAT target, QAT-specific drafter, 2×128k fit, and verified draft-token acceptance.
  3. Reliability limits remain. Hard quality and coding-diff failures still need validators and review for consequential work.
Routing rule

Simple routing rule

  1. Two active local sessions: Gemma 26B QAT+MTP, currently 2×128k on :8080.
  2. Three active local sessions: Gemma 26B QAT no-draft until a smaller/faster drafter proves out.
  3. Short synthesis / quality baseline: Gemma 26B Q5 @ 32k.
  4. Strict structured code/tool tasks: Qwen3-Coder vLLM only when the task justifies model-switch overhead.
  5. Everything else: keep local models assistive, not autonomous.