Eval suite · what it measures · updated 2026‑06‑07

What the benchmark is actually testing.

The suite is a routing test: can a model follow instructions, preserve source truth, resist injection, emit strict tool/diff formats, handle long context, and make reliable agent decisions?

93 core cases · plus vision/runtime smoke checks

Two layers, one decision: where does this model fit?

The first 75 cases are deterministic gates: exact output, JSON shape, no forbidden text, tool traces, fact recall, and diffs that must apply. The v3 quality layer adds 18 harder rubric-scored cases for reasoning, judgment, debugging, planning, and synthesis. Separate smoke checks now cover vision enablement, slot sizing, and live serving shape.

75deterministic
18quality
6vision smoke
2×128klive slots

Suite map

Bars show case counts; the quality layer carries extra scoring weight.
Instruction

Smoke

Basic exactness, simple JSON schema, regex, and “don’t leak forbidden token” checks.

12
cases
12
weight
Agent

Agent core

Source-grounded recall, caveats, contradictions, priority selection, schema output, secret refusal.

9
cases
9
weight
Tool use

Tool calling

Tool choice, terminal/read/search behavior, recovery, multi-tool order, forbidden tools, injection.

9
cases
9
weight
Long context

Long-context fidelity

Needle/fact retrieval, contradictions, negative evidence, JSON, list discipline, injection-in-context.

10
cases
10
weight
Research

Research synthesis

Grounded summaries, caveats, unsupported claims, contradictions, ranked evidence, schema output.

10
cases
10
weight
Injection

Injection resistance

Forbidden disclosure, source hierarchy, roleplay, encoding, JSON/tooltext conflict, instruction priority.

8
cases
8
weight
Reasoning

Reasoning

Sequences, logic, constraints, tables, arithmetic, caveats, ordering, and one raw JSON case.

10
cases
10
weight
Coding

Coding diffs

Seven patch outputs that must be unfenced and actually apply.

7
cases
7
weight
V3 hard layer

Quality rubrics

Reasoning depth, agent judgment, debugging diagnosis, planning tradeoffs, and grounded synthesis.

18
cases
62
weighted
Vision smoke add-on

Multimodal is tested separately

  • QAT exact-file needs mmproj. Without --mmproj, image requests return HTTP 500 and no vision path is active.
  • 6/6 with-image smoke passed. Shapes, OCR, spatial relation, chart, receipt, and image-over-text contradiction all passed.
  • Missing-image controls matter. 4/6 controls correctly refused or asked for an image; 2/6 hallucinated plausible image content.
Runtime smoke add-on

Serving shape is part of the test

  • Context is split by slots. For 2×128k, launch with -c 262144 -np 2; -c 131072 -np 2 is only 2×64k.
  • Live /slots verified. Slot 0 and slot 1 each report n_ctx: 131072.
  • Health is endpoint + smoke. /v1/models, /slots, GPU residency, and exact response QAT-128K-OK all passed.
Scoring philosophy

Deterministic where possible

  • Exact / regex / contains: cheap checks for instruction following.
  • JSON exact / schema: rejects markdown fences and sloppy structure.
  • Forbidden: catches leakage and unsafe compliance.
  • Tool trace: evaluates requested tool actions, not pretty explanations.
  • Diff applies: patches must be raw, unfenced, and usable.
Why the hard layer exists

Harder cases prevent false confidence

  • Mechanical gates were too easy. Several models could pass format checks while still making weak decisions.
  • Rubrics expose partial competence. A model can name the decision but miss the blocker, caveat, or minimal reversal.
  • Hard tasks match real use. Safety gates, debugging diagnosis, tradeoffs, evidence weighting, and “don’t invent facts.”

Quality layer breakdown

18 rubric cases: 10 hard, 8 adversarial.
Category
Cases
Difficulty
What failure looks like
Reasoning hard
5
hard/adversarial
Overweights tempting facts, misses necessary conditions, invents certainty.
Agent judgment
5
hard/adversarial
Asks when it should act, ignores explicit safety gates, or misses the useful next step.
Debugging diagnosis
3
hard
Jumps to a fix without the cheapest discriminating test or confounds benchmark causes.
Planning tradeoffs
3
hard
Produces plausible plans that violate constraints or hide the real decision.
Research synthesis hard
2
adversarial
Flattens conflicting evidence, cites weak signals as proof, or loses caveats.
Interpretation

Use scores as routing evidence

  1. Route by job type. A high score does not mean one model should do everything.
  2. Keep guardrails on hard tasks. Coding diffs, planning, debugging, and safety-sensitive calls still need validation.
  3. Compare only clean runs. Shared VRAM, CPU-bound runs, or different context settings distort results.