Local models · dual DGX Sparks · TP2

How the household models actually score

Not a synthetic leaderboard. This is every Sparks build we have measured on Léo Laporte’s Hermes agent harness — real tools, mail and calendar cases, buried-needle reads, fairness under load, and decode speed — as of 16 September 2026.

Columns are model + quant + recipe. Rows are tests. A dash means we did not run that test on that build, never a fail. Gold cells are the best measured value in the row.

How to read this

Every number comes from the same dual NVIDIA DGX Spark cluster, tensor-parallel 2. The agentic corpus is 78 cases at effort low with a 4,096-token floor. Where a build ran more than once, the cell shows min–max across arms (run-to-run noise is about ±2 cases). “Carried” means the number is from an earlier run of the same weights on this recipe lineage.

GLM · Mia (HEAD → rung C)

GLM-5.3-Flash EXL3 4 bpw. Mia recipe lineage from 9 Sep through fair-v5 rung C (15 Sep). Parked 16 Sep.

GLM · Reederey

Same weights, Reederey87’s self-built ExLlamaV3 1.4.7 recipe. Bake-off 10 Sep, not adopted.

GLM · Mia c190db1

The 1–7 Sep resident (MTP k=2). Four corpus arms on an older harness.

DSV4.1 · EXL3

DeepSeek-V4.1-Flash EXL3 2.9 bpw, shipped recipe, text-only. 13 Sep.

FN · NVFP4

Qwen3.8-Flash-Next, NVIDIA NVFP4 checkpoint, Mia dual recipe. 13 Sep.

FN · FP8 resident

Qwen3.8-Flash-Next, Qwen’s official FP8 checkpoint. Resident since 16 Sep, test drive.

Table A — agentic corpus, 78 cases

Pass / cases, by suite. Bar ≥ 59 on the total. Visual strip uses the high end of any range.

Qwen edges GLM on general knowledge (10/10) and once on coordination. The official FP8 checkpoint does not change Qwen’s shape: +1 total, a reshuffle of the same suites.

Table B — the cases that separate the models

Pass / fail, or k/n across arms. This is where GLM’s quality lead actually lives.

† c190db1 arms ran on harness b24e30c before the needle refit; the 200K needle there is not comparable.

Table C — other quality and reliability gates

Reference points on the same coding axis: frontier pair (Kronk / Pi) = 100; Mac mini qwen3-coder = 53.5. Effort vocabularies differ by build — a rejected medium or high is a 400, not a quality fail.

Table D — speed and capacity

Tests down, models across. Decode is tokens/s. Prefill is wall time to first content. Fairness bar is a short request waiting behind a long read, target < 10 s. Newcomer bar is first token of a 5.5K prompt ten seconds into another thinking reply, target < 20 s.

‡ Reederey 2b5dfdc single-stream thinking figure carried from the 10 Sep session summary; approximate. Only fair-v5 GLM (rung C) is under 5 s behind a long read and under 15 s for a newcomer during decode. Qwen FP8 sits right at the 10 s fairness line.

Table E — late-August GLM serving lineage

Older harness; corpus at high effort; rob-01 at 2048 and 4096. Kept so the September board has a floor.

Table F — checked, but no comparable numbers

Reading the board — 16 September 2026

• Quality: GLM wins, and the gap is in two rows. Precision traps (gate-a3: GLM 6/10, Qwen 3–4, DeepSeek 2) and long-context needles (GLM 3/3, Qwen FP8 1/3, NVFP4 0/3). Qwen edges GLM on general knowledge (10/10) and the coordination suite once; everything else is inside noise.
• Speed: Qwen NVFP4 wins every throughput row and holds 12× the KV. FP8 gives back a quarter of the ×4 and two-thirds of the KV for that +1 corpus point. DeepSeek-V4-Flash was the August decode king; DeepSeek-V4.1 EXL3 is the slowest thing on the board.
• Fairness and newcomer wait: only the fair-v5 GLM (rung C) is under 5 s behind a long read and under 15 s for a newcomer during decode. Qwen FP8 sits right at the 10 s line.
• The live decision: Qwen FP8 (resident, test drive) vs GLM rung C (parked). Table A row 1, Table B needles, Table D decode — that is the whole trade.