ModelSaar

Benchmarks and evidence

Does automatic routing help? What one recorded run shows: the same task score as the strongest single model, better coverage, higher tail latency. Nothing more is claimed. Everything on this page can be regenerated from files in the repository.

Method

modelsaar eval compares auto (live routing through the gateway) against policies that use the same requests, the same gateway and the same budgets:

The task set is 47 original synthetic tasks (conversation, coding, agent/tools, math, extraction with strict schemas, summarisation, translation, RAG-style, decisions, image questions, a tool round trip, mixed-language and deliberately ambiguous prompts), split 15 calibration / 32 test with different templates. Scoring is executable or exact: numbers with tolerance, strict tool-call validation, JSON-schema checks, and Python tasks run inside an isolation sandbox (bubblewrap, macOS sandbox-exec or Docker) or skipped with the changed denominator reported. No LLM judge. Full definition: Evaluation and the fixture README.

Recorded run — 2026-10-11, three models on an ASUS GX10

Deployments: qwen3.6:latest (35.5B Q4_K_M), gemma4:31b (31.3B Q4_K_M), qwen3-coder:latest (30.5B Q4_K_M) served by Ollama 0.32.13; 15 calibration / 32 test tasks; temperature 0, max_tokens 400, seed 7, scorer 2026.10.11-2, code isolation sandbox-exec, gateway at commit 0f8af84b66df. Success is the share of the 32 held-out tasks scored correct; completion is the share that returned HTTP 200; a violation is a 422 for a request some deployment could have served; latencies are end-to-end per request.

policysuccess95 % CIcompletionviolationsp50 msp95 ms
auto (live routing)0.938[0.80, 0.98]1.00005987,958
fixed-strong (calibrated)0.938[0.80, 0.98]0.96916394,760
fixed-fast (calibrated)0.938[0.80, 0.98]0.96916394,760
rules (gold label, specialist tags)0.812[0.65, 0.91]1.00009624,859
random (diagnostic)0.719[0.55, 0.84]1.00004,32024,356
oracle (offline upper bound, not available at runtime)1.000[0.89, 1.00]1.00006394,671

Reading it. Automatic routing and fixed-strong each scored 30 of 32. Automatic routing completed 32 versus 31 requests (the image task the fixed model cannot take), recorded zero versus one capability violation, and had a p95 end-to-end latency of 7.958 s versus 4.760 s because two requests went to a slower thinking model. The two policies differ on exactly two tasks: the image task, solved by automatic routing, and long-01, whose result differed despite selection of the same model; the cause has not been isolated.

What it does not show. n = 32 and every 95 % confidence interval overlaps every other; equal point estimates do not establish equivalence, and nothing here shows general superiority. The 400-token budget systematically penalises the two thinking models (their reasoning counts against it), so the fixed model was also the one that happened to suit the budget; a matched larger budget has not been run. One machine, one runtime, three of eight available models, synthetic tasks.

Routing overhead — the time the gateway spends deciding, which is separate from end-to-end response latency — was mean 0.67 ms, p95 0.98 ms over the 32 auto requests, with the deterministic task label matching the gold label 84 % of the time and the classifier disabled.

Raw evidence and reproduction