Common rules — every measurement, every script

These rules bind every test on every page. Test-specific steps live in the per-test pages; the run loop lives in the checklist. The speed test's own rules (prompts, timings, prompt-cache reuse, the KV type decision) live in context creep.

1. One model on the GPU at a time. Fresh server start per configuration. Long runs go in background tasks.
2. Warmup first. Send a discarded warmup request; measure only after it.
3. Read the server's own numbers, not wall clocks, where the server has them. Each test page says which field to read on which backend.
4. KV cache type is a per-model decision, never a default. f16 and q8_0 are the candidates. q4_0 is banned for quality, with one exception: a vendor ships a per-model calibration for it (a KV bias file), and such a config must pass the EvalPlus gate before serving. The decision is its own test, KV cache pick, and it runs before any other measurement of the model. Rows measured at the other type before a pick existed are hidden until re-measured.
5. API-or-nothing. A config qualifies only if it serves an HTTP API a harness can use. CLI-only inference paths are disqualified.
6. Keep thinking-on AND thinking-off data, both labeled — never replace one with the other. The target setup is mixed: main agent thinks, sub-agents run thinking-off. Report tables show thinking-on; thinking-off goes in note text.
7. Record each result on every surface in the same pass — a result is not recorded until all agree: the model's docs/setups/<setup>/benchmarks/*.md (full data), its docs/setups/<setup>/reports/*.md page including the summary line (it goes stale easily), the setup's comparison.md, the generated tables (models.json + node tools/gen-tables.mjs), and the harness config (npm run pi:models writes the pi entries from the same models.json; --dry-run and --check show the drift first, and it never touches a field it does not own). Every server config gets a copy-paste command block in its report whose alias equals the harness model id. The report and comparison pages show only numbers measured under the CURRENT wired limit; superseded measurements move to the setup's historical.md (benchmarks pages keep the full archive).
8. Run the exact files the runbook names, at the revision and quant it names. A missing or different file is stop-and-ask, never a substitute. Whether a run may download, and what, is decided when the run is planned and written into its runbook; a runbook that says nothing means no download. Exactness stops at identity: the files, the backend, the thinking level, the prompt version, the KV type once picked. The values measured around them are rule 10.
9. Measured parameters come from the newest measurement, never from a planning snapshot. The serving -c, the harness window, the output budget, the drafter depth and every gate input are measurements. A runbook gives the value known when it was written, with its date and source. A newer measurement under the same identity, made in the same run or committed since, replaces it, and the config note names the value and its source. A value frozen on purpose is a dated owner decision with its reason, written as such. Run 11 measured an 82K clean depth in its first block and ran its Mendel rows on a 49152 window because the runbook froze the window; this rule exists so that does not repeat.
10. After tests, check for leftovers and clean up (the checklist has the commands). Do not delete model files or tools early — keep variants for debugging until many successes.
11. The drafter is a speed and memory decision, never a quality one, and it is per build. A speculative decoder accepts a draft token only when it matches the token the target model would have picked, so it preserves the model's output distribution. At temperature 0 that means identical text, which is why a drafter setting never earns its own EvalPlus arm: that test runs at temperature 0. Above temperature 0 it means the same distribution, not the same run. Two agent runs that differ only in drafter setting are two samples, so neither confirms nor refutes a quality difference, and neither is worth spending an agent run on. What the setting does change is decode speed, memory, and therefore the depth the model reaches. Dropping the drafter frees memory that becomes KV cache, measured at about 2 GB on one dense 27B model, which at f16 is roughly 32K more tokens of window. That is the trade to price: shallow speed against depth. Decide it from a creep and a sweep, per build. The head ships inside the build, so its acceptance rate belongs to that build and never carries over from another build of the same model. The sweep runs in both directions and includes no drafter at all. A larger n-max can only buy speed; a smaller one, and dropping the drafter, buys memory and therefore window. Sweeping upward alone answers half the question, which this project did once and had to redo. A creep's window is only valid for the drafter setting it ran with. Record that setting beside the depth, and serve the same setting in every block that uses the window.