EvalPlus (HumanEval+) — the quality gate

Tier 1 of the quality flow: cheap, execution-verified, sensitive to quantization damage. A gate, not a ranking. Survivors go to Mendel. Common rules and the run loop apply (common rules, checklist).

Gate mechanics

• Temperature 0, pass@1, small prompt context (problems are tiny — prompt context does not affect scores).
• Serve through the config you will actually run.
• Which thinking modes and reasoning levels to score is a planning decision per model, from the vendor's documentation and the owner.
• Speculative decoding never changes outputs at temperature 0, so score without a drafter and serve with one freely.
• Timing of runs is a secondary signal; never chase precision. pass@1 is what matters.

Calibrate the output budget FIRST — it affects scores

max_tokens is a separate axis and it DOES affect scores. An undersized budget lets reasoning exhaust the cap and empty completions score as failures (up to 38% of scores lost before this was found).

1. Run benchmarks/calibrate.py (10 fixed problems, cap 30000).
2. Budget = observed max completion × 1.5, floor 8192.
3. For models whose thinking sometimes never converges (finish_reason: length at any budget), the budget is a waste-limiter instead: set it just above the longest SUCCESSFUL completion. Expect and record a real empty rate; do not chase zero empties with ever-larger budgets.
4. Never reuse a thinking-on budget for a thinking-off pass, or across models.

Which serving config to score

The fastest at shallow depth, always. The problems are short and nothing here reaches depth, so a larger window buys nothing and speed is the whole cost of the gate. Where a model has a drafter, that is normally the drafter on, at the n-max its shallow sweep picked (context creep, "The order for a model with a drafter"). An agent run picks differently, on depth; the two tests do not have to serve the same config, and each row says which it used.

A non-converging calibration is this gate's early warning. It costs ten problems to see, and a full run costs hours, so read it before starting one: two length stops in a calibration have preceded a run that spent its budget and returned empties.

Steps

1. Calibrate (above). Calibration files live under the setup, at hardware/<hardware-id>/calibrations/, because a calibration is a measurement of one machine. calibrate.py takes the directory from CALIBRATION_DIR. Pass the thinking mode or reasoning level explicitly on every call, in the extra-body argument, for the calibration and for the full run alike. It is not optional: a call with no extra body gets the chat template's own default, which can be a different level from the one the file is named for. Every row records requested_extra_body and resolved_reasoning_effort; check that the resolved value matches the file name before you read the budget.
2. Start the config's server on port 8081, warm up, start the run watcher (benchmarks/run-watch.sh, checklist step 6: the memory record and the crash signal, exit 42 on a dead server).
3. Run the scoring script (RESULTS_BASE chooses the run dir; the extra body carries chat_template_kwargs for thinking toggles):

   RESULTS_BASE=hardware/m1-max-32gb/benchmarks/benchN/results \
     EVALPLUS_MAX_NEW_TOKENS=BUDGET \
     benchmarks/run-humaneval.sh RUN_NAME MODEL_ID_AS_SERVED [extra-body-json]

4. The script resumes from an existing jsonl automatically (skips existing task_ids). Strip genuinely-empty lines first if they must regenerate.
5. Monitor per the checklist (output growth, not process liveness).
6. Evaluate runs automatically at the end. Record pass@1 base/plus AND the empty count, honestly, on every surface.

The smoke

A fast fixed subset, for two uses: the KV cache pick confirms its candidate with it, and a research run tries a candidate container with it. Neither runs this gate: that is bench work and costs an hour or more per config. They run benchmarks/evalplus-smoke.py instead — four fixed HumanEval+ problems, the same output budget on both sides, once against the config we run today and once against the candidate.

What it can say, and only this: the candidate is level, better or worse on four problems. What it cannot say: a score. The smoke never produces a pass@1 and never reaches the site. Four problems have no resolution to separate two good configs a few tenths of a point apart; they separate a broken config from a working one. A candidate that survives the smoke still has to pass this gate before any number is published.

The subset is fixed in the tool and is not a parameter. Three problems are short and every scored config passes them, so a failure means something changed. The fourth is the problem with the most empty completions on our configs, so the smoke also sees the completion failure mode, not only the wrong-answer one.

The budget is the current config's, on both sides, and the candidate is never calibrated. A candidate that needs a bigger budget to pass is a candidate that costs more.

SMOKE_CALIBRATION=hardware/<hardware-id>/calibrations/calibration-CURRENT_CONFIG.json \
  benchmarks/evalplus-smoke.py LABEL MODEL_ID_AS_SERVED [extra-body-json]

Run it once per side, one at a time, and compare the two SMOKE lines. Level is the same passed and the same empty; better is a higher passed with empty no higher; worse is a lower passed, or an equal passed with a higher empty. Any other mix is not a verdict. A difference of one problem is one problem out of four: never write it as a percentage, and never write it beside a pass@1 number.

Harness patches (do not rediscover these)

EvalPlus 0.3.1 needs local patches, all live in benchmarks/run_codegen_wrapper.py + one venv file: token budget, extra_body passthrough, None-content handling, no signal.alarm + 7200 s client timeout (EvalPlus's own 100 s alarm made long completions retry forever), macOS rlimit in the venv. History in hardware/m1-max-32gb/benchmarks/bench1/state.md and hardware/m1-max-32gb/benchmarks/bench2/state.md.