EvalPlus (HumanEval+) — M1 Max 32 GB

The quality gate: pass@1 at temperature 0, output budget calibrated per model — see the method. Scores are shared across serving configs when thinking mode, effort, and quant match.

These runs are shallow, a few thousand tokens each, so the KV cache type does not move a score and the rows name it only where it is part of the quant, as the fork's calibrated q4_0 KV is. Each model page names the KV type its config serves.

config	budget	pass@1 base	pass@1 plus	empty	completion
Qwen3.8-27B 4-bitmlx-community,mlx_lm.server–f16medium	8192	0.982	0.939	0/164	100%
Qwen3.8-27B IQ3_S-mtpISTA-DASLab,llama-server–mtp/3f16medium	8192	0.976	0.945	1/164	99%
Qwen3.8-27B AD-IQ3_SAtomicChat,llama-server–mtp/3f16medium	8886	0.988	0.927	0/164	100%
Qwen3.8-27B IQ3_S-mtpISTA-DASLab,llama-server–f16low	8192	0.976	0.933	1/164	99%
Qwen3.8-27B IQ3_S-mtpISTA-DASLab,llama-server–f16xhigh	30000	0.945	0.921	5/164	97%
Qwen3.6-35B-A3B UD-Q4_K_XLunsloth,llama-server–mtp/3q8_0on	26624	0.939	0.921	5/164	97%
Qwen3.6-35B-A3B UD-Q4_K_XLunsloth,llama-server–mtp/3q8_0off	8192	0.951	0.915	0/164	100%
Ternary-Bonsai-27B Q2_g64prism-ml,prism-llama–q4_0+biason	10240	0.927	0.890	4/164	98%
Ternary-Bonsai-27B 2-bitprism-ml,mlx_lm.server–f16on	10240	0.915	0.884	5/164	97%
Gemma-4-12B Q4_K_XLunsloth,llama-server–f16off	8192	0.976	0.939	0/164	100%
Gemma-4-12B 4-bitlmstudio-community,lms–f16off	30000	0.909	0.872	0/164	100%
Gemma-4-26B-A4B UD-Q4_K_XLunsloth,llama-server–mtp/2f16off	8192	0.976	0.945	0/164	100%
Gemma-4-26B-A4B UD-Q4_K_XLunsloth,llama-server–mtp/2f16on	30000	0.884	0.860	18/164	89%
Gemma-4-26B-A4B 4-bitmlx-community,mlx_lm.server–f16on	30000	0.713	0.701	46/164	72%
Qwen3.8-27B Q4_K_Mbartowski,llama-server–mtp/3f16xhigh	30000	0.957	0.939	6/164	96%

An empty completion is reasoning that exhausted the output budget; a high empty rate is a real model limit, not a harness bug. Completion is the share of the 164 problems that got an answer at all. Read the two columns together: a low score with a low completion rate measures delivery, not code quality — the withdrawn Gemma-4-12B thinking-on row passed 99% of the answers it gave and never answered 37% of the problems (historical). Raw runs and per-problem results live in the repo's benchmarks/ run kits; each model's data page (first column) carries its scoring notes.