Gemma-4-12B-it on M1 Max 32 GB

Backends: llama-server, LM Studio MLX engine · GGUF on Hugging Face · MLX 4-bit

245Kdeepest step, llama f16 KV, 8.86 tok/s

24.64 tok/sdecode at 4K, llama f16 KV

0.976 / 0.939EvalPlus base / plus, GGUF, thinking off100% completion

2×82Ktwo llama slots at f16 KV, 13.8 GB

Benchmarked 2026-08-25 (llama build 10621, unsloth Q4_K_XL); both depth curves re-measured 2026-09-04 at wired limit 24000, the four-slot config 2026-09-05; EvalPlus scored on the LM Studio MLX container 2026-09-03 and on the GGUF quant 2026-09-05, thinking off; the two do not share a score.

Highlights

• The usable agent configuration is llama-server, f16 KV, no MTP drafter, thinking off. It decodes 24.64 tok/s at 4K and 8.86 at 245K, so it reaches the model's own 262,144 window and stays above the 8 tok/s floor. Wired memory holds flat at 13.9 GB. On the Mendel guided run it replaced 3 of 8 libraries and scored 37.5 capped, ending on the model budget after three nudges.
• The GGUF quant scores 0.976 / 0.939 / 100% with thinking off, all 164 answers delivered. That is 0.067 above the LM Studio MLX entry's 0.909 / 0.872 / 100%, so the two quants do not share a score here. LM Studio is retired on this machine: it kept the fastest curve, 34.19 tok/s at 4K down to 23.23 at 131K, and it never finished an agent task. The full record.
• The KV type sets the depth on this model, not the weights. With q8_0 KV the same server drops under the 8 tok/s floor by 16K. With f16 KV it is 3.2x faster at 16K and stays usable eight times deeper.
• Two slots at f16 KV hold 82K each, and no -c moves that. With no drafter, both slots swept in turn, the clean per-slot depth is 81958 tokens at 15.7 tok/s in 13.8 GB wired. Every larger allocation loads, up to 770048, and stops on swap growth at the step past 82K. Four slots with the drafter hold 49K each in 25.1 GB.

All configs — this model

Model / Config	Ctx	Cap	tok/s	Memory (at max ctx)	EvalPlus	Coding
Gemma-4-12B Q4_K_XLunsloth,llama-server–f16off	245k	mem	† 24.6→ 8.9	13.9 GB	0.976/0.939100% completion	38% / 37.5mendel-guided

† from an earlier serving config or method; re-run pending.

Rows below 100 percent completeness. Completeness counts three measurements: tok/s, EvalPlus and Mendel.

Model / Config	Ctx	Cap	tok/s	Memory (at max ctx)	EvalPlus	Coding
Gemma-4-12B Q4_K_XLunsloth,llama-server–f16off	2x82k	mem	† 25.0→15.7	13.8 GB	0.976/0.939100% completion	pending
Gemma-4-12B Q4_K_XLunsloth,llama-server–mtp/4f16off	4x49k	mem	† 42.9→27.7	25.1 GB	0.976/0.939100% completion	pending
Gemma-4-12B Q4_K_XLunsloth,llama-server–mtp/4q8_0off	16k	speed	† 13.8→ 6.5	10.5 GB	0.976/0.939100% completion	pending
Gemma-4-12B 4-bitlmstudio-community,lms–f16off 💀	131k	mem	34.19 → 23.23	17.2 GB	0.909/0.872100% completion	invalid

† from an earlier serving config or method; re-run pending.

💀 LM Studio is retired here: three agent runs, zero commits, a window that cannot be pinned. Why this runtime is not a candidate.

Retired entries: Gemma-4-12B, LM Studio entry google/gemma-4-12b — thinking-on repetition loop; entry gone from the model store (details).

Configs

Each table row above is one config; start it with its block below.

Gemma-4-12B Q4_K_XLunsloth,llama-server–f16off

pi id gemma-4-12b. Measured 2026-09-04 at wired limit 24000; wired memory stays flat from load to the trained window. The trained window ends at 262,144; the deepest step measured is 245K, still above the floor.

llama-server -hf unsloth/gemma-4-12b-it-GGUF:Q4_K_XL \
  --alias gemma-4-12b --no-mmproj --parallel 1 \
  -ngl 999 -fa on -c 262144 \
  --cache-type-k f16 --cache-type-v f16 \
  --jinja --port 8081

Gemma-4-12B Q4_K_XLunsloth,llama-server–f16off

pi id gemma-4-12b-2x. Measured 2026-09-08 at wired limit 25000, both slots swept in turn. The clean per-slot depth is 81958 tokens, and it does not move with -c: at every allocation from 221184 up, the sweep stopped on swap growth at the step past 81958, and at -c 196608 the same depth ran clean to the slot's own window. A larger -c loads (770048 serves a short completion) and buys no depth. Two slots hold about 71 percent of the single slot's 114718 clean depth at -c 131072. The EvalPlus score is the single-slot config's, same weights and cache type.

llama-server -hf unsloth/gemma-4-12b-it-GGUF:Q4_K_XL \
  --alias gemma-4-12b-2x --no-mmproj --parallel 2 \
  -ngl 999 -fa on -c 196608 \
  --cache-type-k f16 --cache-type-v f16 \
  --jinja --port 8081

Gemma-4-12B Q4_K_XLunsloth,llama-server–mtp/4f16off

pi id gemma-4-12b-4x. Measured 2026-09-05 at f16 KV: 655360 is the largest -c that serves a real completion (688128 loads but fails on compute buffers at the first depth step), 163840 per slot. One slot swept with the other three loaded and idle: swap grew at 66K, so the last clean row is 49K at 27.7 tok/s. The machine ran this sweep with free memory near zero and heavy compaction on every step, with swap already in use at session start; the row is honest to that state and a re-measure after a reboot may read deeper.

llama-server -hf unsloth/gemma-4-12b-it-GGUF:Q4_K_XL \
  --alias gemma-4-12b-4x --no-mmproj \
  --spec-type draft-mtp --spec-draft-n-max 4 --parallel 4 \
  -ngl 999 -fa on -c 655360 \
  --cache-type-k f16 --cache-type-v f16 \
  --jinja --port 8081

Gemma-4-12B Q4_K_XLunsloth,llama-server–mtp/4q8_0off

The q8 KV variant with the MTP drafter. Re-measured 2026-09-03 under wired limit 24000: no OOM at load, unlike the qwen3.6 MTP dagger sweep.

llama-server -hf unsloth/gemma-4-12b-it-GGUF:Q4_K_XL \
  --alias gemma-4-12b --no-mmproj \
  --spec-type draft-mtp --spec-draft-n-max 4 --parallel 1 \
  -ngl 999 -fa on -c 262144 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --jinja --port 8081

Gemma-4-12B 4-bitlmstudio-community,lms–f16off

LM Studio entry gemma-4-12b-it-mlx (lmstudio-community/gemma-4-12B-it-MLX-4bit): thinking is off and the API cannot turn it on (probed 2026-09-04). Single-turn work only — in multi-turn tool work it loops on the thought channel.

~/.cache/lm-studio/bin/lms server start --port 8081
~/.cache/lm-studio/bin/lms load gemma-4-12b-it-mlx --parallel 4 --gpu max -y

Model details and findings

The runtime decides what this model can do. mlx-lm cannot serve it: it lacks the gemma4_unified model type. LM Studio's engine supports that type and gives the flattest decode curve on this machine. llama-server serves the GGUF quant, and it is the only path that survives multi-turn tool work. Serving from the lms CLI is an approved exception to the no-GUI rule: every step runs command-line only, and the model store is shared with the app.

The two backends fail differently, and that decides the pick. llama-server allocates its KV from -c, so wired memory stays flat at 59% of the limit from load to the trained window; it runs out of model, not of machine. The LM Studio engine grows into the cap instead: wired reaches 87% of the limit past 131K, and the sweep stops on swap growth. So LM Studio is faster at every depth it survives (1.4x at 4K, 1.8x at 131K) and llama-server is the one that finishes.

The chat path costs nothing on llama-server. 24.68 tok/s against 24.64 at 4K, and 22.59 against 22.66 at 16K. The raw-prompt curve below transfers to harness use.

The context ceiling of the GGUF is the model, not the machine. GGUF metadata gives a trained context of 262,144, and a sliding window of 1024 on 5 of every 6 layers, so KV grows only about 1 GB per 64K tokens. No Metal OOM appeared at any size tested. Context limits are mode-independent, since KV is preallocated by -c. Four slots at f16 load at 163,840 each (-c 655360); the next step up fails on compute buffers at the first real request, which a trivial warmup does not show.

A -c that loads is not a window, and two slots prove it. At -c 770048 the server loads two slots and serves a short completion, but its own creep stopped on swap growth at 16K, because the KV allocation ate the wired budget before any depth was used. Judged by the creep instead, the per-slot ceiling is 81958 tokens at every allocation from 221184 up, and -c 196608 runs that depth clean to its own window boundary. One slot at -c 131072 creeps to 114718 at 13.6 tok/s, so two slots hold about 71 percent of one slot's depth. A larger -c than the creep needs buys no depth, only KV allocation.

The context window cannot be pinned on the MLX path. The engine ignores every context-length setting for this model: CLI flags, REST body, per-model config file, app default. Auto-fit computes the window from the GPU wired limit and gives 158,464 tokens at a 24000 limit. --parallel is the one load knob that works.

The two quants carry their own scores. The GGUF quant scored 0.976 / 0.939 / 100% thinking off, 0.067 above the LM Studio MLX container's 0.909 / 0.872 / 100%, so the shared-score rule does not apply to this pair. A thinking-on score is pending: the earlier one was measured on a retired entry and moved to the historical page.

Thinking on is a pitfall of this model, on both backends. The evidence, the retired LM Studio entry, and the chat-template history are on the benchmarks page.

Agentic quality — Mendel

Blind test:

config	prompt	window	score	completed	minutes	tokens	peak ctx	compactions	tool calls	commits	loop
Gemma-4-12B 4-bitlmstudio-community,lms–f16high 💀	blind-v1.1	144k	0 (raw 30.5)	0/8/invalid	49.5	218k	28k	0	15	0
Gemma-4-12B Q4_K_XLunsloth,llama-server–f16off	blind-v1.1	256k	0	0/8/invalid	80.3	9,994k	179k	0	92	0

Guided test:

config	prompt	window	score	completed	minutes	tokens	peak ctx	compactions	tool calls	commits	loop
Gemma-4-12B Q4_K_XLunsloth,llama-server–f16off	guided-v3.0	256k	37.5 (raw 58)	3/8/partial	97.6	6,453k	125k	0	132	3	text
Gemma-4-12B 4-bitlmstudio-community,lms–f16high 💀	guided-v3.0	160k	0 (raw 30)	0/8/invalid	46.0	306k	30k	0	21	0
Gemma-4-12B 4-bitlmstudio-community,lms–f16low 💀	guided-v3.0	160k	0 (raw 29.5)	0/8/invalid	99.0	1,971k	45k	3	130	0	tool call

The window cell is the harness context window of that run. Rows before the KV pick of 2026-09-04 carry the type their runbook served, or q8_0 where no record names one.

💀 LM Studio is retired here: every agent run ended with zero commits. Why this runtime is not a candidate.

The full table and the rubric are on the Mendel page.

Decode speed vs used context

Gemma-4-12Boff

One row per configuration, the same shape as the comparison table. Measured 2026-09-04 at wired limit 24000, thinking off on every row; the two-slot row 2026-09-08 at wired limit 25000. llama-server uses the raw completion endpoint with the allocation always above the deepest step; LM Studio uses the chat endpoint with four slots. Pause 25 s per step on the 2026-09-04 rows, 60 s on the two-slot row.

config	@ 4K	@ 16K	@ 33K	@ 65K	@ 82K	@ 131K	@ 245K	capped by
llama, f16 KV, no drafter	24.6	22.7	20.6	17.4		12.7	8.86	mem — the trained window ends at 262144; 14.9 at 98K, 10.7 at 180K, 9.7 at 213K
llama, f16 KV, no drafter, 2 slots, -c 196608	25.0	22.8	20.6	16.9	15.7			mem — swap grew at the step past 82K on every larger -c; 82K per slot is the ceiling
LM Studio MLX engine, f16 KV 💀	34.2	32.1	30.6	27.1		23.2		mem — last stable 131K; the engine grows into the wired cap and swap starts. Retired here
llama, q8_0 KV + MTP	13.8	6.5						speed — under the 8 tok/s floor by 16K

Cells are blank past a config's cap, or where no step was measured at that depth. Wired memory at the deepest row: 13.9 GB on llama f16, 13.8 GB on two slots, 17.2 GB on the LM Studio engine, 10.5 GB on llama q8_0. The two-slot row is slot A; slot B read within 0.2 tok/s of it at every step.

---

Method: fresh server start per configuration; identical curl per run; temperature 0. Full raw numbers, the MTP sweeps and the thinking-on evidence in the benchmarks. Cross-model picks on the comparison page.