Context creep — decode speed against USED context

Allocation is storage; used tokens are what decode pays for. This test finds the depth curve, the floor, and the "capped by" verdict for a config. Common rules and the run loop apply (common rules, checklist).

Requirements

• The server for ONE config, warmed up, with its log in a file.
• One command per sweep, and one output file. The runner samples memory itself and watches liveness itself, so a sweep needs no second process beside it — see The monitor below.
• Append-only prompt growth (prompt-cache rule) — the sweep tool already does this.
• The pause rule: creep slowly, ~60 s between depth steps. The pause simulates real use — an agent's model waits on the user and on tool runs between requests — and it gives macOS time to compress other memory, which raises the measured ceiling (verified on the reference setup: about 2K extra tokens on a 35B MoE MLX config). A no-pause sweep understates the ceiling a real harness reaches. The pause is not overhead to trim. It was 25 s first and gave worse results than 60 s on this hardware, so 60 s is a measured value, not a round number. It also keeps the chip off a thermal ramp that a back-to-back sweep would build and a real agent session never would. A creep that drops the pause to save wall clock is measuring a machine nobody serves from.

Install

The sweep tool is not part of this repo. It lives in slow-context-creep in the local-llm-eval-tools repository, and a runner always fetches and uses the newest copy. Before every session that runs a sweep:

git -C ~/code/local-llm-eval-tools pull --ff-only
git -C ~/code/local-llm-eval-tools rev-parse --short HEAD

Clone it first if it is not there: git clone git@github.com:irae/local-llm-eval-tools.git ~/code/local-llm-eval-tools. Record the commit hash the second command prints beside the sweep's result, the same way a model's revision is recorded. Run the tool from that path: python3 ~/code/local-llm-eval-tools/slow-context-creep/creep.py <backend>.

Speed measurement rules

These rules belong to the speed test only. The common rules still apply on top of them.

• Fixed prompts, temperature 0, fixed token count (we use 256), identical across every model and config:
  • py: Write a Python function that parses ISO dates.
  • js: Write a JavaScript function that deep clones an object. Two languages matter: speculative-decoding acceptance differs by language, so a single-prompt benchmark can mislead.
• Read the server's own timings, not wall clocks, where available. llama-server: .timings from /completion (prompt_per_second, predicted_per_second, draft_n, draft_n_accepted). mlx_lm.server has no per-request timings — measure decode by streaming and timing the token chunks. LM Studio: the server log is ground truth (server lore).
• Reuse the server's prompt cache perfectly. Grow prompts append-only: each request's prompt = the previous prompt + the server's own reply + the new text. Never insert before an existing prefix, and never use a fixed suffix that later steps insert before. llama-server reuses any longest common prefix; mlx_lm.server only reuses strict extensions. Verify reuse via llama's .timings.prompt_n (must be the delta, not the total).
• With a drafter, record draft acceptance beside every tok/s. A speculative-decoding number without its acceptance rate cannot be compared with a later run (research run 2 could not reproduce a published 45.0 py tok/s for that reason alone). Sweep the draft depth per model AND per mode: the optimum shifts with output style (thinking on/off) and with depth.
• A drafter's tok/s at depth comes from a real-text reader at the server's own sampling. The creep sends a fixed prompt and one short completion, so on a drafter row its tok/s is a ceiling test only: a memorised or unsampled completion accepts almost every draft. The number a row publishes at depth comes from a reader that sends real text and lets the server sample (research run 4: llama-benchy), with acceptance beside every cell. A temperature-0 shot is not that number; it reads high. A row whose published tok/s did not come from such a reader carries the stale marker until one reads it (owner rule, 2026-09-12).
• Sweep the draft depth with a few fast readings, not a ladder (owner rule, 2026-09-12). Read each arm at two or three well-placed depths, the shallow one and the row's deep one, and climb from no drafter through n-max 1, 2, 3. Stop the climb as soon as an arm reads slower than the one before it at every depth; when it is slower at one depth and faster at another, take one more arm and stop. The row's served n-max is always read. A full table of every step at every depth costs a machine day and decides nothing the short climb does not.

How a sweep runs

Read this once and the rest of the page is detail.

1. Prepare the machine: checklist, "Before the run".
2. Start the server for ONE config, and send its log to a file.
3. Warm it up with one small request.
4. Run the one command, and keep its whole output:

DEPTH_LIST=4096,8192,16384,24576,32768,49152,65536 \
MODEL=<the id the server answers to> \
SWEEP_BASE=http://127.0.0.1:8081 \
python3 ~/code/local-llm-eval-tools/slow-context-creep/creep.py llama \
  > ~/.local/share/slow-context-creep/<config>-creep.tsv 2>&1

On mlx_lm.server add SERVER_LOG=<the server log>, so the runner sees the death signature, and use mlx in place of llama. On LM Studio use lmstudio. A sweep run and a server log are data worth keeping across a reboot; never write them under /tmp (see the root AGENTS.md). 5. Watch the file grow. Every line is one step row or one event. 6. Read the verdict from the last line and the exit code.

The columns, in order:

Column	What it says
context	Which round-robin context: A, B, ...
depth_tokens	Used context at this step
decode_toks	Decode speed. This is the published number
wired_mb	Wired memory. The meter that cannot lie
free_mb	Free memory. Read its shape, never its level
swap_delta_mb	Swap used now, minus swap used at the start
compress_pages	Pages compressed since the step before
decompress_pages	Pages decompressed since the step before
step_seconds	Wall time of the step

The verdict, from the last line:

Last line	Exit	Verdict
STOP: below N tok/s at depth D	0	speed. The ceiling is the deepest row at or above the floor
STOP: request failed ...	42	OOM, when the server was still fast. The ceiling is the last good row
STOP: silent halt, no tokens ...	42	OOM. The server answered with nothing
STOP: swap grew N MB ...	42	mem. Every number past the last good row times the swap file
STOP: N or more pages compressed ...	42	mem. Compaction onset; the last clean row carries the tok/s
STOP: generation thread died in <log>	42	Dead server, not a ceiling. Restart and run it again
STOP: server dead. The step gave nothing ...	42	The same, found by the probes instead of the log
no ceiling found up to D	0	window, or no ceiling in the swept range

The site publishes the stable value only: the deepest depth that still served correctly, with its tok/s. The death point stays in the run log.

The monitor

A sweep needs no external memory watcher. A scoring run does.

The runner reads vm_stat and vm.swapusage after every step and writes what it read into the row. So the memory evidence sits beside the tok/s it explains, at the same instant, in one file. An external watcher samples on its own timer and writes a second file that somebody must line up by wall clock — which is the manual step this method used to ask for, and the step agents skipped.

The runner also owns the stop conditions that watcher was there to serve: swap growth, sustained material compaction, the floor, a silent halt, a failed request, and a dead server.

Scoring runs (EvalPlus, Mendel) are the other case. Those harnesses sample no memory and run for hours, so benchmarks/run-watch.sh is their one watcher: it writes the memory record and it exits 42 on a dead server. Start it as the checklist step 6 says. A sweep watches itself; a scoring run has exactly one watcher.

Liveness is one signal, not three. /health stays 200 after an mlx_lm.server generation thread dies, so no sweep tool reads it. The runner watches the server log for the backend's death signature, and it probes ONE real completion when a step goes SILENT for STALL_S (default 600 s).

Silence, not slowness, starts a probe. The streaming backends (mlx_lm.server and LM Studio) beat a heartbeat into the runner for every chunk they send, so a step that still produces tokens never stalls. Only a silent phase can — a prefill, a full recompute, or a dead generation thread. The raw llama-server completion path does not stream, so there the whole step is silent and the clock runs from the start of the request.

One failed probe is a suspicion, not a verdict. All three servers hold one slot, so a probe sent while a step is in flight queues behind it and times out exactly like a probe to a dead server. The runner therefore polls the step and the probe together, and it drops the probe the moment the step answers or sends a chunk again. It calls the server dead only after two probes fail, each after its own silent STALL_S. A silent step gets about 2 * (STALL_S + PROBE_TIMEOUT_S) to prove it lives — 30 minutes on the defaults — and then the sweep exits 42.

A probe that finds the server alive still takes a cache slot, so the runner warns that the next step can re-read its prompt and read slow.

The one case the probe cannot tell apart: a prefill that sends nothing for longer than that whole window. The runner would call a live server dead. If a config prefills that slowly, raise STALL_S for it, and record the value you used beside the sweep — a ceiling measured with a changed STALL_S must say so.

The order for a model with a drafter

One creep per configuration, with the tool, and nothing else. No bespoke fixed-depth sweep. The configurations are: no drafter, then the drafter at each n-max worth trying.

Why the creep is not the expensive option. The token cost of reaching a depth is fixed and neither strategy avoids it. Measured on one dense 27B model at 3 bits, 2026-09-09: the creep spent 1050 s of compute arriving at depth 98338 through ten steps, and a single cold prefill to that same depth took about 1080 s. The same. Append-only growth does not save prefill time, so measuring on the way up is nearly free, and a run that prefills to one depth and takes a single reading has paid a full creep's price for one point instead of ten.

Cost of comparing five configurations on that model:

strategy	wall clock	points	ceilings	seconds per point
creep per config, full list	245 min	70	5	210
creep per config, stopped at 98K	138 min	50	0	165
one fixed-depth reading per config	92 min	5	0	1107

The fixed-depth shape saves under an hour and costs the depth curve, every ceiling, and the memory column. It is dominated, and it needs per-run scripting that CONVENTIONS.md forbids in a run folder.

The shallow comparison is the creep's own first rows. Do not run a separate shallow block: rows 4K and 8K cost about 35 s each and give the same reading. To abandon a configuration that is clearly out, stop its creep after those rows; the cost already paid is the two cheap steps.

Three things a drafter creep must record, learned the hard way:

• Memory is not flat across n-max. Turning the drafter on cost about 1.9 GB on that build, and each extra draft token a further 150 to 160 MB, all of it out of the KV budget. So each n-max has its own ceiling. Record wired at load per configuration.
• Sweep down as well as up. The optimum can sit below the value a previous run chose. One earlier comparison started at n3, went up only, and never saw that every step down was faster.
• Read draft acceptance against its own sample size. That log line reports one task's accepted-over-generated. A step that generated 47 tokens and accepted all 47 prints 1.000 and means almost nothing; a task that generated 1536 and accepted 1195 prints 0.778 and means a great deal. Acceptance on that build ran 86.1% at n1 down to 61.8% at n4 and did not improve with depth, against a claim that it did which came from reading 47-token steps as a curve.

Estimating the -c for a configuration you have not served. The memory the drafter frees, divided by the model's KV bytes per token, is how many more tokens fit. Probe that value once, then bisect against the largest -c already known to serve. Two rungs is typical. Climbing in 8192 steps from a known-good value spends an hour to learn what the arithmetic already said.

Picking the config a block will serve. For a short-prompt test such as EvalPlus, take the fastest at shallow depth; the work never reaches depth, so window is irrelevant and speed is the whole cost. For an agent run, take the faster config at the depth the task actually uses, unless the slower one removes a compaction. Window only pays when it removes one: a compaction costs a summarisation turn and a re-read, minutes, while a few percent of decode costs seconds per turn.

Steps

0. The cache type is already picked (KV cache pick). The full creep runs at that type only.
1. Set -c to the model's trained context (GGUF metadata <arch>.context_length, or the vendor card). When that does not load, binary-search the largest -c that does, toward the trained value, and verify each candidate with a real request the size of the work it will serve, never a one-token probe. A server can report "loaded" and still answer every request with a 500 and "Insufficient Memory" in its log, and it can pass a small probe and then fail the first real step: one MoE 35B model at q8_0 KV passed a one-token probe at -c 49920 under wired 24000 and hit a Metal OOM on the first sweep step, while -c 40960 served the whole creep (2026-09-07). The largest -c that serves real traffic is the hardware ceiling and is recorded as one. Grow a prompt in steps: 4K, 8K, 16K, 24K, 32K, then 16K steps.
2. Measure decode tok/s at each depth (server timings, or streamed chunks where the server has none). The runner writes one row per step, with the memory counters of that same step beside the speed. With a drafter, record draft acceptance beside every tok/s.
3. Stop at the first reading under the usability floor (ours: 8 tok/s), at OOM, or at the model's trained context — never earlier. A sweep that stops at an arbitrary depth has not found a ceiling — it has just stopped. A "window" verdict at a -c below the trained context is that mistake; raise -c and continue. Record "no ceiling found up to <max>" only after the sweep actually reached that number. To continue a sweep after raising -c, do not restart from 4K: set DEPTH_LIST to the last verified depth (a control point, which must land within 5% of its earlier reading) plus the new targets. The runner grows the prompt to the control point in one jump. Measured deviation from a slow re-creep: under 3%, on the pessimistic side.
4. Read the memory columns of the same row before you trust a slow step: wired_mb first, then swap_delta_mb, compress_pages and decompress_pages. Free memory is the wrong meter (see the pitfalls below). The runner stops by itself on swap growth and on material compaction that speed does not recover from; a step that shows either and keeps going is still a step to distrust.
5. Verdict. The site publishes two values: speed (decode dropped under the floor while memory still had room) and mem (memory ended the curve). The run log keeps the finer stop reason behind a mem verdict: a larger -c did not load, the server died in flight, compaction or swap onset, or the model's own trained window arrived first. The row note names it. tools/gen-tables.mjs refuses any other value.
6. Record on every surface; the floor — not the window — is where the harness compaction threshold belongs.

LM Studio: the compression-onset criterion

Context length cannot be pinned on LM Studio for some MLX models (auto-fit always wins — server lore), so the raw window is a loader estimate, not a measurement. For LM Studio configs:

• The ceiling is the FIRST depth step whose row shows material compression or swap: compress_pages plus decompress_pages at or above COMPRESS_PAGES (default 5000 — thousands of pages, not the idle noise of about 12 per tick), or any growth in swap_delta_mb.
• The reported tok/s comes from the last clean step before onset.
• The context-window column keeps the auto-fit estimate, flagged as a loader estimate; the trained max goes in a footnote.
• The engine stays functional well past onset — that functional range is worth a note for harness use, but it is not the ceiling.

Multi-context (multi-agent) configs

Reported tok/s comes from ONE slot decoding alone, not all slots decoding at once. Slots are parallel contexts, not parallel use: a sub-agent's slot holds its place while idle, but a main agent and a sub-agent rarely generate at the same instant. Depth-sweep the single slot exactly as any other config; the other slots stay loaded but idle. For N alternating contexts use N_CONTEXTS on the backend's creep script. Round-robin works on all three backends: mlx_lm.server holds several distinct KV caches when started with --prompt-cache-size at least as large as N_CONTEXTS.

The tool

One command picks the backend: python3 creep.py <llama|mlx|lmstudio>, run from the clone (see Install). creep.py owns the method: the 60-second pause as a DEFAULT, append-only growth, round-robin contexts, memory sampling, liveness, and the stop conditions. Each backend module holds only its endpoint, its request shape, how it reads speed, and its two liveness parts. --help at either level prints the environment variables it reads.

• llama — llama-server. ENDPOINT=completion (default) is raw and comparable with every published number here; ENDPOINT=chat is the path a harness uses. llama-server defaults enable_thinking to TRUE on the chat path, so set THINKING=off when that is what you mean.
• lmstudio — LM Studio. Chat endpoint only.
• mlx — mlx_lm.server. Set SERVER_LOG; this backend's generation thread can die while /health stays green.

Stop conditions and their defaults, all overridable by environment variable: the floor (FLOOR_TOKS, 8 tok/s), swap growth above 1 MB, material memory compression (COMPRESS_PAGES, 5000 pages) on six steps in a row without speed recovering to RECOVERY_FRACTION (0.75) of the step before, a silent halt, a failed request, and a dead server (silence for STALL_S, 600 s, then a probe with PROBE_TIMEOUT_S, 300 s; two failed probes end the sweep). Compression under COMPRESS_PAGES in one step is noise on a busy machine and does not count. These are the loosened thresholds found on this machine on 2026-09-07 (hardware/m1-max-32gb/benchmarks/bench12/results.md, "Pre-block prep"); the tighter values used before that date produced a false-positive stop from ordinary speed decay at 16K-32K depth.

Do not try these — see git history

• Parallel slot sweeps. llama_sweep_slot.py used llama.cpp's id_slot to sweep one slot of a parallel server. The project measures round-robin instead: separate sessions keeping their own cache is the real shape, and parallel slots do not fit sub-agent use. Deleted 2026-09-04.
• A "slow" script beside a fast one. mlx_sweep_slow.py differed from mlx_sweep.py by three lines and defaulted its pause to ZERO, so its name asserted behaviour its code did not have. Slow creep is now the default and cannot be lost by picking the wrong file. Deleted 2026-09-04.
• sweep-one.sh and ceilings.sh. Both hardcoded a scratchpad path from a long-dead session and could not run. Deleted 2026-09-04.
• A per-process graphics-resident watcher. proc-mem-watch.sh sampled one PID's vmmap IOAccelerator line. That counter reads near zero for an MLX server holding gigabytes, so it answered one question on one day and nothing since. Wired pages are the meter. Deleted 2026-09-05.
• LM Studio's /v1/completions. Returns garbage on this build and streams the whole reply in one burst with no per-token pacing — verified 2026-08-29. Use chat completions there.

Known pitfalls

• Allocation is not depth. Serving -c 262144 proves the KV fits; it says nothing about decode speed at 262144 USED tokens. Reading one as the other produced a wrong "this row is stale" claim in run 2.
• Allocating far more context than the sweep will reach costs memory for nothing and can push the machine into compaction.
• Free memory is the wrong meter. Read wired_mb (memory ceiling says why).