Method & measurements · Local Model Benchmarks

Run protocols

Most runs here are one-shot. A single task prompt is submitted to the model through a coding agent, and I do not intervene, steer the session, or hand-edit the result. The published build is whatever the run produced.

The archive also includes staged, long-horizon challenges that ask the model to preserve plans and project state across fresh sessions. A deterministic runner can sequence roles without a permanent orchestrator model. Manual continuation, setup drift and incomplete review gates are recorded on the run sheet. Wild West Showdown 3D is the first such attempt here; its game was completed, but its runner stopped before the planned review phase.

That protocol is what makes the builds comparable as artifacts: each one is an honest answer to the same prompt. It is not what makes their measurements comparable. Every workload in this archive records one repetition, so a figure such as a decode rate describes one session rather than a stable property of the model or engine.

Metric definitions

These are the measurement names used across the run sheets, with the statistic each one reports. Where a run recorded a different statistic for the same quantity, that run sheet says so in its own row.

Metric · What it reports
Metric	What it reports
Decode · per-request median	The middle per-request generation rate across the session’s requests. Sensitive to short requests and to cache state.
Decode · output-weighted	Total output tokens divided by the summed per-request decode time. Weights long generations more heavily than the median does.
Prefill · median	The middle prompt-processing rate across requests. Includes cache reuse, so a cached prompt inflates it.
MTP acceptance	The share of drafted speculative tokens the model accepted. A property of the draft configuration and the text, not of raw speed.
TTFT · cached	Request-start to first token when the prompt prefix was already cached. Recorded only where the engine logged it.
Cache hit ratio	Cached input tokens as a share of presented input tokens.
Session elapsed	Wall time from the first request to the final response. Includes coding, tool calls and recorded checks — it is not generation time.
End-to-end output rate	Output tokens divided by task wall time, including reasoning, tool calls and waiting. Deliberately not model decode speed.
Peak KV checkpoint	The largest key/value checkpoint observed in unified memory during the run.
Cumulative input	Input tokens summed across model calls. Repeated conversation context can be counted more than once.

Which figures cannot be compared

Four distinctions account for most of the apparent contradictions between run sheets. Each one is a different question, not a different answer.

Apparent comparison · Why it does not hold
Apparent comparison	Why it does not hold
Output-weighted decode vs. median decode	Different statistics of the same quantity. The TensorFold run reports a median of 22.9 tok/s; the Sushi runs report output-weighted means in the mid-30s. Neither is wrong, and neither converts to the other without the underlying per-request rows, which are not published.
Local decode vs. Codex end-to-end rate	The Codex figures divide output tokens by total task wall time, so reasoning, tool calls, generated tool output and waiting are all inside the denominator. A local engine decode rate has none of that. The cloud run records state this directly.
Prefill across runs	Prefill figures include cache reuse, and recorded cache hit ratios range from 90.23% to 95.17% across these runs. A run with more reuse reports a faster prefill for reasons that have nothing to do with the engine.
Session elapsed across runs	Session time includes coding, tool calls, recorded checks and any backend failure. The TensorFold session contains a recorded 25,718-second outage, so its raw wall time is not an effective elapsed time; that run sheet publishes both figures.

What this archive does not establish

No model ranking. One run per workload cannot rank anything. A completed run is not a better run; it means the build finished and its recorded checks passed.
No repeatability. Every workload here records a single repetition. The LLMProbe suite measured each scenario once and is explicitly indicative.
No peak-memory comparison for the inference suite. Peak RAM was not sampled during the LLMProbe A/B, so the two engines cannot be compared on memory.
No general context ceiling. The MLX-Serve 64K prefill was rejected by its memory-admission policy. That describes the tested cold-prefill scenario, not a maximum context for either engine.
No image or video workloads. These runs cover text-to-code agentic tasks and a text inference suite only.
No thermal, power or sustained-load data. Long-duration memory pressure and battery behaviour are not covered.
Checks were transcribed, not rerun. The verification bullets on each run sheet are transcribed from the session log during archiving. They were not re-executed against the published build.

Reading a run sheet

Each run sheet opens with the generated result, followed by recorded checks and limits, configuration, and measurements. Every metric retains its statistic, unit and sample count. Request provenance is available in a disclosure at the end. Browse the runs, or read a worked example with a recorded backend outage.