2 October 2026 · Inference benchmark · context rungs
LLMProbe performance suite · MLX-Serve 26.10.1
The 4K and 16K measurements completed; the 64K cold prefill was rejected by memory admission.
What was checked
- The matched run used the same Sushi EXL3 checkpoint as the standalone run.
- The 64K cold prefill did not produce a result.
- Peak RAM was not measured.
Limits of this evidence
No raw LLMProbe JSON, coverage entry labels, endpoint, prompt text, or server logs are included. The run used one sample per scenario and did not measure peak RAM.
Configuration
The model, serving engine and agent used for this execution.
| Field | Value | Detail |
|---|---|---|
| Model · primary | Qwen 3.8 Flash Next · primary | EXL3 K=2.625 · 2.625 bpw |
| Runtime | MLX-Serve 26.10.1 · MLX 0.32.3 | inference engine and exact build |
| Harness | LLMProbe 0.6.12 | the agent or client that drove the run |
| Hardware | Apple M5 Pro · 64 GB unified memory | hardware recorded for this execution |
| Workload | LLMProbe performance suite · revision 1 | Matched benchmark-only attempt with 4K, 16K, and 64K context rungs. One sample per scenario; decode samples were fixed at 192 tokens. The run ended when memory admission rejected the cold 64K prefill. |
| Context window | 131,072 tokens | maximum context configured for the serving runtime |
| Repetitions | 1 | executions recorded for this workload; one means an indicative result, not an estimate |
Benchmark suite measurements
LLMProbe scenarios. Every scenario was measured once, so these are indicative rather than stable estimates.
| Metric | Value | Definition |
|---|---|---|
| General decode | 50.6 tok/s | single 192-token decode sample · n=1 |
| Reference TTFT | 293 ms | single sample, 2040 input tokens · n=1 |
| Reference prefill | 970 tok/s | single sample, 2040 input tokens · n=1 |
| MTP · predictable text | 82.1 tok/s | single sample · n=1 |
| MTP · novel text | 42 tok/s | single sample · n=1 |
| Predictable-to-novel MTP ratio | 1.95 ratio | predictable decode / novel decode · n=1 |
| Prefix cache · cold TTFT | 1,662 ms | single sample · n=1 |
| Prefix cache · warm TTFT | 151 ms | single sample · n=1 |
| Four-stream aggregate throughput | 57.8 tok/s | four-stream aggregate · n=1 |
| Four-stream batching efficiency | 0.35 ratio | four-stream run · n=1 |
| Decode drift under measured load | -9.3 % | change from first to last decode measurement · n=1 |
| Decode · 4K rung | 45.8 tok/s | single 4248-token rung · n=1 |
| TTFT · 4K rung | 6,137 ms | single 4248-token rung · n=1 |
| Prefill · 4K rung | 692 tok/s | single 4248-token rung · n=1 |
| Decode · 16K rung | 40.6 tok/s | single 16452-token rung · n=1 |
| TTFT · 16K rung | 22,759 ms | single 16452-token rung · n=1 |
| Prefill · 16K rung | 723 tok/s | single 16452-token rung · n=1 |
Agent session counters
Agent-wide usage totals. These include reasoning, tool calls and waiting, and are not model decode speed.
| Metric | Value | Definition |
|---|---|---|
| Attempt wall time | 1.8 min | wall time; incomplete suite · n=1 |
Run ID: 2026-10-02-qwen38-flash-next-sushi-mlx-serve-llmprobe
One execution, with checks transcribed from the retained record. A completed session is not a quality score. Measurement method · All runs