18 October 2026 · Agentic task loop · video edit
Local Model Benchmarks Promo
The orchestrator reached COMPLETE after 6 h 45 min with no operator action and no failed session, and the final reviewer accepted all 36 requirements on its first cycle. The render follows the storyboard and the site's design system and passes all 23 hidden acceptance cases. Artistically it is modest: it reads like an animated slide deck. Its opening photograph, a generated image from the supplied pack, is visibly AI-made.
What was checked
- Bootstrap was accepted on its first session: 36 requirements and 19 task cards. T008 was split in two; no task blocked.
- Each task was accepted only after verify --all and verify --task <ID> passed and a fresh reviewer returned ACCEPT. One review returned CHANGES (T005); two T011 reviews left no verdict and the review was retried. The final reviewer accepted on the first cycle.
- Independent check after the run: verify --all passed 58/58 in 93 s (limit 240 s). The committed render is 1080×1920, 30 fps, 30.000 s with the narration, and was made after the last composition change.
- Hidden acceptance cases: 23 of 23. Every caption is in sync and stays until the next sentence, each scene has the storyboard's label and headline, the numbers and the 15 / 16 score appear, the background is dark, paper and accent blue in scenes 2–4 as specified, the address and handle close the video, and the last 1.5 s are still (SSIM 1.0000). The first scoring gave 20; all three failures were the cases' own (a headline that reuses a sentence's words counted as a caption, and OCR running two words together and misreading the @ of the handle), and they were corrected before this score.
- Visual judgement (operator, not a score): artistically modest, close to an animated PowerPoint: each scene is a label, a headline, a framed asset and a caption, with simple slides and wipes. In scene 1 the word 'laptop' sits on the lamp's glow and is hard to read, and in scene 3 the build cards are small in a lot of empty space.
- Asset quality (operator): the opening photograph of the open laptop (laptop-night.png, made with Qwen-Image 2.1 for the asset pack, not by the model under test) is AI slop: the screen is not as wide as the keyboard and what it shows makes no sense. The pack's image check missed it.
Limits of this evidence
The asset pack was made for this workload outside the run: site captures with headless Chrome, the 'Local benchmark rig' figure with the anatomy skill (MIT), four photographs with Qwen-Image 2.1 (8-bit, MLX-Serve), and the narration with VoxCPM2 (Apache-2.0) from a written voice description. The engine was shared with another session developing a new runner during the run, which the log shows and the timing notes account for. Every asset was made on this Mac with local models and tools. The generated project, transcripts and engine logs stay on the workstation.
Configuration
The model, serving engine and agent used for this execution.
| Field | Value | Detail |
|---|---|---|
| Model · primary | Qwen 3.8 Flash Next · primary · reasoning medium | EXL3 2.625 bpw (K=2.625; MCG codebook, window 15); n-gram embedding table bf16 (95.4 GB, read from the SSD), as the Sushi startup log confirms |
| Checkpoint · primary | Qwen3.8-Flash-Next-Sushi-2.6bpw, with the bf16 n-gram table of the Sushi-4bpw pack | primary model artifact |
| Runtime | Sushi 1.2.4 · OpenAI-compatible server, greedy sampling (--temp 0), MTP drafts, 8-bit KV cache, SSD prefix cache (emptied before the run), vision on, /metrics enabled; the server also answered another local session during the run mode | inference engine and exact build |
| Harness | Pi coding agent, driven by the task-loop orchestrator, with six HyperFrames skills 1.1.0 | the agent or client that drove the run |
| Hardware | Apple MacBook Pro · 64 GB unified memory | hardware recorded for this execution |
| Workload | Local Model Benchmarks Promo · revision 1 | First run of the Local Model Benchmarks Promo brief: a 30-second vertical promo for this site, edited with HyperFrames from a supplied asset pack (site captures, an anatomy figure, four generated photographs, a VoxCPM2 narration), with an approved storyboard, the site's design system, and six HyperFrames skills including hyperframes-creative and hyperframes-keyframes. It follows Custom Characters Promo to test whether a fuller brief closes the visual gap to the Opus 5.5 render. No operator action during the run. |
| Context window | 131,072 tokens | maximum context configured for the serving runtime |
| Repetitions | 1 | executions recorded for this workload; one means an indicative result, not an estimate |
Inference measurements
Engine-side rates for this run. Each figure states its own statistic; figures from different runs are not interchangeable.
| Metric | Value | Definition |
|---|---|---|
| TTFT · median | 1.79 s | Median engine-measured TTFT from Sushi /metrics deltas around each response with exactly one finished request (p90 6.12 s). · n=744 · 164 responses had no unambiguous /metrics delta, most likely because the other client's requests finished in the same window. Speeds may be lower than on an idle server. |
| Decode · output-weighted | 42.38 tok/s | Output tokens / engine decode seconds over responses with engine timings (output-weighted). · n=744 |
| Draft acceptance | 65.67 % | Accepted MTP draft tokens / drafted tokens from Sushi spec-stats over matched requests. · n=903 · 241,571 accepted of 367,839 drafted. |
| Swapped in during the run | 0.91 GiB | Pages swapped in during the run, summed over sessions (vm_stat Swapins deltas), over 845 samples. · n=845 · Memory pressure reached warning at least once and free memory fell to 0.02 GiB; no request failed. |
Agent session counters
Agent-wide usage totals. These include reasoning, tool calls and waiting, and are not model decode speed.
| Metric | Value | Definition |
|---|---|---|
| Run elapsed | 405.2 min | From the orchestrator's preflight (2026-10-17 20:41:21 UTC) to COMPLETE (2026-10-18 03:26:34 UTC) in one segment, with no block or pause. · n=1 |
| Agent sessions | 57 sessions | Fresh Pi sessions started by the orchestrator: 1 bootstrap, 28 builder, 22 reviewer, 1 split, 4 replan, 1 final reviewer. None stopped or failed. · n=1 |
| Model responses | 908 responses | Assistant responses with usage records across all sessions: 851 tool calls and 57 stops; no engine errors. · n=1 |
| Matched model requests | 903 requests | Pi responses matched in order to the Sushi 1.2.4 process log (PID 78275) by prompt total, output tokens and SSD-restored tokens. · n=1 · Five responses did not match. The log holds 1,441 request summaries: 5 are the orchestrator's preflight and 533 between the run's own requests came from another session on this Mac that was developing a new runner against the same server during the run. Two matched keys also occur elsewhere in the log. |
| Tool calls | 1,049 calls | Pi tool calls across all sessions. · n=1 |
| Context compactions | 0 compactions | Pi context compactions across all sessions. · n=1 |
Context and token counters
Token totals as reported by the harness or engine. Cached and cumulative figures are not additive with fresh input.
| Metric | Value | Definition |
|---|---|---|
| Maximum prompt context | 69,162 tokens | Largest engine-reported prompt among matched requests. · n=903 |
| Fresh input | 1,451,387 tokens | Sum of Pi uncached input counters across all sessions. · n=1 |
| Cached input | 24,580,167 tokens | Sushi SSD-restored tokens over matched requests. · n=903 |
| Prompt-cache token share | 94.44 % | Cached prompt tokens / prompt tokens over matched requests (24,580,167 / 26,025,820). · n=903 |
| Output tokens | 534,902 tokens | Pi output usage across all sessions; includes reasoning tokens. · n=1 |
| of which reasoning output | 261,862 tokens | Pi-reported reasoning tokens included in output tokens (49.0%). · n=1 |
Recorded check counters
Counts from the checks recorded with this run.
| Metric | Value | Definition |
|---|---|---|
| Tasks completed | 20 tasks | Task cards with status done at the end; one more was split. None blocked or dropped. · n=1 |
| Reviews returning changes | 1 reviews | Task reviewer sessions that returned CHANGES (T005). Two T011 reviews left no verdict and were retried. · n=1 |
| Project checks passing | 58 checks | Checks passed by the project's own ./verify --all at the end (93 s in the independent run). · n=1 |
| Hidden acceptance cases passed | 23 checks | Hidden acceptance cases passed, of 23: written from the brief before the run, never shown to the model, and read from the rendered video. · n=1 · Three case defects found on this render were corrected before this score; the first scoring gave 20. No reference render exists yet. |
Request records & provenance
Per-request numbers are not published. 903 of 908 Pi responses were matched in order to the Sushi 1.2.4 process log; TTFT and decode come from Sushi /metrics read before and after each response (744 responses with an unambiguous delta).
Run ID: 2026-10-17-qwen38-flash-next-sushi-v124-pi-task-loop-benchmarks-promo
One execution, with checks transcribed from the retained record. A completed session is not a quality score. Measurement method · All runs