Local Model Benchmarks

Local test machine: Apple M5 Pro · 64 GB unified memoryRun window 29 September — 18 October 202622 recorded runs

Workload · Task-loop orchestrator · video edit · Revision 1

Custom Characters Promo

A video brief for the task-loop orchestrator: a 20-second vertical promo edited with HyperFrames from supplied app recordings, a drawing animation and an English narration. The model must find the right parts of the footage itself, keep captions in sync with the voice, and leave out what the brief excludes.

Compare 2 attempts Task prompt

2 recorded attempts

The builds, side by side

Opening frame of the vertical promo: a child's drawing of a vampire with sparkles, the headline What if their drawing could talk to them? and the first caption
Custom Characters Promo · TensorFold native 1.0.12 · rendered video, hook at 1.5 s

16–17 October 2026

Qwen 3.8 27B

TensorFold native 1.0.12 · Pi 1.1.0 + task-loop orchestrator

Session partial Local run

COMPLETE_WITH_GAPS after 8 h 36 min of run time: 18 tasks, 25 project checks, and 15 of 20 hidden acceptance cases. The engine and orchestrator were fixed for refused images during the run.

Run sheet
Opening frame of the vertical promo: the headline What if their drawing could talk to them? above a child's drawing of a vampire with sparkles, and the first caption
Custom Characters Promo · Sushi 1.2.4 · rendered video, hook at 1.5 s

17 October 2026

Qwen 3.8 Flash Next

Sushi 1.2.4 (bf16 n-gram table) · Pi 1.1.0 + task-loop orchestrator

Session completed Local run

COMPLETE in 3 h 55 min with no operator action: 28 project checks and 19 of 20 hidden acceptance cases (captions hidden in two pauses). Clean but plainer than the Opus 5.5 render of the Spanish brief.

Play the build Run sheet

From the run records

Measured side by side

Wall time
Output tokens
Tool calls
Decode speed
Largest prompt

Each attempt used its own engine, harness version and settings, and some sessions include waiting or outages. Hover a value for its exact definition, and read the run sheets before drawing conclusions. Cloud wall time includes provider queueing; local decode speed is per request on this machine.

Task requirements & full brief

The footage and narration are supplied; the model cuts, places and captions them. Renders are scored with hidden acceptance cases read from the video itself (ffprobe, on-screen text by OCR, and still-frame comparison). The supplied source files are not published; the Sushi attempt's render is, as a build, with its narration made by Fish Audio Drama 3 (Built with Fish Audio).

  • Five beats in 20 s at 1080×1920 and 30 fps: hook, creation form, finished card, video call, call to action.
  • Find the usable parts of two silent screen recordings by looking at frames, and leave out a long generation wait and two dark call screens.
  • Captions that show each sentence of the supplied narration while it is spoken.
  • Narration at full level, a ducked music bed and short chime accents.
  • Offline, with the HyperFrames CLI pinned in the project and four HyperFrames skills loaded.

How to read this comparison

Compare the generated builds and the checks recorded for each attempt. There is no normalised quality score, and the cloud task’s cumulative token totals and end-to-end rate include repeated context, reasoning, tool calls and waiting — they are not comparable with local per-request decode or prefill rates. The method page sets out which figures can and cannot be set against each other.