Local Model Benchmarks

Local test machine: Apple M5 Pro · 64 GB unified memoryRun window 29 September — 18 October 202622 recorded runs

Workload · Task-loop orchestrator · command-line app · Revision 1

Pocket Ledger

A long brief for the task-loop orchestrator: a dependency-free Node.js personal finance tracker. The model plans its own tasks, and fresh Pi sessions build, review and verify them one at a time with no human input.

1 recorded attempt

Recorded attempt

12 October 2026

Qwen 3.8 27B

TensorFold native 1.0.7 · Pi 1.1.0 + task-loop orchestrator

Session completed Local run

COMPLETE in 5 h 9 min with no intervention: 16 tasks, 50 project checks, and a final review accepting all 20 requirements.

Task requirements & full brief

The orchestrator runs one fresh Pi session per unit of work (bootstrap, build, review, replan, final review), keeps all state in the repository, and accepts a task only when the project's verify passes and a fresh reviewer agrees. No transcripts are published.

  • Add, list, edit and remove transactions stored as integer cents in one JSON file.
  • Import bank CSV files with quoted fields, two date formats and deduplication.
  • Categorisation rules, monthly reports, budgets and CSV export that round-trips.
  • A 20-step undo, help text, fixed exit codes and atomic writes.
  • A project verify script whose checks run the program.

How to read this comparison

Compare the generated builds and the checks recorded for each attempt. There is no normalised quality score, and the cloud task’s cumulative token totals and end-to-end rate include repeated context, reasoning, tool calls and waiting — they are not comparable with local per-request decode or prefill rates. The method page sets out which figures can and cannot be set against each other.