Opus 4.5–4.8 Pi Computation Benchmark

A blind, isolated micro-benchmark comparing Claude Opus 4.5, 4.6, 4.7, and 4.8 on a small, verifiable Python task: computing π to 50 decimal places.

TL;DR — recommendation

Opus 4.6 is the best value pick for tasks of this shape:

• Cheapest per run at ~$0.33 avg (vs $0.37 for 4.5, $0.68 for 4.8, $0.92 for 4.7)
• Reasonable wall time (~108s avg — middle of the pack)
• 2/3 correctness on a strict external 50-digit reference check

Caveat: Opus 4.6 had one precision error in 3 runs — run02 produced …7511 as the final digit where the true value is …7510 (a single-digit rounding error in the 50th decimal place). The failure is minor — only the last of 50 digits was wrong, and only in 1 of 3 runs — but it does mean 4.6 is not suitable for tasks requiring provably-correct arbitrary-precision output without an external verifier. If every-digit-correct output is non-negotiable, Opus 4.5 (3/3 correct, only ~$0.04/run more) is the safer choice.

Opus 4.7 and 4.8 were both slower and more expensive than 4.6, with worse correctness, so they are not recommended for this task class.

Task

Each model received an identical prompt (see PROMPT.md) and was asked to build compute_pi.py that:

1. Computes π to ≥50 decimal places using a real algorithm (no hardcoded digits)
2. Supports --digits 50 (prints 3.<50 digits>) and --verify (exit 0 on success)
3. Writes RESULT.md documenting the algorithm, usage, output, line count, clarity

Models ran in 3 isolated runs each (12 total), in separate empty workspaces, with no knowledge of the comparison or each other.

Results

Correctness (external 50-digit π reference check)

Model	Pass rate	Notes
Opus 4.5	3/3 ⭐	All exact
Opus 4.6	2/3	run02 wrong final digit (…7511 vs …7510) — see caveat above
Opus 4.7	2/3	run02 printed only 49 decimals
Opus 4.8	1/3	run01 + run02 failed reference (rounding/truncation)

Wall time (avg of 3 runs)

Model	Avg time	Range
Opus 4.5	88.8s ⭐	86.7–90.1s
Opus 4.6	107.9s	95.4–117.2s
Opus 4.8	121.0s	109.9–132.1s
Opus 4.7	123.2s	110.2–137.7s

Cost & tokens (per run, avg of 3 runs)

Model	Avg cost	Total cost	Avg input	Avg output	Avg cache read	Avg cache write
Opus 4.5	$0.369	$1.11	63	2,727	226,492	29,906
Opus 4.6 ⭐	$0.328	$0.98	12	2,844	250,421	21,089
Opus 4.7	$0.916	$2.75	12	2,546	136,452	125,482
Opus 4.8	$0.675	$2.03	16	2,543	226,916	79,689

Grand total for the batch (including parent orchestration + retry attempts): ~$8.65.

Key findings

• Opus 4.6 is the best value — cheapest per run (~$0.33) with acceptable accuracy (2/3; the one failure was a single-digit error in the 50th place).
• Opus 4.5 was fastest and most accurate — best choice when every digit must be correct (~$0.04/run more than 4.6).
• Opus 4.7 was the most expensive — driven by high cache write tokens, and no correctness or speed advantage.
• All runs passed their own --verify, but 4.6 (1 run), 4.7 (1 run), and 4.8 (2 runs) failed an independent 50-digit reference check despite self-verifying as correct — a useful signal about self-evaluation reliability.
• Chudnovsky was the most common algorithm; mpmath-based approaches correlated with formatting/precision failures.

Variability across runs (n=3 per model)

Sample standard deviation (n−1), variance, min, max, and range for each metric.

Wall time (seconds)

Model	Mean	Std dev	Variance	Min	Max	Range
Opus 4.5	88.8	1.84	3.37	86.7	90.1	3.4
Opus 4.6	107.9	11.25	126.49	95.4	117.2	21.8
Opus 4.7	123.2	13.81	190.60	110.2	137.7	27.5
Opus 4.8	121.0	11.10	123.21	109.9	132.1	22.2

Opus 4.5 was not just the fastest but also the most consistent (σ ≈ 1.8s, range 3.4s). 4.6–4.8 all showed ~6–8× higher timing variance.

Cost (USD)

Model	Mean	Std dev	Variance	Min	Max	Range
Opus 4.5	0.369	0.2325	0.054057	0.157	0.617	0.461
Opus 4.6	0.328	0.0989	0.009782	0.249	0.439	0.190
Opus 4.7	0.916	0.2662	0.070854	0.646	1.178	0.532
Opus 4.8	0.675	0.1626	0.026433	0.547	0.858	0.311

Opus 4.6 had the lowest cost variance (σ ≈ $0.10, range $0.19) — its per-run spend was the most predictable. 4.5 and 4.7 had wider cost swings driven by cache-write variability.

Output tokens

Model	Mean	Std dev	Variance	Min	Max	Range
Opus 4.5	2,727	1,269.6	1,611,993	1,820	4,178	2,358
Opus 4.6	2,844	1,464.8	2,145,784	1,618	4,466	2,848
Opus 4.7	2,546	731.9	535,730	2,091	3,390	1,299
Opus 4.8	2,543	1,003.0	1,005,919	1,810	3,686	1,876

Opus 4.7 produced the most consistent output length; 4.5 and 4.6 swung

2× between min and max runs.

Cache read tokens

Model	Mean	Std dev	Min	Max	Range
Opus 4.5	226,492	60,760	190,218	296,638	106,420
Opus 4.6	250,421	83,925	161,337	327,999	166,662
Opus 4.7	136,452	44,137	91,701	179,948	88,247
Opus 4.8	226,916	116,492	139,264	359,107	219,843

Cache write tokens

Model	Mean	Std dev	Min	Max	Range
Opus 4.5	29,906	27,877	2,505	58,236	55,731
Opus 4.6	21,089	15,109	3,744	31,389	27,645
Opus 4.7	125,482	38,842	83,748	160,575	76,827
Opus 4.8	79,689	35,098	43,972	114,133	70,161

Opus 4.7's high cost is traceable to consistently high cache-write tokens (σ ≈ 39k, mean 125k) — it wrote ~4–6× more cache than 4.5/4.6 on every run.

Methodology

The prompt

The exact task prompt given to every subagent is preserved verbatim in PROMPT.md. It was deliberately framed as a neutral "coding exercise in a private workspace" — no mention of comparisons, other models, benchmarks, or that the result would be measured against other runs. Each subagent saw only its own workspace path and the task.

The prompt required the model to:

1. Build compute_pi.py computing π to ≥50 decimal places via a real algorithm
2. Support --digits 50 and --verify (exit 0 on success) CLI modes
3. Write a RESULT.md documenting algorithm, usage, output, line count, clarity
4. Reply with 10 words or fewer (to keep orchestrator context small)

Isolation was enforced by three layers: a separate empty workspace directory per run, an explicit "only this directory" boundary in the prompt, and a .cursor/rules/isolation.mdc rule (alwaysApply: true) planted in each workspace reinforcing the boundary.

Models involved

Role	Model	Notes
Subjects (benchmarked)	Claude Opus 4.5, 4.6, 4.7, 4.8	3 isolated runs each via Cursor Task subagents
Benchmark orchestrator	Composer-2.5	Designed the prompt, built the harness, launched 12 subagents in parallel, recorded wall-clock metrics, validated outputs, fetched dashboard usage, and aggregated the comparison
Privacy reviewers	6 × GLM-5.2 subagents	Independently reviewed the sanitized export for personal/identifying info before publishing (all 6 returned PASS)
GitHub publisher	GLM-5.2	Created the public repository and pushed the sanitized export via gh CLI

The orchestrators are not among the benchmarked Opus models — they only ran the benchmark, collected metrics, and published results. The benchmarked Opus variants were unaware they were being compared.

Attribution note. The benchmark itself (prompt design, harness, the 12 Opus subagent runs, metrics collection, and result aggregation) was orchestrated by Composer-2.5. The subsequent work — sanitizing the export, launching 6 privacy-review subagents, and creating + pushing this GitHub repository — was done by GLM-5.2. This split is confirmed by Cursor's usage dashboard, which attributed the benchmark-window orchestration cost (~$0.48) to the composer-2.5 billing category. The benchmarked Opus results are unaffected: they came from isolated Task subagents explicitly pinned to each Opus variant's model slug.

Run execution

The 12 subagent runs were launched in parallel via Cursor's Task tool, each pinned to one Opus variant via its model slug:

Subject model	Cursor agent slug
Opus 4.5	claude-4.5-opus-high-thinking
Opus 4.6	claude-4.6-opus-high-thinking
Opus 4.7	claude-opus-4-7-thinking-xhigh
Opus 4.8	claude-opus-4-8-thinking-high

Wall-clock time per run was measured from workspace file mtimes (first→last meaningful file write). Token and cost data was retrieved after the fact from the Cursor usage dashboard API (get-filtered-usage-events), filtered to the benchmark window, and each billing event was matched to the run whose workspace peak mtime was closest to the event timestamp.

Repository layout

PROMPT.md                          # Task prompt (sanitized)
config/opus_bench.yaml             # Model config
harness/                           # Orchestration + validation scripts
  opus_bench_common.py
  run_opus_bench.py
  validate_opus_bench.py
  aggregate_opus_bench.py
  prepare_opus_bench_tasks.py
  finalize_opus_task_run.py
  opus_bench_manifest.py
  finish_opus_bench_batch.py
runs/<model>-run01|02|03/
  compute_pi.py                    # Model's solution
  RESULT.md                        # Model's writeup
  metrics.json                     # Time, tokens, cost
  validation.json                  # Pass/fail checks
results/
  comparison.md                    # Full comparison table
  comparison.csv                   # Same, CSV form
  comparison.json                  # Same, machine-readable
  usage_costs.json                 # Per-run token/cost breakdown
  usage_window.json                # Raw dashboard events (sanitized)

Reproducing

These results were collected by launching 12 Cursor Task subagents (one per run × model) in isolated workspaces, orchestrated by Composer-2.5. The export sanitization, privacy review, and GitHub publishing were done by GLM-5.2. Token/cost data comes from the Cursor usage dashboard API filtered to the benchmark window.

To re-run via the agent CLI instead:

pip install pyyaml
python3 harness/prepare_opus_bench_tasks.py
python3 harness/run_opus_bench.py            # sequential, needs `agent` CLI auth
python3 harness/aggregate_opus_bench.py --validate

Validation requires only Python 3.10+ and pyyaml.

Caveats

• Small sample size (3 runs/model): treat as directional, not statistical.
• Single task (π computation): generalization across tasks is not implied.
• Self-verify vs. external verify: every run passed its own --verify; the failures surfaced only against an independent 50-digit reference. This is itself an interesting finding about self-evaluation reliability.
• Token/cost figures are dashboard-derived and may include small attribution noise from concurrent sessions.

License

MIT — see headers in each model's compute_pi.py for the model's own choice.