SQE Leaderboard/Claude Opus 4.7

Claude Opus 4.7

anthropic/claude-opus-4.7

fourlineClosed API reference SWE-bench Pro 64.3% Closed reference2026-08-05
Methodology

SQE score

99

blended mean, 5 reps

Gate

5/5

all reps counted

Wilson 95% low

≥56.6%

lower confidence bound

Shots to green

1

median of 5 reps

Every published metric

SQE score
Headline blended mean percent across reps (gate 45 / pass 25 / repair 15 / style 10 / thrift 5).
99n=5
Gate
Reps that cleared known and held-out honesty checks, shown as k/n after tainted exclusions.
5/5n=5
Pass
Reps that cleared public verification (compile, known conformance, scenario threshold), as k/n. Distinct from gate, which also requires held-out honesty.
5/5n=5
Wilson 95%
Lower bound of the gate pass rate at 95% confidence.
≥56.6%n=5
Harness ±
Blend of all score components, mean ± spread.
98.6 ±1.8n=5
Shots to green
Feedback rounds needed before tests pass. Lower is better.
1n=5
Scenario pass
Share of graded scenarios passing in the green build.
100%n=5
Trap/spec rate
Share of trap parameter values correctly synthesized from the spec.
100%n=5
1-shot gap
How much the first shot trailed the final result.
-n=5
Effective n
Reps counted after tainted-run exclusion.
5/5n=5
Median tokens / rep
Prompt + completion tokens per rep, median over untainted reps. Lower is better.
0.89M tokn=5 untainted
Min VRAM
Catalog minimum cluster VRAM (GB) from syndicAI referenceGpuRequirements. Closed APIs and packs not in the catalog show as -.
-
SWE-bench Pro
Catalog SWE-bench Pro score for the resolved syndicAI model, or the vendor-published score for closed-reference models. Inferred scores are labelled. Packs without a score show as -.
64.3%

Run configuration

Endpoint model
anthropic/claude-opus-4.7
SWE-bench Pro
64.3% · SWE-bench Pro · Anthropic (Claude Opus 4.7)
Reps
5
Max shots
3
Feedback tier
1
Context cap
200K
Token budget / run
4M
Turn cap / shot
120
Thinking budget
4.096K
Sandbox digest
quality-eval-sandbox@sha256:242ebd2f73c1c1ffa90c3ccb7913574f91e74a89dd1e4cf874f907a3129149c5

Review

Generated by claude-haiku-4-5

Performance Review: Claude Opus 4.7 on Fourline

Overall verdict:
The model delivered a complete, correct implementation of the Fourline Connect-Four game across all five repetitions with 100% test pass rate and zero-failure repair curves. Correctness was bulletproof: every scenario passed (board dimensions, gravity, win detection in four directions, draw logic, UI testid conformance), all trap specifications were extracted accurately, and held-out conformance verification succeeded on both seed-42 and seed-99001 runs. The implementation was production-ready on first attempt every time.

Efficiency was uneven but acceptable: Token thrift ranged from 0.688–0.900 (r2 burned 1.85M prompt tokens over 57 turns, suggesting exploration or re-verification; r4 was lean at 589K over 23 turns). Wall clock times clustered around 200–400ms per run. One rep incurred 5 tool-call errors (r2), but others saw only 1 each, and none escalated into repair loops. Lint output flagged 4 immutable-pattern warnings across multiple runs (use const instead of let), indicating a minor style violation that did not prevent gate passage.

Strengths

  • Perfect correctness & coverage: 5/5 gate pass, 5/5 scenario pass, 100% held-out conformance. Zero overfitting. All 14 scenarios (including hidden diagonal-win and full-board-draw traps) passed on first shot.
  • Trap specification accuracy: board-cols=8, board-rows=7, start-player='Y' extracted correctly every run; no drift.
  • Reliable self-awareness: model proactively ran pnpm typecheck, pnpm lint, pnpm build, and verified all data-testids and keyboard controls before finishing. Made strategic use of single-shot completion every run.
  • Pure architecture: immutable game engine (src/game/) with drop, win-detection, draw-detection logic; clean separation of domain and UI in main.ts.
  • API encapsulation: correctly gated harness window.__fourline to ?test=1 mode only, preserving playable UI in all modes (required).

Failure modes

  • Tool-call error spike in r2: 5 out of 57 turns yielded tool errors (likely file or command glitches), yet no retry cascade occurred and task still finished cleanly. Suggests resilience, but indicates occasional brittleness.
  • Lint warnings not addressed: 4 "Unexpected let, use const" warnings persisted across runs (e.g., r1, r3, r5) in state.ts. Functional ESLint rule violations don't block gate, but a stricter linter would have required remediation.
  • Variable token burn: r2 used ~3× the prompt tokens of r4 (1.85M vs. 589K), hinting that one run may have re-read or re-explored specs unnecessarily. Median was ~870K, close to r1 and r5.

Serving & harness notes

  • No stalls or reasoning-only turns: all 32 turns (r1, r5) and 23 turns (r4) stayed productive; no tool-use loops or context depletion observed.
  • Repair curve: "single-shot" all runs (shotsToGreen=1.0, spread=0.0). No second-attempt scenarios.
  • Conformance replay: both seed-42 and seed-99001 held-out checks passed; no divergence in game state replay logic.
  • Context burn: peak was r2 at 1.85M prompt tokens (acknowledged in run.log); still well below typical 4M budget on this harness.
  • Quantization note: No precision issues observed. Model ID is anthropic/claude-opus-4.7 (native full-precision), so FP8/INT4 considerations do not apply.

Recommendation: Deploy with confidence. The model solves this task reliably, thoroughly, and correctly. Minor lint style issues and occasional tool friction are non-critical given 100% functional correctness. Suitable for production use.

Per-rep results