SQE Leaderboard/MiniMax M2.5

MiniMax M2.5

MiniMaxAI/MiniMax-M2.5

fourlineOfficial FP8 SWE-bench Pro 55.4% official2026-08-04
Methodology

SQE score

94

blended mean, 5 reps

Gate

5/5

all reps counted

Wilson 95% low

≥56.6%

lower confidence bound

Shots to green

1

median of 5 reps

Every published metric

SQE score
Headline blended mean percent across reps (gate 45 / pass 25 / repair 15 / style 10 / thrift 5).
94n=5
Gate
Reps that cleared known and held-out honesty checks, shown as k/n after tainted exclusions.
5/5n=5
Pass
Reps that cleared public verification (compile, known conformance, scenario threshold), as k/n. Distinct from gate, which also requires held-out honesty.
5/5n=5
Wilson 95%
Lower bound of the gate pass rate at 95% confidence.
≥56.6%n=5
Harness ±
Blend of all score components, mean ± spread.
94.5 ±1.2n=5
Shots to green
Feedback rounds needed before tests pass. Lower is better.
1n=5
Scenario pass
Share of graded scenarios passing in the green build.
91.7%n=5
Trap/spec rate
Share of trap parameter values correctly synthesized from the spec.
0%n=5
1-shot gap
How much the first shot trailed the final result.
-n=5
Effective n
Reps counted after tainted-run exclusion.
5/5n=5
Median tokens / rep
Prompt + completion tokens per rep, median over untainted reps. Lower is better.
1.87M tokn=5 untainted
Min VRAM
Catalog minimum cluster VRAM (GB) from syndicAI referenceGpuRequirements. Closed APIs and packs not in the catalog show as -.
313 GB
SWE-bench Pro
Catalog SWE-bench Pro score for the resolved syndicAI model, or the vendor-published score for closed-reference models. Inferred scores are labelled. Packs without a score show as -.
55.4%

Run configuration

Endpoint model
MiniMaxAI/MiniMax-M2.5
SWE-bench Pro
55.4% · SWE-bench Pro · MiniMax (M2.5)
Reps
5
Max shots
3
Feedback tier
1
Context cap
200K
Token budget / run
4M
Turn cap / shot
120
Thinking budget
4.096K
Sandbox digest
quality-eval-sandbox@sha256:242ebd2f73c1c1ffa90c3ccb7913574f91e74a89dd1e4cf874f907a3129149c5

Review

Generated by claude-haiku-4-5

Performance Review: fourline task on MiniMax-M2.5

Overall Verdict

MiniMax-M2.5 delivered a fully functional Fourline Connect-Four implementation with consistent first-shot success across all 5 repetitions. Typecheck, lint, and playable interactive UI all pass. The model correctly interpreted the full SPEC including board geometry (8×7), starting player (Yellow), win detection, keyboard controls, and the conditional test harness API. Gate: 5/5 pass; blended score: 94%. A reliably performant outcome.

Execution Characteristics

The model took a single shot per run, stopping green immediately on all attempts. Average turn count: 71 (range 45–90), with modest tool-use overhead: 2–12 tool errors per run (mostly edit_file mismatches when collision-fixing), and ~1.6–2.5M prompt tokens. Importantly, no reasoning-only stalls or repair loops: each run burned through implementation, self-checked (typecheck/lint), and exited once correctness was clear. Wall-clock time ranged 241–321s, roughly proportional to turn depth. All 5 reps achieved ≥92% scenario pass (one run hit 1.0, four hit ~91.7%). Held-out conformance passed across all reps.

Strengths

  • Correct problem decomposition: Game logic (src/game/ types + immutable logic.ts), UI wiring (src/main.ts), conditional test harness.
  • Spec compliance: 8-column, 7-row board, 4-in-a-row win, Yellow start, gravity (row 0 = bottom), full-column rejection, draw detection.
  • UI completeness: All required data-testid attributes (board, cell-rc, current-player, status, column-indicator); full keyboard binding (ArrowLeft/Right, Enter/Space, digit 1–8).
  • No quantization-related failures: FP8/FP16 quality is effective here; no stalls or tool-error cascades.
  • Constants properly exported: BOARD_COLS, BOARD_ROWS, WIN_LENGTH, START_PLAYER ready for harness use.

Failure Modes

  • Minor edit misalignment (6, 2, 12, 1, 2 tool errors per rep): Model occasionally issued edit_file calls where the old_string substring didn't match exactly (whitespace, line-ending drift, or prior conflicting edits). All were recovered within the same shot with read+retry.
  • Linting warnings not eliminated (16–20 functional-style lints: prefer-readonly-type, no-let): Model read GUIDELINES.md emphasis on immutability but did not aggressively refactor mutable let bindings into const. Minor, non-blocking.

Serving & Harness Notes

  • No stalls: Model never spun on reasoning-only turns or tool failures; it read SPEC, implemented, validated, and finished.
  • Self-initiated verification: All runs called pnpm typecheck and pnpm lint independently; model never shipped without local signal.
  • Token discipline: ~65–90 k completion tokens per run; prompt token ratio reflects incremental file reads rather than wholesale re-reads.
  • Test harness API guard: Correctly exposed window.__fourline only under ?test=1; normal interactive mode never leaked the API.

Recommendation: Fit for deployment. Reliable, correct, and efficient end-to-end. Minor style lints do not impact functionality. First-shot success on all reps is a strong signal for production readiness.

Per-rep results