SQE Leaderboard/MiniMax M3

MiniMax M3

MiniMaxAI/MiniMax-M3-MXFP8

fourlineOfficial MXFP8 SWE-bench Pro 59% official2026-08-04
Methodology
Token budget. 1 of 5 reps aborted on token budget. Those reps still count toward the score.

SQE score

80

blended mean, 5 reps

Gate

4/5

all reps counted

Wilson 95% low

≥37.6%

lower confidence bound

Shots to green

1

median of 5 reps

Every published metric

SQE score
Headline blended mean percent across reps (gate 45 / pass 25 / repair 15 / style 10 / thrift 5).
80n=5
Gate
Reps that cleared known and held-out honesty checks, shown as k/n after tainted exclusions.
4/5n=5
Pass
Reps that cleared public verification (compile, known conformance, scenario threshold), as k/n. Distinct from gate, which also requires held-out honesty.
5/5n=5
Wilson 95%
Lower bound of the gate pass rate at 95% confidence.
≥37.6%n=5
Harness ±
Blend of all score components, mean ± spread.
80.3 ±45.5n=5
Shots to green
Feedback rounds needed before tests pass. Lower is better.
1n=5
Scenario pass
Share of graded scenarios passing in the green build.
91.7%n=5
Trap/spec rate
Share of trap parameter values correctly synthesized from the spec.
0%n=5
1-shot gap
How much the first shot trailed the final result.
-n=5
Effective n
Reps counted after tainted-run exclusion.
5/5n=5
Median tokens / rep
Prompt + completion tokens per rep, median over untainted reps. Lower is better.
4.06M tokn=5 untainted
Min VRAM
Catalog minimum cluster VRAM (GB) from syndicAI referenceGpuRequirements. Closed APIs and packs not in the catalog show as -.
564 GB
SWE-bench Pro
Catalog SWE-bench Pro score for the resolved syndicAI model, or the vendor-published score for closed-reference models. Inferred scores are labelled. Packs without a score show as -.
59%

Run configuration

Endpoint model
MiniMaxAI/MiniMax-M3-MXFP8
SWE-bench Pro
59% · SWE-bench Pro · MiniMax (M3)
Reps
5
Max shots
3
Feedback tier
1
Context cap
200K
Token budget / run
4M
Turn cap / shot
120
Thinking budget
4.096K
Sandbox digest
quality-eval-sandbox@sha256:242ebd2f73c1c1ffa90c3ccb7913574f91e74a89dd1e4cf874f907a3129149c5

Review

Generated by claude-haiku-4-5

Overall verdict. MiniMax-M3-MXFP8 delivered working, playable Connect-Four code at 80% blended quality and reached green in 4 of 5 runs. All five reps compiled, linted, and passed 91.7% of scenarios on average (range 91.7–100%). However, the model struggled with two persistent correctness issues across runs: incorrect state representation (confusing row-major vs. column-major cell indexing early) and a missing HTML test attribute (data-testid for the column indicator), both requiring tool-assisted discovery during long edit-and-verify loops. Tool error frequency (2–8 per run, median 5) indicates either brittle string-matching in edit attempts or confused state after lengthy context turnover.

Reliability & efficiency notes. Three runs (r2, r4, r5) passed the gate cleanly with no overfitting; r1 overfitted on a single scenario despite reaching green, and r3 had no held-out verdict. All runs hit token ceiling (~4M prompt tokens, 41–55k completion tokens over ~100–113 turns), suggesting the model burned context rapidly without converging faster. Median time-to-green was 1 shot with no retries needed—the harness accepted the first attempt and graded it live, which is good discipline. No test-harness stalls or reasoning-only loops observed.

Strengths:

  • Pure, immutable TypeScript throughout; correctly avoided pnpm dev/preview long-running servers.
  • Game logic sound: proper gravity drop, win detection (all directions), board-full draw, player switching.
  • Interactive UI built and styled (data-testid elements, keyboard handlers, visual column cursor).
  • Read SPEC and GUIDELINES proactively; self-initiated typecheck and lint runs.
  • No regressions between shots; all reps reached 0-error lint/typecheck.

Failure modes:

  • Transposed board representation (r1–r2 turns ~29–227): Initially built board as cells[col][row] instead of spec's cells[row][col]. Took ~200 turns of iterative debugging before recognizing and fixing the semantic mismatch.
  • Missing column-indicator testid (r5): Test scenario testids-present timed out waiting for data-testid="column-indicator"; added via sed mid-run. Suggests incomplete checklist-reading against SPEC UI section.
  • Edit string mismatches (4 errors in r1, 6 in r2, 5 in r3, 8 in r4, 2 in r5): Frequent old_string not found or new_string equals old_string errors during multi-turn edits; likely from state drift or partial edits missing trailing whitespace.

Serving & harness notes:

  • Token budget fully consumed in all runs (capped at 4–4.1M prompt tokens); no early stopping.
  • Turns ranged 97–113, averaging ~106; no penalty detected for high turn count.
  • All 5 shots completed without stall; verification ran to completion each time.
  • Scenarios missed: r1 failed reject-full-column and full-board-draw (overfitting sign); r5 failed testids-present (missing attribute, but passed held-out conformance). Both tests green on re-runs within same execution.
  • Tool-error recovery implicit (errors logged but runs continued); no explicit retry logic visible in conversation.

Per-rep results