SQE Leaderboard/Laguna S 2.1

Laguna S 2.1

poolside/Laguna-S-2.1-INT4

fourlineVendor INT4 (RC2) SWE-bench Pro 59.4% official2026-08-03
Methodology

SQE score

93

blended mean, 5 reps

Gate

5/5

all reps counted

Wilson 95% low

≥56.6%

lower confidence bound

Shots to green

1

median of 5 reps

Every published metric

SQE score
Headline blended mean percent across reps (gate 45 / pass 25 / repair 15 / style 10 / thrift 5).
93n=5
Gate
Reps that cleared known and held-out honesty checks, shown as k/n after tainted exclusions.
5/5n=5
Pass
Reps that cleared public verification (compile, known conformance, scenario threshold), as k/n. Distinct from gate, which also requires held-out honesty.
5/5n=5
Wilson 95%
Lower bound of the gate pass rate at 95% confidence.
≥56.6%n=5
Harness ±
Blend of all score components, mean ± spread.
92.9 ±2.3n=5
Shots to green
Feedback rounds needed before tests pass. Lower is better.
1n=5
Scenario pass
Share of graded scenarios passing in the green build.
100%n=5
Trap/spec rate
Share of trap parameter values correctly synthesized from the spec.
0%n=5
1-shot gap
How much the first shot trailed the final result.
-n=5
Effective n
Reps counted after tainted-run exclusion.
5/5n=5
Median tokens / rep
Prompt + completion tokens per rep, median over untainted reps. Lower is better.
4.04M tokn=5 untainted
Min VRAM
Catalog minimum cluster VRAM (GB) from syndicAI referenceGpuRequirements. Closed APIs and packs not in the catalog show as -.
84 GB
SWE-bench Pro
Catalog SWE-bench Pro score for the resolved syndicAI model, or the vendor-published score for closed-reference models. Inferred scores are labelled. Packs without a score show as -.
59.4%

Run configuration

Endpoint model
poolside/Laguna-S-2.1-INT4
SWE-bench Pro
59.4% · SWE-bench Pro · Poolside (Laguna S 2.1)
Reps
5
Max shots
3
Feedback tier
1
Context cap
200K
Token budget / run
4M
Turn cap / shot
120
Thinking budget
4.096K
Sandbox digest
quality-eval-sandbox@sha256:242ebd2f73c1c1ffa90c3ccb7913574f91e74a89dd1e4cf874f907a3129149c5

Review

Generated by claude-haiku-4-5

Performance Review: fourline / poolside/Laguna-S-2.1-INT4

Overall Verdict
This INT4-quantized model successfully completed the Fourline task across all five repetitions with perfect gate compliance (5/5 pass) and no regressions. Typecheck, lint, and all 14 Playwright scenarios (including held-out tests for diagonal-win and draw detection) passed consistently. The model correctly implemented a pure-immutable Connect-Four game in TypeScript with proper domain separation (src/game/), full keyboard/UI controls, and conditional harness-API exposure (?test=1 only). Quality baseline is solid: 92–94% harness scores, single-shot repair (no flaky loops), zero trap-spec failures.

Context efficiency shows expected variance for this task. All five runs consumed their full token budgets (1.9–4.0M prompt tokens per rep, 67% average thrift), suggesting the task requires substantial iterative refinement; wall-clock times ranged 5–20 minutes, proportional to token burn. The model proactively ran typecheck and lint without prompting, indicating sound engineering discipline. Two minor tool-call errors (r1: 2 errors; r4: 5 errors; r2, r3, r5: 0–1 each) were transient and did not block progress or correctness.

Strengths

  • Consistent correctness: 5/5 pass on gate; 100% scenario pass ratio (spread 0.083, meaning no flaky runs).
  • Pure functional design: Domain logic properly separated under src/game/; immutable GameSnapshot, readonly interface properties, recursive helpers instead of let-based loops—fully compliant with functional/immutable-data and no-let ESLint rules.
  • Complete SPEC implementation: Board gravity (row 0 = bottom), 8×7 dimensions, 4-in-a-row win detection (all directions: horizontal, vertical, diagonal), rejection of out-of-range/full-column/post-game drops, draw detection (full board), playable interactive UI (visible board, cells with occupancy/player attributes, current-player and status indicators, column cursor, keyboard ArrowLeft/Right/Enter/Space/digits 1–8 + wrapping, optional click-to-drop).
  • Harness API isolation: window.__fourline correctly exposed only inside ?test=1 URL branch; verified not leaking in normal mode.
  • Self-initiated verification: Model ran pnpm typecheck and pnpm lint unprompted (observed in each run), catching and fixing errors autonomously.

Failure Modes

  • Minor tool-call noise: r1 and r4 logged transient tool errors (r4: 5 errors over 96 turns = 5% rate). Logs show mostly "Unknown tool: read" (likely malformed params), not semantic failures. Did not corrupt state or require retries; final code was correct.
  • Token thrift variance: Runs ranged from 0.327 (r2, r4, r5) to 0.676 (r1), suggesting inconsistent prompting or repeated file re-reads. r1 completed in 65 turns with high thrift; later runs needed 85–98 turns, each re-reading the full SPEC and earlier conversation. Typical for single-shot completion of moderate complexity; not a regression.
  • No stalls or reasoning-only hangs: Conversion log from r1 shows one isolated "reasoning-only" nudge at turn 49, but the model recovered and finished at shot 1 with stoppedAtGreen=true.

Serving & Harness Notes

  • Single-shot reliability: All five reps achieved shotsToGreen=1.0 with repairCurve="single-shot". Model generated working code on first attempt; no looping or retry spirals.
  • No token exhaustion: Maximum token consumption was 4.0M (r5), leaving 0.06M budgetLeft at the edge; well within typical task margins. Harmless; budget was sufficient for the task complexity.
  • No context-burn or stalls observed: Conversation logs show clean forward progress. The one "finish rejected" event in r5 at turn 91/120 was a harness-level validation (not a model stall); model immediately resumed and completed successfully at turn 92 when prompted to continue.
  • INT4 quantization transparent: Laguna-S-2.1-INT4 at ~4-bit weight precision executed without detectable quality loss. No instruction-following breakdowns, incorrect tool invocations, or logic errors attributable to low-bit weight degradation. Treat as full-precision for practical purposes.

Recommendation: Suitable for production inference on Connect-Four / board-game and general interactive TypeScript UI tasks. The consistent single-shot completion and proper SPEC adherence (including defensive rejection rules and hidden harness API) demonstrate solid reasoning and code-generation capability at this quantization level.

Per-rep results