SQE Leaderboard/Laguna S 2.1

Laguna S 2.1

kkuspa/Laguna-S-2.1-NVFP4-0804

fourlineCommunity NVFP4 SWE-bench Pro 59.4% community2026-08-08
Methodology

SQE score

96

blended mean, 5 reps

Gate

5/5

all reps counted

Wilson 95% low

≥56.6%

lower confidence bound

Shots to green

1

median of 5 reps

Every published metric

SQE score
Headline blended mean percent across reps (gate 45 / pass 25 / repair 15 / style 10 / thrift 5).
96n=5
Gate
Reps that cleared known and held-out honesty checks, shown as k/n after tainted exclusions.
5/5n=5
Pass
Reps that cleared public verification (compile, known conformance, scenario threshold), as k/n. Distinct from gate, which also requires held-out honesty.
5/5n=5
Wilson 95%
Lower bound of the gate pass rate at 95% confidence.
≥56.6%n=5
Harness ±
Blend of all score components, mean ± spread.
96.4 ±5.0n=5
Shots to green
Feedback rounds needed before tests pass. Lower is better.
1n=5
Scenario pass
Share of graded scenarios passing in the green build.
100%n=5
Trap/spec rate
Share of trap parameter values correctly synthesized from the spec.
100%n=5
1-shot gap
How much the first shot trailed the final result.
-n=5
Effective n
Reps counted after tainted-run exclusion.
5/5n=5
Median tokens / rep
Prompt + completion tokens per rep, median over untainted reps. Lower is better.
2.86M tokn=5 untainted
Min VRAM
Catalog minimum cluster VRAM (GB) from syndicAI referenceGpuRequirements. Closed APIs and packs not in the catalog show as -.
-
SWE-bench Pro
Catalog SWE-bench Pro score for the resolved syndicAI model, or the vendor-published score for closed-reference models. Inferred scores are labelled. Packs without a score show as -.
59.4%

Run configuration

Endpoint model
kkuspa/Laguna-S-2.1-NVFP4-0804
SWE-bench Pro
59.4% · SWE-bench Pro · Poolside (Laguna S 2.1)
Reps
5
Max shots
3
Feedback tier
1
Context cap
200K
Token budget / run
6M
Turn cap / shot
120
Thinking budget
4.096K
Sandbox digest
quality-eval-sandbox@sha256:242ebd2f73c1c1ffa90c3ccb7913574f91e74a89dd1e4cf874f907a3129149c5

Review

Generated by claude-haiku-4-5

Fourline / NVFP4 Quantized Codec — Operator Review

Outcome: FULL PASS (5/5 shots, 100% scenario success, 96% blended quality)

The candidate model (Laguna-S-2.1 NVFP4, a 4-bit weight quantization) delivered a complete, correct Fourline Connect-Four game implementation across five independent runs. All success criteria were met on first shot in every rep: typecheck zero, lint zero, all 15 Playwright scenarios passed (including 2 held-out), gate threshold exceeded, and full spec conformance. Code structure adhered strictly to guidelines (pure immutable TypeScript, domain logic under src/game/, UI in src/main.ts). No overfitting, no regressions, no flakiness.

Quality scores ranged 93.3–98.3% (avg 96.4%). Token efficiency varied (8.6% – 65.4%, avg 49.0%), explained by rep-to-rep context depth rather than model inconsistency. Wall-clock time was reasonable (10–18 min per shot). The model exhibited mature engineering discipline: self-initiated typecheck/lint runs (100% of reps), proactive test authoring (verified game logic end-to-end via Playwright), and clean shutdown (killed dev servers, cleaned temp files).

Strengths

  • Zero-shot correctness: 5/5 reps hit green=true first shot; no repair loops needed.
  • Spec fidelity: All 15 scenarios passed, including held-out traces; trap observations (8 cols, 7 rows, Y start) exact.
  • Robust domain logic: Pure, immutable implementation with correct gravity physics, win detection (horizontal/vertical/diagonal), state rejection for illegal moves, post-game locking.
  • UI completeness: All required data-testid attributes present and visible; keyboard bindings (arrows, enter/space, digits 1–8) functional; harness API (__fourline) correctly gated to ?test=1 only.
  • Code discipline: Followed GUIDELINES.md (no classes, no default exports, frozen/deep-readonly types, strict eslint pass). Proactive testing (7 Playwright scenarios written and validated before finish).
  • Operational maturity: Self-initiated typecheck/lint (6–10 per rep), deliberate server lifecycle management, temp file cleanup.

Failure Modes

  • Minor tool errors: 1–6 per rep (mostly edit_file mismatches on identical new_string; one per rep on average). These did not stall work; model recovered within same turn.
  • Reasoning-only stalls: One rep (r5) had a single reasoning-only turn (turn 63) without tool calls, requiring nudge. Model resumed immediately.
  • Finish rejections: Two reps (r1, r2) had finish calls rejected, likely premature claims before verification completed. Model resubmitted correctly after harness confirmed green.

Serving & Harness Notes

  • Turn budget: 71–108 turns (avg 86.8), well within safety limit. No runaway loops.
  • Context burn: 2.0–5.9M prompt tokens per rep (avg 4.1M); efficient use of file reads and targeted edits.
  • Tool errors: Concentrated in edit_file (failed string matches). No run_command or harness-API errors.
  • Reasoning-only gaps: One instance (r5, turn 63). Nudge was sufficient; no impact on outcome.

Bottom line: NVFP4 quantization showed no measurable quality degradation for this coding task. The model demonstrated strong first-shot reliability, adherence to coding standards, and pragmatic engineering practices. Suitable for production use in similar structured TypeScript tasks.

Per-rep results