SQE Leaderboard/Qwen3.6 27B

Qwen3.6 27B

Qwen/Qwen3.6-27B-FP8

fourlineOfficial FP8 SWE-bench Pro 53.5% official2026-08-03
Methodology

SQE score

95

blended mean, 5 reps

Gate

5/5

all reps counted

Wilson 95% low

≥56.6%

lower confidence bound

Shots to green

1

median of 5 reps

Every published metric

SQE score
Headline blended mean percent across reps (gate 45 / pass 25 / repair 15 / style 10 / thrift 5).
95n=5
Gate
Reps that cleared known and held-out honesty checks, shown as k/n after tainted exclusions.
5/5n=5
Pass
Reps that cleared public verification (compile, known conformance, scenario threshold), as k/n. Distinct from gate, which also requires held-out honesty.
5/5n=5
Wilson 95%
Lower bound of the gate pass rate at 95% confidence.
≥56.6%n=5
Harness ±
Blend of all score components, mean ± spread.
94.6 ±1.2n=5
Shots to green
Feedback rounds needed before tests pass. Lower is better.
1n=5
Scenario pass
Share of graded scenarios passing in the green build.
100%n=5
Trap/spec rate
Share of trap parameter values correctly synthesized from the spec.
0%n=5
1-shot gap
How much the first shot trailed the final result.
-n=5
Effective n
Reps counted after tainted-run exclusion.
5/5n=5
Median tokens / rep
Prompt + completion tokens per rep, median over untainted reps. Lower is better.
1.63M tokn=5 untainted
Min VRAM
Catalog minimum cluster VRAM (GB) from syndicAI referenceGpuRequirements. Closed APIs and packs not in the catalog show as -.
55 GB
SWE-bench Pro
Catalog SWE-bench Pro score for the resolved syndicAI model, or the vendor-published score for closed-reference models. Inferred scores are labelled. Packs without a score show as -.
53.5%

Run configuration

Endpoint model
Qwen/Qwen3.6-27B-FP8
SWE-bench Pro
53.5% · SWE-bench Pro · aggregator (Qwen3.6-27B)
Reps
5
Max shots
3
Feedback tier
1
Context cap
200K
Token budget / run
4M
Turn cap / shot
120
Thinking budget
4.096K
Sandbox digest
quality-eval-sandbox@sha256:242ebd2f73c1c1ffa90c3ccb7913574f91e74a89dd1e4cf874f907a3129149c5

Review

Generated by claude-haiku-4-5

fourline Task Review – Qwen/Qwen3.6-27B-FP8

Overall Verdict

The model successfully implemented a complete, fully-functional Connect-Four variant (Fourline) that passes all five runs without failures. The implementation achieves 5/5 gate passes, 5/5 scenario pass rate, 1.0 single-shot correctness, with all runs reaching green on the first and only attempt. The code demonstrates proper domain/UI separation, correct game logic (gravity, win detection in 4 directions, draw handling), and full keyboard/click interactivity. No regressions, no overfitting, all held-out scenarios pass. The approach is disciplined and systematic.

The model showed strong engineering practices: it read requirements and guidelines before coding, implemented pure immutable TypeScript per spec, separated concerns cleanly, self-initiated typecheck and lint validation, and handled edge cases correctly (full columns, out-of-range drops, board fills). Scores cluster tightly in the 94–95% range, driven mainly by lint style warnings (functional plugin prefer-readonly-type and no-let in internal loop patterns, which do not fail the gate).

Strengths

  • Perfect correctness pipeline: All 5 runs completed in shot 1 with green verdicts; all 14 scenarios (including 2 held-out) pass consistently.
  • Proper architecture: Domain logic cleanly isolated in src/game/ (types + immutable state functions); UI wiring in src/main.ts with no leakage.
  • Complete interactive UI: Visible board, correct data-testid attributes, keyboard support (arrows, Enter/Space, digit keys 1–8), column indicator, click handlers. Meets SPEC playability requirement.
  • Rigorous test harness guard: window.__fourline API properly gated to ?test=1 only; exposed methods (getState, drop, reset) work reliably.
  • Proactive validation: Model self-initiated typecheck and lint on every run, read GUIDELINES upfront, addressed readonlyness patterns.

Failure Modes

  • None observed: Zero gate failures, zero scenario failures, zero repair loops. No tool errors led to rework; the model's code was correct on first attempt.
  • Lint warnings (non-blocking): 9–17 warnings from @typescript-eslint/functional rules (prefer-readonly, no-let in loops). These are style lints, not errors, and do not block build or grading.

Serving & Harness Notes

Metric Range Notes
Shots to green 1 (all) Single-shot success across all reps.
Tool errors 1–10 per run Minor edit_file old-string mismatches (r1: 1 error; r3: 10 errors; r5: 3 errors). None caused a retry; harness auto-recovered and completed without repair steps.
Turns 43–73 Reasonable distribution. r1 lean (43 turns), r3 higher (73 turns with 10 tool errors), r5 moderate (59 turns).
Context burn 1.01M–2.47M prompt tokens Efficient; token thrift 0.58–0.83. Model made small targeted edits rather than full rewrites.
Reasoning-only stall (r5) 1 nudge at turn 34 Non-critical; model resumed and completed cleanly.
Finish rejection (r5) 1 at turn 56 Early premature finish attempt; harness nudged again, model made final corrections, then successfully finished.
Wall-clock time 1.4–2.9 sec per run Typical for browser-based Playwright verification.

Summary

This is a high-quality, production-ready implementation with a clean architecture, perfect correctness record, and robust compliance to spec. The minor tool errors and lint warnings are cosmetic and do not affect functionality or evaluation. The single-shot success and absence of any repair loop demonstrate strong first-order understanding of the requirements.

Per-rep results