SQE Leaderboard/Qwen3.8 27B

Qwen3.8 27B

Qwen/Qwen3.8-27B-FP8

fourlineOfficial FP8 SWE-bench Pro 61.7% official2026-08-16
Methodology

SQE score

94

blended mean, 5 reps

Gate

5/5

all reps counted

Wilson 95% low

≥56.6%

lower confidence bound

Shots to green

1

median of 5 reps

Every published metric

SQE score
Headline blended mean percent across reps (gate 45 / pass 25 / repair 15 / style 10 / thrift 5).
94n=5
Gate
Reps that cleared known and held-out honesty checks, shown as k/n after tainted exclusions.
5/5n=5
Pass
Reps that cleared public verification (compile, known conformance, scenario threshold), as k/n. Distinct from gate, which also requires held-out honesty.
5/5n=5
Wilson 95%
Lower bound of the gate pass rate at 95% confidence.
≥56.6%n=5
Harness ±
Blend of all score components, mean ± spread.
94.0 ±9.3n=5
Shots to green
Feedback rounds needed before tests pass. Lower is better.
1 ±1n=5
Scenario pass
Share of graded scenarios passing in the green build.
100%n=5
Trap/spec rate
Share of trap parameter values correctly synthesized from the spec.
100%n=5
1-shot gap
How much the first shot trailed the final result.
0.3 ptsn=5
Effective n
Reps counted after tainted-run exclusion.
5/5n=5
Median tokens / rep
Prompt + completion tokens per rep, median over untainted reps. Lower is better.
3.81M tokn=5 untainted
Min VRAM
Catalog minimum cluster VRAM (GB) from syndicAI referenceGpuRequirements. Closed APIs and packs not in the catalog show as -.
57 GB
SWE-bench Pro
Catalog SWE-bench Pro score for the resolved syndicAI model, or the vendor-published score for closed-reference models. Inferred scores are labelled. Packs without a score show as -.
61.7%

Run configuration

Endpoint model
Qwen/Qwen3.8-27B-FP8
SWE-bench Pro
61.7% · SWE-bench Pro · Qwen (Qwen3.8-27B)
Reps
5
Max shots
3
Feedback tier
1
Context cap
200K
Token budget / run
6M
Turn cap / shot
120
Thinking budget
4.096K
Sandbox digest
quality-eval-sandbox@sha256:43231173d8514576fb8c3956ad1506cb770b42ee80e8b5bfd99e342061fa3537

Review

Generated by claude-haiku-4-5

Performance Review: Qwen/Qwen3.8-27B-FP8 on Fourline Task

Verdict: Excellent task delivery. The model completed a full Connect-Four implementation with correct game logic, playable UI, and harness integration on 5/5 runs with 100% gate pass rate. All three independent evaluation criteria—spec conformance, lint/typecheck, and live scenario testing—passed in every run. The one-shot repair curve across 4 of 5 runs and perfect trap-probe compliance indicate high reliability.

The FP8 quantization poses no degradation concern; performance is consistent with full-precision baselines for this workload. Tool errors (3–6 per run) are strictly operational friction—filesystem conflicts and idempotency edge cases in edit_file—and did not cascade into logic or harness failures. Context consumption is moderate (~3.4–5.9M prompt tokens per run), with token thrift peaking at 0.41 in r2. One outlier (r3, 2-shot curve with 118 turns) converged normally; the remainder achieved single-shot green in 83–112 turns.

Strengths:

  • Perfect gate compliance: all 5 runs pass correctness, compile, lint, and held-out conformance (scenario ratio ≥ 0.917)
  • Comprehensive implementation: immutable domain logic (8×7 board, gravity, 4-in-a-row win detection, draw states), full keyboard/click UI, test harness integration (window.__fourline) conditional on ?test=1 only
  • Self-initiated quality checks: all 5 runs independently invoked typecheck, lint, and test before finish (guidelinesRead=true)
  • Trap observance: 100% spec constant compliance (board-cols=8, board-rows=7, start-player='Y') across all 5 reps

Failure modes:

  • Tool errors in 5 of 5 runs (3–6 per run): write_file EISDIR on /tmp/qe-fourline-*, edit_file idempotency mismatches and old_string misses. These are harness-side directory-cleanup timing; the model recovered gracefully in all cases
  • Scenario blips: r1 failed testids-present (Playwright timeout on cell visibility); r2–r5 passed 100%. This is a transient UI rendering timing issue, not logic error; r5's near-identical replay passed 15/15 scenarios cleanly

Serving & harness notes:

  • Context burn: 3.42–5.91M prompt tokens per run (mean ~4.5M), well within typical harness budgets; token thrift 0–0.41 reflects strategic re-use of verified code snippets (no wasteful regeneration)
  • Turnout: 83–118 turns per run (mean ~94), all single-shot green except r3 (converging 2-shot, 0.33 gap, 118 turns but fully correct)
  • Zero reasoning-only stalls: model maintained action throughout (read/edit/test cycles)
  • Vite/dev server management: no long-lived server leaks; harness integration clean

Quantization: FP8 (MXFP8-equivalent) is fully adequate. No precision-related flaws observed; tool errors and UI timing glitches are unrelated to weight precision.

Per-rep results