SQE Leaderboard/DeepSeek V4 Flash 0731

DeepSeek V4 Flash 0731

deepseek/deepseek-v4-flash-0731

fourlineOfficial FP4 + FP8 Mixed SWE-bench Pro 68.6% inferredofficial2026-08-08
Methodology

SQE score

99

blended mean, 5 reps

Gate

5/5

all reps counted

Wilson 95% low

≥56.6%

lower confidence bound

Shots to green

1

median of 5 reps

Every published metric

SQE score
Headline blended mean percent across reps (gate 45 / pass 25 / repair 15 / style 10 / thrift 5).
99n=5
Gate
Reps that cleared known and held-out honesty checks, shown as k/n after tainted exclusions.
5/5n=5
Pass
Reps that cleared public verification (compile, known conformance, scenario threshold), as k/n. Distinct from gate, which also requires held-out honesty.
5/5n=5
Wilson 95%
Lower bound of the gate pass rate at 95% confidence.
≥56.6%n=5
Harness ±
Blend of all score components, mean ± spread.
98.8 ±1.3n=5
Shots to green
Feedback rounds needed before tests pass. Lower is better.
1n=5
Scenario pass
Share of graded scenarios passing in the green build.
100%n=5
Trap/spec rate
Share of trap parameter values correctly synthesized from the spec.
100%n=5
1-shot gap
How much the first shot trailed the final result.
-n=5
Effective n
Reps counted after tainted-run exclusion.
5/5n=5
Median tokens / rep
Prompt + completion tokens per rep, median over untainted reps. Lower is better.
1.52M tokn=5 untainted
Min VRAM
Catalog minimum cluster VRAM (GB) from syndicAI referenceGpuRequirements. Closed APIs and packs not in the catalog show as -.
186 GB
SWE-bench Pro
Catalog SWE-bench Pro score for the resolved syndicAI model, or the vendor-published score for closed-reference models. Inferred scores are labelled. Packs without a score show as -.
68.6%

Run configuration

Endpoint model
deepseek/deepseek-v4-flash-0731
SWE-bench Pro
68.6% · Inferred · cross-benchmark estimate vs. GLM-5.2 (Terminal Bench, NL2Repo, DeepSWE, Toolathlon, Agents' Last Exam, DSBench)
Reps
5
Max shots
3
Feedback tier
1
Context cap
200K
Token budget / run
4M
Turn cap / shot
120
Thinking budget
4.096K
Sandbox digest
quality-eval-sandbox@sha256:242ebd2f73c1c1ffa90c3ccb7913574f91e74a89dd1e4cf874f907a3129149c5

Review

Generated by claude-haiku-4-5

Performance Review: deepseek/deepseek-v4-flash-0731 on fourline

Task Success & Reliability

The model completed the Connect-Four TypeScript task at 5/5 gate pass with 100% scenario conformance across all five runs. All checklist items passed (correctness, overfitting, compile, lint, scenario execution, held-out conformance). First-shot success rate was perfect: every run terminated on shot 1 with stoppedAtGreen=true. Final scores ranged 98.06–99.36% with zero flaky scenarios (0 variance in final-shot outcomes across runs).

Task Execution & Efficiency

The model read SPEC and GUIDELINES, implemented domain logic (game board state, gravity, win detection across horizontal/vertical/diagonals, draw logic, invalid-drop rejection), wired the playable UI with all required data-testid attributes, and added dual-mode harness integration (playable keyboard controls in standard mode; window.__fourline API under ?test=1). Code was idiomatic TypeScript (immutable GameSnapshot, pure functions) and passed all validation: pnpm typecheck, pnpm lint (0 errors, only warnings), and held-out test scenarios including diagonal win and full-board draw. Token thrift varied (0.61–0.87) but was adequate for the scope.

Strengths

  • Perfect gate pass and scenario coverage: 5/5 runs green, 100% scenario ratio, both public and held-out tests passing on first shot.
  • Self-initiated verification discipline: Model proactively ran typecheck, lint, and build checks multiple times across the implementation. Self-verification reduced diagnostic turns.
  • Correct immutable TypeScript: Domain logic followed GUIDELINES: pure drop(snapshot, col) function, no mutations, readonly types. Rejection of invalid drops via unchanged-snapshot return was correct.
  • Dual-mode harness integration: Playable interactive mode (keyboard: arrows, enter, digits 1–8) and test API (getState/drop/reset) coexisted cleanly; no cross-contamination.
  • Spec adherence: Board geometry (8×7, row 0 bottom, gravity), player alternation (Y first), win/draw/rejection logic all matched SPEC exactly.

Failure Modes

  • Minor tool errors: 1–11 tool call errors per run (mostly edit_file old_string mismatches in turns 51, 53, suggesting mid-refinement friction). All were recovered on-turn without branching.
  • Lint warnings retained: 7 functional/no-let warnings in final code. Model correctly identified these as non-blocking (linter exit 0, warnings only), but did not address the stylistic violations (likely acceptable per project config, but slightly untidy).
  • Turn variance: Turn counts ranged 39–80 (mean ~63, SD ≈17), reflecting minor restarts mid-implementation, though all converged to green on shot 1.

Serving & Harness Notes

  • Negligible stall risk: Single-shot repair curve on all runs; no reasoning-only turns. Effective n=5 (no tainted runs excluded).
  • Context management: Prompt tokens peaked at 2.28M (r4); completion tokens 24k–49k. No context-budget stress or truncation observed; build/verification stayed within harness time budgets (283–594 sec wall time).
  • Tool errors were transient: 2–11 edit_file errors per run were immediately corrected on re-attempt (typical "old_string not found" during refactoring). No loops or retries required beyond the same turn.

Recommendation: Model is reliable and high-quality for this task tier. Stable single-shot success, correct domain logic, and good self-verification discipline make it suitable for production coding tasks of similar scope.

Per-rep results