SQE Leaderboard/Kimi K2.7 Code

Kimi K2.7 Code

moonshotai/kimi-k2.7-code

fourlineFlagship open-weight SWE-bench Pro 58.6% official2026-08-11
Methodology

SQE score

99

blended mean, 5 reps

Gate

5/5

all reps counted

Wilson 95% low

≥56.6%

lower confidence bound

Shots to green

1

median of 5 reps

Every published metric

SQE score
Headline blended mean percent across reps (gate 45 / pass 25 / repair 15 / style 10 / thrift 5).
99n=5
Gate
Reps that cleared known and held-out honesty checks, shown as k/n after tainted exclusions.
5/5n=5
Pass
Reps that cleared public verification (compile, known conformance, scenario threshold), as k/n. Distinct from gate, which also requires held-out honesty.
5/5n=5
Wilson 95%
Lower bound of the gate pass rate at 95% confidence.
≥56.6%n=5
Harness ±
Blend of all score components, mean ± spread.
98.7 ±1.4n=5
Shots to green
Feedback rounds needed before tests pass. Lower is better.
1n=5
Scenario pass
Share of graded scenarios passing in the green build.
100%n=5
Trap/spec rate
Share of trap parameter values correctly synthesized from the spec.
100%n=5
1-shot gap
How much the first shot trailed the final result.
-n=5
Effective n
Reps counted after tainted-run exclusion.
5/5n=5
Median tokens / rep
Prompt + completion tokens per rep, median over untainted reps. Lower is better.
1.25M tokn=5 untainted
Min VRAM
Catalog minimum cluster VRAM (GB) from syndicAI referenceGpuRequirements. Closed APIs and packs not in the catalog show as -.
739 GB
SWE-bench Pro
Catalog SWE-bench Pro score for the resolved syndicAI model, or the vendor-published score for closed-reference models. Inferred scores are labelled. Packs without a score show as -.
58.6%

Run configuration

Endpoint model
moonshotai/kimi-k2.7-code
SWE-bench Pro
58.6% · Proxy from Kimi K2.6 SWE-bench Pro (no K2.7 Code score published)
Reps
5
Max shots
3
Feedback tier
1
Context cap
200K
Token budget / run
6M
Turn cap / shot
120
Thinking budget
4.096K
Sandbox digest
quality-eval-sandbox@sha256:242ebd2f73c1c1ffa90c3ccb7913574f91e74a89dd1e4cf874f907a3129149c5

Review

Generated by claude-haiku-4-5

Performance Review: moonshotai/kimi-k2.7-code | Fourline Task

Verdict: Excellent execution on an intermediate-complexity interactive TypeScript task. The model delivered full implementation in a single shot across all 5 repetitions, achieving 100% gate pass and 99% blended score. All functional requirements met: immutable domain logic, interactive UI with correct data-testids, keyboard controls, and test-harness API compliance. No flaky runs; consistent, reliable behavior.

Reliability: The model demonstrated strong understanding of requirements upfront. It read SPEC and GUIDELINES, structured code cleanly under src/game/, wired UI in src/main.ts, and proactively validated via keyboard interaction and harness API calls mid-run (turns 183–187). All 5 runs reached green on first shot with 100% scenario pass rate, including held-out conformance checks.

Efficiency: Variable token efficiency across reps (58–86% thrift) due to differing conversation lengths (46–86 turns). Wall-clock times ranged 93–1.07M ms; variance reflects genuine exploration and manual verification of keyboard controls and test-mode API exposure. Tool errors (3–5 per run) were low-value friction: identical-string edit-file errors and JSON formatting errors on finish call, neither blocking nor architectural.

Strengths

  • First-shot success: All 5 reps passed gate and lint on the initial attempt; no retries needed.
  • Spec compliance: Correctly implemented 8×7 board, gravity drops, 4-in-a-row detection, Yellow-first player order, and rejected moves (full columns, out-of-range, game-over state).
  • Immutable domain: Pure functions in src/game/types.ts, board.ts, drop.ts, checkWin.ts. GameSnapshot immutable; state updates via new object spread.
  • UI integration: All required data-testids (board, cell-r{r}-c{c}, current-player, status, column-indicator) rendered correctly in both normal and test modes.
  • Keyboard controls: Digit keys (1–8), Arrow keys (left/right cursor), Enter/Space verified working in interactive testing (run.log turns 183–187).
  • Test harness isolation: window.__fourline conditionally exposed only under ?test=1; play mode does not expose API.
  • Self-initiated QA: Proactively ran pnpm typecheck and pnpm lint mid-run; manually verified Playwright scenarios against live server (turn 183+).

Failure Modes

  • Tool-call friction: 3–5 edit_file errors per run (e.g., "old_string not found", "identical old_string and new_string"). Low-impact; model recovered by re-reading files and refining patches.
  • JSON formatting on finish: Turns 116–117 (r1) and 66–67 (r5) show finish rejections due to malformed JSON (unclosed quote). Benign; model resubmitted with correct format.
  • Lint warnings (unfixed): r5 final lints shows 4 functional/no-let warnings in src/game/logic.ts (lines 32, 53–55). Flagged but not addressed; does not block green gate (warnings ≠ errors).

Serving & Harness Notes

  • Context burn: r1 consumed 831K prompt tokens (47 turns); r5 consumed 2.5M (86 turns). Steep climb suggests verbose intermediate exploration and conversation overhead, but no evidence of degradation or OOM.
  • Stalls: None observed. All runs completed within allocated turn budget (≤120) and wall-clock limits.
  • Harness interactions: Scenarios 15/15 passed (including held-out win-diagonal and full-board-draw). Conformance replay and trap probe (board-cols: 8, board-rows: 7, start-player: Y) all correct.
  • Manual e2e validation: Turn 183 confirmed UI visible, API hidden in normal mode; turn 187 confirmed digit-press and arrow-key cursor movement; drop(0) state transition verified. No harness re-runs needed.

Recommendation: Ship with confidence. The model is production-ready for this task class. Minor improvements would address lint warnings and reduce tool-call friction, but both are low-severity and do not affect correctness or user experience.

Per-rep results