Step 3.7 Flash

Performance Review: stepfun-ai/Step-3.7-Flash-FP8 on Fourline Task

Overall Verdict

The model achieved 2 of 4 passes (50% gate rate) with wide variance in task completion. On successful runs (r2, r5), it produced correct, playable implementations that passed all scenarios and held-out conformance tests. On failure runs (r3, r4), it exhausted token budget or regressed, leaving critical game logic broken (gravity-drop and keyboard-input failures). The FP8 quantization did not compromise baseline quality; instability stems from agent-loop behavior, excessive tool retries, and context management.

Positives: r2 solved the task in a single shot (38 turns, ~1.3M tokens) with clean typecheck/lint and full scenario pass. r5 also passed but required 101 turns and ~3.7M tokens, showing the model could self-correct when prompted to continue. Both passing runs demonstrate capability to implement immutable TypeScript game logic, UI harness integration, and spec-compliant behavior.

Negatives: r3 and r4 hit irreversible failure modes. r3 ran to token exhaustion (106 turns, ~4M tokens) with 5 tool errors and stalled reasoning loops, achieving only 83% scenario pass before aborting. r4 regressed (final shot 83% → 0% pass ratio) after 95 turns, producing non-compilable code. Neither had a recovery path within the session.

Strengths

• Single-shot success (r2): Correctly implemented game domain (immutable state, win detection, gravity), UI (board render, keyboard controls), and test harness API in 38 turns with 1 tool error tolerated.
• Scenario coverage on green runs: Both r2 and r5 passed 100% of scenarios including held-out (diagonal-win, full-board-draw), indicating spec-compliant logic.
• Self-initiated verification: All runs showed guidelinesRead=true, selfInitiatedTypecheck=true, selfInitiatedLint=true, suggesting proactive quality checks.
• Code structure: When successful, code was well-organized (game logic in src/game/, UI in src/main.ts), followed immutability guidelines, and exposed the harness API correctly under ?test=1 only.

Failure Modes

• Tool-error loops (r3, r4): r3 accumulated 5 tool errors (edit_file string-match failures) across 106 turns, indicating poor state tracking or drift. r4 had 4 errors; both suggest the model lost context on file contents and made identical-change edits.
• Reasoning-only stalls (r3): Turn 25 logged stall (no tools) (reasoning-only), a sign of agent confusion without tool invocation to resolve the impasse.
• Token budget exhaustion: r3 and r4 both burned ~4M prompt tokens without achieving green. r3 hit budget at turn 106 with 2 scenario failures (gravity-drop, play-mode-keydown). r4 exhausted at turn 95, leaving the app non-compilable.
• Regression (r4): Final shot regressed from single-shot 83% to 0% (non-compiling code), with repairCurve: regressing and gap -0.83, indicating the model corrupted working code while attempting fixes.

Serving & Harness Notes

• Turn density: Passing runs (r2: 38 turns; r5: 101 turns) vs. failing runs (r3: 106 turns; r4: 95 turns) show high correlation: runs that succeeded did so early, while failing runs monotonically consumed tokens trying to fix errors.
• Tool error cascade: r3's 5 edit_file failures (old_string not found, identical-change rejects) suggest the model's internal representation of files diverged from actual content—likely due to long context and many edits without explicit re-reads.
• No stall recovery: When stalls occurred (r3, turn 25), the nudge to continue did not reset behavior; the model continued looping.
• Context burn: r3 burned 3.98M prompt tokens in a single shot; r4 burned 3.97M; r5 (which passed) also used 3.69M but achieved green. Prompt size was not the bottleneck; tool-use quality and convergence speed were.
• Shots to green: When green was reached, it came early (r2: 1 shot, 1.5 average turns; r5: 2 shots, converging curve). No case of green on shot 2+ recovery within budget, suggesting the agent does not effectively replan after mid-session failure.

Recommendation

The model is capable but inconsistent on this task. Success depends on early convergence (first 50–70 turns) and avoiding tool-error loops. For production use:

• Accept single-shot success (r2 pattern): ~50% probability of high-