Step 3.7 Flash

Performance Review

Step-3.7-Flash-AWQ-INT4 on the fourline task: Failed comprehensively across 5 runs. Zero gate passes (0/5); only 1 of 5 runs achieved any scenario pass (20% overall). 4 of 5 hit token budget exhaustion; the remaining run plateaued without improvement. Repair attempts were ineffectual (median shotsToGreen: 2 in the single converging case; the rest single-shot or plateauing).

The model demonstrated marked difficulty recovering from tool-use errors and logical mistakes. All trap scenarios (board-cols, board-rows, start-player) yielded neither, indicating the final code failed to execute or validate behavior at all. Most critically, every run failed compilation — the model repeatedly generated syntactically or semantically malformed TypeScript, applied identical edits that had no effect, and ran out of tokens before achieving a correct implementation. Runs r1, r4, r5 produced uncompilable code from the start; r3 achieved 83.3% scenario pass but still failed the ≥90% gate.

Strengths

• Proactive verification: Model consistently initiated pnpm typecheck and pnpm lint self-checks (all 5 reps).
• Tool read discipline: Properly read SPEC and GUIDELINES upfront in most runs.
• Low tool-call errors relative to turns: Runs r2, r3, r4, r5 had 2–5 tool errors across 98–109 turns (relatively low frequency).

Failure Modes

• Widespread compilation failure: 4 of 5 final shots did not compile. Errors include incorrect .ts imports, readonly array type mismatches, unused imports/vars, missing allowImportingTsExtensions in tsconfig, and property not found on Window.
• Ineffectual edits: Multiple runs logged edit_file ERROR: No changes to apply (r1 turns 53, 58, 63, 75; r5 turns 62, 70), suggesting the model made identical old_string and new_string replacements—indicating confusion or failure to track state.
• Stalling (reasoning-only loops): r1 turn 72 and r5 turns 81, 82 entered "no tools" reasoning-only modes, nudged by harness, wasting tokens on internal deliberation.
• finish rejections: r5 called finish prematurely at turn 7, claiming code was done when it was manifestly incomplete.
• Token burn without convergence: All 5 runs exhausted the token budget (2.3–3.9M prompt tokens). No run converged to a green pass; r2's single-shot success (40.14 harnessScore) was an outlier, likely overfitting to known-OK tests but failing held-out conformance.

Serving & Harness Notes

• Context exhaustion: Turns ranged 98–360 per run; most ran 100+ turns hitting token limits. The longest run (r4, 360 turns) achieved only 3.02% harnessScore and spent 82% of token budget with minimal code progress.
• Tool-call error patterns: Sparse but sticky (edit_file no-ops, old_string not found). Not symptomatic of high-bit quantization noise; rather, model state tracking failure.
• Single-shot + plateauing repair curves (r1, r3, r4, r5): Model generated code, ran into errors, made minimal or zero progress on retries. Did not converge.
• Early termination risk: r5 attempted finish at turn 7 with compiling code—model misjudged readiness.

Root Cause

This is not a quantization artifact. AWQ INT4 can sustain competitive coding; the issue is task execution discipline and state tracking under complexity. The model:

• Failed to maintain correctness across sequential edits (copy-paste placeholders, identical edits).
• Did not adequately debug compilation errors (e.g., missing tsconfig options).
• Wasted significant tokens on redundant reasoning or tool loops without recovery strategy.
• Could not scale the multi-file interaction (types, logic, main UI) reliably within a single shot.

Recommendation for ops: This model is not production-ready for medium-complexity TypeScript tasks requiring multi-file coordination and iterative debugging. Simpler single-file tasks may fare better. Consider fallback to a larger or higher-precision variant for this workload.