Qwen3.6 35B-A3B

Performance Review: Qwen3.6-35B-A3B-FP8 on Fourline Task

Verdict: Model demonstrated moderate reliability with critical reliability gaps. Of 5 runs, only 2 (40%) passed the gate; 3 exhibited overfitting to training scenarios that failed held-out conformance. Gate passage requires both correctness and held-out conformance—2/5 at 0.12 blended pass rate reflects a model that solves the immediate task but fails generalization. When successful, output was clean and efficient (89% harness score, good token thrift). Failures stemmed from logical oversights in game-state management, not quantization.

Reliability snapshot: All runs compiled and linted. All 5 achieved 100% scenario pass on known scenarios (trap-spec rate: 1.0), but 3 failed held-out conformance. Converging repair curves (2 shots median, all runs) suggest the model self-corrected incremental bugs but could not escape fundamental logic errors in early shots. Two successful runs demonstrate the model is capable on this task; three failures show inconsistent validation of game rules.

Strengths

• Self-directed quality gates: Model independently initiated pnpm typecheck and pnpm lint in all runs; proactively read guidelines before implementation.
• Debuggable iterations: Failed runs showed structured debugging—reading error logs, attempting surgical edits, re-verifying—rather than thrashing.
• Token efficiency: Successful runs (r1, r4) achieved 32–34% token thrift; even failed r5 hit 23%. No wasteful context burn.
• Architectural cleanliness: Code structure (domain logic in src/game/, UI in src/main.ts) followed GUIDELINES.md immutable-TypeScript patterns when correct.

Failure Modes

• Overfitting to known scenarios (3/5 runs fail held-out): Gravity-drop and play-mode-keydown scenarios passed in training but failed on held-out variants. Root cause: r2 and r3 passed 14/15 known scenarios but broke on full-board-draw (expected 56 moves, got 18) and conformance mismatches. r5 achieved green but message logs show heavy edit-file error churn (20 tool errors across 113 turns).
• Edit-file inefficiency loop: r2 had 23 tool errors, r5 had 20; both show repeated "No changes to apply: old_string and new_string are identical" (e.g., r2: turns 21, 25, 33, 44, 48, 52, 55, 62, 64, 67, 73, 75, 84). This indicates failed substring matching or misalignment of intent—model attempted edits that were already in place.
• Reasoning-only stalls: r2 and r5 both experienced 2–3 "stall (no tools) (reasoning-only)" nudges (r2: turns 57, 85, 92; r5 exceeds), suggesting model got stuck deliberating without meaningful progress.
• Held-out conformance degradation: r2 and r3 both reported heldOutConformance.passed: false with firstDivergenceTick: 0, meaning the game state diverged immediately on replay—indicating rule violations or state initialization bugs not caught by 14 known scenarios.

Serving & Harness Notes

• Token consumption: Runs ranged 3.6–6.0M prompt tokens / 37–95k completion tokens over 1–2 shots. Successful runs capped out at ~4M prompt after heavy context burn; failed runs saturated longer before gate rejection.
• Tool errors vs. logical errors: The 12–23 edit-file errors per run mostly reflect tool-use friction (string not found, identical old/new), not syntax failures. Compiles always passed; lint always passed. This suggests the model understood the target but struggled with precise text replacement.
• Repair curve pattern: All runs converged in 1–2 shots; no exponential explosion. Median "shots to green" = 2.0 with tight spread (1.0–2.0). This is good for convergence but masks the fact that 3 runs converged to a false positive (all known scenarios pass, held-out breaks).
• No stall loops or infinite retries: Longest runs capped at 141 turns (r2) on token budget, not runaway tool loops. Harness nudging on reasoning-only stalls worked.

Recommendation: Model is production-marginal for interactive games requiring stateful validation. Strengths (architecture, self-testing, efficiency) are real; weaknesses (overfitting to training scenarios, edit-file churn, generalization) are critical. Two successful runs (r1, r4) show the model can implement Fourline correctly, but 60% failure rate on held-out conformance disqualifies it for reliable deployment. For open-weight quantized models, FP8