Qwen3.6 27B

Performance Review: cyankiwi/Qwen3.6-27B-AWQ-INT4 on fourline

Verdict

The model achieved 2/5 gate pass (40% success rate), with reliable first-shot performance in passing runs (94% scores when green) but high failure variability due to overfitting to visible training scenarios. In the two passing runs (r1, r2), it correctly produced a complete, working implementation within a single shot — all scenarios passing, typecheck and lint clean. In the three failing runs (r3–r5), it passed 6–8/12 visible scenarios but failed hidden conformance checks, indicating the model learned the visible test cases rather than the underlying specification. The model demonstrates solid tooling discipline and reasoning, but lacks robust generalization to unseen edge cases.

Critical limitation: Overfitting on held-out conformance indicates the model's logic may be superficially correct for known tests but incomplete or fragile. Run r3 needed 2 shots to reach green (converging), while r4–r5 required only 1 shot but still failed hidden checks — suggesting the model gets lucky on first shot in some runs but cannot guarantee specification fidelity.

Strengths

• Self-initiated verification: All runs show guidelinesRead: true and auto-triggered typecheck/lint after key edits, demonstrating proactive quality discipline.
• Single-shot success: Runs r1 and r2 solved the full task in one shot with 0 errors; lint passes with only minor warnings (no blocking issues).
• Tool reliability: Low tool-error rates (2–7 per run) and no catastrophic loops despite edit_file friction (e.g., identical old_string/new_string errors in r1).
• Reasoning-only stall recovery: Turn 52/120 in r1 shows a warning for reasoning-only stall, but the model nudged forward and recovered without retry.

Failure Modes

• Overfitting (runs r3–r5): All three failing runs pass 8–9 of 12 visible scenarios but fail held-out conformance (heldOutConformanceOk=false). Indicates the model may have learned specific test patterns (e.g., exact board layout, sequence of drops) rather than robust game logic.
  • r3: scenarioPassRatio 0.917, fail on full-board-draw (expected 56 moves, received 18); conformance divergence at tick 0.
  • r4–r5: Pass public scenarios but fail hidden conformance on first tick.
• Finish rejection (r3, first shot): At turn 55, model called finish prematurely with a false claim of completeness. Harness rejected with 5 scenario failures. Model recovered on shot 2.
• Token thrift variability: Completion token spread is wide (33k–51k); shot 2 of r3 burned 51k tokens despite converging to the same solution size, suggesting verbose error-recovery dialogue.

Serving & Harness Notes

• No context-burn risk: Prompt tokens ~2.5–3.0M per run; budget utilization reasonable (65% blended).
• Tool errors non-fatal: edit_file errors (old_string not found, identical changes) in r1, r3 are routine; model handles them by re-reading or retrying.
• Turn efficiency: Runs that pass (r1, r2) converge in ~79–88 turns; r3 shot 2 takes 75 turns. No runaway loops; all within safe limits.
• Lint/typecheck integration: Model consistently runs both checks; all final builds pass with 0 errors.

Root Cause Analysis

The INT4 (AWQ) quantization is not a plausible cause for overfitting — overfitting is a training-time generalization failure, not a precision artifact. The model's issue is specification fidelity under variation: it correctly implements visible scenarios (gravity, basic wins, board geometry) but appears to miss edge cases or subtle game-state transitions that the hidden conformance tests expose (e.g., move count in full-board scenarios, state consistency across replay seeds). Likely gaps: move counting, draw detection robustness, or state mutation in unexpected conditions.

Recommendation: Model is suitable for known, bounded tasks with visible test coverage, but not for high-stakes open-spec domains. Consider supplementary unit-test verification or tighter specification validation steps in the agentic loop before finish to catch overfitting earlier.