Performance Review: nvidia/MiniMax-M3-NVFP4 on fourline

Overall Verdict

The model achieved 4/5 gate passes with 5/5 scenario pass-rate and zero flaky runs. All green shots completed in a single attempt. The model demonstrates strong correctness and self-correction behavior, reliably hitting compilation, linting, and conformance targets. However, one gate failure (r4) shows a systematic correctness issue: the draw-detection logic failed a held-out conformance test (reported "won" instead of "draw"), indicating incomplete handling of terminal game states under certain board configurations. The failure occurred despite passing all public scenarios, suggesting an edge case in win-vs-draw precedence or full-board evaluation.

Three of five runs incurred moderate edit-file friction (4–18 tool errors per rep), primarily identical old_string/new_string mismatches. This friction, combined with moderately high tool error rates and token burn (0.33–0.41 blended token thrift), suggests the model sometimes over-edited or re-attempted edits without reading file state, rather than making surgical changes. No stalls or network failures impacted green achievement. The NVFP4 4-bit quantization did not appear to degrade behavioral correctness; failures align with logic bugs and editing friction, not precision loss.

Strengths

• 100% single-shot repair: all reps reached green in one attempt with no multi-shot loops.
• Strong scenario coverage: 5 of 5 reps passed all public scenarios (board cols/rows, gravity, win conditions, UI playability, keybinds).
• Self-directed quality checks: consistently ran pnpm typecheck, pnpm lint, and self-initiated verification without explicit instruction.
• Playable interactive mode: successfully implemented visible board, cell markup, keyboard controls (arrow keys, digit keys 1–8, Enter/Space), and column indicator—all graded in live Playwright tests.
• Test harness API: correctly exposed window.__fourline with getState(), drop(), and reset() (gated to ?test=1).
• No flaky scenarios: deterministic execution; all 5 runs produced consistent test results.

Failure Modes

• Correctness regression in r4: held-out conformance test detected status: "won" when board was full with no four-in-a-row—should be status: "draw". Gate marked overfittingFailure: true. Public scenarios passed, so the failure is an untested edge case, not broad misunderstanding.

• Edit-file friction:

  • r1: 4 tool errors (3× identical old_string/new_string, 1× old_string not found)
  • r2: 12 errors (mostly identical mismatches)
  • r3: 9 errors
  • r4: 18 errors (consistent pattern of idempotent edits)
  • r5: 5 errors (lowest friction)

  Pattern suggests incomplete file reads or stale context before attempting edits, leading to retry attempts.

Serving & Harness Notes

• Tool errors (mean 9.6 per rep): driven by edit-file idempotency mismatches, not tool unavailability. No indication of harness instability.
• Context burn: token usage ranged 3.5M–5.4M prompt tokens, with 4 of 5 reps near or at token budget caps (0.08–0.41 token thrift score). r3 achieved best efficiency (0.0877 thrift) but used the most prompt tokens, suggesting verbose exploration.
• Turn count: 111–120 turns per rep. r4 was capped at 111 by token budget, yet still reached green—indicates deterministic, linear progress rather than search or backtracking.
• Fetch transient retries: r1 log shows 4 transient HTTP failures with exponential backoff; all were retried successfully and did not block green.
• No reasoning-only turns: all turns included tool calls (edit_file, run_command, read_file). No evidence of stuck-in-thinking loops.

Summary

Recommendation: Green-gate success rate (4/5) is acceptable for a coding eval, and the single failure is a correctness bug, not a model capacity issue. Edit-file friction is higher than ideal but did not prevent task completion. NVFP4 quantization is not a limiting factor. Model is suitable for deployment with standard CI/CD gate requirements (typecheck + lint + conformance); consider re-testing draw-vs-win edge case or adding stricter board-full checks upstream.