Beyond SWE-bench: Introducing syndicAI Quality Eval (SQE)

SWE-bench Pro is one of the best quantitative coding benchmarks we have. We use it in the syndicAI catalog, we publish it on model cards, and we still believe it. A high Pro score means a model can survive long-horizon, multi-file patch work under a serious harness.

It does not mean every checkpoint of that model will behave the same on your server, with your tools, across five messy attempts, especially when the weight pack is a weak community quant and the context window fills with failed edits.

That gap is why we built syndicAI Quality Eval (SQE): a small, opinionated, agentic quality benchmark for the kind of programming work squads actually run. Spec-driven. Multi-rep. Held-out checks against overfitting. Scored as an absolute percentage you can compare across models and quants.

Why quantitative benchmarks still matter

Leaderboards are not the enemy. SWE-bench Pro (and peers) buy you three things:

1. A shared yardstick. Vendors, labs, and catalogs can point at the same task family.
2. Scale. Hundreds of repository issues beat five hand-built fixtures for capability coverage.
3. A first filter. If a model cannot clear Everyday or Frontier coding on Pro, you probably should not bet a squad on it.

We wrote about how to read those scores in How to Read SWE-bench Pro. The short version: for human-steered engineering, clearing about 58 on Pro is already a frontier teammate. Chasing 80 is often chasing unsupervised autonomy you did not ask for.

What Pro does not buy you:

• Stability across repeated attempts (pass@1 vibes vs pass@5 reality)
• Behavior under a real tool loop (read, edit, typecheck, recover)
• Sensitivity to which weight quant you load (vendor INT4 vs a weak community AWQ)
• Resistance to "I memorized Connect Four from the internet"

Those are quality and reliability questions. SQE is aimed at them.

What SQE is

SQE is an internal coding-agent benchmark we run against OpenAI-compatible endpoints (including syndicAI squads). Each campaign:

• Spins a Docker sandbox with a seeded TypeScript workspace (SPEC.md + GUIDELINES.md)
• Runs an agentic loop with OpenCode-aligned tools: read_file, write_file, edit_file, run_command, finish
• Allows multiple repair shots after verification feedback
• Compacts long contexts (prune tool output, then summarize) the way a real session does
• Grades with a correctness gate: compile, Playwright scenarios, known-seed conformance, and held-out scenarios the agent never saw in feedback
• Publishes an absolute harness score (0 to 100%) against a theoretical perfect run, not a relative rank against other models

Default protocol: n = 5 independent reps, up to 3 shots each, feedback tier 1. Per-run token budgets are 800K (Tally), 6M (Fourline), and 20M (Blockfall). Five is enough to estimate reliability without pretending we have SWE-scale coverage. One lucky green is not a product claim. Three of five failing on token budget is a product claim. With n = 5, treat one-rep flips as noise: we report gate as k/n and round blended harness scores to whole percent.

The absolute score (headline %)

Gate pass rate (k/n) remains the honesty check. We also publish public pass rate (k/n): compile, known conformance, and scenario threshold without the held-out honesty cut. The headline SQE harness score blends five components (methodology v2):

Component	Weight	Idea
Gate	45%	Binary: full gate (correctnessPassed), including held-out honesty
Pass	25%	Binary: public verification (knownOk)
Repair	15%	Prefer one-shot greens (1 / shotsToGreen); unlocks on pass
Style / traps	10%	House-rule immutability + following the written SPEC constants; unlocks on pass
Token thrift	5%	Mild credit for not burning the whole run budget

A perfect run is 100%. Claude Opus 4.7 lands around 98% on our Fourline set (5/5 gate, 5/5 pass): full gate, all one-shots, with a few points left on style and tokens. That is the scale. Not "best in this spreadsheet equals 100%."

An overfit rep (public green, held-out fail) can still earn pass, repair, style, and thrift. It caps near 55%. Gate stays the heaviest slice.

Three tasks, one philosophy

SQE currently ships three greenfield TypeScript fixtures:

Task	Vibe	Token budget / run	Spec twist
Tally	Tiny counter UI	800K	Starts at 7, steps by 3, clamps to [0, 40]
Fourline	Connect-Four-style browser game	6M	8×7 board, win length 4, Yellow starts, row 0 is bottom
Blockfall	Falling-block puzzle	20M	12×26 field (22 visible + 4 buffer), never named "Tetris", no T-spins

Why games (and tiny UIs)

Games force the model to implement a closed world: state machine, illegal moves, win/draw, keyboard UX, and a test harness API. They compile. They lint. They run in Playwright. Failures are crisp ("scenario gravity-drop failed") instead of "the PR looks fine to me."

They also match how developers use coding agents: invent a small system from a written spec, keep guidelines, iterate until green.

Spec-driven, not vibe-driven

Every task leads with a hard SPEC and our TypeScript guidelines (pure helpers, immutability, layering). The agent is graded on following that document, not on "make something fun." Hidden verification never ships in the seed. If the model freestyles a 7×6 Connect Four because that is what Wikipedia says, the trap and held-out scenario checks catch it.

Anti-pollution by design

Training data is soaked in classic game rules. So we deliberately deviate:

• Fourline is not 7×6 with Red first. It is 8×7 with Yellow first.
• Blockfall is not a 10×20 Tetris clone. Dimensions and naming are wrong on purpose.
• Tally is not += 1 from zero.

The model has to read the SPEC and invent, not regurgitate a memorized engine. That is the same spirit as contamination-resistant SWE sets, applied to a harness we can run in an afternoon.

The agentic harness (briefly)

Under the hood, SQE is a sibling to our inference-stress tooling:

• OpenAI-compatible chat against a squad URL or any compatible API
• Tool calls parsed from the model (reasoning models keep a thinking budget)
• Sandbox isolation so pnpm / Playwright cannot touch the host
• Artifacts per rep: transcript, conversation dump, diffs, verification JSON, run log
• Re-grade without re-spend: grade reads the filesystem; you can change the scorer and backfill

We are not claiming this replaces vendor SWE harnesses. We are claiming it answers a different question with per-run receipts (transcripts, grades, and reports) we keep internally for every campaign.

Fourline results (selected models)

Protocol: task fourline, n = 5, absolute SQE harness score (blended mean, whole percent), August 2026 internal runs. Claude Opus 4.7 is the closed reference. Everyone else is an open-weight (or open-weight-quant) endpoint we actually serve or evaluated for the catalog. The live, filterable board with per-rep receipts is on the SQE Leaderboard.

Model	Quant / notes	Gate	Pass	SQE harness
Claude Opus 4.7	Closed API reference	5/5	5/5	98%
GLM-5.2	Flagship open-weight	5/5	5/5	95%
Qwen3.6 27B	Official FP8	5/5	5/5	94%
MiniMax M2.5	Official FP8	5/5	5/5	94%
Laguna S 2.1	Vendor INT4 (RC2)	5/5	5/5	93%
MiniMax M3	Official MXFP8	4/5	5/5	79%
Step 3.7 Flash	Official FP8	2/4	2/4	45%

Footnotes: Step's gate is 2/4 because one of five reps was excluded after a tripwire found tool access to leaked verification artifacts in an older harness revision (effective n = 4). We treat that rep as invalid, not as a pass.

Reading the table the way we read SWE: Opus, GLM-5.2, Qwen FP8, MiniMax M2.5 FP8, and Laguna INT4 (RC2) are in the same reliability band on this fixture. MiniMax M3 MXFP8 sits below that band with a 4/5 gate despite 5/5 public pass (one overfit). Step sits clearly lower once the tainted rep is removed. The leaderboard and the squad do not always agree.

4-bit is the default path. Quant quality is the variable.

On syndicAI, 4-bit (and other compact) quants are still how we recommend most squads run. They cut VRAM and GPU-hour cost roughly in half or better versus FP8 of the same family, which is the whole point of owning the stack: more context, more concurrency, or a smaller box for the same model class.

The mistake is treating "4-bit" as one product. A strong vendor INT4 and a weak community AWQ can share a bit width and diverge hard on agent reliability.

The upside of a good 4-bit

Laguna S 2.1 INT4 (vendor pack, RC2) posts 93% SQE on Fourline with a 5/5 gate: same reliability band as official MiniMax M2.5 FP8 and Qwen3.6 27B FP8. That is the pattern we want. In the wider ecosystem, carefully authored vendor recipes (including official low-bit DeepSeek V4 packs) are aimed at preserving tool-use behavior, not only shrinking disks.

So the catalog recommendation stays: prefer compact quants when they are good, and prefer vendor or well-calibrated packs when the catalog offers them.

Where weak packs show up (fully automated stress)

SQE is a harsh reader. It is closer to an unsupervised overnight agent than to a human-steered Cursor session. Under that lens, some community AWQ packs of otherwise strong bases fall off:

Family	Higher precision	SQE	Weaker community 4-bit	SQE
Qwen3.6 27B	Official FP8	94% (5/5 gate)	Community AWQ-INT4	65% (2/5 gate, 5/5 pass)
MiniMax M2.5	Official FP8	94% (5/5 gate)	Community AWQ	56% (3/5 gate)
Step 3.7 Flash	Official FP8	45% (2/4 gate)	Community AWQ-INT4	8% (0/5 gate, 1/5 pass)

Read that figure as a pack-quality warning, not as "avoid 4-bit." We ship compact quants because the cost curve is real. We also measure them, because a cheap checkpoint that cannot finish the tool loop is not cheap.

Human in the loop softens the cliff

Most syndicAI usage is steered: you scope the change, you catch a bad edit, you nudge the agent. In that mode, the gap between a mediocre AWQ and FP8 is usually smaller than SQE's fully automated n = 5 suggests. The model still helps. You pay more in attention and retries, not necessarily in a total washout.

SQE is useful precisely because it removes that safety net. If a pack fails here, expect more babysitting in production agents, longer sessions, and more burnt context. That may still be the right trade if the GPU bill halves and your workflow stays human-paced. It is a worse trade if you want long unattended runs.

What weak-quant failure looks like when the harness is alone

When a weak pack "almost works," the transcripts rhyme:

• Token burn without progress. Runs hit multi-million token budgets while still red.
• Edit loops. edit_file returns "old_string not found" or no-op applies; the model retries the same patch.
• Reasoning stalls. Long stretches of thinking with no tool call until the harness nudges.
• Fragile recovery. One compile error cascades; the agent cannot rebuild a mental model of the files.
• Overfit then fail held-out. Occasional known-scenario greens that die on held-out scenarios withheld from feedback.

Strong FP8 and strong vendor INT4 still make mistakes. They usually recover. That recovery is most of what SQE's gate and repair weights are paying for.

Why activation-aware quants need a good recipe

Methods like AWQ are only as good as the calibration story. A data-free or poorly calibrated community pack can keep the right file sizes and the wrong decision boundary for tool use. You still download ~100 GB+. You still pay for the GPUs. You get a model that reads the SPEC and then loses the plot in the repair loop.

Net: 4-bit remains the recommended default for cost. Choose high-quality 4-bit (vendor or proven packs) when you can. Treat anonymous AWQ mirrors as something to validate on a quality harness, not as free lunch from the SWE footnote of the base model. The tradeoff is real either way: fewer bits, lower $/hour, and a reliability curve you should actually look at.

Methodology v2 (August 2026)

In August 2026 we split the old single correctness component into gate (45%) and pass (25%). Repair and style now unlock on public pass, not only on the full gate. Token thrift is unchanged. Composite score is gone.

Later the same month we raised per-run token budgets from 500K / 4M / 15M to 800K / 6M / 20M (Tally / Fourline / Blockfall). Median successful runs were already sitting near the old ceilings, and the abort threshold was clipping recoveries that would finish with a little more room. On syndicAI people pay GPU hours, not tokens, so the budget stays a fairness stop, not a pretend scarcity signal. Runs that never hit the old abort stay valid; token thrift is re-scored against the new published budgets. Reps that aborted on exhaustion are re-run under the new caps.

Why: a public-green overfit rep used to read as ~2% in the campaign mean because failing the held-out check wiped correctness, repair, and style. That contradicted the per-rep detail page and made the leaderboard hard to trust. Gate is still nearly half the score and still heavier than pass. An overfit rep still cannot clear the gate, and it still caps near 55%.

Before / after on the packs that moved (blended mean, whole percent; 5/5 band barely moved):

Campaign	Gate	Pass	v1	v2
Qwen3.6 27B AWQ-INT4	2/5	5/5	38%	65%
MiniMax M3 MXFP8	4/5	5/5	69%	79%
MiniMax M3 NVFP4	3/5	4/5	54%	64%
Step 3.7 Flash AWQ-INT4	0/5	1/5	0%	8%
Claude Opus 4.7 (5/5 band)	5/5	5/5	95%	95%

Notable crossover: Qwen AWQ (2/5 gate, 5/5 pass) now ranks just above MiniMax M3 NVFP4 (3/5 gate, 4/5 pass) on the blended score. Read the gate k/n column for the honesty distinction. That decomposition is the point of v2.

All published leaderboard numbers are v2. Prior grade files remain on disk under grades/1.2.0.json for audit.

What we will do with SQE

SQE will not replace SWE-bench Pro on our model cards. Pro stays the wide capability signal. SQE is how we decide whether a checkpoint is fit for a syndicAI squad: parsers, KV settings, and quant included. Browse the full set on the SQE Leaderboard.

If you are choosing infrastructure for agentic coding, ask both questions:

1. Is the model capable on a contamination-resistant patch benchmark?
2. Is this build reliable under a real tool loop, five times in a row?

We built syndicAI so squads can own that second answer on hardware they control, without a token tax.

Early access is open. Join the waiting list at app.syndicai.dev.