Quants & context for coding agents
Why this matters on syndicAI
syndicAI is built around a simple idea: sharing a dedicated GPU in a squad usually beats metered API usage for serious coding, especially if you run agents for many hours or push long contexts. Power users especially benefit from predictable GPU time instead of usage that spikes with every long agent turn. You are picking a model variant and how much history you load so the run stays fast, stable, and good enough on real hardware.
Agentic coding sends long, noisy prompts: trees, tool output, diffs, and multi-turn history. Two levers matter most:
- Quantization, how weights (and sometimes activations) are stored on the vLLM GPU server.
- Effective context length, how much of that history fits in memory at once. Longer is not always better: KV cache grows with sequence length, so huge windows eat VRAM and can slow generation even when the model weights fit.
The sections below are practical advice for squads, grounded in what vLLM supports and what syndicAI actually provisions.
W×A notation (quick primer)
Quant labels often read W{weights}A{activations}:
| Notation | Meaning | Examples on syndicAI |
|---|---|---|
| W8A8 | 8-bit weights and 8-bit activations | FP8, MXFP8 |
| W4A16 | 4-bit weights, 16-bit (BF16/FP16) activations | AWQ, INT4, NVFP4 W4A16 |
| W4A4 | 4-bit weights and 4-bit activations | NVFP4 (default ModelOpt recipe) |
| W16A16 | Full 16-bit | BF16 / FP16 checkpoints |
Lower bit widths shrink weight memory and bandwidth. Whether activations are also quantized affects quality, speed, and which GPU tensor cores you need.
Quantization formats at a glance
| Format | Typical W×A | Weight size vs BF16 | Quality (coding) | syndicAI takeaway |
|---|---|---|---|---|
| BF16 / FP16 | W16A16 | 1× (largest) | Reference | Rarely worth the VRAM on syndicAI when FP8 exists for the same model. |
| FP8 | W8A8 | ~½× | Very close to BF16 on broad evals (vLLM FP8) | Default “quality first” when available. |
| MXFP8 | W8A8 + block scales | ~½× (similar to FP8) | Comparable to FP8; block-32 microscaling can track outliers better on supported hardware | Best on Blackwell; official checkpoints (e.g. MiniMax M3 MXFP8) or vLLM online --quantization mxfp8. |
| AWQ | W4A16 | ~¼× | Strong; occasional edge-case slips vs FP8 | Workhorse for big MoE on finite GPUs — syndicAI reference config (MiniMax M2.5 AWQ on 2× RTX PRO 6000). |
| INT4 | W4A16 | ~¼× | Similar footprint to AWQ; format depends on checkpoint (e.g. compressed-tensors) | Same practical advice as AWQ — watch your failure modes. |
| MXFP4 | W4A4 (OCP microscaling) | ~⅛× | Good 4-bit quality for the size; different format from NVFP4 | Needs Blackwell (native) or Hopper (emulated). Open OCP standard — also runs on AMD CDNA4. |
| NVFP4 | W4A4 or W4A16 | ~⅛× (W4A4) | W4A4 is fastest on Blackwell; W4A16 keeps activations in BF16/FP16 for a quality bump | Blackwell only (B200/B300, RTX 50-series, RTX PRO 6000 Blackwell). Not interchangeable with MXFP4. |
Rule of thumb: prefer FP8 (or MXFP8 on Blackwell) for quality; use AWQ / INT4 to fit frontier MoE on fewer GPUs; reach for MXFP4 / NVFP4 when you have Blackwell and want the smallest weight footprint. If AWQ quality slips, try FP8 before BF16.
GPU compatibility
Not every quant runs on every card. syndicAI matches offers using the same architecture buckets you see in the create-squad wizard. Below is the global compatibility matrix (specific models can be stricter via overrides).
| Quant | Native (full-speed tensor cores) | Emulated (software path, ~1.05–1.2× VRAM) | Not supported |
|---|---|---|---|
| FP8 | Ada · L40S, RTX 40 · Hopper · H100/H200 · Blackwell · B200/B300, RTX 50, RTX PRO Blackwell · AMD MI300/MI350 | Ampere DC · A100, A30 | Volta, Turing, Ampere consumer (A40, RTX 30) |
| AWQ / INT4 | Turing onward · T4, RTX 20+ · all Ampere, Ada, Hopper, Blackwell · AMD MI300/MI350 | Volta · V100 | — |
| BF16 | All NVIDIA from Volta onward | AMD MI300/MI350 (FP16 path) | — |
| MXFP8 | Blackwell DC/consumer · B200/B300, RTX 50, RTX PRO Blackwell · AMD MI350 | Hopper · H100/H200 · Ada · L40S, RTX 40 | Volta, Turing, Ampere, AMD MI300 |
| MXFP4 | Blackwell · B200/B300, RTX 50, RTX PRO Blackwell · AMD MI350 | Hopper · H100/H200 | Ada and older |
| NVFP4 | Blackwell only · B200/B300, RTX 50, RTX PRO Blackwell | — | Hopper, Ada, Ampere, and everything older |
What “native” vs “emulated” means
- Native — the GPU has matching low-precision tensor cores (or a validated kernel). This is what you want for production coding agents.
- Emulated — vLLM falls back to a dequant or Marlin-style path. It works, but uses more VRAM and is usually slower. syndicAI may still provision emulated paths when no native offer exists, but prefer native when you can.
- Not supported — the quant cannot run reliably on that architecture; pick a different variant or GPU generation.
syndicAI hardware in practice
- Reference squad config (Ada): 2× RTX PRO 6000 (96 GB) running MiniMax M2.5 AWQ — AWQ and FP8 are native; MXFP8, MXFP4, and NVFP4 require Blackwell.
- Blackwell datacenter (B200/B300): unlocks NVFP4, MXFP4, and MXFP8 at full speed — the formats NVIDIA and vLLM optimize hardest for in 2026.
- Blackwell consumer (RTX 5090, RTX PRO 6000 Blackwell): also runs NVFP4 and MXFP4/8 natively; best for fitting 400B+ MoE into a single node.
- Hopper (H100/H200): strong home for FP8; MXFP4 and MXFP8 run but via emulated paths with extra VRAM headroom.
When you pick a model in the app, the requirements card shows exactly which architectures are native vs emulated for that checkpoint — use it as the source of truth for your selection.
Format details
FP8 (W8A8)
Per-tensor or per-channel FP8 is vLLM’s mainstream production quant: roughly half the weight memory of BF16 with small accuracy deltas on broad benchmarks. On Ada, Hopper, and Blackwell it hits native FP8 tensor cores. A100 (Ampere DC) can run FP8 via an emulated kernel with a ~20% VRAM penalty.
MXFP8 (microscaling FP8)
MXFP8 is an OCP microscaling variant: FP8 element type plus block-32 scale factors. vLLM supports pre-quantized ModelOpt checkpoints (quantization="modelopt_mxfp8") and online conversion (--quantization mxfp8) for both dense and MoE layers.
Specialty: on Blackwell, MXFP8 uses hardware block-scaling paths (FlashInfer / CuTeDSL kernels) that plain per-tensor FP8 does not. On Hopper and Ada, vLLM dequantizes the block scales and runs the standard FP8 path — emulated, with extra VRAM.
AWQ and INT4 (W4A16)
AWQ protects “salient” weight channels using activation statistics — a mature W4A16 format with fast Marlin/GPTQ-style kernels on almost every GPU syndicAI rents (Turing and newer).
INT4 checkpoints (e.g. some vendor compressed-tensors builds) occupy a similar footprint. Treat them like AWQ: excellent for fitting 200B+ MoE models, with occasional quality gaps vs FP8 on edge cases. Move to FP8 if you see systematic mistakes.
MXFP4 vs NVFP4 — not the same 4-bit
Both are ~4-bit, but the on-disk format and scaling rules differ:
| MXFP4 (OCP) | NVFP4 (NVIDIA) | |
|---|---|---|
| Standard body | Open Compute Project microscaling | NVIDIA ModelOpt / TensorRT-LLM ecosystem |
| Typical W×A | W4A4 | W4A4 (default) or W4A16_NVFP4 |
| Native GPUs | Blackwell + AMD CDNA4; Hopper emulated | Blackwell only (datacenter + consumer) |
| vLLM flag | mxfp4 / compressed-tensors |
modelopt_fp4, quant_algo: NVFP4 or W4A16_NVFP4 |
| Cross-vendor | Yes (OCP; AMD MI350) | No — needs NVIDIA FP4 tensor cores |
Do not assume an MXFP4 checkpoint runs as NVFP4 or vice versa. Pick the variant that matches your GPU generation.
NVFP4 and W4A16_NVFP4
NVFP4 W4A4 quantizes both weights and activations to 4-bit — maximum throughput on Blackwell FP4 tensor cores. MoE models may route through FlashInfer fused-MoE FP4 backends.
W4A16_NVFP4 (ModelOpt recipe) keeps 4-bit NVFP4 weights but runs BF16/FP16 activations. vLLM dispatches this to the Marlin FP4 kernel — slightly more VRAM than W4A4, often better quality on hard coding tasks. Checkpoints declare quant_algo: W4A16_NVFP4 in hf_quant_config.json.
NVFP4 does not run on Hopper or Ada in syndicAI’s compatibility policy, even though vLLM has experimental Marlin emulation on older cards — the performance and MoE coverage on pre-Blackwell hardware is not production-grade for the models we serve.
BF16 / FP16
Full-width 16-bit weights. Useful when no quantized sibling exists or a vendor ships BF16-only. Expect 2× the weight VRAM of FP8 and 4× that of 4-bit formats — fine for smaller dense models, expensive for 400B+ MoE.
Choosing a variant (decision guide)
- Same model, multiple quants in the catalog? Check the requirements card — if FP8 is native on your tier’s GPUs, start there.
- VRAM tight? AWQ or INT4 (W4A16) is the proven way to run frontier MoE on 2× 96 GB Ada cards.
- On Blackwell and want smallest weights? NVFP4 W4A4 for max throughput; W4A16_NVFP4 if W4A4 shows quality issues. MXFP4 is the open-standard alternative if you care about OCP portability or AMD CDNA4.
- Quality regression? AWQ → FP8 → BF16, in that order. Jumping straight to BF16 rarely pays for the VRAM cost.
- MoE + FP4? Prefer Blackwell datacenter or RTX PRO Blackwell; MoE FP4 kernels are the most mature there.
Context length: KV cache and hardware limits
Transformers keep a key/value cache for past positions in the thread. For a fixed model, KV memory grows with sequence length (layers, heads, precision, length). So very long single prompts or threads stress VRAM and attention work, even when weights already fit. That is a hardware and latency issue for your squad server, not a line item on a usage-priced API.
Why ~100k tokens (effective) is often enough
For everyday agent work, up to roughly 100k tokens of effective context is often enough and lighter on KV: you keep more headroom on the same GPU, snappier iterations, and fewer surprises when the session grows. Many workflows should not rely on one giant 200k-token blob anyway: scoped reads, retrieval, and rolling summaries reduce noise and often work better than stuffing everything into one window.
When >100k or >200k is worth it
Reach for very large windows when attention really must cover distant pieces at once, e.g. huge cross-package refactors or migrations where dropping a file from context is risky. Even then, planning and chunking sometimes beats raw length.
Summarization and tooling
Good agent setups compress history on purpose. With that discipline, sub-100k effective context is sufficient for many teams, and it keeps KV pressure down on shared GPU time.
Architectures that ease KV pressure
Some models cut KV and attention work on long sequences instead of only adding RAM. Example: MiMo-V2-Flash mixes sliding-window and global attention and reports roughly 6× less KV-cache storage and attention compute vs full attention on long contexts (technical report). If you compare models, look at total memory and speed including such tricks, not only the marketing context size number.
Practical checklist for squads
- Prefer FP8 or MXFP8 when native on your GPU — closest to BF16 in practice.
- Use AWQ / INT4 (W4A16) when you need the smallest weights on Ada/Hopper; accept occasional quantization artifacts.
- On Blackwell, consider NVFP4 or MXFP4 for the most aggressive compression; use W4A16_NVFP4 if W4A4 quality slips.
- Check the requirements card for native vs emulated architectures before committing to a quant.
- If quality slips (systematic mistakes, bad edits): switch to FP8 (or W4A16_NVFP4 from W4A4) before assuming you need BF16.
- Default to a moderate effective context (often ≤ ~100k tokens) unless you need a single ultra-long attention pass; use summaries and retrieval to avoid pointless KV bloat.
- Huge refactors: long context or better chunking and plans — sometimes the second is more reliable.
References
- Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, arXiv:2306.00978
- MiMo-V2-Flash Technical Report (hybrid attention, KV savings), arXiv:2601.02780
- vLLM quantization overview: Quantization
- vLLM FP8 W8A8: FP8 W8A8
- vLLM NVIDIA ModelOpt (FP8, NVFP4, MXFP8): Model Optimizer
- OCP Microscaling Formats (MXFP4, MXFP8): OCP MX spec
- “Give Me BF16 or Give Me Death”? Accuracy-Performance Trade-Offs in LLM Quantization (broader quant study), ACL 2025