Inference Sizing and Capacity
What you tell us
You give the instance three things when you create it:
- Peak requests per minute. The most requests you expect in one minute, at your busiest. Size for the peak, not the average.
- A typical request. The number of tokens going in and coming out. The wizard offers presets for each model kind and lets you edit the numbers under Advanced.
- A context limit per request. A hard limit on one request. Requests above it are rejected with an error that names the limit. A higher limit reserves more GPU memory.
| Preset | Kind | Tokens in | Tokens out |
|---|---|---|---|
| Short chat | Chat | 600 | 300 |
| Long document | Chat | 6,000 | 800 |
| Receipt or form | OCR | 900 | 250 |
| OCR page | OCR | 2,500 | 1,000 |
| Dense page or table | OCR | 4,000 | 2,500 |
| Search query | Embeddings | 64 | 0 |
| Embedding batch | Embeddings | 512 | 0 |
| Long passage | Embeddings | 4,000 | 0 |
For OCR, the tokens in include the page image. An image costs a fixed number of tokens, not one per pixel.
What we calculate
From your load and the model, syndicAI estimates four things for every GPU on offer.
1. GPU memory
A GPU must hold three things:
- The model weights, the size of the checkpoint.
- A runtime reserve for the engine, its buffers and, with image input on, the vision encoder.
- The cache for requests in flight, the key-value cache. It grows with the number of requests running at once and with the context they use.
Only part of each GPU's memory is usable by the engine, and the rest is left free. The wizard shows the total as "GPU memory needed" with its three parts. For a large model the weights dominate. That is why you can be told you need a bigger GPU, or several, even though the parallel capacity looks high: the model has to fit before anything else matters.
2. Time to first token
The time before the first token comes back on an idle instance. It is set by reading your prompt (prefill), which is limited by the GPU's compute for the precision the model runs in. A long prompt takes longer. For OCR the figure is the time per page.
3. Decode speed
Once the first token is out, each new token needs the model to read its active weights from memory, so decoding is limited by memory bandwidth, not compute. Many requests can share one pass over the weights, so total throughput rises with the number of requests in flight until compute becomes the limit. Mixture-of-experts models read only the active experts, so they decode much faster than their size suggests.
4. Parallel requests
How many requests must run at the same time to serve your peak:
requests in flight = requests per second x time one request takes
time one request takes = time to first token + tokens out / decode speed
The decode speed falls as more requests share the GPU, so the estimate is solved iteratively. We then compare it with the parallel capacity: the number of requests the GPU can hold at once, limited by memory (the cache) or by compute (keeping each stream at 15 tokens per second or faster). The wizard shows which one limits you.
The verdict
| Verdict | Meaning |
|---|---|
| Fits | The capacity is at least 1.5 times the parallel requests you need, and compute is busy at most two thirds of the time. There is room for bursts. |
| Tight | The capacity covers the parallel requests you need, but with less than 1.5 times headroom or with busy compute. It works at your peak and has little room above it. |
| Does not fit | The GPU cannot carry the load: not enough memory, or compute saturated. |
The tier cards use the same estimate. Budget is the cheapest GPU that can carry your load, even a tight one. Performance is the cheapest GPU with clearly more headroom, and Balanced sits between. When GPUs are close in headroom, fewer cards are shown.
How far to trust it
The estimate is built from each GPU's published memory bandwidth and tensor throughput, and its constants are fitted to measured runs. We measured on an RTX 5090 and an H100 SXM; for GPUs we have not measured, the datasheet figures are used with a conservative derating, and the card says so. It is an estimate, not a guarantee. It does not know your exact prompts, your client's behaviour or how bursty your traffic is.
Before you rely on an instance, test it with the load that matters. The usage tab charts requests per minute against your peak, so you can see how close you run. If you run close to it, choose the Performance card, or create a new instance sized for the higher peak.