Inference Limits and Rate Control

Peak requests per minute

The peak you enter when you create an instance is a sizing input. It decides which GPU can carry your load and what the instance costs. It is not an enforced limit on the instance as a whole. If more requests arrive than the GPU can serve at once, they typically wait their turn and responses get slower. The usage tab charts your requests per minute against your peak so you can see when you run close to it.

Per-key request caps

To protect the instance from one misbehaving client, give each API key its own cap.

• What it is: the most requests per minute one key may send, counted over a sliding 60 second window.
• Default: a new key takes the instance's peak requests per minute. You can set any whole number from 1 to 100,000.
• Per key: each key has its own counter. One key hitting its cap does not affect another.
• Change it anytime: edit a key's cap from the keys dialog. It applies at once, without restarting the instance.
• Inference instances only: keys of coding instances are not capped.

A key over its cap gets HTTP 429 and a Retry-After header with the seconds to wait (at least 1):

{
  "error": {
    "message": "Rate limit exceeded",
    "type": "rate_limit_error",
    "code": "rate_limit_exceeded"
  }
}

On the Anthropic-style /v1/messages path the body is {"type": "error", "error": {"type": "rate_limit_error", "message": "Rate limit exceeded"}}.

The usage tab lists, per key, how many requests were rejected, so you can tell whether a cap is too tight or a client is misbehaving.

Handling 429 in your client

Wait for the time in Retry-After, then retry. Add a little random jitter so many clients do not retry at the same moment.

import random, time
import requests

def call(payload, headers, url, attempts=6):
    for attempt in range(attempts):
        r = requests.post(url, json=payload, headers=headers, timeout=120)
        if r.status_code not in (429, 503):
            return r
        wait = int(r.headers.get("Retry-After", 2 ** attempt))
        time.sleep(wait + random.random())
    return r

Request size limits

Limit	Value
Request body, text-only instance	20 MiB
Request body, image input on	64 MiB
Images per request	4
Context per request	The limit you chose when you created the instance

A body over the limit is rejected with HTTP 413 and "code": "payload_too_large". A request over the context limit is rejected with an error that names the limit. See Inference API for the image limits.

When the instance is paused or starting

An instance that pauses when idle, or restarts after a host interruption, answers while it is not ready with HTTP 503 and an error code:

code	Meaning	Retry-After
instance_paused	The instance is paused. An authenticated chat, completions or embeddings request starts it.	none
server_activating	The instance is starting. It is usually ready in about 2 minutes.	120 or 60 seconds
server_reprovisioning	The host was lost. A new one is being set up.	1200 seconds

Only an authenticated POST to an inference endpoint can wake a paused instance. Listing models or an unauthenticated request does not. While it starts, further requests do not trigger a second start. Build clients that retry on 503 and honour Retry-After.