Inference Models
Categories
Every model has one or more categories. The model picker filters by them.
- Chat: text in, text out. Assistants, summaries, extraction, classification.
- Vision: chat models that also read images. Describe a screenshot, read a chart, answer a question about a photo.
- Embeddings: text in, a vector out. Search, retrieval, deduplication, clustering.
- OCR: page images in, text out. Invoices, forms, scans, dense tables.
A model can sit in several categories. A coding model that also chats appears in both pickers it belongs to; coding servers show coding models only.
The catalogue
| Model | Category | Active parameters | Weights | Context | Image input |
|---|---|---|---|---|---|
| Qwen3.8 27B (FP8, NVFP4) | Chat, Vision | 27.8B | 29 GB (FP8), 22 GB (NVFP4) | 262K | Yes |
| Qwen3.6 35B-A3B (FP8) | Chat, Vision | 3B | 35 GB | 262K | Yes |
| Gemma 4 31B-it (INT4) | Chat, Vision | 31.3B | 22 GB | 262K | Yes |
| gpt-oss 120B (MXFP4) | Chat | 5.1B | 61 GB | 131K | No |
| gpt-oss 20B (MXFP4) | Chat | 3.6B | 13 GB | 131K | No |
| Qwen3 Embedding 8B | Embeddings | 8B | 15 GB | 32K | No |
| Qwen3 Embedding 0.6B | Embeddings | 0.6B | 2 GB | 32K | No |
| Harrier OSS v1 0.6B | Embeddings | 0.6B | 2 GB | 32K | No |
| GLM-OCR | OCR | 1.3B | 3 GB | 131K | Yes (always on) |
| PaddleOCR-VL 1.6 | OCR | 1B | 2 GB | 131K | Yes (always on) |
| DeepSeek-OCR 2 | OCR | 0.6B | 7 GB | 8K | Yes (always on) |
"Active parameters" is what a token passes through. Mixture-of-experts models such as gpt-oss 120B, gpt-oss 20B and Qwen3.6 35B-A3B have many weights but use few per token, so they decode fast for their size and need memory for the full weights.
Choosing a model
- A chatbot or assistant: start with a mid-size chat model. gpt-oss 120B is a strong all-rounder that fits on a single 96 GB card. Qwen3.8 27B and Gemma 4 31B are smaller and also read images. For a simple support or FAQ bot on a modest GPU, gpt-oss 20B fits a 24 GB card; it knows less and handles fewer languages than the larger models, so test it on your own questions first.
- Questions about images: pick a vision model and keep image input on. See Inference API for the request format and limits.
- Search and retrieval: Qwen3 Embedding 0.6B is small and fast; the 8B variant gives better vectors when quality matters more than cost.
- Documents and forms: an OCR model reads a page image and returns text. They are small, so one modest GPU can read many pages a minute. DeepSeek-OCR 2 has an 8K context and suits single pages.
The very large coding models (hundreds of gigabytes of weights) are for coding instances only: they are no sensible backend for a chatbot or a document pipeline. The catalogue grows, and the picker is the source of truth for what you can create today.
Image input
Vision models take image input when you turn it on (it is on by default for inference instances) and accept up to 4 images per request. OCR models always run in vision mode and have no switch. Turning image input on reserves some GPU memory for the vision encoder, so the memory estimate in the wizard rises a little. Changing the setting on an existing instance re-provisions it.