Skip to main content
Inference is limited by your organization’s tier. Limits apply at three scopes at once: each API key, each project (1.5× the key limit), and the organization (2×). Tokens per day are limited at the organization only. Requests, tokens per minute, and requests per day use sliding windows. Your tier rises automatically as you pay; a plan can also grant a tier.

​
How tokens are counted

A request is admitted with an estimate: its input text (about 4 bytes per token) plus the output it may generate (max_tokens, or the model’s default). When it finishes, the counters are corrected to the actual usage. A request bigger than a whole window’s budget is still admitted into an empty window.

​
Headers

Every inference response carries OpenAI-style headers: and IETF headers, such as RateLimit-Policy: "rpm";q=60;w=60, "tpm";q=1000000;w=60 and RateLimit: "rpm";r=59;t=1. count_tokens sends only the rpm entries. When several scopes apply, the headers report the one closest to its limit.

​
When you hit a limit

You get 429 rate_limited with a Retry-After header in seconds, and a message naming the scope and metric. The SDKs wait and retry for you. Spread bursts out, or move traffic to another key or project if one scope is the bottleneck.

​
Other endpoints