The AI Gateway is usually adopted for one reason — a single API key for hundreds of models — and then configured with a single model and nothing else. The fields that matter most are the ones nobody fills in: a models array for cross-model failover, per-provider timeouts for fast failover, and budgets that reject requests before a retry storm becomes an invoice. Here are the exact config blocks, the metadata that proves they fired, and the caveats the docs bury.
There is a particular shape of incident that shows up in every team that has put an LLM on a request path. A provider degrades — not fails, degrades. Latency climbs from 800ms to 40 seconds. The application's own retry logic, written months ago and never revisited, sees a timeout and retries. The retry also hangs. The queue behind it fills. Somewhere around minute four, the provider starts returning 529s, and now every request in flight is retrying against a wall.
Nobody gets paged for the model being slow. They get paged for the queue depth. And afterwards, in the retro, someone says the thing that always gets said: we should route to a backup provider.
That work usually gets scheduled and never done, because doing it properly means writing a client abstraction over two SDKs, normalizing their error taxonomies, deciding what counts as a retryable failure, and then maintaining that forever. It is a two-week project that competes with shipping features, and it loses.
Vercel's AI Gateway makes most of that two-week project a four-line config block. The interesting part is that almost nobody uses it. The gateway gets adopted for the headline feature — one API key, hundreds of models, no markup on tokens — and then wired into an app with a single model string and no provider options at all. The reliability machinery ships in the box, unopened.
!The unopened half of the gateway config
What you already get without configuring anything
Worth establishing the baseline first, because part of why teams skip the config is a reasonable assumption that the defaults are doing something.
They are. By default the gateway picks providers dynamically based on recent uptime and latency, and it automatically retries against a different provider when one fails. If you call anthropic/claude-sonnet-5 and Bedrock is having a bad afternoon, the gateway can serve the same model through Anthropic directly or through Vertex without your code knowing.
That is real failover, and it covers the most common outage shape: one provider's infrastructure is unhealthy while the model itself is fine everywhere else.
What it does not cover is the model being the problem. Rate limits scoped to a model. A capacity crunch on a newly-released model that every provider hosts and none can serve. A model deprecation you did not track. In all of those cases, provider-level failover exhausts every route to the same model and then fails.
That is the gap the models array closes, and it is the field most config blocks are missing.
The fallback chain
The syntax is a single array under providerOptions.gateway:
``typescript
// app/api/chat/route.ts
import { streamText } from 'ai';
export async function POST(request: Request) {
const { prompt } = await request.json();
const result = streamText({
model: 'anthropic/claude-fable-5', // Primary
prompt,
providerOptions: {
gateway: {
models: ['anthropic/claude-opus-5', 'google/gemini-3.1-pro-preview'],
},
},
});
return result.toUIMessageStreamResponse();
}
`
The evaluation order is worth stating precisely, because it is two nested loops and the nesting direction determines your latency budget:
The gateway routes to the primary model — the one in the model parameter.
For that model, it tries every eligible provider, applying order or only if you set them.
If all providers for that model fail, it moves to the next entry in models.
The response comes from the first model/provider combination that succeeds.
Providers are the inner loop, models are the outer loop. A three-model chain where each model has three providers is up to nine attempts before you get an error back. That is a feature when the failures are fast and a problem when they are slow, which is the reason provider timeouts exist further down this post.
The same option works across every API surface the gateway speaks — AI SDK, OpenAI Chat Completions, Anthropic Messages, OpenAI Responses, OpenResponses. On the OpenAI and Anthropic SDKs, providerOptions is not in the SDK's TypeScript types, so it needs a // @ts-expect-error above it. In Python, the same object goes through the SDK's extra_body parameter.
One design note that is easy to miss: the fallback list is a list of model strings, not a list of clients. Nothing in your application code changes when you extend the chain. There is no second SDK, no second key, no error-mapping layer. Adding a cross-vendor backup is one string.
Provider preference, and the three fields that control it
models handles which model. Three sibling fields handle which provider:
`typescript
providerOptions: {
gateway: {
order: ['bedrock', 'anthropic'], // Preferred sequence
only: ['bedrock', 'anthropic'], // Hard allowlist
sort: 'cost', // Rank by 'cost' 'tps'
},
},
`
order is a preference — providers not listed still get used, just after the ones that are. only is a restriction; if none of the listed providers serve the model, the request fails with an error naming what was allowed. When both are set, only defines the allowed set and order sets priority inside it, so only: ['anthropic', 'vertex'] with order: ['vertex', 'bedrock', 'anthropic'] resolves to vertex → anthropic and drops Bedrock silently.
sort is the one that is genuinely underused. It takes 'cost' (lowest first), 'ttft' (lowest median time to first token first), or 'tps' (highest median throughput first), and it ranks providers by live metrics rather than a list you hardcoded in March.
It also has a guardrail that makes it safe to leave on: the gateway penalizes degraded and recovering providers in the ranking, and always sorts down providers last, regardless of how cheap or fast they look on paper. You get cost optimization that will not happily route all your traffic into a provider that is currently on fire.
When sort is active, the response metadata tells you exactly what it decided:
`json
{
"gateway": {
"routing": {
"sort": {
"option": "cost",
"executionOrder": ["anthropic", "bedrock", "vertex"],
"metrics": { "anthropic": 0.003, "bedrock": 0.003, "vertex": 0.005 },
"deprioritizedProviders": []
}
}
}
}
`
Combining them is the common production shape: only to satisfy a compliance or data-residency requirement, sort to optimize inside the allowed set, models to survive the model itself going away.
`typescript
providerOptions: {
gateway: {
models: ['openai/gpt-5.4-nano', 'anthropic/claude-opus-5'],
order: ['azure', 'openai'],
},
},
`
That block tries the primary model via Azure then OpenAI, then gpt-5.4-nano via Azure then OpenAI, then Claude Opus via whatever serves it.
!Two nested loops: models on the outside, providers on the inside
Fast failover, and the caveat the docs put in a footnote
A fallback chain only helps if failures are fast. A provider that returns a 500 in 280ms is cheap to fall away from. A provider that accepts the connection and then sits there for 90 seconds before timing out will burn your entire request budget on attempt one.
providerTimeouts sets a per-provider deadline in milliseconds:
`typescript
providerOptions: {
gateway: {
order: ['anthropic', 'bedrock', 'vertex'],
providerTimeouts: {
byok: {
anthropic: 10000,
bedrock: 15000,
// vertex omitted — uses the default gateway timeout
},
},
},
},
`
Two details matter more than the syntax.
The timeout measures time-to-first-token, not total duration. Once the first token arrives — including thinking tokens from a reasoning model — the timer is cleared and will not fire mid-stream. So a 10-second timeout on a model that reasons for two minutes is not a truncation risk; it is a "did this provider acknowledge us" check.
Provider timeouts apply to BYOK credentials only. This is the caveat worth reading twice. If you are running on gateway-managed credentials, setting providerTimeouts does nothing. The fast-failover story is only available to teams that have brought their own provider keys. The docs also note that some providers do not support stream cancellation, so a timed-out request may still be billed by that provider.
Allowed range is 1,000ms to 789,000ms. When a timeout fires, it is visible in the metadata rather than being silently folded into a generic error:
`json
{
"provider": "anthropic",
"credentialType": "byok",
"success": false,
"error": "PROVIDER_TIMEOUT",
"providerTimeout": true,
"configuredTimeoutMs": 10000
}
`
Reading what actually happened
The reason to care about that metadata block is that a fallback chain is a piece of infrastructure that, when working correctly, is invisible. Your dashboards look identical whether the primary model served every request or failed every request and the third fallback carried your entire Tuesday.
Every response carries a modelAttempts array with the full trace. Each entry has two identifiers that look interchangeable and are not: canonicalSlug is the gateway's normalized name (always creator/model-name), and modelId is the provider's own internal ID (provider:model). The same canonicalSlug can appear against several providers, each reporting a different modelId. If you are aggregating on the wrong one, your "which model served this" chart is wrong in a way that looks plausible.
The minimum useful instrumentation is one line:
`typescript
const meta = await result.providerMetadata;
console.log(JSON.stringify(meta?.gateway?.routing, null, 2));
`
What you want out of it, structured and shipped to whatever you already use:
• finalProvider and canonicalSlug — what actually served the request
• modelAttemptCount — anything above 1 means the primary failed
• totalProviderAttemptCount — your retry amplification factor
• gateway.cost — the inference cost for that request, as a decimal string
That last one deserves a note: cost covers inference only. It does not include Custom Reporting writes or Zero Data Retention surcharges, so it is the right number for per-request unit economics and the wrong number for reconciling an invoice.
For anything beyond a console log, Trace Drains forward an OpenTelemetry trace of every request — provider-attempt spans included — to your own collector. That is the version you want if fallback behavior is going to be part of an SLO conversation.
Budgets: the thing that stops the retry storm from becoming an invoice
Fallback chains and cost control pull in opposite directions. A chain that tries three models across nine providers is, on a bad day, nine times the request volume. Every one of those attempts that reaches a provider and fails partway through can still be billable.
Budgets are the backstop. They exist at three scopes — team, project, API key — and the interaction is not obvious:
• Every request counts toward the team budget.
• An OIDC token from a deployment counts toward that project's budget.
• An API key counts toward that key's budget.
• A request must pass every budget in scope. One exhausted budget rejects it even if the others have room.
The consequence that surprises people: an API key's spend is never attributed to a project. A key used from inside storefront does not draw down storefront's budget. Only OIDC tokens from that project's deployments do. If your budget dashboard shows a project sitting at zero while your bill climbs, that is usually why.
Setting one is a single command:
`bash
vercel ai-gateway budgets set team --limit 500 --refresh-period monthly
vercel ai-gateway budgets set project my-project --limit 200 --refresh-period monthly
vercel ai-gateway budgets defaults set project --limit 50 --refresh-period monthly
`
Refresh periods are daily, weekly, monthly (the default for team/project), or none for a cumulative cap that never resets. All reset at midnight UTC — weekly on Monday, monthly on the first.
When a budget is exhausted, the gateway returns HTTP 402 with type: "quota_for_entity_exceeded" and a message naming the scope, current spend, and limit. Handle it explicitly; a 402 is not a transient error and retrying it is pure waste:
`typescript
if (res.status === 402) {
// Budget exhausted for some scope. Do not retry.
// Serve cached/degraded output and alert — the reset is on a UTC boundary.
}
`
Three caveats the docs are honest about and that are easy to skim past:
• A budget is a soft cap. The check runs at the start of each request, so the request that crosses the line still completes. Final spend lands slightly over the limit.
• BYOK spend is not counted. If you brought your own provider keys, that spend is metered separately and does not count toward any budget. The teams most likely to have configured provider timeouts are exactly the teams whose budgets do not protect them.
• Enforcement lags. A new budget can take up to a minute or two to start counting, and edits take tens of seconds to a few minutes to propagate.
Spend alerts are off by default. Turning on 50/75/100% thresholds when you create the budget costs nothing and is the difference between finding out on Tuesday and finding out on the invoice.
!Budget scopes stack: team, project, and key each get a veto
Routing rules: changing behavior without a deploy
The config discussed so far lives in your application code, which means changing it means a deploy. Routing rules — currently in beta, CLI-only — move that decision to the gateway.
They are firewall-style rules applied to every request made with your team's gateway credentials:
`bash
Transparently substitute one model for another
vercel ai-gateway rules add --type rewrite \
--source anthropic/claude-opus-4.8 \
--destination anthropic/claude-haiku-4.5
Block a model outright — requests get a 403
vercel ai-gateway rules add --type deny --source openai/gpt-5.5
`
rewrite is the incident tool: a model is unavailable, or retired, or suddenly too expensive for the traffic hitting it, and you reroute every service at once without touching a repository. deny is the governance tool — stopping a team from quietly putting an expensive model on a high-volume path.
The honest framing is that this is a blunt instrument with a wide blast radius. It applies to every request from every service using those credentials. It is very good for the thirty minutes of an incident and a poor place for permanent architecture.
Try It Yourself
Fifteen minutes, and you end with a chain you can prove fired.
Set up a project and key.
`bash
mkdir gateway-fallback-demo && cd gateway-fallback-demo
pnpm init
pnpm add ai dotenv @types/node tsx typescript
`
Create a key from the AI Gateway API Keys page in the Vercel dashboard, then:
`bash
.env.local
AI_GATEWAY_API_KEY=your_ai_gateway_api_key
`
Write a chain that reports on itself. Save as index.ts:
`typescript
import { streamText } from 'ai';
import 'dotenv/config';
async function main() {
const result = streamText({
model: 'anthropic/claude-fable-5',
prompt: 'Explain a fallback chain in two sentences.',
providerOptions: {
gateway: {
models: ['anthropic/claude-opus-5', 'google/gemini-3.1-pro-preview'],
sort: 'ttft',
},
},
});
for await (const part of result.textStream) process.stdout.write(part);
const routing = (await result.providerMetadata)?.gateway?.routing as any;
console.log('\n---');
console.log('served by :', routing?.finalProvider, routing?.canonicalSlug);
console.log('model attempts :', routing?.modelAttemptCount);
console.log('provider tries :', routing?.totalProviderAttemptCount);
}
main().catch(console.error);
`
`bash
pnpm tsx index.ts
`
Force the fallback. Change the primary model to a deliberately invalid slug — anthropic/claude-does-not-exist — and run it again. The output still arrives, modelAttemptCount is greater than 1, and finalProvider names the model that caught it. That output is the proof the chain works, and it is the thing to assert on in a test rather than trusting the config by inspection.
Cap the blast radius.
`bash
vercel ai-gateway budgets set project my-project --limit 25 --refresh-period monthly
vercel ai-gateway budgets list
`
Confirm the spend is visible.
`bash
curl "https://ai-gateway.vercel.sh/v1/report?start_date=2026-08-01&end_date=2026-08-31&group_by=model" \
-H "Authorization: Bearer $AI_GATEWAY_API_KEY"
`
Group by model, user, tag, provider, or credential_type. If you attach user and tags to requests, per-customer cost attribution comes out of the same endpoint.
The part worth keeping
The specific syntax here will drift; gateway products iterate quickly and half these field names may be different in a year. The pattern underneath is what transfers.
Reliability across model providers used to be application code — an abstraction layer, an error taxonomy, a retry policy, a circuit breaker, all of it yours to write and yours to maintain. It is now configuration, and configuration is cheap enough that the reason to skip it stops being effort and starts being that nobody read past the quickstart.
The quickstart gets you a working request. The four fields after it — models, order/sort, providerTimeouts`, and a budget — are what makes that request survive a bad Tuesday. They take about ten minutes, and the metadata tells you honestly whether they are doing anything.
Most gateway configs in production are one line long. That is a choice, even when it does not feel like one.