Reliability and fallbacks

How the gateway handles provider failures: opt-in retries and same-family fallbacks, provider timeouts, and how your client should retry using Retry-After.

By default the gateway calls the requested model's provider once for each request and returns what happens: it does not retry a failed call or try another model, and the provider SDK's own retries are turned off. The one exception is a parameter the provider rejects, which the gateway adjusts and sends again to the same provider (see Chat completions). You can opt in to one retry and to fallbacks within the same model family. Your client should still retry the errors that are worth retrying.

Retries and fallbacks are opt-in

Retries and fallbacks are off by default. You can turn them on in two places:

For the workspace. A tenant admin or super admin turns on Allow retries and fallbacks under Usage, Credits and limits, Gateway settings.
For one request. Send provider.allow_fallbacks in the request body. It overrides the workspace setting in either direction, so a request can also turn them off.

curl https://api.safeguard.sh/gpt/v1/chat/completions \
  -H "Authorization: Bearer $SAFEGUARD_INFERENCE_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MODEL_ID",
    "messages": [{"role": "user", "content": "Hello"}],
    "provider": {"allow_fallbacks": true}
  }'

When they are off, a provider failure comes straight back to you.

What happens when they are on

If the provider returns a server error, does not answer in time, cannot be reached or rate limits the call, the gateway tries the same model once more.
If that fails too, it moves on to the model's fallbacks, in order. Each fallback gets the same single retry.
If the provider refuses Safeguard's credentials, the gateway skips the retry and moves straight to the next fallback.
If the provider rejects the request as invalid (upstream_rejected), the gateway stops. That request is never retried and never sent to a fallback.

Fallbacks follow strict rules:

Safeguard lists the fallbacks for each model in its model catalog. You cannot choose them.
A fallback is only ever a model from the same family, meaning the same model maker, or the same model served by another host.
A fallback has to be available through the API in its own right. If your workspace or the request requires zero data retention, the fallback has to offer it too.
A fallback is billed at its own price. The credit hold before the call covers the most expensive model the request might use.
The response tells you what answered: model is the model that served the request, and provider names the provider the gateway called. The request log lists the request under that model.

For a streamed request, retries and fallbacks happen only before the first chunk reaches you. After that, see Streams that fail partway.

No fallback to Safeguard's platform providers

Your requests go only to the provider of the model you asked for, or of one of its listed fallbacks. They never fall back to the providers behind Safeguard's own AI features. The gateway calls providers with its own credentials, kept apart from the ones those features use, and automated tests check that the two paths stay separate.

Timeouts

Safeguard sets how long the gateway waits for each provider. The default is 120 seconds, and no provider gets more than 600 seconds. A provider that does not answer in time gives you a 504 upstream_timeout.
A streamed response is cut off after 15 minutes in total. The stream then ends with an upstream_timeout error event.
With retries and fallbacks on, one request can make several attempts, each with its own timeout. Set your client's timeout with that in mind, or stream the response.

Retrying in your client

Retry only the errors that can succeed on a second try, and wait between attempts.

Response	Retry?	How long to wait
429 `rate_limited`	Yes	The `Retry-After` header: the seconds until your key's one-minute window has room again. A refused request does not count toward the limit.
429 `upstream_rate_limited`	Yes	`Retry-After` is 5 seconds. Back off further if it repeats.
502 `upstream_error`, 504 `upstream_timeout`	Yes, a few times	Exponential backoff with jitter.
503 `auth_unavailable`, 503 `billing_unavailable`	Yes	Exponential backoff with jitter.
503 `model_unavailable`	Only when the message says the catalog is temporarily unavailable	Exponential backoff. For any other message, pick another model.
500 `internal_error`	Once or twice	Exponential backoff.
502 `upstream_auth`	No	The provider refused Safeguard's credentials. Try another model.
400, 401, 402, 403, 404, 413	No	Fix the request, the key, the credit or the limit first.

When Retry-After is present, wait at least that many seconds. It is a whole number.
Otherwise, use exponential backoff with jitter, for example about 1, 2 and then 4 seconds plus a random fraction, and cap the number of attempts.
Each retry is a new request, with its own credit hold and its own charge.
OpenAI's SDKs can retry some of these responses on their own (max_retries in Python, maxRetries in Node). Check that setting, so you do not run two retry loops at once.

A small retry loop in Python:

import os
import random
import time

import httpx

URL = "https://api.safeguard.sh/gpt/v1/chat/completions"
RETRY = {
    "rate_limited", "upstream_rate_limited", "upstream_error", "upstream_timeout",
    "auth_unavailable", "billing_unavailable", "internal_error",
}


def create(body, attempts=4):
    headers = {"Authorization": f"Bearer {os.environ['SAFEGUARD_INFERENCE_KEY']}"}
    for attempt in range(attempts):
        r = httpx.post(URL, json=body, headers=headers, timeout=300)
        if r.status_code < 400:
            return r.json()
        try:
            code = r.json()["error"]["code"]
        except (ValueError, KeyError, TypeError):
            code = None
        if code not in RETRY or attempt == attempts - 1:
            r.raise_for_status()
        wait = r.headers.get("Retry-After")
        time.sleep(int(wait) if wait else 2 ** attempt + random.random())

Streams that fail partway

Once the first chunk is on its way, the HTTP status is already 200. If the provider fails after that, the stream ends with an error event, followed by [DONE]:

data: {"error": {"message": "The provider stream ended with an error.", "type": "api_error", "code": "upstream_error", "param": null}}

data: [DONE]

Treat the answer as incomplete. The tokens produced before the failure are charged, from the provider's usage figures or, when it sent none, the gateway's estimate. If you asked for usage with stream_options.include_usage, the final usage chunk still arrives before [DONE]. See Streaming.

Closing the connection does not stop the charge. If your client disconnects mid-stream, the gateway keeps reading the provider's answer to the end, up to the 15-minute cap, and charges for all of it. The request log shows client_disconnected. To keep answers short, set max_tokens.