Production reliability for LLM APIs: rate limits, retries and failover

Calling a model once in a notebook is trivial. Running real traffic is where reliability matters: upstreams have transient errors, rate limits, and the occasional slow response. These are the patterns that keep your app stable, and they're the same whether you call OpenAI or a Chinese model through an OpenAI-compatible gateway.

1. Retry with exponential backoff

Transient 429 (rate limit) and 5xx errors should be retried, with increasing delays so you don't hammer a struggling upstream.

import time, os
from openai import OpenAI, APIError, RateLimitError

client = OpenAI(
    api_key=os.environ["TURILOOP_API_KEY"],
    base_url="https://api.turiloop.com/v1",
)

def call_with_retry(model, messages, retries=4):
    for attempt in range(retries):
        try:
            return client.chat.completions.create(
                model=model, messages=messages, timeout=60,
            )
        except (RateLimitError, APIError) as e:
            if attempt == retries - 1:
                raise
            time.sleep(2 ** attempt)   # 1s, 2s, 4s, 8s

2. Always set a timeout

A request with no timeout can hang forever and block a worker. Set one (say 60s for chat, longer for image generation) and treat a timeout as a retryable error.

3. Fall back to a second model

If your primary model is failing or saturated, degrade gracefully to another. Every model is one model string away, so failover is trivial:

def robust_call(messages):
    for model in ["deepseek-v4-pro", "glm-5.1", "deepseek-v4-flash"]:
        try:
            return call_with_retry(model, messages)
        except Exception:
            continue   # try the next model
    raise RuntimeError("all models failed")

A single OpenAI-compatible key reaching DeepSeek, GLM, Kimi and Claude makes this a real, cheap safety net rather than a second integration.

4. Idempotency and partial-failure handling

• For agent loops, cap iterations so a stuck conversation can't run up an unbounded bill.
• Log the model and token usage per call so you can reconcile spend and spot anomalies.
• For non-streaming batch jobs, make retries idempotent (same input, same dedup key) so a retry doesn't double-process.

5. Let the gateway help

A good relay already does some of this for you: multi-channel routing and failover at the upstream level, so a single provider wobble doesn't surface as an error in your app. Turiloop runs redundant upstreams with automatic failover behind one key. You still add app-level retries and a fallback model, but the floor is higher.

Build these four patterns once, wrap them in a helper, and your LLM calls stay up under real load. International card, pay-as-you-go.