🔀 Fallbacks

Automatic failover between AI providers and models. When your primary provider encounters an issue, the system seamlessly switches to a backup provider without interrupting your application.

---

✨ Core Features

Automatic Provider Failover

Fallbacks provide automatic failover capabilities when your primary AI provider encounters issues. Whether it's rate limiting, service outages, or model unavailability, the system will automatically try backup providers in the order you specify until one succeeds.

When a failover is triggered, the system treats it as a completely new request — all configured plugins (caching, governance, logging, etc.) are re-run for the backup provider, ensuring consistent behavior across all providers.

---

⚙️ How Fallbacks Work

When fallbacks are configured, the system follows this processing flow:

flowchart LR
    A[Receive user request] --> B[Try primary provider/model]
    B --> C{Request successful?}
    C -->|Yes| D[Return success response]
    C -->|No| E[Try next fallback provider in order]
    E --> F{All providers failed?}
    F -->|No| E
    F -->|Yes| G[Return original error from primary provider]

1. Primary Attempt: First try your primary provider/model
2. Auto-Detection: If the primary fails (network error, rate limit, model unavailable), the system detects the failure
3. Sequential Failover: Try each backup provider in order until one succeeds
4. Success Response: Return the response from the first successful provider
5. Complete Failure: If all providers fail, return the primary provider's original error

Each failover attempt is treated as a completely new request, so all your configured plugins (semantic cache, governance rules, monitoring) will apply to whichever provider ultimately processes the request.

---

🚀 Implementation Examples

Gateway Method

# Chat completion request with multiple fallback options
curl "https://ai-tokenhub.com/api/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $YOUR_API_KEY" \
  -d '{
    "model": "deepseek-v4-pro-ga-260813",
    "messages": [
      {"role": "user", "content": "Hello, how are you?"}
    ],
    "temperature": 0.7,
    "max_tokens": 100
  }'

Response (from any successful provider)

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "The capital of France is Paris."
      }
    }
  ],
  "created": 1677652288,
  "id": "chatcmpl-123",
  "model": "deepseek-v4-pro-ga-260813",
  "object": "chat.completion",
  "system_fingerprint": "fp_44709d6fcb",
  "usage": {
    "completion_tokens": 10,
    "prompt_tokens": 25,
    "total_tokens": 35
  }
}

---

🎯 Real-World Scenarios

Scenario	Description	Value
🚦 Rate Limiting	Primary provider OpenAI triggers rate limit → Backup provider Anthropic handles it successfully	Your application continues running without interruption
❌ Model Unavailable	Primary provider's specific model is unavailable → Switch to another provider's model with similar capabilities	Seamless transition to equivalent capability
🚫 Provider Outage	Primary provider experiences an outage → Backup provider takes over	Business continuity is maintained
💰 Cost Optimization	Primary provider uses high-quality paid model → Budget exceeded, switch to economical backup option	Governance rules can trigger failover based on usage

---

🔍 Failover Behavior Details

Situations That Trigger Failover

• 🔌 Network connection issues
• ⚠️ Provider API errors (500, 502, 503, 504)
• 🚦 Rate limit errors (429)
• ❌ Model unavailable
• ⏱️ Request timeout
• 🔑 Authentication failure

Situations That Preserve the Original Error

• ❌ Request validation errors (malformed requests)
• 🛡️ Plugin enforcement blocks (governance rule violations)
• ⚠️ Certain provider-specific errors marked as non-retryable

---

🔌 Plugin Integration Mechanism

When a failover is triggered, the failover request is treated as a completely independent new request:

• 🔍 Semantic cache checks are re-run (different providers may have different cached responses)
• 📜 Governance rules apply to the new provider
• 📊 Logging captures the failover attempt process
• 🔌 All configured plugins are re-executed for the backup provider

Plugin Control Over Failover

Plugins can control whether failover should be triggered based on their own logic. For example:

• Custom plugins may block failover for certain error types
• Security plugins may disable failover for compliance reasons

When a plugin decides that failover should not be attempted, it can completely block the failover mechanism, ensuring the original error is returned immediately.

This ensures consistent behavior regardless of which provider ultimately processes your request, while giving plugins full control over the failover decision process. And you can always see which provider actually handled the request through the extra_fields field.