Request Routing

Tianshu AI-TokenHub provides a multi-level, highly elastic intelligent request routing mechanism, supporting distribution of requests to the most suitable LLM providers based on different strategies, achieving a balance of cost optimization, high availability, and optimal performance.

Core Routing Types

Tianshu AI-TokenHub supports a total of 5 routing strategies to meet the needs of different scenarios:

---

1. Exact Routing - Based on Model Name Prefix

This is the most direct and deterministic routing method, allowing precise specification of which provider a request should be sent to.

Rule

Use the provider/model format in the model field of the API request. The system will directly parse the prefix and forward the request to the corresponding provider.

Example

{
  "model": "openai/gpt-4o",
  "messages": [
    {"role": "user", "content": "Hello!"}
  ]
}

Other examples:

• anthropic/claude-3-opus
• deepseek/deepseek-chat
• zhipu/glm-4

Workflow

1. The system parses the model prefix openai/
2. Directly forwards the request to the OpenAI provider
3. Uses the suffix gpt-4o as the actual model name in the request

Applicable Scenarios

• Debugging functionality of a specific provider
• When you need to ensure requests are sent to a specific backend (e.g., that provider has an exclusive model or better pricing)
• Forcing bypass of load balancing rules to prevent requests from being distributed to other providers

---

2. Load-Balanced Routing - Based on Virtual Key Weights

This is Tianshu AI-TokenHub's most powerful routing feature, allowing requests to be distributed across multiple providers by ratio, achieving the optimal balance of cost, performance, and availability.

Rule

Configure weight strategies through Virtual Keys. When creating a virtual key, you can add multiple providers and set a weight value for each. The model field in the request does not need to include a provider prefix (e.g., just write "gpt-4o").

Workflow

1. The system looks up the virtual key and model name from the request
2. Searches the Model Catalog for all healthy providers that support that model
3. Randomly selects one provider based on the weight ratios configured in the virtual key and forwards the request

Weight Calculation Example

Virtual key configuration:

Provider	Weight	Traffic Share
OpenAI	70	~70%
Azure	30	~30%

Result: Approximately 70% of gpt-4o requests are sent to OpenAI, and 30% to Azure OpenAI.

Applicable Scenarios

• Cost Optimization: Direct the majority of traffic to cheaper providers (e.g., DeepSeek, Moonshot)
• A/B Testing: Compare response quality or latency of the same model across different providers
• Failover: Combined with health checks, automatically switch to backup providers when the primary is unavailable

---

3. Fallback Routing

Tianshu AI-TokenHub has built-in resilience mechanisms to handle provider failures or model unavailability, improving overall service availability.

Rule Types

1. Exact Routing Fallback: When the provider specified by exact routing returns an error (e.g., 5xx server error, 429 rate limit) or times out, the system can be configured to automatically try other providers in the Model Catalog that support the same model.
2. Load-Balanced Fallback: When the provider selected by load balancing fails, the system re-selects from the remaining healthy providers by weight and retries automatically.

Configuration

Enable the Fallback Enabled option in virtual key settings, and optionally set the maximum number of retries (default: 2).

Applicable Scenarios

• Improve high availability (HA) in production environments, avoiding single points of failure
• Handle sudden LLM provider outages or rate limiting events
• Reliability assurance for critical business paths

---

4. Conditional Routing - Based on Request Headers or Parameters

Tianshu AI-TokenHub supports dynamically modifying routing behavior based on specific request metadata, enabling highly customizable routing strategies.

Common Conditional Rule Examples

• User/Key-Based Routing Override: Even if the request specifies openai/gpt-4, if the virtual key configuration enforces a specific routing policy, the latter takes precedence
• Stream Mode Based: Configure non-streaming requests to go to Provider A and streaming requests to go to Provider B (since some providers have better streaming support)
• Content Length Based: Automatically route long texts to providers that support larger context windows at lower prices
• Priority Tag Based: High-priority business requests are preferentially routed to more stable, lower-latency providers

Implementation

Implemented through Tianshu AI-TokenHub's Plugin System or Middleware with custom logic, supporting fully programmable routing rules.

---

5. Cache Routing

While not strictly a provider routing mechanism, it determines whether a request is responded to directly by Tianshu AI-TokenHub without going through a backend LLM, significantly reducing costs and latency.

Rule

With Semantic Cache or Exact Match Cache enabled, qualifying requests are returned directly from cache without calling any backend LLM.

Routing Priority

Cache hit > Exact Routing / Load-Balanced Routing. If the cache is hit, the request is never sent to any LLM provider.

---

Routing Decision Flow

When you initiate a request on Tianshu AI-TokenHub, the routing decision follows this logic:

Request enters
    │
    ├─ 1. Check cache -> Hit? -> Return cached result (end)
    │               No ↓
    ├─ 2. Parse virtual key / authentication info
    │
    ├─ 3. Check if model field has a provider prefix?
    │      ├─ Has prefix -> **Exact Routing**: Send directly to the specified provider
    │      │              │
    │      │              ├─ Success? -> Return result
    │      │              └─ Failure? -> Decide whether to try other providers based on fallback config
    │      │
    │      └─ No prefix -> **Load-Balanced Routing**:
    │                     ├─ Query the Model Catalog for healthy providers supporting this model
    │                     ├─ Combine with weight configuration in virtual key
    │                     ├─ Randomly select a provider by weight
    │                     └─ Send request (retry by weight on failure)

API Configuration Examples

Request Routing Parameters Example

  curl "https://ai-tokenhub.com/api/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $YOUR_API_KEY" \
  -d '{
    "model": "deepseek-v4-pro-ga-260813",
    "messages": [
      {"role": "user", "content": "Hello, how are you?"}
    ],
    "temperature": 0.7,
    "max_tokens": 100
  }'

Routing Information in Response

Each API response returns the actual routing information in the response headers:

X-Routed-Provider: openai
X-Routed-Model: deepseek-v4-pro-ga-260813
X-Cache-Hit: false
X-Response-Time: 234ms