---

⏱️ Async Inference

Async Inference is an asynchronous request mechanism provided by the platform, allowing clients to initiate inference requests in a "fire-and-forget" manner. A task ID (job_id) is returned immediately, and the client can poll for the final result later using that ID. This feature is only available in Gateway mode, not supported by Go SDK, and requires Logs Store to be configured.

---

✨ Core Features

• Non-blocking Interaction: The client receives an immediate response after submitting a request without waiting for inference to complete, avoiding long-lived connection timeouts
• Task Persistence: Task state and results are persistently stored, supporting on-demand queries even after network interruptions
• Only supports text-to-image and text-to-video inference scenarios
• Idempotent Queries: Results can be queried repeatedly; repeated polling incurs no additional cost
• Peak Shaving: Naturally provides traffic buffering — sudden traffic spikes are automatically queued, preventing gateway overload
• Full Ecosystem Compatibility: Background execution triggers the full plugin chain, with governance, logging, and cost tracking fully consistent with synchronous requests

---

🔄 Workflow

flowchart LR
    A[Client submits async request] --> B[Return 202 Accepted + job_id]
    B --> C[Task enters background queue]
    C --> D[Worker picks up task for inference]
    D --> E[Result persisted to storage]
    F[Client polls job_id] --> E
    E --> F[Return final result]

1. Request Submission: The client sends a request to the async endpoint, and the gateway immediately returns a task ID
2. Queue & Execution: The task enters the background queue and is executed asynchronously by a worker thread
3. Result Storage: After inference completes, the result and status are persisted
4. Result Query: The client polls for the execution result (success/failure) using the job_id

---

📋 Supported Endpoints

Request Type	Submit Endpoint (POST)	Poll Endpoint (GET)
Image Generation (Text-to-Image)	/v1/async/images/generations	/v1/async/images/generations/{job_id}
Video Generation (Text-to-Video)	/v1/async/videos/generations	/v1/async/videos/generations/{job_id}

⚠️ Async endpoints do not support streaming.

---

Submitting a Request

Use the same JSON body as the synchronous endpoint, but switch to the /v1/async/ path.

curl -X POST http://localhost:8080/v1/async/images/generations \
  -H "Content-Type: application/json" \
  -H "x-bf-vk: sk-bf-your-virtual-key" \
  -H "x-bf-async-job-result-ttl: 3600" \
  -d '{
    "model": "stabilityai/stable-diffusion-xl-base-1.0",
    "prompt": "A beautiful sunset over the ocean, 4k, high resolution",
    "n": 1,
    "size": "1024x1024"
  }'

Response (202 Accepted)

{
  "id": "1e89b165-d4fe-49e8-beb2-3e157f2df02f",
  "status": "pending",
  "created_at": "2026-02-19T08:10:17.831Z"
}

Polling for Results

Use GET on the matching endpoint with the returned job_id.

curl -X GET http://localhost:8080/v1/async/images/generations/1e89b165-d4fe-49e8-beb2-3e157f2df02f \
  -H "x-bf-vk: sk-bf-your-virtual-key"

Response codes:

• 202 Accepted: job is still pending or processing
• 200 OK: job is completed or failed

Pending example (202)

{
  "id": "1e89b165-d4fe-49e8-beb2-3e157f2df02f",
  "status": "pending",
  "created_at": "2026-02-19T08:10:17.831Z"
}

Completed example (200)

{
  "id": "1e89b165-d4fe-49e8-beb2-3e157f2df02f",
  "status": "completed",
  "created_at": "2026-02-19T08:10:17.831Z",
  "completed_at": "2026-02-19T08:10:19.412Z",
  "expires_at": "2026-02-19T09:10:19.412Z",
  "status_code": 200,
  "result": {
    "id": "img-123",
    "object": "images.generation",
    "created": 1740288617,
    "data": [
      {
        "url": "https://example.com/generated-sunset.png",
        "revised_prompt": "A beautiful sunset over the ocean, 4k, high resolution"
      }
    ]
  }
}

Failed example (200)

{
  "id": "1e89b165-d4fe-49e8-beb2-3e157f2df02f",
  "status": "failed",
  "created_at": "2026-02-19T08:10:17.831Z",
  "completed_at": "2026-02-19T08:10:19.412Z",
  "expires_at": "2026-02-19T09:10:19.412Z",
  "status_code": 429,
  "error": {
    "error": {
      "message": "rate limit exceeded",
      "type": "rate_limit_error"
    }
  }
}

---

📊 Task Lifecycle States

State	Meaning	Trigger
pending	Task created and entered queue	Returned immediately upon submission
processing	Background worker has picked up the task	Worker begins execution
completed	Operation succeeded, result stored	Provider call completed successfully
failed	Operation failed, error stored	Provider call returned an error

---

⏳ Result TTL & Expiry Cleanup

• Default TTL: 3600 seconds (1 hour), counting from task completion time, not submission time
• Global Config: Set the server-side default via client.async_job_result_ttl
• Request-level Override: Use the request header x-bf-async-job-result-ttl to specify seconds; if the value is invalid or ≤0, fall back to the default TTL
• Expiry Behavior: Polling an expired task returns 404 Job not found or expired
• Cleanup Mechanism: Expired tasks are automatically cleaned up once per minute

---

🔐 Virtual Key Authorization

• Task Binding: If a virtual key (x-bf-vk) was used at submission, that key identity is bound to the task
• Polling Verification: The same virtual key value must be provided when polling, otherwise a 404 is returned
• Public Tasks: Tasks created without a virtual key are not restricted to any caller; any request authenticated through the gateway can poll them

---

📈 Observability

• Log Marking: Async execution logs are recorded just like synchronous requests, with isAsyncRequest: true in the log metadata, displaying an Async badge in the Logs UI
• Full Plugin Chain: Background execution still triggers the complete plugin chain (governance, logging, cost tracking, etc.), ensuring audit and billing consistency

---

🎯 Core Value

Synchronous Request Pain Points	Async Inference Solution
🕒 Long-running tasks block connections: Complex inference (e.g., image generation) may take tens of seconds or even minutes, HTTP connections easily time out, occupying gateway threads	Client receives a job_id immediately after submission, the connection can close right away without maintaining a long-lived connection
🔄 Difficult client retries: When a synchronous request fails (network interruption, timeout), it's hard for the client to determine whether the server already processed it; retries may cause duplicate consumption or data inconsistency	Task state is persisted; clients can poll results idempotently at any time, and precisely retrieve error information on failure
📈 Uncontrollable backend pressure: Synchronous request pile-ups during traffic spikes can exhaust gateway resources, causing a cascade failure	Requests enter a queue first and are consumed by background workers at their capacity, naturally providing peak-shaving
❓ No progress visibility: Synchronous requests only have two terminal states (success/failure), lacking intermediate states	Provides lifecycle visibility of pending → processing → completed/failed
💸 Resource waste: While waiting for AI provider responses in synchronous requests, gateway threads are in a blocked state; high concurrency consumes memory and file descriptors	The gateway only needs to handle lightweight queue writes and status queries; inference execution is fully delegated to the background worker pool

---

📌 Typical Application Scenarios

1. Long-Running Inference Tasks

• High-resolution image generation: Stable Diffusion and similar models can take tens of seconds to generate 4K images
• Long video generation: Generating AI videos of several minutes or longer can take tens of seconds to minutes

2. Business-Critical Operations Requiring Reliability

• Batch image/video generation: Submit a large number of generation tasks at once, no need to wait, retrieve all results together when complete
• Content moderation workflows: After AI reviews image/video content, results need to be reliably stored for human review; async mode ensures results are not lost

3. Resource-Constrained Clients

• Mobile/IoT devices: Unstable networks cannot maintain long-lived HTTP connections; async polling is more robust
• Serverless functions: Cloud function execution time is usually limited (e.g., AWS Lambda max 15 minutes); async tasks easily bypass this limit

4. Service Decoupling in Microservice Architectures

Downstream services don't need to wait for AI service responses; they can write the task ID to a message queue for other services to consume results asynchronously, enabling event-driven architectures

---

💡 Integration with Existing Platform Capabilities

• Virtual Key Isolation: Async tasks from different tenants are isolated via virtual keys, preventing interference
• Fallbacks: If the primary provider fails during background inference, the Fallbacks mechanism still triggers; async tasks also have high availability
• Cost Tracking: Usage data from completed async tasks is properly accounted in cost statistics and governance logs; financial consistency is maintained

---

⚠️ Prerequisites

Async Inference is not enabled by default. The following are required:

1. Deployment mode is Gateway (not embedded SDK)
2. Logs Store is configured (for persisting task state)
3. The client must implement polling logic (the platform currently only supports polling mode; Webhook callback is under development)

In short, Async Inference upgrades the platform from a simple API proxy to an AI gateway with task queue capabilities, especially suitable for long-running, high-value inference scenarios in production environments.