Semantic Cache

1. What is Semantic Cache?

Semantic Cache uses vector similarity search to intelligently cache AI responses. Even if two requests are not literally identical, as long as they are semantically similar, the cached result can be returned directly, significantly reducing the cost and latency of repeated LLM calls.

Four Core Advantages

• Cost Reduction: Avoids repeatedly calling expensive LLM APIs for similar requests
• Performance Improvement: Sub-millisecond cache reads vs. second-level API calls
• Intelligent Matching: Goes beyond exact text matching to achieve semantic similarity recognition
• Streaming Support: Full support for caching and sequential restoration of streaming responses

2. Core Mechanism: Dual-Layer Caching

Semantic Cache includes two matching strategies:

Mode	Mechanism	Description
Exact Hash	Computes a deterministic hash of the request content	Identical requests → immediate hit, fastest speed, no embedding model required
Semantic	Generates vector embeddings for similarity search	Different wording but similar semantics → cache hit, embedding model required

Default Behavior: First attempts exact hash matching; if not hit, falls back to semantic similarity search.

3. Required Component: Vector Store

Semantic Cache must be configured with a vector database as the storage backend. Supported types include:

• Weaviate (production-grade gRPC support)
• Redis / Valkey (high-performance in-memory storage, recommended for pure hash mode)
• Qdrant (written in Rust, supports advanced filtering)
• Pinecone (managed service, supports Serverless)

Key Code: Initialize Vector Store (Weaviate Example)

import (
    "context"
    "github.com/maximhq/bifrost/framework/vectorstore"
)

vectorConfig := &vectorstore.Config{
    Enabled: true,
    Type:    vectorstore.VectorStoreTypeWeaviate,
    Config: vectorstore.WeaviateConfig{
        Scheme: "http",
        Host:   "localhost:8080",
    },
}

store, err := vectorstore.NewVectorStore(context.Background(), vectorConfig, logger)
if err != nil {
    log.Fatal("Failed to create vector store:", err)
}

4. Core Configuration: Enabling Semantic Cache

Key Code: Create and Configure Semantic Cache Plugin (Go SDK)

import (
    "time"
    "github.com/maximhq/bifrost/plugins/semanticcache"
    "github.com/maximhq/bifrost/core/schemas"
)

cacheConfig := &semanticcache.Config{
    // [Required] Embedding model configuration (if semantic matching is enabled)
    Provider:       schemas.OpenAI,
    Keys:          []schemas.Key{{Value: "sk-..."}},
    EmbeddingModel: "text-embedding-3-small",
    Dimension:     1536,
    
    // Cache behavior configuration
    TTL:       5 * time.Minute,  // Cache validity period
    Threshold: 0.8,              // Similarity threshold (default 0.8)
    CleanUpOnShutdown: true,     // Whether to clear cache on shutdown
    
    // Conversation processing configuration
    ConversationHistoryThreshold: 5,        // Do not cache conversations exceeding this message count (prevent false positives)
    ExcludeSystemPrompt: bifrost.Ptr(false), // Whether to exclude system prompt from calculation
    
    // Advanced isolation options
    CacheByModel:    bifrost.Ptr(true), // Cache isolation by model
    CacheByProvider: bifrost.Ptr(true), // Cache isolation by provider
}

plugin, err := semanticcache.Init(context.Background(), cacheConfig, logger, store)
if err != nil {
    log.Fatal("Failed to create semantic cache plugin:", err)
}

// Add plugin to Bifrost configuration
bifrostConfig := schemas.BifrostConfig{
    LLMPlugins: []schemas.LLMPlugin{plugin},
    // ... other configuration
}

5. Pure Exact Hash Mode (No Embedding Model)

If you don't want to call any embedding API (e.g., only need exact deduplication, no external network environment), you can enable pure hash mode.

Key Code: Pure Exact Hash Mode Configuration

cacheConfig := &semanticcache.Config{
    // Do not provide Provider, Keys, EmbeddingModel
    Dimension: 1, // Placeholder value; actual storage has no vectors, only metadata
    TTL:       5 * time.Minute,
    CleanUpOnShutdown: true,
    CacheByModel:      bifrost.Ptr(true),
    CacheByProvider:   bifrost.Ptr(true),
}

plugin, err := semanticcache.Init(ctx, cacheConfig, logger, store)

Note

A vector database is still required in pure hash mode; Redis / Valkey is recommended (no real vector fields needed).

6. Required Trigger: Setting a Cache Key

Semantic Cache only takes effect when a cache key is provided in the request context. Requests without a cache key will completely bypass the cache.

Key Code: Injecting a Cache Key into Requests

import "github.com/maximhq/bifrost/plugins/semanticcache"

// Set a cache key in the context; this request will be cached
ctx = context.WithValue(ctx, semanticcache.CacheKey, "session-123")
response, err := client.ChatCompletionRequest(schemas.NewBifrostContext(ctx, schemas.NoDeadline), request)

// Context without a cache key → this request will not trigger caching
response, err := client.ChatCompletionRequest(schemas.NewBifrostContext(context.Background(), schemas.NoDeadline), request)

7. Advanced Control Options

1. Per-Request Dynamic TTL and Threshold Override

ctx = context.WithValue(ctx, semanticcache.CacheKey, "session-123")
ctx = context.WithValue(ctx, semanticcache.CacheTTLKey, 30*time.Second)
ctx = context.WithValue(ctx, semanticcache.CacheThresholdKey, 0.9)

2. Force Specify Cache Type (only valid in dual mode)

// Force only exact hash matching
ctx = context.WithValue(ctx, semanticcache.CacheTypeKey, semanticcache.CacheTypeDirect)

// Force only semantic similarity matching
ctx = context.WithValue(ctx, semanticcache.CacheTypeKey, semanticcache.CacheTypeSemantic)

3. Prevent Storing This Response (Read-Only)

ctx = context.WithValue(ctx, semanticcache.CacheNoStoreKey, true)

4. Manual Cache Entry Cleanup

// Delete a specific entry by request ID
err := plugin.ClearCacheForRequestID("550e8400-e29b-41d4-a716-446655440000")

// Delete all related entries by cache key
err := plugin.ClearCacheForKey("support-session-456")

8. Debug Information After Cache Hit

After a cache hit, the response object's ExtraFields.CacheDebug field will contain a JSON object with the following key fields:

Exact Hash Hit

{
  "cache_hit": true,
  "hit_type": "direct",
  "cache_id": "550e8500-e29b-41d4-a725-446655440001"
}

Semantic Similarity Hit

{
  "cache_hit": true,
  "hit_type": "semantic",
  "cache_id": "550e8500-e29b-41d4-a725-446655440001",
  "threshold": 0.8,
  "similarity": 0.95,
  "provider_used": "openai",
  "model_used": "gpt-4o-mini",
  "input_tokens": 100
}

Tip

Use cache_id for precise cache entry management.