AI Gateway
Every AI Search instance is connected to a Cloudflare AI Gateway. The model calls that AI Search makes for embedding, query rewriting, reranking, and response generation run through this gateway. By configuring the connected gateway, you can observe and control those model calls.
To choose or change which gateway your instance uses, see Models.
AI Gateway records the model requests that run through it, so you can see what your instance is doing.
- Analytics: Track the number of requests, tokens used, cost, latency, and errors across your model calls.
- Logs: Inspect individual requests and responses, including the effective system prompt, rewritten queries, and generated answers.
By default, AI Search uses Workers AI models. To use models from other providers, such as OpenAI or Anthropic, add your provider keys to AI Gateway and select those models in AI Search.
- Add your provider keys with Bring Your Own Keys.
- Connect the gateway and select the models in your AI Search settings. For details, see Models.
Use AI Gateway Guardrails to screen the prompts and responses that flow through your instance and block content that is unsafe or inappropriate. To detect and handle sensitive information, such as personal or financial data, use Data Loss Prevention (DLP).
Configure request retries and model fallbacks so that a model call can automatically retry or fall back to another model when a provider returns an error.
Some AI Gateway features act on every request that passes through the gateway. Because your AI Search instance shares this gateway for its internal model calls, a few features can interfere with indexing and querying.
Do not turn on AI Gateway caching for the gateway connected to your AI Search instance. This matters most for embedding requests. AI Search relies on fresh embeddings to build its vector index and to match each query against it, so serving cached embeddings can store or return incorrect vectors and quietly degrade the accuracy of your search results. To cache search results, use AI Search's own Similarity cache instead.
Similarly, avoid setting rate limiting on this gateway. Rate limits apply to AI Search's own model calls, including the many embedding requests made while indexing, and can interrupt indexing and querying.