--- title: Cloudflare Workers AI description: Workers AI allows you to run AI models in a serverless way, without having to worry about scaling, maintaining, or paying for unused infrastructure. You can invoke models running on GPUs on Cloudflare's network from your own code — from Workers, Pages, or anywhere via the Cloudflare API. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) ### Tags [ AI ](https://developers.cloudflare.com/search/?tags=AI) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/index.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Cloudflare Workers AI Run machine learning models, powered by serverless GPUs, on Cloudflare's global network. Available on Free and Paid plans Workers AI allows you to run AI models in a serverless way, without having to worry about scaling, maintaining, or paying for unused infrastructure. You can invoke models running on GPUs on Cloudflare's network from your own code — from [Workers](https://developers.cloudflare.com/workers/), [Pages](https://developers.cloudflare.com/pages/), or anywhere via [the Cloudflare API](https://developers.cloudflare.com/api/resources/ai/methods/run/). Workers AI gives you access to: * **50+ [open-source models](https://developers.cloudflare.com/workers-ai/models/)**, available as a part of our model catalog * Serverless, **pay-for-what-you-use** [pricing model](https://developers.cloudflare.com/workers-ai/platform/pricing/) * All as part of a **fully-featured developer platform**, including [AI Gateway](https://developers.cloudflare.com/ai-gateway/), [Vectorize](https://developers.cloudflare.com/vectorize/), [Workers](https://developers.cloudflare.com/workers/) and more... [ Get started ](https://developers.cloudflare.com/workers-ai/get-started)[ Watch a Workers AI demo ](https://youtu.be/cK%5FleoJsBWY?si=4u6BIy%5FuBOZf9Ve8) Custom requirements If you have custom requirements like private custom models or higher limits, complete the [Custom Requirements Form ↗](https://forms.gle/axnnpGDb6xrmR31T6). Cloudflare will contact you with next steps. Workers AI is now Generally Available To report bugs or give feedback, go to the [#workers-ai Discord channel ↗](https://discord.cloudflare.com). If you are having issues with Wrangler, report issues in the [Wrangler GitHub repository ↗](https://github.com/cloudflare/workers-sdk/issues/new/choose). --- ## Features ### Models Workers AI comes with a curated set of popular open-source models that enable you to do tasks such as image classification, text generation, object detection and more. [ Browse models ](https://developers.cloudflare.com/workers-ai/models/) --- ## Related products **[AI Gateway](https://developers.cloudflare.com/ai-gateway/)** Observe and control your AI applications with caching, rate limiting, request retries, model fallback, and more. **[Vectorize](https://developers.cloudflare.com/vectorize/)** Build full-stack AI applications with Vectorize, Cloudflare’s vector database. Adding Vectorize enables you to perform tasks such as semantic search, recommendations, anomaly detection or can be used to provide context and memory to an LLM. **[Workers](https://developers.cloudflare.com/workers/)** Build serverless applications and deploy instantly across the globe for exceptional performance, reliability, and scale. **[Pages](https://developers.cloudflare.com/pages/)** Create full-stack applications that are instantly deployed to the Cloudflare global network. **[R2](https://developers.cloudflare.com/r2/)** Store large amounts of unstructured data without the costly egress bandwidth fees associated with typical cloud storage services. **[D1](https://developers.cloudflare.com/d1/)** Create new serverless SQL databases to query from your Workers and Pages projects. **[Durable Objects](https://developers.cloudflare.com/durable-objects/)** A globally distributed coordination API with strongly consistent storage. **[KV](https://developers.cloudflare.com/kv/)** Create a global, low-latency, key-value data storage. --- ## More resources [Get started](https://developers.cloudflare.com/workers-ai/get-started/workers-wrangler/) Build and deploy your first Workers AI application. [Plans](https://developers.cloudflare.com/workers-ai/platform/pricing/) Learn about Free and Paid plans. [Limits](https://developers.cloudflare.com/workers-ai/platform/limits/) Learn about Workers AI limits. [Use cases](https://developers.cloudflare.com/use-cases/ai/) Learn how you can build and deploy ambitious AI applications to Cloudflare's global network. [Storage options](https://developers.cloudflare.com/workers/platform/storage-options/) Learn which storage option is best for your project. [Developer Discord](https://discord.cloudflare.com) Connect with the Workers community on Discord to ask questions, share what you are building, and discuss the platform with other developers. [@CloudflareDev](https://x.com/cloudflaredev) Follow @CloudflareDev on Twitter to learn about product announcements, and what is new in Cloudflare Workers. ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}}]} ``` --- --- title: Getting started description: There are several options to build your Workers AI projects on Cloudflare. To get started, choose your preferred method: image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/get-started/index.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Getting started There are several options to build your Workers AI projects on Cloudflare. To get started, choose your preferred method: * [ Workers Bindings ](https://developers.cloudflare.com/workers-ai/get-started/workers-wrangler/) * [ REST API ](https://developers.cloudflare.com/workers-ai/get-started/rest-api/) * [ Dashboard ](https://developers.cloudflare.com/workers-ai/get-started/dashboard/) Note These examples are geared towards creating new Workers AI projects. For help adding Workers AI to an existing Worker, refer to [Workers Bindings](https://developers.cloudflare.com/workers-ai/configuration/bindings/). ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/get-started/","name":"Getting started"}}]} ``` --- --- title: Dashboard description: Follow this guide to create a Workers AI application using the Cloudflare dashboard. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/get-started/dashboard.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Dashboard Follow this guide to create a Workers AI application using the Cloudflare dashboard. ## Prerequisites Sign up for a [Cloudflare account ↗](https://dash.cloudflare.com/sign-up/workers-and-pages) if you have not already. ## Setup To create a Workers AI application: 1. In the Cloudflare dashboard, go to the **Workers & Pages** page. [ Go to **Workers & Pages** ](https://dash.cloudflare.com/?to=/:account/workers-and-pages) 2. Select **Create application**. 3. Under **Select a template**, select **LLM Chat App**. 4. Select **Deploy**. 5. Name your Worker, then select **Create and deploy**. 6. Preview your Worker at its provided [workers.dev](https://developers.cloudflare.com/workers/configuration/routing/workers-dev/) subdomain. ## Development ### Dashboard Editing in the dashboard is helpful for simpler use cases. Once you have created your Worker script, you can edit and deploy your Worker using the Cloudflare dashboard: 1. In the Cloudflare dashboard, go to the **Workers & Pages** page. [ Go to **Workers & Pages** ](https://dash.cloudflare.com/?to=/:account/workers-and-pages) 2. Select your application. 3. Select **Edit Code**. ![Edit code directly within the Cloudflare dashboard](https://developers.cloudflare.com/_astro/workers-edit-code.CKxxvQSe_11id2b.webp) ### Wrangler CLI To develop more advanced applications or [implement tests](https://developers.cloudflare.com/workers/testing/), start working in the Wrangler CLI. 1. Install [npm ↗](https://docs.npmjs.com/getting-started). 2. Install [Node.js ↗](https://nodejs.org/en/). Node.js version manager Use a Node version manager like [Volta ↗](https://volta.sh/) or [nvm ↗](https://github.com/nvm-sh/nvm) to avoid permission issues and change Node.js versions. [Wrangler](https://developers.cloudflare.com/workers/wrangler/install-and-update/), discussed later in this guide, requires a Node version of `16.17.0` or later. 1. Run the following command, replacing the value of `[]` which the location you want to put your Worker Script. npm yarn pnpm ``` npm create cloudflare@latest -- [] --type=pre-existing ``` ``` yarn create cloudflare [] --type=pre-existing ``` ``` pnpm create cloudflare@latest [] --type=pre-existing ``` After you run this command - and work through the prompts - your local changes will not automatically sync with dashboard. So, once you download your script, continue using the CLI. ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/get-started/","name":"Getting started"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/get-started/dashboard/","name":"Dashboard"}}]} ``` --- --- title: REST API description: Use the Cloudflare Workers AI REST API to deploy a large language model (LLM). image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/get-started/rest-api.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # REST API This guide will instruct you through setting up and deploying your first Workers AI project. You will use the Workers AI REST API to experiment with a large language model (LLM). ## Prerequisites Sign up for a [Cloudflare account ↗](https://dash.cloudflare.com/sign-up/workers-and-pages) if you have not already. ## 1\. Get API token and Account ID You need your API token and Account ID to use the REST API. To get these values: 1. In the Cloudflare dashboard, go to the **Workers AI** page. [ Go to **Workers AI** ](https://dash.cloudflare.com/?to=/:account/ai/workers-ai) 2. Select **Use REST API**. 3. Get your API token: 1. Select **Create a Workers AI API Token**. 2. Review the prefilled information. 3. Select **Create API Token**. 4. Select **Copy API Token**. 5. Save that value for future use. 4. For **Get Account ID**, copy the value for **Account ID**. Save that value for future use. Note If you choose to [create an API token](https://developers.cloudflare.com/fundamentals/api/get-started/create-token/) instead of using the template, that token will need permissions for both `Workers AI - Read` and `Workers AI - Edit`. ## 2\. Run a model via API After creating your API token, authenticate and make requests to the API using your API token in the request. You will use the [Execute AI model](https://developers.cloudflare.com/api/resources/ai/methods/run/) endpoint to run the [@cf/meta/llama-3.1-8b-instruct](https://developers.cloudflare.com/workers-ai/models/llama-3.1-8b-instruct/) model: Terminal window ``` curl https://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/ai/run/@cf/meta/llama-3.1-8b-instruct \ -H 'Authorization: Bearer {API_TOKEN}' \ -d '{ "prompt": "Where did the phrase Hello World come from" }' ``` Replace the values for `{ACCOUNT_ID}` and `{API_TOKEN}`. The API response will look like the following: ``` { "result": { "response": "Hello, World first appeared in 1974 at Bell Labs when Brian Kernighan included it in the C programming language example. It became widely used as a basic test program due to simplicity and clarity. It represents an inviting greeting from a program to the world." }, "success": true, "errors": [], "messages": [] } ``` This example execution uses the `@cf/meta/llama-3.1-8b-instruct` model, but you can use any of the models in the [Workers AI models catalog](https://developers.cloudflare.com/workers-ai/models/). If using another model, you will need to replace `{model}` with your desired model name. By completing this guide, you have created a Cloudflare account (if you did not have one already) and an API token that grants Workers AI read permissions to your account. You executed the [@cf/meta/llama-3.1-8b-instruct](https://developers.cloudflare.com/workers-ai/models/llama-3.1-8b-instruct/) model using a cURL command from the terminal and received an answer to your prompt in a JSON response. ## Related resources * [Models](https://developers.cloudflare.com/workers-ai/models/) \- Browse the Workers AI models catalog. * [AI SDK](https://developers.cloudflare.com/workers-ai/configuration/ai-sdk) \- Learn how to integrate with an AI model. ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/get-started/","name":"Getting started"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/get-started/rest-api/","name":"REST API"}}]} ``` --- --- title: Workers Bindings description: Deploy your first Cloudflare Workers AI project using the CLI. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/get-started/workers-wrangler.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Workers Bindings This guide will instruct you through setting up and deploying your first Workers AI project. You will use [Workers](https://developers.cloudflare.com/workers/), a Workers AI binding, and a large language model (LLM) to deploy your first AI-powered application on the Cloudflare global network. 1. Sign up for a [Cloudflare account ↗](https://dash.cloudflare.com/sign-up/workers-and-pages). 2. Install [Node.js ↗](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm). Node.js version manager Use a Node version manager like [Volta ↗](https://volta.sh/) or [nvm ↗](https://github.com/nvm-sh/nvm) to avoid permission issues and change Node.js versions. [Wrangler](https://developers.cloudflare.com/workers/wrangler/install-and-update/), discussed later in this guide, requires a Node version of `16.17.0` or later. ## 1\. Create a Worker project You will create a new Worker project using the `create-cloudflare` CLI (C3). [C3 ↗](https://github.com/cloudflare/workers-sdk/tree/main/packages/create-cloudflare) is a command-line tool designed to help you set up and deploy new applications to Cloudflare. Create a new project named `hello-ai` by running: npm yarn pnpm ``` npm create cloudflare@latest -- hello-ai ``` ``` yarn create cloudflare hello-ai ``` ``` pnpm create cloudflare@latest hello-ai ``` Running `npm create cloudflare@latest` will prompt you to install the [create-cloudflare package ↗](https://www.npmjs.com/package/create-cloudflare), and lead you through setup. C3 will also install [Wrangler](https://developers.cloudflare.com/workers/wrangler/), the Cloudflare Developer Platform CLI. For setup, select the following options: * For _What would you like to start with?_, choose `Hello World example`. * For _Which template would you like to use?_, choose `Worker only`. * For _Which language do you want to use?_, choose `TypeScript`. * For _Do you want to use git for version control?_, choose `Yes`. * For _Do you want to deploy your application?_, choose `No` (we will be making some changes before deploying). This will create a new `hello-ai` directory. Your new `hello-ai` directory will include: * A `"Hello World"` [Worker](https://developers.cloudflare.com/workers/get-started/guide/#3-write-code) at `src/index.ts`. * A [wrangler.jsonc](https://developers.cloudflare.com/workers/wrangler/configuration/) configuration file. Go to your application directory: Terminal window ``` cd hello-ai ``` ## 2\. Connect your Worker to Workers AI You must create an AI binding for your Worker to connect to Workers AI. [Bindings](https://developers.cloudflare.com/workers/runtime-apis/bindings/) allow your Workers to interact with resources, like Workers AI, on the Cloudflare Developer Platform. To bind Workers AI to your Worker, add the following to the end of your Wrangler file: * [ wrangler.jsonc ](#tab-panel-6941) * [ wrangler.toml ](#tab-panel-6942) ``` { "ai": { "binding": "AI" } } ``` ``` [ai] binding = "AI" ``` Your binding is [available in your Worker code](https://developers.cloudflare.com/workers/reference/migrate-to-module-workers/#bindings-in-es-modules-format) on [env.AI](https://developers.cloudflare.com/workers/runtime-apis/handlers/fetch/). You can also bind Workers AI to a Pages Function. For more information, refer to [Functions Bindings](https://developers.cloudflare.com/pages/functions/bindings/#workers-ai). ## 3\. Run an inference task in your Worker You are now ready to run an inference task in your Worker. In this case, you will use an LLM, [llama-3.1-8b-instruct](https://developers.cloudflare.com/workers-ai/models/llama-3.1-8b-instruct/), to answer a question. Update the `index.ts` file in your `hello-ai` application directory with the following code: * [ JavaScript ](#tab-panel-6943) * [ TypeScript ](#tab-panel-6944) index.js ``` export default { async fetch(request, env) { const response = await env.AI.run("@cf/meta/llama-3.1-8b-instruct", { prompt: "What is the origin of the phrase Hello, World", }); return new Response(JSON.stringify(response)); }, }; ``` index.ts ``` export interface Env { // If you set another name in the Wrangler config file as the value for 'binding', // replace "AI" with the variable name you defined. AI: Ai; } export default { async fetch(request, env): Promise { const response = await env.AI.run("@cf/meta/llama-3.1-8b-instruct", { prompt: "What is the origin of the phrase Hello, World", }); return new Response(JSON.stringify(response)); }, } satisfies ExportedHandler; ``` Up to this point, you have created an AI binding for your Worker and configured your Worker to be able to execute the Llama 3.1 model. You can now test your project locally before you deploy globally. ## 4\. Develop locally with Wrangler While in your project directory, test Workers AI locally by running [wrangler dev](https://developers.cloudflare.com/workers/wrangler/commands/general/#dev): Terminal window ``` npx wrangler dev ``` Workers AI local development usage charges Using Workers AI always accesses your Cloudflare account in order to run AI models and will incur usage charges even in local development. You will be prompted to log in after you run `wrangler dev`. When you run `npx wrangler dev`, Wrangler will give you a URL (most likely `localhost:8787`) to review your Worker. After you go to the URL Wrangler provides, a message will render that resembles the following example: ``` { "response": "Ah, a most excellent question, my dear human friend! *adjusts glasses*\n\nThe origin of the phrase \"Hello, World\" is a fascinating tale that spans several decades and multiple disciplines. It all began in the early days of computer programming, when a young man named Brian Kernighan was tasked with writing a simple program to demonstrate the basics of a new programming language called C.\nKernighan, a renowned computer scientist and author, was working at Bell Labs in the late 1970s when he created the program. He wanted to showcase the language's simplicity and versatility, so he wrote a basic \"Hello, World!\" program that printed the familiar greeting to the console.\nThe program was included in Kernighan and Ritchie's influential book \"The C Programming Language,\" published in 1978. The book became a standard reference for C programmers, and the \"Hello, World!\" program became a sort of \"Hello, World!\" for the programming community.\nOver time, the phrase \"Hello, World!\" became a shorthand for any simple program that demonstrated the basics" } ``` ## 5\. Deploy your AI Worker Before deploying your AI Worker globally, log in with your Cloudflare account by running: Terminal window ``` npx wrangler login ``` You will be directed to a web page asking you to log in to the Cloudflare dashboard. After you have logged in, you will be asked if Wrangler can make changes to your Cloudflare account. Scroll down and select **Allow** to continue. Finally, deploy your Worker to make your project accessible on the Internet. To deploy your Worker, run: Terminal window ``` npx wrangler deploy ``` ``` https://hello-ai..workers.dev ``` Your Worker will be deployed to your custom [workers.dev](https://developers.cloudflare.com/workers/configuration/routing/workers-dev/) subdomain. You can now visit the URL to run your AI Worker. By finishing this tutorial, you have created a Worker, connected it to Workers AI through an AI binding, and ran an inference task from the Llama 3 model. ## Related resources * [Cloudflare Developers community on Discord ↗](https://discord.cloudflare.com) \- Submit feature requests, report bugs, and share your feedback directly with the Cloudflare team by joining the Cloudflare Discord server. * [Models](https://developers.cloudflare.com/workers-ai/models/) \- Browse the Workers AI models catalog. * [AI SDK](https://developers.cloudflare.com/workers-ai/configuration/ai-sdk) \- Learn how to integrate with an AI model. ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/get-started/","name":"Getting started"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/get-started/workers-wrangler/","name":"Workers Bindings"}}]} ``` --- --- title: Models image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Copy page # Models Task Type Text-to-Speech Summarization Text Embeddings Text Classification Text Generation Object Detection Text-to-Image Automatic Speech Recognition Translation Image-to-Text Image Classification Voice Activity DetectionCapabilitiesModel Reasoning Function calling VisionPlatform Batch Partner Real-time LoRAAuthors Deepgram facebook baai thebloke DeepSeek HuggingFace lykon Google tiiuae Black Forest Labs aisingapore Zhipu AI OpenAI IBM nousresearch ai4bharat Moonshot AI Meta meta-llama llava-hf Leonardo myshell-ai MistralAI MistralAI NVIDIA openchat Microsoft pfnet Qwen pipecat-ai defog runwayml Stability.ai bytedance nexusflow tinyllama unum fblgit [📌![Moonshot AI logo](https://developers.cloudflare.com/_astro/moonshotai.CZZI-XBJ.svg)kimi-k2.5Text Generation • Moonshot AIKimi K2.5 is a frontier-scale open-source model with a 256k context window, multi-turn tool calling, vision inputs, and structured outputs for agentic workloads.BatchFunction callingReasoningVision](https://developers.cloudflare.com/workers-ai/models/kimi-k2.5)[📌![Zhipu AI logo](https://developers.cloudflare.com/_astro/zai-org.B9OfhU57.svg)glm-4.7-flashText Generation • Zhipu AIGLM-4.7-Flash is a fast and efficient multilingual text generation model with a 131,072 token context window. Optimized for dialogue, instruction-following, and multi-turn tool calling across 100+ languages.Function callingReasoning](https://developers.cloudflare.com/workers-ai/models/glm-4.7-flash)[📌![OpenAI logo](https://developers.cloudflare.com/_astro/openai.ChTKThcR.svg)gpt-oss-120bText Generation • OpenAIOpenAI’s open-weight models designed for powerful reasoning, agentic tasks, and versatile developer use cases – gpt-oss-120b is for production, general purpose, high reasoning use-cases.Function callingReasoning](https://developers.cloudflare.com/workers-ai/models/gpt-oss-120b)[📌![Meta logo](https://developers.cloudflare.com/_astro/meta.x5nlFKBG.svg)llama-4-scout-17b-16e-instructText Generation • MetaMeta's Llama 4 Scout is a 17 billion parameter model with 16 experts that is natively multimodal. These models leverage a mixture-of-experts architecture to offer industry-leading performance in text and image understanding.BatchFunction callingVision](https://developers.cloudflare.com/workers-ai/models/llama-4-scout-17b-16e-instruct)[![NVIDIA logo](https://developers.cloudflare.com/_astro/nvidia.CRWb-0oU.svg)nemotron-3-120b-a12bText Generation • NVIDIANVIDIA Nemotron 3 Super is a hybrid MoE model with leading accuracy for multi-agent applications and specialized agentic AI systems.Function callingReasoning](https://developers.cloudflare.com/workers-ai/models/nemotron-3-120b-a12b)[![Black Forest Labs logo](https://developers.cloudflare.com/_astro/blackforestlabs.Ccs-Y4-D.svg)flux-2-klein-9bText-to-Image • Black Forest LabsFLUX.2 \[klein\] 9B is an ultra-fast, distilled image model with enhanced quality. It unifies image generation and editing in a single model, delivering state-of-the-art quality enabling interactive workflows, real-time previews, and latency-critical applications.Partner](https://developers.cloudflare.com/workers-ai/models/flux-2-klein-9b)[![Black Forest Labs logo](https://developers.cloudflare.com/_astro/blackforestlabs.Ccs-Y4-D.svg)flux-2-klein-4bText-to-Image • Black Forest LabsFLUX.2 \[klein\] is an ultra-fast, distilled image model. It unifies image generation and editing in a single model, delivering state-of-the-art quality enabling interactive workflows, real-time previews, and latency-critical applications.Partner](https://developers.cloudflare.com/workers-ai/models/flux-2-klein-4b)[![Black Forest Labs logo](https://developers.cloudflare.com/_astro/blackforestlabs.Ccs-Y4-D.svg)flux-2-devText-to-Image • Black Forest LabsFLUX.2 \[dev\] is an image model from Black Forest Labs where you can generate highly realistic and detailed images, with multi-reference support.Partner](https://developers.cloudflare.com/workers-ai/models/flux-2-dev)[![Deepgram logo](https://developers.cloudflare.com/_astro/deepgram.DVGPhlbc.svg)aura-2-esText-to-Speech • DeepgramAura-2 is a context-aware text-to-speech (TTS) model that applies natural pacing, expressiveness, and fillers based on the context of the provided text. The quality of your text input directly impacts the naturalness of the audio output.BatchPartnerReal-time](https://developers.cloudflare.com/workers-ai/models/aura-2-es)[![Deepgram logo](https://developers.cloudflare.com/_astro/deepgram.DVGPhlbc.svg)aura-2-enText-to-Speech • DeepgramAura-2 is a context-aware text-to-speech (TTS) model that applies natural pacing, expressiveness, and fillers based on the context of the provided text. The quality of your text input directly impacts the naturalness of the audio output.BatchPartnerReal-time](https://developers.cloudflare.com/workers-ai/models/aura-2-en)[![IBM logo](https://developers.cloudflare.com/_astro/ibm.RSRiCjkw.svg)granite-4.0-h-microText Generation • IBMGranite 4.0 instruct models deliver strong performance across benchmarks, achieving industry-leading results in key agentic tasks like instruction following and function calling. These efficiencies make the models well-suited for a wide range of use cases like retrieval-augmented generation (RAG), multi-agent workflows, and edge deployments.Function calling](https://developers.cloudflare.com/workers-ai/models/granite-4.0-h-micro)[![Deepgram logo](https://developers.cloudflare.com/_astro/deepgram.DVGPhlbc.svg)fluxAutomatic Speech Recognition • DeepgramFlux is the first conversational speech recognition model built specifically for voice agents.PartnerReal-time](https://developers.cloudflare.com/workers-ai/models/flux)[pplamo-embedding-1bText Embeddings • pfnetPLaMo-Embedding-1B is a Japanese text embedding model developed by Preferred Networks, Inc. It can convert Japanese text input into numerical vectors and can be used for a wide range of applications, including information retrieval, text classification, and clustering.](https://developers.cloudflare.com/workers-ai/models/plamo-embedding-1b)[agemma-sea-lion-v4-27b-itText Generation • aisingaporeSEA-LION stands for Southeast Asian Languages In One Network, which is a collection of Large Language Models (LLMs) which have been pretrained and instruct-tuned for the Southeast Asia (SEA) region.](https://developers.cloudflare.com/workers-ai/models/gemma-sea-lion-v4-27b-it)[aindictrans2-en-indic-1BTranslation • ai4bharatIndicTrans2 is the first open-source transformer-based multilingual NMT model that supports high-quality translations across all the 22 scheduled Indic languages](https://developers.cloudflare.com/workers-ai/models/indictrans2-en-indic-1B)[![Google logo](https://developers.cloudflare.com/_astro/google.C4p59fss.svg)embeddinggemma-300mText Embeddings • GoogleEmbeddingGemma is a 300M parameter, state-of-the-art for its size, open embedding model from Google, built from Gemma 3 (with T5Gemma initialization) and the same research and technology used to create Gemini models. EmbeddingGemma produces vector representations of text, making it well-suited for search and retrieval tasks, including classification, clustering, and semantic similarity search. This model was trained with data in 100+ spoken languages.](https://developers.cloudflare.com/workers-ai/models/embeddinggemma-300m)[![Deepgram logo](https://developers.cloudflare.com/_astro/deepgram.DVGPhlbc.svg)aura-1Text-to-Speech • DeepgramAura is a context-aware text-to-speech (TTS) model that applies natural pacing, expressiveness, and fillers based on the context of the provided text. The quality of your text input directly impacts the naturalness of the audio output.BatchPartnerReal-time](https://developers.cloudflare.com/workers-ai/models/aura-1)[![Leonardo logo](https://developers.cloudflare.com/_astro/leonardo.CTjvkdQs.svg)lucid-originText-to-Image • LeonardoLucid Origin from Leonardo.AI is their most adaptable and prompt-responsive model to date. Whether you're generating images with sharp graphic design, stunning full-HD renders, or highly specific creative direction, it adheres closely to your prompts, renders text with accuracy, and supports a wide array of visual styles and aesthetics – from stylized concept art to crisp product mockups.Partner](https://developers.cloudflare.com/workers-ai/models/lucid-origin)[![Leonardo logo](https://developers.cloudflare.com/_astro/leonardo.CTjvkdQs.svg)phoenix-1.0Text-to-Image • LeonardoPhoenix 1.0 is a model by Leonardo.Ai that generates images with exceptional prompt adherence and coherent text.Partner](https://developers.cloudflare.com/workers-ai/models/phoenix-1.0)[![OpenAI logo](https://developers.cloudflare.com/_astro/openai.ChTKThcR.svg)gpt-oss-20bText Generation • OpenAIOpenAI’s open-weight models designed for powerful reasoning, agentic tasks, and versatile developer use cases – gpt-oss-20b is for lower latency, and local or specialized use-cases.Function callingReasoning](https://developers.cloudflare.com/workers-ai/models/gpt-oss-20b)[psmart-turn-v2Voice Activity Detection • pipecat-aiAn open source, community-driven, native audio turn detection model in 2nd versionBatchReal-time](https://developers.cloudflare.com/workers-ai/models/smart-turn-v2)[![Qwen logo](https://developers.cloudflare.com/_astro/qwen.B8ST_F2H.svg)qwen3-embedding-0.6bText Embeddings • QwenThe Qwen3 Embedding model series is the latest proprietary model of the Qwen family, specifically designed for text embedding and ranking tasks. ](https://developers.cloudflare.com/workers-ai/models/qwen3-embedding-0.6b)[![Deepgram logo](https://developers.cloudflare.com/_astro/deepgram.DVGPhlbc.svg)nova-3Automatic Speech Recognition • DeepgramTranscribe audio using Deepgram’s speech-to-text modelBatchPartnerReal-time](https://developers.cloudflare.com/workers-ai/models/nova-3)[![Qwen logo](https://developers.cloudflare.com/_astro/qwen.B8ST_F2H.svg)qwen3-30b-a3b-fp8Text Generation • QwenQwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models. Built upon extensive training, Qwen3 delivers groundbreaking advancements in reasoning, instruction-following, agent capabilities, and multilingual support.BatchFunction callingReasoning](https://developers.cloudflare.com/workers-ai/models/qwen3-30b-a3b-fp8)[![Google logo](https://developers.cloudflare.com/_astro/google.C4p59fss.svg)gemma-3-12b-itText Generation • GoogleGemma 3 models are well-suited for a variety of text generation and image understanding tasks, including question answering, summarization, and reasoning. Gemma 3 models are multimodal, handling text and image input and generating text output, with a large, 128K context window, multilingual support in over 140 languages, and is available in more sizes than previous versions.LoRA](https://developers.cloudflare.com/workers-ai/models/gemma-3-12b-it)[![MistralAI logo](https://developers.cloudflare.com/_astro/mistralai.Bn9UMUMu.svg)mistral-small-3.1-24b-instructText Generation • MistralAIBuilding upon Mistral Small 3 (2501), Mistral Small 3.1 (2503) adds state-of-the-art vision understanding and enhances long context capabilities up to 128k tokens without compromising text performance. With 24 billion parameters, this model achieves top-tier capabilities in both text and vision tasks.Function calling](https://developers.cloudflare.com/workers-ai/models/mistral-small-3.1-24b-instruct)[![Qwen logo](https://developers.cloudflare.com/_astro/qwen.B8ST_F2H.svg)qwq-32bText Generation • QwenQwQ is the reasoning model of the Qwen series. Compared with conventional instruction-tuned models, QwQ, which is capable of thinking and reasoning, can achieve significantly enhanced performance in downstream tasks, especially hard problems. QwQ-32B is the medium-sized reasoning model, which is capable of achieving competitive performance against state-of-the-art reasoning models, e.g., DeepSeek-R1, o1-mini.LoRAReasoning](https://developers.cloudflare.com/workers-ai/models/qwq-32b)[![Qwen logo](https://developers.cloudflare.com/_astro/qwen.B8ST_F2H.svg)qwen2.5-coder-32b-instructText Generation • QwenQwen2.5-Coder is the latest series of Code-Specific Qwen large language models (formerly known as CodeQwen). As of now, Qwen2.5-Coder has covered six mainstream model sizes, 0.5, 1.5, 3, 7, 14, 32 billion parameters, to meet the needs of different developers. Qwen2.5-Coder brings the following improvements upon CodeQwen1.5:LoRA](https://developers.cloudflare.com/workers-ai/models/qwen2.5-coder-32b-instruct)[bbge-reranker-baseText Classification • baaiDifferent from embedding model, reranker uses question and document as input and directly output similarity instead of embedding. You can get a relevance score by inputting query and passage to the reranker. And the score can be mapped to a float value in \[0,1\] by sigmoid function.](https://developers.cloudflare.com/workers-ai/models/bge-reranker-base)[![Meta logo](https://developers.cloudflare.com/_astro/meta.x5nlFKBG.svg)llama-guard-3-8bText Generation • MetaLlama Guard 3 is a Llama-3.1-8B pretrained model, fine-tuned for content safety classification. Similar to previous versions, it can be used to classify content in both LLM inputs (prompt classification) and in LLM responses (response classification). It acts as an LLM – it generates text in its output that indicates whether a given prompt or response is safe or unsafe, and if unsafe, it also lists the content categories violated.LoRA](https://developers.cloudflare.com/workers-ai/models/llama-guard-3-8b)[![DeepSeek logo](https://developers.cloudflare.com/_astro/deepseek.Dn1KbMH4.svg)deepseek-r1-distill-qwen-32bText Generation • DeepSeekDeepSeek-R1-Distill-Qwen-32B is a model distilled from DeepSeek-R1 based on Qwen2.5\. It outperforms OpenAI-o1-mini across various benchmarks, achieving new state-of-the-art results for dense models.Reasoning](https://developers.cloudflare.com/workers-ai/models/deepseek-r1-distill-qwen-32b)[![Meta logo](https://developers.cloudflare.com/_astro/meta.x5nlFKBG.svg)llama-3.3-70b-instruct-fp8-fastText Generation • MetaLlama 3.3 70B quantized to fp8 precision, optimized to be faster.BatchFunction calling](https://developers.cloudflare.com/workers-ai/models/llama-3.3-70b-instruct-fp8-fast)[![Meta logo](https://developers.cloudflare.com/_astro/meta.x5nlFKBG.svg)llama-3.2-1b-instructText Generation • MetaThe Llama 3.2 instruction-tuned text only models are optimized for multilingual dialogue use cases, including agentic retrieval and summarization tasks.](https://developers.cloudflare.com/workers-ai/models/llama-3.2-1b-instruct)[![Meta logo](https://developers.cloudflare.com/_astro/meta.x5nlFKBG.svg)llama-3.2-3b-instructText Generation • MetaThe Llama 3.2 instruction-tuned text only models are optimized for multilingual dialogue use cases, including agentic retrieval and summarization tasks.](https://developers.cloudflare.com/workers-ai/models/llama-3.2-3b-instruct)[![Meta logo](https://developers.cloudflare.com/_astro/meta.x5nlFKBG.svg)llama-3.2-11b-vision-instructText Generation • Meta The Llama 3.2-Vision instruction-tuned models are optimized for visual recognition, image reasoning, captioning, and answering general questions about an image.LoRAVision](https://developers.cloudflare.com/workers-ai/models/llama-3.2-11b-vision-instruct)[![Black Forest Labs logo](https://developers.cloudflare.com/_astro/blackforestlabs.Ccs-Y4-D.svg)flux-1-schnellText-to-Image • Black Forest LabsFLUX.1 \[schnell\] is a 12 billion parameter rectified flow transformer capable of generating images from text descriptions. ](https://developers.cloudflare.com/workers-ai/models/flux-1-schnell)[![Meta logo](https://developers.cloudflare.com/_astro/meta.x5nlFKBG.svg)llama-3.1-8b-instruct-awqText Generation • MetaQuantized (int4) generative text model with 8 billion parameters from Meta.](https://developers.cloudflare.com/workers-ai/models/llama-3.1-8b-instruct-awq)[![Meta logo](https://developers.cloudflare.com/_astro/meta.x5nlFKBG.svg)llama-3.1-8b-instruct-fp8Text Generation • MetaLlama 3.1 8B quantized to FP8 precision](https://developers.cloudflare.com/workers-ai/models/llama-3.1-8b-instruct-fp8)[mmelottsText-to-Speech • myshell-aiMeloTTS is a high-quality multi-lingual text-to-speech library by MyShell.ai.](https://developers.cloudflare.com/workers-ai/models/melotts)[![Meta logo](https://developers.cloudflare.com/_astro/meta.x5nlFKBG.svg)llama-3.1-8b-instructText Generation • MetaThe Meta Llama 3.1 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction tuned generative models. The Llama 3.1 instruction tuned text only models are optimized for multilingual dialogue use cases and outperform many of the available open source and closed chat models on common industry benchmarks.](https://developers.cloudflare.com/workers-ai/models/llama-3.1-8b-instruct)[bbge-m3Text Embeddings • baaiMulti-Functionality, Multi-Linguality, and Multi-Granularity embeddings model.](https://developers.cloudflare.com/workers-ai/models/bge-m3)[mmeta-llama-3-8b-instructText Generation • meta-llamaGeneration over generation, Meta Llama 3 demonstrates state-of-the-art performance on a wide range of industry benchmarks and offers new capabilities, including improved reasoning. ](https://developers.cloudflare.com/workers-ai/models/meta-llama-3-8b-instruct)[![OpenAI logo](https://developers.cloudflare.com/_astro/openai.ChTKThcR.svg)whisper-large-v3-turboAutomatic Speech Recognition • OpenAIWhisper is a pre-trained model for automatic speech recognition (ASR) and speech translation. Batch](https://developers.cloudflare.com/workers-ai/models/whisper-large-v3-turbo)[![Meta logo](https://developers.cloudflare.com/_astro/meta.x5nlFKBG.svg)llama-3-8b-instruct-awqText Generation • MetaQuantized (int4) generative text model with 8 billion parameters from Meta.](https://developers.cloudflare.com/workers-ai/models/llama-3-8b-instruct-awq)[lllava-1.5-7b-hfBetaImage-to-Text • llava-hfLLaVA is an open-source chatbot trained by fine-tuning LLaMA/Vicuna on GPT-generated multimodal instruction-following data. It is an auto-regressive language model, based on the transformer architecture.](https://developers.cloudflare.com/workers-ai/models/llava-1.5-7b-hf)[funa-cybertron-7b-v2-bf16BetaText Generation • fblgitCybertron 7B v2 is a 7B MistralAI based model, best on it's series. It was trained with SFT, DPO and UNA (Unified Neural Alignment) on multiple datasets.Deprecated](https://developers.cloudflare.com/workers-ai/models/una-cybertron-7b-v2-bf16)[![OpenAI logo](https://developers.cloudflare.com/_astro/openai.ChTKThcR.svg)whisper-tiny-enBetaAutomatic Speech Recognition • OpenAIWhisper is a pre-trained model for automatic speech recognition (ASR) and speech translation. Trained on 680k hours of labelled data, Whisper models demonstrate a strong ability to generalize to many datasets and domains without the need for fine-tuning. This is the English-only version of the Whisper Tiny model which was trained on the task of speech recognition.](https://developers.cloudflare.com/workers-ai/models/whisper-tiny-en)[![Meta logo](https://developers.cloudflare.com/_astro/meta.x5nlFKBG.svg)llama-3-8b-instructText Generation • MetaGeneration over generation, Meta Llama 3 demonstrates state-of-the-art performance on a wide range of industry benchmarks and offers new capabilities, including improved reasoning.](https://developers.cloudflare.com/workers-ai/models/llama-3-8b-instruct)[![MistralAI logo](https://developers.cloudflare.com/_astro/mistralai.Bn9UMUMu.svg)mistral-7b-instruct-v0.2BetaText Generation • MistralAIThe Mistral-7B-Instruct-v0.2 Large Language Model (LLM) is an instruct fine-tuned version of the Mistral-7B-v0.2\. Mistral-7B-v0.2 has the following changes compared to Mistral-7B-v0.1: 32k context window (vs 8k context in v0.1), rope-theta = 1e6, and no Sliding-Window Attention.LoRA](https://developers.cloudflare.com/workers-ai/models/mistral-7b-instruct-v0.2)[![Google logo](https://developers.cloudflare.com/_astro/google.C4p59fss.svg)gemma-7b-it-loraBetaText Generation • Google This is a Gemma-7B base model that Cloudflare dedicates for inference with LoRA adapters. Gemma is a family of lightweight, state-of-the-art open models from Google, built from the same research and technology used to create the Gemini models.LoRA](https://developers.cloudflare.com/workers-ai/models/gemma-7b-it-lora)[![Google logo](https://developers.cloudflare.com/_astro/google.C4p59fss.svg)gemma-2b-it-loraBetaText Generation • GoogleThis is a Gemma-2B base model that Cloudflare dedicates for inference with LoRA adapters. Gemma is a family of lightweight, state-of-the-art open models from Google, built from the same research and technology used to create the Gemini models.LoRA](https://developers.cloudflare.com/workers-ai/models/gemma-2b-it-lora)[mllama-2-7b-chat-hf-loraBetaText Generation • meta-llamaThis is a Llama2 base model that Cloudflare dedicated for inference with LoRA adapters. Llama 2 is a collection of pretrained and fine-tuned generative text models ranging in scale from 7 billion to 70 billion parameters. This is the repository for the 7B fine-tuned model, optimized for dialogue use cases and converted for the Hugging Face Transformers format. LoRA](https://developers.cloudflare.com/workers-ai/models/llama-2-7b-chat-hf-lora)[![Google logo](https://developers.cloudflare.com/_astro/google.C4p59fss.svg)gemma-7b-itBetaText Generation • GoogleGemma is a family of lightweight, state-of-the-art open models from Google, built from the same research and technology used to create the Gemini models. They are text-to-text, decoder-only large language models, available in English, with open weights, pre-trained variants, and instruction-tuned variants.LoRA](https://developers.cloudflare.com/workers-ai/models/gemma-7b-it)[nstarling-lm-7b-betaBetaText Generation • nexusflowWe introduce Starling-LM-7B-beta, an open large language model (LLM) trained by Reinforcement Learning from AI Feedback (RLAIF). Starling-LM-7B-beta is trained from Openchat-3.5-0106 with our new reward model Nexusflow/Starling-RM-34B and policy optimization method Fine-Tuning Language Models from Human Preferences (PPO).Deprecated](https://developers.cloudflare.com/workers-ai/models/starling-lm-7b-beta)[nhermes-2-pro-mistral-7bBetaText Generation • nousresearchHermes 2 Pro on Mistral 7B is the new flagship 7B Hermes! Hermes 2 Pro is an upgraded, retrained version of Nous Hermes 2, consisting of an updated and cleaned version of the OpenHermes 2.5 Dataset, as well as a newly introduced Function Calling and JSON Mode dataset developed in-house.Function calling](https://developers.cloudflare.com/workers-ai/models/hermes-2-pro-mistral-7b)[![MistralAI logo](https://developers.cloudflare.com/_astro/mistralai.Bn9UMUMu.svg)mistral-7b-instruct-v0.2-loraBetaText Generation • MistralAIThe Mistral-7B-Instruct-v0.2 Large Language Model (LLM) is an instruct fine-tuned version of the Mistral-7B-v0.2.LoRA](https://developers.cloudflare.com/workers-ai/models/mistral-7b-instruct-v0.2-lora)[![Qwen logo](https://developers.cloudflare.com/_astro/qwen.B8ST_F2H.svg)qwen1.5-1.8b-chatBetaText Generation • QwenQwen1.5 is the improved version of Qwen, the large language model series developed by Alibaba Cloud.Deprecated](https://developers.cloudflare.com/workers-ai/models/qwen1.5-1.8b-chat)[uuform-gen2-qwen-500mBetaImage-to-Text • unumUForm-Gen is a small generative vision-language model primarily designed for Image Captioning and Visual Question Answering. The model was pre-trained on the internal image captioning dataset and fine-tuned on public instructions datasets: SVIT, LVIS, VQAs datasets.](https://developers.cloudflare.com/workers-ai/models/uform-gen2-qwen-500m)[fbart-large-cnnBetaSummarization • facebookBART is a transformer encoder-encoder (seq2seq) model with a bidirectional (BERT-like) encoder and an autoregressive (GPT-like) decoder. You can use this model for text summarization.](https://developers.cloudflare.com/workers-ai/models/bart-large-cnn)[![Microsoft logo](https://developers.cloudflare.com/_astro/microsoft.BfW2Sks3.svg)phi-2BetaText Generation • MicrosoftPhi-2 is a Transformer-based model with a next-word prediction objective, trained on 1.4T tokens from multiple passes on a mixture of Synthetic and Web datasets for NLP and coding.](https://developers.cloudflare.com/workers-ai/models/phi-2)[ttinyllama-1.1b-chat-v1.0BetaText Generation • tinyllamaThe TinyLlama project aims to pretrain a 1.1B Llama model on 3 trillion tokens. This is the chat model finetuned on top of TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T.Deprecated](https://developers.cloudflare.com/workers-ai/models/tinyllama-1.1b-chat-v1.0)[![Qwen logo](https://developers.cloudflare.com/_astro/qwen.B8ST_F2H.svg)qwen1.5-14b-chat-awqBetaText Generation • QwenQwen1.5 is the improved version of Qwen, the large language model series developed by Alibaba Cloud. AWQ is an efficient, accurate and blazing-fast low-bit weight quantization method, currently supporting 4-bit quantization.Deprecated](https://developers.cloudflare.com/workers-ai/models/qwen1.5-14b-chat-awq)[![Qwen logo](https://developers.cloudflare.com/_astro/qwen.B8ST_F2H.svg)qwen1.5-7b-chat-awqBetaText Generation • QwenQwen1.5 is the improved version of Qwen, the large language model series developed by Alibaba Cloud. AWQ is an efficient, accurate and blazing-fast low-bit weight quantization method, currently supporting 4-bit quantization.Deprecated](https://developers.cloudflare.com/workers-ai/models/qwen1.5-7b-chat-awq)[![Qwen logo](https://developers.cloudflare.com/_astro/qwen.B8ST_F2H.svg)qwen1.5-0.5b-chatBetaText Generation • QwenQwen1.5 is the improved version of Qwen, the large language model series developed by Alibaba Cloud.Deprecated](https://developers.cloudflare.com/workers-ai/models/qwen1.5-0.5b-chat)[tdiscolm-german-7b-v1-awqBetaText Generation • theblokeDiscoLM German 7b is a Mistral-based large language model with a focus on German-language applications. AWQ is an efficient, accurate and blazing-fast low-bit weight quantization method, currently supporting 4-bit quantization.Deprecated](https://developers.cloudflare.com/workers-ai/models/discolm-german-7b-v1-awq)[tfalcon-7b-instructBetaText Generation • tiiuaeFalcon-7B-Instruct is a 7B parameters causal decoder-only model built by TII based on Falcon-7B and finetuned on a mixture of chat/instruct datasets.Deprecated](https://developers.cloudflare.com/workers-ai/models/falcon-7b-instruct)[oopenchat-3.5-0106BetaText Generation • openchatOpenChat is an innovative library of open-source language models, fine-tuned with C-RLFT - a strategy inspired by offline reinforcement learning.Deprecated](https://developers.cloudflare.com/workers-ai/models/openchat-3.5-0106)[dsqlcoder-7b-2BetaText Generation • defogThis model is intended to be used by non-technical users to understand data inside their SQL databases. ](https://developers.cloudflare.com/workers-ai/models/sqlcoder-7b-2)[![DeepSeek logo](https://developers.cloudflare.com/_astro/deepseek.Dn1KbMH4.svg)deepseek-math-7b-instructBetaText Generation • DeepSeekDeepSeekMath-Instruct 7B is a mathematically instructed tuning model derived from DeepSeekMath-Base 7B. DeepSeekMath is initialized with DeepSeek-Coder-v1.5 7B and continues pre-training on math-related tokens sourced from Common Crawl, together with natural language and code data for 500B tokens.Deprecated](https://developers.cloudflare.com/workers-ai/models/deepseek-math-7b-instruct)[fdetr-resnet-50BetaObject Detection • facebookDEtection TRansformer (DETR) model trained end-to-end on COCO 2017 object detection (118k annotated images).](https://developers.cloudflare.com/workers-ai/models/detr-resnet-50)[bstable-diffusion-xl-lightningBetaText-to-Image • bytedanceSDXL-Lightning is a lightning-fast text-to-image generation model. It can generate high-quality 1024px images in a few steps.](https://developers.cloudflare.com/workers-ai/models/stable-diffusion-xl-lightning)[ldreamshaper-8-lcmText-to-Image • lykonStable Diffusion model that has been fine-tuned to be better at photorealism without sacrificing range.](https://developers.cloudflare.com/workers-ai/models/dreamshaper-8-lcm)[rstable-diffusion-v1-5-img2imgBetaText-to-Image • runwaymlStable Diffusion is a latent text-to-image diffusion model capable of generating photo-realistic images. Img2img generate a new image from an input image with Stable Diffusion. ](https://developers.cloudflare.com/workers-ai/models/stable-diffusion-v1-5-img2img)[rstable-diffusion-v1-5-inpaintingBetaText-to-Image • runwaymlStable Diffusion Inpainting is a latent text-to-image diffusion model capable of generating photo-realistic images given any text input, with the extra capability of inpainting the pictures by using a mask.](https://developers.cloudflare.com/workers-ai/models/stable-diffusion-v1-5-inpainting)[tdeepseek-coder-6.7b-instruct-awqBetaText Generation • theblokeDeepseek Coder is composed of a series of code language models, each trained from scratch on 2T tokens, with a composition of 87% code and 13% natural language in both English and Chinese.Deprecated](https://developers.cloudflare.com/workers-ai/models/deepseek-coder-6.7b-instruct-awq)[tdeepseek-coder-6.7b-base-awqBetaText Generation • theblokeDeepseek Coder is composed of a series of code language models, each trained from scratch on 2T tokens, with a composition of 87% code and 13% natural language in both English and Chinese.Deprecated](https://developers.cloudflare.com/workers-ai/models/deepseek-coder-6.7b-base-awq)[tllamaguard-7b-awqBetaText Generation • theblokeLlama Guard is a model for classifying the safety of LLM prompts and responses, using a taxonomy of safety risks.Deprecated](https://developers.cloudflare.com/workers-ai/models/llamaguard-7b-awq)[tneural-chat-7b-v3-1-awqBetaText Generation • theblokeThis model is a fine-tuned 7B parameter LLM on the Intel Gaudi 2 processor from the mistralai/Mistral-7B-v0.1 on the open source dataset Open-Orca/SlimOrca.Deprecated](https://developers.cloudflare.com/workers-ai/models/neural-chat-7b-v3-1-awq)[topenhermes-2.5-mistral-7b-awqBetaText Generation • theblokeOpenHermes 2.5 Mistral 7B is a state of the art Mistral Fine-tune, a continuation of OpenHermes 2 model, which trained on additional code datasets.Deprecated](https://developers.cloudflare.com/workers-ai/models/openhermes-2.5-mistral-7b-awq)[tllama-2-13b-chat-awqBetaText Generation • theblokeLlama 2 13B Chat AWQ is an efficient, accurate and blazing-fast low-bit weight quantized Llama 2 variant.Deprecated](https://developers.cloudflare.com/workers-ai/models/llama-2-13b-chat-awq)[tmistral-7b-instruct-v0.1-awqBetaText Generation • theblokeMistral 7B Instruct v0.1 AWQ is an efficient, accurate and blazing-fast low-bit weight quantized Mistral variant.Deprecated](https://developers.cloudflare.com/workers-ai/models/mistral-7b-instruct-v0.1-awq)[tzephyr-7b-beta-awqBetaText Generation • theblokeZephyr 7B Beta AWQ is an efficient, accurate and blazing-fast low-bit weight quantized Zephyr model variant.Deprecated](https://developers.cloudflare.com/workers-ai/models/zephyr-7b-beta-awq)[![Stability.ai logo](https://developers.cloudflare.com/_astro/stabilityai.CWXCgVjU.svg)stable-diffusion-xl-base-1.0BetaText-to-Image • Stability.aiDiffusion-based text-to-image generative model by Stability AI. Generates and modify images based on text prompts.](https://developers.cloudflare.com/workers-ai/models/stable-diffusion-xl-base-1.0)[bbge-large-en-v1.5Text Embeddings • baaiBAAI general embedding (Large) model that transforms any given text into a 1024-dimensional vectorBatch](https://developers.cloudflare.com/workers-ai/models/bge-large-en-v1.5)[bbge-small-en-v1.5Text Embeddings • baaiBAAI general embedding (Small) model that transforms any given text into a 384-dimensional vectorBatch](https://developers.cloudflare.com/workers-ai/models/bge-small-en-v1.5)[![Meta logo](https://developers.cloudflare.com/_astro/meta.x5nlFKBG.svg)llama-2-7b-chat-fp16Text Generation • MetaFull precision (fp16) generative text model with 7 billion parameters from Meta](https://developers.cloudflare.com/workers-ai/models/llama-2-7b-chat-fp16)[![MistralAI logo](https://developers.cloudflare.com/_astro/mistralai.Bn9UMUMu.svg)mistral-7b-instruct-v0.1Text Generation • MistralAIInstruct fine-tuned version of the Mistral-7b generative text model with 7 billion parametersLoRA](https://developers.cloudflare.com/workers-ai/models/mistral-7b-instruct-v0.1)[bbge-base-en-v1.5Text Embeddings • baaiBAAI general embedding (Base) model that transforms any given text into a 768-dimensional vectorBatch](https://developers.cloudflare.com/workers-ai/models/bge-base-en-v1.5)[![HuggingFace logo](https://developers.cloudflare.com/_astro/huggingface.DHiS2HZA.svg)distilbert-sst-2-int8Text Classification • HuggingFaceDistilled BERT model that was finetuned on SST-2 for sentiment classification](https://developers.cloudflare.com/workers-ai/models/distilbert-sst-2-int8)[![Meta logo](https://developers.cloudflare.com/_astro/meta.x5nlFKBG.svg)llama-2-7b-chat-int8Text Generation • MetaQuantized (int8) generative text model with 7 billion parameters from Meta](https://developers.cloudflare.com/workers-ai/models/llama-2-7b-chat-int8)[![Meta logo](https://developers.cloudflare.com/_astro/meta.x5nlFKBG.svg)m2m100-1.2bTranslation • MetaMultilingual encoder-decoder (seq-to-seq) model trained for Many-to-Many multilingual translationBatch](https://developers.cloudflare.com/workers-ai/models/m2m100-1.2b)[![Microsoft logo](https://developers.cloudflare.com/_astro/microsoft.BfW2Sks3.svg)resnet-50Image Classification • Microsoft50 layers deep image classification CNN trained on more than 1M images from ImageNet](https://developers.cloudflare.com/workers-ai/models/resnet-50)[![OpenAI logo](https://developers.cloudflare.com/_astro/openai.ChTKThcR.svg)whisperAutomatic Speech Recognition • OpenAIWhisper is a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multitasking model that can perform multilingual speech recognition, speech translation, and language identification.](https://developers.cloudflare.com/workers-ai/models/whisper)[![Meta logo](https://developers.cloudflare.com/_astro/meta.x5nlFKBG.svg)llama-3.1-70b-instructText Generation • MetaThe Meta Llama 3.1 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction tuned generative models. The Llama 3.1 instruction tuned text only models are optimized for multilingual dialogue use cases and outperform many of the available open source and closed chat models on common industry benchmarks.](https://developers.cloudflare.com/workers-ai/models/llama-3.1-70b-instruct)[![Meta logo](https://developers.cloudflare.com/_astro/meta.x5nlFKBG.svg)llama-3.1-8b-instruct-fastText Generation • Meta\[Fast version\] The Meta Llama 3.1 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction tuned generative models. The Llama 3.1 instruction tuned text only models are optimized for multilingual dialogue use cases and outperform many of the available open source and closed chat models on common industry benchmarks.](https://developers.cloudflare.com/workers-ai/models/llama-3.1-8b-instruct-fast) ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/models/","name":"Models"}}]} ``` --- --- title: Agents image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/agents.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Agents Build AI assistants that can perform complex tasks on behalf of your users using Cloudflare Workers AI and Agents. [ Go to Agents documentation ](https://developers.cloudflare.com/agents/) ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/agents/","name":"Agents"}}]} ``` --- --- title: Playground image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/playground.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Playground ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/playground/","name":"Playground"}}]} ``` --- --- title: REST API reference image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/api-reference.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # REST API reference ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/api-reference/","name":"REST API reference"}}]} ``` --- --- title: Changelog description: Review recent changes to Cloudflare Workers AI. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/changelog.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Changelog [ Subscribe to RSS ](https://developers.cloudflare.com/workers-ai/changelog/index.xml) ## 2026-03-19 **Moonshot AI Kimi K2.5 now available on Workers AI** * [@cf/moonshotai/kimi-k2.5](https://developers.cloudflare.com/workers-ai/models/kimi-k2.5/) now available on Workers AI. A frontier-scale open-source model with a 256k context window, multi-turn tool calling, vision inputs, and structured outputs for agentic workloads. Read [changelog](https://developers.cloudflare.com/changelog/post/2026-03-19-kimi-k2-5-workers-ai/) to get started. * New [Prompt caching](https://developers.cloudflare.com/workers-ai/features/prompt-caching/) documentation. Send the `x-session-affinity` header to route requests to the same model instance and maximize prefix cache hit rates across multi-turn conversations. * Redesigned [Asynchronous Batch API](https://developers.cloudflare.com/workers-ai/features/batch-api/) with a pull-based system that processes queued requests as capacity becomes available, avoiding out-of-capacity errors for durable workflows. ## 2026-03-11 **NVIDIA Nemotron 3 Super now available on Workers AI** * [@cf/nvidia/nemotron-3-120b-a12b](https://developers.cloudflare.com/workers-ai/models/nemotron-3-120b-a12b/) now available on Workers AI! A hybrid MoE model with 120B total parameters and 12B active, optimized for multi-agent and agentic AI workloads. Read [changelog](https://developers.cloudflare.com/changelog/post/2026-03-11-nemotron-3-super-workers-ai/) to get started. ## 2026-03-06 **Deepgram Nova-3 now supports 10 languages with regional variants** * [@cf/deepgram/nova-3](https://developers.cloudflare.com/workers-ai/models/nova-3/) now supports 10 languages with regional variants for real-time transcription. Supported languages include English, Spanish, French, German, Hindi, Russian, Portuguese, Japanese, Italian, and Dutch — with regional variants like `en-GB`, `fr-CA`, and `pt-BR`. ## 2026-02-17 **Chat Completions API support for gpt-oss models and tool calling improvements** * [@cf/openai/gpt-oss-120b](https://developers.cloudflare.com/workers-ai/models/gpt-oss-120b/) and [@cf/openai/gpt-oss-20b](https://developers.cloudflare.com/workers-ai/models/gpt-oss-20b/) now support Chat Completions API format. Use `/v1/chat/completions` with a `messages` array, or use `/ai/run` which dynamically detects your input format and accepts Chat Completions (`messages`), legacy Completions (`prompt`), or Responses API (`input`). * **\[Bug fix\]** Fixed a bug in the schema for multiple text generation models where the `content` field in message objects only accepted string values. The field now properly accepts both string content and array content (structured content parts for multi-modal inputs). This fix applies to all affected chat models including GPT-OSS models, Llama 3.x, Mistral, Qwen, and others. * **\[Bug fix\]** Tool call round-trips now work correctly. The binding no longer rejects `tool_call_id` values that it generated itself, fixing issues with multi-turn tool calling conversations. * **\[Bug fix\]** Assistant messages with `content: null` and `tool_calls` are now accepted in both the Workers AI binding and REST API (`/v1/chat/completions`), fixing tool call round-trip failures. * **\[Bug fix\]** Streaming responses now correctly report `finish_reason` only on the usage chunk, matching OpenAI's streaming behavior and preventing duplicate finish events. * **\[Bug fix\]** `/v1/chat/completions` now preserves original tool call IDs from models instead of regenerating them. Previously, the endpoint was generating new IDs which broke multi-turn tool calling because AI SDK clients could not match tool results to their original calls. * **\[Bug fix\]** `/v1/chat/completions` now correctly reports `finish_reason: "tool_calls"` in the final usage chunk when tools are used. Previously, it was hardcoding `finish_reason: "stop"` which caused AI SDK clients to think the conversation was complete instead of executing tool calls. ## 2026-02-13 **GLM-4.7-Flash, @cloudflare/tanstack-ai, and workers-ai-provider v3.1.1** * [@cf/zai-org/glm-4.7-flash](https://developers.cloudflare.com/workers-ai/models/glm-4.7-flash/) is now available on Workers AI! A fast and efficient multilingual text generation model optimized for multi-turn tool calling across 100+ languages. Read [changelog](https://developers.cloudflare.com/changelog/2026-02-13-glm-4.7-flash-workers-ai/) to get started. * New [@cloudflare/tanstack-ai](https://www.npmjs.com/package/@cloudflare/tanstack-ai) package for using Workers AI and AI Gateway with TanStack AI. * [workers-ai-provider v3.1.1](https://www.npmjs.com/package/workers-ai-provider) adds transcription, text-to-speech, and reranking capabilities. ## 2026-01-28 **Black Forest Labs FLUX.2 \[klein\] 9B now available** * [@cf/black-forest-labs/flux-2-klein-9b](https://developers.cloudflare.com/workers-ai/models/flux-2-klein-9b/) now available on Workers AI! Read [changelog](https://developers.cloudflare.com/changelog/2026-01-28-flux-2-klein-9b-workers-ai/) to get started ## 2026-01-15 **Black Forest Labs FLUX.2 \[klein\] 4b now available** * [@cf/black-forest-labs/flux-2-klein-4b](https://developers.cloudflare.com/workers-ai/models/flux-2-klein-4b/) now available on Workers AI! Read [changelog](https://developers.cloudflare.com/changelog/2026-01-15-flux-2-klein-4b-workers-ai/) to get started ## 2025-12-03 **Deepgram Flux promotional period over on Dec 8, 2025 - now has pricing** * Check out updated pricing on the [@cf/deepgram/flux](https://developers.cloudflare.com/workers-ai/models/flux/) model page or [pricing](https://developers.cloudflare.com/workers-ai/platform/pricing/) page * Pricing will start Dec 8, 2025 ## 2025-11-25 **Black Forest Labs FLUX.2 dev now available** * [@cf/black-forest-labs/flux-2-dev](https://developers.cloudflare.com/workers-ai/models/flux-2-dev/) now available on Workers AI! Read [changelog](https://developers.cloudflare.com/changelog/2025-11-25-flux-2-dev-workers-ai/) to get started ## 2025-11-13 **Qwen3 LLM and Embeddings available on Workers AI** * [@cf/qwen/qwen3-30b-a3b-fp8](https://developers.cloudflare.com/workers-ai/models/qwen3-30b-a3b-fp8/) and [@cf/qwen/qwen3-embedding-0.6b](https://developers.cloudflare.com/workers-ai/models/qwen3-embedding-0.6b) now available on Workers AI ## 2025-10-21 **New voice and LLM models on Workers AI** * Deepgram Aura 2 brings new text-to-speech capabilities to Workers AI. Check out [@cf/deepgram/aura-2-en](https://developers.cloudflare.com/workers-ai/models/aura-2-en/) and [@cf/deepgram/aura-2-es](https://developers.cloudflare.com/workers-ai/models/aura-2-es/) on how to use the new models. * IBM Granite model is also up! This new LLM model is small but mighty, take a look at the docs for more [@cf/ibm-granite/granite-4.0-h-micro](https://developers.cloudflare.com/workers-ai/models/granite-4.0-h-micro/) ## 2025-10-02 **Deepgram Flux now available on Workers AI** * We're excited to be a launch partner with Deepgram and offer their new Speech Recognition model built specifically for enabling voice agents. Check out [Deepgram's blog](https://deepgram.com/flux) for more details on the release. * Access the model through [@cf/deepgram/flux](https://developers.cloudflare.com/workers-ai/models/flux/) and check out the [changelog](https://developers.cloudflare.com/changelog/2025-10-02-deepgram-flux/) for in-depth examples. ## 2025-09-24 **New local models available on Workers AI** * We've added support for some regional models on Workers AI in support of uplifting local AI labs and AI sovereignty. Check out the [full blog post here](https://blog.cloudflare.com/sovereign-ai-and-choice). * [@cf/pfnet/plamo-embedding-1b](https://developers.cloudflare.com/workers-ai/models/plamo-embedding-1b) creates embeddings from Japanese text. * [@cf/aisingapore/gemma-sea-lion-v4-27b-it](https://developers.cloudflare.com/workers-ai/models/gemma-sea-lion-v4-27b-it) is a fine-tuned model that supports multiple South East Asian languages, including Burmese, English, Indonesian, Khmer, Lao, Malay, Mandarin, Tagalog, Tamil, Thai, and Vietnamese. * [@cf/ai4bharat/indictrans2-en-indic-1B](https://developers.cloudflare.com/workers-ai/models/indictrans2-en-indic-1B) is a translation model that can translate between 22 indic languages, including Bengali, Gujarati, Hindi, Tamil, Sanskrit and even traditionally low-resourced languages like Kashmiri, Manipuri and Sindhi. ## 2025-09-23 **New document formats supported by Markdown conversion utility** * Our [Markdown conversion utility](https://developers.cloudflare.com/workers-ai/features/markdown-conversion/) now supports converting `.docx` and `.odt` files. ## 2025-09-18 **Model Catalog updates (types, EmbeddingGemma, model deprecation)** * Workers AI types got updated in the upcoming wrangler release, please use `npm i -D wrangler@latest` to update your packages. * EmbeddingGemma model accuracy has been improved, we recommend re-indexing data to take advantage of the improved accuracy * Some older Workers AI models are being deprecated on October 1st, 2025\. We reccommend you use the newer models such as [Llama 4](https://developers.cloudflare.com/workers-ai/models/llama-4-scout-17b-16e-instruct/) and [gpt-oss](https://developers.cloudflare.com/workers-ai/models/gpt-oss-120b/). The following models are being deprecated: * @hf/thebloke/zephyr-7b-beta-awq * @hf/thebloke/mistral-7b-instruct-v0.1-awq * @hf/thebloke/llama-2-13b-chat-awq * @hf/thebloke/openhermes-2.5-mistral-7b-awq * @hf/thebloke/neural-chat-7b-v3-1-awq * @hf/thebloke/llamaguard-7b-awq * @hf/thebloke/deepseek-coder-6.7b-base-awq * @hf/thebloke/deepseek-coder-6.7b-instruct-awq * @cf/deepseek-ai/deepseek-math-7b-instruct * @cf/openchat/openchat-3.5-0106 * @cf/tiiuae/falcon-7b-instruct * @cf/thebloke/discolm-german-7b-v1-awq * @cf/qwen/qwen1.5-0.5b-chat * @cf/qwen/qwen1.5-7b-chat-awq * @cf/qwen/qwen1.5-14b-chat-awq * @cf/tinyllama/tinyllama-1.1b-chat-v1.0 * @cf/qwen/qwen1.5-1.8b-chat * @hf/nexusflow/starling-lm-7b-beta * @cf/fblgit/una-cybertron-7b-v2-bf16 ## 2025-09-05 **Introducing EmbeddingGemma from Google** * We’re excited to be a launch partner alongside Google to bring their newest embedding model to Workers AI. We're excited to introduce EmbeddingGemma delivers best-in-class performance for its size, enabling RAG and semantic search use cases. Take a look at [@cf/google/embeddinggemma-300m](https://developers.cloudflare.com/workers-ai/models/embeddinggemma-300m) for more details. Now available to use for embedding in AI Search too. ## 2025-08-27 **Introducing Partner models to the Workers AI catalog** * Read the [blog](https://blog.cloudflare.com/workers-ai-partner-models) for more details * [@cf/deepgram/aura-1](https://developers.cloudflare.com/workers-ai/models/aura-1) is a text-to-speech model that allows you to input text and have it come to life in a customizable voice * [@cf/deepgram/nova-3](https://developers.cloudflare.com/workers-ai/models/nova-3) is speech-to-text model that transcribes multilingual audio at a blazingly fast speed * [@cf/pipecat-ai/smart-turn-v2](https://developers.cloudflare.com/workers-ai/models/smart-turn-v2) helps you detect when someone is done speaking * [@cf/leonardo/lucid-origin](https://developers.cloudflare.com/workers-ai/models/lucid-origin) is a text-to-image model that generates images with sharp graphic design, stunning full-HD renders, or highly specific creative direction * [@cf/leonardo/phoenix-1.0](https://developers.cloudflare.com/workers-ai/models/phoenix-1.0) is a text-to-image model with exceptional prompt adherence and coherent text * WebSocket support added for audio models like `@cf/deepgram/aura-1`, `@cf/deepgram/nova-3`, `@cf/pipecat-ai/smart-turn-v2` ## 2025-08-05 **Adding gpt-oss models to our catalog** * Check out the [blog](https://blog.cloudflare.com/openai-gpt-oss-on-workers-ai) for more details about the new models * Take a look at the [gpt-oss-120b](https://developers.cloudflare.com/workers-ai/models/gpt-oss-120b) and [gpt-oss-20b](https://developers.cloudflare.com/workers-ai/models/gpt-oss-20b) model pages for more information about schemas, pricing, and context windows ## 2025-04-09 **Pricing correction for @cf/myshell-ai/melotts** * We've updated our documentation to reflect the correct pricing for melotts: $0.0002 per audio minute, which is actually cheaper than initially stated. The documented pricing was incorrect, where it said users would be charged based on input tokens. ## 2025-03-17 **Minor updates to the model schema for llama-3.2-1b-instruct, whisper-large-v3-turbo, llama-guard** * [llama-3.2-1b-instruct](https://developers.cloudflare.com/workers-ai/models/llama-3.2-1b-instruct/) \- updated context window to the accurate 60,000 * [whisper-large-v3-turbo](https://developers.cloudflare.com/workers-ai/models/whisper-large-v3-turbo/) \- new hyperparameters available * [llama-guard-3-8b](https://developers.cloudflare.com/workers-ai/models/llama-guard-3-8b/) \- the messages array must alternate between `user` and `assistant` to function correctly ## 2025-02-21 **Workers AI bug fixes** * We fixed a bug where `max_tokens` defaults were not properly being respected - `max_tokens` now correctly defaults to `256` as displayed on the model pages. Users relying on the previous behaviour may observe this as a breaking change. If you want to generate more tokens, please set the `max_tokens` parameter to what you need. * We updated model pages to show context windows - which is defined as the tokens used in the prompt + tokens used in the response. If your prompt + response tokens exceed the context window, the request will error. Please set `max_tokens` accordingly depending on your prompt length and the context window length to ensure a successful response. ## 2024-09-26 **Workers AI Birthday Week 2024 announcements** * Meta Llama 3.2 1B, 3B, and 11B vision is now available on Workers AI * `@cf/black-forest-labs/flux-1-schnell` is now available on Workers AI * Workers AI is fast! Powered by new GPUs and optimizations, you can expect faster inference on Llama 3.1, Llama 3.2, and FLUX models. * No more neurons. Workers AI is moving towards [unit-based pricing](https://developers.cloudflare.com/workers-ai/platform/pricing) * Model pages get a refresh with better documentation on parameters, pricing, and model capabilities * Closed beta for our Run Any\* Model feature, [sign up here](https://forms.gle/h7FcaTF4Zo5dzNb68) * Check out the [product announcements blog post](https://blog.cloudflare.com/workers-ai) for more information * And the [technical blog post](https://blog.cloudflare.com/workers-ai/making-workers-ai-faster) if you want to learn about how we made Workers AI fast ## 2024-07-23 **Meta Llama 3.1 now available on Workers AI** Workers AI now suppoorts [Meta Llama 3.1](https://developers.cloudflare.com/workers-ai/models/llama-3.1-8b-instruct/). ## 2024-06-27 **Introducing embedded function calling** * A new way to do function calling with [Embedded function calling](https://developers.cloudflare.com/workers-ai/function-calling/embedded) * Published new [@cloudflare/ai-utils](https://www.npmjs.com/package/@cloudflare/ai-utils) npm package * Open-sourced [ai-utils on Github](https://github.com/cloudflare/ai-utils) ## 2024-06-19 **Added support for traditional function calling** * [Function calling](https://developers.cloudflare.com/workers-ai/function-calling/) is now supported on enabled models * Properties added on [models](https://developers.cloudflare.com/workers-ai/models/) page to show which models support function calling ## 2024-06-18 **Native support for AI Gateways** Workers AI now natively supports [AI Gateway](https://developers.cloudflare.com/ai-gateway/usage/providers/workersai/#worker). ## 2024-06-11 **Deprecation announcement for \`@cf/meta/llama-2-7b-chat-int8\`** We will be deprecating `@cf/meta/llama-2-7b-chat-int8` on 2024-06-30. Replace the model ID in your code with a new model of your choice: * [@cf/meta/llama-3-8b-instruct](https://developers.cloudflare.com/workers-ai/models/llama-3-8b-instruct/) is the newest model in the Llama family (and is currently free for a limited time on Workers AI). * [@cf/meta/llama-3-8b-instruct-awq](https://developers.cloudflare.com/workers-ai/models/llama-3-8b-instruct-awq/) is the new Llama 3 in a similar precision to your currently selected model. This model is also currently free for a limited time. If you do not switch to a different model by June 30th, we will automatically start returning inference from `@cf/meta/llama-3-8b-instruct-awq`. ## 2024-05-29 **Add new public LoRAs and note on LoRA routing** * Added documentation on [new public LoRAs](https://developers.cloudflare.com/workers-ai/fine-tunes/public-loras/). * Noted that you can now run LoRA inference with the base model rather than explicitly calling the `-lora` version ## 2024-05-17 **Add OpenAI compatible API endpoints** Added OpenAI compatible API endpoints for `/v1/chat/completions` and `/v1/embeddings`. For more details, refer to [Configurations](https://developers.cloudflare.com/workers-ai/configuration/open-ai-compatibility/). ## 2024-04-11 **Add AI native binding** * Added new AI native binding, you can now run models with `const resp = await env.AI.run(modelName, inputs)` * Deprecated `@cloudflare/ai` npm package. While existing solutions using the @cloudflare/ai package will continue to work, no new Workers AI features will be supported. Moving to native AI bindings is highly recommended ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/changelog/","name":"Changelog"}}]} ``` --- --- title: Vercel AI SDK description: Workers AI can be used with the Vercel AI SDK for JavaScript and TypeScript codebases. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/configuration/ai-sdk.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Vercel AI SDK Workers AI can be used with the [Vercel AI SDK ↗](https://sdk.vercel.ai/) for JavaScript and TypeScript codebases. ## Setup Install the [workers-ai-provider provider ↗](https://sdk.vercel.ai/providers/community-providers/cloudflare-workers-ai): npm yarn pnpm bun ``` npm i workers-ai-provider ``` ``` yarn add workers-ai-provider ``` ``` pnpm add workers-ai-provider ``` ``` bun add workers-ai-provider ``` Then, add an AI binding in your Workers project Wrangler file: ``` [ai] binding = "AI" ``` ## Models The AI SDK can be configured to work with [any AI model](https://developers.cloudflare.com/workers-ai/models/). JavaScript ``` import { createWorkersAI } from "workers-ai-provider"; const workersai = createWorkersAI({ binding: env.AI }); // Choose any model: https://developers.cloudflare.com/workers-ai/models/ const model = workersai("@cf/meta/llama-3.1-8b-instruct", {}); ``` ## Generate Text Once you have selected your model, you can generate text from a given prompt. JavaScript ``` import { createWorkersAI } from 'workers-ai-provider'; import { generateText } from 'ai'; type Env = { AI: Ai; }; export default { async fetch(_: Request, env: Env) { const workersai = createWorkersAI({ binding: env.AI }); const result = await generateText({ model: workersai('@cf/meta/llama-2-7b-chat-int8'), prompt: 'Write a 50-word essay about hello world.', }); return new Response(result.text); }, }; ``` ## Stream Text For longer responses, consider streaming responses to provide as the generation completes. JavaScript ``` import { createWorkersAI } from 'workers-ai-provider'; import { streamText } from 'ai'; type Env = { AI: Ai; }; export default { async fetch(_: Request, env: Env) { const workersai = createWorkersAI({ binding: env.AI }); const result = streamText({ model: workersai('@cf/meta/llama-2-7b-chat-int8'), prompt: 'Write a 50-word essay about hello world.', }); return result.toTextStreamResponse({ headers: { // add these headers to ensure that the // response is chunked and streamed 'Content-Type': 'text/x-unknown', 'content-encoding': 'identity', 'transfer-encoding': 'chunked', }, }); }, }; ``` ## Generate Structured Objects You can provide a Zod schema to generate a structured JSON response. JavaScript ``` import { createWorkersAI } from 'workers-ai-provider'; import { generateObject } from 'ai'; import { z } from 'zod'; type Env = { AI: Ai; }; export default { async fetch(_: Request, env: Env) { const workersai = createWorkersAI({ binding: env.AI }); const result = await generateObject({ model: workersai('@cf/meta/llama-3.1-8b-instruct'), prompt: 'Generate a Lasagna recipe', schema: z.object({ recipe: z.object({ ingredients: z.array(z.string()), description: z.string(), }), }), }); return Response.json(result.object); }, }; ``` ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/configuration/","name":"Configuration"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/configuration/ai-sdk/","name":"Vercel AI SDK"}}]} ``` --- --- title: Workers Bindings description: Workers provides a serverless execution environment that allows you to create new applications or augment existing ones. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/configuration/bindings.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Workers Bindings ## Workers [Workers](https://developers.cloudflare.com/workers/) provides a serverless execution environment that allows you to create new applications or augment existing ones. To use Workers AI with Workers, you must create a Workers AI [binding](https://developers.cloudflare.com/workers/runtime-apis/bindings/). Bindings allow your Workers to interact with resources, like Workers AI, on the Cloudflare Developer Platform. You create bindings on the Cloudflare dashboard or by updating your [Wrangler file](https://developers.cloudflare.com/workers/wrangler/configuration/). To bind Workers AI to your Worker, add the following to the end of your Wrangler file: * [ wrangler.jsonc ](#tab-panel-6921) * [ wrangler.toml ](#tab-panel-6922) ``` { "ai": { "binding": "AI" // i.e. available in your Worker on env.AI } } ``` ``` [ai] binding = "AI" ``` ## Pages Functions [Pages Functions](https://developers.cloudflare.com/pages/functions/) allow you to build full-stack applications with Cloudflare Pages by executing code on the Cloudflare network. Functions are Workers under the hood. To configure a Workers AI binding in your Pages Function, you must use the Cloudflare dashboard. Refer to [Workers AI bindings](https://developers.cloudflare.com/pages/functions/bindings/#workers-ai) for instructions. ## Methods ### async env.AI.run() `async env.AI.run()` runs a model. Takes a model as the first parameter, and an object as the second parameter. JavaScript ``` const answer = await env.AI.run('@cf/meta/llama-3.1-8b-instruct', { prompt: "What is the origin of the phrase 'Hello, World'" }); ``` **Parameters** * `model` ` string ` required * The model to run. **Supported options** * `stream` ` boolean ` optional * Returns a stream of results as they are available. JavaScript ``` const answer = await env.AI.run('@cf/meta/llama-3.1-8b-instruct', { prompt: "What is the origin of the phrase 'Hello, World'", stream: true }); return new Response(answer, { headers: { "content-type": "text/event-stream" } }); ``` ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/configuration/","name":"Configuration"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/configuration/bindings/","name":"Workers Bindings"}}]} ``` --- --- title: Hugging Face Chat UI description: Use Workers AI with Chat UI, an open-source chat interface offered by Hugging Face. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/configuration/hugging-face-chat-ui.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Hugging Face Chat UI Use Workers AI with [Chat UI ↗](https://github.com/huggingface/chat-ui?tab=readme-ov-file#text-embedding-models), an open-source chat interface offered by Hugging Face. ## Prerequisites You will need the following: * A [Cloudflare account ↗](https://dash.cloudflare.com) * Your [Account ID](https://developers.cloudflare.com/fundamentals/account/find-account-and-zone-ids/) * An [API token](https://developers.cloudflare.com/workers-ai/get-started/rest-api/#1-get-api-token-and-account-id) for Workers AI ## Setup First, decide how to reference your Account ID and API token (either directly in your `.env.local` using the `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` variables or in the endpoint configuration). Then, follow the rest of the setup instructions in the [Chat UI GitHub repository ↗](https://github.com/huggingface/chat-ui?tab=readme-ov-file#text-embedding-models). When setting up your models, specify the `cloudflare` endpoint. ``` { "name" : "nousresearch/hermes-2-pro-mistral-7b", "tokenizer": "nousresearch/hermes-2-pro-mistral-7b", "parameters": { "stop": ["<|im_end|>"] }, "endpoints" : [ { "type": "cloudflare", // optionally specify these if not included in .env.local "accountId": "your-account-id", "apiToken": "your-api-token" // } ] } ``` ## Supported models This template works with any [text generation models](https://developers.cloudflare.com/workers-ai/models/) that begin with the `@hf` parameter. ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/configuration/","name":"Configuration"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/configuration/hugging-face-chat-ui/","name":"Hugging Face Chat UI"}}]} ``` --- --- title: OpenAI compatible API endpoints description: Workers AI supports OpenAI compatible endpoints for text generation (/v1/chat/completions) and text embedding models (/v1/embeddings). This allows you to use the same code as you would for your OpenAI commands, but swap in Workers AI easily. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/configuration/open-ai-compatibility.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # OpenAI compatible API endpoints Workers AI supports OpenAI compatible endpoints for [text generation](https://developers.cloudflare.com/workers-ai/models/) (`/v1/chat/completions`) and [text embedding models](https://developers.cloudflare.com/workers-ai/models/) (`/v1/embeddings`). This allows you to use the same code as you would for your OpenAI commands, but swap in Workers AI easily. ## Usage ### Workers AI Normally, Workers AI requires you to specify the model name in the cURL endpoint or within the `env.AI.run` function. With OpenAI compatible endpoints, you can leverage the [openai-node sdk ↗](https://github.com/openai/openai-node) to make calls to Workers AI. This allows you to use Workers AI by simply changing the base URL and the model name. OpenAI SDK Example ``` import OpenAI from "openai"; const openai = new OpenAI({ apiKey: env.CLOUDFLARE_API_KEY, baseURL: `https://api.cloudflare.com/client/v4/accounts/${env.CLOUDFLARE_ACCOUNT_ID}/ai/v1`, }); // Use chat completions const chatCompletion = await openai.chat.completions.create({ messages: [{ role: "user", content: "Make some robot noises" }], model: "@cf/meta/llama-3.1-8b-instruct", }); // Use responses const response = await openai.responses.create({ model: "@cf/openai/gpt-oss-120b", input: "Talk to me about open source", }); const embeddings = await openai.embeddings.create({ model: "@cf/baai/bge-large-en-v1.5", input: "I love matcha", }); ``` cURL example ``` curl --request POST \ --url https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/v1/chat/completions \ --header "Authorization: Bearer {api_token}" \ --header "Content-Type: application/json" \ --data ' { "model": "@cf/meta/llama-3.1-8b-instruct", "messages": [ { "role": "user", "content": "how to build a wooden spoon in 3 short steps? give as short as answer as possible" } ] } ' ``` ### AI Gateway These endpoints are also compatible with [AI Gateway](https://developers.cloudflare.com/ai-gateway/usage/providers/workersai/#openai-compatible-endpoints). ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/configuration/","name":"Configuration"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/configuration/open-ai-compatibility/","name":"OpenAI compatible API endpoints"}}]} ``` --- --- title: Asynchronous Batch API description: Asynchronous batch processing lets you send a collection (batch) of inference requests in a single call. Instead of expecting immediate responses for every request, the system queues them for processing and returns the results later. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/features/batch-api/index.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Asynchronous Batch API Asynchronous batch processing lets you send a collection (batch) of inference requests in a single call. Instead of expecting immediate responses for every request, the system queues them for processing and returns the results later. Batch processing is useful for large workloads such as summarization or embeddings when there is no human interaction. Using the batch API will guarantee that your requests are fulfilled eventually, rather than erroring out if Cloudflare does not have enough capacity at a given time. When you send a batch request, the API immediately acknowledges receipt with a status like `queued` and provides a unique `request_id`. This ID is later used to poll for the final responses once the processing is complete. You can use the Batch API by either creating and deploying a Cloudflare Worker that leverages the [Batch API with the AI binding](https://developers.cloudflare.com/workers-ai/features/batch-api/workers-binding/), using the [REST API](https://developers.cloudflare.com/workers-ai/features/batch-api/rest-api/) directly or by starting from a [template ↗](https://github.com/craigsdennis/batch-please-workers-ai). Note Ensure that the total payload is under 10 MB. ## Demo application If you want to get started quickly, click the button below: [![Deploy to Workers](https://deploy.workers.cloudflare.com/button)](https://deploy.workers.cloudflare.com/?url=https://github.com/craigsdennis/batch-please-workers-ai) This will create a repository in your GitHub account and deploy a ready-to-use Worker that demonstrates how to use Cloudflare's Asynchronous Batch API. The template includes preconfigured AI bindings, and examples for sending and retrieving batch requests with and without external references. Once deployed, you can visit the live Worker and start experimenting with the Batch API immediately. ## Supported Models Refer to our [model catalog](https://developers.cloudflare.com/workers-ai/models/?capabilities=Batch) for supported models. ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/features/","name":"Features"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/features/batch-api/","name":"Asynchronous Batch API"}}]} ``` --- --- title: REST API description: If you prefer to work directly with the REST API instead of a Cloudflare Worker, below are the steps on how to do it: image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/features/batch-api/rest-api.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # REST API If you prefer to work directly with the REST API instead of a [Cloudflare Worker](https://developers.cloudflare.com/workers-ai/features/batch-api/workers-binding/), below are the steps on how to do it: ## 1\. Sending a Batch Request Make a POST request using the following pattern. You can pass `external_reference` as a unique ID per-request that will be returned in the response. Sending a batch request ``` curl "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai/run/@cf/baai/bge-m3?queueRequest=true" \ --header "Authorization: Bearer $API_TOKEN" \ --header 'Content-Type: application/json' \ --json '{ "requests": [ { "query": "This is a story about Cloudflare", "contexts": [ { "text": "This is a story about an orange cloud" }, { "text": "This is a story about a llama" }, { "text": "This is a story about a hugging emoji" } ], "external_reference": "reference-1" } ] }' ``` ``` { "result": { "status": "queued", "request_id": "768f15b7-4fd6-4498-906e-ad94ffc7f8d2", "model": "@cf/baai/bge-m3" }, "success": true, "errors": [], "messages": [] } ``` ## 2\. Retrieving the Batch Response After receiving a `request_id` from your initial POST, you can poll for or retrieve the results with another POST request: Retrieving a response ``` curl "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai/run/@cf/baai/bge-m3?queueRequest=true" \ --header "Authorization: Bearer $API_TOKEN" \ --header 'Content-Type: application/json' \ --json '{ "request_id": "" }' ``` ``` { "result": { "responses": [ { "id": 0, "result": { "response": [ { "id": 0, "score": 0.73974609375 }, { "id": 1, "score": 0.642578125 }, { "id": 2, "score": 0.6220703125 } ] }, "success": true, "external_reference": "reference-1" } ], "usage": { "prompt_tokens": 12, "completion_tokens": 0, "total_tokens": 12 } }, "success": true, "errors": [], "messages": [] } ``` ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/features/","name":"Features"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/features/batch-api/","name":"Asynchronous Batch API"}},{"@type":"ListItem","position":5,"item":{"@id":"/workers-ai/features/batch-api/rest-api/","name":"REST API"}}]} ``` --- --- title: Workers Binding description: You can use Workers Bindings to interact with the Batch API. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/features/batch-api/workers-binding.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Workers Binding You can use Workers Bindings to interact with the Batch API. ## Send a Batch request Send your initial batch inference request by composing a JSON payload containing an array of individual inference requests and the `queueRequest: true` property (which is what controlls queueing behavior). Note Ensure that the total payload is under 10 MB. src/index.ts ``` export interface Env { AI: Ai; } export default { async fetch(request, env): Promise { const embeddings = await env.AI.run( "@cf/baai/bge-m3", { requests: [ { query: "This is a story about Cloudflare", contexts: [ { text: "This is a story about an orange cloud", }, { text: "This is a story about a llama", }, { text: "This is a story about a hugging emoji", }, ], }, ], }, { queueRequest: true }, ); return Response.json(embeddings); }, } satisfies ExportedHandler; ``` ``` { "status": "queued", "model": "@cf/baai/bge-m3", "request_id": "000-000-000" } ``` You will get a response with the following values: * **`status`**: Indicates that your request is queued. * **`request_id`**: A unique identifier for the batch request. * **`model`**: The model used for the batch inference. Of these, the `request_id` is important for when you need to [poll the batch status](#poll-batch-status). ### Poll batch status Once your batch request is queued, use the `request_id` to poll for its status. During processing, the API returns a status `queued` or `running` indicating that the request is still in the queue or being processed. src/index.ts ``` export interface Env { AI: Ai; } export default { async fetch(request, env): Promise { const status = await env.AI.run("@cf/baai/bge-m3", { request_id: "000-000-000", }); return Response.json(status); }, } satisfies ExportedHandler; ``` ``` { "responses": [ { "id": 0, "result": { "response": [ { "id": 0, "score": 0.73974609375 }, { "id": 1, "score": 0.642578125 }, { "id": 2, "score": 0.6220703125 } ] }, "success": true, "external_reference": "reference-1" } ], "usage": { "prompt_tokens": 12, "completion_tokens": 0, "total_tokens": 12 } } ``` When the inference is complete, the API returns a final HTTP status code of `200` along with an array of responses. Each response object corresponds to an individual input prompt, identified by an `id` that maps to the index of the prompt in your original request. ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/features/","name":"Features"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/features/batch-api/","name":"Asynchronous Batch API"}},{"@type":"ListItem","position":5,"item":{"@id":"/workers-ai/features/batch-api/workers-binding/","name":"Workers Binding"}}]} ``` --- --- title: Fine-tunes description: Learn how to use Workers AI to get fine-tuned inference. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/features/fine-tunes/index.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Fine-tunes Learn how to use Workers AI to get fine-tuned inference. ### Fine-tuned inference with LoRAs Upload a LoRA adapter and run fine-tuned inference with one of our base models. [ Run inference with LoRAs ](https://developers.cloudflare.com/workers-ai/features/fine-tunes/loras/) --- ## What is fine-tuning? Fine-tuning is a general term for modifying an AI model by continuing to train it with additional data. The goal of fine-tuning is to increase the probability that a generation is similar to your dataset. Training a model from scratch is not practical for many use cases given how expensive and time consuming they can be to train. By fine-tuning an existing pre-trained model, you benefit from its capabilities while also accomplishing your desired task. [Low-Rank Adaptation ↗](https://arxiv.org/abs/2106.09685) (LoRA) is a specific fine-tuning method that can be applied to various model architectures, not just LLMs. It is common that the pre-trained model weights are directly modified or fused with additional fine-tune weights in traditional fine-tuning methods. LoRA, on the other hand, allows for the fine-tune weights and pre-trained model to remain separate, and for the pre-trained model to remain unchanged. The end result is that you can train models to be more accurate at specific tasks, such as generating code, having a specific personality, or generating images in a specific style. ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/features/","name":"Features"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/features/fine-tunes/","name":"Fine-tunes"}}]} ``` --- --- title: Using LoRA adapters description: Upload and use LoRA adapters to get fine-tuned inference on Workers AI. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/features/fine-tunes/loras.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Using LoRA adapters Workers AI supports fine-tuned inference with adapters trained with [Low-Rank Adaptation ↗](https://blog.cloudflare.com/fine-tuned-inference-with-loras). This feature is in open beta and free during this period. ## Limitations * We only support LoRAs for a [variety of models](https://developers.cloudflare.com/workers-ai/models/?capabilities=LoRA) (must not be quantized) * Adapter must be trained with rank `r <=8` as well as larger ranks if up to 32\. You can check the rank of a pre-trained LoRA adapter through the adapter's `config.json` file * LoRA adapter file must be < 300MB * LoRA adapter files must be named `adapter_config.json` and `adapter_model.safetensors` exactly * You can test up to 100 LoRA adapters per account --- ## Choosing compatible LoRA adapters ### Finding open-source LoRA adapters We have started a [Hugging Face Collection ↗](https://huggingface.co/collections/Cloudflare/workers-ai-compatible-loras-6608dd9f8d305a46e355746e) that lists a few LoRA adapters that are compatible with Workers AI. Generally, any LoRA adapter that fits our limitations above should work. ### Training your own LoRA adapters To train your own LoRA adapter, follow the [tutorial](https://developers.cloudflare.com/workers-ai/guides/tutorials/fine-tune-models-with-autotrain/). --- ## Uploading LoRA adapters In order to run inference with LoRAs on Workers AI, you'll need to create a new fine tune on your account and upload your adapter files. You should have a `adapter_model.safetensors` file with model weights and `adapter_config.json` with your config information. _Note that we only accept adapter files in these types._ Right now, you can't edit a fine tune's asset files after you upload it. We will support this soon, but for now you will need to create a new fine tune and upload files again if you would like to use a new LoRA. Before you upload your LoRA adapter, you'll need to edit your `adapter_config.json` file to include `model_type` as one of `mistral`, `gemma` or `llama` like below. ``` { "alpha_pattern": {}, "auto_mapping": null, ... "target_modules": [ "q_proj", "v_proj" ], "task_type": "CAUSAL_LM", "model_type": "mistral", } ``` ### Wrangler You can create a finetune and upload your LoRA adapter via wrangler with the following commands: wrangler CLI ``` npx wrangler ai finetune create #🌀 Creating new finetune "test-lora" for model "@cf/mistral/mistral-7b-instruct-v0.2-lora"... #🌀 Uploading file "/Users/abcd/Downloads/adapter_config.json" to "test-lora"... #🌀 Uploading file "/Users/abcd/Downloads/adapter_model.safetensors" to "test-lora"... #✅ Assets uploaded, finetune "test-lora" is ready to use. npx wrangler ai finetune list ┌──────────────────────────────────────┬─────────────────┬─────────────┐ │ finetune_id │ name │ description │ ├──────────────────────────────────────┼─────────────────┼─────────────┤ │ 00000000-0000-0000-0000-000000000000 │ test-lora │ │ └──────────────────────────────────────┴─────────────────┴─────────────┘ ``` ### REST API Alternatively, you can use our REST API to create a finetune and upload your adapter files. You will need a Cloudflare API Token with `Workers AI: Edit` permissions to make calls to our REST API, which you can generate via the Cloudflare Dashboard. #### Creating a fine-tune on your account Required API token permissions At least one of the following [token permissions](https://developers.cloudflare.com/fundamentals/api/reference/permissions/)is required: * `Workers AI Write` Create a new Finetune ``` curl "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai/finetunes" \ --request POST \ --header "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \ --json '{ "model": "SUPPORTED_MODEL_NAME", "name": "FINETUNE_NAME", "description": "OPTIONAL_DESCRIPTION" }' ``` #### Uploading your adapter weights and config You have to call the upload endpoint each time you want to upload a new file, so you usually run this once for `adapter_model.safetensors` and once for `adapter_config.json`. Make sure you include the `@` before your path to files. You can either use the finetune `name` or `id` that you used when you created the fine tune. cURL ``` ## Input: finetune_id, adapter_model.safetensors, then adapter_config.json ## Output: success true/false curl -X POST https://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/ai/finetunes/{FINETUNE_ID}/finetune-assets/ \ -H 'Authorization: Bearer {API_TOKEN}' \ -H 'Content-Type: multipart/form-data' \ -F 'file_name=adapter_model.safetensors' \ -F 'file=@{PATH/TO/adapter_model.safetensors}' ``` #### List fine-tunes in your account You can call this method to confirm what fine-tunes you have created in your account Required API token permissions At least one of the following [token permissions](https://developers.cloudflare.com/fundamentals/api/reference/permissions/)is required: * `Workers AI Write` * `Workers AI Read` List Finetunes ``` curl "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai/finetunes" \ --request GET \ --header "Authorization: Bearer $CLOUDFLARE_API_TOKEN" ``` ``` { "success": true, "result": [ [ { "id": "00000000-0000-0000-0000-000000000", "model": "@cf/meta-llama/llama-2-7b-chat-hf-lora", "name": "llama2-finetune", "description": "test" }, { "id": "00000000-0000-0000-0000-000000000", "model": "@cf/mistralai/mistral-7b-instruct-v0.2-lora", "name": "mistral-finetune", "description": "test" } ] ] } ``` --- ## Running inference with LoRAs To make inference requests and apply the LoRA adapter, you will need your model and finetune `name` or `id`. You should use the chat template that your LoRA was trained on, but you can try running it with `raw: true` and the messages template like below. * [ workers ai sdk ](#tab-panel-6923) * [ rest api ](#tab-panel-6924) JavaScript ``` const response = await env.AI.run( "@cf/mistralai/mistral-7b-instruct-v0.2-lora", //the model supporting LoRAs { messages: [{ role: "user", content: "Hello world" }], raw: true, //skip applying the default chat template lora: "00000000-0000-0000-0000-000000000", //the finetune id OR name }, ); ``` Terminal window ``` curl https://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/ai/run/@cf/mistral/mistral-7b-instruct-v0.2-lora \ -H 'Authorization: Bearer {API_TOKEN}' \ -d '{ "messages": [{"role": "user", "content": "Hello world"}], "raw": "true", "lora": "00000000-0000-0000-0000-000000000" }' ``` ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/features/","name":"Features"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/features/fine-tunes/","name":"Fine-tunes"}},{"@type":"ListItem","position":5,"item":{"@id":"/workers-ai/features/fine-tunes/loras/","name":"Using LoRA adapters"}}]} ``` --- --- title: Public LoRA adapters description: Cloudflare offers a few public LoRA adapters that are immediately ready for use. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/features/fine-tunes/public-loras.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Public LoRA adapters Cloudflare offers a few public LoRA adapters that can immediately be used for fine-tuned inference. You can try them out immediately via our [playground ↗](https://playground.ai.cloudflare.com). Public LoRAs will have the name `cf-public-x`, and the prefix will be reserved for Cloudflare. Note Have more LoRAs you would like to see? Let us know on [Discord ↗](https://discord.cloudflare.com). | Name | Description | Compatible with | | ---------------------------------------------------------------------------- | ---------------------------------- | ------------------------------------------------------------------------- | | [cf-public-magicoder ↗](https://huggingface.co/predibase/magicoder) | Coding tasks in multiple languages | @cf/mistral/mistral-7b-instruct-v0.1 @hf/mistral/mistral-7b-instruct-v0.2 | | [cf-public-jigsaw-classification ↗](https://huggingface.co/predibase/jigsaw) | Toxic comment classification | @cf/mistral/mistral-7b-instruct-v0.1 @hf/mistral/mistral-7b-instruct-v0.2 | | [cf-public-cnn-summarization ↗](https://huggingface.co/predibase/cnn) | Article summarization | @cf/mistral/mistral-7b-instruct-v0.1 @hf/mistral/mistral-7b-instruct-v0.2 | You can also list these public LoRAs with an API call: Required API token permissions At least one of the following [token permissions](https://developers.cloudflare.com/fundamentals/api/reference/permissions/)is required: * `Workers AI Write` * `Workers AI Read` List Public Finetunes ``` curl "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/ai/finetunes/public" \ --request GET \ --header "Authorization: Bearer $CLOUDFLARE_API_TOKEN" ``` ## Running inference with public LoRAs To run inference with public LoRAs, you just need to define the LoRA name in the request. We recommend that you use the prompt template that the LoRA was trained on. You can find this in the HuggingFace repos linked above for each adapter. ### cURL Terminal window ``` curl https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/run/@cf/mistral/mistral-7b-instruct-v0.1 \ --header 'Authorization: Bearer {cf_token}' \ --data '{ "messages": [ { "role": "user", "content": "Write a python program to check if a number is even or odd." } ], "lora": "cf-public-magicoder" }' ``` ### JavaScript JavaScript ``` const answer = await env.AI.run("@cf/mistral/mistral-7b-instruct-v0.1", { stream: true, raw: true, messages: [ { role: "user", content: "Summarize the following: Some newspapers, TV channels and well-known companies publish false news stories to fool people on 1 April. One of the earliest examples of this was in 1957 when a programme on the BBC, the UKs national TV channel, broadcast a report on how spaghetti grew on trees. The film showed a family in Switzerland collecting spaghetti from trees and many people were fooled into believing it, as in the 1950s British people didnt eat much pasta and many didnt know how it was made! Most British people wouldnt fall for the spaghetti trick today, but in 2008 the BBC managed to fool their audience again with their Miracles of Evolution trailer, which appeared to show some special penguins that had regained the ability to fly. Two major UK newspapers, The Daily Telegraph and the Daily Mirror, published the important story on their front pages.", }, ], lora: "cf-public-cnn-summarization", }); ``` ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/features/","name":"Features"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/features/fine-tunes/","name":"Fine-tunes"}},{"@type":"ListItem","position":5,"item":{"@id":"/workers-ai/features/fine-tunes/public-loras/","name":"Public LoRA adapters"}}]} ``` --- --- title: Function calling description: Function calling enables people to take Large Language Models (LLMs) and use the model response to execute functions or interact with external APIs. The developer usually defines a set of functions and the required input schema for each function, which we call tools. The model then intelligently understands when it needs to do a tool call, and it returns a JSON output which the user needs to feed to another function or API. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) ### Tags [ LLM ](https://developers.cloudflare.com/search/?tags=LLM) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/features/function-calling/index.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Function calling Function calling enables people to take Large Language Models (LLMs) and use the model response to execute functions or interact with external APIs. The developer usually defines a set of functions and the required input schema for each function, which we call `tools`. The model then intelligently understands when it needs to do a tool call, and it returns a JSON output which the user needs to feed to another function or API. In essence, function calling allows you to perform actions with LLMs by executing code or making additional API calls. ## How can I use function calling? Workers AI has [embedded function calling](https://developers.cloudflare.com/workers-ai/features/function-calling/embedded/) which allows you to execute function code alongside your inference calls. We have a package called [@cloudflare/ai-utils ↗](https://www.npmjs.com/package/@cloudflare/ai-utils) to help facilitate this, which we have open-sourced on [Github ↗](https://github.com/cloudflare/ai-utils). For industry-standard function calling, take a look at the documentation on [Traditional Function Calling](https://developers.cloudflare.com/workers-ai/features/function-calling/traditional/). To show you the value of embedded function calling, take a look at the example below that compares traditional function calling with embedded function calling. Embedded function calling allowed us to cut down the lines of code from 77 to 31. * [ Embedded ](#tab-panel-6925) * [ Traditional ](#tab-panel-6926) Terminal window ``` # The ai-utils package enables embedded function calling npm i @cloudflare/ai-utils ``` Embedded function calling example ``` import { createToolsFromOpenAPISpec, runWithTools, autoTrimTools, } from "@cloudflare/ai-utils"; export default { async fetch(request, env, ctx) { const response = await runWithTools( env.AI, "@hf/nousresearch/hermes-2-pro-mistral-7b", { messages: [{ role: "user", content: "Who is Cloudflare on github?" }], tools: [ // You can pass the OpenAPI spec link or contents directly ...(await createToolsFromOpenAPISpec( "https://gist.githubusercontent.com/mchenco/fd8f20c8f06d50af40b94b0671273dc1/raw/f9d4b5cd5944cc32d6b34cad0406d96fd3acaca6/partial_api.github.com.json", { overrides: [ { // for all requests on *.github.com, we'll need to add a User-Agent. matcher: ({ url, method }) => { return url.hostname === "api.github.com"; }, values: { headers: { "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36", }, }, }, ], }, )), ], }, ).then((response) => { return response; }); return new Response(JSON.stringify(response)); }, }; ``` Traditional function calling example ``` export default { async fetch(request, env, ctx) { const response = await env.AI.run( "@hf/nousresearch/hermes-2-pro-mistral-7b", { messages: [{ role: "user", content: "Who is Cloudflare on GitHub?" }], tools: [ { name: "getGithubUser", description: "Provides publicly available information about someone with a GitHub account.", parameters: { type: "object", properties: { username: { type: "string", description: "The handle for the GitHub user account.", }, }, required: ["username"], }, }, ], }, ); const selected_tool = response.tool_calls[0]; let res; if (selected_tool.name == "getGithubUser") { try { const username = selected_tool.arguments.username; const url = `https://api.github.com/users/${username}`; res = await fetch(url, { headers: { // Github API requires a User-Agent header "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36", }, }).then((res) => res.json()); } catch (error) { return error; } } const finalResponse = await env.AI.run( "@hf/nousresearch/hermes-2-pro-mistral-7b", { messages: [ { role: "user", content: "Who is Cloudflare on GitHub?", }, { role: "assistant", content: JSON.stringify(selected_tool), }, { role: "tool", content: JSON.stringify(res), }, ], tools: [ { name: "getGithubUser", description: "Provides publicly available information about someone with a GitHub account.", parameters: { type: "object", properties: { username: { type: "string", description: "The handle for the GitHub user account.", }, }, required: ["username"], }, }, ], }, ); return new Response(JSON.stringify(finalResponse)); }, }; ``` ## What models support function calling? There are open-source models which have been fine-tuned to do function calling. When browsing our [model catalog](https://developers.cloudflare.com/workers-ai/models/), look for models with the function calling property beside it. For example, [@hf/nousresearch/hermes-2-pro-mistral-7b](https://developers.cloudflare.com/workers-ai/models/hermes-2-pro-mistral-7b/) is a fine-tuned variant of Mistral 7B that you can use for function calling. ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/features/","name":"Features"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/features/function-calling/","name":"Function calling"}}]} ``` --- --- title: Embedded description: Cloudflare has a unique embedded function calling feature that allows you to execute function code alongside your tool call inference. Our npm package @cloudflare/ai-utils is the developer toolkit to get started. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/features/function-calling/embedded/index.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Embedded Cloudflare has a unique [embedded function calling ↗](https://blog.cloudflare.com/embedded-function-calling) feature that allows you to execute function code alongside your tool call inference. Our npm package [@cloudflare/ai-utils ↗](https://www.npmjs.com/package/@cloudflare/ai-utils) is the developer toolkit to get started. Embedded function calling can be used to easily make complex agents that interact with websites and APIs, like using natural language to create meetings on Google Calendar, saving data to Notion, automatically routing requests to other APIs, saving data to an R2 bucket - or all of this at the same time. All you need is a prompt and an OpenAPI spec to get started. REST API support Embedded function calling depends on features native to the Workers platform. This means that embedded function calling is only supported via [Cloudflare Workers](https://developers.cloudflare.com/workers-ai/get-started/workers-wrangler/), not via the [REST API](https://developers.cloudflare.com/workers-ai/get-started/rest-api/). ## Resources * [ Get Started ](https://developers.cloudflare.com/workers-ai/features/function-calling/embedded/get-started/) * [ Examples ](https://developers.cloudflare.com/workers-ai/features/function-calling/embedded/examples/) * [ API Reference ](https://developers.cloudflare.com/workers-ai/features/function-calling/embedded/api-reference/) * [ Troubleshooting ](https://developers.cloudflare.com/workers-ai/features/function-calling/embedded/troubleshooting/) ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/features/","name":"Features"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/features/function-calling/","name":"Function calling"}},{"@type":"ListItem","position":5,"item":{"@id":"/workers-ai/features/function-calling/embedded/","name":"Embedded"}}]} ``` --- --- title: API Reference description: Learn more about the API reference for embedded function calling. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/features/function-calling/embedded/api-reference.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # API Reference Learn more about the API reference for [embedded function calling](https://developers.cloudflare.com/workers-ai/features/function-calling/embedded). ## runWithTools This wrapper method enables you to do embedded function calling. You pass it the AI binding, model, inputs (`messages` array and `tools` array), and optional configurations. * `AI Binding`Ai * The AI binding, such as `env.AI`. * `model`BaseAiTextGenerationModels * The ID of the model that supports function calling. For example, `@hf/nousresearch/hermes-2-pro-mistral-7b`. * `input`Object * `messages`RoleScopedChatInput\[\] * `tools`AiTextGenerationToolInputWithFunction\[\] * `config`Object * `streamFinalResponse`boolean optional * `maxRecursiveToolRuns`number optional * `strictValidation`boolean optional * `verbose`boolean optional * `trimFunction`boolean optional - For the `trimFunction`, you can pass it `autoTrimTools`, which is another helper method we've devised to automatically choose the correct tools (using an LLM) before sending it off for inference. This means that your final inference call will have fewer input tokens. ## createToolsFromOpenAPISpec This method lets you automatically create tool schemas based on OpenAPI specs, so you don't have to manually write or hardcode the tool schemas. You can pass the OpenAPI spec for any API in JSON or YAML format. `createToolsFromOpenAPISpec` has a config input that allows you to perform overrides if you need to provide headers like Authentication or User-Agent. * `spec`string * The OpenAPI specification in either JSON or YAML format, or a URL to a remote OpenAPI specification. * `config`Config optional - Configuration options for the createToolsFromOpenAPISpec function * `overrides`ConfigRule\[\] optional * `matchPatterns`RegExp\[\] optional * `options` Object optional {`verbose` boolean optional } ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/features/","name":"Features"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/features/function-calling/","name":"Function calling"}},{"@type":"ListItem","position":5,"item":{"@id":"/workers-ai/features/function-calling/embedded/","name":"Embedded"}},{"@type":"ListItem","position":6,"item":{"@id":"/workers-ai/features/function-calling/embedded/api-reference/","name":"API Reference"}}]} ``` --- --- title: Use fetch() handler description: Learn how to use the fetch() handler in Cloudflare Workers AI to enable LLMs to perform API calls, like retrieving a 5-day weather forecast using function calling. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) ### Tags [ AI ](https://developers.cloudflare.com/search/?tags=AI) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/features/function-calling/embedded/examples/fetch.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Use fetch() handler **Last reviewed:** over 1 year ago A very common use case is to provide the LLM with the ability to perform API calls via function calling. In this example the LLM will retrieve the weather forecast for the next 5 days. To do so a `getWeather` function is defined that is passed to the LLM as tool. The `getWeather`function extracts the user's location from the request and calls the external weather API via the Workers' [Fetch API](https://developers.cloudflare.com/workers/runtime-apis/fetch/) and returns the result. Embedded function calling example with fetch() ``` import { runWithTools } from '@cloudflare/ai-utils'; type Env = { AI: Ai; }; export default { async fetch(request, env, ctx) { // Define function const getWeather = async (args: { numDays: number }) => { const { numDays } = args; // Location is extracted from request based on // https://developers.cloudflare.com/workers/runtime-apis/request/#incomingrequestcfproperties const lat = request.cf?.latitude const long = request.cf?.longitude // Interpolate values for external API call const response = await fetch( `https://api.open-meteo.com/v1/forecast?latitude=${lat}&longitude=${long}&daily=temperature_2m_max,precipitation_sum&timezone=GMT&forecast_days=${numDays}` ); return response.text(); }; // Run AI inference with function calling const response = await runWithTools( env.AI, // Model with function calling support '@hf/nousresearch/hermes-2-pro-mistral-7b', { // Messages messages: [ { role: 'user', content: 'What the weather like the next 5 days? Respond as text', }, ], // Definition of available tools the AI model can leverage tools: [ { name: 'getWeather', description: 'Get the weather for the next [numDays] days', parameters: { type: 'object', properties: { numDays: { type: 'numDays', description: 'number of days for the weather forecast' }, }, required: ['numDays'], }, // reference to previously defined function function: getWeather, }, ], } ); return new Response(JSON.stringify(response)); }, } satisfies ExportedHandler; ``` ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/features/","name":"Features"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/features/function-calling/","name":"Function calling"}},{"@type":"ListItem","position":5,"item":{"@id":"/workers-ai/features/function-calling/embedded/","name":"Embedded"}},{"@type":"ListItem","position":6,"item":{"@id":"/workers-ai/features/function-calling/embedded/examples/","name":"Examples"}},{"@type":"ListItem","position":7,"item":{"@id":"/workers-ai/features/function-calling/embedded/examples/fetch/","name":"Use fetch() handler"}}]} ``` --- --- title: Use KV API description: Learn how to use Cloudflare Workers AI to interact with KV storage, enabling persistent data handling with embedded function calling in a few lines of code. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) ### Tags [ AI ](https://developers.cloudflare.com/search/?tags=AI) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/features/function-calling/embedded/examples/kv.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Use KV API Interact with persistent storage to retrieve or store information enables for powerful use cases. In this example we show how embedded function calling can interact with other resources on the Cloudflare Developer Platform with a few lines of code. ## Pre-Requisites For this example to work, you need to provision a [KV](https://developers.cloudflare.com/kv/) namespace first. To do so, follow the [KV - Get started ](https://developers.cloudflare.com/kv/get-started/) guide. Importantly, your Wrangler file must be updated to include the `KV` binding definition to your respective namespace. ## Worker code Embedded function calling example with KV API ``` import { runWithTools } from "@cloudflare/ai-utils"; type Env = { AI: Ai; KV: KVNamespace; }; export default { async fetch(request, env, ctx) { // Define function const updateKvValue = async ({ key, value, }: { key: string; value: string; }) => { const response = await env.KV.put(key, value); return `Successfully updated key-value pair in database: ${response}`; }; // Run AI inference with function calling const response = await runWithTools( env.AI, "@hf/nousresearch/hermes-2-pro-mistral-7b", { messages: [ { role: "system", content: "Put user given values in KV" }, { role: "user", content: "Set the value of banana to yellow." }, ], tools: [ { name: "KV update", description: "Update a key-value pair in the database", parameters: { type: "object", properties: { key: { type: "string", description: "The key to update", }, value: { type: "string", description: "The value to update", }, }, required: ["key", "value"], }, function: updateKvValue, }, ], }, ); return new Response(JSON.stringify(response)); }, } satisfies ExportedHandler; ``` ## Verify results To verify the results, run the following command Terminal window ``` npx wrangler kv key get banana --binding KV --local ``` ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/features/","name":"Features"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/features/function-calling/","name":"Function calling"}},{"@type":"ListItem","position":5,"item":{"@id":"/workers-ai/features/function-calling/embedded/","name":"Embedded"}},{"@type":"ListItem","position":6,"item":{"@id":"/workers-ai/features/function-calling/embedded/examples/","name":"Examples"}},{"@type":"ListItem","position":7,"item":{"@id":"/workers-ai/features/function-calling/embedded/examples/kv/","name":"Use KV API"}}]} ``` --- --- title: Tools based on OpenAPI Spec description: Oftentimes APIs are defined and documented via OpenAPI specification. The Cloudflare ai-utils package's createToolsFromOpenAPISpec function creates tools from the OpenAPI spec, which the LLM can then leverage to fulfill the prompt. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) ### Tags [ AI ](https://developers.cloudflare.com/search/?tags=AI) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/features/function-calling/embedded/examples/openapi.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Tools based on OpenAPI Spec Oftentimes APIs are defined and documented via [OpenAPI specification ↗](https://swagger.io/specification/). The Cloudflare `ai-utils` package's `createToolsFromOpenAPISpec` function creates tools from the OpenAPI spec, which the LLM can then leverage to fulfill the prompt. In this example the LLM will describe the a Github user, based Github's API and its OpenAPI spec. Embedded function calling example from OpenAPI Spec ``` import { createToolsFromOpenAPISpec, runWithTools } from '@cloudflare/ai-utils'; type Env = { AI: Ai; }; const APP_NAME = 'cf-fn-calling-example-app'; export default { async fetch(request, env, ctx) { const toolsFromOpenAPISpec = [ // You can pass the OpenAPI spec link or contents directly ...(await createToolsFromOpenAPISpec( 'https://gist.githubusercontent.com/mchenco/fd8f20c8f06d50af40b94b0671273dc1/raw/f9d4b5cd5944cc32d6b34cad0406d96fd3acaca6/partial_api.github.com.json', { overrides: [ { matcher: ({ url }) => { return url.hostname === 'api.github.com'; }, // for all requests on *.github.com, we'll need to add a User-Agent. values: { headers: { 'User-Agent': APP_NAME, }, }, }, ], } )), ]; const response = await runWithTools( env.AI, '@hf/nousresearch/hermes-2-pro-mistral-7b', { messages: [ { role: 'user', content: 'Who is cloudflare on Github and how many repos does the organization have?', }, ], tools: toolsFromOpenAPISpec, } ); return new Response(JSON.stringify(response)); }, } satisfies ExportedHandler; ``` ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/features/","name":"Features"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/features/function-calling/","name":"Function calling"}},{"@type":"ListItem","position":5,"item":{"@id":"/workers-ai/features/function-calling/embedded/","name":"Embedded"}},{"@type":"ListItem","position":6,"item":{"@id":"/workers-ai/features/function-calling/embedded/examples/","name":"Examples"}},{"@type":"ListItem","position":7,"item":{"@id":"/workers-ai/features/function-calling/embedded/examples/openapi/","name":"Tools based on OpenAPI Spec"}}]} ``` --- --- title: Get Started description: This guide will instruct you through setting up and deploying your first Workers AI project with embedded function calling. You will use Workers, a Workers AI binding, the ai-utils package, and a large language model (LLM) to deploy your first AI-powered application on the Cloudflare global network with embedded function calling. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/features/function-calling/embedded/get-started.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Get Started This guide will instruct you through setting up and deploying your first Workers AI project with embedded function calling. You will use Workers, a Workers AI binding, the [ai-utils package ↗](https://github.com/cloudflare/ai-utils), and a large language model (LLM) to deploy your first AI-powered application on the Cloudflare global network with embedded function calling. ## 1\. Create a Worker project with Workers AI Follow the [Workers AI Get Started Guide](https://developers.cloudflare.com/workers-ai/get-started/workers-wrangler/) until step 2. ## 2\. Install additional npm package Next, run the following command in your project repository to install the Worker AI utilities package. npm yarn pnpm bun ``` npm i @cloudflare/ai-utils ``` ``` yarn add @cloudflare/ai-utils ``` ``` pnpm add @cloudflare/ai-utils ``` ``` bun add @cloudflare/ai-utils ``` ## 3\. Add Workers AI Embedded function calling Update the `index.ts` file in your application directory with the following code: * [ JavaScript ](#tab-panel-6927) * [ TypeScript ](#tab-panel-6928) index.js ``` import { runWithTools } from "@cloudflare/ai-utils"; export default { async fetch(request, env, ctx) { // Define function const sum = (args) => { const { a, b } = args; return Promise.resolve((a + b).toString()); }; // Run AI inference with function calling const response = await runWithTools( env.AI, // Model with function calling support "@hf/nousresearch/hermes-2-pro-mistral-7b", { // Messages messages: [ { role: "user", content: "What the result of 123123123 + 10343030?", }, ], // Definition of available tools the AI model can leverage tools: [ { name: "sum", description: "Sum up two numbers and returns the result", parameters: { type: "object", properties: { a: { type: "number", description: "the first number" }, b: { type: "number", description: "the second number" }, }, required: ["a", "b"], }, // reference to previously defined function function: sum, }, ], }, ); return new Response(JSON.stringify(response)); }, }; ``` index.ts ``` import { runWithTools } from "@cloudflare/ai-utils"; type Env = { AI: Ai; }; export default { async fetch(request, env, ctx) { // Define function const sum = (args: { a: number; b: number }): Promise => { const { a, b } = args; return Promise.resolve((a + b).toString()); }; // Run AI inference with function calling const response = await runWithTools( env.AI, // Model with function calling support "@hf/nousresearch/hermes-2-pro-mistral-7b", { // Messages messages: [ { role: "user", content: "What the result of 123123123 + 10343030?", }, ], // Definition of available tools the AI model can leverage tools: [ { name: "sum", description: "Sum up two numbers and returns the result", parameters: { type: "object", properties: { a: { type: "number", description: "the first number" }, b: { type: "number", description: "the second number" }, }, required: ["a", "b"], }, // reference to previously defined function function: sum, }, ], }, ); return new Response(JSON.stringify(response)); }, } satisfies ExportedHandler; ``` This example imports the utils with `import { runWithTools} from "@cloudflare/ai-utils"` and follows the API reference below. Moreover, in this example we define and describe a list of tools that the LLM can leverage to respond to the user query. Here, the list contains of only one tool, the `sum` function. Abstracted by the `runWithTools` function, the following steps occur: sequenceDiagram participant Worker as Worker participant WorkersAI as Workers AI Worker->>+WorkersAI: Send messages, function calling prompt, and available tools WorkersAI->>+Worker: Select tools and arguments for function calling Worker-->>-Worker: Execute function Worker-->>+WorkersAI: Send messages, function calling prompt and function result WorkersAI-->>-Worker: Send response incorporating function output The `ai-utils package` is also open-sourced on [Github ↗](https://github.com/cloudflare/ai-utils). ## 4\. Local development & deployment Follow steps 4 and 5 of the [Workers AI Get Started Guide](https://developers.cloudflare.com/workers-ai/get-started/workers-wrangler/) for local development and deployment. Workers AI Embedded Function Calling charges Embedded function calling runs Workers AI inference requests. Standard charges for inference (e.g. tokens) usage will be charged. Resources consumed (e.g. CPU time) during embedded functions' code execution will be charged just as any other Worker's code execution. ## API reference For more details, refer to [API reference](https://developers.cloudflare.com/workers-ai/features/function-calling/embedded/api-reference/). ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/features/","name":"Features"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/features/function-calling/","name":"Function calling"}},{"@type":"ListItem","position":5,"item":{"@id":"/workers-ai/features/function-calling/embedded/","name":"Embedded"}},{"@type":"ListItem","position":6,"item":{"@id":"/workers-ai/features/function-calling/embedded/get-started/","name":"Get Started"}}]} ``` --- --- title: Troubleshooting description: This section will describe tools for troubleshooting and address common errors. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/features/function-calling/embedded/troubleshooting.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Troubleshooting This section will describe tools for troubleshooting and address common errors. ## Logging General [logging](https://developers.cloudflare.com/workers/observability/logs/) capabilities for Workers also apply to embedded function calling. ### Function invocations The invocations of tools can be logged as in any Worker using `console.log()`: Logging tool invocations ``` export default { async fetch(request, env, ctx) { const sum = (args: { a: number; b: number }): Promise => { const { a, b } = args; // Logging from within embedded function invocations console.log(`The sum function has been invoked with the arguments a: ${a} and b: ${b}`) return Promise.resolve((a + b).toString()); }; ... } } ``` ### Logging within `runWithTools` The `runWithTools` function has a `verbose` mode that emits helpful logs for debugging of function calls as well input and output statistics. Enabled verbose mode ``` const response = await runWithTools( env.AI, '@hf/nousresearch/hermes-2-pro-mistral-7b', { messages: [ ... ], tools: [ ... ], }, // Enable verbose mode { verbose: true } ); ``` ## Performance To respond to a LLM prompt with embedded function, potentially multiple AI inference requests and function invocations are needed, which can have an impact on user experience. Consider the following to improve performance: * Shorten prompts (to reduce time for input processing) * Reduce number of tools provided * Stream the final response to the end user (to minimize the time to interaction). See example below: Streamed response example ``` async fetch(request, env, ctx) { const response = (await runWithTools( env.AI, '@hf/nousresearch/hermes-2-pro-mistral-7b', { messages: [ ... ], tools: [ ... ], }, { // Enable response streaming streamFinalResponse: true, } )) as ReadableStream; // Set response headers for streaming return new Response(response, { headers: { 'content-type': 'text/event-stream', }, }); } ``` ## Common Errors If you are getting a `BadInput` error, your inputs may exceed our current context window for our models. Try reducing input tokens to resolve this error. ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/features/","name":"Features"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/features/function-calling/","name":"Function calling"}},{"@type":"ListItem","position":5,"item":{"@id":"/workers-ai/features/function-calling/embedded/","name":"Embedded"}},{"@type":"ListItem","position":6,"item":{"@id":"/workers-ai/features/function-calling/embedded/troubleshooting/","name":"Troubleshooting"}}]} ``` --- --- title: Traditional description: This page shows how you can do traditional function calling, as defined by industry standards. Workers AI also offers embedded function calling, which is drastically easier than traditional function calling. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/features/function-calling/traditional.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Traditional This page shows how you can do traditional function calling, as defined by industry standards. Workers AI also offers [embedded function calling](https://developers.cloudflare.com/workers-ai/features/function-calling/embedded/), which is drastically easier than traditional function calling. With traditional function calling, you define an array of tools with the name, description, and tool arguments. The example below shows how you would pass a tool called `getWeather` in an inference request to a model. Traditional function calling example ``` const response = await env.AI.run("@hf/nousresearch/hermes-2-pro-mistral-7b", { messages: [ { role: "user", content: "what is the weather in london?", }, ], tools: [ { name: "getWeather", description: "Return the weather for a latitude and longitude", parameters: { type: "object", properties: { latitude: { type: "string", description: "The latitude for the given location", }, longitude: { type: "string", description: "The longitude for the given location", }, }, required: ["latitude", "longitude"], }, }, ], }); return new Response(JSON.stringify(response.tool_calls)); ``` The LLM will then return a JSON object with the required arguments and the name of the tool that was called. You can then pass this JSON object to make an API call. ``` [ { "arguments": { "latitude": "51.5074", "longitude": "-0.1278" }, "name": "getWeather" } ] ``` For a working example on how to do function calling, take a look at our [demo app ↗](https://github.com/craigsdennis/lightbulb-moment-tool-calling/blob/main/src/index.ts). ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/features/","name":"Features"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/features/function-calling/","name":"Function calling"}},{"@type":"ListItem","position":5,"item":{"@id":"/workers-ai/features/function-calling/traditional/","name":"Traditional"}}]} ``` --- --- title: JSON Mode description: When we want text-generation AI models to interact with databases, services, and external systems programmatically, typically when using tool calling or building AI agents, we must have structured response formats rather than natural language. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) ### Tags [ JSON ](https://developers.cloudflare.com/search/?tags=JSON) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/features/json-mode.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # JSON Mode When we want text-generation AI models to interact with databases, services, and external systems programmatically, typically when using tool calling or building AI agents, we must have structured response formats rather than natural language. Workers AI supports JSON Mode, enabling applications to request a structured output response when interacting with AI models. ## Schema JSON Mode is compatible with OpenAI’s implementation; to enable add the `response_format` property to the request object using the following convention: ``` { response_format: { title: "JSON Mode", type: "object", properties: { type: { type: "string", enum: ["json_object", "json_schema"], }, json_schema: {}, } } } ``` Where `json_schema` must be a valid [JSON Schema ↗](https://json-schema.org/) declaration. ## JSON Mode example When using JSON Format, pass the schema as in the example below as part of the request you send to the LLM. ``` { "messages": [ { "role": "system", "content": "Extract data about a country." }, { "role": "user", "content": "Tell me about India." } ], "response_format": { "type": "json_schema", "json_schema": { "type": "object", "properties": { "name": { "type": "string" }, "capital": { "type": "string" }, "languages": { "type": "array", "items": { "type": "string" } } }, "required": [ "name", "capital", "languages" ] } } } ``` The LLM will follow the schema, and return a response such as below: ``` { "response": { "name": "India", "capital": "New Delhi", "languages": [ "Hindi", "English", "Bengali", "Telugu", "Marathi", "Tamil", "Gujarati", "Urdu", "Kannada", "Odia", "Malayalam", "Punjabi", "Sanskrit" ] } } ``` As you can see, the model is complying with the JSON schema definition in the request and responding with a validated JSON object. ## Supported Models This is the list of models that now support JSON Mode: * [@cf/meta/llama-3.1-8b-instruct-fast](https://developers.cloudflare.com/workers-ai/models/llama-3.1-8b-instruct-fast/) * [@cf/meta/llama-3.1-70b-instruct](https://developers.cloudflare.com/workers-ai/models/llama-3.1-70b-instruct/) * [@cf/meta/llama-3.3-70b-instruct-fp8-fast](https://developers.cloudflare.com/workers-ai/models/llama-3.3-70b-instruct-fp8-fast/) * [@cf/meta/llama-3-8b-instruct](https://developers.cloudflare.com/workers-ai/models/llama-3-8b-instruct/) * [@cf/meta/llama-3.1-8b-instruct](https://developers.cloudflare.com/workers-ai/models/llama-3.1-8b-instruct/) * [@cf/meta/llama-3.2-11b-vision-instruct](https://developers.cloudflare.com/workers-ai/models/llama-3.2-11b-vision-instruct/) * [@hf/nousresearch/hermes-2-pro-mistral-7b](https://developers.cloudflare.com/workers-ai/models/hermes-2-pro-mistral-7b/) * [@hf/thebloke/deepseek-coder-6.7b-instruct-awq](https://developers.cloudflare.com/workers-ai/models/deepseek-coder-6.7b-instruct-awq/) * [@cf/deepseek-ai/deepseek-r1-distill-qwen-32b](https://developers.cloudflare.com/workers-ai/models/deepseek-r1-distill-qwen-32b/) We will continue extending this list to keep up with new, and requested models. Note that Workers AI can't guarantee that the model responds according to the requested JSON Schema. Depending on the complexity of the task and adequacy of the JSON Schema, the model may not be able to satisfy the request in extreme situations. If that's the case, then an error `JSON Mode couldn't be met` is returned and must be handled. JSON Mode currently doesn't support streaming. ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/features/","name":"Features"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/features/json-mode/","name":"JSON Mode"}}]} ``` --- --- title: Markdown Conversion description: Markdown is essential for text generation and large language models (LLMs) in training and inference because it can provide structured, semantic, human, and machine-readable input. Likewise, Markdown facilitates chunking and structuring input data for better retrieval and synthesis in the context of RAGs, and its simplicity and ease of parsing and rendering make it ideal for AI Agents. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/features/markdown-conversion/index.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Markdown Conversion [Markdown ↗](https://en.wikipedia.org/wiki/Markdown) is essential for text generation and large language models (LLMs) in training and inference because it can provide structured, semantic, human, and machine-readable input. Likewise, Markdown facilitates chunking and structuring input data for better retrieval and synthesis in the context of RAGs, and its simplicity and ease of parsing and rendering make it ideal for AI Agents. For these reasons, document conversion plays an important role when designing and developing AI applications. Workers AI provides the `toMarkdown` utility method that developers can use from the [env.AI](https://developers.cloudflare.com/workers-ai/features/markdown-conversion/usage/binding/) binding or the [REST APIs](https://developers.cloudflare.com/workers-ai/features/markdown-conversion/usage/rest-api/) for quick, easy, and convenient conversion and summary of documents in multiple formats to Markdown language. ## Pricing `toMarkdown` is free for most format conversions. In some cases, like image conversion, it can use Workers AI models for object detection and summarization, which may incur additional costs if it exceeds the Workers AI free allocation limits. Refer to [what models we use](https://developers.cloudflare.com/workers-ai/features/markdown-conversion/how-it-works/) and the [Workers AI pricing page](https://developers.cloudflare.com/workers-ai/platform/pricing/) for more details. ## Other Markdown conversion features * Browser Rendering [/markdown](https://developers.cloudflare.com/browser-rendering/rest-api/markdown-endpoint/) REST API supports markdown conversion if you need to render a dynamic page or application in a real browser before converting it. * [Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/) allows real-time document conversion for Cloudflare zones using content negotiation headers. ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/features/","name":"Features"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/features/markdown-conversion/","name":"Markdown Conversion"}}]} ``` --- --- title: Conversion Options description: By default, the toMarkdown service extracts text content from your files. To further extend the capabilities of the conversion process, you can pass options to the service to control how specific file types are converted. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/features/markdown-conversion/conversion-options.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # Conversion Options By default, the `toMarkdown` service extracts text content from your files. To further extend the capabilities of the conversion process, you can pass options to the service to control how specific file types are converted. Options are organized by file type and are all optional. ## Available options ### Images TypeScript ``` { image?: { descriptionLanguage?: 'en' | 'it' | 'de' | 'es' | 'fr' | 'pt'; } } ``` * `descriptionLanguage`: controls the language of the AI-generated image descriptions. Warning This option works on a _best-effort_ basis: it is not guaranteed that the resulting text will be in the desired language. ### HTML TypeScript ``` { html?: { hostname?: string; cssSelector?: string; } } ``` * `hostname`: string to use as a host when resolving relative links inside the HTML. * `cssSelector`: string containing a CSS selector pattern to pick specific elements from your HTML. Refer to [how HTML is processed](https://developers.cloudflare.com/workers-ai/features/markdown-conversion/how-it-works/#html) for more details. ### PDF TypeScript ``` { pdf?: { metadata?: boolean; } } ``` * `metadata`: Previously, all converted PDF files always included metadata information when converted. This option allows you to opt-out of this behavior. ## Examples ### Binding To configure custom options, pass a `conversionOptions` object inside the second argument of the binding call, like this: TypeScript ``` await env.AI.toMarkdown(..., { conversionOptions: { html: { ... }, pdf: { ... }, ... } }) ``` ### REST API Since the REST API uses file uploads, the request's `Content-Type` will be `multipart/form-data`. As such, include a new form field with your stringified object as a value: Terminal window ``` curl https://api.cloudflare.com/client/v4/accounts/{ACCOUNT_ID}/ai/tomarkdown \ -X POST \ -H 'Authorization: Bearer {API_TOKEN}' \ ... -F 'conversionOptions={ "html": { ... }, ... }' ``` ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"item":{"@id":"/directory/","name":"Directory"}},{"@type":"ListItem","position":2,"item":{"@id":"/workers-ai/","name":"Workers AI"}},{"@type":"ListItem","position":3,"item":{"@id":"/workers-ai/features/","name":"Features"}},{"@type":"ListItem","position":4,"item":{"@id":"/workers-ai/features/markdown-conversion/","name":"Markdown Conversion"}},{"@type":"ListItem","position":5,"item":{"@id":"/workers-ai/features/markdown-conversion/conversion-options/","name":"Conversion Options"}}]} ``` --- --- title: How it works description: When parsing files before converting them to Markdown, there are some cleanup tasks we do depending on the type of file you are trying to convert. image: https://developers.cloudflare.com/dev-products-preview.png --- [Skip to content](#%5Ftop) Was this helpful? YesNo [ Edit page ](https://github.com/cloudflare/cloudflare-docs/edit/production/src/content/docs/workers-ai/features/markdown-conversion/how-it-works.mdx) [ Report issue ](https://github.com/cloudflare/cloudflare-docs/issues/new/choose) Copy page # How it works ## Pre-processing When parsing files before converting them to Markdown, there are some cleanup tasks we do depending on the type of file you are trying to convert. ### HTML When we detect an HTML file, a series of things happen to the HTML content before it is converted: * Some elements are ignored, including `script` and `style` tags. * Meta tags are extracted. These include `title`, `description`, `og:title`, `og:description` and `og:image`. * [JSON-LD ↗](https://json-ld.org/) content is extracted, if it exists. This will be appended at the end of the converted markdown. * The base URL to use for resolving relative links is extracted from the `` element1, if it exists, according to the spec (that is, only the first instance of the base URL is counted). * If the `cssSelector` option is: * present, then only those elements that match the selector are kept for further processing; * missing, then elements such as `
`, `