Skip to content
Cloudflare Docs
DirectoryAPISDKsChangelog

Fetch and index single web pages

This guide builds a Worker that fetches a single web page's rendered HTML with the Browser Run /content endpoint and uploads it to an AI Search instance's built-in storage using the Items API. AI Search then indexes the page so it is searchable, the same as any other uploaded document. The Worker also exposes a /search endpoint that queries the indexed pages, so one service both indexes and searches.

When to use this pattern

Use this pattern to index one page, or a small hand-picked set of pages, on demand. To crawl and continuously index an entire site, use the AI Search website data source instead.

Both Browser Run and the AI Search instance are reached through bindings, so a single Worker can fetch a page and index it without a public endpoint in between.

Prerequisites

  1. Sign up for a Cloudflare account ↗.
  2. Install Node.js ↗.

Node.js version manager

Use a Node version manager like Volta ↗ or nvm ↗ to avoid permission issues and change Node.js versions. Wrangler, discussed later in this guide, requires a Node version of 16.17.0 or later.

You also need an AI Search instance to upload to. To create one, refer to Get started. This guide uploads to the instance's built-in storage, so the instance does not need an external data source.

1. Create a Worker project

Create a new Worker project using the create-cloudflare CLI (C3). C3 ↗ is a command-line tool designed to help you set up and deploy new applications to Cloudflare.

Create a new project named fetch-and-index by running:

npm create cloudflare@latest -- fetch-and-index

For setup, select the following options:

  • For What would you like to start with?, choose Hello World example.
  • For Which template would you like to use?, choose Worker only.
  • For Which language do you want to use?, choose TypeScript.
  • For Do you want to use git for version control?, choose Yes.
  • For Do you want to deploy your application?, choose No (we will be making some changes before deploying).

Go to your application directory:

Terminal window
cd fetch-and-index

2. Configure Wrangler

Add both bindings to your Wrangler configuration file: a browser binding for Browser Run and an AI Search namespace binding for uploads. The /content endpoint runs through the browser binding, so you do not need to install Puppeteer or any other package.

JSONC
{
  "$schema": "./node_modules/wrangler/config-schema.json",
  "name": "fetch-and-index",
  "main": "src/index.ts",
  // Set this to today's date
  "compatibility_date": "2026-07-15",
  "browser": {
    "binding": "BROWSER",
    "remote": true
  },
  "ai_search_namespaces": [
    {
      "binding": "AI_SEARCH",
      "namespace": "default",
      "remote": true
    }
  ]
}

The browser binding's quickAction method requires a compatibility date of 2026-03-24 or later, and is not supported in local development without remote mode. Setting remote = true on the browser binding enables remote mode for wrangler dev. The remote option on the AI Search binding proxies uploads to your deployed instance, since AI Search does not run locally.

3. Add the Worker code

Update src/index.ts. This Worker has two routes: a request with a ?url= parameter fetches that page's rendered HTML and indexes it, and a request to /search?q= queries the indexed content. Replace my-instance with the name of your instance.

src/index.js
// The instance that indexes the fetched page.
const INSTANCE_ID = "my-instance";


// Build a stable item key that ends in .html, so AI Search converts the HTML
// to Markdown before indexing it.
function itemKey(pageUrl) {
  const slug = `${pageUrl.hostname}${pageUrl.pathname}`
    .replace(/[^a-zA-Z0-9]+/g, "-")
    .replace(/^-+|-+$/g, "");
  return `${slug || "index"}.html`;
}


export default {
  async fetch(request, env) {
    const url = new URL(request.url);


    // Search route: query the indexed content and return the matching chunks.
    if (url.pathname === "/search") {
      const query = url.searchParams.get("q");
      if (!query) {
        return new Response("Add a ?q= query parameter", { status: 400 });
      }
      const results = await env.AI_SEARCH.get(INSTANCE_ID).search({ query });
      return Response.json({
        query: results.search_query,
        results: results.chunks.map((chunk) => ({
          key: chunk.item.key,
          score: chunk.score,
          text: chunk.text,
        })),
      });
    }


    // Index route: fetch a URL's rendered HTML and index it.
    const target = url.searchParams.get("url");
    if (!target) {
      return new Response(
        "Add a ?url= parameter to index a page, or use /search?q= to search",
        { status: 400 },
      );
    }


    const pageUrl = new URL(target);


    // Fetch the fully rendered HTML with the Browser Run /content endpoint.
    // networkidle2 waits until the page has no more than two network
    // connections for at least 500 ms, giving client-side JavaScript time
    // to render the content.
    const response = await env.BROWSER.quickAction("content", {
      url: pageUrl.toString(),
      gotoOptions: {
        waitUntil: "networkidle2",
        timeout: 30000,
      },
    });


    if (!response.ok) {
      const detail = (await response.text()).slice(0, 500);
      return new Response(
        `Browser Run failed with ${response.status}: ${detail}`,
        { status: 502 },
      );
    }


    // The /content endpoint returns a JSON envelope with the rendered HTML
    // in the result field.
    const data = await response.json();


    if (!data.success || typeof data.result !== "string") {
      return new Response("Browser Run returned an unsuccessful response", {
        status: 502,
      });
    }


    const html = data.result;


    // Upload the rendered HTML to built-in storage. uploadAndPoll waits until
    // the page is indexed and searchable.
    const item = await env.AI_SEARCH.get(INSTANCE_ID).items.uploadAndPoll(
      itemKey(pageUrl),
      html,
      { timeoutMs: 60_000 },
    );


    return Response.json({ key: item.key, status: item.status });
  },
};

The .html item key tells AI Search to run the content through Markdown conversion, which strips boilerplate such as the header and footer before indexing.

4. Attach metadata for filtering

This step is optional. Because this Worker controls the upload, you can enrich each page with structured metadata, such as its title and section, and then filter searches by those fields. This is something the built-in crawler cannot do on its own.

First, define the custom metadata fields on your instance. If you are creating the instance now, pass them to create:

Terminal window
npx wrangler ai-search create my-instance --type builtin --custom-metadata title:text --custom-metadata section:text

To add fields to an existing instance, use the dashboard under Settings, or the update() binding method. An instance supports up to five custom fields, and each field can be a text, number, boolean, or datetime type. Changing the schema re-indexes existing documents.

Next, use the Browser Run /json endpoint to extract those fields from the same page. It runs through the same browser binding and returns structured JSON that matches a schema you provide. In the fetch handler from step 3, after you have the rendered html and before the upload, add:

TypeScript
// Extract structured metadata from the page with the /json endpoint.
// response_format constrains the model to the fields you defined above.
// Treat extraction as best-effort: if it fails, index the page without metadata.
const metadata: Record<string, string> = {};
try {
  const jsonResponse = await env.BROWSER.quickAction("json", {
    url: pageUrl.toString(),
    prompt: "Extract the page title and its top-level section.",
    response_format: {
      type: "json_schema",
      json_schema: {
        type: "object",
        properties: {
          title: { type: "string" },
          section: { type: "string" },
        },
        required: ["title"],
      },
    },
  });


  const extracted = (await jsonResponse.json()) as {
    result?: Record<string, unknown>;
  };


  // Metadata values must be strings, so coerce each value and drop empty ones.
  for (const [key, value] of Object.entries(extracted.result ?? {})) {
    if (value) metadata[key] = String(value);
  }
} catch {
  // Ignore extraction errors and index the page without metadata.
}

Then pass metadata in the upload options:

TypeScript
const item = await env.AI_SEARCH.get(INSTANCE_ID).items.uploadAndPoll(
  itemKey(pageUrl),
  html,
  { timeoutMs: 60_000, metadata },
);

Once indexed, you can restrict queries to pages in a given section, for example. Refer to Filtering for the query syntax.

5. Run and deploy

Start a local development server. Because remote = true is set on the browser binding, wrangler dev runs the /content endpoint in remote mode:

Terminal window
npx wrangler dev

Index a page by passing its URL:

Terminal window
curl "http://localhost:8787/?url=https://example.com/"

The response contains the item key and its status (completed once indexed):

{ "key": "example-com.html", "status": "completed" }

Then query the indexed content through the same Worker's /search endpoint:

Terminal window
curl "http://localhost:8787/search?q=what+is+this+domain+for"
{
  "query": "what is this domain for",
  "results": [
    {
      "key": "example-com.html",
      "score": 0.75,
      "text": "# Example Domain\nThis domain is for use in documentation examples..."
    }
  ]
}

Log in with your Cloudflare account, then deploy your Worker to make it accessible on the Internet:

Terminal window
npx wrangler login
npx wrangler deploy

Next steps

Browser Run /content endpoint Fetch the fully rendered HTML of a page after JavaScript runs, with options for load behavior and blocking.
Items Workers binding Full reference for uploading, listing, and deleting documents in built-in storage.
Website data source Crawl and index a domain you own automatically, following its sitemap.