---
title: "Public Agent Web"
description: "Make public routes crawlable, readable, citable, and optionally callable by agents — robots.txt, llms.txt, markdown mirrors, JSON-LD, and a public MCP surface."
---

# Public Agent Web

The public agent web makes public Agent-Native routes easy for agents to crawl, read, cite, and call. The goal is not to make every app endpoint public. The goal is to publish a clean public surface for pages that are already public, while keeping private data and tool access behind explicit controls.

The docs site is the reference implementation. Today it ships:

- `/robots.txt` with a crawler policy that allows retrieval but disallows training by default.
- `/sitemap.xml` with absolute canonical URLs and `lastmod` when the source file exposes it.
- `/llms.txt` and `/llms-full.txt` for agent-friendly content discovery.
- Markdown mirrors such as `/docs/getting-started.md`.
- `Accept: text/markdown` responses for public docs pages after a production build.
- JSON-LD for base organization, website, and page metadata.
- An audit CLI (`npx @agent-native/core@latest audit-agent-web`) that checks all of the above.

Setting `publicMcp: true` additionally exposes opted-in actions as a public MCP endpoint, allowing external agents to call them directly (see [MCP Protocol](/docs/mcp-protocol)).

<Diagram id="doc-block-1guzidr" title="What a public route publishes" summary={"A public route fans out into agent-friendly representations. Reading the route is separate from calling tools — tool access stays opt-in."}>

```html
<div class="diagram-web">
  <div class="diagram-box" data-rough>
    Public route<br /><small class="diagram-muted"
      >derived from route access settings</small
    >
  </div>
  <div class="diagram-arrow diagram-muted" aria-hidden="true">&rarr;</div>
  <div class="diagram-grid">
    <span class="diagram-pill">robots.txt</span
    ><span class="diagram-pill">sitemap.xml</span
    ><span class="diagram-pill">llms.txt</span
    ><span class="diagram-pill">.md mirror</span
    ><span class="diagram-pill">JSON-LD</span
    ><span class="diagram-pill">text/markdown</span>
  </div>
  <div class="diagram-arrow diagram-muted" aria-hidden="true">&rarr;</div>
  <div class="diagram-col gate">
    <span class="diagram-pill warn">Tools stay private</span
    ><small class="diagram-muted"
      >publicMcp + publicAgent.expose required</small
    >
  </div>
</div>
```

```css
.diagram-web {
  display: flex;
  align-items: center;
  gap: 14px;
  flex-wrap: wrap;
}
.diagram-web .diagram-arrow {
  font-size: 22px;
  line-height: 1;
}
.diagram-web .diagram-grid {
  display: grid;
  grid-template-columns: repeat(2, minmax(0, 1fr));
  gap: 8px;
}
.diagram-web .gate {
  display: flex;
  flex-direction: column;
  gap: 4px;
  align-items: flex-start;
}
```

</Diagram>

## Configuration {#config}

Add `agentWeb` under the existing workspace app config (in your app's `package.json` under the `agent-native` key — or equivalently `workspace.agentWeb`, `agentWeb`, or `root.agentWeb`). The public route list is still derived from the app's route access settings; `agentWeb` controls how that public surface is represented to agents.

```json
{
  "agent-native": {
    "workspaceApp": {
      "audience": "public",
      "protectedPaths": ["/admin/*"],
      "agentWeb": {
        "discoverable": true,
        "markdownTwins": true,
        "llmsTxt": true,
        "jsonLd": true,
        "publicAgentCard": true,
        "publicMcp": false,
        "crawlerPolicy": "discoverable-no-training",
        "crawlers": {
          "training": "disallow",
          "search": "allow",
          "userTriggered": "allow",
          "codingAgents": "allow",
          "autonomousAgents": "allow"
        }
      }
    }
  }
}
```

For most apps, leave the defaults alone. If an app has any public route, `discoverable` defaults on. The default crawler policy is "discoverable, not trainable": search, user-triggered retrieval, coding agents, and autonomous browsing agents are allowed; training crawlers are disallowed.

## Route source of truth {#route-source}

Agent Web discovery follows the route access model:

- Public apps expose every route except `protectedPaths`.
- Internal apps expose only `publicPaths`.
- Public share and form pages can be readable by agents.
- Submitted private data, authenticated dashboards, and user/org state are never included just because a nearby page is public.

This keeps mixed apps natural. A forms app can expose a public form page and keep submissions private. A content app can expose published posts and keep the editor private. A docs site can expose everything except admin tools.

## Public pages are not public tools {#public-tools}

Public page access and public tool access are separate. A route being public only means agents can read that route as HTML, Markdown, sitemap entries, llms entries, and structured data.

<Callout id="doc-block-s50w8y" tone="warning">

**A public page is not a public tool.** Making a route crawlable never exposes an action. Tool access requires an explicit `publicAgent.expose` opt-in on the action _and_ `publicMcp: true` on the app.

</Callout>

To expose an action through a public agent protocol, the action must opt in:

<AnnotatedCode
  id="doc-block-ak7w4m"
  title="Opting one safe action onto the public surface"
  filename="actions/search-docs.ts"
  language="ts"
  code={
    'export default defineAction({\n  description: "Search published docs",\n  readOnly: true,\n  publicAgent: {\n    expose: true,\n    readOnly: true,\n    requiresAuth: false,\n    isConsequential: false,\n    title: "Search published docs",\n  },\n  run: async (args) => {\n    // ...\n  },\n});'
  }
  annotations={[
    {
      lines: "4",
      label: "Explicit opt-in",
      note: "Without `publicAgent.expose === true`, the action never appears on any public agent surface — no matter how public its routes are.",
    },
    {
      lines: "5-7",
      label: "Self-describe safety",
      note: "Mark it read-only, declare whether it needs auth, and flag whether it is consequential. Public MCP excludes consequential/write actions unless policy explicitly allows them.",
    },
  ]}
/>

`agentWeb.publicMcp` stays `false` by default. When public MCP is enabled, the server should expose only actions with `publicAgent.expose === true`, and should still exclude consequential or write actions unless the action and auth policy explicitly allow them.

## Build-time files {#build-time}

Framework utilities in `@agent-native/core/agent-web` generate the common files from one page list:

```ts
import {
  buildAgentWebStaticFiles,
  normalizeAgentWebConfig,
} from "@agent-native/core/agent-web";

const config = normalizeAgentWebConfig(
  { crawlerPolicy: "discoverable-no-training" },
  { hasPublicRoutes: true },
);

const files = buildAgentWebStaticFiles({
  siteName: "My Agent-Native App",
  siteUrl: "https://example.com",
  description: "Public docs for my app.",
  config,
  pages: [
    {
      path: "/docs",
      title: "Docs",
      description: "Start here.",
      markdown: "# Docs\n\nStart here.\n",
      markdownPath: "/docs/getting-started.md",
      lastmod: new Date(),
    },
  ],
});
```

Vite apps can use `createAgentWebVitePlugin` from `@agent-native/core/vite` to write those files into `public`, `dist`, `dist/client`, `dist/server/public`, or `build/client` during production builds.

## Audit a site {#audit}

Use the CLI audit against a deployed site or a local production server:

```bash
npx @agent-native/core@latest audit-agent-web --url https://www.agent-native.com
```

The audit checks for:

- SSR-visible HTML.
- Canonical URLs.
- JSON-LD.
- `robots.txt` policy and absolute sitemap URL.
- Absolute sitemap entries.
- `/llms.txt` and `/llms-full.txt`.
- Markdown mirrors.
- `Accept: text/markdown`.
- No accidental 401/403 blocks for common agent retrieval user agents.

The audit exits non-zero if a required public surface is missing.

## What's next

- [**Actions**](/docs/actions) — how to opt actions into the public agent protocol
- [**MCP Protocol**](/docs/mcp-protocol) — the MCP surface that `publicMcp: true` enables
- [**Deployment**](/docs/deployment) — where these static files are written during builds