Optimization Station
AI Agent Readiness
The 20 web standards that make your site discoverable and usable by AI agents — robots.txt, Link headers, Markdown negotiation, MCP, Agent Skills, OAuth, and agentic commerce — plus Reaudit’s 0-5 readiness levels and how to fix every check.
AI agents — ChatGPT, Claude, Perplexity, Copilot and the autonomous tools built on them — increasingly visit websites on a person’s behalf to read content, call APIs, authenticate, and even transact. They do not browse like a human. They look for machine-readable signals: files, response headers, and well-known endpoints that tell them what your site offers and how to use it. Agent readiness is how completely your site exposes those signals.
Reaudit’s Agent Readiness scan (in the Optimization Station) checks your site against 20 of these standards across five categories and rolls the result into a single 0-5 readiness level. This article explains every standard, why agents rely on it, and exactly how to implement it.
The five readiness levels
Your readiness level is the highest tier whose requirements you fully meet — each level builds on the one below it.
- Level 0 — Not Ready: fewer than two of robots.txt, sitemap, and Link headers are present.
- Level 1 — Basic Web Presence: at least two of robots.txt, sitemap, and Link headers.
- Level 2 — Bot-Aware: Level 1 plus AI bot rules in robots.txt and Content Signals.
- Level 3 — Agent-Readable: Level 2 plus Markdown content negotiation.
- Level 4 — Agent-Integrated: Level 3 plus at least one of MCP Server Card, A2A Agent Card, Agent Skills index, or API Catalog.
- Level 5 — Agent-Native: Level 4 plus at least two of Web Bot Auth, all four integration documents, and authentication metadata (OAuth discovery or Auth.md).
Start with the easy wins
A valid robots.txt with AI bot rules, a sitemap, and a single discovery Link header on your homepage already moves most sites from Level 0 to Level 2.
robots.txt
A valid
robots.txt at your domain root with explicit crawl rules is the baseline signal that your site is intentionally machine-accessible. Agents and crawlers read it first to learn what they may fetch. Publish it per the Robots Exclusion Protocol — RFC 9309 — with User-agent lines and Allow / Disallow rules.Sitemap
A
sitemap.xml lists your canonical URLs so agents can enumerate your content instead of guessing. Publish one per the Sitemaps protocol and reference it from robots.txt with a Sitemap: line so it is discoverable from the root.Link headers
HTTP
Link response headers let your homepage advertise machine-readable resources without an agent parsing any HTML. Per RFC 8288, return a Link header using registered relations such as api-catalog, service-desc, service-doc, or describedby pointing at your discovery documents.DNS for AI Discovery (DNS-AID)
DNS-AID lets agents discover your agent endpoints straight from DNS, before fetching a single page. Publish
SVCB/HTTPS records under your domain’s _agents namespace (for example _a2a._agents.example.com) with alpn and connection parameters, and sign the zone with DNSSEC so resolvers return authenticated data.Markdown negotiation
Agents understand clean Markdown far better than rendered HTML full of navigation and scripts. With Markdown content negotiation, when a request includes
Accept: text/markdown your server returns a Markdown representation of the page with Content-Type: text/markdown, while humans still get HTML by default. This pairs naturally with an llms.txt overview of your site.AI bot rules
Beyond generic crawl rules, declare explicit
User-agent rules for AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and others — in robots.txt. This tells answer engines whether they may read your content, which is a prerequisite for being cited by them.Content Signals
Content Signals express *how* AI may use the content it crawls. A
Content-Signal: directive in robots.txt declares preferences such as ai-train, search, and ai-input (each yes or no). See contentsignals.org. Reaudit recommends an open posture (ai-train=yes, search=yes, ai-input=yes) for brands that want maximum AI visibility.Web Bot Auth
Web Bot Auth lets a site cryptographically prove which automated requests it sends, so receiving sites can verify them. Publish a JSON Web Key Set (JWKS) at
/.well-known/http-message-signatures-directory containing at least one public key, and sign outbound bot requests with Signature-Agent and Signature-Input headers. The approach is being standardized by the IETF Web Bot Auth working group.API Catalog
An API Catalog is a single machine-readable index of your APIs. Per RFC 9727, serve
/.well-known/api-catalog with Content-Type: application/linkset+json and a linkset array; each entry carries an anchor URL and relations like service-desc (an OpenAPI spec), service-doc (docs), and optionally status (a health endpoint).OAuth discovery
OAuth/OIDC discovery metadata lets an agent learn how to authenticate with your APIs automatically. Serve
/.well-known/oauth-authorization-server (RFC 8414) or /.well-known/openid-configuration including issuer, authorization_endpoint, token_endpoint, and jwks_uri, plus the supported grant and response types.OAuth Protected Resource
Protected Resource Metadata tells an agent which authorization server guards a given resource. Per RFC 9728, serve
/.well-known/oauth-protected-resource with a resource identifier and an authorization_servers array, and optionally return a WWW-Authenticate header with resource_metadata on 401 responses.Auth.md
Auth.md is a human- and agent-readable
/auth.md document that explains, in plain language, how an agent registers and connects on a user’s behalf — which OAuth endpoints to use, the supported identity and credential types, and any prerequisites (for example, that an account must already have an active subscription). It complements the machine metadata above with onboarding instructions.MCP Server Card
The Model Context Protocol (MCP) lets agents call your tools directly. An MCP Server Card advertises your hosted MCP server so agents can discover it with no prior configuration. Serve
/.well-known/mcp/server-card.json with serverInfo (name and version), a transport endpoint (for example a Streamable HTTP /mcp URL), and the capabilities your server exposes.A2A Agent Card
An A2A (Agent-to-Agent) Agent Card lets other agents discover and delegate to your agent. Per the A2A Protocol, serve
/.well-known/agent-card.json with name, version, description, the interfaces it supports (service URL and transport), and the capabilities and skills it offers.Agent Skills
An Agent Skills index publishes reusable skills that code-capable agents can discover, fetch, and run. Serve
/.well-known/agent-skills/index.json with a $schema and a skills array; each skill carries a name, type, description, url, and a content digest. See agentskills.io.WebMCP
WebMCP exposes interactive tools to an agent running inside the browser, on the page itself. A page registers tools at load time by calling
navigator.modelContext.provideContext() with tool definitions (name, description, input schema, and an execute handler), letting an in-page agent act on the live DOM. It is an emerging W3C-incubated API.A note on WebMCP detection
WebMCP tools are registered by JavaScript at runtime, so a static scan can only detect the markers in your page source — not tools added dynamically. Treat a WebMCP result as a strong hint rather than a definitive verdict.
x402
x402 revives the dormant HTTP
402 Payment Required status as an agent-native payment flow: a protected endpoint responds with 402 and machine-readable payment requirements, the agent pays, and retries. Learn more at x402.org.A note on x402 detection
x402 has no fixed discovery document — payment requirements appear only when a protected route is hit. A homepage scan can therefore miss an x402 integration that lives behind a paywalled API path.
MPP
MPP (Machine Payment Protocol) describes payable operations inside your OpenAPI definition. Publish
/openapi.json with x-payment-info extensions on the operations that require payment so an agent can understand cost and payment method before calling them.UCP
UCP (Universal Commerce Protocol) lets agents transact for content and services through a published profile. Serve
/.well-known/ucp describing the protocol version, the services you offer, your capabilities, and the endpoints an agent should call.ACP
ACP (Agentic Commerce Protocol) lets an agent complete a purchase on a user’s behalf. Publish an
/.well-known/acp.json discovery document with protocol.name set to acp, a protocol.version, an absolute api_base_url, a non-empty transports array, and a non-empty capabilities.services array. See agenticcommerce.dev.How Reaudit’s Agent Readiness scan works
Open your project’s Optimization Station and the Agent Readiness panel runs a live scan of your domain: it inspects your homepage response and headers, your
robots.txt and sitemap, the relevant /.well-known/ documents, content negotiation, and DNS. Every standard is reported as pass, fail, or unable to check, grouped by category, with your overall 0-5 level and what to do to reach the next one. Use Re-scan any time after you ship a change.Fixing failing checks with your coding agent
Each failing check includes a concrete fix and a link back to this guide. To fix several at once, use Copy fix instructions in the Agent Readiness panel: it assembles a ready-to-paste prompt listing your failing standards and how to implement them. Paste it into your coding agent — Cursor, Claude Code, Copilot, or similar — and let it open the changes in your codebase.
Do it conversationally
Connect the Reaudit MCP server and ask your assistant to run the agent readiness scan for your project, then implement the top fixes — it can read your current level and the failing checks directly.