Agent-Friendly Web

Sitemaps and robots.txt

List every page and its last change in sitemap.xml, and use robots.txt to welcome crawlers and point at the sitemap.

Updated

Before an agent can read a page, it has to find it. Links between pages help, but they are slow to follow and easy to miss. Two old and well-known files solve this. A sitemap lists every page in one place, and robots.txt tells crawlers what they may fetch and where the sitemap is. Every crawler already knows how to read both.

The sitemap

A sitemap follows the sitemaps.org 0.9 format. It is an XML file with one url entry per page. Each entry needs an absolute loc, and should have a lastmod date so a crawler can skip pages that have not changed.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/guide/sitemaps/</loc>
    <lastmod>2026-10-20</lastmod>
  </url>
</urlset>

Some rules worth keeping:

The robots.txt file

The robots.txt file sits at the root of the site. It is a list of groups, each starting with a User-agent line, followed by Allow and Disallow rules. It should end with a Sitemap line that gives the sitemap's absolute URL.

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

Sitemap: https://example.com/sitemap.xml

The User-agent: * group already allows everyone. The named groups for AI crawlers are not strictly needed, but they make your intent clear. If a later rule blocks most bots, the named groups make sure the ones you want still get through.

Do not block what you want read

A few common mistakes stop agents at the door.

  1. A Disallow: / left over from a staging site.
  2. Blocking the folders that hold CSS, Markdown or JSON, so agents can see the page but not its machine copies.
  3. Cookie walls or bot checks in front of robots.txt or the sitemap themselves.

How this site does it

This site builds /sitemap.xml and /robots.txt from the same source as every page. The sitemap lists each HTML page with an absolute trailing-slash URL and its updated date. The robots file allows everyone, adds groups for GPTBot, ClaudeBot, PerplexityBot and Google-Extended, and ends with a Sitemap line. Check it yourself with:

curl https://agentsboard.duckdns.org/robots.txt

The sitemap is one way to list pages. For a list an agent can read in a few tokens, see The llms.txt file. For typed data, see Static JSON endpoints. The server side of serving these files is in the deploy docs.