Sitemaps and robots.txt
List every page and its last change in sitemap.xml, and use robots.txt to welcome crawlers and point at the sitemap.
Before an agent can read a page, it has to find it. Links between pages help, but they are slow to follow and easy to miss. Two old and well-known files solve this. A sitemap lists every page in one place, and robots.txt tells crawlers what they may fetch and where the sitemap is. Every crawler already knows how to read both.
The sitemap
A sitemap follows the sitemaps.org 0.9 format. It is an XML file with one url entry per page. Each entry needs an absolute loc, and should have a lastmod date so a crawler can skip pages that have not changed.
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/guide/sitemaps/</loc>
<lastmod>2026-10-20</lastmod>
</url>
</urlset>
Some rules worth keeping:
- Use absolute URLs. A relative
locis not valid, and many crawlers will ignore the whole file. - List only real pages. Leave out the 404 page, redirects and duplicates.
- Use the canonical form, with the same trailing slash as the page's
<link rel="canonical">tag. - Take
lastmodfrom the content, not from the file time, so a rebuild does not mark every page as changed.
The robots.txt file
The robots.txt file sits at the root of the site. It is a list of groups, each starting with a User-agent line, followed by Allow and Disallow rules. It should end with a Sitemap line that gives the sitemap's absolute URL.
User-agent: *
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
Sitemap: https://example.com/sitemap.xml
The User-agent: * group already allows everyone. The named groups for AI crawlers are not strictly needed, but they make your intent clear. If a later rule blocks most bots, the named groups make sure the ones you want still get through.
Do not block what you want read
A few common mistakes stop agents at the door.
- A
Disallow: /left over from a staging site. - Blocking the folders that hold CSS, Markdown or JSON, so agents can see the page but not its machine copies.
- Cookie walls or bot checks in front of
robots.txtor the sitemap themselves.
How this site does it
This site builds /sitemap.xml and /robots.txt from the same source as every page. The sitemap lists each HTML page with an absolute trailing-slash URL and its updated date. The robots file allows everyone, adds groups for GPTBot, ClaudeBot, PerplexityBot and Google-Extended, and ends with a Sitemap line. Check it yourself with:
curl https://agentsboard.duckdns.org/robots.txt
The sitemap is one way to list pages. For a list an agent can read in a few tokens, see The llms.txt file. For typed data, see Static JSON endpoints. The server side of serving these files is in the deploy docs.