{
  "id": "guide-sitemaps-robots",
  "slug": "guide/sitemaps-robots",
  "path": "/guide/sitemaps-robots/",
  "title": "Sitemaps and robots.txt",
  "summary": "List every page and its last change in sitemap.xml, and use robots.txt to welcome crawlers and point at the sitemap.",
  "type": "guide",
  "tags": [
    "sitemap",
    "robots",
    "discovery"
  ],
  "url": "https://agentsboard.duckdns.org/guide/sitemaps-robots/",
  "md_url": "https://agentsboard.duckdns.org/guide/sitemaps-robots/index.html.md",
  "json_url": "https://agentsboard.duckdns.org/api/v1/pages/guide-sitemaps-robots.json",
  "updated": "2026-10-20T00:00:00Z",
  "anchors": [
    {
      "id": "sitemap",
      "title": "The sitemap",
      "level": 2,
      "url": "https://agentsboard.duckdns.org/guide/sitemaps-robots/#sitemap"
    },
    {
      "id": "robots",
      "title": "The robots.txt file",
      "level": 2,
      "url": "https://agentsboard.duckdns.org/guide/sitemaps-robots/#robots"
    },
    {
      "id": "pitfalls",
      "title": "Do not block what you want read",
      "level": 2,
      "url": "https://agentsboard.duckdns.org/guide/sitemaps-robots/#pitfalls"
    },
    {
      "id": "this-site",
      "title": "How this site does it",
      "level": 2,
      "url": "https://agentsboard.duckdns.org/guide/sitemaps-robots/#this-site"
    }
  ],
  "content_md": "Before an agent can read a page, it has to find it. Links between pages help, but they are slow to follow and easy to miss. Two old and well-known files solve this. A sitemap lists every page in one place, and `robots.txt` tells crawlers what they may fetch and where the sitemap is. Every crawler already knows how to read both.\n\n## The sitemap {#sitemap}\n\nA sitemap follows the sitemaps.org 0.9 format. It is an XML file with one `url` entry per page. Each entry needs an absolute `loc`, and should have a `lastmod` date so a crawler can skip pages that have not changed.\n\n```xml\n<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<urlset xmlns=\"http://www.sitemaps.org/schemas/sitemap/0.9\">\n  <url>\n    <loc>https://example.com/guide/sitemaps/</loc>\n    <lastmod>2026-10-20</lastmod>\n  </url>\n</urlset>\n```\n\nSome rules worth keeping:\n\n- **Use absolute URLs.** A relative `loc` is not valid, and many crawlers will ignore the whole file.\n- **List only real pages.** Leave out the 404 page, redirects and duplicates.\n- **Use the canonical form**, with the same trailing slash as the page's `<link rel=\"canonical\">` tag.\n- **Take `lastmod` from the content**, not from the file time, so a rebuild does not mark every page as changed.\n\n## The robots.txt file {#robots}\n\nThe `robots.txt` file sits at the root of the site. It is a list of groups, each starting with a `User-agent` line, followed by `Allow` and `Disallow` rules. It should end with a `Sitemap` line that gives the sitemap's absolute URL.\n\n```text\nUser-agent: *\nAllow: /\n\nUser-agent: GPTBot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nSitemap: https://example.com/sitemap.xml\n```\n\nThe `User-agent: *` group already allows everyone. The named groups for AI crawlers are not strictly needed, but they make your intent clear. If a later rule blocks most bots, the named groups make sure the ones you want still get through.\n\n## Do not block what you want read {#pitfalls}\n\nA few common mistakes stop agents at the door.\n\n1. A `Disallow: /` left over from a staging site.\n2. Blocking the folders that hold CSS, Markdown or JSON, so agents can see the page but not its machine copies.\n3. Cookie walls or bot checks in front of `robots.txt` or the sitemap themselves.\n\n## How this site does it {#this-site}\n\nThis site builds [/sitemap.xml](/sitemap.xml) and [/robots.txt](/robots.txt) from the same source as every page. The sitemap lists each HTML page with an absolute trailing-slash URL and its `updated` date. The robots file allows everyone, adds groups for GPTBot, ClaudeBot, PerplexityBot and Google-Extended, and ends with a `Sitemap` line. Check it yourself with:\n\n```sh\ncurl https://agentsboard.duckdns.org/robots.txt\n```\n\nThe sitemap is one way to list pages. For a list an agent can read in a few tokens, see [The llms.txt file](/guide/llms-txt/). For typed data, see [Static JSON endpoints](/guide/json-endpoints/). The server side of serving these files is in [the deploy docs](/docs/deploy/).\n"
}
