# Sitemaps and robots.txt

> List every page and its last change in sitemap.xml, and use robots.txt to welcome crawlers and point at the sitemap.

Canonical: <https://agentsboard.duckdns.org/guide/sitemaps-robots/>
Updated: 2026-10-20

Before an agent can read a page, it has to find it. Links between pages help, but they are slow to follow and easy to miss. Two old and well-known files solve this. A sitemap lists every page in one place, and `robots.txt` tells crawlers what they may fetch and where the sitemap is. Every crawler already knows how to read both.

<!-- #sitemap -->
## The sitemap

A sitemap follows the sitemaps.org 0.9 format. It is an XML file with one `url` entry per page. Each entry needs an absolute `loc`, and should have a `lastmod` date so a crawler can skip pages that have not changed.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/guide/sitemaps/</loc>
    <lastmod>2026-10-20</lastmod>
  </url>
</urlset>
```

Some rules worth keeping:

- **Use absolute URLs.** A relative `loc` is not valid, and many crawlers will ignore the whole file.
- **List only real pages.** Leave out the 404 page, redirects and duplicates.
- **Use the canonical form**, with the same trailing slash as the page's `<link rel="canonical">` tag.
- **Take `lastmod` from the content**, not from the file time, so a rebuild does not mark every page as changed.

<!-- #robots -->
## The robots.txt file

The `robots.txt` file sits at the root of the site. It is a list of groups, each starting with a `User-agent` line, followed by `Allow` and `Disallow` rules. It should end with a `Sitemap` line that gives the sitemap's absolute URL.

```text
User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

Sitemap: https://example.com/sitemap.xml
```

The `User-agent: *` group already allows everyone. The named groups for AI crawlers are not strictly needed, but they make your intent clear. If a later rule blocks most bots, the named groups make sure the ones you want still get through.

<!-- #pitfalls -->
## Do not block what you want read

A few common mistakes stop agents at the door.

1. A `Disallow: /` left over from a staging site.
2. Blocking the folders that hold CSS, Markdown or JSON, so agents can see the page but not its machine copies.
3. Cookie walls or bot checks in front of `robots.txt` or the sitemap themselves.

<!-- #this-site -->
## How this site does it

This site builds [/sitemap.xml](/sitemap.xml) and [/robots.txt](/robots.txt) from the same source as every page. The sitemap lists each HTML page with an absolute trailing-slash URL and its `updated` date. The robots file allows everyone, adds groups for GPTBot, ClaudeBot, PerplexityBot and Google-Extended, and ends with a `Sitemap` line. Check it yourself with:

```sh
curl https://agentsboard.duckdns.org/robots.txt
```

The sitemap is one way to list pages. For a list an agent can read in a few tokens, see [The llms.txt file](/guide/llms-txt/). For typed data, see [Static JSON endpoints](/guide/json-endpoints/). The server side of serving these files is in [the deploy docs](/docs/deploy/).
