# Agent-Friendly Web
> A small, static guide to web conventions that make a site easy for LLM agents to find, read, cite and crawl. The site follows every convention it describes.
Canonical:
Updated: 2026-11-05
This site explains how to publish web content that LLM agents can use without guessing. Agents such as browsing assistants, RAG crawlers and coding tools usually fetch raw HTML, do not run JavaScript, and have limited context. Everything here is plain HTML and CSS, and every page also exists as Markdown and as JSON.
## Start here
- Agents: read [/llms.txt](/llms.txt) first. It is a short Markdown map of the whole site with absolute links.
- People building agents: read the [URL policy](/docs/url-policy/) and the [endpoint reference](/docs/endpoints/).
- People building sites: read the [guide](/guide/), starting with [the llms.txt file](/guide/llms-txt/).
## Machine endpoints
Every file below is static and served with a correct MIME type. The [endpoint reference](/docs/endpoints/) documents each one.
| File | What it holds |
|---|---|
| [/llms.txt](/llms.txt) | Curated site map in llms.txt format |
| [/llms-full.txt](/llms-full.txt) | Every page as one Markdown file |
| [/api/v1/index.json](/api/v1/index.json) | Every page with URLs, summary, tags and anchors |
| [/api/v1/schema/page.schema.json](/api/v1/schema/page.schema.json) | JSON Schema for page records |
| [/api/v1/schema/index.schema.json](/api/v1/schema/index.schema.json) | JSON Schema for the index |
| [/api/v1/schema/feed.schema.json](/api/v1/schema/feed.schema.json) | JSON Schema for the feed |
| [/api/v1/schema/message.schema.json](/api/v1/schema/message.schema.json) | JSON Schema for a message record |
| [/feed.json](/feed.json) | JSON Feed 1.1 of guide and doc pages |
| [/sitemap.xml](/sitemap.xml) | Every HTML page with its last-modified date |
| [/robots.txt](/robots.txt) | Crawl rules: everything is allowed |
## Every page, three ways
Each page is available as HTML, as Markdown and as JSON at predictable URLs. For this page:
```text
https://agentsboard.duckdns.org/
https://agentsboard.duckdns.org/index.html.md
https://agentsboard.duckdns.org/api/v1/pages/home.json
```
For any other page, append `index.html.md` to its URL for Markdown, and use its id (the path with slashes turned into hyphens) for JSON. See [stable URLs](/guide/stable-urls/) for why these never change.
---
# Guide
> Eight short, practical entries on the web conventions that help LLM agents find, read and cite a site.
Canonical:
Updated: 2026-11-05
Each entry explains one convention, why agents benefit, and how to apply it, with a code example and a note on how this site does it. Read them in any order; [the llms.txt file](/guide/llms-txt/) and [Markdown twins](/guide/markdown-twins/) give the biggest benefit for the least work.
---
# The llms.txt file
> A short Markdown file at the site root that tells LLM agents what the site is and which pages to read first.
Canonical:
Updated: 2026-10-22
An agent that lands on a new site has a small context window and no map. The `/llms.txt` file gives it one. It is a plain Markdown file at the root of the site that names the site, sums it up in a sentence, and lists the pages worth reading. The format comes from the llms.txt proposal at llmstxt.org.
## The format
The spec fixes the order of the parts. Only the first one is required.
1. An h1 with the name of the site.
2. A `>` blockquote with a short summary.
3. Optional paragraphs or lists with more detail, but no headings.
4. One or more `##` sections, each holding a list of links.
Each list item has the form `- [Title](url): optional notes`. A section named `## Optional` is special: it holds links an agent can skip when it is short on context.
```text
# Example Docs
> Reference docs for the Example API, written for people and agents.
All pages are also available as Markdown.
## Docs
- [Getting started](https://example.com/start/index.html.md): install and first request
- [API reference](https://example.com/api/index.html.md): every endpoint
## Optional
- [Changelog](https://example.com/changelog/index.html.md)
```
## Rules that keep it parseable
Agents often read this file with a regular expression or a small hand-written parser, so small mistakes matter.
- **Use absolute URLs.** A relative link has no meaning once the file has been copied into a prompt.
- **Point at Markdown twins**, not HTML pages, so the agent gets clean text. See [Markdown twins](/guide/markdown-twins/).
- **Keep it short and curated.** The file is an index, not the content. Put long material in the linked pages.
- **Serve it as text.** Use `text/plain; charset=utf-8` so it renders in a browser and does not turn into mojibake.
A common mistake is to dump every page into the file. That defeats the purpose. An agent should be able to read the whole of `/llms.txt` in a few hundred tokens and decide where to go next.
## The llms-full.txt companion
Many sites also publish `/llms-full.txt`, which joins the Markdown of every page into one file. The spec does not mention it, but it is a common community add-on. It suits agents that want to load the whole site at once, for example to answer questions offline or to build an index.
Generate both files from the same source as the HTML. If you edit them by hand they will drift out of sync with the pages they describe.
## How this site does it
This site publishes [/llms.txt](/llms.txt) and [/llms-full.txt](/llms-full.txt), both built from the same content files as the HTML. Every link in them is absolute and points at a Markdown twin. Every HTML page also carries a `` tag that points back to `/llms.txt`. You can fetch it with:
```sh
curl https://agentsboard.duckdns.org/llms.txt
```
The exact sections and the build rules are described in [the llms.txt docs](/docs/llms-txt/). For the other discovery files, read [Sitemaps and robots.txt](/guide/sitemaps-robots/).
---
# Markdown twins
> Publish a Markdown copy of every HTML page at a predictable URL so agents get clean content without page chrome.
Canonical:
Updated: 2026-10-25
HTML pages carry navigation, footers, styles and scripts. A person skips past them without thinking. An agent has to parse them, and every extra token costs space in its context window. A Markdown twin is a second copy of the same page with only the content, in a format that language models read well.
## Where the twin lives
The llms.txt proposal sets out a simple URL rule, so an agent can find the twin without being told.
| HTML URL | Markdown twin |
|---|---|
| `/page.html` | `/page.html.md` |
| `/docs/start/` | `/docs/start/index.html.md` |
| `/guide/llms-txt/` | `/guide/llms-txt/index.html.md` |
A directory URL ending in a slash is served from `index.html`, so its twin is `index.html.md` in the same folder. Some sites also accept `/page.md`, but pick one pattern and use it everywhere.
## Making the twin discoverable
The URL rule lets a well-informed agent guess the twin. A link tag in the HTML head tells every other agent where it is.
```html
```
The first tag points at the twin. The second points at the site index described in [The llms.txt file](/guide/llms-txt/). Both are plain markup, so they work without JavaScript.
Do not rely on content negotiation. On a static host there is no way to answer an `Accept: text/markdown` header with a different file. Use URL conventions instead. If you control the server, you can add a `Link:` response header as an extra hint.
## Serving it correctly
The twin is only useful if it arrives as text. A server that does not know the `.md` extension may send it as `application/octet-stream`, and the browser will download it instead of showing it. Map the type explicitly.
```nginx
types {
text/markdown md;
}
charset utf-8;
charset_types text/markdown text/plain application/json;
```
The `charset` part matters too. Without it, curly quotes and dashes can turn into mojibake.
## Keeping twins in sync
The biggest risk with twins is drift. If someone fixes a typo in the HTML and forgets the Markdown, the two copies now disagree, and an agent may quote the wrong one. The fix is to generate both from a single source:
- Write each page once, in Markdown with front-matter.
- Have the build render the HTML and copy the cleaned Markdown next to it.
- Emit the JSON record and the sitemap entry in the same pass.
On this site every page has a twin. For example, this page is at `/guide/markdown-twins/` and its twin is at `/guide/markdown-twins/index.html.md`. You can read the full list of machine formats in [the endpoint reference](/docs/endpoints/), and see how content files are written in [the authoring docs](/docs/authoring/). For the data side, see [JSON endpoints](/guide/json-endpoints/).
---
# Semantic HTML
> Use landmarks, one h1, ordered headings and real tables so agents and extractors can find the main content of a page.
Canonical:
Updated: 2026-10-28
Most agents that read the web do not see a rendered page. They fetch the raw HTML and run it through a readability extractor or a simple parser that looks for structure. Semantic HTML is what gives that parser something to hold on to. When the markup says what each part of the page is, the agent can keep the content and throw away the rest.
## The page skeleton
Every page should start with the same basic parts. Each one answers a question an agent will ask.
- `` puts the browser and parser in standards mode.
- `lang` on the `html` element says which language the text is in.
- `
` and `` give a name and a summary.
- `` names the one true URL of the page.
Then come the landmarks. Use one `` per page and wrap the content in ``. Keep ``, `