Agent-Friendly Web

Markdown twins

Publish a Markdown copy of every HTML page at a predictable URL so agents get clean content without page chrome.

Updated

HTML pages carry navigation, footers, styles and scripts. A person skips past them without thinking. An agent has to parse them, and every extra token costs space in its context window. A Markdown twin is a second copy of the same page with only the content, in a format that language models read well.

Where the twin lives

The llms.txt proposal sets out a simple URL rule, so an agent can find the twin without being told.

HTML URLMarkdown twin
/page.html/page.html.md
/docs/start//docs/start/index.html.md
/guide/llms-txt//guide/llms-txt/index.html.md

A directory URL ending in a slash is served from index.html, so its twin is index.html.md in the same folder. Some sites also accept /page.md, but pick one pattern and use it everywhere.

Making the twin discoverable

The URL rule lets a well-informed agent guess the twin. A link tag in the HTML head tells every other agent where it is.

<link rel="alternate" type="text/markdown" href="/guide/markdown-twins/index.html.md">
<link rel="describedby" href="/llms.txt">

The first tag points at the twin. The second points at the site index described in The llms.txt file. Both are plain markup, so they work without JavaScript.

Do not rely on content negotiation. On a static host there is no way to answer an Accept: text/markdown header with a different file. Use URL conventions instead. If you control the server, you can add a Link: response header as an extra hint.

Serving it correctly

The twin is only useful if it arrives as text. A server that does not know the .md extension may send it as application/octet-stream, and the browser will download it instead of showing it. Map the type explicitly.

types {
    text/markdown  md;
}
charset utf-8;
charset_types text/markdown text/plain application/json;

The charset part matters too. Without it, curly quotes and dashes can turn into mojibake.

Keeping twins in sync

The biggest risk with twins is drift. If someone fixes a typo in the HTML and forgets the Markdown, the two copies now disagree, and an agent may quote the wrong one. The fix is to generate both from a single source:

On this site every page has a twin. For example, this page is at /guide/markdown-twins/ and its twin is at /guide/markdown-twins/index.html.md. You can read the full list of machine formats in the endpoint reference, and see how content files are written in the authoring docs. For the data side, see JSON endpoints.