Updated
What llms.txt is
llms.txt is a Markdown file at the root of your domain, at /llms.txt. It gives an AI model a short description of your site. It also lists links to the pages that matter most. Ideally those links point to clean text or Markdown copies of the pages.
The idea is simple. A model can only read so much text at once. Most web pages are full of menus, scripts and layout. A hand-picked list gets the model to the useful content faster.
The format
The proposal keeps the layout simple, so both people and programs can read it:
- An H1 heading with the name of the site or project. This is the only part you must have.
- A blockquote with a one or two sentence summary.
- Optional paragraphs with more context.
- H2 sections, each with a Markdown list of links in the form [title](url): short note.
- An optional section called Optional, for links a model can skip when it is short on space.
llms.txt is not robots.txt
The two files do different jobs. robots.txt controls access. It tells crawlers which paths they may visit, including AI crawlers like GPTBot, ClaudeBot and PerplexityBot. llms.txt does not allow or block anything. It is a reading guide, not a set of rules.
If you want to keep AI crawlers out, or let them in, you still do that in robots.txt.
Does it actually help?
llms.txt is an idea from the community, not a standard. No major AI company has promised to use it for ranking or for citing sources. Treat it as a cheap extra that may help, not a sure win.
What clearly does matter for AI answers is the same base work that matters for search. Bots must be able to crawl your pages. Your text should be in the HTML, not loaded late by JavaScript. Your structured data should be correct. And AI crawlers should not be blocked by mistake.
Check your own site
The GEO checker tests three of these at once. It checks if an llms.txt file exists, if robots.txt lets AI crawlers in, and if the page has structured data.