Your website has two types of visitors now: humans using browsers, and AI agents fetching content to answer questions. Most sites are only optimized for one of them. The ones that figure out both will dominate visibility in the next two years.
This is not speculation. Right now, ChatGPT-User, Claude-User, PerplexityBot, and Gemini-Deep-Research are visiting web pages the moment someone asks a question. They fetch your content in real-time, parse it, and decide whether to cite you in their answer. If your site is not readable by these agents, you do not get cited. Someone else does. Your traditional SEO ranking becomes irrelevant if the AI never surfaces your content in its response.
The /llms.txt standard
There is a proposed standard called /llms.txt that addresses this directly. The idea is straightforward: you place a markdown file at the root of your domain that maps your site for AI models. It tells them what you are, what you offer, and provides links to clean markdown versions of your key pages. Think of it as robots.txt for comprehension rather than access control.
Where robots.txt says "you may or may not crawl these paths," llms.txt says "here is how to understand what this site is about, and here are the pages that matter most, in a format you can actually parse." The distinction matters because AI agents have limited context windows. They cannot process your entire site. They need a guided entry point, and llms.txt gives them one.
I added one to my own site. It took ten minutes. The file is plain markdown: a title, a brief description, and a list of URLs pointing to the content I want AI agents to find first. No build tooling, no framework, no dependencies. If your site has a dozen important pages, your llms.txt is probably fifteen lines long. The barrier to entry is essentially zero, which makes the number of sites that still do not have one surprising.
Clean content wins
Beyond llms.txt, the sites that get cited by AI agents share a common trait: their content is clean and parseable. AI assistants struggle with JavaScript-heavy single-page applications, cookie consent walls, and navigation-cluttered HTML. They perform best with pages that have clear, direct answers early in the content. Structure and semantic HTML matter more than keyword density ever did.
This is good news for anyone who cares about writing quality. The same things that make content good for human readers (clear structure, direct answers, minimal noise) are exactly what makes content good for AI agents. The sites being cited are not the ones gaming keywords. They are the ones with genuinely useful, well-organized information. If you have been writing clearly and structuring your pages with semantic HTML all along, you are already ahead of most of the web.
Two types of bots, two different decisions
Not all AI bots are doing the same thing, and understanding the difference changes how you handle them. Training crawlers like GPTBot and CCBot download your content to use in model training. You can block them via robots.txt if you do not want your content baked into future model weights. That is a legitimate choice, and many publishers are making it.
But AI Assistants (ChatGPT-User, Claude-User, PerplexityBot) are different. They fetch pages in real-time, in response to a specific user query, to generate an answer right now. Blocking these means your content never appears in AI-generated answers, even if your Google ranking is perfect. You become invisible to an increasingly large segment of how people find information.
The decision is not binary. You can block training crawlers while allowing assistant bots. You can serve different content quality to each. But you need to make this decision consciously rather than inheriting a default robots.txt that was written before AI assistants existed. Most sites I see have not updated their robots.txt in years. They are making a choice by not making a choice, and that default is increasingly costly.
The new question
SEO used to be about one thing: do I rank on page one? Now there is a second question that matters just as much: am I the source the AI cites when it answers? These are different problems with different solutions. Traditional SEO optimizes for Google's ranking algorithm. AI visibility optimizes for parseability, clarity, and being the most useful source an agent can find quickly.
Both audiences want the same fundamental thing: clear, authoritative, well-structured content. The difference is that AI agents are less forgiving of noise. They will not scroll past your cookie banner or wait for your JavaScript to render. They will not infer meaning from your elaborate navigation structure. They will move on to the next source that gives them a clean answer in the first few hundred tokens. If that source is your competitor, the AI cites your competitor. Permanently.