llms.txt and Google Are Not Contradicting Each Other: What Website Owners Actually Need to Know

Google appears to be saying two different things about llms.txt.

Google Search Central says website owners do not need an llms.txt file to appear in Google Search, AI Overviews, or AI Mode. Google's Chrome documentation, however, describes llms.txt as a useful convention for helping AI agents understand a website's structure and primary content.

Both statements are correct. They refer to different products, different use cases, and different outcomes.

Short answer: Google Search does not use llms.txt as a ranking or eligibility requirement for Google Search, AI Overviews, or AI Mode. Chrome documents it as an optional convention that may help some browser agents understand a website faster. Therefore, llms.txt is optional infrastructure, not an SEO or AEO growth hack.

The practical conclusion is simple: llms.txt is an optional aid for some AI agents, not a Google Search ranking factor or a proven generative-engine-optimization shortcut. For most businesses, the priority should remain crawlable websites, valuable content, clear information architecture, credible external references, and measurement of how AI systems actually describe and cite the brand.

Last reviewed: September 2026. AI crawler behavior, platform documentation, and experimental standards may change.

What is llms.txt?

llms.txt is a proposed Markdown file placed at a website's root or within a subdirectory. It summarizes the site or documentation area and links agents to important pages. A root-level file might be available at https://example.com/llms.txt, while a documentation-specific file could appear at https://example.com/docs/llms.txt.

The proposal behind the convention is intended to help large language models and other agents orient themselves when a website contains complex HTML, JavaScript, advertising, navigation, or more information than an agent can efficiently process at once.[2]

Unlike robots.txt, llms.txt does not control crawler access, block content, prevent indexing, or guarantee that an AI system will cite a page. It is an optional summary and navigation aid.

The source of the confusion

The term llms.txt sounds like an AI equivalent of robots.txt. That comparison creates the wrong expectation.

A robots.txt file communicates crawler-access preferences. It tells a crawler which URLs it may access, although it is not a security mechanism and does not guarantee that a URL will stay out of Google's index. Google explains that robots.txt is primarily used to manage crawler traffic, while noindex, authentication, or removal are the appropriate tools for preventing indexing.[1]

An llms.txt file has a different proposed purpose. It can summarize what the site does and point an AI agent toward important pages or cleaner documentation.

In other words, robots.txt is about access instructions, while llms.txt is about optional context and orientation. Neither file is a substitute for a well-built website.

What Google Search Central actually says

Google Search's position is specific. Its guidance concerns visibility in Google Search features, including AI Overviews and AI Mode.

Google says that its generative AI features are grounded in the core Search index and ranking systems. They use techniques such as retrieval-augmented generation and query fan-out to identify relevant pages, assess information, and present supporting links.[3]

For a page to be eligible as a supporting link in AI Overviews or AI Mode, Google says that it must be indexed and eligible to appear in Search with a snippet. There are no additional technical requirements for these AI features.[4]

Google's AI optimization guide goes further. In its mythbusting section, Google says that website owners do not need to create llms.txt files or other special machine-readable files, AI text files, or special markup to appear in Google Search, including its generative AI capabilities. Google Search does not use those files for this purpose. Creating one for another service is allowed, but Google says it will neither help nor harm Search visibility or rankings.[3]

That is not a claim that no AI agent anywhere can use llms.txt. It is a claim about Google Search's systems.

What Chrome's Lighthouse documentation actually says

Chrome's documentation addresses a different question: how prepared a website is for agentic browsing.

An agentic browser agent may inspect a page's rendered appearance, Document Object Model, accessibility tree, and content in order to complete a task for a user. For example, an agent might compare products, identify a company's pricing terms, or find the correct documentation page.

Chrome describes llms.txt as an "emerging convention" that provides a machine-readable summary of a website for large language models and AI agents. The documentation says that, without the file, agents may need to crawl more of the site to understand its high-level structure and primary content.[5]

The important qualification is in the same documentation: the Lighthouse audit is part of experimental agentic-browsing audits, and a missing llms.txt file returns Not Applicable because the file is optional.[5]

Chrome is therefore documenting a possible usability aid for agents. It is not announcing that llms.txt is a requirement for Google Search, a ranking signal, or a guaranteed path into AI answers.

The two Google teams are answering different questions:

• Do I need llms.txt to appear in Google Search, AI Overviews, or AI Mode? Google Search Central: No. Follow foundational SEO practices instead.
• Could a browser agent use llms.txt to orient itself more quickly? Chrome Lighthouse: Potentially. It is an emerging, optional convention.
• Does the file control crawler access or prevent content from being used? robots.txt and preview-control documentation: No. Use the appropriate crawl, indexing, or access controls.
• Has llms.txt been proven to increase AI citations or traffic? Available independent studies: No reliable causal evidence exists yet.

Does llms.txt improve Google rankings?

No. Google says that Google Search does not use llms.txt for Search rankings or generative AI visibility.[3]

There is also no strong independent evidence that publishing the file improves Google rankings, AI Overview inclusion, ChatGPT mentions, or citations in other AI search systems.

SE Ranking analyzed nearly 300,000 domains and found that 10.13% had an llms.txt file. Its statistical analysis and machine-learning model found no correlation between the file and how often a domain was cited by AI systems. In fact, removing the llms.txt variable improved the model's predictive accuracy.[6]

Ahrefs analyzed 137,210 domains that received traffic in May 2026. In its sample, 28% published an llms.txt file, but 97% of those files received no requests during the month. Among files that did receive requests, most requests came from bots, and a substantial share came from SEO tools, research systems, or other non-search activity.[7] Ahrefs also noted that its sample was more technically sophisticated than the web as a whole, so its adoption figure should not be treated as a universal rate.

A smaller Search Engine Land analysis followed ten websites for 90 days before and after implementation. Two sites experienced increases in AI traffic, but those sites had also launched new functional assets, improved technical SEO, published extractable information, or earned significant media coverage. Eight sites saw no measurable improvement, and one declined.[8]

These findings do not prove that llms.txt can never be useful. They show that the file should not be treated as a dependable growth lever. A correlation between a file and visibility has not been established, and the available evidence does not justify prioritizing it over proven work.

The independent studies are useful directional evidence, not definitive causal experiments. Adoption samples differ, AI crawler behavior changes quickly, and citation measurement varies by provider. The safest conclusion is therefore not that llms.txt can never help, but that its visibility impact has not been demonstrated reliably.

Does llms.txt help AI Overviews or AI Mode?

There is no requirement to publish it, and Google says that no special file or markup is needed for AI Overviews or AI Mode.[4]

Google recommends the same foundational practices that support ordinary Search visibility: allow crawling, make important pages discoverable through internal links, provide important information in text, maintain a good page experience, use accurate structured data, and keep business information current.[4]

A page must be indexed and eligible to appear with a snippet before it can be considered as a supporting link. An llms.txt file does not bypass that requirement.

Is llms.txt the same as robots.txt?

No.

robots.txt communicates crawler-access preferences. It is relevant to crawlers such as Googlebot and provider-specific AI search bots. It is not a security mechanism, and it does not by itself remove a page from an index.[1]

llms.txt is an optional Markdown summary for agents. It does not grant or deny access, prevent training, control search inclusion, or guarantee citations.

AI platforms may also use separate crawlers for search, model training, advertising, and user-initiated browsing. OpenAI, for example, distinguishes OAI-SearchBot, which is used to surface websites in ChatGPT search results, from GPTBot, which is associated with crawling content that may be used to improve OpenAI's foundation models. OpenAI also documents ChatGPT-User as a user-initiated agent that is separate from automatic search crawling.[10]

Website owners should therefore review each provider's official crawler documentation rather than assuming that llms.txt controls all AI access.

When publishing llms.txt can make sense

There are legitimate reasons to publish the file.

A developer platform, API provider, documentation-heavy product, or open-source project may benefit from giving coding agents a concise route into its documentation. In those settings, the agent is not merely trying to cite a brand. It may need to locate an API reference, understand authentication, compare implementation paths, or select the correct tutorial. A concise Markdown index can reduce the effort required to find that information if the agent chooses to use it.

The case is weaker for a typical marketing website. A company selling professional services, software, consumer products, or local services usually gains more from improving its actual pages than from adding an index file that no major AI provider has established as a general visibility requirement across the open web.

A sensible decision framework looks like this:

• You need Google Search, AI Overviews, or AI Mode visibility: Do not create llms.txt as an SEO requirement. Improve technical SEO, content quality, and authority.
• You operate developer documentation or an API: Consider testing a concise file that links to authoritative, maintained documentation.
• You have an agent-heavy workflow and can inspect server logs: Experiment if the implementation is inexpensive, then measure actual requests and outcomes.
• A vendor says the file guarantees AI citations or rankings: Treat the claim skeptically. Ask for methodology, controls, and independent evidence.
• Your site has crawl, indexing, content, or reputation problems: Fix those issues before spending time on llms.txt.

If a team publishes the file, it should be treated as low-risk documentation infrastructure rather than a campaign promise. It should be short, accurate, maintained, and limited to pages that genuinely help an agent understand the business or product. It should not contain inflated claims, obsolete URLs, hidden instructions, or content that contradicts the visible website.

What should businesses optimize instead?

1. Make the important information accessible

Google recommends allowing crawling, using internal links to make important pages discoverable, and ensuring that key content is available in text form.[4] These fundamentals also help AI systems that retrieve and interpret public web pages.

Review whether your most important pages are indexable, linked from relevant navigation or content, rendered clearly, and free from technical barriers. Check canonicalization, redirects, status codes, mobile usability, page speed, and Search Console coverage. An llms.txt file cannot compensate for a page that an agent or search crawler cannot reliably access.

Google also recommends checking that it can see a page in substantially the same way as a user. Important CSS, JavaScript, and content should not be hidden from crawlers when those resources are necessary to understand the page.[12]

2. Publish information that answers real questions

Google's guidance emphasizes helpful, reliable, people-first content rather than content produced primarily to manipulate an AI system.[3] This means explaining products clearly, documenting limitations, answering comparison questions, stating prices or service boundaries where appropriate, and supporting claims with evidence.

In practice, clear definitions, specific facts, comparison information, and well-structured pages are easier for both people and retrieval systems to interpret. Clear headings, concise definitions, comparison tables, examples, original research, expert commentary, and well-maintained FAQs can make a page more useful without making the writing unnatural or repetitive.

3. Establish a coherent entity and trustworthy source

AI systems need to understand what a company is, what it offers, who operates it, where it serves customers, and how its claims are supported.

Make that information consistent across the website and reputable external sources. Maintain a clear About page, named authors and reviewer credentials where relevant, accurate contact information, consistent company and product names, service descriptions, customer evidence, case studies, and visible publication or update dates.

Use Organization, Person, Article, Service, and Breadcrumb structured data where appropriate, but ensure that the markup matches visible page content. Structured data can clarify entities and relationships; it cannot turn unsupported claims into trusted facts.

4. Build external credibility

AI answers do not depend only on what a company says about itself. Models may also use editorial coverage, reputable directories, customer discussions, expert publications, videos, documentation, and other third-party sources.

The goal should not be to manufacture mentions. Google specifically warns against pursuing inauthentic mentions as an AI visibility tactic.[3] Instead, earn references by publishing useful material, contributing expertise, maintaining accurate profiles, developing partnerships, and creating products or data that others have a reason to discuss.

5. Make the website agent-friendly

For agentic browsing, the website's DOM, accessibility tree, visible text, and interaction design may matter more than the presence of a single Markdown file. Chrome's agent-friendly website guidance recommends semantic links and buttons, clear form labels, stable layouts, visible actions, and interfaces that are understandable through both HTML and accessibility information.[11]

Prefer real <a> and <button> elements over clickable <div> elements. Associate labels with form fields. Avoid transparent overlays that obscure controls. Keep important actions stable and visually clear. These practices improve accessibility and human usability as well as agent performance.

This is a different optimization problem from AI search retrieval. A page can be easy for a search system to cite but difficult for an agent to operate. Conversely, an agent-friendly interface does not guarantee that the page will be selected as an authoritative source. The two goals overlap, but they are not identical.

6. Measure outcomes instead of file adoption

The meaningful question is not "Do we have an llms.txt file?" The meaningful questions are:

• Does the brand appear when customers ask relevant questions?
• Which sources do AI systems cite?
• Are competitors mentioned more often or more favorably?
• What factual inaccuracies or reputation risks appear?
• Which pages and third-party sources are associated with visibility?
• Do AI-referred visitors engage, convert, or request a meeting?

Google recommends using Search Console to monitor performance in its generative AI features.[3] Google has also introduced generative-AI visibility controls and reporting in Search Console. Depending on availability and account rollout, website owners can review which pages appear in AI responses, where impressions occur, and how pages perform in generative AI features.[9]

For broader AI-search monitoring, teams can track model-specific mentions, cited domains, sentiment, competitor visibility, prompts, and changes over time. Tools such as Anny (https://anny.dodoxhq.com/) are designed for this wider measurement problem: understanding what AI systems say about a brand, which sources they rely on, and where visibility gaps exist across platforms such as ChatGPT, Gemini, Perplexity, Grok, Claude, Google AI Overviews, and AI Mode.

Measurement also makes experiments more honest. If a team publishes llms.txt, it can compare server-log requests, AI crawler activity, brand mentions, cited URLs, referral traffic, and conversions before and after implementation. If nothing changes, the team has evidence rather than an assumption. If something changes, it can investigate whether content, links, technical fixes, media coverage, or seasonality better explains the result.

The bottom line

Google is not contradicting itself.

Google Search Central is saying that llms.txt is not needed and is not used as a requirement or ranking mechanism for Google Search's generative AI features. Chrome Lighthouse is saying that an optional llms.txt document may help some browser agents orient themselves more efficiently, and it is testing an audit for that emerging use case.

Those statements can both be true because Google Search visibility, AI training controls, and agent usability are different objectives.

For most businesses, llms.txt belongs in the "optional experiment" category. For documentation-heavy developer products, it may be a reasonable convenience for agent-assisted research or implementation. In neither case should it replace technical SEO, useful content, accurate structured data, external credibility, semantic accessibility, or ongoing measurement.

The strongest AI-search strategy is not the one with the most special files. It is the one that makes a brand easy to understand, easy to verify, technically accessible, and genuinely useful when customers ask difficult questions.

If your team wants to move from speculation to measurement, Anny's AI search analytics platform (https://anny.dodoxhq.com/) tracks brand mentions, cited sources, competitors, sentiment, and visibility across major AI platforms. For teams that need implementation support, Anny also offers custom strategy, managed execution, ongoing audits, and team training (https://anny.dodoxhq.com/services). You can also request a free AI Visibility Audit (https://cal.com/dodox/quick-chat).

References

1. Introduction to robots.txt, Google Search Central — https://developers.google.com/search/docs/crawling-indexing/robots/intro
2. The /llms.txt file, v2 — https://llmstxt.org/
3. Optimizing your website for generative AI features on Google Search, Google Search Central — https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
4. AI features and your website, Google Search Central — https://developers.google.com/search/docs/appearance/ai-features
5. llms.txt, Chrome for Developers — https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt
6. Does llms.txt impact your AI visibility and citations? No, according to research, SE Ranking — https://seranking.com/blog/llms-txt/
7. We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read, Ahrefs — https://ahrefs.com/blog/llmstxt-study/
8. We analyzed llms.txt across 10 websites. Only two saw AI traffic increases, and it wasn't because of the file, Search Engine Land — https://searchengineland.com/does-llms-txt-matter-467740
9. New opportunities, control and insights for website owners, Google — https://blog.google/products-and-platforms/products/search/new-controls-website-owners/
10. Overview of OpenAI Crawlers, OpenAI — https://developers.openai.com/api/docs/bots
11. Build agent-friendly websites, web.dev — https://web.dev/articles/ai-agent-site-ux
12. SEO Starter Guide, Google Search Central — https://developers.google.com/search/docs/fundamentals/seo-starter-guide