AI SEO checklist: llms.txt, structured data and AI crawlers explained
There is no shortage of advice about what your website “must have” to be visible in AI search. Here is a sober checklist: what actually affects whether ChatGPT, Copilot, Perplexity and Google can find and use your content – and what you can safely skip.
In this article
- 1. Let the right crawlers in
- 2. Check the generative AI setting in Search Console
- 3. Make sure pages can be indexed and shown with snippets
- 4. Keep important text in HTML
- 5. Structured data: useful, not magic
- 6. llms.txt: what it is and whether you need it
- 7. Speed and stability
- 8. Clear structure and internal links
- 9. One page per service and location – but not a hundred
- 10. Fresh information
- 11. Measure the effect
- The checklist in one place
- Frequently asked questions
There is no shortage of advice about what your website “must have” to be visible in AI search. Some of it is right, much of it is guesswork. This checklist sticks to what the AI companies themselves document, and says clearly when something is uncertain.
1. Let the right crawlers in
AI companies use different crawlers for different purposes. The key distinction is between crawlers that fetch content for search and crawlers that collect text for training models. You can let one in and keep the other out.
| Crawler | Company | Used for | If you block it |
|---|---|---|---|
| Googlebot | Google Search, including AI Overviews and AI Mode | You disappear from Google | |
| Google-Extended | Gemini training and use as a source in the Gemini app | Does not affect Google Search or AI Overviews, but may make you less visible in the Gemini app | |
| Bingbot | Microsoft | Bing and Microsoft Copilot | You disappear from Bing and weaken your visibility in Copilot |
| OAI-SearchBot | OpenAI | ChatGPT search | You are not shown in ChatGPT search answers |
| GPTBot | OpenAI | Training OpenAI’s models | Your content is not used for training. Does not affect ChatGPT search |
| ChatGPT-User | OpenAI | Visits a user asks ChatGPT to make | robots.txt may not apply |
| PerplexityBot | Perplexity | Perplexity search | You are not shown as a source in Perplexity |
| Claude-SearchBot | Anthropic | Search in Claude | May make you less visible in Claude’s search answers |
| ClaudeBot | Anthropic | Training Claude | Your content is not used for training |
For most businesses that want to be found, the simplest option is to let all of these in. If you do not want your content used for training, you can block GPTBot and ClaudeBot without affecting search. In robots.txt it can look like this:
# Search crawlers are allowed by default.
# Optional: stop collection for training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
WordPress lets all crawlers in by default. So check security plugins, firewalls and services such as Cloudflare in particular, which have their own settings for blocking AI bots. It is easy to lock out search crawlers without knowing it.
2. Check the generative AI setting in Search Console
Search Console now has a control for Google’s generative AI features (Search generative AI) under Settings. There you choose whether your content can appear in AI Overviews, AI Mode and AI features in Discover. By default your site is included. According to Google’s help pages, the choice does not affect regular rankings or AI training – the latter is controlled by Google-Extended.
3. Make sure pages can be indexed and shown with snippets
Google only uses pages that are indexed and can be shown with a snippet. Check that important pages have no noindex, are not blocked in robots.txt, and do not use nosnippet or a very low max-snippet without reason. Use URL Inspection in Search Console to see how Google sees the page.
4. Keep important text in HTML
Crawlers read text. If your prices are only in an image, your services only in a PDF, or your content loads through heavy scripts after the page opens, it may not be read. Put the key information in plain text on the page: what you offer, where, for whom, the price and contact details.
5. Structured data: useful, not magic
Structured data (schema.org) is code that tells machines what content is: a business, an article, a service or a question with an answer. Google writes in its guide to generative AI features that structured data is not required and that there is no special markup for AI. It is still useful, because it makes information unambiguous and can enable rich results.
Prioritise these types:
- Organization or LocalBusiness with name, logo, contact details and area served.
- Article on articles, with author and date.
- BreadcrumbList for a clear structure.
- FAQPage where you genuinely have questions and answers. Google only shows FAQ rich results for well-known government and health websites, but the markup makes the content clearer.
In WordPress, an SEO plugin such as Rank Math can add most of this automatically.
6. llms.txt: what it is and whether you need it
llms.txt is a standard proposed in 2024: a simple text file in the root of your site that summarises the content for language models, a bit like a table of contents. The idea is good, but support is thin.
- Google writes in its guide that Google Search does not use llms.txt, and that the file neither helps nor harms search visibility.
- Google’s John Mueller has compared the file to the old meta keywords tag, which search engines stopped using long ago.
- Google’s Lighthouse tool checks for an llms.txt from version 13.3, in an experimental category for AI agents that use the browser. That is about agents, not rankings (Search Engine Journal, May 2026).
Our view: if your SEO plugin can generate the file automatically, you might as well switch it on. But do not spend time on it until everything else on this list is in place.
7. Speed and stability
Slow pages are bad for visitors, and AI tools that fetch pages in real time may give up if a page takes too long. Fast hosting, compressed images and few heavy plugins help both people and machines.
8. Clear structure and internal links
One H1 per page, logical headings below it, and internal links between related pages make it easier for both Google and AI tools to understand what you are good at. Link from articles to service pages, and the other way round.
9. One page per service and location – but not a hundred
If you offer several services or operate in several places, each deserves its own well-written page. But avoid creating many near-identical pages for every possible search variation. Google treats that as spam (scaled content abuse) when it is done to manipulate rankings, including when the pages are made with AI.
10. Fresh information
Show when content was last updated, and keep prices, opening hours and offers current. AI tools answering “what does it cost” and “are they open” need information they can trust.
11. Measure the effect
- Search Console: Use the generative AI performance report, and follow impressions and clicks for web search.
- Google Analytics 4: Create an exploration or segment where the traffic source contains
chatgpt,perplexity,copilot,geminiorclaudeto see traffic from AI assistants. - Server logs or Cloudflare: Check whether OAI-SearchBot, PerplexityBot and the others actually visit your site.
The checklist in one place
- robots.txt lets in Googlebot, Bingbot, OAI-SearchBot and PerplexityBot
- No firewall or CDN setting blocks AI search crawlers
- The generative AI setting in Search Console is set the way you want
- Important pages are indexed, without noindex or nosnippet
- Prices, services and contact details are in plain text
- Organization or LocalBusiness, Article and BreadcrumbList are in place
- The site is registered in Bing Webmaster Tools
- Pages load fast on mobile
- One well-written page per service
- Update dates and current prices
- Measurement in Search Console and Google Analytics
- llms.txt: optional, low priority
Sources: Google Search Central (guide to generative AI features, July 2026; AI features and your website, December 2025; Search Console help on the generative AI control), OpenAI and Perplexity (crawler documentation), Anthropic (help pages on crawlers) and Search Engine Journal (May 2026).
Frequently asked questions
Should I block GPTBot?
That is a business decision. GPTBot collects text for training OpenAI’s models. Blocking it does not affect your visibility in ChatGPT search, which uses OAI-SearchBot.
Do I need llms.txt to be visible in AI search?
No. Google Search does not use the file, and it is not a requirement for visibility. It does no harm, but it is a low priority.
Does FAQ schema help in AI search?
Not directly. Google says structured data is not required for its AI features. What helps is that the questions and answers are actually on the page. The markup just makes them clearer to machines.
How do I know if AI crawlers visit my website?
Check your server logs or the traffic analytics in a service such as Cloudflare, where crawlers show up with names like OAI-SearchBot, GPTBot, PerplexityBot and ClaudeBot.
Does WordPress block AI crawlers by default?
No. WordPress lets all crawlers in by default. It is security plugins, firewalls and services such as Cloudflare you should check.
Modestas Urbonas is the founder of Symbio Partners in Bergen and holds a bachelor’s degree in marketing management from BI Norwegian Business School. He works daily with SEO, Google Ads and websites for businesses in Norway.
Want a technical AI check of your website?
We review robots.txt, Search Console, structured data and your business information, and test what AI tools say about you. Fixed price.