Search engines and AI assistants read a website in much the same way. Automated programs find your pages, fetch them, pull out the words and the structure, and store or summarize what they find. Nothing in that process rewards tricks. It rewards pages that say clearly what the business does, in HTML that carries the full text, with facts that match from page to page. A site that a careful stranger can understand quickly is a site a machine can read and repeat accurately.

How do search engines crawl and index a website?

Google describes the process plainly. Google Search works in three stages: crawling, indexing, and serving search results. In crawling, automated programs called crawlers download text, images, and videos from pages Google has found. In indexing, Google analyzes that content and stores the information in its index. In serving, Google returns information that is relevant to a person's search.

Two points from that page matter for owners. First, Google says that during the crawl it renders the page and runs any JavaScript it finds using a recent version of Chrome. Second, Google says it does not guarantee that it will crawl, index, or serve any page, even one that follows its own guidelines. Being readable is the entry ticket. It is not a promise of a result.

What do robots.txt and sitemaps actually do?

A robots.txt file sits at the root of a site and gives instructions to crawlers. Google's documentation says robots.txt tells crawlers which URLs they can access, and that it is used mainly to avoid overloading a site with requests. Google is explicit that it is not a way to keep a page out of Google. For that, Google points to a noindex rule or password protection.

A sitemap is a file that lists the pages and files on a site. Google says a sitemap helps search engines discover URLs, but that it does not guarantee every item in it will be crawled and indexed. We treat both files as basic housekeeping. They help a crawler work efficiently. They do not make a weak page strong.

How does ChatGPT find and cite websites?

AI companies publish documentation for their crawlers, and the details matter. OpenAI's documentation names separate user agents with separate jobs. OAI-SearchBot is used to surface websites in ChatGPT's search features. GPTBot crawls content that may be used to train OpenAI's foundation models. OpenAI says each setting is independent of the others. It also says that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.

There is a third agent. When a ChatGPT user asks a question, ChatGPT may visit a page with the ChatGPT-User agent. OpenAI says that because these actions are initiated by a user, robots.txt rules may not apply.

Which AI crawlers can a site owner control in robots.txt?

Anthropic lists three agents in its guidance for site owners. ClaudeBot collects content that may be used for training. Claude-SearchBot improves search results. Claude-User fetches pages when a person asks Claude a question. Anthropic states that its bots honor robots.txt directives, and that disabling Claude-User prevents its system from retrieving a site's content in response to a user query.

Perplexity's crawler documentation says PerplexityBot surfaces and links websites in Perplexity's search results, and is not used to crawl content for AI foundation models. Its user-initiated fetcher, Perplexity-User, works differently. Perplexity says that since a user requested the fetch, this fetcher generally ignores robots.txt rules.

Google keeps one control for Search. Its page on AI features says robots.txt directives for Googlebot are how site owners manage crawling for Search, and it points to a separate token, Google-Extended, for AI training and grounding in some of Google's other systems.

The practical reading is simple. Training, search, and user-requested fetches are separate decisions, and each company handles them in its own way. Whether to allow each one is a business choice. Blocking a search crawler can keep a company out of that assistant's answers. OpenAI and Perplexity both describe their crawler settings as independent, so blocking a training crawler is a separate choice from blocking a search crawler. We set these rules on purpose and write them down, rather than inheriting whatever a plugin or a host chose.

Do AI Overviews need special files or markup?

No. Google is direct about this. On its AI features page, Google says there are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary. To be eligible as a supporting link, a page must be indexed and eligible to be shown in Google Search with a snippet. Google adds: "You don't need to create new machine readable files, AI text files, or markup to appear in these features."

What is llms.txt?

llms.txt is a file some sites now publish to summarize their content for AI tools. Its own site describes it as a proposal to standardise on using an /llms.txt file. It is a community proposal, not a formal web standard. Google's messages differ by product. Search says no AI text files are needed for its AI features, while Chrome's experimental Lighthouse agentic browsing audit does check for one. Our view is that an accurate llms.txt does no harm. It is a courtesy, not a source of visibility, and it never replaces clear pages.

Can AI crawlers read JavaScript?

For Google, the answer is on record: it renders pages and runs JavaScript during the crawl. For other crawlers, we make no assumption. Instead, we remove the question. When a page is rendered on the server, the HTML that arrives already contains every heading, paragraph, price list, and link. Any reader gets the full page on the first request, whether it runs scripts or not. Interactive features can still use JavaScript. The words that describe the business should not depend on it.

Does schema markup help AI assistants understand a page?

Schema markup is structured data that labels what a page contains: an organization, a service, an article, a set of questions and answers. Google says it uses structured data to understand the content of the page. Google's structured data guidelines also say the markup must be a true representation of the page content, and warn against marking up content that is not visible to readers.

Two limits follow. Google says there is no special schema.org markup needed for its AI features. And we do not claim to know how each AI assistant uses schema. So we add structured data where it is accurate and useful, validate it, and make sure it says exactly what the visible page says. It supports clear writing. It does not stand in for it.

How do you make a website easier for AI assistants to understand?

The same habits that help a buyer help a machine. These are the ones we apply on every site:

  • One clear purpose per page. Each important page answers one set of questions, with headings that follow the order a buyer asks them.
  • Direct answers near the top. The first paragraph says what the page is about and answers the main question. Detail comes after.
  • Consistent facts across pages. The company name, services, credentials, and process read the same everywhere. Conflicting facts give a summary nothing solid to repeat.
  • Server-rendered HTML. The full text is in the page the server sends.
  • Accurate structured data. Markup matches the visible text, and nothing more.
  • Deliberate crawler rules and a current sitemap. Access is a decision, not an accident.

None of this guarantees a mention or a position. Google's own guidance says no one can guarantee a #1 ranking on Google, and the same honesty applies to AI answers. What a business controls is whether its site is easy to read and right in what it says. The same facts feed any tool that speaks for the business, including an on-site assistant, which is why we set rules for what an AI concierge should and should not do.

What to do next

Start by reading your own site as a crawler would. View the page source of your most important pages and check whether the words are there. Compare the facts on your home page, service pages, and contact page. Check what your robots.txt allows. If the structure itself is the problem, our guide to deciding whether to rebuild or repair your website sets out how we make that call.

If you want this done as part of the site, our search and AI visibility practice covers crawling, structure, structured data, and crawler rules. You can also read how we run an engagement in our approach.