robots.txt for AI crawlers
robots.txt is a plain text file at the root of your site that tells crawlers which parts they may fetch. For AI visibility, it decides whether AI crawlers like GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot are allowed in at all.
How it works
Each rule in the file names a crawler and what it may or may not read. If a store disallows the AI crawlers, or a security layer blocks them, those systems never see the store, so it cannot appear in their answers.
A commerce example
This is a surprisingly common way to be invisible. Cloudflare has blocked AI crawlers by default for domains added since mid-2025, and several security plugins have an "AI bots" toggle that does the same, so a store can be shut out of AI answers without the owner ever choosing that.
What this means for your store
Open yourstore.com/robots.txt and check that GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot are not disallowed, then check your firewall or security plugin for an AI-bot setting. Blocking training crawlers is a fair choice; just keep the search-related ones allowed so you stay in AI answers.