Retrieval crawler

A retrieval crawler, also called a search crawler, is an AI crawler that fetches pages to answer a shopper's question right now, such as OpenAI's OAI-SearchBot or PerplexityBot. Blocking it removes you from those live AI answers.

Last updated

What this means for your store

This is the crawler that actually decides whether you can appear in AI shopping answers, so it is the one to keep allowed even if you block training crawlers. Check your robots.txt and firewall so the retrieval crawlers are not accidentally shut out.

See where your own store stands.

One product link, about a minute, and every finding shows its evidence.