Training crawler
A training crawler is an AI crawler that gathers web content to train future models, rather than to answer a live question. OpenAI's GPTBot is the best-known example.
Last updated
What this means for your store
Blocking training crawlers keeps your content out of model training, which some brands choose for control or copyright reasons. Important nuance: it does not remove you from live AI answers, because those are fetched by a different, retrieval crawler. Blocking training and blocking search are two separate decisions.