GEO & AI Search
Should we block AI crawlers in robots.txt?
Only if you understand which decision you are making. Blocking training crawlers such as GPTBot is a legitimate content-licensing choice with no direct search penalty. Blocking retrieval crawlers such as OAI-SearchBot removes you from ChatGPT search entirely. These are two separate decisions that most sites make with a single line.
The distinction matters commercially. Training agents (GPTBot, ClaudeBot, Google-Extended, CCBot) affect whether your content contributes to model training. Retrieval agents (OAI-SearchBot, PerplexityBot, Claude-SearchBot) determine whether you can appear in answers at all. Blocking Googlebot removes you from search and from AI Overviews eligibility, since Google gates those on ordinary indexing.
This is the most common and most expensive error we find in audits: a site blanket-blocks AI agents, then commissions work to improve AI visibility it has already made impossible.
Informational · Updated 2026-08-10
Related Services
Related Reading
Related Questions
- How do AI engines decide which businesses to cite?They draw on a trust graph rather than a ranking: whether your organisation resolves as a clear entity, whether independ…Read the full answer→
- What is GEO (Generative Engine Optimization)?GEO is the practice of making a business legible, verifiable and consistently described across the web so AI answer engi…Read the full answer→
Strategy Call
See Exactly Where You Stand.
Every relationship starts with intelligence, not a proposal. A strategy call gives you a clear picture of your AI visibility, search authority, and competitive gaps — and a realistic view of what is achievable.