Free tool · AI crawlers
Can AI crawlers read your website?
We check robots.txt for 16 AI and search crawlers, send requests as ChatGPT and Perplexity to catch server or Cloudflare blocks, and look at whether your content is visible without JavaScript.
The check takes 5–15 seconds. We send a few requests to your site and keep nothing beyond a basic log.
Which crawlers we check
Search crawlers and user-triggered fetchers decide whether an AI assistant can cite you. Training crawlers only collect data for future models; blocking them is a business decision and doesn’t directly affect AI search.
| Search | |
|---|---|
| OAI-SearchBot | OpenAI · ChatGPT search |
| Googlebot | Google · Google Search, AI Overviews, AI Mode |
| Bingbot | Microsoft · Bing and Copilot |
| PerplexityBot | Perplexity · Perplexity search |
| Claude-SearchBot | Anthropic · Claude search |
| Applebot | Apple · Siri and Spotlight |
| DuckAssistBot | DuckDuckGo · DuckAssist answers |
| User-triggered visits | |
| ChatGPT-User | OpenAI · ChatGPT (user-triggered visits) |
| Perplexity-User | Perplexity · Perplexity (user-triggered visits) |
| Claude-User | Anthropic · Claude (user-triggered visits) |
| Gemini (training and answers) | |
| Google-Extended | Google · Gemini apps (training and grounding) |
| Model training | |
| GPTBot | OpenAI · model training |
| ClaudeBot | Anthropic · model training |
| Applebot-Extended | Apple · Apple Intelligence training |
| meta-externalagent | Meta · Meta AI training |
| CCBot | Common Crawl · open dataset used to train many models |