People usually just talk about AI crawlers. We looked at our own server logs instead. Eleven days, August 22 to September 1, request counts by each crawler's name. The site is Kazakh, tens of thousands of pages, the topic is services
| crawler | owner | requests over 11 days | days out of 11 |
|---|---|---|---|
| ClaudeBot | Anthropic | 25,434 | 9 |
| Amazonbot | Amazon | 24,697 | 7 |
| Googlebot | Google, search | 13,391 | 11 |
| YandexBot | Yandex, search | 10,906 | 11 |
| GPTBot | OpenAI, training | 6,600 | 6 |
| meta-externalagent | Meta | 4,541 | 8 |
| OAI-SearchBot | OpenAI, search | 3,045 | 10 |
| Applebot | Apple | 2,006 | 7 |
| ChatGPT-User | chat referral | 776 | 11 |
| bingbot | Microsoft, search | 392 | 11 |
| PerplexityBot | Perplexity | 122 | 7 |
AI crawlers passed the search crawlers. ClaudeBot alone made more requests than Googlebot and YandexBot combined. A year ago this line didn't exist in the logs at all
Bing barely shows up. 392 requests against Googlebot's 13,391, 34 times fewer. For a site that wants to appear in Copilot's answers, this takes separate work: Bing follows external links, and we have almost none
ChatGPT-User is the only line here backed by a real person. It marks a click from a link inside a chat answer. 776 visits in eleven days show the model's answer already sends real people to the site
Check your `robots.txt` by name. A general rule for all crawlers doesn't mean a specific crawler is allowed. Many sites block GPTBot and ClaudeBot without noticing, just by copying someone else's file. If you want to appear in AI answers, these names need to be allowed explicitly
Check which addresses they request. A crawler pulls pages, styles, images, and scripts. Without them it can't assemble the page or show an image in an answer. A blocked styles folder costs more than it looks
And count your own numbers. You already have a server log. All you need to count are the names in the `User-Agent` line. Eleven days is enough to see the pattern