ooon.ai
all articles

Who actually visits the Kazakh site: 11 days of server logs

People usually just talk about AI crawlers. We looked at our own server logs instead. Eleven days, August 22 to September 1, request counts by each crawler's name. The site is Kazakh, tens of thousands of pages, the topic is services

crawlerownerrequests over 11 daysdays out of 11
ClaudeBotAnthropic25,4349
AmazonbotAmazon24,6977
GooglebotGoogle, search13,39111
YandexBotYandex, search10,90611
GPTBotOpenAI, training6,6006
meta-externalagentMeta4,5418
OAI-SearchBotOpenAI, search3,04510
ApplebotApple2,0067
ChatGPT-Userchat referral77611
bingbotMicrosoft, search39211
PerplexityBotPerplexity1227

Three things you see right away

AI crawlers passed the search crawlers. ClaudeBot alone made more requests than Googlebot and YandexBot combined. A year ago this line didn't exist in the logs at all

Bing barely shows up. 392 requests against Googlebot's 13,391, 34 times fewer. For a site that wants to appear in Copilot's answers, this takes separate work: Bing follows external links, and we have almost none

ChatGPT-User is the only line here backed by a real person. It marks a click from a link inside a chat answer. 776 visits in eleven days show the model's answer already sends real people to the site

What this means for a site owner

Check your `robots.txt` by name. A general rule for all crawlers doesn't mean a specific crawler is allowed. Many sites block GPTBot and ClaudeBot without noticing, just by copying someone else's file. If you want to appear in AI answers, these names need to be allowed explicitly

Check which addresses they request. A crawler pulls pages, styles, images, and scripts. Without them it can't assemble the page or show an image in an answer. A blocked styles folder costs more than it looks

And count your own numbers. You already have a server log. All you need to count are the names in the `User-Agent` line. Eleven days is enough to see the pattern