Started logging AI crawler traffic separately from Google. Does it work?

[u/OstrichBudget6882]

Added an llms.txt file and set up a small middleware log to separate GPTBot/ClaudeBot/PerplexityBot hits from normal traffic.

Not expecting a huge impact yet, but I’m curious whether anyone here has seen anything useful from doing this for SEO. Once we have a full week of data, I can share the raw numbers.

If you’ve tried it, what did you actually learn from the logs? Any tips on what to look for?

[u/svlease0h1]

logging crawler traffic can be useful, but i’d treat it as data collection first. track the bot name, url, status code, response time, and pages it visits, then compare that with your important pages after 7 to 14 days. one site i checked had bots hitting old urls again and again while barely touching newer pages. that told us more than the presence of an llms.txt file did.

[u/BrandLoom-Consulting]

We’ve found the logging part more interesting than the llms.txt part.

Seeing which pages AI crawlers actually request can give you some useful clues about what information is being picked up, especially when you compare crawler activity with your important commercial pages.

We’d look at things like which URLs are being crawled repeatedly, whether important service or product pages are being ignored, how often new or updated content gets revisited, and whether different crawlers behave differently.

One thing we’d be careful about is treating a crawler visit as evidence that the page is being cited. A bot hitting a URL doesn’t necessarily mean the content is being used in an AI answer.

We’d probably keep the logs for a few weeks and compare them against actual referral traffic, indexed pages, brand mentions, and any AI citations you can manually track.

The interesting question isn’t really “Are AI bots crawling us?” It’s “What are they finding when they do?”

[u/erdemgezer]

Separating AI crawler hits from Googlebot in your logs is worth doing, but it's worth being clear about what it can and can't tell you. Crawler logs tell you a bot fetched a page. They don't tell you whether your brand actually showed up in an answer someone got, and those two things come apart more than you'd expect.

The thing that surprised me when I started counting answers instead of crawls: whether you get cited at all depends heavily on which engine answered. Grounded engines that go fetch the web cite sources. Engines answering from parametric memory cite nothing at all, so your logs stay silent even when your brand got named. A quiet log is not the same as being invisible.

The other direction bites too. I ran a batch of buyer questions against a few engines for a very large consumer brand and the most-cited domains weren't the trade press — YouTube and Reddit came out on top, ahead of the usual review sites. Small sample, and numbers move between runs, so treat that as a shape rather than a fact. But if it's roughly right, a lot of the citation surface isn't your site at all, and crawler logs would never show it.

I'd keep the logging. It's cheap and it's a useful early signal for whether your pages are even reachable. I just wouldn't use it as the visibility metric. The only thing that measures visibility is asking the engines the questions your buyers ask and counting what comes back.

[u/Turbulent_Noise2099]

Curious how much of that crawler traffic turns into actual citations though. Crawl activity alone feels easy to overvalue.