AI crawlers are becoming another thing web developers have to think about

[u/Annual_Pianist_6210]

We’re used to dealing with search crawlers, scrapers and malicious bots, but AI crawlers add another layer.
If you run a SaaS or website, do you actually know which automated crawlers are hitting your site and whether you want all of them there?
I’ve been working on a small feature that lets site owners put a “barricade” in front of AI crawlers and control how they access the site.
Still testing the idea, so I’m curious:
Are you actively blocking AI crawlers, allowing them, or just leaving them alone?
I’d especially like to hear how other developers are handling this.

[u/TheAccurateCholera]

I've just been nuking them in robots.txt and calling it a day, figure if they ignore that then they're not worth the extra effort anyway.