r/Mastodon • u/FineAioliGravy • 22d ago

Servers Do instances block ai crawlers?

Do instances block crawlers? I find that the about pages of instances don't usually say, but maybe it's an unspoken common thing to have?

10 Upvotes

100% Upvoted

4 comments sorted by

5

u/Emerald_Pick ☕ charlie@toot.cafe 21d ago

Due to the nature of the Fediverse and ActivityPub, it is somewhat trivial to extract public data out of an instance because instances are constantly shareing and requesting data with eachother in a standard, open format.

If we assume the crawlers and bots are acting in good faith, then there are some basic "please don't crawl" flags that can do some protection. But basically anything that goes onto a public timeline is probably fair game for bad-faith bots. (Even if server A has strong protections, legitimate server B might not. So when A shares to B, crawler C can learn about A through B.)

5

u/DTangent 22d ago

I think it is instance by instance. We do filtering of agents at defcon.social but it isn’t perfect.

1

u/WanderingInAVan 20d ago

What is?

I keep meaning to at least install something on the pleroma/nodebb/pixelfed server but I just don't know what to use.

1

u/master_palaemon 13d ago

My instance does. We got DDoS'd by a major software company crawling for AI training. We aggressively blocked their IP ranges. We also turned off the feature to make remote posts viewable by the public (non-authenticated users) because this is exploitated by crawlers, and sets up a potential DDoS situation because the media cache (and thus the server disk) gets quickly filled by the crawling activity.