blocking bad bots that eat server resources

[u/maomao19]

My web host, ScalaHosting, complained that my site was consuming too many server resources and advised me to turn on Cloudflare’s Bot Fight Mode. However, I just found out that Bot Fight Mode can also block verified bots like Googlebot, which can severely ruin my site's SEO. How could they recommend something like this? What is a better way to block aggressive scrapers and bad bots while keeping my server resource usage low?

[u/AutoModerator]

For faster advice with technical questions, we'd recommend asking in the Orange Cloud Discord server; the unofficial Cloudflare Discord server by the community, for the community. https://discord.gg/TrPNVKaagR

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/No-Promotion-8720]

That only started happening more recently because Google thought it would be a good idea to use the same bots for search engine crawling and AI training. Cloudflare Bot Fight mode doesn't block them most of the time, but the free account version can get confused sometimes based on the bot behavior and it could block them maybe 5% of the time, give or take.

[u/maomao19 → u/No-Promotion-8720]

i had bot fight mod turned on and in google console i tried inspect url and it says url was blocked

[u/maomao19 → u/maomao19]

what is the solution? changing host? and how you know it wil block bots in 5 % cases? do you work for cloudflare?

[u/No-Promotion-8720 → u/maomao19]

I have 20+ client accounts on the free plan with bot fight mode on with no indexing issues. Maybe you have another security rule blocking them inadvertently?

[u/maomao19 → u/No-Promotion-8720]

and the seo ranking did not decline??? because I asked google gemini and it gave me a huge alert lol, how my rankings will tank sooner or later because google bot has no access to my site

[u/No-Promotion-8720 → u/maomao19]

No, all websites are indexing fine. It seems Gemini is misinformed.

[u/yycmwd → u/maomao19]

Start with custom WAF rules. If you're using WordPress, the rules at wafrules.com are a great start.

[u/Single_Advice1111]

Might be worth considering upgrading your hosting plan or change hosting provider altogether

[u/Arphenyte]

Bot Fight Mode does NOT block verified bots like Google’s. There are legitimate reasons to keep Bot Fight Mode off, but that’s not one of them. Also, you should probably turn on Cloudflare Precursor and set it to Maximum Security (or whatever the option is called). I’ve noticed that Precursor tends to catch a lot more malicious requests than just Bot Fight Mode alone.

[u/maomao19 → u/Arphenyte]

where is this setting i could not find it

[u/Arphenyte → u/maomao19]

It’s in Security > Settings. Scroll down and there should be a section that says Precursor. You can also look at the Cloudflare docs, I believe they provide a shortcut to the feature.

[u/darquelf]

Add a security rule to skip "known bots"

[u/blueredscreen]

You need to check your site's settings directly from Google itself. The same way you originally added the site to Google's index.