Skip to main content ↑ ↓
Tips for Bots Started by vbgamer45 · · Read 197 times 0 Members and 1 Guest are viewing this topic. previous topic - next topic

Tips for Bots

Tips for Bots
Use cloudflare for geo blocking of countries/asn's works great. You can also challenge users instead of block if you are concerned.
Code: [Select]
(ip.src.country eq "CN") or (ip.src.country eq "HK") or (ip.src.country eq "VN") or (ip.src.country eq "BR") or (ip.src.country eq "AR") or (ip.src.country eq "EC") or (ip.src.country eq "UY") or (ip.src.country eq "IR") or (ip.src.country eq "SG") or (ip.src.country eq "IQ") or (ip.src.country eq "BD") or (ip.src.country eq "VE") or (ip.src.country eq "CL") or (ip.src.country eq "PY") or (ip.src.country eq "MX") or (ip.src.country eq "PA") or (ip.src.country eq "BG") or (ip.src.asnum eq 136907) or (ip.src.country eq "SN")

Block old chrome versions or challenge if using cloudflare and block empty user agents
For apache httpd.conf below
Code: [Select]
# Block empty user agents
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^$ [NC]
RewriteRule .* - [F,L]

# Block Chrome below 120
RewriteCond %{HTTP_USER_AGENT} Chrome/([1-9][0-9]|10[0-9]|11[0-9])\. [NC]
RewriteRule .* - [F,L]
Clouldflare block chrome
Code: [Select]
(http.user_agent contains "Chrome/100." or http.user_agent contains "Chrome/101." or http.user_agent contains "Chrome/102." or http.user_agent contains "Chrome/103." or http.user_agent contains "Chrome/104." or http.user_agent contains "Chrome/105." or http.user_agent contains "Chrome/106." or http.user_agent contains "Chrome/107." or http.user_agent contains "Chrome/108." or http.user_agent contains "Chrome/109." or http.user_agent contains "Chrome/110." or http.user_agent contains "Chrome/111." or http.user_agent contains "Chrome/112." or http.user_agent contains "Chrome/113." or http.user_agent contains "Chrome/114." or http.user_agent contains "Chrome/115." or http.user_agent contains "Chrome/116." or http.user_agent contains "Chrome/117." or http.user_agent contains "Chrome/118." or http.user_agent contains "Chrome/119." or http.request.uri.query contains "action=printpage" or http.request.uri.path contains "printpage")


Turn off certain forum features for guests.

Make sure your site supports HTTP2 for your webserver.

Tweak your PHP/Database settings. User latest versions.

But in generally tweak, all settings, from webserver, php, database. The defaults are not enough for bigger sites.
Last Edit: February 26, 2026, 08:44:49 pm by vbgamer45
ElkarteMods.com - Addons, Products and more!

Re: Tips for Bots

Reply #1 –

Glad I'm not alone on this I've been in bot fighting mode on my sites for the last few days!

Other items that may help, depending on your site, traffic, location, etc.

Many requests are coming in coming in on groups of ipv4 /16, which is a group of ~65.5K address (xxx.xxx.123.123). For my sites that is not normal traffic but YMMV.  I wrote a script that grouped those /16 hits (from the access log) and if it finds more than xx IP's in a group (i use 10) in the last 15min's then I write it to a log file and use fail2ban to block that entire xxx.xxx.0.0 sub (use ipsec).  If you have some really small local group you can whitelist that sub. I now have over 400 of those subs blocked.

nginx has geoip2 (via max mind) so you can use that to GeoIP fence to countries and block ones you know are not in your zone. I know some folks take issue with that but honestly to bad, you have to work through an attack!  I will say, however, most of the bot traffic was out of US address (proxies) so Virginia TX and WA were common whois endpoints, but still that drops some of the crap.

Last thing that can be helpful is bots tend to flood on connection attempts.  Another script, this one groups connection limit failures (from the error.log) over a given limit/time threshold that also have PHPSESSION in the url and -> ban.  Guests are not opening 30+ connections to login or browse a site, and to be honest even with cache off and trying to beat on a site from your own IP, you will not trigger that either.

I may add that low chrome version check, more bot pain! I've seen high values but those are from variants (vivaldi for example), but i did not consider old cruft thanks for the idea!
 

Re: Tips for Bots

Reply #2 –

Yeah it was wrecking me and I had hardware firewall/software firewall, had to pull out all the stops.  Did switch to cloudflare as well which helped a ton.
I have done so much apache/fastcgi tweaking along with the database to handle the loads.    The worst is when I would get hit 50k to 100k bots at one time.
I do a lot of research the asn's i use ip2location.co.m If you use cloudflare be careful with your mx record proxing/email sending I run reports via https://mxwhiz.com/ to do double check  (mine btw)

Downside of blocking older chrome versions is windows 7 users would be cut off if they used chrom.