21
Chit Chat / Re: Tips for Bots
Last post by Spuds -Other items that may help, depending on your site, traffic, location, etc.
Many requests are coming in coming in on groups of ipv4 /16, which is a group of ~65.5K address (xxx.xxx.123.123). For my sites that is not normal traffic but YMMV. I wrote a script that grouped those /16 hits (from the access log) and if it finds more than xx IP's in a group (i use 10) in the last 15min's then I write it to a log file and use fail2ban to block that entire xxx.xxx.0.0 sub (use ipsec). If you have some really small local group you can whitelist that sub. I now have over 400 of those subs blocked.
nginx has geoip2 (via max mind) so you can use that to GeoIP fence to countries and block ones you know are not in your zone. I know some folks take issue with that but honestly to bad, you have to work through an attack! I will say, however, most of the bot traffic was out of US address (proxies) so Virginia TX and WA were common whois endpoints, but still that drops some of the crap.
Last thing that can be helpful is bots tend to flood on connection attempts. Another script, this one groups connection limit failures (from the error.log) over a given limit/time threshold that also have PHPSESSION in the url and -> ban. Guests are not opening 30+ connections to login or browse a site, and to be honest even with cache off and trying to beat on a site from your own IP, you will not trigger that either.
I may add that low chrome version check, more bot pain! I've seen high values but those are from variants (vivaldi for example), but i did not consider old cruft thanks for the idea!
