Our forum was being knocked down by lots of spiders, now it's being knocked down by the bad behavior module.
The mariadb processlist looks like this:
https://gist.github.com/kode54/70577e3f67c430442675c456314484ba
select count(*) shows there are 275147 entries, and the date of the first entry is from 6 days ago.
E: I see there is a toggle. Good.
Speaking of errors, our error log has over a million entries, after just a month.
Glad you found the toggle, that is alot of spider traffic for sure. I had some issues with traffic a while back and ended up setting up a couple of Nginx connection and request limits which if you exceeded tossed that IP to fail2ban for a short while. Anyway something to consider looking into, I'd imagine there is something like that for Apache as well.
If you could summarize the main contributors to the log spam that would help, with that many entries there is probably just 1 or 2 things causing the log spam, usually from an addon, you know the drill.
I stupidly deleted the log, but then I turned on logging again and received 75 pages worth of log items within about a minute, all:
Details: Header 'Referer' present but blank
All various Chinese language spiders.
We’re already using Caddy to block a large amount of requests. Prior to that, we would have 1500-6000 “guests” around the clock.
Your forum must be into the spider-verse. :D
Oh yeah, I thought I’d mention, rate limiting per ip probably won’t help much, since they seem to be rotating through a whole lot of random ip addresses.
They did that to one of my sites recently. Hundreds of bots at a time. They scraped a couple of days, disappeared a few, then came back for a couple days. There is probably now a Chinese duplicate somewhere.
Indeed, that makes things more difficult when they have access to a large set of IP's .... I've had some "success" setting up limit_req_zone for number of simultaneous connections from a single IP which at least slows what a single IP can do, and then both req_limit and php_limit, so if they are just moving from page to page to page the rate limit will also trip. I only do a short ban time as well, just slows things down but can't stop it.
Its always a balance to find the right numbers so you don't get legit users but still slow down the bots. When my sites are being trolled like that I turn on the limits for a few days and go back to normal. Of course there is always cloudflare as well.
Unfortunately, Cloudflare is not an option for me, as I do not control the domains we’re using. I would need to coordinate with the current owner of the domains if I ever want to try placing Cloudflare in front of the site.