Re: Disable bad behavior logging
Reply #1 –
Glad you found the toggle, that is alot of spider traffic for sure. I had some issues with traffic a while back and ended up setting up a couple of Nginx connection and request limits which if you exceeded tossed that IP to fail2ban for a short while. Anyway something to consider looking into, I'd imagine there is something like that for Apache as well.
If you could summarize the main contributors to the log spam that would help, with that many entries there is probably just 1 or 2 things causing the log spam, usually from an addon, you know the drill.
Re: Disable bad behavior logging
Reply #2 –
I stupidly deleted the log, but then I turned on logging again and received 75 pages worth of log items within about a minute, all:
Details: Header 'Referer' present but blank
All various Chinese language spiders.
We’re already using Caddy to block a large amount of requests. Prior to that, we would have 1500-6000 “guests” around the clock.
Re: Disable bad behavior logging
Reply #4 –
Oh yeah, I thought I’d mention, rate limiting per ip probably won’t help much, since they seem to be rotating through a whole lot of random ip addresses.
Re: Disable bad behavior logging
Reply #5 –
They did that to one of my sites recently. Hundreds of bots at a time. They scraped a couple of days, disappeared a few, then came back for a couple days. There is probably now a Chinese duplicate somewhere.
Re: Disable bad behavior logging
Reply #7 –
Unfortunately, Cloudflare is not an option for me, as I do not control the domains we’re using. I would need to coordinate with the current owner of the domains if I ever want to try placing Cloudflare in front of the site.