Skip to main content ↑ ↓
Recent Posts
91
Addons ideas and questions / Re: Updated Spider List
Last post by Spuds -
That was Karl Benson (sp?) He used to have a long list of spiders that you could add.  No idea if it has been updated, but TBH given todays internet its a full time effort to keep it updated.

There is a good overall blocker for either Apache or Nginx available here. https://github.com/mitchellkrogza Thats a bit more effort than just a robots.txt list but really blocks them early in the process.

If you just want a list of bad spiders/bots/AI agents ... here is the robot.txt file from the above. https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/blob/master/robots.txt/robots.txt I think there may be a couple of blocked ones that are arguably SEO useful, but YMMV.

I use that robot.txt file to politely ask them to not scan/scrape  (which is all that really does, ask) and then the things I have in the ElkArte spiders list are what I consider "safe" or necessary/good for SEO, just so I can see who is doing what/when. 

If you have a really problem with bots, then installing the full code from above is the way to go as it uses iptables and fail2ban among others.

I've attempted to keep an updated list of interesting ones (a.k.a probably SEO useful) for 2.0, here is the current install section
Code: [Select]
		return $this->db->insert('ignore',
			'{db_prefix}spiders',
			array('spider_name' => 'string', 'user_agent' => 'string', 'ip_info' => 'string'),
			array(
				array('Amazon', 'Amazonbot', ''),
				array('Anthropic-AI', 'anthropic-ai', ''),
				array('Anthropic-AI (Bot)', 'ClaudeBot', ''),
				array('Anthropic-AI (Claude)', 'claude-Web', ''),
				array('Apple', 'Applebot', ''),
				array('Baidu', 'Baiduspider', ''),
				array('Bing', 'bingbot', ''),
				array('Bing (Preview)', 'BingPreview', ''),
				array('CCBot', 'CCBot', ''),
				array('Diffbot', 'Diffbot', ''),
				array('DoCoMo', 'DoCoMo', ''),
				array('DuckDuckGo', 'duckduckgo', ''),
				array('DuckDuckGo (Assist)', 'DuckAssistBot', ''),
				array('Ecosia', 'Ecosia', ''),
				array('Exabot', 'Exabot', ''),
				array('Google', 'Googlebot', ''),
				array('Google (AdSense)', 'Mediapartners-Google', ''),
				array('Google (Adwords)', 'AdsBot-Google', ''),
				array('Google (Bard)', 'Google-Extended', ''),
				array('Google (Image)', 'Googlebot-Image', ''),
				array('Google (ImageProxy)', 'GoogleImageProxy', ''),
				array('Google (Mobile)', 'Googlebot-Mobile', ''),
				array('Google (News)', 'Googlebot-News', ''),
				array('Google (Video)', 'Googlebot-Video', ''),
				array('Gravityscan', 'Gravityscan', ''),
				array('InternetArchive', 'ia_archiver-web.archive.org', ''),
				array('Jakarta', 'Jakarta Commons', ''),
				array('Kraken', 'Kraken', ''),
				array('LinkedIn', 'LinkedInBot', ''),
				array('MegaIndex', 'MegaIndex.ru', ''),
				array('Meta/Facebook', 'FacebookBot', ''),
				array('Meta/Facebook', 'meta-externalagent', ''),
				array('Meta/Facebook (Hit)', 'facebookexternalhit', ''),
				array('MSN', 'msnbot', ''),
				array('MSN (Mobile)', 'MSNBOT_Mobile', ''),
				array('Omgili', 'Omgili', ''),
				array('Open-AI (Bot)', 'GPTBot', ''),
				array('Open-AI (SearchBot)', 'OAI-SearchBot', ''),
				array('Open-AI (User)', 'ChatGPT-User', ''),
				array('Perplexity (User)', 'Perplexity-User', ''),
				array('PerplexityBot (Bot)', 'PerplexityBot', ''),
				array('Slack', 'Slackbot', ''),
				array('Sogou', 'Sogou', ''),
				array('Teoma', 'teoma', ''),
				array('Tik-Tok', 'Bytespider', ''),
				array('Timpi', 'TimpiBot', ''),
				array('Twitter', 'TwitterBot', ''),
				array('Yahoo!', 'slurp', ''),
				array('Yahoo! (Blogs)', 'Yahoo-Blogs', ''),
				array('Yahoo! (Feeds)', 'YahooFeedSeeker', ''),
				array('Yahoo! (Image)', 'Yahoo-MMCrawler', ''),
				array('Yahoo! (Mobile)', 'YahooSeeker/M1A1-R2D2', ''),
				array('Yandex', 'YandexBot', ''),
				array('Yandex (Blogs)', 'YandexBlogs', ''),
				array('Yandex (Images)', 'YandexImages', ''),
				array('Yandex (Media)', 'YandexMedia', ''),
				array('Yandex (Video)', 'YandexVideo', '')
			);
	}
94
Support / Re: Antechinus needs help logging in!
Last post by Antechinus -
Oh yeah, we're thinking of moving the thing from tld to a directory of its own. I notice the existing repair_settings.php download is from 2014. Does it still work on 1.1.9?

If it's a bit long in the tooth I can always do settings manually.
96
Support / Re: Antechinus needs help logging in!
Last post by Spuds -
I would recommend that https://github.com/elkarte/Elkarte/compare/development...patch_1-1-10

I do run that on a few sites so I know its stable on 8.2+

I recall it changed a bunch of items with theme.php so it may not play nice with a custom theme that has its own theme.php file, so best to set things to default and update and then do any theme fixes as needed.

I'll attach update and full install files that I basically found on my desktop, give these a whirl
97
Support / Re: Antechinus needs help logging in!
Last post by Antechinus -
Sounds like that might be it. All good now.

Spoiler (click to show/hide)

We're going to be updating an archived board that is way out of date. Would you recommend 1.1.10 for that?
98
Support / Re: Antechinus needs help logging in!
Last post by Spuds -
I saw that and to be honest I was impressed.   I'll see if I can change that.

Yes 2.0 has a few bugs here and there, will get to working on them as some projects that have been holding me captive are "finishing"

1.1.9 was the last official release, although 1.1.10 was packaged, I just never released it.

ETA: I was able to change it, so prob just a permissions issue somewhere
100
Support / Re: Antechinus needs help logging in!
Last post by oblivion -
Thanks for that!

Antechinus says it looks like he'll need a hand-crafted artisanal password change, unfortunately.

He also asks that you edit out the partial email address now that he knows for sure which one he used to set up his account, just to be safe.

eta Hi Spuds! Yes, he'd like a password reset.