Text only | Text with Images

ElkArte Community

Extending Elk => Addons => Addons ideas and questions => Topic started by: Burke Knight on July 06, 2015, 06:03:58 am

Title: More Spiders
Post by: Burke Knight on July 06, 2015, 06:03:58 am
More Spiders (http://custom.simplemachines.org/mods/index.php?mod=1157)

Not too sure if the database entries for search engine spiders has been changed, but I'd love to be able to use this mod.
I had it in the old forum, but I guess the converter did not do them.

Any chance of getting this mod ported over?
Title: Re: More Spiders
Post by: emanuele on July 06, 2015, 10:17:50 am
I wrote a quick scheduled task to fetch the list from http://user-agents.org/ but is slightly overwhelming... LOL
The final table is 1000+ entries and most are probably no more relevant.

This one may be interesting as well:
https://github.com/monperrus/crawler-user-agents/blob/master/crawler-user-agents.json
though it lacks some info that would be nice to show.
TL;DR: it may be nice to have a repo to add to the list...
Title: Re: More Spiders
Post by: Burke Knight on July 06, 2015, 11:08:20 am
I'm not certain what the license is on the SMF mod, nor if it's on GH.
I can't PM Sleepy to ask, either. :P LOL
Title: Re: More Spiders
Post by: Spuds on July 07, 2015, 07:23:40 am
I guess the more spiders list is a bit outdated,  it was last updated in 08?  Some are still right, some are gone, some have changed the crawler id.    The list emanuele linked to looks like an interesting updated to the list.
Text only | Text with Images