I see that ElkArte doesn't use any robots.txt and in google webmaster tool Google asks me to submit robots.txt file for better indexing
Do we really need it and if so what can we allow and disallow. Any sample robots.txt of elkarte available?
I'm also interested in this argument. Here is what I have in my robots.txt (it's an old file I have created for SMF and never adapted for Elkarte)
User-agent: *
Disallow: /*sort=*
Disallow: /*msg*
Disallow: /index.php?action=activate*
Disallow: /index.php?action=admin*
Disallow: /index.php?action=calendar*
Disallow: /index.php?action=emailuser*
Disallow: /index.php?action=findmember*
Disallow: /index.php?action=help*
Disallow: /index.php?action=helpadmin*
Disallow: /index.php?action=login*
Disallow: /index.php?action=mlist*
Disallow: /index.php?action=modifykarma*
Disallow: /index.php?action=pm*
Disallow: /index.php?action=post*
Disallow: /index.php?action=printpage*
Disallow: /index.php?action=profile*
Disallow: /index.php?action=recent*
Disallow: /index.php?action=register*
Disallow: /index.php?action=reminder*
Disallow: /index.php?action=search*
Disallow: /index.php?action=unread*
Disallow: /index.php?action=unreadreplies*
Disallow: /index.php?action=verificationcode*
Disallow: /index.php?action=who*
Disallow: /index.php?action=stats*
Disallow: /index.php?theme*
Disallow: /index.php?*wap*
Disallow: /index.php?*wap2*
Disallow: /index.php?*imode*
Disallow: /index.php?*rss*
Disallow: /Sources
Great, can i use this in Elkarte now..any edits needed?
you can use it, you may also remove these lines:
Disallow: /index.php?*wap*
Disallow: /index.php?*wap2*
Disallow: /index.php?*imode*
Disallow: /index.php?*rss*
Thank you so much
@radu81