Re: Robots.txt needed for a site?
Reply #1 –
I'm also interested in this argument. Here is what I have in my robots.txt (it's an old file I have created for SMF and never adapted for Elkarte)
User-agent: *
Disallow: /*sort=*
Disallow: /*msg*
Disallow: /index.php?action=activate*
Disallow: /index.php?action=admin*
Disallow: /index.php?action=calendar*
Disallow: /index.php?action=emailuser*
Disallow: /index.php?action=findmember*
Disallow: /index.php?action=help*
Disallow: /index.php?action=helpadmin*
Disallow: /index.php?action=login*
Disallow: /index.php?action=mlist*
Disallow: /index.php?action=modifykarma*
Disallow: /index.php?action=pm*
Disallow: /index.php?action=post*
Disallow: /index.php?action=printpage*
Disallow: /index.php?action=profile*
Disallow: /index.php?action=recent*
Disallow: /index.php?action=register*
Disallow: /index.php?action=reminder*
Disallow: /index.php?action=search*
Disallow: /index.php?action=unread*
Disallow: /index.php?action=unreadreplies*
Disallow: /index.php?action=verificationcode*
Disallow: /index.php?action=who*
Disallow: /index.php?action=stats*
Disallow: /index.php?theme*
Disallow: /index.php?*wap*
Disallow: /index.php?*wap2*
Disallow: /index.php?*imode*
Disallow: /index.php?*rss*
Disallow: /Sources