Skip to main content
Topic: Robots.txt needed for a site? (Read 3972 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Robots.txt needed for a site?

I see that ElkArte doesn't use any robots.txt and in google webmaster tool Google asks me to submit robots.txt file for better indexing

Do we really need it and if so what can we allow and disallow. Any sample robots.txt of elkarte available?

Re: Robots.txt needed for a site?

Reply #1

I'm also interested in this argument. Here is what I have in my robots.txt (it's an old file I have created for SMF and never adapted for Elkarte)
 
Code: [Select]
  
User-agent: *
Disallow: /*sort=*
Disallow: /*msg*
Disallow: /index.php?action=activate*
Disallow: /index.php?action=admin*
Disallow: /index.php?action=calendar*
Disallow: /index.php?action=emailuser*
Disallow: /index.php?action=findmember*
Disallow: /index.php?action=help*
Disallow: /index.php?action=helpadmin*
Disallow: /index.php?action=login*
Disallow: /index.php?action=mlist*
Disallow: /index.php?action=modifykarma*
Disallow: /index.php?action=pm*
Disallow: /index.php?action=post*
Disallow: /index.php?action=printpage*
Disallow: /index.php?action=profile*
Disallow: /index.php?action=recent*
Disallow: /index.php?action=register*
Disallow: /index.php?action=reminder*
Disallow: /index.php?action=search*
Disallow: /index.php?action=unread*
Disallow: /index.php?action=unreadreplies*
Disallow: /index.php?action=verificationcode*
Disallow: /index.php?action=who*
Disallow: /index.php?action=stats*
Disallow: /index.php?theme*
Disallow: /index.php?*wap*
Disallow: /index.php?*wap2*
Disallow: /index.php?*imode*
Disallow: /index.php?*rss*
Disallow: /Sources
sorry for my bad english

Re: Robots.txt needed for a site?

Reply #2

Great, can i use this in Elkarte now..any edits needed?

Re: Robots.txt needed for a site?

Reply #3

you can use it, you may also remove these lines:
Code: [Select]
Disallow: /index.php?*wap*
Disallow: /index.php?*wap2*
Disallow: /index.php?*imode*
Disallow: /index.php?*rss*
sorry for my bad english

Re: Robots.txt needed for a site?

Reply #4

Thank you so much @radu81