Robots.txt needed for a site?

July 23, 2018, 02:22:12 pm

I see that ElkArte doesn't use any robots.txt and in google webmaster tool Google asks me to submit robots.txt file for better indexing

Do we really need it and if so what can we allow and disallow. Any sample robots.txt of elkarte available?

Re: Robots.txt needed for a site?

Reply #1 – July 23, 2018, 04:26:44 pm

I'm also interested in this argument. Here is what I have in my robots.txt (it's an old file I have created for SMF and never adapted for Elkarte)

Code: [Select]

  
User-agent: *
Disallow: /*sort=*
Disallow: /*msg*
Disallow: /index.php?action=activate*
Disallow: /index.php?action=admin*
Disallow: /index.php?action=calendar*
Disallow: /index.php?action=emailuser*
Disallow: /index.php?action=findmember*
Disallow: /index.php?action=help*
Disallow: /index.php?action=helpadmin*
Disallow: /index.php?action=login*
Disallow: /index.php?action=mlist*
Disallow: /index.php?action=modifykarma*
Disallow: /index.php?action=pm*
Disallow: /index.php?action=post*
Disallow: /index.php?action=printpage*
Disallow: /index.php?action=profile*
Disallow: /index.php?action=recent*
Disallow: /index.php?action=register*
Disallow: /index.php?action=reminder*
Disallow: /index.php?action=search*
Disallow: /index.php?action=unread*
Disallow: /index.php?action=unreadreplies*
Disallow: /index.php?action=verificationcode*
Disallow: /index.php?action=who*
Disallow: /index.php?action=stats*
Disallow: /index.php?theme*
Disallow: /index.php?*wap*
Disallow: /index.php?*wap2*
Disallow: /index.php?*imode*
Disallow: /index.php?*rss*
Disallow: /Sources

Re: Robots.txt needed for a site?

Reply #2 – July 24, 2018, 12:38:06 pm

Great, can i use this in Elkarte now..any edits needed?

Re: Robots.txt needed for a site?

Reply #3 – July 24, 2018, 07:03:54 pm

you can use it, you may also remove these lines:

Code: [Select]

Disallow: /index.php?*wap*
Disallow: /index.php?*wap2*
Disallow: /index.php?*imode*
Disallow: /index.php?*rss*

Re: Robots.txt needed for a site?

Reply #4 – July 25, 2018, 12:28:11 pm

Thank you so much @radu81