Robots.txt File

General General Questions
1

A question that is more than likley very basic but i am unsure as to what is correct.



What info should be in the public_html/robots.txt file, the reason i ask is i checked my url with Google’s robots.txt Tester and found that the home page shows an error red warning with Disallow: / highlighted in red.



I changed this to Allow: / and it now returns green but have i done something that may comprimise the site’s security?

2

Hello Swifty,



Did you find the answers to your question? I would love to know the same thing



Thanks in advance

Olof

3

robots.txt has nothing to do with security of your site.

It is only a small text file that Search Engine robots can read when they spider your site. It is only telling the robot where you allow it to read and index. It is not respected by ALL robots, but big SE like Google and Bing will not index what you have identified as “disallowed”.

http://www.robotstxt.org/