robots.txt - why they matter
The robots.txt file tells spiders what they can and cannot read.
Having one is not absolutely necessary. However, if you do have
one, you can keep certain files from being searched via Google,
Yahoo, etc. You might want to keep pictures or other information
private.
If you do have a robots.txt, we will flag it if it looks like this:
User-agent: *
Disallow: *
That's a robots.txt that blocks everybody. If that's what you want,
great. But usually it's not. Which is why we flag it.
close window