FAQ: robots.txt

Why do I see hits on a file called "robots.txt" in my statistics?

Short Answer

robots.txt is a file that tells search engine spiders and robots what they can do, so a hit on robots.txt means that a spider visited your site.

Long Answer

robots.txt is a "standard" file that appears at the root level of many web sites to tell search engine robots what to do on the site. Robots, also known as spiders, are computer programs that attempt to systematically visit and catalog all the pages on the Web. robots.txt tells the robots what they can or can't do on the site (whether they can index the site, which pages they may not index, etc.). Any correctly written robot will hit that page first, and follow the instructions it finds there. So the hits you're seeing are from robots.

If you don't have a robots.txt file on your site, the robots don't actually get any information--they get a "404 File Not Found" error instead, which they generally interpret as "index whatever you want."