Robots Files
Uploading a Robots.txt file to your website’s root directory is an important factor in your site’s SEO. You can write a robots.txt file yourself, if you like. It’s not really that difficult. Here is an example, which allows all bots to crawl and index all pages of your site:
User-agent: *
Disallow:
There are lots and lots of variables you can add to a robots.txt file. Some will prevent the bots from crawling certain folders; some will prevent only certain bots from crawling; some will prevent the crawling and indexing of any of your pages at all, and there are any number of variables in between.
If you’re not sure you want to create your own robots.txt file, then you can use free online generators instead.
If you have a Google Webmaster Tools account, you can also generate the file in there. And you can also add a meta tag just for the robots.txt file:
<meta name=”Robots” content=”index, follow”>
This is the default behavior of the spiders, however, so this tag is not really required.
