How to Use Robots.txt?

[King]

Robots.txt is pretty easy to use, to create you just need to use notepad and follow the right syntax. Luckily this is also very similar and is comprised of only a few elements.

The file is made up of several records and each record features a user-agent section and a disallow section. You may choose to include comments with the use of a "#" at the start of a line.

The "user-agent" section defines which robots should follow the command. To list multiple robots simply use multiple user-agent lines. You can also use the wildcard character "*" to force ALL robots to obey the following command. An example of this line is below.
User-agent: *

The "Disallow" section is used for specifying the directory or file that should not be accessed. It is fairly simply sytax and you just need to include the directory (excluding you base URL) or file similar to the following example.

Again you can have multiple lines to disallow a selection of files or directories.
Disallow: /folder1/

The "allow" command can also come in useful, it lets you specify a specific file to allow in a directory you may have disallowed. It still needs to be paired with a User-Agent command but can be added in with disallow commands. Here is an example of a full record using all of the above.
User-Agent: *
Disallow: /folder1/
Disallow: /folder2/
Disallow: /folder3/
Allow: /folder1/important.html
Allow All

Happy with all the robots getting to your site? Just use this simple code to open the gates and ensure you’re not blocking anything
User-Agent: *
Disallow:
Block All

Site down for a while for construction? This will block everything form spidering. Don’t forget to remove it when you’re done!
User-Agent: *
Disallow: /

Linking to your Sitemap

A handy little trick that some robots allow is to link to your sitemap inside your robots file. Simply use the following example and modify the URL to suite your own domain.
Sitemap:http://www.yourdomain.com/sitemap.xml

[Imran]

wow
great explanation
thanks for it

[urvi88]

Thanks for such good information.

[django]

Thanks just added my sitemap and blog sitemap to my robots.txt file

[jingwen]

Yes,robots.txt could control the spider crawled your site,you can choose which you want it crawl or not.Very usefull.

[Sam]

Ok thanks for such good information.


________________________________
Software Development Company | SEO Forums

[imbitch]

"Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.

[liquidgraph]

That is amazing information, I am not familiar at al with using the robots.txt, and I believe that is a great start
Thank you!

[chatterbox]

Could someone give me a basic idea as to what is Robots.txt? Have heard about it numerous times but never bothered to find out. Any advice would be appreciated.