When search engine’s crawler visits your website, one special file it will look for on your website is the “robots.txt” file.

The robots.txt file is a simple text file that resides in the root directory of your domain. The content of a robots.txt file consists of information that inform the search engines which directories or pages of your website should be indexed and which should be ignored.

For blogs that use WordPress, a SEO friendly content management software, an example of the robots.txt is as follow:

User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /category
Disallow: /page
Disallow: /tag

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:
Allow: /*

Sitemap: http://www.yourdomain.com/sitemap.xml”