When robots (like the Googlebot) crawl your site, they begin by requesting http://example.com/robots.txt and checking it for special instructions. Use this plugin to create and ...

# robots.txt file for YouTube . User-agent: Mediapartners-Google* Disallow: User-agent: * Disallow: /videos. Disallow: /bulletin. Disallow: /comment. Disallow: /forgot

Web Spiders, often called Robots, are WWW search engines that "crawl" across the Internet and index pages on Web servers. A Web Spider will then catalog that information and ...

PC Robots.txt. PC Robots.txt creates a virtual robots.txt file for your blog. Your robots.txt file can be easily edited from the plugin options page.

A Web site administrator can indicate which parts of the site should not be visited by a robot by providing a specially formatted file on their site in http://.../robots.txt.

The default robots.txt file in Drupal 5.* has some problems. Also, the more modules one adds, the more duplicate content and low-quality URLs are created.

Aim-pro Tutorial: Using the Robots.txt File

The robot.txt (http://www.robotstxt.org/) is a publicly available file and when used properly is a very good way to control what search engines crawl and what

Tutorial on setting up a robots.txt to exclude search engine robots/spiders as part of the Robots Exclusion Standard.

Opinions expressed here are my own and do not necessarily reflect those of TMCnet. Some entries are excerpted from my personal site: www.hashemian.com.