10/6/2011

Web crawlers can be be instructed not to index certain web pages. This is called robot exclusion protocol. Crawlers look for robots.txt at highest directory in domain level or even a specific page only can be shielded. But these shielding will never lock the page from crawling but it is only a compliance that most of the search engines follow.

-Bharath