Crawling - Robots.txt file is included in most sites, which lists criteria about how deep crawling can occur. It also lists exclusions of particular parts of the sites which are not supposed to be crawled. Its not mandatory and crawlers can ignore robots.txt file.
--Shreejay
