10/06/11

The main issues with design of web crawlers are:

--> It Shouldn't overload web servers.

--> Avoid downloading same page multiple times.

-->Should be Robust when failures happen

-Aamir