|
Crawler is another name for a search engine spider. A robotic program that search and roam web pages and stores them to be indexed by the search engines.
Crawling the web is a job of the search engine crawler. It will follow a set list of pre-defined links that are deemed worth to follow. |
