Resources

Crawl the web - SEO Glossary

Crawl the web - SEO Glossary

Crawling the web is job of the search engine crawler. It will follow a set list of pre-defined links that are deemed worth while to follow.