How a search engine works - SEO Tips
There are 5 main parts to a search engine. All of the large main players in the search industry will have a system that is based on the very basics of what is described below:
A spider is a robotic program that downloads web pages and looks at the code of the page. The spider doesn't have any visual components at all, if you right click on most web pages and go to view source that is what the spider can see.
While the spider is downloading pages, it is the crawler's job to strip apart the page and look for links. It is the crawler's job to look at the links and then decide where the spider should go next determined on the links or a pre-defined list of URL's.
It is the indexers job to analyse different aspects of a web page. It looks at Entities such as, titles, headings, links, text, constructs, bold, italic, and other style portions of a page that are ripped apart and analysed.
The database is where all the results and pages are stored, considering that there are an estimated 10 billion web pages on the Internet this can take a huge amount of storage space, which will be increasing indefinitely.
- Search Engine Results Page:
The results page is where the clever stuff takes place; this is where the engine decides which page matches the query that was entered in the search box of their home page.