It will then access all of the links on the website. This is how search engines are able to find a user"s site regardless of whether or not they register their URL with the search engine. Submitting an URL however gets the user on the database a lot faster. It notifies a crawler to visit and index the user"s site instead of waiting for it to eventually find it through one of the external links. .

             Robots will revisit the site periodically to refresh the page"s information. This allows the search engine to automatically find "dead links", or pages that are no longer there. The robot will try unsuccessfully to update the information on a dead link and realize it isn"t there. .

             One of the things that separate all of the different search engines is how they are organized and what they include in their databases when "crawling". Some engines only record the contents of the main page of a site, where others record all pages in a site, effectively recording every page on the Internet. .

             Also unique are the criteria different search engines use to organize information for its users. Some use a system known as link popularity, listing the results of a user's search according to which sites have the most links from other sites. Other search engines sort results according to the summary information in a site"s Meta tags, and others look for common themes used throughout a site. There are many other ways to organise results, and the leading search engines use a combination of several of them.

             There is a different sort of content database on the Internet, very similar to a search engine. This is called a directory. The main difference between a search engine and a directory is that a directory will not list an URL if it has not been manually registered with them. They do not use indexing software and so have no way of knowing of content that is not introduced to them.