Go to: /articles/2010/07/06/ for other articles.

What are Web Spiders? How Do they Work?

Do you know how search engines work? Since the search engines finally bring your precious website to the notice of your potential customer it is best that you understand the basics of their operation. No I won't go into the full details because if I'm totally honest I don't have sufficient technical knowledge to understand it all (a bit like the "black arts" to me). However I can give you the basics which are all you need really.

"Spiders" are robots used by Search Engines to index websites. A "web spider" is an automated program which is run by the search engine system.

Once you have completed the required submission page following presenting your website to a search engine the spider will visit the site. This may not be immediate however. What the web spider does is that it reads the content and visits each link you have on your website. This information is then relayed to a central depository where the pages are then indexed. A spider can index up to a million pages a day and in fact be likened to a book containing all the information for all the websites that it finds during the search.

You should bear in mind that the web spider will periodically return to your site to see if any information has changed. The frequency of this return visit is decided by each individual search engine management.

Another point to consider is that some spiders will only index a certain number of pages so try to avoid creating a site containing hundreds!

So what happens to all this information which has been collated? When information is requested from a search engine it actually searches through the index and is not searching the web as many people think. However it should be noted that because of the different *algorithms each search engine uses rankings will differ.

An important point to remember is that search engine algorithms scan for the frequency and location of the keywords on a web page.. Since the* algorithms can analyze the way pages link to other pages on the website they can determine how pages link to each other and what the page is all about. However they can also detect unnecessary usage of artificial keywords. So be on your guard and don't try to be too clever.

I hope this short report has been of use to you but for the more technically minded amongst you I suggest you might like to visit some of the search engine websites to get more information. This should not be necessary though to make full use of search engines in my opinion.

John Beaumont Internet Marketer

*An algorithm is an effective method for solving a problem using a finite sequence of instructions.

About the Author:

Get your free report about the Truth of My Internet Business and Online Marketing here; tinyurl.com/3xfzld2