What are Web Spiders? How Do they Work?
Do you know how search engines work? Since the search engines
finally bring your precious website to the notice of your
potential customer it is best that you understand the basics
of their operation. No I won't go into the full details
because if I'm totally honest I don't have sufficient
technical knowledge to understand it all (a bit like the
"black arts" to me). However I can give you the basics which
are all you need really.
"Spiders" are robots used by Search Engines to index
websites. A "web spider" is an automated program which is
run by the search engine system.
Once you have completed the required submission page
following presenting your website to a search engine the
spider will visit the site. This may not be immediate
however. What the web spider does is that it reads the
content and visits each link you have on your website. This
information is then relayed to a central depository where
the pages are then indexed. A spider can index up to a
million pages a day and in fact be likened to a book
containing all the information for all the websites that it
finds during the search.
You should bear in mind that the web spider will
periodically return to your site to see if any information
has changed. The frequency of this return visit is decided
by each individual search engine management.
Another point to consider is that some spiders will only
index a certain number of pages so try to avoid creating a
site containing hundreds!
So what happens to all this information which has been
collated? When information is requested from a search engine
it actually searches through the index and is not searching
the web as many people think. However it should be noted
that because of the different *algorithms each search engine
uses rankings will differ.
An important point to remember is that search engine
algorithms scan for the frequency and location of the
keywords on a web page.. Since the* algorithms can analyze
the way pages link to other pages on the website they can
determine how pages link to each other and what the page is
all about. However they can also detect unnecessary usage of
artificial keywords. So be on your guard and don't try to be
I hope this short report has been of use to you but for the
more technically minded amongst you I suggest you might like
to visit some of the search engine websites to get more
information. This should not be necessary though to make
full use of search engines in my opinion.
John Beaumont Internet Marketer
*An algorithm is an effective method for solving a problem
using a finite sequence of instructions.
About the Author:
Get your free report about the Truth of My Internet Business
and Online Marketing here;