Hashemian Blog Web Tools, Financial Markets, Technology Saturday, July 19, 2008 Yahoo Slurp Versus Googlebot  As a part of my early morning routine at my job I scan a daily notification on our web site visitor patterns, specifically looking for abusive or abnormal page requests to identify rogue hosts. I wrote the program years ago, but it has become one of my most effective tools in identifying abuse. No surprise that search engine robots (spiders or crawlers) always top the list. Spiders are necessary evils. You want them crawling and indexing all your pages frequently, but that leads to spiders with voracious appetites that can rob the infrastructure of performance. Speaking of voracious appetites, for years Googlebot used to top our crawl volume list, far and away from the second position, usually (but not always) occupied by Yahoo Slurp (Yahoo's spider). As of a few weeks ago I am noticing that Yahoo Slurp crawl volume is surpassing that of Googlebot. The first few days I considered it an anomaly, but the levels have remained consistent, knocking Googlebot out of its first position. Googlebot still registers with more or less the same numbers, but Yahoo Slurp has been consistently beating those numbers, sometimes by twice as many. That's a considerable spike in Yahoo's crawl activity. I find it interesting that Yahoo's crawl volume started to rise in the midst of its take-over battles with Microsoft. Search engines have always competed on who has the most indexed pages. It's a bragging right, more than it is a practical or useful matter in search accuracy. But accuracy in search results is subjective, while page volumes are concrete numbers and that's why they are sometimes used as key differentiators between competing search engines. Still, the timing of the jump in Yahoo's crawling activity is intriguing. Could the motive be to enhance its index size (considered a key asset), thereby adding to the company's value? One wonders. Meanwhile, sites need to consider the battle repercussions between the search engine titans as their increased crawl rates could put additional pressure on their infrastructures. One way to mitigate the effects is to set up mirror servers and deflect the spider/robot traffic to them using application-aware appliances, but that there's some effort involved and there's room for error as the servers need to be synchronized in real-time, specially for those such as news-related sites that continuously publish new pages and fresh content. The other consideration is bandwidth management to accommodate increased traffic. At the least, faster servers and fatter pipes are in order. As a reference here are the current signatures left behind by Google and Yahoo crawlers: Googlebot's user-agent: Mozilla/5.0+(compatible;+Googlebot/2.1;++http://www.google.com/bot.html) Yahoo Slurp's user-agent: Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp) google,yahoo,search engines,http,seo,googlebot,web Labels: web <Yahoo Slurp Versus Googlebot> // posted by rh 0 comments 0 Comments: Post a Comment

Links Technorati Profile TMCnet.com ARCHIVES 09/01/2003 - 10/01/2003 03/01/2004 - 04/01/2004 04/01/2004 - 05/01/2004 05/01/2004 - 06/01/2004 06/01/2004 - 07/01/2004 07/01/2004 - 08/01/2004 08/01/2004 - 09/01/2004 09/01/2004 - 10/01/2004 10/01/2004 - 11/01/2004 11/01/2004 - 12/01/2004 12/01/2004 - 01/01/2005 01/01/2005 - 02/01/2005 02/01/2005 - 03/01/2005 03/01/2005 - 04/01/2005 04/01/2005 - 05/01/2005 05/01/2005 - 06/01/2005 06/01/2005 - 07/01/2005 07/01/2005 - 08/01/2005 08/01/2005 - 09/01/2005 09/01/2005 - 10/01/2005 10/01/2005 - 11/01/2005 11/01/2005 - 12/01/2005 12/01/2005 - 01/01/2006 01/01/2006 - 02/01/2006 02/01/2006 - 03/01/2006 03/01/2006 - 04/01/2006 04/01/2006 - 05/01/2006 05/01/2006 - 06/01/2006 06/01/2006 - 07/01/2006 07/01/2006 - 08/01/2006 08/01/2006 - 09/01/2006 09/01/2006 - 10/01/2006 10/01/2006 - 11/01/2006 11/01/2006 - 12/01/2006 12/01/2006 - 01/01/2007 01/01/2007 - 02/01/2007 02/01/2007 - 03/01/2007 03/01/2007 - 04/01/2007 04/01/2007 - 05/01/2007 05/01/2007 - 06/01/2007 06/01/2007 - 07/01/2007 07/01/2007 - 08/01/2007 08/01/2007 - 09/01/2007 09/01/2007 - 10/01/2007 10/01/2007 - 11/01/2007 11/01/2007 - 12/01/2007 12/01/2007 - 01/01/2008 01/01/2008 - 02/01/2008 02/01/2008 - 03/01/2008 03/01/2008 - 04/01/2008 04/01/2008 - 05/01/2008 05/01/2008 - 06/01/2008 06/01/2008 - 07/01/2008 07/01/2008 - 08/01/2008 08/01/2008 - 09/01/2008

هاشمیان.com

 Home

 Blog

 Web Tools
Usage

 News

 Articles

 FAQ

 About

 Contact

 Financial Markets Book
Read Complete Book

BOOK
Tug of War: Classical Versus "Modern" Dressage: Why Classical Training Works and How Incorrect Riding Negatively Affects Horses' Health
Gerd Heuschmann
$24.95

BOOK
Scott Pilgrim, Vol. 2: Scott Pilgrim Versus The World
Bryan Lee O'Malley
$11.95

BOOK
A Friendly Letter to Skeptics and Atheists: Musings on Why God Is Good and Faith Isn't Evil
David G. Myers
$16.95

BOOK
Richard Brautigan's Trout Fishing in America, The Pill versus The Springhill Mine Disaster, and In Watermelon Sugar
Richard Brautigan
$15.00

BOOK
U.S. Versus Them: How a Half-Century of Conservatism Has Undermined America's Security
J. Peter Scoblic
$25.95

|yahoo-slurp-versus-googlebot|

more…