August 21, 2002
Google increasing spidering frequency: Covering more news-sites & weblogs

SearchEngineWatch reports that Google has started adding more "fresh pages":

Last December, Google announced that it was spidering 3 million pages each day where freshness had been determined to be crucial. Now the search engine has gone beyond those core pages, though it won't say exactly how far.
"We are significantly increasing the size of the daily crawl and have been doing over the past month," said spokesperson Nate Tyler.
I've noticed that my blog entries have started receiving hits from Google only one to two days after publishing, so this seems to be a pretty widespread crawl they have started... It certainly makes it easier to track the "living web" (news sources, weblogs and more) - an area up to now covered almost exclusively by Daypop...

It seems the "Google Dance" is taking place more often than before as well - a new one started today, and it's only a week and a half since last time! Previously, average frequency has been approximately every 30 days...

Sponsored links
Related Entries

Google doesn't respect / crawl
non english code pages like iso-8859-2

Posted by: adbent on March 17, 2005 10:15 PM
Post a comment

(Will not be displayed if you enter a website below. Otherwise, it will be displayed "spam protected")

(if you have one)

What do you want to say?
(please don't bother posting "spam" (pornography, viagra-sales etc - I will delete such comments anyway))

Remember info?

Referrers to this page
TrackBack URL for this entry:

[an error occurred while processing this directive]

© Anders Jacobsen
[ photography]