Most of the people would definitely say Googlebot without hesitation, but my recent observations seem to suggest otherwise.

I had recently upgraded my website from the site I wrote to the more powerful dasBlog, and since the directory structures are drastically different between dasBlog and the website I wrote, I was expecting the crawlers to have some lag before catching up. I deliberately not to leave any old link references to point to the new pages (a practice that is used in professional websites), because I would like to see how fast would these crawlers catch up.

To my surprise, only after the second day, my new pages are already showing up in MSN search. But Googlebots seem to be well behind. Not only the search results are stale, their crawlers kept trying to go to the old links (this can be seen from IIS log). It will be interesting to see how long before my new pages appear in Google search.

Be Sociable, Share!