phil ringnalda

Back when Google purchased Blogger, there was a lot of talk about what Blogger could do for Google, and what Google could do better with and for Blogger. One of the most frequently mentioned things was indexing: either Google would selectively index Blogger-powered blogs faster and better, or they would not be evil, and would index all blogs faster and better.

Well, doesn’t much look like it, at least for Blogger-powered blogs on Blog*Spot. Go to any random Blog*Spot address and look for the robots.txt file, that tells search engine crawlers what they can and can’t access. My FAQ blog will do. Hit refresh a few times. As of this moment, what you should see is one of two things, for any given refresh, either:

User-agent: msnbot
Disallow: /

or:

User-agent: msnbot
Disallow: /
User-agent: googlebot
Disallow: /

Completely banning Google’s crawler from any page on a given Blog*Spot blog, sometimes, randomly, maybe a bit less that half the time, doesn’t exactly strike me as good synergy. Does seem like it would explain why I’ve heard people complaining that their Google-powered site search for their Blog*Spot blog isn’t working right anymore, though. If half the time Googlebot gets told to stay away, you’re not going to be very well indexed. Maybe Yahoo! has a similar site-search form?

Oh, and the banning of msnbot? That’s just rude. Funny, but rude.