phil ringnalda

Does anyone happen to know what Newzcrawler‘s up to? I was just watching tail -f access.log (like TV, but even more boring), when suddenly everything was one IP address requesting the RSS feed for entry plus comments for every post I’ve made since August, with a request for /favicon.ico in between every request. Luckily, that amounts to just a whole lot of 304 Not Modifieds, but a little grepping says that since it happens every 30 minutes, it also amounts to 6520 requests over the last 24 hours. From one IP address. With two entries and seven comments in that 24 hours, things were far more active than usual, but still, 6520 requests seems a little overboard. I’ve got gobs of bandwidth, and Apache shouldn’t have any problem with someone doing five or six requests a second for thirty seconds or so, but that’s one IP address. Switch a couple hundred people over to Newzcrawler, so it’s 1,304,000 requests a day, and I think it might get my host’s attention.

So, does anyone know what Newzcrawler’s doing, or what I’ve done wrong to confuse it into thinking it should request every single RSS feed I’ve done since last August, every thirty minutes? I’m willing to take the blame for that, and fix whatever I’ve done, but that bit about requesting /favicon.ico over and over, once for every request on the same host? That’s a bug.

Update : A bug, and one that’s already fixed in the current release. From the comments and Andrew’s answer in his forums, it’s just what I should have thought of, if it was before midnight or if I ever slept anymore: combine the good thing that’s storing old posts so a user can refer back to them with the good thing that’s fetching wfw:commentRss feeds for entries that you know about, so you can show the user the comments, and you have a whole lot of requests.

If you’ve got a better idea for how to keep those two good things than what’s in my comments or the Newzcrawler forum, I’d appreciate it if you would post it (over there, so Andrew doesn’t have to keep checking my comments. Well, unless he’s subscribed to the commentRss feed ;)).