Another RSS feed-related element needed

Browsing Jon Udell’s feeds, I noticed that he’s subscribed to a Yahoo Finance beta feed, with the channel title “Yahoo Finance RSS Test is Over…” and one item reading “Our RSS test is over. We appreciate all the feedback.”

Thing is, the way most aggregators work these days, unless you unsubscribe when you first see that item, you’ll never see it again. You’ll just keep hammering away at their server, hour after hour getting the same file (or the same HEAD, if your aggregator is that kind), waiting and waiting for a new item. Just 404ing the feed would be fairly effective, but it’s not very nice, what with all the people who have linked to it. Far better if there was a <blogChannel:dead> or <admin:feedLink um, er something> element, to tell aware aggregators that it’s gone and won’t be back.

28 Comments

Comment by Dave Winer #
2002-10-18 03:46:57

I agree. Another example are feeds for discussion threads, like this one. Eventually the thread will die down, and people who are subscribed to it can unsubscribe. But will they remember to do it? Right now, some of our users subscribe to over 300 feeds. I bet that goes up over time (of course it will). Managing what today seem like large numbers of feeds will become important.

 
Comment by Morbus Iff #
2002-10-18 04:51:59

I prefer a ”automatically remove this feed if it hasn’t been updated for a month” setting. If a feed is dying/dead, there’s a good chance it’s for lack of interest, and there’s not going to be enough impetus for the producer to do the ”right thing” with an RSS module.

 
Comment by Dave Winer #
2002-10-18 05:09:16

Morbus, we do that in Radio. The problem is that it may not make sense to delete the feed on the server side. Why would Phil want to get rid of the RSS for this thread? But people can stop polling it a month from now. See the diff?

 
Comment by Morbus Iff #
2002-10-18 06:03:08

Dave – didn’t know that. Is that a user customizable setting (”a month”, no, no, ”two months!”)? Some feeds may indeed ONLY update once a month (a web-zine, perhaps). For a bit I was thinking of, instead of automatics, do a ”hey! this feed hasn’t been updated for xx days. would you like me to remove it?” software prompt.

As for ”the diff”… are we talking about a user subscription or a feed on the server? If this comment feed stopped being updated for a month, then Radio or AmphetaDesk could remove it from the user’s subscriptions – it wouldn’t affect Phil’s server whatsoever (where it would still exist). How did ”delete the feed on the server side” come up?

 
Comment by Will Cox #
2002-10-18 06:32:28

Proper appplication of 410 Gone on the server end, and paying atttention to ”clients … should delete references” on the client-side would help in an HTTP environment. As does backing off on fetching the feed if it isn’t updated, and/or only fetching the feed if you are notified that it has been updated.

w/r/t a 410, that has the same problems in implementation that a 301 does: end-users don’t usually have the abilty to issue that status code. So I can see the usefullness of an element to indicate the status of a feed. I seem to recall that a status element is already in one of the modules.

 
Comment by Mark #
2002-10-18 06:55:23

Will: do you mean mod_servicestatus?

http://web.resource.org/rss/1.0/modules/servicestatus/

That won’t work for this; it solves a different problem. In this case, the server is fine, it just doesn’t have anything more to say to you (at this URL). Don’t take it personally. :)

 
Comment by Phil Ringnalda #
2002-10-18 07:33:51

I’d say there are cases where it needs to be done on each side: I sometimes stay subscribed to a feed for a while, hoping that the person hasn’t really thrown in the towel: my aggregator should prompt me to accept the inevitable. Other times, they know that they are throwing in the towel for good (at least at that URI), but want to leave a file explaining why.

 
Comment by Will Cox #
2002-10-18 07:36:15

I saw that mod_servicestatus wasn’t exactly in tune with this. Too much information running through my brain: I can no longer discern between what I read and what I imagine I read. :-)

 
Comment by Phil Ringnalda #
2002-10-18 07:38:50

Also, wrt discussion threads: someone slap me for suggesting this, but: after you have subscribed to the feed for this thread alone for four or five days tops, if you are still sort of interested, you should unsubscribe to this thread, and subscribe to my ”recent comments on all posts” feed. How you would generalize that? onUnSubscribe?

 
Comment by Will Cox #
2002-10-18 07:50:37

dc:isReplacedBy ? ;-)

 
Comment by Will Cox #
2002-10-18 07:51:16

typo: dcterms:isReplacedBy

 
Comment by Phil Ringnalda #
2002-10-18 07:57:45

Nah, I’m not shutting off the feed, and if I did then recent comments on other threads wouldn’t be a replacement. Just a suggestion: ”you’re tired of this feed, try this one instead.” Could even be generalized to ”if you’re tired of reading my blog, this guy covers the same sort of things.” I doubt that anyone would want to support an element like that, though.

 
Comment by Dave Winer #
2002-10-18 08:38:41

I need a macro that groans for me when someone says D & C or R, D & F. You can quote me on that.

 
Comment by Will Cox #
2002-10-18 11:30:32

Turns out the element I was thinking of when I jokingly suggested dcterms:isReplacedBy was rdfs:seeAlso. Looking for that, I ran across this
mention by Bill Kearney of moreInfo element. But I don’t think that seeAlso will work unless the main comments feed contains information about the inactive thread.

You’re looking for a feed consumer to act on an element that points to moreLikeThis when the consumer unsubscribes from this. I think onUnsubscribe confuses the event with the data acted on during the event.

 
Comment by Michael Bernstein #
2002-10-18 11:36:31

A straightforward indication thta the feed will no longer be updated (for whatever reason), seems simplest and most utilitarian.

For that purpose, a single empty element (in whatever namespace) will suffice.

The use cases of pointing to a potential replacement feed seem to be too varied to be easily encapsulated. I can imagine, for example, situations where an aggregator would want to select a new feed to try as a replacement based on taxonomic information for example, or (assuming that a feed had multiple contributors) googling for another feed that most of the same folks were participating in.

Let’s keep it simple, and not add a URI or URL to the element (at least not yet). If it turns out that aggregators in general need a specific kind of locational hint later, then we can add a URI attribute (or element contents) with a well defined semantic meaning at that time.

This way, the element can be automatically added to a category feed if the category is removed, or to a comments feed when comments for a posting are turned off, or to a whole blog’s feed, if it is simply shut down and archived.

I can imagine a situation where a feed is shut down and moved to a permanent archival location, where the old location has both newFeedLocation and a deadFeed elements, and the new location just a deadFeed element (and possible ’mirror’ elements as well pointing to other archival locations).

It will also serve as a warning if someone follows a link and tries to subscribe to a dead feed. The client software can prompt the user ”this feed is marked as dead, are you sure you want to subscribe?”.

 
Comment by Will Cox #
2002-10-18 11:54:02

Getting back to the original question,

  1. pay attention to 410 Gone when fetching over HTTP
  2. Should there be an element that defines when something is gone, or
  3. should the aggregator just back off and ignore feeds that haven’t been updated, or
  4. both

There’s a need for something to help automate subscription maintainance, whether for temporary feeds disappearing, or for ad hoc feeds petering out, or for feeds moving.

 
Comment by Phil Wolff #
2002-10-18 11:58:18

I’m glad to see life-cycle thinking for rss feeds. Where else can we apply life-cycles to syndication? Do items have lives (expire after…)? Do feed suppliers?

 
Comment by Ed Nicholson #
2002-10-18 12:04:56

pardon the no-tech comment – but once a thread is done I would like the option of receiving a nicely packaged archive ready (maybe RDF ready) gift of the entire discourse.

each thread could become a supporting element in a knowledge base if post-processed with that in mind.

another option?

 
Comment by Phil Ringnalda #
2002-10-18 14:29:49

If someone is afraid to post something here saying ”I’m a user who wants to be able to do this”, then I’ve failed.

 
Comment by Dave Cantrell #
2002-10-18 19:15:35

Is there any reason this couldn’t be handled simply with a <status/> element?

e.g. <meta:status state=”dead” href=”http://www.mysite.com/reason.html”/>

State would provide the state of the feed using an enum, say something like ”dead|active|suspended” etc. Href would be an optional link to an HTML page with more information on the status. Default status could be ”active”, and for backwards-compatibility this default could be inferred by the lack of the element entirely.

Just an idea…

 
Comment by Lach #
2002-10-19 03:11:53

I need a macro that groans for me when someone says D & C or R, D & F. You can quote me on that.

I need a macro that groans whenever Dave Winer opens his mouth… Seriously Dave, where is your vision? Just because you don’t see a use for RDF or DC doesn’t mean there isn’t a valid one, and just because you don’t see a use now doesn’t mean you won’t in the future.

As for the actual issue at hand, what about <admin:feedLink somens:status="discontinued" rdfs:seeAlso="http://yoururl.com/moreinfo.html"/> where rdfs see also would be a url explaining why it had been discontinued and was optional, and somens was replaced by a real ns (preferably one which already exists and has a status attribute).

 
Comment by Dave Winer #
2002-10-19 15:41:54

Dave Cantrell, I like your idea.

To Lach, I can accept that you’d like a macro that groans whenever I open my mouth. I don’t agree that RDF is needed in this context, and while I would like to work with the people on the RSS-DEV list, productively and with mutual respect, I don’t want to open the door for them to once again control what we do. We’ve been through that for over two years, and we’ve gotten nowhere. Now, the market has grown and users have needs, and sorry, RDF simply doesn’t have the answers.

We’re going to deploy in the next few days. Radio will support an XML-level redirect as well as a way to tell Radio that a feed is finished and that aggregators should no longer scan it. We already support HTTP-based methods for both.

The reason we’re moving is that our users need it, and the network we’re building needs it. I remember very well how much congestion Pointcast created. We’re on the verge of getting into the same bind they were in. We must take care of it, and that’s going to take some working together among competitors. Luckily the market is still quite small. It doesn’t have anything to do with RDF, it has to do with the products, the installed base, tool vendors, aggregators and users. The conversation so far has been far too focused on RDF. That was a mistake.

 
Comment by Phil Ringnalda #
2002-10-19 18:52:20

I say, do it. I went into this wanting two things: solutions, and a way to bring 1.0 and 2.0 closer together. The RSS-DEV folks, or at least the #rss-dev folks, have said that the only compromise they’ll accept is 100% their way, so lets at least have solutions. If they can produce their own solutions, and get someone to implement them, so much the better.

 
Comment by Dave Winer #
2002-10-19 21:41:15

Hi Phil, thanks for the green light. I will go ahead and do it. I started from scratch and did only what was needed to specify a redirect. I managed to overcome Mark Nottingham’s objection that it would only work for RSS. It also covers the situation where the feed is being redirected nowhere, that is, isn’t updating any longer. I’ll make the necessary changes to the Radio code when I’m fresh tomorrow morning, and release the parts, and move on to evangelizing and working on other projects.

 
Comment by Michael Bernstein #
2002-10-20 14:48:58

Phil,

I’m a user who wants to do this:

  1. Have an aggregator that warns me I am trying to subscribe to a feed marked ’dead’.
  2. Have an aggregator that notifies me that a feed I am currently subscribed to has been marked ’dead’.

As a producer (and tool developer) I’d like to be able to do this:

  1. Mark a comment feed as ’dead’ automatically, when I disable commenting for an item.
  2. Mark a category feed as dead, if I remove the category (which is not the same as removing the category archive).
  3. mark my main blog feed as ’dead’, if I shut down the weblog, and/or decide to move it to some archival location (pointing to the archive, which would also be marked as ’dead’, would be the responsibility of a different element).

As I’ve mentioned before, it would be very nice if the mechanism for this worked equally well in RSS 1.0 and 2.0. To that end, I would prefer a simple element whose presence indicated ’deadness’. No URI or URL information is necessary for this purpose, so the element-contents-vs-rdf-resource issue doesn’t arise.

I realize that this all may be now moot, but I figured I’d post it anyway.

If I understand Dave’s post correctly, ’deadness’ is semantically indicated by using a redirection element with no URI. While I’m sure this will work just fine in practice, overloading the redirection element in this way feels, well, crufty.

As I mentioned up-thread, deadness and redirection can both be used together, when you are indicating that a certain resource will no longer be updated, and also indicating a permanent archival location for it (also marked as ’dead’).

I realize that this scenario is accomplishable with the old feed redirecting to the archival location, and the archival location redirecting nowhere, but it still doesn’t feel right, and it requires another HTTP transaction (if the aggregator just wanted to unsubscribe).

Oh well. Maybe it’s just me.

 
Comment by Phil Ringnalda #
2002-10-20 15:14:21

No, it’s not just you. Reading through, I’m nodding the whole way. If we lived in a world where everyone interested in syndication could act like adults most of the time, we could have a bulletin board/wiki that gateways well to email and RSS, with moderators who could split off issues and help identify when we were as close to consensus as we could get, and we could design a format that would make people say ”RSS? Isn’t that a grade of rubber?” In the ugly, bitter world we live in, I’m happy enough to see something, anything happen that even looks like progress.

 
Comment by Dave Winer #
2002-10-21 07:54:24

How to Redirect an RSS Feed. ”You’ve just moved your weblog or news site, and the RSS feed has moved too. You want people who are subscribed to your RSS feed to automatically start reading the feed at its new location. This document explains how to do that.”

 
Comment by Michael Bernstein #
2002-10-21 11:35:05

Hmm. Dave, I had no idea that you were contemplating providing a new document type as a substitute for HTTP response codes rather than trying to shoehorn this information in-band in the RSS feed itself.

This does look much more reusable, but if you’re going to do this, why go to the trouble of inventing a new syntax? just reuse the existing http response codes (this also points the way forward for more codes in the future)

Rather than:

<?xml version=”1.0”?>
<redirect>
<newLocation/>
</redirect>

How about something like:

<?xml version=”1.0”?>
<301>
<location>http://www.newlocation.com/</location&gt;
</301>

And:

<?xml version=”1.0”?>
<404/>

What do you think?

 
Name (required)
E-mail (required - never shown publicly)
URI
Your Comment (smaller size | larger size)
You may use <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <del datetime="" cite=""> <dd> <dl> <dt> <em> <i> <ins datetime="" cite=""> <kbd> <li> <ol> <p> <pre> <q cite=""> <samp> <strong> <sub> <sup> <ul> in your comment.