Breaking the world of syndication

It’s time for the users of syndicated XML feeds to stand up and be heard. We’ve been the victims of far too many politicized developer fights, as they push whatever their new idea is, at the expense of whatever we’re already happily using. One thing after another, they decide that just because they’ve had a brain fart and like the smell, that every single one of us should start producing something new, and should turn in whatever programs we are currently using to read feeds for a new version, or if we can’t get a new version, switch to some other program. I say, enough! The things that worked before would still work just fine, if only they would get out of their ivory towers, get off their high horses, and quit breaking the stuff we actually use for their strange ideas of format elegance.

Movable Type: don’t replace the RSS link element with the RSS guid element!

The spec makes it absolutely, unambiguously clear that when you have an excerpt in the description element, the link element is the URL for the full item:

An item may represent a “story” — much like a story in a newspaper or magazine; if so its description is a synopsis of the story, and the link points to the full story. An item may also be complete in itself, if so, the description contains the text (entity-encoded HTML is allowed), and the link and title may be omitted.

If you want to include the guid element, fine. If you want to make the guid a permalink, fine. But don’t break the hundreds of scripts and programs out there that only know title/link/description just because it doesn’t look pretty to have the same data in two elements.

(What, you thought I was talking about something else?)


Comment by GiacomoL #
2004-05-11 01:36:10

Absolutely with you.

Comment by Neil T. #
2004-05-11 01:50:42

I make sure that the guid isn’t a permalink in my feeds. The filename that link points to in my case is generated from the title, and so if I change the title for whatever reason, the link changes too. Therefore I use guid with permalink=”false” and then use MTEntryID@MTBlogURL as the ID in the tags, since that will be unique for an entry and won’t change if I edit the title of the entry.

You comment about retaining link for backwards compatibility is a good one though.

Comment by Geodog #
2004-05-11 03:23:01

Thanks for speaking up — I hope they listen.

Magpie won’t parse Salon Radio blogs RSS 2.0 because the newest update did away with link and title. Now I don’t read some of my favorite blogs.

Comment by Dave Winer #
2004-05-11 03:44:18

Geodog, I would be very surprised if ”the newest update did away with link and title.”

In Radio, if the user has titles and links turned on, then the feed will have titles and links. If the user doesn’t have them turned on, the feed will not have titles and links. That’s the way it’s been since early 2002, well over two years.

Aggregators have to be prepared to handle feeds that don’t have titles and links. But there may be apps that were built to work with a small number of MT sites, a private app, that depends on the link element being there. Why break them? Better yet, let’s not break them.

I think Phil’s advice is good (although a little colorful for my taste) and I sent an email to Anil and Mena saying that and pointing to this post.

If you really think that the Salon blogs all changed, then that’s a serious issue and we’d better look into it, asap.

Comment by Geodog #
2004-05-11 14:29:29

I’m not an RSS geek, just a user. Maybe you can debug this for us. Compare:

Real Live Preacher: (titles & links) vs. (neither)

Scott Rosenberg of Salon (titles & links) vs. (neither)


Comment by Dave Winer #
2004-05-11 18:54:39

Geodog, we don’t know how they have things set in their content system. It could be that they are formating the text that way, rather than entering a title into the Title field of the form. It’s impossible to tell from here, you’d have to ask Scott and the Preacher how they have set things up.

Comment by Geodog #
2004-05-11 21:45:10

Thanks for checking. I emailed both of them a while ago about it, and
they didn’t know anything about it — they were just using the
default setup that Salon/Radio gave them. I had to explain to one of
them what RSS was.

I guess maybe this is a matter of changing the default Salon setup,
and I don’t have clue who to go to about that.

P.S. Phil, I like the pink, but where is Barbie?

Comment by Dave Winer #
2004-05-12 02:44:41

Geodog, that explains it. If they have the default install, then they need to turn it on. Both probably should since they think in terms of titles for blog posts.

If anyone asks, this is where you turn on item-level Title and Link.

That link works only if you have Radio running.

Comment by Phil Ringnalda #
2004-05-12 10:00:44

Heh. I’d forgotten about that very slick aspect of running your app’s UI through a local webserver. Beats descriptions like ”On the Tools menu choose options, then Fields is the third tab on the second row…”

Comment by Matthew Ernest #
2004-05-11 20:23:30

On the other hand has titles and links. It’s not some effect of Salon’s community server, so it must something done on the client end. And that could be anything.

Comment by Phil Ringnalda #
2004-05-11 08:48:15

Oops, Ben says my leftover crapflood throttle’s kicking in (apparently I never expected to get more than 100 comments in 24 hours).

Anyway, his comment would have been something along the lines of:

Hi Phil, Thanks for the (subtle :) ) note. The removal of the item-level link element in the latest beta was an accident that has now been fixed.

Cool, thanks!

Now, about Sam’s both link and guid, but different URLs… :)

Comment by Randy Charles Morin #
2004-05-11 18:35:22

Using guid instead of link is valid. If your aggregator doesn’t like it, then your aggregator needs an upgrade. Nevertheless, change for change-sake is never good. MTs guid has always been less than useful, as it’s not a permalink. I like the dual link/guid concept. Last, since MT is highly configurable, then it doesn’t really matter what they do, does it? Just reconfigure it.

Comment by Mark #
2004-05-12 04:59:19

Further reading:

  1. The Lizard Brain of RSS, in which we learn that we’ve all been using <link> incorrectly for 4 years.
  2. There is no FAQ, in which we argue about #1.
  3. Leave RSS Alone, in which we learn that most aggregators don’t support guid as a permalink, and that my changing my RSS feed to look exactly like Dave Winer’s RSS feed is a nefarious act of malice designed to break the world of syndication and push everyone to alternative formats.
Comment by James Robertson #
2004-05-12 05:38:30

I don’t know what other aggregators do, but BottomFeeder uses the GUID as a permalink if

— the link isn’t there
— the GUID looks like an URL

Comment by Mark #
2004-05-12 06:36:30

Upon encountering a guid, my Universal Feed Parser will

  1. Perform an HTTP HEAD request on the RSS 2.0 specification to see if it has been updated since the parser was last run, since it seems to change subtlely on a monthly basis without any warning, notification, or documentation.
  2. If so, download the specification and search for the phrase ”<guid> sub-element of <item>” and attempt to determine what we should be treating as permalinks today.
  3. If not, download the RSS feed and, using a complex natural language parser of my own devising, attempt to determine if Dave has capriciously decided to change the semantics of any RSS elements lately. Although technically not spec text, is the only true source of RSS semantics, since so many rules never seem to make it into specs and yet competitors are supposed to follow them religiously.
  4. If Dave has not changed the rules recently, or if natural language parsing fails (a common occurrence), then the parser will treat guid as a permalink, if and only if the optional <link> element is missing and any of the attributes isPermaLink, isPermalink, ispermalink, permalink, or thisIsTheLinkIReallyMeanItThisTimeDamnIt is true, True, TRUE, TrUe, 1, MoreLikelyThanNot, or if all of these attributes is missing, since <guid> defaults to acting like a permalink (at least today) if no attributes are present.

I hope this clarifies the situation for other aggregator developers. Remember: RSS is simple! Rejoice, relax, and bask in the radioactive glow of simplicity!

Comment by Andrew Grumet #
2004-05-12 19:19:39

Mark, that’s a pretty serious allegation you made about the RSS spec document. I asked Dave Winer on the phone today, point blank, have you edited this document since posting. He said yes, there were a few minor navigation changes, but gave his word that the content of the specification document hasn’t been changed.

Dave’s word was enough to satisfy me, but I went ahead and checked the Internet archive, which made a copy of the spec on July 22, 2003. Here is a color-highlighted diff.

Most of the changes inconsequential, style changes and whitespace. Also, note that the spec page now has a news pane, which will tend to change the size and content of the document but not in a substantial way. The only significant changes I found were in the grey-backgrounded ”Misc changes” section. Just as Dave said, there were only a few minor tweaks to wording and navigation.

Please correct me if I’ve erred somewhere, but I don’t see any substance in your post.

Comment by Michael Pate #
2004-05-12 20:52:34


Not being Mark, I can’t be 100% sure as to what he is referring to but I am willing to hazard a guess. In The myth of RSS compatibility, Mark refers to two instances of changes.

The possible values in skipHours were changed . Apparently, the entire process took less than 16 hours from beginning to end.

Later, the rating element returned. I didn’t find any evidence of the process used in that case.

Hopefully, the RSS Advisory Board will develop a method for making formal announcements of changes and their timing so we don’t have to rely on discovering them by reading random weblogs entries.

Comment by Dave Winer #
2004-05-13 01:15:34

>>changes and their timing so we don’t have to rely on discovering them by reading random weblogs entries.

Comment by Michael Pate #
2004-05-13 05:57:30


I assumed that was the document that Mark would be performing the HTTP HEAD on. It documents the changes extremely well.

The issue is, short of visiting that page on an extremely regular basis, how do you know when it changes? And when you are aware of it, how much time will it take to implement the change specified? And how much time will it take other people to discover and implement the change specified?

Having a mechanism in place is worth considering even if the spec never advances past 2.0.1 rev 2.

Comment by Dave Winer #
2004-05-13 06:54:33

As you can see the changes are few and far between. But if you want to be in the loop, which seems reasonable, we can post notices on the home page of the Tech site, which has an RSS feed that you can subscribe to.

Make sense?

Comment by Shelley #
2004-05-12 21:45:47

You’re right Andrew. Let’s take Mark out behind the bowling alley and beat the crap out of him. He still hasn’t gone pink yet. The cad.

Who cares about syndication feeds. If I depended on my syndication feed, I wouldn’t have seen Phil’s lovely move to pink.

But Phil, where’s the big, hard, nippleless breasted doll?

(PS, Michael Hanscome has _really_ gone pink. See post)

Comment by Mark #
2004-05-13 06:05:33

Well, among other undocumented changes, the spec used to include a link to with the text ”Here’s an example of a file that makes use of elements in namespaces, authored by Mark Pilgrim.” That link and text were quietly removed after Dave arbitrarily changed the rules about what namespace elements were acceptable. (See his laughable ”political FAQ” for details on the current rules. Quick, before they change again!)

Andrew, the sooner you get that I know what I’m talking about, the sooner your world gets better.

Comment by Andrew Grumet #
2004-05-13 07:29:53

Specifics, good.

But first let’s go back to your original statement that the spec document ”seems to change subtlely on a monthly basis without any warning, notification, or documentation.” Looking at the past 9 months, and taking into consideration only changes in the content of the spec, this would appear to be false. Can we agree on that?

Now, as best I can tell, you’re right about that hyperlink getting removed. In fact, the change history document indicates that it was added on 10/1/02. However, for all the handwringing, the content of the spec remains unchanged. Feeds that were valid before July 2003 were also valid after. Can we agree on that?

Comment by Mark #
2004-05-13 07:52:30

I have not seen any changes in the past 9 months. There were several changes in the first 9 months. Some were documented, some were not. Some were discussed openly, some were not.

I think the initial inclusion of a link to a ”funky RSS” feed (and the later undocumented removal) is pretty damning all by itself, despite your technically correct assertion that it didn’t change the ”content” of the spec. The content of the spec states that any element in any namespace may be used anywhere in any RSS feed for any reason, but we all know that’s not the reality for publishers.

Other changes are meticulously documented here: The Myth of RSS Compatibility. Despite some flamethrowing from the RSS camp, I have never seen anyone disprove any of the factual evidence presented in that piece.

RSS 2.0 is not ”frozen”. It has never been ”frozen”. It has been capriciously and backwardly incompatibly changed multiple times without community involvement or discussion. Examples that were in the original version have been silently removed when their existence was deemed politically inconvenient. Exactly one man (who, ironically enough, claims not to ”control” RSS) has been responsible for all of these changes.

Agree on *that* and get back to me.

Comment by Pete Goldstein #
2004-05-13 08:35:00

Mark- these are nits. work to make Atom better, stop prosecuting this ’case’… its such a sad thing. ok, you hate winer. I dont care. Don’t expect anyone to go with you on your little trip.

Comment by Geodog #
2004-05-14 03:05:40

Off topic, which, might a be a good idea here:

Phil, do I have to search through cyberspace to find your comments on the MT 3.0/release/license/pricing issue, or do you plan on making some on your own blog? I’m interested in what a usually calm person who is the author of ”there is no they” is going to say. I’ll wager you are a lot less hotheaded than some.

Comment by Phil Ringnalda #
2004-05-16 22:02:02

I’ll wager you are a lot less hotheaded than some.

Nope, just know more languages in which I can count backward ;)

Sorry for the delayed answer, but between background distractions and the ”pure evil betrayal of teh beta testers!!1!” part, which did have me counting backward, it took a while before I knew what I thought for sure.

Name (required)
E-mail (required - never shown publicly)
Your Comment (smaller size | larger size)
You may use <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <del datetime="" cite=""> <dd> <dl> <dt> <em> <i> <ins datetime="" cite=""> <kbd> <li> <ol> <p> <pre> <q cite=""> <samp> <strong> <sub> <sup> <ul> in your comment.