Aggregating RSS feeds with Delicious, Yahoo! Pipes, FeedBurner, and CFML

I saved major time creating a feed aggregator by using a mashup of Delicious, Yahoo! Pipes, FeedBurner, and the <cffeed> tag in CFML.

Over the weekend, I created a couple simple feed aggregators on the CFWheels Community page. I saved major time by using a mashup of Delicious, Yahoo! Pipes, FeedBurner, and the tag in CFML. This was far easier than creating my own feed aggregator by hand.

Basically, I wanted to show 2 streams:

Latest 5 posts from a handpicked selection of blogs
Latest 5 articles that I’ve found on the Web relating to Wheels

For #1, I was able to aggregate the feeds using Yahoo! Pipes. For #2, I created a Delicious account that I can post articles to. All very dynamic. I’ll explain later how I used FeedBurner and CFML to complete the solution.

The end goal

The finished product looks like this (click for larger version):

The easy one: Delicious, FeedBurner, and CFML

We’ll do the easy one first because I’m lazy. For the “Articles and Discussion” section on the right, I have it set up where I can just post links to the Delicious account. Once I got that set up, I burned the account’s RSS feed using FeedBurner.

Why FeedBurner?

Why would I use FeedBurner? FeedBurner acts as an intermediary between the user and your RSS feed. By exchanging a little bit of your info with Google (what you pay), you get some great benefits:

Normalizes your feed into a valid/compatible format
Track how your feed is being used and how many users are subscribed to it
The ability to substitute your own title and description (useful in cases where you’re using a third party feed like Delicious)
Google takes care of the feed hosting for you
Automatic blog service pinging

I also make the feeds available on the CFWheels site so that others can subscribe to them. It’s great to track how those feeds are being used and by how many people.

Using CFML to display the feed

After getting the RSS feed set up, I was able to use one line of CFML to fetch the feed:

view raw cffeed.cfm hosted with ❤ by GitHub

But I hate to say that it can’t be this simple. I had to implement a solution to cache the feed for at least 5 minutes. If we fetched the feed in real time every time the page was loaded, Google would get pissed for hitting their servers so often.

To do this, I created a partial called articles. I could then cache the partial so that at most, the feed would only be accessed by my web server every 5 minutes. Here’s how the call in views/community/index.cfm works:

	#includePartial(
	partial="articles",
	articlesRss=articlesRss,
	articlesUrl=articlesUrl,
	numToDisplay=5,
	cache=5
	)#

view raw index.cfm hosted with ❤ by GitHub

The partial at views/community/_articles.cfm calls a view helper to get the feed data and displays the data. I had to use a view helper to link the partial back to the data that we needed. This was done so the caching could be controlled at the partial level.

	<!--- URL to RSS feed --->
	<cfparam name="arguments.articlesRss" type="url">
	<!--- URL to Delicious page --->
	<cfparam name="arguments.articlesUrl" type="url">
	<!--- Number of items to display from feed --->
	<cfparam name="arguments.numToDisplay" type="numeric" default="10">

	<!--- Note: we are breaking the convention of data access in the view here so that we can cache this partial --->
	<cfset articles = articles(arguments.articlesRss)>

	<cfoutput>

	<div id="community-articles">
	<h3>
	#Replace(articles.title, "ColdFusion on Wheels ", "")#
	<a href="#arguments.articlesRss#" class="image">#imageTag(source="feed-icon-14x14.png", alt="Articles Feed")#</a>
	</h3>
	<p>#HtmlEditFormat(articles.description)#</p>
	<dl>
	<cfloop from="1" to="#arguments.numToDisplay#" index="i">
	<dt>
	<a href="#articles.item[i].link#">#articles.item[i].title#</a>
	<span class="date">(discovered #Replace(timeAgoInWords(articles.item[i].pubDate), " ", " ", "all")# ago)</span>
	</a>
	</dt>
	<dd>#articles.item[i].description.value#</dd>
	</cfloop>
	</dl>
	<h4>Read More Articles</h4>
	<ul class="calls-to-action">
	<li class="delicious"><a href="#arguments.articlesUrl#">All Articles on Delicious</a></li>
	<li class="feed"><a href="#arguments.articlesRss#" title="Articles Feed"><abbr title="Really Simple Syndication">RSS</abbr> Feed</a></li>
	</ul>
	</div>

	</cfoutput>

view raw _articles.cfm hosted with ❤ by GitHub

And lastly, here was the view helper that I stored in views/community/helpers.cfm. It basically fetches the feed appropriately depending on which environment I’m in.

	<cffunction name="articles" hint="Fetches latest article data from specified RSS feed.">
	<cfargument name="rssUrl" type="string" hint="URL of RSS feed to check.">

	<cfset var loc = {}>

	<!--- Design mode will not cache, so let's store data in session to keep FeedBurner happy --->
	<cfif get("environment") is "design">
	<cfif not StructKeyExists(session, "articles")>
	<cffeed name="session.articles" source="#arguments.rssUrl#">
	</cfif>
	<cfset loc.articles = session.articles>
	<!--- This is cached in production mode --->
	<cfelse>
	<cffeed name="loc.articles" source="#arguments.rssUrl#">
	</cfif>

	<cfreturn loc.articles>
	</cffunction>

view raw helpers.cfm hosted with ❤ by GitHub

Voila! A cached list of articles from the Delicious feed!

Feed aggregation using Yahoo! Pipes

Yahoo! Pipes is a visual tool for aggregating different types of data, sorting it, filtering it, and then exporting it to RSS or JSON. It’s best used for my use case of having a tool where I can go in and manually add data to the aggregation. There are a lot of different data sources that you can use, but I’ve primarily used the tool for creating uber RSS feeds.

This is the pipe layout that I created (click to view larger):

Each item that I mention is available in a menu to the left of the canvas. You drag the items onto the canvas and then click and drag connectors of each item to connect them.

Here are the basic steps that I followed to create the uber feed.

Drag a Fetch Feed item from the left onto the canvas. Enter the URL of the feed. (In this case, I tried to just grab each blog’s CFWheels category.)
Chain each feed object into a Union operator.
Chain the Union items together if you need more than one.
Tie the last Union operator to a Sort item and set that to sort by date descending.
Tie the Sort item to the Output item that comes with the canvas.
Save and name the feed.
Go back to the My Pipes screen and grab the RSS or JSON URL of the feed.

As you’re doing all of these steps, you can click the Output item at the bottom and see a preview of what your feed data will look like in the Debugger panel. If you want to debug any part of the layout, you just click that element to see the preview as well. How slick is that?!

Overall, I recommend playing with Pipes for an hour or 2 to see what kind of data mashups you can come up with. I’m sure there are other uses that I just haven’t thought of yet.

Tying it to FeedBurner and CFML

Because the output of this Pipe is RSS, I basically copied most of the steps that I described above for the Delicious feed. I burned it with FeedBurner and then created a cached partial similar to the one in the Articles example above.

There are some differences in how I displayed the Articles vs. Bloggers feeds, so I could probably spend some time refactoring the common stuff into a base partial. Perhaps that is a discussion for a future post.