blog

So have you been hearing about mashups recently? A data mashup is basically a web app that combines data from several sources into a single source. An example would be to combine a photo feed and google maps. The result is a map with points showing exactly where your photos where taken. Sounds cool hu? Well yahoo has created a visual tool that makes creating mashups relatively easy and fun called Yahoo Pipes. And of course Yahoo Pipes uses the YUI AJAX library so it's full of drag-n-drop goodness. Below is screenshot of the interface

So you maybe thinking, OK this looks pretty neat, but I still don't really get it. You can think of Yahoo Pipes as it's own functional programming language. It can take input through various means such as RSS feeds, CSV files, Web Pages, or Web Services. It also contains operators and helper functions to manipulate your data. Finally you can output your data to something a web browser can read (like an RSS feed). It also has a built in debugger so you can debug your "application" each step of the way.

Still fuzzy? Let me give you an example of a Pipe I just created to choose a winner on my Flickr Challenge Group. The winner must meet the following criteria:

1. The photo must be taken with a Nikon D300 Camera
   
2. The user must submit the photo to the weekly thread
   
3. The photo must have be taken that week
   
4. The user must have the most amount of votes to win
   

Sounds simple enough, but it is an extremely tedious process that just gets worse as more people join and there is no automated way on flickr to do this. Here is how I automated the process:

1. First the Pipe takes a few parameters: Week Number (i.e. week 18) and valid date ranges
   
2. Next we find the thread for that week and get all the photos submited
   
3. Next we count all the votes that have been accumulated for each photo in the thread
   
4. Next we make sure the photo was taken with a Nikon D300 and is within the valid date ranges
   
5. Finally, we output the data as an RSS feed in order by the number of votes descending

Below are the results in an RSS feed format. By clicking the link, you get taken to the photo.

You can checkout and view the source from this page:

http://pipes.yahoo.com/pipes/pipe.info?_id=e9e03c4f1214d97f5453aa825b9aab3e

So this is just one example of what can be done, but think of all the possibilities. And because it supports web services, you can even deal we real data objects! So next time your thinking of regexping a web page to mine some data, try automating it with a Pipe first.