My Data Stream

Posted on Feb 19, 2007

After a year and a half of using social applications heavily, I recently had to revisit the plan to aggregate all my activity into one data stream. As the calendar rolled to 2007, I kept wishing I could look at all my social activity from 2006 in context: time, date, type of activity, location, memory, information interest, and so on. What was I bookmarking, blogging about, listening to, going to, and thinking about? I still had the urge to have an information and online activity mash-up that would allow me to discover my own patterns and to share my activity across the web in one chronological stream of data (to start with anyway).

While there are some great services like SuprGlu for aggregating feeds into a browseable blog format (see my SuprGlu here), it’s still a hosted service and I need something that runs on my own server so I can have greater flexibility.

Others have been toying with this idea as well. A few months ago, Jeremy Keith wrote about his experience with this very issue in “Streaming my life away.” He decided to write a PHP script that would track several RSS feeds (Twitter, Flickr, Del.icio.us, Last.fm, and blog posts) by time-stamp and then display them in chronological order. It doesn’t do any caching but “the important thing is that it’s keeping the context of the permalinks (song, link, photo, or blog post) and displaying them ordered by date and time.” His experience resonated with me. I tried his PHP script but it didn’t quite work on my blog. Jeff Croft also has a similar implementation at his tumblelog, which appears to be down now.

The night I was playing with this idea, I twittered about it and heard from Boris Anthony who showed me his ggalaxy multi-contextual aggregator, a great system for seeing all of your friends’ (or favorite feeds) output in one place. In this case, the river of news is limited to 7 days and not archived. ggalaxy will be a WordPress plugin.

For my immediate purposes, I wanted to have the data from my selection of RSS feeds parsed and actually imported into my own MySQL database so it was archived and referenced in my own system. After doing some more research to see if anything was out there specifically for my blog software, I found feedgrab, an amazing plugin by Andrew Weaver in the ExpressionEngine forums. It worked well but needed the ability to import data from RSS feeds with unique criteria. After an email exchange with Andrew, he informed me this was already in the works. He updated the plugin and added new features within a couple of days. Big thanks to Andrew for writing this (and all the great programmers that support the open source community with their expertise).

My data stream:
http://www.emilychang.com/stream

I set up a separate blog for each of my services with custom fields for the data and I’m tracking my RSS feeds from:

The plugin script fetches the RSS feed from each service and then parses the XML and imports the data from pre-determined fields into an ExpressionEngine blog database. This is then displayed on my stream page by time-stamp with an icon to indicate the service or my blog source. You can click on the mini-calendar to view activity by day.

The script runs automatically every day and imports data from the various services. I can now search that database with the ExpressionEngine search engine by keyword(s). While this is internal to me at the moment, I’ll make the search public after I’ve customized it a bit. If I search for “urban” I would see results for all my bookmarks tagged red in stylehive, del.icio.us, music I had listened to through last.fm with the title or band “red,” photos I posted to flickr with that tag, status messages at twitter, events I was watching or attending at upcoming, Ideacodes news, or eHub apps with a match.

When I wrote about this back in September 2005, I was thinking that this type of aggregating by tag would also be helpful.

But, I’m also finding that the more tagging I do in various places and for different types of content, the more I feel my metadata becoming siloed into these other databases. Even though today’s new applications have moved beyond rigid categories and allow me to tag my own data, the current state of tagging doesn’t allow for us to map these information relationships from one site to another – or perhaps more aptly, to see a snapshot of our metadata universe. I’m still looking for a way to see how my current tags crossover with one another, and with tags made by others – the aerial view.

This type of attention stream seems harder to do right now since each web service does tags quite differently within RSS enclosures.

For now, this activity stream idea is providing the start to a holistic view of my activity across online networks: both my own and the ones I use. In turn, this acts as a conduit for you, the reader. Rather than just a static “recommended links” page or a blogroll, the data stream opens up my activity to you in semi-realtime and at one website. I’ll continue tweaking the content, navigation, and layout so watch for changes. Let me know what you think!

My data stream:
http://emilychang.com/stream

UPDATES:

See discussion about my post on Techmeme

March 4: Watching the stream by Jeremy Keith

Life Data Stream by Thomas Vander Wal

Traffic And Flow by Stowe Boyd

Attention Streams – Your life in feed by Chris Saad

Taming your own river of news by Grant Robertson on DownloadSquad

Everyone’s data streams for everything visible everywhere by Ross Dawson

When What i Say and do Probably Matters the Most by Daniela Barbosa

Marketing 2.0 by Robyn Tippins

My Life Stream by Jack Vinson

Lifestreams could help create new personalised discovery engines by Sam Sethi

IDEA #23 – Your Syndicated Chronological Life by Steve Poland

Lifestream – Could it be the next big thing? by Mark Krynsky

39 Comments

  1. Sam
    February 19, 2007

    Check out Yahoo! Pipes for doing things like this easily without any real coding:

    http://pipes.yahoo.com

  2. Edison Thomaz
    February 19, 2007

    Hey Emily,

    We have just released a new web service last week that you might want to check. It’s called Slifeshare (http://www.slifeshare.com). It’s a space where you can share with people (or just check for youself) what you’ve been doing, pages you’ve been visiting, music you’ve been listening to, etc. It’s all happens with the Slife client (http://www.slifelabs.com).

    It’s a bit wacky in that you can expose a lot of stuff, but at the same time we have lots of privacy controls. You only publish what you, live stream of activities, just stats, etc.

    Take Care,

    Thomaz

  3. leafar
    February 19, 2007

    Very smart move.

    very interesting … and happy to try all the services in the comments.

    My url is not a service, it’s just a mapping of the different piece of identity puzzle : http://dev.culturels.eu/myspace/Leafar.html

  4. John Tropea
    February 19, 2007

    There are many meta-identity services that collate all your stuff, even within a social network…I probably found all these sites via eHub.

    But I see your method is incorporating it into your own site…nifty!

    List:

    Ziki, Lijit, Jaiku, ProfileLinker, SocialNetwork.In, Pidy, ozmozr, Profilactic, mugshot

    http://libraryclips.blogsome.com/2007/02/19/mugshot-meta-content-stream-social-network/

  5. Chris Saad
    February 19, 2007

    Hi Emily – not a bad idea. I have posted how Touchstone might make this easier for other people to implement.

    Manual Trackback:

    http://www.touchstonelive.com/blog/2007/02/attention-streams-your-life-in-feed.html

    Cheers!

  6. Narendra
    February 19, 2007

    Emily,

    You really should have a look at 30 Boxes because we have been doing this sort of thing for a while and continuing to improve on it.

    1) we have a built in lifestream style reporting about your buddies

    http://www.flickr.com/photos/factoryjoe/375079039/

    2) your social identity is automatically spun out as an aggregated public buddy page

    http://www.30boxes.com/user/3/Narendra

    3) and we let you visualize all sorts of date stamped data with a slick ajax calendar UI

    http://www.30boxes.com/boxed

    for example, here is a bunch of your stuff!

    http://tinyurl.com/2y9nlg

  7. Alan Keister
    February 19, 2007

    This is almost exactly what we have built into AIM.  Check out AIM Buddy Updates where you can enter your flickr, delicious, youtube, personal blog, and any feed about you.  We show an indicator next to your name in the buddy list when you have updates.  We also provide an aggregated RSS feed that looks like http://buddyupdates.aim.com/YOUR_AIM_NAME/rss.xml that anyone can subscribe to.  If you want to see screenshots, look here http://farm1.static.flickr.com/167/395377424_061a226c7f_o.png.  You need an AIM Screenname to use this but you don’t have to use our IM client.

  8. Paulo Diniz
    February 20, 2007

    Nice points Emily. I have a concept that is similar to what you’ve described on the back of my head for some time, and i document my ideas for it on my blog:

    http://notariussystem.blogspot.com

    The main post on it is:

    http://notariussystem.blogspot.com/2006/11/taglogger-idea-for-tag-based-pimgeneral.html

  9. leafar
    February 20, 2007

    You should take a look at Nicholas Felton’s lifelog (it’s an annual publication that includes manual data).

    It’s brilliant and fun :

    http://feltron.com/06report_index.html

  10. Christian Carter
    February 20, 2007

    Hi Emily,

    stumbled upon your post here, and I actually kind of have this built.  YOu can see it in action at http://tumblr.metacampsite.com.

    If you are interested, contact me at cdcarter NOSPAM @ gmail.com

  11. hubs
    February 20, 2007

    just a little more info for you wordpress users out there.

    I have a “lifestream” on my wordpress blog (http://www.artifacting.com/blog/life) that incorporates the work that incorporates the work of chris j davis (http://chrisjdavis.org/2006/11/26/lifestream-ala-wordpress/)

    elliot back has created a wordpress plugin for lifestreams: http://elliottback.com/wp/archives/2006/12/14/lifestream-wp-plugin-for-wordpress/

  12. Emily Chang
    February 20, 2007

    Hi all, thanks for the comments so far.  Good to see all these resources, most of which I’ve come across.  Some are new to me so I’ll check them out. 

    Narendra, I’ve definitely played with the great tools you have at 30boxes for mashing up feeds and pointed my client, Stylehive to it.  They wrote about it here. Chris, I’ll ping you about Touchstone…don’t think I have my login.  Alan, the AIM feature would be great for those hooked on IM.  And, Sam, I’ve just started playing with Pipes. Grant at Downloadsquad wrote about my data stream and the next day, decided to do the same aggregation of his feeds using Pipes

    For me, having a portion of that data I create in my own system is important to me, but for others, web services will work well.  RSS provides a conduit to work with data and now we’re seeing more and more services focused on aggregation, personalization, and attention streams. As Stowe writes in his post, there are other types of traffic and flow that are also part of this model. 

    I’ve been calling it a stream but knowing that it will eventually turn into an ecosystem – one where we can get a holistic view of our own and other’s flow of information. 

    In terms of my data stream, there are other simple hacks I might try in terms of having another UI or interaction layer beyond the simple list or calendar format.  Still pondering this.

    Let me know if you have other resources I should check out.

  13. Steve
    February 28, 2007

    I love this post. I’ll bookmark this page.. Please keep me updated..Thank u so much..

  14. Stephane Cheikh
    March 2, 2007

    Hi Emily,

    Interesting post that made me think about work! Yes, work, where I would be very interested to get a “My Data Stream” that will aggregate all my working activities into one data stream. 

    Imagine if you could see, per year, in this context: time, date, type of activity, location, task, information, interest, request, projects, and so on. 

    What are the corporate applications I visited, with a summary of what was completed, trainings, events, seminars I took during the year, events I participated in, blogging about, what I wrote, accomplished, etc…

    All that information would be an online activity mash-up that would allow each staff member to discover their own patterns and to share their activities across the intranet web in one chronological stream of data.

    This is getting me going! I will check if anything can be installed within a corporate network and beta test.

  15. Petteri Koponen
    March 4, 2007

    Moi Emily —

    A very interesting post! Your data stream is pretty similar to what we are doing at Jaiku, although we have called our chronological list of activities a presence stream (or lifestream a la Stowe).

    Coming from Finland, the land of Santa Claus, sauna, and Nokia, we believe that the mobile aspect is especially important in services like this: People are carrying mobile handsets with them all the time and a lot of interesting actions are recorded by the handset. Luckily, the handsets are increasingly connected to the Internet over GPRS, EDGE, and even WiFi, and therefore the mobile lifestream can be shared as well. And, even more importantly, one can be constantly aware of the life of their friends and family via the mobile service.

    Jyri Engestrm recently gave a mobile-focused speach on the subject at ETel, in addition to introducing the latest version of our service.

  16. Bart
    March 5, 2007

    Emily,

    I’ve been searching for a tool like that, and found this open source aggregator: http://gregarius.net/.

    It collects your RSS-feeds.

    You can see it in action at: http://www.bartdewaele.be (my lifestream, in Dutch).

  17. Mark Krynsky
    March 6, 2007

    Great post Emily. It’s nice to see how your post and stream implementation has proliferated across the blogosphere and is generating strong interest. I recently have become quite obsessed with the Lifestream concept and am excited about the possibilities in seeing this it evolve. I’ve become quite an evangelist for it at my site having recently written ”Lifestream – Could it be the next big thing?”. It would be nice to see a standards based approach develop. I look forward to following your site for updates.

    Mark

  18. hypotheek
    March 6, 2007

    I’ve use the Magpie RSS class to feed my site with rss feeds.

    It works great including a cache function to prevent massive requests.

  19. Jean-Patrick Smith
    March 9, 2007

    Wow, learning about new web 2.0 technologies gives me butterflies in my stomach.

    I was actually thinking about this concept earlier today, and BHAM! It’s out there already.

    My idea was to automatically be able to post to those various services as well, kind of like a “home base” for all your networked sites.

    Have a wordpress blog? Post from home. Wanna upload a flickr picture? Do it with ease, all from the same application.

    What needs to be done, and is probably in the works already, is a microformat API something or other, a new standard, that will allow one application to easily control all the facets of any service.

    It’s what will most likely happen in the future.

  20. k malone
    March 15, 2007

    Hi Emily,

    Great post – a lot of great links in there. Another great resource that some might find interesting is http://www.kushcash.com – feedbcak or questions form readers would be greatly appreciated!

  21. Ian
    March 19, 2007

    thats awesome! i had a brief idea a couple weeks ago but havent really developed the purpose let alone the ui…

    http://iancamarillo3.blogspot.com/2007/03/this-website-tracks-your-web-usage-for.html

  22. MJ
    April 2, 2007

    Hi Emily

    This was very interesting – thanks!

  23. Tranix
    April 7, 2007

    Interesting…

  24. Jack
    April 8, 2007

    Ah, a post for my bookmarks! Thx for all the links in comments as well.

  25. Rob
    May 29, 2007

    I like your data stream. I have tried istalkr.com and several others and the only I could get to work well pulling in various RSS feeds has been Tumblr. I have mine set up at http://robslifestream.tumblr.com/

  26. Kamran
    June 4, 2007

    Quite good, but there is no control on privacy and chances are there for the exposure. I have tried the magnificent tools available at the 30 boxes and came out with great results. 

  27. The Mobile Phone Man
    July 21, 2007

    Hi Emily thanks for the time and effort you put into this blog its great.

    Thanks,
    Samuel

  28. Postman
    August 28, 2007

    This is a great article and idea. Having a data stream is something extra for readers. I tried some of the above suggestions and if I get it working (I’m not that technical) I will implementate one in my blog.

  29. Steven Hodson
    September 1, 2007

    As cool as this whole “lifestream” idea is people seem to be ignoring the downside to it.

    http://www.winextra.com/2007/09/01/who-needs-carnivore-when-you-have-lifestreams/

  30. Billy
    September 14, 2007

    Hi Emily

    This was very interesting – thanks!

  31. Christophe
    September 18, 2007

    Kudos to you Emily, the data stream ‘s nice and thanks to Chris aswell for the Touchstone link which does make it easy to implement.

  32. Chris Saad
    September 18, 2007

    Emily – since this post seems to keep getting traffic – I might post a pointer to a blog post about converting your Lifestream into APML as well.

    http://blog.engagd.com/2007/08/creating-apml-file-from-your-lifestream.html

    APML, of course, being an XML format for storing your interests – your “Attention Profile”

  33. Lketophone
    October 20, 2007

    Great blog Emily, keep up the good work :-)

  34. Malz
    October 23, 2007

    Love your posts! Trying to start my own..

  35. Jan-Willem Bobbink Skeeleren
    February 2, 2008

    It’s a really nice concept. I’ve build a RSS feed processor myself for all the different websites. Thanks for sharing this project. Quite useful!

  36. Jason
    February 2, 2008

    Very smart move.

    very interesting … and happy to try all the services in the comments.

  37. Francis
    August 4, 2008

    You wrote an amazing and interesting article. Thank you and hope you keep us updated.

  38. hid
    January 30, 2009

    Excellent read, I just passed this onto a colleague who was doing a little research of data streams in general—he bought me lunch because I found it for him haha. so Thanks for lunch! :)