My Data Stream
After a year and a half of using social applications heavily, I recently had to revisit the plan to aggregate all my activity into one data stream. As the calendar rolled to 2007, I kept wishing I could look at all my social activity from 2006 in context: time, date, type of activity, location, memory, information interest, and so on. What was I bookmarking, blogging about, listening to, going to, and thinking about? I still had the urge to have an information and online activity mash-up that would allow me to discover my own patterns and to share my activity across the web in one chronological stream of data (to start with anyway).
While there are some great services like SuprGlu for aggregating feeds into a browseable blog format (see my SuprGlu here), it’s still a hosted service and I need something that runs on my own server so I can have greater flexibility.
Others have been toying with this idea as well. A few months ago, Jeremy Keith wrote about his experience with this very issue in “Streaming my life away.” He decided to write a PHP script that would track several RSS feeds (Twitter, Flickr, Del.icio.us, Last.fm, and blog posts) by time-stamp and then display them in chronological order. It doesn’t do any caching but “the important thing is that it’s keeping the context of the permalinks (song, link, photo, or blog post) and displaying them ordered by date and time.” His experience resonated with me. I tried his PHP script but it didn’t quite work on my blog. Jeff Croft also has a similar implementation at his tumblelog, which appears to be down now.
The night I was playing with this idea, I twittered about it and heard from Boris Anthony who showed me his ggalaxy multi-contextual aggregator, a great system for seeing all of your friends’ (or favorite feeds) output in one place. In this case, the river of news is limited to 7 days and not archived. ggalaxy will be a WordPress plugin.
For my immediate purposes, I wanted to have the data from my selection of RSS feeds parsed and actually imported into my own MySQL database so it was archived and referenced in my own system. After doing some more research to see if anything was out there specifically for my blog software, I found feedgrab, an amazing plugin by Andrew Weaver in the ExpressionEngine forums. It worked well but needed the ability to import data from RSS feeds with unique criteria. After an email exchange with Andrew, he informed me this was already in the works. He updated the plugin and added new features within a couple of days. Big thanks to Andrew for writing this (and all the great programmers that support the open source community with their expertise).
My data stream:
http://www.emilychang.com/stream
I set up a separate blog for each of my services with custom fields for the data and I’m tracking my RSS feeds from:
The plugin script fetches the RSS feed from each service and then parses the XML and imports the data from pre-determined fields into an ExpressionEngine blog database. This is then displayed on my stream page by time-stamp with an icon to indicate the service or my blog source. You can click on the mini-calendar to view activity by day.
The script runs automatically every day and imports data from the various services. I can now search that database with the ExpressionEngine search engine by keyword(s). While this is internal to me at the moment, I’ll make the search public after I’ve customized it a bit. If I search for “urban” I would see results for all my bookmarks tagged red in stylehive, del.icio.us, music I had listened to through last.fm with the title or band “red,” photos I posted to flickr with that tag, status messages at twitter, events I was watching or attending at upcoming, Ideacodes news, or eHub apps with a match.
When I wrote about this back in September 2005, I was thinking that this type of aggregating by tag would also be helpful.
But, I’m also finding that the more tagging I do in various places and for different types of content, the more I feel my metadata becoming siloed into these other databases. Even though today’s new applications have moved beyond rigid categories and allow me to tag my own data, the current state of tagging doesn’t allow for us to map these information relationships from one site to another – or perhaps more aptly, to see a snapshot of our metadata universe. I’m still looking for a way to see how my current tags crossover with one another, and with tags made by others – the aerial view.
This type of attention stream seems harder to do right now since each web service does tags quite differently within RSS enclosures.
For now, this activity stream idea is providing the start to a holistic view of my activity across online networks: both my own and the ones I use. In turn, this acts as a conduit for you, the reader. Rather than just a static “recommended links” page or a blogroll, the data stream opens up my activity to you in semi-realtime and at one website. I’ll continue tweaking the content, navigation, and layout so watch for changes. Let me know what you think!
My data stream:
http://emilychang.com/stream
UPDATES:
See discussion about my post on Techmeme
March 4: Watching the stream by Jeremy Keith
Life Data Stream by Thomas Vander Wal
Traffic And Flow by Stowe Boyd
Attention Streams – Your life in feed by Chris Saad
Taming your own river of news by Grant Robertson on DownloadSquad
Everyone’s data streams for everything visible everywhere by Ross Dawson
When What i Say and do Probably Matters the Most by Daniela Barbosa
Marketing 2.0 by Robyn Tippins
My Life Stream by Jack Vinson
Lifestreams could help create new personalised discovery engines by Sam Sethi
IDEA #23 – Your Syndicated Chronological Life by Steve Poland
Lifestream – Could it be the next big thing? by Mark Krynsky
39 Comments
Join the discussion and tell us your opinion.
Check out Yahoo! Pipes for doing things like this easily without any real coding:
http://pipes.yahoo.com
Hey Emily,
We have just released a new web service last week that you might want to check. It’s called Slifeshare (http://www.slifeshare.com). It’s a space where you can share with people (or just check for youself) what you’ve been doing, pages you’ve been visiting, music you’ve been listening to, etc. It’s all happens with the Slife client (http://www.slifelabs.com).
It’s a bit wacky in that you can expose a lot of stuff, but at the same time we have lots of privacy controls. You only publish what you, live stream of activities, just stats, etc.
Take Care,
Thomaz
Very smart move.
very interesting … and happy to try all the services in the comments.
My url is not a service, it’s just a mapping of the different piece of identity puzzle : http://dev.culturels.eu/myspace/Leafar.html
There are many meta-identity services that collate all your stuff, even within a social network…I probably found all these sites via eHub.
But I see your method is incorporating it into your own site…nifty!
List:
Ziki, Lijit, Jaiku, ProfileLinker, SocialNetwork.In, Pidy, ozmozr, Profilactic, mugshot
http://libraryclips.blogsome.com/2007/02/19/mugshot-meta-content-stream-social-network/
Hi Emily – not a bad idea. I have posted how Touchstone might make this easier for other people to implement.
Manual Trackback:
http://www.touchstonelive.com/blog/2007/02/attention-streams-your-life-in-feed.html
Cheers!
Emily,
You really should have a look at 30 Boxes because we have been doing this sort of thing for a while and continuing to improve on it.
1) we have a built in lifestream style reporting about your buddies
http://www.flickr.com/photos/factoryjoe/375079039/
2) your social identity is automatically spun out as an aggregated public buddy page
http://www.30boxes.com/user/3/Narendra
3) and we let you visualize all sorts of date stamped data with a slick ajax calendar UI
http://www.30boxes.com/boxed
for example, here is a bunch of your stuff!
http://tinyurl.com/2y9nlg
This is almost exactly what we have built into AIM. Check out AIM Buddy Updates where you can enter your flickr, delicious, youtube, personal blog, and any feed about you. We show an indicator next to your name in the buddy list when you have updates. We also provide an aggregated RSS feed that looks like http://buddyupdates.aim.com/YOUR_AIM_NAME/rss.xml that anyone can subscribe to. If you want to see screenshots, look here http://farm1.static.flickr.com/167/395377424_061a226c7f_o.png. You need an AIM Screenname to use this but you don’t have to use our IM client.
Nice points Emily. I have a concept that is similar to what you’ve described on the back of my head for some time, and i document my ideas for it on my blog:
http://notariussystem.blogspot.com
The main post on it is:
http://notariussystem.blogspot.com/2006/11/taglogger-idea-for-tag-based-pimgeneral.html
You should take a look at Nicholas Felton’s lifelog (it’s an annual publication that includes manual data).
It’s brilliant and fun :
http://feltron.com/06report_index.html
Hi Emily,
stumbled upon your post here, and I actually kind of have this built. YOu can see it in action at http://tumblr.metacampsite.com.
If you are interested, contact me at cdcarter NOSPAM @ gmail.com
just a little more info for you wordpress users out there.
I have a “lifestream” on my wordpress blog (http://www.artifacting.com/blog/life) that incorporates the work that incorporates the work of chris j davis (http://chrisjdavis.org/2006/11/26/lifestream-ala-wordpress/)
elliot back has created a wordpress plugin for lifestreams: http://elliottback.com/wp/archives/2006/12/14/lifestream-wp-plugin-for-wordpress/
Hi all, thanks for the comments so far. Good to see all these resources, most of which I’ve come across. Some are new to me so I’ll check them out.
Narendra, I’ve definitely played with the great tools you have at 30boxes for mashing up feeds and pointed my client, Stylehive to it. They wrote about it here. Chris, I’ll ping you about Touchstone…don’t think I have my login. Alan, the AIM feature would be great for those hooked on IM. And, Sam, I’ve just started playing with Pipes. Grant at Downloadsquad wrote about my data stream and the next day, decided to do the same aggregation of his feeds using Pipes.
For me, having a portion of that data I create in my own system is important to me, but for others, web services will work well. RSS provides a conduit to work with data and now we’re seeing more and more services focused on aggregation, personalization, and attention streams. As Stowe writes in his post, there are other types of traffic and flow that are also part of this model.
I’ve been calling it a stream but knowing that it will eventually turn into an ecosystem – one where we can get a holistic view of our own and other’s flow of information.
In terms of my data stream, there are other simple hacks I might try in terms of having another UI or interaction layer beyond the simple list or calendar format. Still pondering this.
Let me know if you have other resources I should check out.
I love this post. I’ll bookmark this page.. Please keep me updated..Thank u so much..
Hi Emily,
Interesting post that made me think about work! Yes, work, where I would be very interested to get a “My Data Stream” that will aggregate all my working activities into one data stream.
Imagine if you could see, per year, in this context: time, date, type of activity, location, task, information, interest, request, projects, and so on.
What are the corporate applications I visited, with a summary of what was completed, trainings, events, seminars I took during the year, events I participated in, blogging about, what I wrote, accomplished, etc…
All that information would be an online activity mash-up that would allow each staff member to discover their own patterns and to share their activities across the intranet web in one chronological stream of data.
This is getting me going! I will check if anything can be installed within a corporate network and beta test.
Moi Emily —
A very interesting post! Your data stream is pretty similar to what we are doing at Jaiku, although we have called our chronological list of activities a presence stream (or lifestream a la Stowe).
Coming from Finland, the land of Santa Claus, sauna, and Nokia, we believe that the mobile aspect is especially important in services like this: People are carrying mobile handsets with them all the time and a lot of interesting actions are recorded by the handset. Luckily, the handsets are increasingly connected to the Internet over GPRS, EDGE, and even WiFi, and therefore the mobile lifestream can be shared as well. And, even more importantly, one can be constantly aware of the life of their friends and family via the mobile service.
Jyri Engestrm recently gave a mobile-focused speach on the subject at ETel, in addition to introducing the latest version of our service.
Stephanie,
Check out my post on datablogging:
http://libraryclips.blogsome.com/2007/01/26/datablogging-personal-knowledge-blogging
Emily,
I’ve been searching for a tool like that, and found this open source aggregator: http://gregarius.net/.
It collects your RSS-feeds.
You can see it in action at: http://www.bartdewaele.be (my lifestream, in Dutch).
Great post Emily. It’s nice to see how your post and stream implementation has proliferated across the blogosphere and is generating strong interest. I recently have become quite obsessed with the Lifestream concept and am excited about the possibilities in seeing this it evolve. I’ve become quite an evangelist for it at my site having recently written ”Lifestream – Could it be the next big thing?”. It would be nice to see a standards based approach develop. I look forward to following your site for updates.
Mark
I’ve use the Magpie RSS class to feed my site with rss feeds.
It works great including a cache function to prevent massive requests.
Wow, learning about new web 2.0 technologies gives me butterflies in my stomach.
I was actually thinking about this concept earlier today, and BHAM! It’s out there already.
My idea was to automatically be able to post to those various services as well, kind of like a “home base” for all your networked sites.
Have a wordpress blog? Post from home. Wanna upload a flickr picture? Do it with ease, all from the same application.
What needs to be done, and is probably in the works already, is a microformat API something or other, a new standard, that will allow one application to easily control all the facets of any service.
It’s what will most likely happen in the future.
Hi Emily,
Great post – a lot of great links in there. Another great resource that some might find interesting is http://www.kushcash.com – feedbcak or questions form readers would be greatly appreciated!
thats awesome! i had a brief idea a couple weeks ago but havent really developed the purpose let alone the ui…
http://iancamarillo3.blogspot.com/2007/03/this-website-tracks-your-web-usage-for.html
Hi Emily
This was very interesting – thanks!
Interesting…
Ah, a post for my bookmarks! Thx for all the links in comments as well.
I like your data stream. I have tried istalkr.com and several others and the only I could get to work well pulling in various RSS feeds has been Tumblr. I have mine set up at http://robslifestream.tumblr.com/
Quite good, but there is no control on privacy and chances are there for the exposure. I have tried the magnificent tools available at the 30 boxes and came out with great results.
Hi Emily thanks for the time and effort you put into this blog its great.
Thanks,
Samuel
This is a great article and idea. Having a data stream is something extra for readers. I tried some of the above suggestions and if I get it working (I’m not that technical) I will implementate one in my blog.
As cool as this whole “lifestream” idea is people seem to be ignoring the downside to it.
http://www.winextra.com/2007/09/01/who-needs-carnivore-when-you-have-lifestreams/
Hi Emily
This was very interesting – thanks!
Kudos to you Emily, the data stream ‘s nice and thanks to Chris aswell for the Touchstone link which does make it easy to implement.
Emily – since this post seems to keep getting traffic – I might post a pointer to a blog post about converting your Lifestream into APML as well.
http://blog.engagd.com/2007/08/creating-apml-file-from-your-lifestream.html
APML, of course, being an XML format for storing your interests – your “Attention Profile”
Great blog Emily, keep up the good work :-)
Love your posts! Trying to start my own..
It’s a really nice concept. I’ve build a RSS feed processor myself for all the different websites. Thanks for sharing this project. Quite useful!
Very smart move.
very interesting … and happy to try all the services in the comments.
You wrote an amazing and interesting article. Thank you and hope you keep us updated.
Excellent read, I just passed this onto a colleague who was doing a little research of data streams in general—he bought me lunch because I found it for him haha. so Thanks for lunch! :)