Webtrends Streams: Behind the Scenes

Webtrends Streams is amazing.  It's different.  A fresh, revolutionary idea (or, set of ideas really), that will fundamentally change how we think of digital analytics.  You will want to see it, very soon, and it will blow you away.

I'm proud to be part of the team putting this innovation together.  What we originally set out to do was to reinvent real-time analytics.  It was a great time for us to rethink our approach to real-time data for several cool reasons:

1. Campaigns, and verticals like media, retail and travel, are requiring analysis sooner:  In the summer of 2011 we had just created a powerful new tool to collect and analyze Facebook campaigns.  In doing so we analyzed tens of thousands of Facebook posts and determined that any given post at that time had a shelf-life of 11 hours.  That was a real eye-opener.  Marketers only have a few short hours to determine whether a post (a micro-campaign if you will) is succeeding or not, and whether to double-down on the post (sponsored post, other potential sources), or whether to get the next post ready to rollout.
2. Real-time innovation through Reinvigorate: We were lucky enough to acquire Reinvigorate, still the best real-time solution available on the market today.  Behind the excellent product is some very smart IP that we immediately poured into a new Heatmaps solution, creating our first real (very) big data infrastructure.  Also behind that was a new approach to real-time data (well done Sean!).  With Reinvigorate, you know, immediately, who is on your site (or using your app).  You can tell how many active visitors are currently viewing any page.  It's very powerful, and got us thinking.
3. Killer technology shared by other creative companies: Twitter, LinkedIn, Facebook, and others, were starting to release and share more mature open source projects that begin to offer incredibly reliable, stable, rapid data movement to support the difficult task of true real-time requirements.

So, we started prototyping new real-time analysis tools...and it didn't quite feel right.  It felt incomplete, and "only" evolutionary.  It was incredibly robust, enriched with additional data, and faster than anything else available, but that didn't feel like enough of a win to make a major investment to complete.  There are definitely places where real-time data is important, and it can fundamentally move the needle for organizations that need it.  But we thought we could do more.

We started experimenting with the idea of expressing the data we collect differently.  What might it look like to enrich the data we're collecting, and then send it out an API as soon as it is collected?  What if we could also offer our customers the ability to subscribe to receive all events, or just a subset of events they are interested in?  And if they only need certain variables (parameters), let's reduce the stream of data to only include those variables on each event.

We ended up building new data viz's as well...

I still remember the time Andrew showed me the data flowing through the Streams API.  I work with a bunch of very smart people, and we were all blown away at what we were seeing.  Now it started getting fun.  We knew were were on to something special.

For the first time, we could "see" all views, events, clicks, plays, pauses, opens, closes, Likes, comments, add-to-carts, purchases, scenario steps, searches - EVERYTHING - as it happens.  Yep, as it happens.

Oh, and are you interested in focusing the data to specifically see product views, add-to-cart, or other scenario events?  Just filter the stream to only send those events.  What about watching referring sources, campaign IDs, and resulting landing page?  No problem.  How about all video starts, or % complete, or errors even?  Easy.  Curious how many people are using your mobile app right now, and which content is being viewed broken down by city and mobile device type?  You got it.

We're really proud of how flexible and powerful Streams is.  And we are just getting started...wait till you see what we have in store next.  Let me know if you want to see it...it's very cool.  I'll post more later, but for now...start here: http://trnd.me/PMdrl5