A Mashup: Mapping Visitors

I thought it might be interesting to see a map of where visitors to the LiquidSculpture site are located. I first started down the Google maps approach, but then I came across a very creative Excel macro that Jeffrey McManus created for Yahoo! Maps that was easy to modify and use (thanks Jeffrey!).

Data Gathering
The idea is relatively simple. Create a spreadsheet of geography data from your analysis tool, then run a macro against it to create the XML data to feed into Yahoo! Maps. So, let's start with gathering the geography data.

Using WebTrends, I ran a Geography Drilldown report for the month of October. You can see in this pie chart that ~61% of the visits to LiquidSculpture were from North America. The Yahoo! Maps data is limited to North America as far as I can tell, so we'll stick to that geography for now.

Drilldown Data
This drilldown shows further granularity into the data (North America -> Canada -> British Columbia -> Vancouver), noting that visits from visitors in Vancouver, B.C., make up .77% of the overall visits during the month. Martin demonstrated his photography at a show in the San Juan Islands a year ago, perhaps his Vancouver visitors should get a special invitation next time he's up in that neck of the woods.
Get Out The Map
So, let's get some data on a map. Now that we have the data, we need to export it. After running the Geography Drilldown report in WebTrends, I exported the data to a CSV file (you could use the Excel export functionality too, but the CSV export works great in this example). I then modified Jeffrey's Excel template to select the cities out of the exported data to create the map below.

You can see the map above by referring to this link which pulls the XML data from a saved file.


Technical Notes
Yahoo! Maps is easy to use and very flexible. You create a XML file and feed it to their API. There are other tools available via the API that I'm not taking advantage of - perhaps if there's enough interest, I'll dig into it further.

If you use WebTrends, you should be able to use the same Excel template I used to map your visit data on Yahoo maps. Please note that this is not a WebTrends product. It is not endorsed or supported by WebTrends. I'd be happy to help provide some help if you are having difficulty using it. Even if you don't use WebTrends, but your analytics package provides you with geography data that you'd like to map, you're welcome to use my template, or grab Jeffrey's original template and hack away.

My template includes a few changes to the original:

• I created "groups" which allow a categorization of the data for different colored icons. I set them to greater than 1%, .05-1% and less than .05%. Feel free to modify those values in the macro if you need to - I've highlighted the two areas that need to be changed with a *** in the comments
• I have modified the instruction box to note that you should sort the data by visits (and pageviews) before mapping it. The reason is that the WebTrends CSV export sorts the data by geography first. If you have a large dataset, you may miss other geographies unless you sort the data by visits first.
• The macro is looking for a few specific items that are in the WebTrends CSV exported file. If you're not using WebTrends, and are having difficulty making it work, try Jeffrey's original template.

Let me know if you have any feedback or questions on this.

Filed in: map geo liquidsculpture webtrends analytics yahoo

Managing an Operation

Running an internet-based service is exhausting, as I'm sure many of us in the web analytics world can attest. With careful planning and a watchful eye to the metrics of your environment, you can stay ahead of the game and avoid outages.

The folks over at Typepad have had some operational pains recently. According to their apology, they are moving systems between data centers (running out of space, power, etc.), and have experienced some outages in hardware that they've never had problems with before. They mention storage equipment and network equipment among the recent failures. Ouch. All this on top of 10-20% growth in bandwidth each month (currently running at 250Mbps - impressive!). They are modifying and fixing the airplane while it is in flight!

Customers, observers and even Robert Scoble have commented on this issue. Most of the press has been very generous toward the Six Apart team, especially given that they were getting many complaints about the performance of the service.

I've been a Movable Type customer for a couple of years. Six Apart develops terrific software, and they've migrated their software to a hosted service extremely well. I'm sure they're going to quickly overcome the issues they've faced, and the service will be stronger for it. It's very impressive how they continue to innovate, grow very quickly, and still communicate well with their customers.

Filed in: operations

Do I Have Your Attention?

I'm intrigued by AttentionTrust's efforts on storing clickstream data. It's obviously very similar data to that which we collect to provide web analytics reporting. It is a different frame of reference though in that they are putting control in the hands of each visitor.

The information collected would be extremely valuable to marketers. What might Amazon "suggest" I consider looking at knowing that I read various Web Analytics and Search sites? Surely, topical books by Eric and John would magically appear as I browse the site. And that's without even logging in or previously visiting Amazon as far as they know (because, of course, I've deleted my cookies ;-)

According to the AttentionTrust site, there is still only one demo service provider collecting attention data. As they've noted, they have a chicken/egg problem: they need service providers and they need individuals to use the service.

So who gets it all started? Does it require a big player like Amazon to throw its weight behind the effort? Does the omnipresent Google get involved to add credibility, and balance doing no evil with collecting all information about everyone (of course...they already do this...without needing to share!)?

Or does the effort get a grassroots kick start? For example, could del.icio.us become a trusted service? Many individuals already trust del.icio.us to store some of their web history (made even easier to do with Flock). Wouldn't it be great to have them collect all of my clickstream data, allowing me to annotate (tag) particular pages of interest as I go (a browser plugin perhaps)? Then as I visit other sites, my del.icio.us username could be sent in the header of each request, allowing those sites to "see" my interests, for which I would be rewarded with generous discounts and offers - right?

Analytics products might be looking for new sources of information in this new world. There are still plenty of important metrics to be captured per web property, but there would be much deeper visitor data available to mine, given that we can solve a plethora of privacy issues first!

Stan James wrote the AttentionTrust ATX (firefox plugin). He's also the author of Outfoxed, which is a very interesting mashup of social networks and web interaction. He's taken the concept of trust to a different level, allowing folks to define other individuals they trust, to allow them a richer experience when searching and browsing. Very cool stuff.

Filed in: attention analytics

LiquidSculpture Analysis - BoingBoing Bounce

In the months and weeks leading up to the "event", those visitors who were lucky enough to find LiquidSculpture were among a select few internet travelers to come across this pioneering work.

In August and through most of September, Martin's site averaged around 30 visits per day. On September 19, BoingBoing posted a link to the site and since then, things haven't quite been the same. Traffic spiked up to 20,000 visits on that day, and has bounced around quite a lot in these subsequent weeks. Note this graph showing the number of daily visits across the month of September...yes, there are visits for the first 18 days of the month...they are obviously dwarfed by the traffic late in the month. BoingBoing's post was clearly bringing visitors to the site.

What's extraordinary about the nature of the reach of sites like BoingBoing is how quickly posts are spread through other bloggers. Many small sites picked up the post, as did some larger sites. As you can see from the visits graph above, traffic tapered off after a day or two from BoingBoing as it was no longer on the home page. But enough other smaller blogs were able to make up the difference and then some through the next few days.

Location, location, location
So, who are these visitors? Are blog-referred (blogerred?) visitors any different than other visitors? As BoingBoing is savvy enough to post their own stats, we know that the site is viewed by over 2M unique visitors per month. Unfortunately, their "countries" report is broken at the moment, so we can't determine the geographical distribution of their reach directly. But here's a geography drilldown report showing where the visitors to LiquidSculpture were coming from on September 20 with Western Europe expanded to show how many visits were from Sweden!

Referrers make all the difference
When analyzing blog/RSS data, one very important metric is referrers. Bloggers should keep a close eye on who is driving traffic to their site. The graph below is terrific as it shows a few things pretty clearly. 1) Make sure your ISP is ready for peak traffic! As you can see, there was a bit of a drop (ok...a complete drop) in traffic as Martin's ISP responded to the load of the day. 2) The referring traffic mix changed over the day as MetaFilter then Digg followed up on the BoingBoing post. 3) Many visitors followed up by posting tags to Del.icio.us, where it became one of the popular items for the day.


Another thing that jumps out when analyzing this data: Why are there so many "direct traffic" referrers? Are people actually typing in the URL into their browser? Or getting an email and clicking on a link (from Outlook or some other non-browser based email tool)? I believe it's something new that web analytics tools are only beginning to comprehend properly - these visitors are looking at sites like BoingBoing through RSS/news readers. They are clicking on the link from within their RSS reader and opening a new browser...thus no referrer. Note that you do get a referrer from visitors using web-based browsers like Bloglines (Bloglines was the 8th most common referrer for September 20th - the highest referrer for any feed aggregator - I sure wish we knew which readers folks were using!).

Side Bar
Note that LiquidSculpture has not been picked up by slashdot, but if it is, referrers will be much easier to measure. Slashdot does not provide links in their feeds, so readers who wish to find out more information about a referenced site need to first go through slashdot. Smart marketing? Hmmmm. Regardless, it is definitely true that BoingBoing is not completely accurately measuring their readership (they don't know if I read an article or not, or visit a link that they reference), nor can sites accurately know how much BoingBoing traffic has been referred to them.

Money matters
Of course, Martin does sell his unique art, but he does not have any commerce mechanism directly on his site. Visitors to his "ordering" page can email him to make orders, or visit cafepress for a few items they distribute for him. I was curious whether BoingBoing readers were more or less likely to visit the ordering page, so I ran a report to breakdown visits to the ordering page by initial referrer to the site. As you can see, even though BoingBoing readers directly accounted for 22% of the visits to the site (from the stats above), they accounted for 30% of the visits to the ordering page. MetaFilter visitors and Digg visitors were less likely to visit the ordering page based on overall visits from those sites.



What about the browser war?
And the browser of choice for all of these fabulous new visitors? Let's just say it isn't IE - which only accounted for 33% of the visits on September 20. Firefox led the way with over 50% of the visits, and Safari came in 3rd at 10% of the visits. I've been pondering why the number of Firefox visits is so much higher than what BoingBoing sees, but their stats are showing hits and I'm using visits, so it's obviously not a perfect comparison. Also, I imagine that a lot of RSS readers send the agent string for IE + the reader name, so they probably get lumped into the IE stats.

Let me know if you have any comments or questions about this data. Many thanks to Martin for the wonderful art, access to some excellent data, and for putting up with my endless questions!

Filed in: liquidsculpture analysis

LiquidSculpture Analysis - AOL impact

Martin cornered me yesterday and said, "I've got three little letters for you...AOL". His LiquidSculpture site was featured on the front page of AOL. Very cool.

Of course, that meant another bump in traffic. What was the impact? Not as much as we had thought it might be. You can see from this hourly graph that there was definitely an uptick in visits when the page was featured on the site.

The traffic tapered off pretty quickly though, and although the link is still available on a second-level page of AOL, it obviously isn't generating as much interest as yesterday.

We're puzzled by one aspect of this traffic though, and that is with the lack of referrers. We assume that AOL is stripping the referrer information out of the requests. They've been known to be creative with their customer's traffic, and of course, it is their network, so they can do whatever they want!

And how did LiquidSculpture end up on AOL's homepage? Martin has no idea. Someone obviously saw his site on BoingBoing or Fark or the NYTimes. I think we should start a competition though. We'll call it "MakeMeHotOrNot". We'll invite all of the portals and search engines to try to drive traffic to his site, and we'll measure it. Who do you think would drive the most?

Filed in: liquidsculpture analysis aol

How Prescient Art Can Be...

Andy Grove used to say (and perhaps he still does!) that only the paranoid survive. Google is helping us paranoid folks survive! I love this view of the future. Although, given Google's track record, I'd say it will be more like 2010 when all of these services should be available (beta starting next year of course!).

Thanks James!

Filed in: funny google art

Blog Analytics Tool - MyBlogLog

MyBlogLog is a simple tool that provides bloggers with a very straight-forward way to determine what external links their visitors are choosing.

Blogs and feeds have different requirements for analytics than most web properties. MyBlogLog helps answer an important question for bloggers: What external links are my visitors choosing? MyBlogLog provides their customers with some javascript to allow them to wrap an onclick event around external links.

Good idea. It probably needs to get wrapped into a larger web analytics offering to provide richer overall traffic data. But for a free tool that's so easy to deploy, this is a smart start for bloggers who would like to understand more about where their site is driving traffic.

Filed in: blog technology

Paying (for) Attention

For awhile now I've been watching a parallel analytics field getting started and I believe we ought to be discussing the implications within the web analytics world. I'm talking about attention. It's based on a specification that the fine folks at Technorati started, with many others joining them in the definition process, but no analytics vendors as far as I can tell.

In short, the idea behind attentiontrust is that my analytics data (called attention data) is mine. Here are the "rights" as defined by attention.org:

Property: You own your attention and can store it wherever you wish. You have CONTROL.
Mobility:You can securely move your attention wherever you want whenever you want to. You have the ability to TRANSFER your attention.
Economy: You can pay attention to whomever you wish and receive value in return. Your attention has WORTH.
Transparency: You can see exactly how your attention is being used. You can DECIDE who you trust.

Expect to see some interesting work around this area. A local Portland company, Attensa, has released a slick RSS reader for Outlook, and is working on other tools to help folks manage their feeds and leverage the attention spec.

At Web2.0 this week, the attentiontrust folks released a Firefox recorder extension that sends your attention data to the attention service you choose. It seems clear that this is an early attempt to get this tool out to kick the tires a bit. The tool sends your "clickstream" data to a "trusted" service. The only service available currently is with acmeattentionservice (who I believe should add a bit more to their website if they want to be perceived as trusted!).

I'll talk more about the underlying technology in a subsequent post. If you'd like to understand more about what the founders of attentiontrust are thinking, check out this very long but quite informative article.

Note too that Paul Miller and Michael Arrington over at Tech Crunch have written about attentiontrust's efforts this week.

Filed in: attention

Amazing photography and case study

I'll run a few articles on a new case study that has quite literally dropped into our laps. We're going to keep tabs on an analytics event as it unfolds, triggered through a slashdot effect (although this one was kicked off by boingboing). The impact is quite fantastic.

But first, let's talk about art. Martin Waugh is a colleague of mine (and a heck of a guy to boot) who has managed to develop several terrific photography jigs to take some truly wonderful photographs of drops. His work has recently been featured on boingboing, fark, yahoo picks (they did a very nice write-up), the nytimes, and of course, by our local WWeek.

Enjoy the great photography. I'll talk more about what's been happening to his traffic in the past few weeks since the boingboing bounce (a newly coined phrase!).

Here are a few samples:



Check them all out on Martin's site (http://www.liquidsculpture.com).

Email/blog etiquette

off topic...
Tim O'Reilly noted that he had received an email with the following note as part of the signature:

this email is: [ ] bloggable [x] ask first [ ] private

Apparently this signature has been described previously by Ross Mayfield. I've said similar things in previous emails, but hadn't thought of how to keep it simple like this. Good call.

Scaling a startup

Richard McManus has noted that Bloglines is losing ground in the RSS world, noting that they are "already no longer the market darling amongst bloggers". I like the simplicity of Bloglines and use it almost daily to keep tabs (literally!) on the world.

Richard's complains that Bloglines isn't adding new features fast enough. I agree that vendors need to keep their first-to-market edge. Bloglines has a large user-base and many of us a very devoted fans.

Mark Fletcher started Bloglines after already developing another excellent program, ONElist, which Yahoo bought (and is now well known to many people: Yahoo Groups). Mark responded directly to a post from Russell Beattie saying that Bloglines has been "...working hard on the back-end of the system." Russell was also noting that Bloglines has been "...really suffering since it got bought by Ask Jeeves".

The issue at this moment for Bloglines seems to be one of scaling. This is what I'd like to talk about now.

Developing a web application is relatively easy these days. Especially with the fantastic tools available and access to data ala web 2.0-style apps. Making the application scale requires a bit more work. Really making it scale requires a lot more work.

I don't know Mark, but I'm sure he knew this when he started. He had previously built, released and subsequently sold an amazing tool. He must have gone through the startup scaling lifecycle (I'll talk more about this in a subsequent post). Given what he knew, and given that he must have had access to at least a little bit of money to help get Bloglines off the ground, it's interesting to note that he still built Bloglines on the cheap.

In March of this year, Mark presented to eTech. I didn't attend the presentation, but I read through his materials and really enjoyed his approach. In his presentation materials he talked about his "garage philosophy" of startups. He talked about using "cheap technologies" and one note that I really liked was to "release early/release often". I believe a lot of folks have this same approach, and it serves them very well at the start.

What happens next though? Can a startup application scale quickly for rapid adoption as Bloglines has, then successfully transition to a truly scalable and stable architecture?

Again, I don't know Mark. But I do like that he's sold Bloglines to Ask Jeeves and he's still actively involved with the business - especially now that they have to do the less-than-glamorous job of trying to make the tool scalable and stable. I hope he/they succeed!

Dig it!

Dig is an old standby DNS utility. As internet technologies have matured, the use for dig has gone down, but it's still very useful for finding out who owns which domains or IP's and where they point on the internet.

I always forget what dig means, so of course, I consulted the DNS wiki, reminding me that dig is the "domain information groper". True name...I couldn't have made up anything better than that.

My favorite dig site is menandmice. Very simple, straight-forward and a good one to bookmark. Use it to confirm DNS configurations or find out what's going on behind the scenes on a site. To see an example of a fairly confusing DNS configuration, look up yahoo: http://www.menandmice.com/cgi-bin/DoDig?host=&domain=www.yahoo.com&type=A&recur=on. You'll see that they CNAME (or alias) their www site to Akamai, which has DNS servers all over the world (providing speedy, redundant and DOS-proof (mostly) DNS).

Why is this interesting? DNS is a core foundational component of the internet. All web requests must first be resolved by DNS. In order to make web analytics data collection fast and reliable, everyone must have a reliable DNS infrastructure and really understand how it works. Dig is a great tool for taking a closer look.