Search and Best Results Analytics

Prefetching content is a clever way to speed up your browsing experience. Simply put, given certain conditions, your browser can prefetch web pages, so when you do visit those pages, they are already in your cache, thus speeding the time to display them in your browser.

Google started prefetching content for Firefox users in March of '05. Note that this doesn't work in IE (although StumbleUpon does), nor even in Chrome (although Chrome does prefetch/pre-resolve DNS information). They don't prefetch for all searches...only those where there's one result that has a high likelyhood of being selected...the "best result".

Here's how Google's search prefetching works. You perform a search, say for "apple". You get back something like these results.

Behind the scenes, google has also added an extra tag to one of the links instructing the browser to prefetch the Apple site (again, only Firefox does it) of http://www.apple.com/

For those who want more details, the tag which generates the prefetch includes: link rel="prefetch"

And to tell you maybe even a little more than you want to know (but highlights an important fact), when the browser does prefetch this information, it doesn't fetch any additional objects referenced on the page. Only the HTML content of the link. (yep...if you're using a javascript tagging solution for your analytics, you are not capturing that prefetch visit...)

Now, how does Google determine what link to prefetch? What if I really wanted info on the actual fruit? Google doesn't divulge how they actually determine when to insert a link, I think it's an interesting metric that is worth investigating. I think of it this way. Google is basically saying, "I have a pretty good idea what you're going to do next, so I'm going to make that next site you're going to visit a little faster for you".

We all pay pretty good money to GYM (Google, Yahoo and Microsoft) and others to send search traffic to us. We also spend money on tools to help us better understand what search terms folks use to get to us even when we're not paying GYM and others. And of course, we all track this important information by paying WebTrends and others for excellent tools to make sense of all of our efforts.

However, unless you've made plans to intercept this prefetch information, you are not collecting this important data. Why is it important? Only Google knows how many times your site is returned in search results. You only know:

1. How many times it was returned in paid results
2. How many times a visitor clicked through to your site in search results (paid or organic)

What's missing is how many times your site was returned in organic searching (does anyone know how to get absolute search info from GYM?). And what would also be excellent to know is how many times Google thought that the visitor was going to select your site - based on the best result. Again, here's what I'm thinking:

Google thinks a visitor is interested in your site. This tells you:

1. The visitor searched for something that likely would have led them to you.
2. Google thought it was relevant enough to even prefetch the content for the visitor.
3. The visitor may or may not actually make it to your site (why or why not?).
4. Which search terms were actually deemed relevant enough to make Google think your site was the right destination?

Ok, so how can you capture this information? If you're using web logs, you need to make sure you're capturing request headers so you can pull out a special header that is sent with each request and put it in your logs. The header is: X-Moz: prefetch

If you're using tagging, here's an approach that I put together a couple of years ago so we could watch this traffic over time: intercept the request and generate a request to your Web Analytics vendor data collection environment. I'd be happy to show you how to do this if you're interested.

Now you can track this traffic over time, and per visitor (depending on your WA solution you're using). What are these "best results" that these visitors are searching for? Did they actually select your site? Assuming you want them to visit your site, how might you change things so they do reach you? If they do make it to your site, which paths did they follow? Which scenarios did they fall into?

Questions? Thoughts? Comments?