Jonathan Mendez's Blog

Multivariate Testing Overview

Part 1: Introduction to Factorial Designs
I'm not sure how many parts this overview will be but since it seems MVT has finally "arrived" I'm starting at the beginning and hope to pound out a complete overview in the next month or two of every important factor to understanding and being successful with multivariate testing.

Multivariate testing (also called MVT, multivariable testing or experimental design) is one of the three common techniques for landing page and onsite optimization (A/B testing and onsite targeting being the other two) though it can be used effectively for ad testing as well. While multivariate testing does not usually provide the performance lifts of A/B testing or the ability of onsite targeting to deliver relevance, when used properly it is an essential part of ongoing and iterative online marketing testing and optimization.

MVT provides incredible intelligence and learning. The greatest benefit to MVT is what is referred to as element contribution. Element contribution informs the marketer of the percentage contribution each tested element has on conversion rate. This learning is highly valuable as it provides understanding of the triggers that influence behavior and can lead to numerous iterative test ideas that can provide further learning and improvements in conversion.

There are two main design of experiments that marketers use and multivariate testing vendors base their technology around. These are fractional factorial design and full factorial design. Let’s examine the pros and cons of each.

Fractional Factorial Testing
Fractional factorial testing (also commonly referred to as Taguchi Methodology) requires the marketer to choose two or more elements to test using one or more variations of the selected elements. This method is called fractional because not every page element is tested. The marketer selects what they believe are the most important elements based on what they are testing.

The combination of elements and alternatives is called an array and is used to create the pages used for testing. Certain arrays mitigate the effects of aliased interactions better than others. In my experience the L8 and L9 arrays are very good at this. L8 arrays are used for 7x2 (7 elements and two variations) and L9 for 4x3 (4 elements and 3 variations) MVT.

Example of a 7x2 L8 Array

Pros
The benefit of fractional factorial testing is that less data is required in order to reach statistical confidence in the results. This means that with a solid test design results can usually be achieved in shorter time periods. Usually weeks. This is highly beneficial for marketers that like to “fail faster” and take more iterative approaches to testing.

Cons
Since not every element is being tested certain interactions are aliased. This means that while the results may achieve statistical confidence full interactions between all elements are not measured. If the marketer selected the most important elements these interaction effects can be mitigated. Of course it's not always easy to determine the most important test elements.

Full Factorial Testing
Full factorial testing uses every element of the page to create a test array measuring all interactions. Because so many interactions are measured this method requires a great amount of data (thus time) to achieve results.

Pros
As mentioned each interaction is accounted for so when confidence in the results are achieved there are no questions about possible interactions skewing the results.

Cons
Since every element must be considered and tested a large amount of data is necessary to achieve results. Incorporating variations of each element can also be a taxing creative effort. Because of these reasons full factorial tests need a much longer period of time to gain results, usually months. SInce online behavior can change (for many reasons) over such long periods results may tend to benefit certain temporal swings.

Summary:
As a marketer I have always used fractional methodology and with tremendous success. I like the idea of velocity in marketing -- test, learn, test, learn, test. Instead of one large test I prefer focusing attention on certain areas or elements to achieve deeper understanding. I also believe this iterative approach lends itself the dynamic nature of online marketing.

Part 2 will focus on designing MVT test arrays based on traffic analysis, page design and conversions.

iTuning Madison Avenue

Facebook's hiring of Sheryl Sandberg who helped build Adwords is another big step forward on the path of media buying automation and optimization. In two years everyone will be within reach for every brand and business large or small. It will all be about targeting.

Google, MSFT, FIM and Facebook, who between them touch just about every online user in the USA (and moving toward that globally) will take care of the targeting. Their core focus now is getting highly targeted quality impressions in every channel -- search, contextual, display, TV, radio, games -- because higher quality means higher spend.

Of course this rapid transformation of the current media structure into one of optimization and automation has tremendous implications. Technology and market forces have created incredible efficiencies for product (ad) distribution. In fact, these changes are starting to resemble that of another big industry. The music business.

Is it possible that Google et al are iTuning Madison Ave? Are the Big Dumb Agencies the new record labels and as such will meet they meet the same fate? The parallels present in consumption patterns give this theory even more credence.

No one’s buying albums anymore!

Services have fractured and are more distributed than ever. Display, SEO, SEM, Reputation Management, Creative, Testing, Targeting, Analytics. The BDA’s are trying to fix this by buying their way into being one-stop shops. The thought being that clients want the whole album. But as with most albums there’s always a lot of filler that you are paying for. Is that what clients really need? Is that what they really want?

If efficiencies and fragmentations are happening where are the opportunities to improve your ad services business? Again, I say look at the new paradigms in the music business. I see the same three core strategies to making money whether you are a record label or an advertising agency.

1) Merchandising: Sell, upsell, re-market those with an expressed interest and affinity.

In music that means t-shirts and fan clubs. In digital marketing it is called search, site optimization and email.

2) Distribution: Generate revenue from new distribution models.

In music it's ringtones and soundtracks. In digital marketing it is called affiliate marketing and social media.

3) Experience: Add to the bottom line by creating new and improved experiences.

In music it's VIP packages. In digital marketing the improved experiences will take root in new platforms & applications

What advertisers need to understand is that not only did the record labels get killed because they didn’t handle the new digital models with foresight, their clients, the artists also got the shaft. This lead to the artists taking more control. You can already get a whiff of that with the rise of in-house search.

How far and how fast? Seven years after its release Apple’s iTunes accounts for 70% of worldwide digital music sales and record labels are mostly irrelevant. At this late stage it looks the only thing that can save Madison Avenue is its own version of American Idol. Simon?

The $130 Billion Dollar Landing Page

Google is a fully digital company. Everything Google has done and is today revolves around the web. Google is also by every account one of the top 3 brands in the world. Until billboards this past year Google has never spent a dollar on advertising. No TV, no print, no radio, No banners, no search ads (that it is pays for ;) - Nothing.

So how did Google build their brand?

It stared with a landing page. The Google homepage is the greatest landing page ever created.

Of course there’s more to it. Google, and Marissa Mayer in particular have been insanely focused on user experience forever. It’s no accident all the links are blue underlined text links or that testing takes place regularly and is a core Google belief.

The other part of the user experience of course is their ability use technology to deliver of relevance.

Still, it started with a landing page. A very simple landing page.

So if you can build a brand -- maybe the world’s leading brand, through digital experience alone what does this mean for brand marketing once everything is digital? What does it mean when a killer landing page may hold the key to your digital success?

What it means is that the experiences people have with your digital footprint will define your brand. They will build it or the can take it down.

Crappy website? Crappy brand!
Irrelevant advertising? Irrelevant brand!
Bad landing page? End of brand conversation!

Of course the offline experience will flow back online as well. Rats in the kitchen? Fell asleep on the couch? Battery issues? These all become part of your brand’s digital footprint - forever!

Success for almost every brand and business is now defined by bringing people to your digital presence and optimizing their experience once there. Time and money spent now on anything else but optimizing your digital footprint is short sighted and likely will be wasted. At this stage of the digital game when the tools and technologies are accessible to everyone there are simply no excuses for a failure to optimize.

The $130 Billion Dollars? That’s Google’s market cap today. Not a bad ROI on testing and optimization.

Chris Anderson on the Two Dimensions of Quality

There have been a few occasions where clients have taken my page layout and messaging work in powerpoint, skipped the design stage altogether and converted these glorified wireframes to HTML. One time it was spite, another time it was money (isn’t this an Eagles song) and another time it was time. In all three events the pre-designed pages outperformed fully designed high production control pages they were being tested against.

Chris Anderson, Editor of Wired understands why. This exchange is from last Thursday’s Charlie Rose show on PBS.

Charlie Rose: What will the web do to content in terms of high cost, expensive, time consuming, content?

Chris Anderson: It turns out there’s two dimensions of what we think of as quality. So quality wins but we think of quality as being production quality. Lighting and sound and film and script and acting - those kind of things. It turns out there’s another dimension of quality that’s as important. That’s relevance.

Chris ended with a question every business needs to know the answer to..."How do you remain relevant in a world that’s fragmenting?”

Speaking at Web Analytics Wednesday - Cambridge, MA

If you’re in the Boston area please come out to this month’s Web Analytics Wednesday on March 19, 2008 at 6:00 PM. I’ll be speaking and presenting on the “Crawl, Walk, Run” approach to Onsite Targeting. The event takes place at Asgard in Cambridge and admission and food is free!

Judah Phillips hosts the event and Omniture is sponsoring it. I’ve heard nothing but great things about these events so I’m thrilled to be actively involved and look forward to “comeasureating” with the analytic luminaries of Boston and meeting some O&P readers there. Note to self: remove Giants Super Bowl Champions XLII magnet from car prior to parking.

SMX West Afterthoughts

I wasn’t sure what to expect with SMX West as this was the first large SMX event. I heard great things about at last summer’s SMX Advanced from clients and friends but there was short notice to fit this event into annual budgets of attendees and exhibitors. Now that it’s over I can say that the event was a big success all the way around and portends well for the SMX brand.

The first thing that made an impression on me was the free wifi throughout the conference area and table seating in the sessions. This allowed easy use of laptops for note taking and increased productivity during the sessions. With one exception the sessions that I sat in and spoke on were excellent. Jonah Stein who’s been to many a search conference commented that the landing page session was one of the best sessions he’s ever attended and modestly I agree.

The only negative I took away from the presentations were from the search engines themselves. The presentations (at least the half dozen I saw) from Google, Yahoo and Live were consistently weak. I eagerly anticipated The Blended Search Revolution session only to be bored to tears as the engines added little more insight than a basic product demonstration and no more substance than a cry of “use us!”

The exhibit hall had many of the usual suspects and as always a few new faces. Yahoo was MIA again as they were in SES Chicago. One thing that continues to amaze me is the ever increasing number of SEM tools for campaign and bid management. There has to be some shakeout in the next 12 months here as I don’t know how the SMB or Mid Agency market is going to afford anything like 5% of spend when GOOG and MSFT are rocking free tools.

The “most crowded booth award” went to SEOmoz. I swear there were times where the overflow in front of their booth completely blocked anyone from walking down the aisle. Now that’s a great tactic to build booth traffic -- block the aisle! Congrats to Gillian and Rand on a home run with their first exhibit.

My speaking slots were both on Day 2 and my first session was Landing Pages & Multivariate Testing. I was very impressed by the panel especially Sandra Niehaus, Vice President User Experience & Creative Director, Closed Loop Marketing whom I had not met before. She accurately summarized the UX and creative benefits of MVT and also emphasized the need for clients to quantify goals and objectives of testing prior to launching into it. All great points.

I lead off the session focusing on the basics of landing page optimization and testing. I’ve been trying to make my presentations more conversational and less all about bullet points so I’m not sure how well they’ll play on Slideshare but I’m posting them here anyway for those interested. Also, I'm aware the font looks terrible - it's really Bank Gothic, I swear.

| View | Upload your own

My second session was Search Ads & Behavioral Targeting. Once again I lead off so I provided an overview of segmentation and targeting methodology. I then showed some of the great site targeting abilities and success I’ve had with the Omniture Testing and Optimization tool. This dovetailed into some of the ad targeting work we’re doing at RAMP within Social Media using the great Lotame Crowd Control technology and with adplications developed using the incredible Dapper technology. More on that work in future posts.

| View | Upload your own

Somehow I double booked dinners on Wednesday so I brought together both parties and headed out to steaks with Ian White of Urban Mapping and his investor/local search guru Jake Baille as well as some of my good friends from the Omniture SF office. If you've never checked out Urban Mapping do yourself a favor - they have some of the most killer geo applications out there.

Thursday was spent catching up with work, visiting with some potential clients and partners around the valley and then 4 hours of work (including this post) at SJC. This was my third time in Silicon Valley in the past few months and I'm still not sure what to make of it. I guess it's like Hollywood for geeks (no, really).

Overall a fantastic start to the big SMX shows. I met a lot of great folks and as always enjoyed reconnecting and talking shop with the search pros I admire. Congrats to Danny, Chris and everyone at Third Door Media. It was an honor to be part of of the event.