Jonathan Mendez's Blog

Multivariate Test Results: Confidence, Stability & Determining a Winner

Getting your MVT tests designed, developing your creative and implementing code are all just precursors to getting your test live. However, once your tests are live success will be determined by how well you did all those things. At this point you are just monitoring results - so relax and kick your feet up...if only it were that easy.

Most of the questions I’ve received through the years have been about determining when a test is over. The first thing we need to understand as testers is that when a test is live, results are always going to be changing. But testing forever defeats the purpose. Every test needs a time to cut the cord.

So, how to figure out when you’re finished? You need to weight the following factors:

Confidence: Confidence is simply calculating the discrepancy between the results. The better the performance of one test recipe vs. another the more statistical confidence will be achieved. For example if our success metric is conversion rate and one test recipe is A at 5.0% and recipe B is 5.5% there will not be a lot of confidence (that B is 10% better). However, if A was at 5.0% and B was at 10.0% there would be a great deal of confidence (that B was 100% better than A).

Important Point: The confidence metric is based on the data that has been collected. This is not a predictive calculation.

Margin of Error: MOE simply looks at the confidence stats and factors in the amount of data that has been collected. The more data the smaller the increments for MOE. I generally don’t pay much attention to MOE as these swings can be very wide. I know some stat heads might get their panties in a bunch about this but as a marketer who relies on speed this can be a paralyzing metric since so much data is needed in most cases, even with fractional factorial testing.

Stability: Stability coupled with confidence are the two most important things to look at in determining if your test is over. There are two graphs you want to be looking at one is cumulative stability and the other is daily results. Let’s see what this looks like in the Omniture Test&Target tool.

Cumulative

The main things we're looking for here is trending and consistency. One things seem to level off for a period of a week or so we're looking good.

Daily

The main things we're looking for here is outliers and fluctuation. Once we have a recipe that wins most of the days we're looking good.

Account for Temporal Changes!

Generally a best practice is to let your multivariate tests run a minimum of two weeks. This way you can get week over week results and see if there are any strange temporal behaviors that could be skewing the results. Here it is helpful to look at the daily results. I’m hoping Omniture’s Test&Target will soon be able to graph results week over week (or in other comparative timeframes) like Google can.

Don’t look back!

Successful multivariate testing is about speed (how quickly), velocity (how many) and iteration (how intelligent) based on analytic data. I’ve never regretted stopping a test with a big winner because even after is test is done you are going to be monitoring results. More often than not early results hold up as winners even if the overall improvement levels subside a little bit. For best results I’d much rather run 10 small, quick tests over a month period than 2 large ones.

This post effectively wraps up my multivariate testing overview in six parts. My final thoughts:

Multivariate testing can be a tremendous amount of fun and get you great results but it requires highly dedicated marketers and great creative methodology. Matt Roche the founder of Offermatica once shared three learnings from his time building the most successful multivariate testing tool. I'll end with his great advice for digital marketers.

1. Great marketing comes from great marketers, machines help them aim better

2. Engaged marketers lead to engaged customers

3. Speed is everything

Happy testing!

Speaking on Landing Page Optimization & Widget Ads

I have a couple of upcoming speaking engagements on landing page optimization. If you’re going to be attending shoot me an email and let’s meet-up.

MediaPost Search Insider Summit
May 18th-21st Captiva Island, Fl

SMX Advanced
June 3rd,4th Seattle, WA

Also, I’m very excited that I was just asked to speak at the WidgetWebExpo. I’ll be sharing some of RAMP Digital’s recent work on Adplications, discussing how the Intelligent Web and the use of APIs and Semantics is going to redefine display advertising and showing the first ever Google Gadget Ad that RAMP has been working with Dapper and Google on.

Setting Up a Great Multivariate Test

Part 5 in my MVT series will focus on the key considerations and decisions for setting up your multivariate test so that you get the most learning and best results possible. I’m not going to get too technical here since I am not a developer however I want to share what I feel will ensure happy testing.

JavaScript Implementation
The first and often most difficult part of setting up MVT is getting the javascript implemented on the pages to serve and change the test content and getting the proper tracking in place. I have seen it done in days I have seen it done in weeks and I have seen it done in months. You probably know your site architecture and IT department well enough to know how long this will take. Plan accordingly. The good news is that once that JavaScript is in place it can be used for future tests and content targeting. Every vendor has implementation guides and most of the implementation with JavaScript based testing tools is very similar.

No Ad-Side Variables
Ads can be quite influential on landing page performance, so they are a great (and rather easy) thing to test. In fact MVT done across ads and landing pages most often you will see that ad elements haves the higher factors of influence on conversion than page elements. Because of this, just like if we were working in a science lab we want to control as many variables as possible to prevent noise from getting into our test results. For a multivariate test on a landing page this means having only a single ad that sends people to the page.

Clearly Defined Channel Sources
It’s incredibly important to look at your results by segments. Channel segments like Natural Search, Paid Search (down to the keyword level), Email and others can have be very high impact and may deliver different results. Make sure that your tests are set-up to recognize the proper channel attributions (usually URL parameters) so your results can be sliced and diced and your learning can be many.

Segmentation (Profiles)
As soon as people are coming into your environment you should have your campaigns set-up so they can create non-PII (Personally Identifiable Information) rules based profiles. These profiles can create constant real-time segment creation based on behavior, source (mentioned above), temporal and environment. This profile create no only lets you look at your data in more helpful ways but it allows you to create test and target rules for these visitors either in-session or on a return visit based on what you know about their intentions and affinity.

Tagging the Funnel
Understanding how your test campaign users navigate through your experience can be very useful information for optimization and future test ideas for content and routing. You want to make sure that each step of your conversion path is being tracked. I have seen horrendous conversion funnels that have nullified some great landing page test results. Keep this in mind. Every page has a goal -- and every user has a goal on that page. The goal of the landing page is to meet or exceed the interest created in the ad and get people into the funnel. Keep this in mind as you develop testing success metrics discussed in Part 3.

Value Attribution (AOV, RPV)
Higher conversion rates with lower orders values is usually not a good thing. All this testing work is about two things. How much money did I spend and how much money did I make. Therefore it is imperative that you feed back price attribution into the test results so you can look at your results by what matters. With Omniture Test&Target this is a pretty simple process. This also allows you to do what is many times the most important testing you can be doing, price testing. If you are a publisher site you can create your own value attributions or scoring systems to help quantify testing results. This is a great idea that is unfortunately hardly ever used.

Proper test set-up is an essential part of testing. Make sure you allocate the time and manpower necessary for code implementation and QA. If your site uses redirects you want to take extra precaution to make sure that the first party cookie is passed consistently across domains throughout the user experience.

Once you are set up you can go live and start the best part of testing, monitoring the results. The next part of this series will look at results monitoring and will attempt to answer the question I get asked the most, how do I know when my test is over?

Adobe Moves Towards Open Source Flash

Adobe is removing all licensing restrictions from Flash and creating the "Open Screen Project." This is huge news and a giant step forward toward truly open source Flash. They can't be happy over in Redmond.

From Adobe's CEO:

“A consistent, more open platform for developers will drive rapid innovation, vastly improving the user experience.”

The cost barriers to build applications keep getting lower and lower. This promotes testing and optimization. Why? They more prototypes I can create, the more applications I can the launch, the faster I fail, the better my end result will be. This is the classic movie studio or record label model. All you need is one big creative hit to drive your whole business. The more creative you release, the better your chances. There is really no doubt that this must be the new creative model for agencies.

Here's an older post on Optimizing Flash Video for Engagement.

RAMP Digital’s First Steps

The first two months of RAMP have been glorious insanity. Besides continued success with Landing Page Optimization and testing I’m focused on a few core digital marketing areas where RAMP will change the world and define next-gen digital agency services. Most of these changes and leadership will revolve around optimized marketing solutions RAMP is creating on top of what is becoming known as the intelligent web. They involve APIs, mashups, personalization and the semantic web.

Web sites are fast becoming web services. In the words of Tim-Berners Lee, “the semantic web will bring change to our everyday life magnitudes greater than the document web has.” Advertising and marketing will not be spared. Semantic advertising and the use of open data and APIs to create marketing mashups will fundamentally change every part of the digital buy-side and sell-side experience by delivering a higher degree of relevance, both contextually and through user control. That’s Relevance AMPlified!

RAMP is actively working on a number of cutting-edge web projects. We believe we are the first agency using the intelligent web to create, deliver and optimize ads, landing pages and digital marketing experiences. We are using advanced marketing technology to create and optimize digital marketing solutions for some of the world’s leading businesses. We are working with technology partners to make their products better. We are using semantic web to help target and personalize experiences. We are also building our own experiences and mashups that solve problems.

We’ve named our initial products Adplications™ and MashLandings™ adding to the lexicon of digital marketing. We’re optimizing ROI performance by creating dynamic ads from the actions of users and dynamic landing pages from the actions in the ads. We’re creating useful widgets because an open and distributed web demands it. We’re building a vertical search engine because there’s nothing more important than search in all of marketing. And since testing and targeting is part of our DNA everything we create will continue to gain more intelligence and become optimized.

I founded RAMP because of two marketing tsunamis. On the buy-side media exchanges and auctions are going to crush every media-buying model in their path. This means performance results will rule. On the sell-side the death of the document (or site) web and the emergence of the data (or services) web can deliver higher intelligence but requires new kinds of creativity. Everything is being disrupted, especially agencies. Five years from now, today’s digital marketing landscape will look like the dark ages. RAMP isn’t anticipating the tsunami, RAMP is riding the wave. And we’re stoked!

I invite you to check out our newest digital experience, rampdigital.com