Do not turn a blind eye to website optimization (part two)
Oct13
In the posts about website optimizing, Thomas Veltman, explains how A/B testing can deliver added value to website optimizing. This post is his second.
In part one of this blog we discussed the problem many companies face with their web presence: How do you increase the effectiveness of your website as much as possible? In this second part of “Do not turn a blind eye to website optimization” we describe one technique for improving conversion rates of your website.
The technique is called A/B or multivariate testing.
A/B Testing tests the effectiveness of many different versions of your website against each other.
How does is it work? The main principle is simple. Instead of keeping just one version of your website online, you keep two or even more versions online. One version is the main version; most of the websites’ visitors are guided to this site. A smaller percentage of visitors is guided to another version of the website.
The conversion and other performance indicators are measured for both versions of the website and compared. After that, the version with the best results becomes the main website.
And then the process starts over: another change is implemented in another new version of the website and that version is put alongside the main version the compare.
Dependent on the number of visitors to a website, it’s also possible to implement several changes in several other versions of the website. That way, more versions of the website can be compared. Since we are working with statistical figures here, the group of visitors has to be of a considerable size to have a dependable measurement. This limits the number of website version that one can have online concurrently
This approach is effective for two main reasons:
- * Other (usability) test methods always measure some kind of simulation, weather in a laboratory or elsewhere. This measurement is taken on the real visitors, the real costumers, while they are using the product in real life. Very dependable, and without having to worry about factors that might alter the results of a test.
- * If you make a change in your site and you have only one version, and it has a dramatic effect on the performance of the site, the conversion of the site and with that the turnover of the company can decrease dramatically (in the video which was linked in a response in my last blog a drop of 90% conversion rate was reported because of the introduction of a coupon code). By testing the change in this way, the performance will drop only for a smaller number of visitors and the risk is decreased.
There are several tools to implement such a test. Even free ones, such as Google website optimizer.
In the first post I talked about the challenge of website optimization being similar to that of a blind man climbing a mountain. The A/B testing method helps companies with their website optimization in the same way a mountain climber can be helped by being tied with a rope to someone else. If one falls into a gorge, the other person is there to haul them out of it. It helps companies gradually and structurally improve their websites!
October 13th, 2009 at 13:23
Hi Thomas,
Nice and interesting article. Maybe you can tell something more about setting up an optimization process in your next blog.
Regards,
Richard
October 13th, 2009 at 19:32
I wonder if this could/should be generalized using orthogonal arrays, especially for web platforms that have a significant amount of user traffic already.
October 13th, 2009 at 23:27
@everybody: richard has been a big inspiration for this article. He wrote his thesis about this subject, and i supervised this.
Richard, i’m working on this. If you have any suggestions feel free to suggest them.
October 14th, 2009 at 13:00
Ewald and Andreas,
If it is not obvious already, I’m a big fan of your blog. I like the way you guys think. These comments are primarily for your readers.
I couldn’t agree more about the benefits available by creating simple experiments to test our hypotheses (whether we’re talking, as here, about website optimization) or whether we want to test two methods of software testing to find out which one is more effective in a given project. By observing actual data, we can really learn and improve.
Danil brings up a good point when he mentions orthogonal arrays. Oversimplifying things a bit, an “A/B” test might change only one thing on a page (e.g., “coupon code included” vs. “coupon code excluded) – and could well be an OFAT test (AKA a One Factor At a Time experiment) in which only one factor is changed between test runs. These are fine to do and, as compared to doing nothing, represent a huge improvement to standard practices.
Orthogonal Arrays are one method of creating MFAT (Many Factors at a Time) experiments in which multiple factors are intelligently changed at a time between each test run. MFAT approaches are often extremely efficient and effective. See how youtube increased their sign in rates by 16% in a 1,024 recipe experiment, for example: http://youtube-global.blogspot.com/2009/08/look-inside-1024-recipe-multivariate.html
Changing multiple factors at a time with each test run used to make people in manufacturing nervous 40 years ago (“if we change so much stuff, how will be able to figure what changes impacted what outcomes? Wouldn’t it be safer and more straightforward to stick with OFAT testing methods? etc.) In manufacturing today, MFAT testing (often relying on OA’s) is widely seen as a dramatically more efficient and effective way to go about designing experiments. There’s no longer any debate when building prototypes for example (at least between people who know what they are talking about).
Switching topics slightly away from web site optimization techniques to the familiar topic of trying to find as many defects as possible in as few tests as possible, MFAT approaches are equally applicable. The adoption of MFAT methods in the software testing industry trails the adoption of MFAT methods in manufacturing by a couple decades. It is only a matter of time before increasing number of influential software testers (like you guys) will help get the word out. Here’s a link to a paper I wrote highlighting pretty dramatic findings. When we gave 1/2 a team access to a MFAT/pairwise/combinatorial test design tool, not only did that half of the team create tests much more quickly (30-40% faster), they also found 2.4X as many defects per hour as compared to the other half of the team. This wasn’t just a one-off fluke event, either. The article summarizes 10 sample projects. I’ve been involved with dozens of such proof of concept pilots to measure the efficiency and effectiveness gains available through well-structured MFAT test design approaches a with similar results. http://csrc.nist.gov/groups/SNS/acts/documents/kuhn-kacker-lei-hunter09.pdf
- Justin
Blog: http://hexawise.wordpress.com
Company: http://hexawise.com
New Testing Forum: http://testing.stackexchange.com
October 14th, 2009 at 14:20
@Justin,
Thanks for the kind words. I will look into your article. Currently I’m busy with how MFAT and OFAT can be used for me. My finding agree with yours, only no experience yet. I’ll come back to it.
- Ewald
October 14th, 2009 at 19:40
Ewald,
Sure. You bet. Keep up the good work. I would recommend other good articles on MFAT for software testing (AKA combinatorial testing) at: http://www.combinatorialtesting.com/clear-introductions-1
And, to see some of the reasons not to get too excited about the method as a “solution for everything,” see James Bach’s good article “Pairwise Testing: a Best Practice that Isn’t” – http://www.testingeducation.org/wtst5/PairwisePNSQC2004.pdf
There aren’t enough good articles published out there about lessons learned from using these approaches in practice (e.g., what works and what doesn’t, etc.) but the above are pretty good. I hope to start writing a few “how to” articles / blog posts when I make the time.
- Justin