How does A/B testing work?

The idea is to change some part of your website and see if the new version does better than the old.  A good place to start is just changing words – maybe a headline. You call the existing version “A” and the new version “B“.

You already know that the page is converting at 2% so you could just try the new headline for a few days and check the new conversion rate.

Trouble with this method is that conversion rates change from day to day. You can guess at some of the reasons: weather, day-of-week, competitor activity.  So you may be trying a bad headline on a good day or a great headline on a lousy day – and make the wrong decision about the headline.

Focus on what has changed

We can remove these other factors from the test. We show A and B at the same time. Software randomly decides if a new visitor to the site is going to see A or see B, then tracks visitors through to conversion and measures if A or B is doing better.

• If it’s raining – people will see both A and B.
• If it’s sunny – people will see both A and B.
• If a rival starts a price war – people will see both A and B.

You have removed the other factors.

Is it significant?

We use a statistical test to check that results are significant – and not just random effects. Sometimes improvements are small, but still worth having. Several 2-3% increases start to add up.

Is it just Connected doing split testing?

Amazon, Zappos, Google, eBay, EasyJet – all the big names use A/B split testing. Join the big boys – with our help.

The art of improvement is simple; make a change and then test it. If it’s better then use the latest version as the best and then cycle round again. The compounding effect of making lots of small improvements gives a number of key benefits:

• Continual improvement drives the performance up, relentlessly
• A frequent change culture encourages the testing of lots of ideas, sometimes the daftest or smallest of ideas have a major impact on performance
• Reduction of risk, a poorer performing idea is quickly and efficiently scrapped
• Much wider pool of thinking, more ideas from more people adheres closely to the concept of “the wisdom of the masses”
• Avoids the “Highest paid persons opinion” (HiPPO) quandary where decisions are made at the wrong level by the wrong people
• Compounding small improvement generates a huge improvement over time
• Real learning is possible and this helps the feedback loop for the next test
• No idea is right or wrong – it simply generates learning

The downside of kaizen in the digital world is that most web-site environments are simply not built to allow the kind of rapid change, test function that is required to get the most from this approach. VITES™ at it’s core is built around this philosophy.

Trial and error culture

Connected adopt a Kaizen approach to ongoing development and conversion rates where staying agile and making lots and lots of small changes – each time testing them to measure success – and then moving on again. Connected offer a number of services to help make this happen:-

• Fixed Kaizen contracts. A fixed number of tests or hours in a given period with a planned target to reach.
• A/B testing suites. Utilising the VITES™ platform to provide accurate, reliable A/B testing.
• Profile/persona-based testing. VITES™ supports unlimited profiles and profiles types, each can be tested independently or raced against a “banker”.
• Micro-site build & development. Avoiding the need to completely re-develop a client web-site and provide much of the improvement via a micro or landing-page site.
• Business Intelligence reporting. VITES™ Passport ™ provides a huge mine of data that can be accessed to provide unique learning. Provided as a service or as an API.
• Landing-page heavy lifting projects. The key first-wins for most commercial traffic sites, provided as a service, turnkey or consultancy.
• Performance contracts. Fixed target-based contract rewarded around specific goals and milestones.

Connected provide these services either standalone or linked with existing build-and-manage contracts.

What results can you expect?

Depending on the start point most clients can expect to achieve a doubling of conversion rates during the first year. The law of diminishing returns can apply and often the major advances are made in the early part of the project. Pushing down the “long tail” still has great value and the ongoing refining of the solution frequently uncovers surprises and big jumps on online conversion rates.

Keen to find out more?

Thinking of entering this world? Talk to Liam, Nick or Martin on 0845 051 4228 and he’ll sit down with you and go through the various options available and how to best approach improving your digital offering.