Testing 2..3..5

The world is full of people with opinions; many of them are wrong. This loses organisations a great of money. Testing is about revealing hard data, facts.

There are in fact two things, science and opinion; the former begets knowledge, the latter ignorance. - Hippocrates.

“One accurate measurement is worth more than a thousand expert opinions” - Admiral Grace Hopper

The internet provides a unique environment to quickly evaluate ideas in a controlled environment. These are commonly called randomized experiments, A/B tests, split tests, Control/Treatment, Banker/Pretender and parallel flights. In real-time users are randomly assigned to one of two (or more) variants including the Control (or banker), which is usually the current live version and the Treatment (or Pretender) which is usually a new version that is being tested.

Data is collected as to the absolute efficacy of each version. Statistical tests are then conducted on this data to evaluate whether there is a statistically significant difference between the two variants. This dictates whether to retain or reject the (null) hypothesis that there is no difference between the versions.

At it’s heart is the realisation that in a random world even identical versions of tests will produce differing results so using the null hypothesis allows us to predict with a certain probability that the result is caused by the difference and not by the random spread of data itself.

We specialise in is the creation of the test harness, the KPI the test is trying to measure and the analysis of the data that comes out of the test. We’ve been running A/B test in the web environment since 2003 with staggeringly successful results and with over 50,000 live test executed we’ve also banked away a great deal of (provable) experience that we can bring to bear when needed.