We’ve just finished our financial year and, feck me, it’s been a busy old year. I’ll do a fuller post highlighting how the year went in more detail but the big thing we’ve seen this year is the explosive growth and (almost) 100% adoption of web site testing. From just 35% of our clients embracing testing in our last FY.

The growth has been so dramatic that we actually completed over 250 individual tests in May, June and July versus just 100 in the same period last year. So what is being tested and what’s working?

1. Page load speed is still an important factor. Landing pages, especially, benefit from slimming classes and whilst the need to have skinny 16k versions seems to be less critical than last year the target is still under 40k as you can expect to achieve twice the conversion versus a 120k page.

2. Relevancy is now a prerequisite. Dedicated landing pages for advertising terms out perform generic home pages by a factor of 2 and edge in front of deep-links by 25%. It’s worth double checking your traffic supplier or Adwords campaign to ensure you don’t have any old orphan campaigns that go to non-dedicated pages, far too many still slip through the net and this costs your conversion rate heavily.

3. Video is starting to make in-roads into certain areas and, critically, at certain times of the day and to certain audiences. In a group of personalised tests we found that certain groups of visitors (eg those that have generic hotmail, MSN or yahoo email addresses) respond far better to video-based pages than standard landing versions. Its not just video, the time to start using profiling pro-actively has arrived and start serving differing landing and content pages to distinct groups. Personalized targeting has shown a typical 10% lift in conversion over non-personalized content.

4. Numbers rock. Testing quantitative versus qualitative in copy is starting to show that using hard numbers work much better than general (soft) marketing messaging. Be mindful that the numbers you use have to be impressive in the context of the visitors needs.

The biggest lesson is the big picture here, continual testing shows continual improvement without the expected plateau that you would think occur. In fact the reverse is true, testing a little can be dangerous as you run the risk of testing items that are too small or too insignificant – think big, be brave and keep a close eye on the numbers so you can cut failed tests and rev-up the successful ones.

I expect our testing load to double in the next year. Bring it on

Thankfully, this is not the case. Much like the focus on “bounce rate” it all ended up being a huge Red Herring. It took a number of scientifically accurate tests to prove that the fold is not quite as critical as you might imagine. Don’t get me wrong, it can make a difference but, for gods sake, test it first as we found out this year with two big tests concerning what is and what isn’t “above the fold”. In both cases it made no difference so we can welcome back “long copy” that until recently has been consigned to the bin.

We all know what really matters is the customer journey and overall conversion and not whether all the content and actions are above the fold on the home page. Or, as an aside, does the bounce rate have special meaning.

How does A/B testing work?

The idea is to change some part of your website and see if the new version does better than the old.  A good place to start is just changing words – maybe a headline. You call the existing version “A” and the new version “B“.

You already know that the page is converting at 2% so you could just try the new headline for a few days and check the new conversion rate.

Trouble with this method is that conversion rates change from day to day. You can guess at some of the reasons: weather, day-of-week, competitor activity.  So you may be trying a bad headline on a good day or a great headline on a lousy day – and make the wrong decision about the headline.

Focus on what has changed

We can remove these other factors from the test. We show A and B at the same time. Software randomly decides if a new visitor to the site is going to see A or see B, then tracks visitors through to conversion and measures if A or B is doing better.

• If it’s raining – people will see both A and B.
• If it’s sunny – people will see both A and B.
• If a rival starts a price war – people will see both A and B.

You have removed the other factors.

Is it significant?

We use a statistical test to check that results are significant – and not just random effects. Sometimes improvements are small, but still worth having. Several 2-3% increases start to add up.

Is it just Connected doing split testing?

Amazon, Zappos, Google, eBay, EasyJet – all the big names use A/B split testing. Join the big boys – with our help.

So in many ways it’s been a rear-guard action as we’ve had to fight against the client’s, co-supplier’s and consultant’s wishes to drive home the philosophy of continual testing and letting the masses decide which page, landing zone, micro-site, call-to-action and design is better.

I’d be lying if I said it wasn’t a hard slog and at a number of points we were on the point of giving up but after 2 months showing the results of real numbers and metrics they’re sold and quite rightly they should after seeing a dramatic upswing in conversion rate.

We’re in the final throes of putting together a case study on this client and I hope it will be ready sometime in April so if you want to reserve a PDF version of the case study now then drop me an email and I’ll make sure you get one of the first batch.

This will be the first time we’ve openly shared so much information about how we work and we’ll be showing actual test results with annotated screenshots. The case study is part of our 64Monkeys projects that was kicked-off in 2007 as an internal knowledge base holding the planning and results behind every test we’ve carried out (around 4,500 so far). We’re planning an alpha roll-out of 64Monkeys later this year to openly share and collaborate with our clients and encourage a great deal of cross-fertilisation.

These are all questions typically running around marketing departments at the moment and the stock answer is to “get a new one designed” on the basis that it must be better than the old one as it’s newer and we’ve learned things (have we?) about the current versions of the landing pages.

Trial and error economics

Don’t guess, don’t risk the return on the heavy-lifters. Your current series of pages act as a ‘banker’ – put simply the new stuff has to race the best of the best you have already. When testing the new pages against the current bankers use a reliable testing method (such as the null hypothesis) that gives results you can be confident in. If you can be confident in the results then you can queue up hundreds of ideas to be tested and leave the testing harness to do the hard work or evaluating the changes.

The risk is low because:

• Pages that perform a lot worse than the banker will show up as failing very quickly and can be removed from the test quickly.
• Pages that perform a lot better than the banker will show up as succeeding very quickly and can replace the banker quickly.
• Pages that perform similarly will take a lot longer to determine their value but as they are not hurting (or helping) the conversion rate there is no loss associated with leaving them in test except the loss of the opportunity to run another test.
• Testing small changes can help with specific learning. For example, the data-entry form might perform better with a solid blue background versus white. This is real learning and can be applied (after testing) across other landing pages.

In fact, taken as a continuous process (Kaizen) optimising the landing pages can be the easiest and fastest way to continually improve a site, although not very glamorous for a marketing department.

About VITES™ as a testing tool

Split testing using the null hypothesis is built-into the core of VITES™ and offers a fast, reliable and repeatable test harness. Testing can be done via profile, traffic type, campaign or any other superset of visitor data (postcode range, for example) and is not limited to A/B testing with support for 26 concurrent tests running in each profile.  Traffic can be split in any range, typically the fastest results are achieved using a 50:50 ratio in an A/B test but 80:20 tests are commonly used when clients are nervous about radical changes to high net-worth landing pages.

The art of improvement is simple; make a change and then test it. If it’s better then use the latest version as the best and then cycle round again. The compounding effect of making lots of small improvements gives a number of key benefits:

• Continual improvement drives the performance up, relentlessly
• A frequent change culture encourages the testing of lots of ideas, sometimes the daftest or smallest of ideas have a major impact on performance
• Reduction of risk, a poorer performing idea is quickly and efficiently scrapped
• Much wider pool of thinking, more ideas from more people adheres closely to the concept of “the wisdom of the masses”
• Avoids the “Highest paid persons opinion” (HiPPO) quandary where decisions are made at the wrong level by the wrong people
• Compounding small improvement generates a huge improvement over time
• Real learning is possible and this helps the feedback loop for the next test
• No idea is right or wrong – it simply generates learning

The downside of kaizen in the digital world is that most web-site environments are simply not built to allow the kind of rapid change, test function that is required to get the most from this approach. VITES™ at it’s core is built around this philosophy.

Trial and error culture

Connected adopt a Kaizen approach to ongoing development and conversion rates where staying agile and making lots and lots of small changes – each time testing them to measure success – and then moving on again. Connected offer a number of services to help make this happen:-

• Fixed Kaizen contracts. A fixed number of tests or hours in a given period with a planned target to reach.
• A/B testing suites. Utilising the VITES™ platform to provide accurate, reliable A/B testing.
• Profile/persona-based testing. VITES™ supports unlimited profiles and profiles types, each can be tested independently or raced against a “banker”.
• Micro-site build & development. Avoiding the need to completely re-develop a client web-site and provide much of the improvement via a micro or landing-page site.
• Business Intelligence reporting. VITES™ Passport ™ provides a huge mine of data that can be accessed to provide unique learning. Provided as a service or as an API.
• Landing-page heavy lifting projects. The key first-wins for most commercial traffic sites, provided as a service, turnkey or consultancy.
• Performance contracts. Fixed target-based contract rewarded around specific goals and milestones.

Connected provide these services either standalone or linked with existing build-and-manage contracts.

What results can you expect?

Depending on the start point most clients can expect to achieve a doubling of conversion rates during the first year. The law of diminishing returns can apply and often the major advances are made in the early part of the project. Pushing down the “long tail” still has great value and the ongoing refining of the solution frequently uncovers surprises and big jumps on online conversion rates.

Keen to find out more?

Thinking of entering this world? Talk to Liam, Nick or Martin on 0845 051 4228 and he’ll sit down with you and go through the various options available and how to best approach improving your digital offering.

This is a classic sales process for many organisations, often referred to as the sales funnel, yet this methodology on the web is not widely available. Below I will provide a brief overview of a client that uses this technology to stunning effect.

Laser eye surgery company

Founded over 10 years ago on a traditional clinical sales model (information -> consultation -> sale -> treatment) the emergence of the Internet offered this client a unique opportunity to map their current sales process onto an online model. At the start of this project in 2002 less than 20% of treated individuals ever touched their web presence. In 2009, nearly 90% of treated patients went through some element of the various web assets – they have, in effect, transformed into a dot com business in just a few years with spectacular results.

The challenges

Mapping a sales transition curve onto an online system is relatively simple to do with the right platform and this was achieved quite quickly and easily. Using the VITES™ platform a number of sales steps were created; first-timers, information gatherers, thinkers, consultation bookers, consultation attenders and treated patients. At all stages in the journey help and assistance is made available. Testimonials re-enforcing and evangelising the positives in the process are posted.

To grow the business and generate sustainable sales, every person in the process has to go through (almost) every stage and this process is not always linear, some steps are missed out and some are clumped together depending on the needs, fears, issues and financial standing of the customer.

At each stage there are various challenges faced by the company and the customer. Here is a summary of the stages and the challenges:

First timers – Stage A

Customer : Who is this company? Are they right for me? What’s involved in surgery? How much? Will it hurt?

Company : Quick engagement, make sure they stay or leave a method for re-engaging (email, address)

Information gatherers – Stage B

Customer : Where is the information I need? How much will it hurt? How can I afford it?

Company : Getting the “this is simple & painless” message across. We can make this affordable. We are the best.

Thinking about it – Stage C

Customer : Really not sure about surgery? Will I go blind if it goes wrong? It’s expensive.

Company : Gotta keep ‘em warm, if they disengage we stand a good chance of losing them. It really *is* safe.

Consultation bookers – Stage D

Customer : Can I see someone near me? Does it cost? Will it hurt? Credit or cash purchase?

Company : Consultations are free, no obligation, no sales pressure. We are the best!

Consultation attenders – Stage E

Customer : Hmm, I’m suitable now – should I go ahead? What happens if I go blind? What about post-op complications?

Company : Gotta get them to buy, this is the most expensive stage to lose ‘em, momentum is the game

Treated patients – Stage F

Customer : Wow, it’s fab – who else can I tell this about, can’t believe I didn’t do this sooner, it didn’t hurt

Company : Lets get these guys talking to the wide world and also to the people in the early stages

There are clear themes that run throughout most of the process, fear and cost for example, but at each stage there are subtly different issues, challenges and therefore different messages, different calls to action and different ways of dealing with the visitor.

The solution

Using the VITES™ Journey Platform, rules define that each visitor is tagged at each stage and “dropped” into a profile that fits the stage that they are at. In each profile there are different ways of dealing with the various challenges and these are served up as custom content based on what is known about the visitor. Additionally, there is external learning that can be transferred into the system; for example an incoming first-time visitor who has typed “laser eye treatment” into Google has a higher preponderance to move faster through the sales cycle than, say, an individual who has responded to a “win FREE laser eye surgery” banner on Hotmail.

A bespoke forum comprising FAQ’s and posts is available to the visitor. It is important that platform technology and the quick and easy availability of “help” work together to enable the journey to continue.

Below are some rule examples:

• If you are a stage A and have typed “laser eye treatment” into Google then show a “Consultation request” form
• If you are a stage A from a (filler) banner campaign based around “Win FREE” then show a “Win FREE form”
• If you are a stage A person and leave the site then flood the visitors banner network with “Win FREE” banners to get you back.
• If you have been on stage B for more than 30 days show an “Interest Free Offer”
• If you have just become a stage B then show a “Consult Booking” option
• If you are at stage C then show testimonials from people like you, treated near you and show forum posts from stage F people (how wonderful it is)
• If you are stage D then show special offers during week 6 to 9

The rules are not complex but there are a great deal of them and with around 3,500 entrance points to the site used by approximately 1,000,000 visitors per month the automated system simply chugs away delivering the right content at the right time. To further expand the model, to say “show a consult booking” option is wider than just online; it includes the use of personalised email and also communicates this information to (and from) the call-centre to ensure that the sales message is consistent across all communication platforms.

See it working?

The average time from first-arrival to sale is around 90 days and with the VITES™ platform being self learning and with many of the rules making quite subtle changes it’s actually very difficult to see the process in action – which is good. You don’t want the visitor to have a “jolting” experience every time he or she returns to the site. The very nature of VITES™ is that it is subtle.

Trial and error economics

The very nature of the platform encourages testing, seeking improvements in all the stages. A few percentage point improvements in the stages generates huge compounded gains. For example, the overall conversion rate from visitor to client has doubled over the last 3 years and the cost per sale has dropped 80% over that time period whilst the time-to-close has fallen by over a third. These are all substantial gains and one of the reasons that the work done on this client is often quoted as real “best practice” across the world and used by some of the top Internet gurus as the basis for workshops, seminars and events.

This is borne out of being able to have a complete view of what happens when we make changes to a site (thanks to our VITES™ platform); when combined with statistically correct sample sizes and A/B testing, this gives you the clearest (probable) indicator of how successful a change is. I note that a search on Google turns up some interesting results, with some companies, such as Nike, embracing the idea, and no less than Forbes Magazine citing it as the reason for the wealth of creativity that comes out of our US cousins across the pond.

Thomas Edison was once quoted as saying “I HAVE not failed. I have just found 10,000 ways that won’t work.” This approach has its problems which need to be addressed, but the greatest problem is with its name- Trial and ***ERROR*** – which states clearly that mistakes will be made. Sadly most of today’s inwardly-focussed corporates seem to lack the courage to take this step. Even worse are the companies that make sweeping changes without trialling them first; in the case of AOL a $75m loss in revenue. The companies that do test seem to fly; the ones that don’t end up stagnating and die.

The moral? To succeed you have to fail occasionally

The argument also holds true only if the creator learns from the early mistakes and this means that there has be be intermediate measurement (or review) of the progress so far and also assumes that there is either a) a target to achieve or b) a reference point (teacher?) that can judge progress.

The theory of a million monkeys with a million typewriters does hold true but that evolutionary approach seems to lack learning outside of the project being worked on. Experience can be brought to bear to speed-up the process.