Good lessons in 250 A/B tests

We’ve just finished our financial year and, feck me, it’s been a busy old year. I’ll do a fuller post highlighting how the year went in more detail but the big thing we’ve seen this year is the explosive growth and (almost) 100% adoption of web site testing. From just 35% of our clients embracing testing in our last FY.

The growth has been so dramatic that we actually completed over 250 individual tests in May, June and July versus just 100 in the same period last year. So what is being tested and what’s working?

1. Page load speed is still an important factor. Landing pages, especially, benefit from slimming classes and whilst the need to have skinny 16k versions seems to be less critical than last year the target is still under 40k as you can expect to achieve twice the conversion versus a 120k page.

2. Relevancy is now a prerequisite. Dedicated landing pages for advertising terms out perform generic home pages by a factor of 2 and edge in front of deep-links by 25%. It’s worth double checking your traffic supplier or Adwords campaign to ensure you don’t have any old orphan campaigns that go to non-dedicated pages, far too many still slip through the net and this costs your conversion rate heavily.

3. Video is starting to make in-roads into certain areas and, critically, at certain times of the day and to certain audiences. In a group of personalised tests we found that certain groups of visitors (eg those that have generic hotmail, MSN or yahoo email addresses) respond far better to video-based pages than standard landing versions. Its not just video, the time to start using profiling pro-actively has arrived and start serving differing landing and content pages to distinct groups. Personalized targeting has shown a typical 10% lift in conversion over non-personalized content.

4. Numbers rock. Testing quantitative versus qualitative in copy is starting to show that using hard numbers work much better than general (soft) marketing messaging. Be mindful that the numbers you use have to be impressive in the context of the visitors needs.

The biggest lesson is the big picture here, continual testing shows continual improvement without the expected plateau that you would think occur. In fact the reverse is true, testing a little can be dangerous as you run the risk of testing items that are too small or too insignificant – think big, be brave and keep a close eye on the numbers so you can cut failed tests and rev-up the successful ones.

I expect our testing load to double in the next year. Bring it on