Statistics, rogues and support: A lesson in reporting

WordPress Support.

We pride ourselves on the quality and consistency of support we offer our WordPress clients. But sometimes it can go terribly wrong (on the face of it).

We’ve run a dedicated WordPress support service for over 3 years now – we’ve learned a lot and during that time on of the most useful tools we’ve had is the independent Zendesk response measurement metric. It forms the basis of how we measure ourselves. And mostly it works well – but sometimes it’s just horrendous.

Lies, damned lies and statistics

Most mathematicians (or more specifically, statisticians) will tell you that Average is a terrible measurement of many things. But it’s widely understood and for simplistic reasons it’s how we measure our response to WordPress support tickets. This is represented as a number, measured in hours, from a ticket being raised to the time we start on the fix. On the face of it, it’s a great measurement tool. Or is it.

Take the example of this month’s support tickets:

Support from March 1st to March 14th

The headline figure shows an average response time of 35.9 hours – yikes, that’s terrible! Way worse than our SLA and implies there is something deeply wrong with support. Except, you can see a huge spike on one day, March the 2nd, where it shows a glacial first response time of over 600hrs. WTF.

This is, as it turns out, one single rogue ticket raised back in February that was added from a phone call and the first response was done over the phone so not logged onto the ticket. When we came to close the ticket off nearly a month later, Zendesk then recorded the closing of the ticket as the first response hence the enormous time recorded. But, surely just one ticket out of 250 so far this month surely doesn’t make a huge difference. Except it does – if the report is run again, starting on March 3rd then the performance of support looks far more normal:

Support in march after the rogue

You can see now that the support response time has dropped to (a rather impressive) 3.9hrs and the slowest ticket to be picked up was just over 9 hours. We’ve had rogue tickets before so we’re not alarmed by massive spikes and we’re still maintaining a 100% client satisfaction score so we must be doing something right.

The moral: Don’t rely on basic numbers, make sure you understand the real meaning behind them.