DTC+: A/B Tests Are Useless

We stopped running A/B tests for all our clients.

For a few weeks now, we've made it part of our service to ONLY run multi-variation tests.

What’s better than an A/B test?

A/B/C/D/E test.

It does require more design and development resources, but the velocity of insights and results we get has 3-4x’d.

It’s our way of challenging ourselves to get more reps in, get better at our craft and be more valuable as a partner.

You can also be smart about it, the variations don’t need to be complete overhauls.

For example, let’s say we want to test a product page redesign.

The bulk of the design work can go into redesigning the whole page (I suggest focusing only on the first 2 folds, that’s the needle mover) and then for the variations you can take some components and move them around, or test different copy/visuals.

Positioning, copy and visuals are the biggest levers you can pull to increase conversions, so including variations of them in a split test will give you great insights.

This way, you do a little bit more work, but get 4x the insights.

That bump in insights velocity really compounds over time, and don’t forget, CRO is not only about running tests.

Those insights can be communicated to other parts/teams of the business to get some wins there as well - ads, email, content etc.

It’s like betting on four horses, instead of one, way more fun.

Pro tip: Have a process in place that monitors how the test is going, in case there are some big losing variations and you can just pull the plug on those and let the others run wild. We check them three times a week - Monday, Wednesday and Friday.

You do need higher traffic/revenue volume to implement multi-variation testing, but we’ve seen good results with brands as small as $750K annual revenue.

Two tips in that case:

• Run the test longer, always the minimum of 2 weeks, but for this size potentially around the one month mark

• Make sure no other active tests are conflicting/impacting the same area of the website, to keep the results more isolated/accurate.

So, next time you want to test something, put in 1.5x the work and get back 4x the results.

Julian

PS - When you start doing multi-variation testing, instead of just A/B

Needle Moving CRO here.

Share this newsletter with someone who will find it valuable/interesting.
This newsletter was sent to you? Sign up here.