Every public facing design you push live – which has the potential to affect core metrics – should be A/B tested. Period.
A couple of reasons:
- Data helps you make informed decisions.
- You test to learn. You’ll find out what works, and what doesn’t. You can then share what you’ve learned with others, and apply what you’ve learned to future hypotheses.
- As much as you’d like to think that you can predict success, humans are terrible at it. There are no exceptions to this statement. That’s not to say that you don’t have a wealth of knowledge and experience that you can leverage. You’re always going to be fairly confident that each test you run will lead to an increase in your core metrics (else why would you run the test in the first place). But just understand up front that half of the designs you release are going to be a bust. That’s just the nature of the game, and if you’re not testing, you won’t know which half.
- If you launch 6 new features in a month, and as a result, a month later you start to see a slump in your core metrics, which of the 6 features do you attribute the slump to? Or is it something else completely? If you don’t test everything, you’ll be in the dark.
While testing may seem burdensome (and perhaps pointless), I promise that as you start doing it, it will get easier. Eventually, you’ll be able to launch tests in a matter of minutes, and you’ll begin to see with absolute certainty what is working, and what isn’t. It actually becomes very addicting, and can be very fun.
Again, by all means, please leverage your instinct and your experience as a designer to come up with the best design possible. But then please test each design, as a safety mechanism to make sure your assumptions were correct.
If the test succeeds, celebrate! Great job.
If the test fails, celebrate! You just learned something new about what doesn’t work, and you avoided launching it to your users! Now share what you learned and use your new found knowledge to keep iterating.
The important bits
Eliminate ego – In order for any of this to work, you’ve got to drop your ego. You’re going to instinctively want to mask every test you run as winning in some way. Don’t do it. Just know that 50% of your designs are going to be flops, and less than 5% of your designs are going to be major improvements.
Get in the habit of testing everything – When you launch designs without measuring them, you learn very little about which designs work and which don’t. That’s a shame, and a waste of your time/resources.
Focus on learning – Whether a test is a success or a failure is unimportant. The important part is that you learn something new with every design you ship, and that you keep on iterating.
I don’t do comments on this site, but if you have thoughts, I’d love to hear them. You can email me at firstname.lastname@example.org.