Test Every Assumption

Let’s say I’m at work and we’re discussing a potential change that we’re thinking about making to our product (or even the marketing site). If the phrase “best practice” or “common knowledge” get’s tossed out, a little red flag always pops up in my head.

“common knowledge” is just an unproven assumption

It’s a mirage I tell you!

Just because a technique supposedly worked well some place else does not mean that it will work well for your app. Time and again I’ve seen instances where after testing something that felt like a “sure fire win”, we discovered that not only is it NOT a sure thing, it results in a shocking loss of signups or revenue.

But this is a no brainer! It’s obvious!

Wrong.

Test it. You’ll see. There’s generally a 50% chance that you’re wrong.

I get it… You’ve got a huge backlog of stuff that needs to get done. Taking the time to set up an A/B test for this feels like a big waste of time—time you could be spending on your big list of todo’s.

Is it much easier to side with the assumption that “loads of other apps do it this way”, push a deploy out the door, and move on to something else?

Of course.

But don’t be fooled

In what area of your life has the easy route ever been the best route? 😉 It’s no different here.

If a change that you’re making has the potential to affect any number that you care about (user happiness, signups, revenue, retention), you should find the time to test it.

There’s no safety in assumptions

In the end, it’s always best to test all “best practice” or “common knowledge”.

Comments

I don’t do comments on this site, but if you have thoughts, I’d love to hear them. You can email me at its@davemart.in.