The Rules of A/B Testing by Tyler Durden

1st Rule: You don’t talk about A/B Testing.

Don’t tell your visitors they are part of an A/B test. It would contaminate the test and would make them feel awkward. Try to hide the test as much as you can. If you have redirects or URL parameters showing up, make them not too obvious to the visitors.

2nd Rule: You DO NOT talk about A/B Testing.

Stop talking and start testing. Way too much time is spent discussing what to test and how to test and not enough time is actually spent testing stuff. Start small and iterate. Every minute without testing is a lost opportunity to learn something.

3rd Rule: If someone says “stop” or goes limp, taps out the test is over.

If you are testing on a large environment and someone reports an issue somehow related to your test, you stop it immediately. You fix the problem and then you start a new test. Tests need to be seamless, they should not disrupt the daily business.

4th Rule: Only two variations to a test.

If you’re not a master at testing, stick to a simple A/B test. Multi-variant tests are very difficult to master and require much longer to be statistically significant.

5th Rule: One test at a time.

Make sure you are running isolated tests. Keep them simple and iterate fast.

6th Rule: No header, no footer.

A/B testing works best if you limit your scope to simple elements of design or content. A/B test your copy, your images, your buttons. Only when you are a master try with complete user flows and bigger elements.

7th Rule: Tests will go on as long as they have to.

Significance, significance, significance. Keep your tests running as long as you have a statistical significance. Don’t cut corners and don’t cut tests short. It’s a good rule to keep tests running for a whole week, at least, in order to avoid short term seasonality in users’ behaviors.

8th Rule: If this is your first approach at A/B Testing, you HAVE to test stuff.

The only way to get good tests is to test often and fail quickly. Don’t spend too much time refining things at the very beginning, work on low hanging fruits and then work on the details, but definitely go and test NOW!

More about Tyler Durden





12 responses to “The Rules of A/B Testing by Tyler Durden”

  1. […] Sartoni writes The Rules of A/B Testing by Tyler Durden. “1st Rule: You don’t talk about A/B […]

  2. Chris Lauzon Avatar

    Reblogged this on Portal of Delusion and commented:

    Great article by Luca!

  3. […] That’s the beauty of A/B testing. A feature request in our bug tracker is a stone being added to a wall with mortar: it becomes a seemingly permanent piece of your architecture, of your user interface and of your responsibility. But an A/B test, why, that’s just a test! We have made no commitments to keeping it around. We’re not even sure we like it! Implementation only took a few hours so we could rip it out at any time for any reason and respond, “It was just a test.” Simply doing it, rather than talking about it, is rule #2 of A/B testing fight club. […]

  4. rajatgarg79 Avatar

    Good post as A/B testing is still not properly understood!
    Few points to consider –
    1) Try small tests in start – change in color of button, font size, less content vs more content, call to action vs two different page designs
    2) Have a clear goal – lots of companies don’t have a clear goal like people who reach URL Y or, % of people who added product to cart when I increase size of button

  5. […] The Rules of A/B Testing by Tyler Durden – […]

  6. […] few months back I came up with the eight rules of A/B testing, a direct adaptation of the eight rules of Fight […]

  7. […] August 2014 I published the 8 rules of A/B testing. It was a fun take on a serious business. Over time I adapted a few of them, but they still […]

  8. judi rolet online uang asli Avatar

    I enjoy that which you guys are usually way up too.
    This sort involving clever work and insurance! Keep up the fantastic works guys I’ve extta you guys to my personal blogroll.

  9. situs judi tembak ikan uang asli Avatar

    You truly would more than visitors’ anticipations.
    Thank you foor making these helpful, trusted, edifying and also
    cool views oon the topic in order to Kate.

Leave a Reply

Your email address will not be published. Required fields are marked *