Tomi Mester – A/B Test like a Pro! (online course)

Tomi Mester – A/B Test like a Pro! (online course)

Understand the science and the business logic behind A/B testing — and achieve real, significant, long-term increases in conversion!

• What is A/B testing exactly? (3:01)
• The correlation vs. causation issue (4:47)
• Why do you need A/B testing? (a story and the real reason) (4:23)
• What can you A/B test? (8:52)
• EXTRA: Hand-picked A/B Testing case studies
• ACTION: The A/B tests you’ve run so far — and the A/B tests you want to run in the future

• Important A/B Testing mindset: Conversion Rate Optimization or Research? (3:42)
• Two key questions before ing an A/B Test. (And when is A/B testing a really bad idea?) (6:45)
• When you don’t need to run experiments (6:32)
• When you need to run experiments (5:30)
• A/B test only when you need it (2:09)
• EXTRA: the “can I run this A/B test” checklist
• ACTION: Filter out the bad A/B test ideas from your backlog!

• Statistics is important (1:30)
• Randomization (we all are terrible “random-generators”) (3:01)
• Statistical significance and the certainty of your results (4:37)
• An intuitive way of interpreting the importance of statistical significance (2:02)
• 80%? 95%? 99%? What’s the right confidence level? (6:45)
• Calculating the required sample size before the test (4:57)
• EXTRA: online calculators (statistical significance and sample size)
• ACTION: Review your A/B testing backlog!

• Your questions answered.

• The four steps of executing an A/B test (1:39)
• The Research Framework (14:14)
• Your success metric(s) to evaluate your test (7:36)
• A few words about setting a hypothesis (5:05)
• EXTRA: A/B Testing Hypothesis Form
• ACTION: Run your first research round!

• My favorite A/B testing tools (6:06)
• Tool demo (Google Optimize) (16:01)
• Avoiding website flickering in an A/B test (4:38)
• The change-one-thing-at-a-time myth, multivariate tests, A/B/n tests and unusual audience splits (7:04)
• A few typical mistakes at the implementation phase (3:13)
• EXTRA: The “can I the A/B test?” checklist
• ACTION: Try out the simplest ever A/B test!

• How long should an A/B test run? (2:42)
• Which part of the year/week shouldn’t you run an A/B test? (1:53)
• Typical mistakes while running an A/B test (5:57)

• Don’t lie to yourself! (4:36)
• What to do when a test didn’t perform the way you expected (2:28)
• What to do after evaluating your experiment (6:29)
• ACTION: your own knowledge base!

• The limitations of A/B testing (2:26)
• The art and science of A/B testing (3:15)
• Segmentation (2:37)
• EXTRA: Further reads and videos

• Summary & The right mindset to be successful with A/B testing (2:29)
• Where to go next

• Your questions answered

• Why?
• 1) Traffic allocation: sending visitors to the old/new landing pages (with JavaScript)
• 2) Data collection and visualization (Bash + SQL + Google Data Studio)
• 3) Evaluation: significance calculations (using Python)
• Conclusion