A/B Testing and Experimentation in Python Course
A/B Testing and Experimentation is an intermediate-level course of 5 lessons (about 2 h 44 min), with 15 exercises and 15 quiz questions. Prerequisites: Probability and Inferential Statistics, Data Analysis with pandas and NumPy. Exercises run in the browser, with nothing to install.
- Statistics
- Intermediate
- 5 lessons
- 2 h 44 min
- 15 exercises
- Exercises run in the browser
The course in brief
This course teaches you to run an A/B experiment end to end. You will write an experiment plan, calculate sample size and duration, analyze conversion and revenue metrics with the right tests and the bootstrap, then learn to avoid the pitfalls that invalidate so many tests: allocation imbalance, peeking, and multiple comparisons.
What you will be able to do
- Write an experiment plan with a hypothesis, a primary metric, and guardrails
- Implement a stable random assignment using hashing
- Calculate a test's sample size, power, and duration
- Analyze a conversion test (z-test, confidence interval of the difference, uplift)
- Analyze a revenue metric with Welch's test and the bootstrap
- Detect an SRM, avoid peeking, and correct for multiple comparisons
- Formulate a well-argued decision from the results
Syllabus: 5 lessons
- Formulate a testable hypothesis and choose a primary metric and guardrail metrics
- Set up a random, stable, and balanced assignment of users
Experiment hypothesisPrimary metric and guardrailsRandomization
- 02
Sample Size, Minimum Detectable Effect, and Power
- Calculate the sample size needed to compare two proportions
- Explain the relationships between alpha, power, minimum detectable effect, and test duration
- 03
Analyzing a Conversion Test: The Two-Proportion Test
- Perform a z-test comparing two proportions
- Calculate the confidence interval of the difference and the relative uplift
- Formulate a decision from the results
- 04
Continuous Metrics: Welch's Test and the Bootstrap
- Analyze a continuous metric (revenue per user) with Welch's test
- Build a confidence interval using the bootstrap
- Avoid the pitfalls of ratio metrics
- 05
A/B Testing Pitfalls: SRM, Peeking, and Multiple Comparisons
- Detect an allocation imbalance (SRM) with a chi-square test
- Understand why watching results continuously inflates false positives
- Correct for multiple comparisons with Bonferroni or Holm
Final project
Complete Analysis of a Checkout Page A/B Test
An online shop tested a new checkout page. In Python (fixed seed), generate a user-level dataset: group, exposure date, conversion, revenue, device. Deliberately introduce a slight effect on conversion and a novelty effect. Write the experiment plan, size the test, then carry out the complete analysis through to a well-argued recommendation.
Deliverables
- The experiment plan (hypothesis, primary metric, guardrails, unit, alpha, power, MDE, duration)
- The sample size and duration calculation
- The SRM check
- The analysis of conversion (z-test, CI, uplift) and revenue per user (Welch and bootstrap)
- The week-by-week effect analysis and a device analysis corrected with Holm
- A one-page recommendation for the product team
Assessment criteria
- The plan is written before the analysis and followed
- The calculations are correct and reproducible
- The unit of analysis matches the unit of randomization
- The pitfalls (SRM, peeking, multiple comparisons, novelty) are explicitly addressed
- The recommendation relies on the effect and its uncertainty, not solely on the p-value
Frequently asked questions
How long should an A/B test run?
The duration follows from the required sample size, calculated from the baseline rate, the minimum detectable effect, the alpha level, and the power, divided by daily traffic. You round it up to full weeks.
Can you stop an A/B test as soon as it is significant?
No, unless you use a sequential method designed for it. Checking the p-value continuously and stopping at the first significant result greatly increases the false positive rate.
Do I need to know statistics before this course?
Yes, the basics of hypothesis testing and confidence intervals (the Probability and Inferential Statistics course) as well as pandas are recommended.
What is an SRM?
A sample ratio mismatch is a significant gap between the observed split of users and the expected one. It often reveals a bug and must be explained before interpreting the results.
This course, told for your job
Say what you want to be able to do: “A/B Testing and Experimentation” takes its place in your path, with examples from your field. The concepts stay those of the course.