Designing an A/B Experiment

An A/B test is the most reliable way to know whether a change (new button, new price, new email) causes a change in behavior. You show the current version (A, the control) to one group, the new version (B, the variant) to another, and you compare.

Why randomize?

Comparing “before” and “after” a release mixes the effect of the change with everything else: seasonality, marketing campaigns, the news. If users are split at random between A and B at the same time, the two groups are comparable on everything, including what you don’t measure. The only systematic difference is the change: the observed difference is therefore attributable to it, up to chance, which statistics measure.

The experiment plan, written before launch

1. Hypothesis: “Showing shipping costs on the product page will increase the order rate, because customers will be less surprised at checkout.”
2. Primary metric (just one): order rate per visitor.
3. Guardrail metrics: average basket, refund rate, load time. They must not degrade.
4. Randomization unit: the user.
5. Sample size, duration, and alpha threshold (next lesson).

Fixing these choices in advance prevents you from “hunting” for a favorable metric after the fact.

Choosing the randomization unit

• By user: the most common; everyone always sees the same version, which avoids inconsistencies.
• By session or page view: more data, but the same customer may see both versions.
• By region or store when individuals interact (marketplace, in-store pricing).

The metric must be computed at the same level as the unit: if you randomize users, you compute a rate per user.

Assigning in a stable way: hashing

Drawing at random on every visit would make a user switch groups. Instead, you use a hash function of the identifier: the result looks random, but it is always the same for the same user.

import hashlib

def assigner(user_id, experience, part_b=0.5):
    # we combine the experiment and the user: two experiments don't share the same split
    cle = f"{experience}:{user_id}".encode()
    h = int(hashlib.sha256(cle).hexdigest()[:8], 16)   # integer between 0 and 16**8 - 1
    position = h / 16**8                               # number between 0 and 1
    return "B" if position < part_b else "A"

print(assigner("u_123", "frais_livraison"))   # always the same answer

Then you check that the split is balanced:

import pandas as pd
groupes = pd.Series([assigner(f"u_{i}", "frais_livraison") for i in range(10_000)])
print(groupes.value_counts(normalize=True))   # close to 50% / 50%

Duration

Even if the sample size is reached in three days, you run the experiment for at least one full week: Monday behavior is not Saturday behavior.

Key takeaways

• Randomization makes the comparison causal.
• One hypothesis, one primary metric, guardrails, all defined before launch.
• Stable assignment by hashing, at the level of the randomization unit.