Probability, Independence and Conditional Probability

Inferential statistics rest on probability: it lets us quantify uncertainty. Let’s start with the basic rules.

Events and probabilities

A probability is a number between 0 (impossible) and 1 (certain). If all outcomes are equally likely: P(A) = number of favorable cases / number of possible cases. With a die, P(“even”) = 3 / 6 = 0.5.

Essential rules:

• Complement: P(not A) = 1 - P(A).
• Union: P(A or B) = P(A) + P(B) - P(A and B). We subtract the intersection, otherwise it would be counted twice.

Conditional probability

P(A given B), written P(A | B), is the probability of A when we know that B has occurred: P(A | B) = P(A and B) / P(B).

Example with 1,000 website visitors:

import pandas as pd

# 1000 visitors: device x purchase
tab = pd.DataFrame({"achat": [60, 40], "pas_achat": [340, 560]},
                   index=["mobile", "ordinateur"])
total = tab.values.sum()                                   # 1000
p_achat = tab["achat"].sum() / total                       # 0.10
p_mobile = tab.loc["mobile"].sum() / total                 # 0.40
p_achat_et_mobile = tab.loc["mobile", "achat"] / total     # 0.06
p_achat_sachant_mobile = p_achat_et_mobile / p_mobile      # 0.15

Given that a visitor is on mobile, they buy with a probability of 15%, versus 10% overall.

Independence

A and B are independent if knowing B changes nothing about the probability of A: P(A | B) = P(A), which is equivalent to P(A and B) = P(A) x P(B). Here, 0.15 is different from 0.10: purchase and device are not independent.

Two coin tosses are independent: P(two heads) = 0.5 x 0.5 = 0.25.

Bayes’ theorem

It lets you invert a conditional probability: P(A | B) = P(B | A) x P(A) / P(B).

A classic example: a disease affects 1% of the population. A test detects it in 99% of sick people, but also gives 5% false positives among healthy people. If the test is positive, what is the probability of being sick?

p_m = 0.01               # P(sick)
p_pos_m = 0.99           # P(positive | sick), the sensitivity
p_pos_sain = 0.05        # P(positive | healthy), the false positive rate
p_pos = p_pos_m * p_m + p_pos_sain * (1 - p_m)   # total probability
print(p_pos_m * p_m / p_pos)                     # about 0.167

Only 17%! Because the disease is rare, most positives are false positives. Confusing P(positive | sick) with P(sick | positive) is a very common mistake, including in business (fraud alerts, risk scores).

Estimating by simulation

When the calculation is hard, you can simulate: the frequency observed over a large number of trials gets close to the probability.

import numpy as np
rng = np.random.default_rng(42)       # seed for a reproducible result
des = rng.integers(1, 7, size=100_000)
print((des % 2 == 0).mean())          # close to 0.5

Key takeaways

• P(A | B) restricts the universe to the cases where B is true.
• Independence: P(A and B) = P(A) x P(B).
• Bayes inverts conditional probabilities; the base rate (here 1%) matters enormously.