Classification Metrics: Beyond Accuracy

Accuracy (the proportion of correct predictions) is intuitive, but it can be misleading. Take a fraud detection problem with 10% positive cases:

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.dummy import DummyClassifier

X, y = make_classification(n_samples=3000, n_features=12, n_informative=6, weights=[0.9],
                           class_sep=1.5, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, stratify=y, random_state=0)

naif = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
print(naif.score(X_test, y_test))   # about 0.90 without learning anything!

A model that always answers “no fraud” gets 90% accuracy… and detects no fraud at all. You need metrics that look at each type of error.

The confusion matrix

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix

clf = make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train)
y_pred = clf.predict(X_test)
print(confusion_matrix(y_test, y_pred))
# [[663   9]     rows = true classes, columns = predicted classes
#  [ 28  50]]

In the binary case, with positive class 1:

	predicted 0	predicted 1
true 0	TN: true negatives (663)	FP: false positives (9)
true 1	FN: false negatives (28)	TP: true positives (50)

ConfusionMatrixDisplay.from_estimator(clf, X_test, y_test) gives a graphical version.

Precision, recall, F1

• Precision = TP / (TP + FP) = 50 / 59 ≈ 0.85: when the model raises an alert, it is right 85% of the time.
• Recall (sensitivity) = TP / (TP + FN) = 50 / 78 ≈ 0.64: the model detects 64% of the fraud cases.
• F1 = 2 × precision × recall / (precision + recall) ≈ 0.73: the harmonic mean, high only if both are high.

from sklearn.metrics import precision_score, recall_score, f1_score, classification_report

print(precision_score(y_test, y_pred), recall_score(y_test, y_pred), f1_score(y_test, y_pred))
print(classification_report(y_test, y_pred, digits=3))

classification_report shows these metrics for each class, along with the macro average (simple mean of the classes, which gives as much weight to the rare class) and the weighted average (weighted by class size). The balanced accuracy (balanced_accuracy_score), the mean of the recalls of each class, is 0.5 for the naive model: it is not fooled by imbalance.

Which metric should you choose?

It all depends on the cost of errors:

• Medical screening, fraud: a false negative is costly → favor recall (while keeping an eye on precision).
• Spam filter, recommendation: a false positive annoys the user → favor precision.
• Balanced trade-off: F1.
• Balanced classes and errors of similar cost: accuracy remains acceptable.

For multiclass problems, precision_score(..., average="macro") or "weighted" summarizes the classes in one number. Set the metric before comparing models, in agreement with the business teams: changing metrics after the fact to flatter a model is a form of bias.

Finally, note that these metrics depend on the decision threshold (0.5 by default). The next lesson shows how to evaluate them independently of the threshold, then how to choose it.