Transform Numerical Variables

Feature engineering means transforming raw data to make the information easier for the model to use. On tabular data, this is often where most of the performance is won. The value of a transformation depends on the model: a linear model benefits a lot, a decision tree much less, because it is insensitive to any monotonic transformation (one that preserves the order of values).

Correcting skewness

Income, purchase amounts, number of visits: these variables often have a long right tail. A few very large values overwhelm the others and pull on the coefficients of a linear model.

import numpy as np
import pandas as pd

rng = np.random.default_rng(0)
revenu = pd.Series(rng.lognormal(mean=10, sigma=1, size=1000))
print(revenu.skew())             # about 2.8: strong skewness
print(np.log1p(revenu).skew())   # close to 0 after transformation

np.log1p(x) computes log(1 + x) and accepts zeros. In a pipeline, you wrap it in a FunctionTransformer(np.log1p). For an automatically learned transformation:

• PowerTransformer(method="yeo-johnson") (default) looks for the power that makes the distribution as Gaussian as possible, negative values included; method="box-cox" requires strictly positive values;
• QuantileTransformer(output_distribution="normal") forces a target distribution based on ranks, very robust to extreme values.

If it is the regression target that is skewed, use TransformedTargetRegressor(regressor=..., func=np.log1p, inverse_func=np.expm1): the model learns on the log, and predictions are automatically brought back to the original scale.

Discretizing

KBinsDiscretizer splits a continuous variable into intervals:

from sklearn.preprocessing import KBinsDiscretizer

kb = KBinsDiscretizer(n_bins=4, encode="ordinal", strategy="quantile")
tranche = kb.fit_transform(revenu.to_frame())   # 0, 1, 2 or 3 (quartiles)
print(kb.bin_edges_[0])                          # edges learned from the data

strategy="quantile" creates bins with the same number of observations, "uniform" bins of the same width. With encode="onehot" (the default), each bin becomes a binary column, which lets a linear model learn a stepwise nonlinear effect. Discretization loses fine-grained information: use it when the effect really works by thresholds or for readability (age brackets).

Interactions and polynomials

A linear model adds up effects. If the effect of one variable depends on another (for example surface area × neighborhood), you must give it the product:

from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.model_selection import cross_val_score

X = rng.uniform(-2, 2, size=(400, 2))
y = 3 * X[:, 0] * X[:, 1] + X[:, 0] + rng.normal(0, 0.3, 400)   # pure interaction

lineaire = LinearRegression()
poly = make_pipeline(PolynomialFeatures(degree=2, include_bias=False), LinearRegression())
print(cross_val_score(lineaire, X, y, cv=5).mean())   # about 0.05
print(cross_val_score(poly, X, y, cv=5).mean())       # about 0.99
print(poly.fit(X, y)[0].get_feature_names_out(["a", "b"]))   # ['a' 'b' 'a^2' 'a b' 'b^2']

Beware of combinatorial explosion: with 50 features and degree=2, you get 1,325 columns. Use interaction_only=True to keep only the products, combine with regularization (Ridge), and standardize afterwards. SplineTransformer is often a more stable alternative to polynomials for modeling curves.