Linear Regression and Regression Metrics

Regression means predicting a numeric value: a price, a duration, a consumption. The simplest model, and often an excellent starting point, is linear regression:

ŷ = b + w₁·x₁ + w₂·x₂ + … + wₚ·xₚ

Each feature xⱼ receives a weight wⱼ, and b is the intercept. Training looks for the weights that minimize the sum of squared errors (ordinary least squares).

Training and reading the model

The built-in load_diabetes dataset describes 442 patients with 10 measurements (age, BMI, blood pressure…) already centered and scaled. The target is a measure of disease progression one year later.

from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression

X, y = load_diabetes(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)

reg = LinearRegression().fit(X_train, y_train)
print(reg.intercept_)                                   # constant b
print(dict(zip(X.columns, reg.coef_.round(1))))         # one weight per feature

A positive coefficient means that, with other variables held constant, the target increases when the feature increases. Comparing coefficients with one another only makes sense if the features are on the same scale, which is the case here.

Measuring the error

from sklearn.metrics import mean_absolute_error, root_mean_squared_error, r2_score

y_pred = reg.predict(X_test)
print("MAE :", mean_absolute_error(y_test, y_pred))      # about 45
print("RMSE:", root_mean_squared_error(y_test, y_pred))  # about 56
print("R²  :", r2_score(y_test, y_pred))                 # about 0.36

• MAE (mean absolute error): the average gap, in the unit of the target. Easy to explain.
• RMSE (root mean squared error): also in the unit of the target, but large errors weigh more. Always greater than or equal to the MAE.
• R²: share of the variance of y explained by the model. 1 = perfect; 0 = no better than predicting the mean; negative = worse than the mean. This is the value returned by reg.score.

Note: in recent versions of scikit-learn, you use root_mean_squared_error; the old squared=False argument of mean_squared_error has been removed.

Always compare to a baseline

Is an R² of 0.36 good? To find out, compare with a naive model:

from sklearn.dummy import DummyRegressor

baseline = DummyRegressor(strategy="mean").fit(X_train, y_train)
print(mean_absolute_error(y_test, baseline.predict(X_test)))   # about 58

The linear model reduces the average error from about 58 to 45: it brings real information, even though a good part of the variability remains unexplained. In a business context, it is often this comparison, expressed in the unit of the target, that convinces.

Limits

Linear regression assumes an additive, linear relationship. If reality is curved or involves interactions, you will need to create suitable features (DS5 course) or use nonlinear models (lessons 4 and 5). It is also sensitive to extreme values and to highly correlated features, which brings us to regularization.