Tensors and Automatic Differentiation

First lesson of the Deep Learning with PyTorch course, free to read: about 15 minutes, 3 exercises and 3 quiz questions. This is the catalogue's base lesson; in the student space, it is rewritten for your profile.

Objectives

Create and manipulate PyTorch tensors (shape, type, device)
Understand the computation graph and compute gradients with autograd
Check a gradient with finite differences

Tensor Autograd Gradient

PyTorch rests on two building blocks: tensors, multidimensional arrays close to numpy, and autograd, an engine that automatically computes derivatives. All of deep learning builds on them.

Create and manipulate tensors

import torch

x = torch.tensor([[1.0, 2.0], [3.0, 4.0]])   # from a list
z = torch.zeros(3, 4)                         # shape (3, 4)
r = torch.randn(2, 3)                         # standard normal distribution
print(x.shape, x.dtype)                       # torch.Size([2, 2]) torch.float32

y = x @ x.T            # matrix product
m = x.mean(dim=0)      # mean per column
v = x.reshape(4)       # change of shape (a view if possible)
a = torch.from_numpy(np.ones(3))  # shares memory with numpy (import numpy as np)

Broadcasting works as in numpy: x + torch.tensor([10.0, 20.0]) adds the vector to each row. By default, floats are float32, the standard type in deep learning.

Choose the device

if torch.cuda.is_available():
    device = "cuda"
elif torch.backends.mps.is_available():   # Apple chips
    device = "mps"
else:
    device = "cpu"
x = x.to(device)

With recent versions of PyTorch (this course uses 2.15), torch.accelerator gives the same result without naming each hardware: device = torch.accelerator.current_accelerator() or "cpu".

Two tensors must be on the same device to be combined, otherwise PyTorch raises an explicit error.

Autograd: the computation graph

When a tensor has requires_grad=True, PyTorch records every operation that uses it. A call to .backward() walks this graph in reverse (backpropagation) and computes the derivatives:

w = torch.tensor(2.0, requires_grad=True)
b = torch.tensor(0.5, requires_grad=True)
x, y_vrai = torch.tensor(3.0), torch.tensor(7.0)

y_pred = w * x + b                 # 6.5
loss = (y_pred - y_vrai) ** 2      # 0.25
loss.backward()                    # computes dloss/dw and dloss/db

print(w.grad)   # 2 * (y_pred - y_vrai) * x = -3.0
print(b.grad)   # 2 * (y_pred - y_vrai)     = -1.0

Three important rules:

.backward() is called on a scalar (a loss).
Gradients accumulate in .grad: without a reset, a second backward() adds the new gradients to the old ones. This is why training loops call optimizer.zero_grad().
For evaluation, you turn off tracking, which saves memory and time:

with torch.no_grad():          # or torch.inference_mode()
    prediction = w * x + b

tensor.detach() returns a copy outside the graph, and .item() extracts the Python value from a scalar tensor (useful for logging a loss).

Check a gradient

A gradient is checked with centered finite differences: (f(x + h) - f(x - h)) / (2h) approximates the derivative for a small h. This is how you test a hand-written implementation, and it is this lesson’s exercise in numpy.

import numpy as np

def f(x):
    return np.sum(x ** 2)

x0, h = np.array([1.0, -2.0]), 1e-5
approx = [(f(x0 + h * e) - f(x0 - h * e)) / (2 * h) for e in np.eye(2)]
print(approx)   # close to 2 * x0 = [2, -4]

Key takeaways

A tensor has a shape, a type, and a device. Autograd computes the derivatives of a scalar loss with respect to the parameters; you reset the gradients to zero at each iteration and turn off tracking at inference.

Exercises

Solutions and tests stay on the server. In the student space, your code is run and checked, and the tutor gives hints before the solution.

CodeDifficulty 2Python in the browser2 hints

Gradient by finite differences

Write gradient_numerique(f, x, h=1e-5) that returns a numpy array containing the approximation of each partial derivative of f at x using centered finite differences.

import numpy as np

def gradient_numerique(f, x, h=1e-5):
    x = np.asarray(x, dtype=float)
    grad = np.zeros_like(x)
    # TODO : pour chaque coordonnée i, (f(x + h e_i) - f(x - h e_i)) / (2h)
    return grad

CodeDifficulty 2Python in the browser2 hints

Backpropagation by hand

For a linear model y_pred = w * x + b and the MSE loss L = mean((y_pred - y)^2), write gradients_mse(w, b, x, y) that returns the tuple (dL/dw, dL/db) computed analytically (this is what autograd would do for you).

import numpy as np

def gradients_mse(w, b, x, y):
    x, y = np.asarray(x, float), np.asarray(y, float)
    # TODO
    pass

TaskDifficulty 12 hints

Autograd in your environment

In your working environment, open a notebook or a Python script and:

print torch.__version__ and the available device;
reproduce the lesson’s w * x + b example and print w.grad and b.grad;
call backward() a second time, recomputing the loss, without resetting, and observe w.grad;
reset the gradients to zero with w.grad.zero_() and check.

The PyTorch version and the device (cpu/cuda/mps) are printed
w.grad equals -3.0 and b.grad equals -1.0 after the first backward
After the second backward without a reset, w.grad equals -6.0 (accumulation)
After zero_(), w.grad equals 0

End-of-lesson quiz

What happens if you call loss.backward() twice (recomputing loss) without resetting?

The gradients are replaced
The gradients are added together in .grad
PyTorch always raises an error
The weights are updated automatically

What is the with torch.no_grad(): block for?

Speeding up training
Disabling the recording of the computation graph, for example for evaluation
Resetting the gradients to zero
Moving the model to the GPU

You add a tensor on ‘cuda’ and a tensor on ‘cpu’. What happens?

PyTorch automatically copies to the GPU
An error says the tensors are on different devices
The result is on the CPU
The computation is done in float64