Normalization in networks stabilizes activations. BatchNorm normalizes each channel over the current batch (different behavior in training and evaluation), LayerNorm normalizes across the features of each example, independently of the batch, and dominates in transformers. It speeds up and stabilizes training.
Example
nn.BatchNorm1d(64)
nn.LayerNorm(64)