Narrow and wide transformations

A narrow transformation (filter, select, map) works partition by partition, with no data exchange. A wide transformation (groupBy, join, distinct, repartition) requires a shuffle: rows are redistributed among the executors over the network. The shuffle is the most expensive operation in a Spark job.

Example

df.groupBy('pays').sum('montant')  # wide transformation, causes a shuffle

Related concepts

The whole glossary