repartition and coalesce

repartition(n) redistributes the data into n partitions with a full shuffle, and can increase or decrease their number or balance their size. coalesce(n) reduces the number of partitions by merging neighboring ones without a full shuffle, which is cheaper but can leave partitions unbalanced.

Example

df.coalesce(10).write.parquet('sortie')

Related concepts

The whole glossary