Pruning and columnar formats

Pruning avoids reading unnecessary data: partition pruning skips the folders excluded by the filter, and column pruning reads only the requested ones. Columnar formats like Parquet or ORC, with per-block statistics and predicates pushed down to the source, make these optimizations possible.

Example

spark.read.parquet('ventes').select('id', 'montant').filter('annee = 2025')

Related concepts

The whole glossary