Job diagnosis

Diagnosing a slow Spark job means reading the Spark UI: spotting the longest stage, comparing task durations, and measuring the volume of shuffle, disk spill, and partition distribution. You then fix the cause, for example skew, an unsuitable number of partitions, or a poorly chosen join.

Related concepts

The whole glossary