Schema and file reading

A DataFrame's schema defines the names and types of its columns. When reading files, Spark can infer it, which costs a pass over the data and can be wrong, or receive it explicitly via a StructType, which is faster and safer. Formats like Parquet already embed their schema.

Example

schema = StructType([StructField('id', IntegerType()), StructField('montant', DoubleType())])
df = spark.read.schema(schema).csv('ventes.csv', header=True)

Related concepts

The whole glossary