UDFs and pandas UDFs

A UDF (user-defined function) lets you apply Python code to columns when no native function fits. It is opaque to the optimizer and slow, because each row is serialized to Python. A pandas UDF works in batches via Apache Arrow and is significantly faster.

Example

@F.udf('string')
def nettoyer(s): return s.strip().lower()

Related concepts

The whole glossary