Text normalization

Text normalization standardizes raw text before analysis: lowercasing, removing or handling punctuation, accents and superfluous whitespace, Unicode normalization. It reduces vocabulary variability, but must be chosen carefully because it can remove useful information, such as a capital letter or a distinctive accent.

Example

texte.lower().strip()

Related concepts

The whole glossary