Tokenization splits a text into units called tokens: words, subwords or characters. It is the first step of any text processing, and each token is then mapped to a numeric identifier. Modern language models use subword tokens, which avoid unknown words.
Example
'apprentissage' -> ['appr', 'entissage']