Indexing pipeline

The indexing pipeline prepares documents for search: text extraction, splitting into chunks, adding metadata, computing embeddings, then storing in an index. It runs ahead of the questions and is replayed when the documents change.

Example

documents -> text -> chunks -> embeddings -> index

Related concepts

The whole glossary