Splitting and Indexing Documents

From Demo to Production

A RAG prototype fits in a few lines: split a text every 200 words, compute vectors, find the nearest ones, paste them into the prompt. In production, the difficulties are elsewhere: hundreds of heterogeneous documents, daily updates, different access rights depending on the team, and users who want to verify the source of an answer. This course tackles these questions, starting from the foundation: a reliable indexing pipeline.

The Indexing Pipeline

Indexing is a separate process from querying, run as a background task whenever a document is added or modified:

1. Extract the text (PDF, web pages, documentation tool, tickets). The quality of the extraction determines everything else: repeated page headers, flattened tables, and mixed-up columns degrade search.
2. Clean: remove repetitive elements, normalize whitespace, keep the headings.
3. Split into chunks.
4. Encode each chunk as a vector (lesson 2).
5. Store text, vector, and metadata in the index.

The pipeline must be reproducible: rerun on the same documents, it produces the same chunks with the same identifiers.

Splitting by Structure

Fixed-size splitting cuts in the middle of sentences and mixes two sections in the same chunk. Company documents almost always have a structure (Markdown or HTML headings, heading styles in a word processor): use it.

def split_sections(markdown, doc_id):
    sections, path, buffer = [], [], []
    for line in markdown.splitlines():
        match = re.match(r"^(#{1,6})\s+(.+)$", line)
        if match:
            # save the previous section, then update the path
            ...
            level = len(match.group(1))
            path[:] = path[:level - 1] + [match.group(2).strip()]
        else:
            buffer.append(line)

The heading path is kept: a paragraph “Carry-over is limited to 5 days” only makes sense with its context “Paid leave > Carry-over”.

Then Bound the Size

A section can be very long. You then group its paragraphs into chunks that do not exceed a maximum size (in words here, in tokens in production), never cutting a paragraph if it fits:

• add paragraphs one by one as long as the limit is not exceeded;
• when the next one no longer fits, close the chunk;
• the new chunk starts with the last paragraph of the previous one (overlap), if that fits within the limit.

Overlap prevents information that straddles two chunks from being unfindable. It has a cost: duplicated text in the index and sometimes in the prompt. A single paragraph longer than the limit forms its own chunk (or is split again by sentences).

Identifiers and Metadata

Each chunk gets a stable identifier, for example politique-conges#3 (document and rank). It is used for citations, evaluation, and updates. You attach metadata to it:

{
    "chunk_id": "politique-conges#3",
    "doc_id": "politique-conges",
    "path": "Paid leave > Carry-over",
    "text": "Carry-over is limited to 5 days...",
    "embed_text": "Paid leave > Carry-over\n\nCarry-over is limited to 5 days...",
    "organization_id": 12, "visibility": "rh", "version": "2026-09",
}

embed_text (path + text) is what you encode; text is what you display and cite. The organization_id and visibility fields will be used for filtering by access rights (lesson 5).

Tuning and Checking

There is no universal ideal size. Chunks that are too small lose their context; too large, they dilute the information and fill the prompt. Start from a reasonable value (a few hundred tokens), then measure on a set of reference questions (lesson 4), comparing several settings. Also inspect random chunks: it is the fastest way to spot a failed extraction.

Key Takeaways

• Indexing is a reproducible pipeline, separate from querying.
• Split by structure, bound the size, add an overlap.
• Stable identifiers and metadata make citations, access rights, and updates possible.