Skip to main content

What are loaders and chunkers

Loaders read files of each format into text when you add them. Chunkers then split that text into pieces small enough to embed and extract from.

How loaders & chunkers work in Rust

Loaders handle file-format reading at ingest: a loader registry dispatches by MIME type / extension to per-format loaders (text, PDF, CSV, HTML, image, audio, and the unstructured office formats), most behind feature flags. They run at add, which stores the extracted text (an image’s vision-LLM description, an audio file’s transcript) and keeps the source file in original_data_location. Chunkers then segment a document into token-bounded pieces through a word → sentence → paragraph hierarchy, sizing chunks with a pluggable TokenCounter (WordCounter, or the feature-gated HuggingFace / tiktoken counters). Rust: loaders (LoaderRegistry, DocumentLoader) in cognee-ingestion; chunking (text_chunker, the Chunker trait that chunk_strategy selects from, TokenCounter) in cognee-chunking. Token-counter selection is configured in configuration.

Add

How add picks a loader

Cognify configuration

Chunking and tokenizer settings

Cognify

Where chunking runs