What are loaders and chunkers
Loaders read files of each format into text when you add them. Chunkers then split that text into pieces small enough to embed and extract from.How loaders & chunkers work in Rust
Loaders handle file-format reading at ingest: a loader registry dispatches by MIME type / extension to per-format loaders (text, PDF, CSV, HTML, image, audio, and theunstructured office formats), most behind feature flags. They
run at add, which stores the extracted text (an image’s vision-LLM
description, an audio file’s transcript) and keeps the source file in
original_data_location. Chunkers
then segment a document into token-bounded pieces through a word → sentence →
paragraph hierarchy, sizing chunks with a pluggable TokenCounter
(WordCounter, or the feature-gated HuggingFace / tiktoken counters).
Rust: loaders (LoaderRegistry, DocumentLoader) in
cognee-ingestion; chunking (text_chunker, the
Chunker trait that chunk_strategy selects from,
TokenCounter) in cognee-chunking. Token-counter
selection is configured in configuration.
Related
Add
How add picks a loader
Cognify configuration
Chunking and tokenizer settings
Cognify
Where chunking runs