Skip to main content
Chunkers are responsible for splitting large documents into smaller, manageable pieces called chunks. This is a crucial step before embedding and graph extraction, as most embedding models have a limit on the amount of text they can process at once.

Token-Based Sizing

Cognee uses token-based sizing for chunks, rather than character counts. This means that chunk_size refers to the maximum number of tokens allowed in a chunk, which is directly tied to the tokenizer used by your embedding model. This ensures that chunks are always within the model’s context window. The token-counting tokenizer is auto-selected to match your embedding model, so chunk sizes (and the --dry-run token estimate) reflect how the model actually tokenizes text. If a matching tokenizer cannot be loaded, Cognee logs a non-fatal advisory warning and falls back to the TikToken tokenizer — token counts become approximate but ingestion continues. To restore exact counts and silence the warning, set HUGGINGFACE_TOKENIZER to a tokenizer that matches your embedding model (Ollama), or install cognee[huggingface] if the warning says transformers is not installed. See Embedding Providers for the per-provider tokenizer mapping and override details. The chunk_size ceiling is respected wherever the text gives Cognee a boundary to split on; a run with no boundary at all is emitted whole as a single oversized chunk rather than cut mid-run — see Unsplittable Text and Oversized Chunks below.

Available Chunkers

Cognee provides several built-in chunkers to handle different types of content:
  • TextChunker: The default chunker. It splits text by paragraphs while respecting the token limit. It tries to keep paragraphs together but will split them if they exceed the chunk_size.
  • CsvChunker: Designed specifically for CSV data. It splits by rows, ensuring that each chunk contains complete rows and does not break data in the middle of a record. Each row is chunked independently — row-level state (text, size, and chunk index) is reset at each row boundary, so one row’s fields are never accumulated onto the next row’s chunk.
  • LangchainChunker: Wraps LangChain’s RecursiveCharacterTextSplitter. It splits text recursively by characters (e.g., \n\n, \n, ) and supports chunk_overlap (in words; default: 10). Requires pip install cognee[langchain].
  • TextChunkerWithOverlap: A paragraph-based chunker that supports overlap via a chunk_overlap_ratio (a fraction of chunk_size, e.g. 0.2 = 20% overlap). Useful for maintaining context across chunk boundaries.
Every built-in chunker copies the document’s importance_weight onto each chunk it yields, so the weight you pass to remember() or add() reaches retrieval ranking whichever chunker you select.
Overlap only works with LangchainChunker and TextChunkerWithOverlap. The default TextChunker splits strictly at paragraph boundaries and does not use overlap. Calling cognee.config.set_chunk_overlap() has no effect when using TextChunker.

Default chunker for a deployment

A call that passes no chunker uses the chunker the DEFAULT_CHUNKER setting names. When the setting is unset or blank, that is TextChunker. This covers cognify(), remember(), update(), the CLI, the HTTP API and Cognee’s own pipelines. A chunker class passed to a call always wins over the setting. DEFAULT_CHUNKER takes a name, never an import path. Valid names are TextChunker and any name added with register_chunker(name, cls):
register_chunker() checks the class:
  • It must subclass Chunker and declare a chunker_id.
  • The id must be its own: one used by another registered or built-in chunker is refused, and so is code_docs, which the code-docs pipeline stamps on its chunks.
  • "TextChunker" always names the built-in TextChunker and cannot be registered for another class.
Never reuse an id, not even that of a chunker you no longer register: documents it cut still carry the id, and only currently registered and built-in ids are checked. A subclass that cuts differently from its parent declares a new id.
  • Register in every process that reads the setting: API workers, the MCP server, the CLI and scripts. If DEFAULT_CHUNKER names a chunker that is not registered, every write fails with CogneeConfigurationError, which lists the registered names. remember() checks this before anything is stored.
  • The HTTP API has no chunker field. To make server-side cognify and remember use a custom chunker, register it when the server starts and set DEFAULT_CHUNKER.
  • Changing the default and update(): the chunk-level path of update() needs the chunker that cut the stored chunks. After DEFAULT_CHUNKER changes, the first update of each existing document rebuilds it once with the new chunker (fallback reason unsupported_chunker). Later updates are chunk-level again. cognify() on its own never re-cuts a document it has already processed.

Additional Information

Chunk overlap causes consecutive chunks to share a portion of text, which helps preserve context at chunk boundaries and can improve entity extraction quality — at the cost of more LLM calls and a slightly larger graph.When standard document readers instantiate your chunker, they call it as chunker_cls(document, get_text=..., max_chunk_size=...). The max_chunk_size value is the chunk_size you pass to remember() or cognify(). Extra chunking parameters such as overlap are not forwarded, so set them by subclassing the chunker and hard-wiring them inside __init__.
TextChunkerWithOverlap takes a chunk_overlap_ratio between 0.0 and 1.0 (fraction of chunk_size). Because this value cannot be passed through cognee.config.set_chunk_overlap() at runtime, configure it by subclassing:
You can create a custom chunker by inheriting from the Chunker base class and implementing the read method. Your chunker must yield DocumentChunk objects.
DocumentChunk.importance_weight defaults to 0.5, so a custom chunker that omits the field ranks every chunk at the default and ignores the document’s importance_weight.
Chunker ids are stamped for you. Cognee gives every chunk a chunker yields without a chunker_id the chunker’s own chunker_id, both during extraction and when update() re-cuts an edited region. This applies to any chunker that declares an id, whether it is registered or passed per call with chunker=..., so your read() can leave the field unset. A chunk yielded with a different id fails the run with CogneeConfigurationError (“Build every chunk with chunker_id=self.chunker_id, or leave it unset”), because stored with it the document would later be updated as if that other chunker had cut it.update() compares this id with the one on the stored chunks to decide whether it can replace only the edited chunks. For that path the chunker also has to cut without overlap or loss: its chunk texts, joined in order, must be the document’s text. Otherwise update() rebuilds the document in full on every edit (fallback reason chunks_not_tiling). update() re-runs the chunker over the edited region alone, so a cut should depend only on the text it is given. See update().To make a custom chunker the default for every call, register it — see Default chunker for a deployment. The full example, a SectionChunker that cuts Markdown at its headings, is examples/guides/custom_chunker.py.
The /cognify REST API endpoint does not accept a chunk_size parameter directly. When you use Cognee through Docker or the HTTP API, set the chunk size via environment variables instead:
With Docker Compose, add these to your .env before starting:
Then start the server:
With docker run, pass them inline:
The chunk_size environment variable is read once at startup. Restart the container after changing it.
The API also takes no chunker parameter. The server cuts with the chunker the DEFAULT_CHUNKER setting names. See Default chunker for a deployment.
The chunk_size passed to remember() directly affects how the knowledge graph is built:If chunk_size is not set, Cognee auto-calculates it as the minimum of the embedding token limit and half of your LLM’s context window. The embedding half is itself min(EMBEDDING_MAX_COMPLETION_TOKENS, the model's own input limit) — the variable is a ceiling (unset means 4096), and Cognee detects what the model actually accepts from the provider (fastembed’s loaded tokenizer, Ollama’s /api/show, LiteLLM’s model table, or the model’s HuggingFace tokenizer).An explicit chunk_size above that embedding limit is lowered to it, with a warning naming which of the two bounds bit — it is not honored. Text past the limit would otherwise be dropped from the embedding or rejected by the provider. The LLM half-context bound applies only to the automatic size. See Max Completion Tokens.
chunk_size is a ceiling Cognee respects wherever the text gives it a boundary to respect. A run with no sentence or paragraph boundary in it — a protein or DNA sequence, a base64 blob, a very long URL, a private-use glyph run left by PDF extraction — offers none, so it is emitted whole as a single oversized chunk rather than cut mid-run. This holds wherever the run sits in the document, including when it is the last thing in the document; a trailing run previously raised ValueError: Input word … longer than chunking size …, which failed the entire pipeline run rather than just that document.For a trailing run, Cognee logs a non-fatal advisory warning naming the token count and the limit; a mid-document oversized run is emitted silently, so this warning is the only log signal an oversized chunk ever produces, and only in the trailing case:
It is worth watching, because such a chunk can exceed your embedding model’s input limit downstream. If you see it and the source is splittable in a way Cognee can’t infer, pre-split the content before ingesting or write a custom chunker that understands your format (see the Custom Chunkers accordion above).
For structured data, the goal is to make chunk boundaries follow the shape of the source. TextChunker (and the underlying chunk_by_paragraph) preserves sentence and paragraph boundaries where possible, batching complete sentence-sized units up to chunk_size before starting a new chunk. It does not understand a “rule”, XML element, or JSON record by itself, so treat chunk_size as a guardrail rather than a semantic guarantee.If you have procedural rules, configuration, or other structured content where each unit must stay intact, choose the smallest source unit you would be comfortable retrieving on its own:
  1. Raise chunk_size so the whole document fits in one chunk. Because chunk_size is a token budget, any content that fits under it is kept together. Set it large enough to hold your entire ruleset and no splitting occurs:
    This has a ceiling: the value is capped at what the embedding model accepts, and at EMBEDDING_MAX_COMPLETION_TOKENS when that is lower — unset, it is 4096, so the 8000 above would be lowered to 4096 with a warning. Raise EMBEDDING_MAX_COMPLETION_TOKENS alongside chunk_size, and no higher than the model’s own limit (8191 for OpenAI’s text-embedding-3-*). If your ruleset cannot fit under that limit, use option 2 instead.
  2. Add each rule as its own data item so every rule becomes an independent document and is processed on its own:
  3. Write a custom chunker (see the Custom Chunkers section above) that splits on your own boundaries (e.g. one chunk per rule, per XML element, or per JSON record) when the natural unit isn’t a paragraph.
Cognee is strongest when the data has concepts, entities, and relationships to reason over. For workflows that must reproduce deterministic API calls or fixed rules verbatim, keep the canonical procedure in a regular file or source system, then use Cognee for the parts that benefit from semantic search and graph reasoning.