Token-Based Sizing
Cognee uses token-based sizing for chunks, rather than character counts. This means thatchunk_size refers to the maximum number of tokens allowed in a chunk, which is directly tied to the tokenizer used by your embedding model. This ensures that chunks are always within the model’s context window.
The token-counting tokenizer is auto-selected to match your embedding model, so chunk sizes (and the --dry-run token estimate) reflect how the model actually tokenizes text. If a matching tokenizer cannot be loaded, Cognee logs a non-fatal advisory warning and falls back to the TikToken tokenizer — token counts become approximate but ingestion continues. To restore exact counts and silence the warning, set HUGGINGFACE_TOKENIZER to a tokenizer that matches your embedding model (Ollama), or install cognee[huggingface] if the warning says transformers is not installed. See Embedding Providers for the per-provider tokenizer mapping and override details.
The chunk_size ceiling is respected wherever the text gives Cognee a boundary to split on; a run with no boundary at all is emitted whole as a single oversized chunk rather than cut mid-run — see Unsplittable Text and Oversized Chunks below.
Available Chunkers
Cognee provides several built-in chunkers to handle different types of content:- TextChunker: The default chunker. It splits text by paragraphs while respecting the token limit. It tries to keep paragraphs together but will split them if they exceed the
chunk_size. - CsvChunker: Designed specifically for CSV data. It splits by rows, ensuring that each chunk contains complete rows and does not break data in the middle of a record. Each row is chunked independently — row-level state (text, size, and chunk index) is reset at each row boundary, so one row’s fields are never accumulated onto the next row’s chunk.
- LangchainChunker: Wraps LangChain’s
RecursiveCharacterTextSplitter. It splits text recursively by characters (e.g.,\n\n,\n,) and supportschunk_overlap(in words; default:10). Requirespip install cognee[langchain]. - TextChunkerWithOverlap: A paragraph-based chunker that supports overlap via a
chunk_overlap_ratio(a fraction ofchunk_size, e.g.0.2= 20% overlap). Useful for maintaining context across chunk boundaries.
importance_weight onto each chunk it yields, so the weight you pass to remember() or add() reaches retrieval ranking whichever chunker you select.
Overlap only works with
LangchainChunker and TextChunkerWithOverlap. The default TextChunker splits strictly at paragraph boundaries and does not use overlap. Calling cognee.config.set_chunk_overlap() has no effect when using TextChunker.Default chunker for a deployment
A call that passes nochunker uses the chunker the DEFAULT_CHUNKER setting names. When the setting is unset or blank, that is TextChunker. This covers cognify(), remember(), update(), the CLI, the HTTP API and Cognee’s own pipelines. A chunker class passed to a call always wins over the setting.
DEFAULT_CHUNKER takes a name, never an import path. Valid names are TextChunker and any name added with register_chunker(name, cls):
register_chunker() checks the class:
- It must subclass
Chunkerand declare achunker_id. - The id must be its own: one used by another registered or built-in chunker is refused, and so is
code_docs, which the code-docs pipeline stamps on its chunks. "TextChunker"always names the built-inTextChunkerand cannot be registered for another class.
- Register in every process that reads the setting: API workers, the MCP server, the CLI and scripts. If
DEFAULT_CHUNKERnames a chunker that is not registered, every write fails withCogneeConfigurationError, which lists the registered names.remember()checks this before anything is stored. - The HTTP API has no chunker field. To make server-side
cognifyandrememberuse a custom chunker, register it when the server starts and setDEFAULT_CHUNKER. - Changing the default and
update(): the chunk-level path ofupdate()needs the chunker that cut the stored chunks. AfterDEFAULT_CHUNKERchanges, the first update of each existing document rebuilds it once with the new chunker (fallback reasonunsupported_chunker). Later updates are chunk-level again.cognify()on its own never re-cuts a document it has already processed.
Additional Information
Using a Specific Chunker
Using a Specific Chunker
Using Chunk Overlap
Using Chunk Overlap
Chunk overlap causes consecutive chunks to share a portion of text, which helps preserve context at chunk boundaries and can improve entity extraction quality — at the cost of more LLM calls and a slightly larger graph.When standard document readers instantiate your chunker, they call it as
chunker_cls(document, get_text=..., max_chunk_size=...). The max_chunk_size
value is the chunk_size you pass to remember() or cognify(). Extra
chunking parameters such as overlap are not forwarded, so set them by
subclassing the chunker and hard-wiring them inside __init__.- TextChunkerWithOverlap
- LangchainChunker
TextChunkerWithOverlap takes a chunk_overlap_ratio between 0.0 and 1.0 (fraction of chunk_size). Because this value cannot be passed through cognee.config.set_chunk_overlap() at runtime, configure it by subclassing:Custom Chunkers
Custom Chunkers
You can create a custom chunker by inheriting from the Chunker ids are stamped for you. Cognee gives every chunk a chunker yields without a
Chunker base class and implementing the read method. Your chunker must yield DocumentChunk objects.DocumentChunk.importance_weight defaults to 0.5, so a custom chunker that omits the field ranks every chunk at the default and ignores the document’s importance_weight.chunker_id the chunker’s own chunker_id, both during extraction and when update() re-cuts an edited region. This applies to any chunker that declares an id, whether it is registered or passed per call with chunker=..., so your read() can leave the field unset. A chunk yielded with a different id fails the run with CogneeConfigurationError (“Build every chunk with chunker_id=self.chunker_id, or leave it unset”), because stored with it the document would later be updated as if that other chunker had cut it.update() compares this id with the one on the stored chunks to decide whether it can replace only the edited chunks. For that path the chunker also has to cut without overlap or loss: its chunk texts, joined in order, must be the document’s text. Otherwise update() rebuilds the document in full on every edit (fallback reason chunks_not_tiling). update() re-runs the chunker over the edited region alone, so a cut should depend only on the text it is given. See update().To make a custom chunker the default for every call, register it — see Default chunker for a deployment. The full example, a SectionChunker that cuts Markdown at its headings, is examples/guides/custom_chunker.py.Configuring chunk size via Docker or REST API
Configuring chunk size via Docker or REST API
The With Docker Compose, add these to your Then start the server:With The API also takes no chunker parameter. The server cuts with the chunker the
/cognify REST API endpoint does not accept a chunk_size parameter directly. When you use Cognee through Docker or the HTTP API, set the chunk size via environment variables instead:.env before starting:docker run, pass them inline:The
chunk_size environment variable is read once at startup. Restart the container after changing it.DEFAULT_CHUNKER setting names. See Default chunker for a deployment.Chunk Size and Graph Quality
Chunk Size and Graph Quality
The
chunk_size passed to remember() directly affects how the knowledge graph is built:If
chunk_size is not set, Cognee auto-calculates it as the minimum of the embedding token limit and half of your LLM’s context window. The embedding half is itself min(EMBEDDING_MAX_COMPLETION_TOKENS, the model's own input limit) — the variable is a ceiling (unset means 4096), and Cognee detects what the model actually accepts from the provider (fastembed’s loaded tokenizer, Ollama’s /api/show, LiteLLM’s model table, or the model’s HuggingFace tokenizer).An explicit chunk_size above that embedding limit is lowered to it, with a warning naming which of the two bounds bit — it is not honored. Text past the limit would otherwise be dropped from the embedding or rejected by the provider. The LLM half-context bound applies only to the automatic size. See Max Completion Tokens.Unsplittable Text and Oversized Chunks
Unsplittable Text and Oversized Chunks
chunk_size is a ceiling Cognee respects wherever the text gives it a boundary to respect. A run with no sentence or paragraph boundary in it — a protein or DNA sequence, a base64 blob, a very long URL, a private-use glyph run left by PDF extraction — offers none, so it is emitted whole as a single oversized chunk rather than cut mid-run. This holds wherever the run sits in the document, including when it is the last thing in the document; a trailing run previously raised ValueError: Input word … longer than chunking size …, which failed the entire pipeline run rather than just that document.For a trailing run, Cognee logs a non-fatal advisory warning naming the token count and the limit; a mid-document oversized run is emitted silently, so this warning is the only log signal an oversized chunk ever produces, and only in the trailing case:Preserving structured or procedural data (rules, XML, JSON)
Preserving structured or procedural data (rules, XML, JSON)
For structured data, the goal is to make chunk boundaries follow the shape of the source.
TextChunker (and the underlying chunk_by_paragraph) preserves sentence and paragraph boundaries where possible, batching complete sentence-sized units up to chunk_size before starting a new chunk. It does not understand a “rule”, XML element, or JSON record by itself, so treat chunk_size as a guardrail rather than a semantic guarantee.If you have procedural rules, configuration, or other structured content where each unit must stay intact, choose the smallest source unit you would be comfortable retrieving on its own:-
Raise
chunk_sizeso the whole document fits in one chunk. Becausechunk_sizeis a token budget, any content that fits under it is kept together. Set it large enough to hold your entire ruleset and no splitting occurs:This has a ceiling: the value is capped at what the embedding model accepts, and atEMBEDDING_MAX_COMPLETION_TOKENSwhen that is lower — unset, it is4096, so the8000above would be lowered to4096with a warning. RaiseEMBEDDING_MAX_COMPLETION_TOKENSalongsidechunk_size, and no higher than the model’s own limit (8191for OpenAI’stext-embedding-3-*). If your ruleset cannot fit under that limit, use option 2 instead. -
Add each rule as its own data item so every rule becomes an independent document and is processed on its own:
- Write a custom chunker (see the Custom Chunkers section above) that splits on your own boundaries (e.g. one chunk per rule, per XML element, or per JSON record) when the natural unit isn’t a paragraph.
Cognee is strongest when the data has concepts, entities, and relationships to reason over. For workflows that must reproduce deterministic API calls or fixed rules verbatim, keep the canonical procedure in a regular file or source system, then use Cognee for the parts that benefit from semantic search and graph reasoning.