What is the cognify operation
cognify turns data stored by add into a knowledge graph plus vector indexes. remember runs it for you.
What happens under the hood
Turns stored data into a knowledge graph in five stages: classify documents → chunk text → extract entities/relationships (LLM, batched) and summarize (conditional) — one fused stage running both concurrently over the same chunks → add data points (six vector collections, plusDltRow_text for DLT rows, and provenance to the relational DB) →
extract DLT FK edges. A DLT row (a document whose external_metadata has
source: "dlt") is one DltRow chunk indexed into DltRow_text, not
DocumentChunk_text; the last step adds its SchemaTable, SchemaRelationship
and DltColumn nodes (vector collections SchemaTable_name,
SchemaTable_description, SchemaRelationship_name,
SchemaRelationship_description and DltColumn_properties) and the row edges.
Datasets cognified before this layout need a re-cognify after deleting their
old graph and vector rows. Configurable via CognifyConfig
(chunk strategy, custom prompts/schemas). Pipeline:
cognee-cognify (cognify() / cognify_datasets()).
Writing vectors
The add data points stage embeds and writes each vector collection in windows of about 16 MiB of vector data (cognee_vector::write_window_points,
around 2,700 points at 1536 dimensions), so memory on the write path does not
grow with the corpus. By default the generated vectors are not kept in the
result: CognifyResult.embeddings is empty unless
CognifyConfig::with_retained_embeddings(true) (field retain_embeddings,
default false) is set. The vectors are in the vector store either way, and
CognifyResult.embedding_count reports how many chunk, entity and summary
embeddings the run generated. Code that read embeddings.len() should read
embedding_count instead.
Dates and Timestamp nodes
Dates come out of the default extraction, with no separate temporal pipeline (the event-graph temporal cognify was removed; see Temporal Cognify): a node the LLM typesTimestamp whose name is an absolute time
(1867, 1969-07, 23 March 1947, from 1803 to 1815) becomes a shared
Timestamp graph node — one per normalized time string, carrying
time_at/time_until (ms since the epoch, UTC, half-open) and its
precision — that *_at edges point at and every chunk naming it contains.
Timestamps are not embedded. A name that does not parse stays an ordinary
entity. Year-less dates in the text (“27 April”, “four weeks later”) are
resolved against the last stated year earlier in the document and handed to the
default prompt as normalization hints; a custom prompt receives none.
Entity ids
An extracted entity’s id derives from its name (Entity.id_for(name)), so
the same name from two chunks or documents is one node; several nodes
sharing a name inside one chunk’s graph get deterministic ordinal ids instead.
This is a breaking change: earlier Rust releases keyed entities on the LLM’s
node id, so re-cognifying a dataset built before it adds the name-based entities
beside the old ones. Rebuild such a dataset from scratch to avoid the duplicates.
An extracted node with an empty type gets no EntityType and no is_a edge.
COGNEE_COGNIFY_EXTRACTION_STRATEGY=cascade swaps the single extraction call
for a multi-round cascade (see
configuration).
When a cognify run fails
A chunk or a file the pipeline cannot process is not a bare error: the chunking, graph-extraction and summarization stages collect their failures, and the run result carries a report naming which file, which chunk, which stage and what went wrong.CognifyResult.failures carries it on success;
CognifyError::RunFailed carries the same report when the run failed.
What happens to what the run already wrote is decided at the end, from two
independent settings — when to stop (FailFast, the default, or RunToEnd)
and what to sweep (WholeRun, the default, FailedItems, or Nothing). The
full matrix is in
Cognify failure handling; the end
states are:
- Default (
FailFast+WholeRun). The run stops at the first failed batch — not the first failed file, see below — everything it created is removed, the run is recordedERROREDand the call returnsErr. The store converges to its pre-run state — what earlier runs completed is untouched, because the sweep selects only rows naming this run. FailedItemsbelow the failure ratio. The run completes. Only the failed files’ contributions are removed; the files that finished are kept, indexed and marked complete; and the call returnsOkwith the failed-file list in the report, so the next run knows exactly what to redo.FailedItemsover the ratio escalates to a whole-run sweep and an errored run — a large enough share of failed chunks means something systemic, not one bad file.Nothingremoves nothing. The escape hatch (code only — not selectable from the environment).
FailFast a run partitions its files three ways at the abort point:
complete (every chunk extracted — persisted and marked), failed (at least
one chunk failed) and unreached (never attempted). Only complete files are
persisted, which is what keeps a data item all-or-nothing; failed and unreached
files are indistinguishable to the next run and are simply redone.
The abort point is a batch boundary, not a file boundary, and how much work
FailFast actually saves depends on how big that batch is. Graph extraction
stops after the batch that contained the failure — chunks_per_batch, default
2000, so a dataset under 2000 chunks is one batch and every extraction call is
dispatched before the failure is noticed. Summarization has no batch loop: it stops
dispatching as soon as a fatal failure comes back, but the calls already in
flight (up to max_parallel_extractions) are neither cancelled nor discarded.
FailFast is an upper bound on wasted spend, not a promise of none.
A sweep deliberately keeps anything still claimed from outside its scope: an
entity a surviving file also produced, an artifact another run or another
dataset also names, or a row written before ownership tracking existed. The
worst case is a surplus artifact that keeps an owner, never an artifact with no
owner at all.
That claim is recorded even when the artifact is not written. Graph extraction
filters out edges the graph already holds, so a file cognified in a later run
adds no second copy of an edge an earlier file created — but it still records
ownership of it. Without that row the earlier file would be the edge’s only
owner, and deleting it would take the edge’s vectors away from a file that still
references it. Ownership is per (dataset, data item): an edge shared across two
datasets is still governed by each dataset’s own exclusivity check, the same
limitation entity ownership has.
One caveat. Cancellation is swept — though there is no
caller-facing cancel handle on cognify() today, and simply dropping the future
runs no sweep at all.
For a run that was killed (not failed in-process), see
Serialization and recovery of pipeline runs
and cognee-cli edge-reindex in the CLI Reference.
Re-cognifying is incremental
incremental_loading is on by default and now does what it says. A successful
run marks each data item it finished, in Python’s own pipeline_status format,
and the next run skips the marked items before it builds a pipeline — no
classification, no chunking, no LLM calls.
A failed run marks nothing, and a sweep clears the markers of whatever it rolled
back, so the next run redoes exactly the work that was lost.
The markers live under Python’s one cognify_pipeline key. See
Completion markers and incremental loading.
Related
Add
The step before cognify
Search
Query the graph
Cognify configuration
Chunking, extraction and failure settings