Skip to main content

What is the cognify operation

cognify turns data stored by add into a knowledge graph plus vector indexes. remember runs it for you.

What happens under the hood

Turns stored data into a knowledge graph in five stages: classify documents → chunk text → extract entities/relationships (LLM, batched) and summarize (conditional) — one fused stage running both concurrently over the same chunks → add data points (six vector collections, plus DltRow_text for DLT rows, and provenance to the relational DB) → extract DLT FK edges. A DLT row (a document whose external_metadata has source: "dlt") is one DltRow chunk indexed into DltRow_text, not DocumentChunk_text; the last step adds its SchemaTable, SchemaRelationship and DltColumn nodes (vector collections SchemaTable_name, SchemaTable_description, SchemaRelationship_name, SchemaRelationship_description and DltColumn_properties) and the row edges. Datasets cognified before this layout need a re-cognify after deleting their old graph and vector rows. Configurable via CognifyConfig (chunk strategy, custom prompts/schemas). Pipeline: cognee-cognify (cognify() / cognify_datasets()).

Writing vectors

The add data points stage embeds and writes each vector collection in windows of about 16 MiB of vector data (cognee_vector::write_window_points, around 2,700 points at 1536 dimensions), so memory on the write path does not grow with the corpus. By default the generated vectors are not kept in the result: CognifyResult.embeddings is empty unless CognifyConfig::with_retained_embeddings(true) (field retain_embeddings, default false) is set. The vectors are in the vector store either way, and CognifyResult.embedding_count reports how many chunk, entity and summary embeddings the run generated. Code that read embeddings.len() should read embedding_count instead.

Dates and Timestamp nodes

Dates come out of the default extraction, with no separate temporal pipeline (the event-graph temporal cognify was removed; see Temporal Cognify): a node the LLM types Timestamp whose name is an absolute time (1867, 1969-07, 23 March 1947, from 1803 to 1815) becomes a shared Timestamp graph node — one per normalized time string, carrying time_at/time_until (ms since the epoch, UTC, half-open) and its precision — that *_at edges point at and every chunk naming it contains. Timestamps are not embedded. A name that does not parse stays an ordinary entity. Year-less dates in the text (“27 April”, “four weeks later”) are resolved against the last stated year earlier in the document and handed to the default prompt as normalization hints; a custom prompt receives none.

Entity ids

An extracted entity’s id derives from its name (Entity.id_for(name)), so the same name from two chunks or documents is one node; several nodes sharing a name inside one chunk’s graph get deterministic ordinal ids instead. This is a breaking change: earlier Rust releases keyed entities on the LLM’s node id, so re-cognifying a dataset built before it adds the name-based entities beside the old ones. Rebuild such a dataset from scratch to avoid the duplicates. An extracted node with an empty type gets no EntityType and no is_a edge. COGNEE_COGNIFY_EXTRACTION_STRATEGY=cascade swaps the single extraction call for a multi-round cascade (see configuration).

When a cognify run fails

A chunk or a file the pipeline cannot process is not a bare error: the chunking, graph-extraction and summarization stages collect their failures, and the run result carries a report naming which file, which chunk, which stage and what went wrong. CognifyResult.failures carries it on success; CognifyError::RunFailed carries the same report when the run failed. What happens to what the run already wrote is decided at the end, from two independent settings — when to stop (FailFast, the default, or RunToEnd) and what to sweep (WholeRun, the default, FailedItems, or Nothing). The full matrix is in Cognify failure handling; the end states are:
  • Default (FailFast + WholeRun). The run stops at the first failed batch — not the first failed file, see below — everything it created is removed, the run is recorded ERRORED and the call returns Err. The store converges to its pre-run state — what earlier runs completed is untouched, because the sweep selects only rows naming this run.
  • FailedItems below the failure ratio. The run completes. Only the failed files’ contributions are removed; the files that finished are kept, indexed and marked complete; and the call returns Ok with the failed-file list in the report, so the next run knows exactly what to redo.
  • FailedItems over the ratio escalates to a whole-run sweep and an errored run — a large enough share of failed chunks means something systemic, not one bad file.
  • Nothing removes nothing. The escape hatch (code only — not selectable from the environment).
Under FailFast a run partitions its files three ways at the abort point: complete (every chunk extracted — persisted and marked), failed (at least one chunk failed) and unreached (never attempted). Only complete files are persisted, which is what keeps a data item all-or-nothing; failed and unreached files are indistinguishable to the next run and are simply redone. The abort point is a batch boundary, not a file boundary, and how much work FailFast actually saves depends on how big that batch is. Graph extraction stops after the batch that contained the failure — chunks_per_batch, default 2000, so a dataset under 2000 chunks is one batch and every extraction call is dispatched before the failure is noticed. Summarization has no batch loop: it stops dispatching as soon as a fatal failure comes back, but the calls already in flight (up to max_parallel_extractions) are neither cancelled nor discarded. FailFast is an upper bound on wasted spend, not a promise of none. A sweep deliberately keeps anything still claimed from outside its scope: an entity a surviving file also produced, an artifact another run or another dataset also names, or a row written before ownership tracking existed. The worst case is a surplus artifact that keeps an owner, never an artifact with no owner at all. That claim is recorded even when the artifact is not written. Graph extraction filters out edges the graph already holds, so a file cognified in a later run adds no second copy of an edge an earlier file created — but it still records ownership of it. Without that row the earlier file would be the edge’s only owner, and deleting it would take the edge’s vectors away from a file that still references it. Ownership is per (dataset, data item): an edge shared across two datasets is still governed by each dataset’s own exclusivity check, the same limitation entity ownership has. One caveat. Cancellation is swept — though there is no caller-facing cancel handle on cognify() today, and simply dropping the future runs no sweep at all. For a run that was killed (not failed in-process), see Serialization and recovery of pipeline runs and cognee-cli edge-reindex in the CLI Reference.

Re-cognifying is incremental

incremental_loading is on by default and now does what it says. A successful run marks each data item it finished, in Python’s own pipeline_status format, and the next run skips the marked items before it builds a pipeline — no classification, no chunking, no LLM calls.
Behaviour change: re-cognifying an already-complete dataset is a no-op, returning Ok with already_completed = true. Deployments that relied on cognify re-processing everything on every call will see it “stop doing anything”; set with_incremental_loading(false) to restore the previous behaviour.
A failed run marks nothing, and a sweep clears the markers of whatever it rolled back, so the next run redoes exactly the work that was lost. The markers live under Python’s one cognify_pipeline key. See Completion markers and incremental loading.

Add

The step before cognify

Search

Query the graph

Cognify configuration

Chunking, extraction and failure settings