> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognee.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Rust SDK Cognify

> cognify in cognee-rust: turn stored data into a knowledge graph and vectors, with failure handling and incremental re-runs.

## What is the cognify operation

`cognify` turns data stored by [add](https://docs.cognee.ai/rust/core-concepts/main-operations/legacy-operations/add.md) into a knowledge graph plus vector indexes. [remember](https://docs.cognee.ai/rust/core-concepts/main-operations/remember.md) runs it for you.

## What happens under the hood

Turns stored data into a knowledge graph in five stages: **classify** documents →
**chunk** text → **extract** entities/relationships (LLM, batched) **and
summarize** (conditional) — one fused stage running both concurrently over the
same chunks → **add data
points** (six vector collections, plus `DltRow_text` for DLT rows, and provenance to the relational DB) →
**extract DLT FK edges**. A DLT row (a document whose `external_metadata` has
`source: "dlt"`) is one `DltRow` chunk indexed into `DltRow_text`, not
`DocumentChunk_text`; the last step adds its `SchemaTable`, `SchemaRelationship`
and `DltColumn` nodes (vector collections `SchemaTable_name`,
`SchemaTable_description`, `SchemaRelationship_name`,
`SchemaRelationship_description` and `DltColumn_properties`) and the row edges.
Datasets cognified before this layout need a re-cognify after deleting their
old graph and vector rows. Configurable via `CognifyConfig`
(chunk strategy, custom prompts/schemas). Pipeline:
[`cognee-cognify`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/) (`cognify()` / `cognify_datasets()`).

## Writing vectors

The **add data points** stage embeds and writes each vector collection in
windows of about 16 MiB of vector data (`cognee_vector::write_window_points`,
around 2,700 points at 1536 dimensions), so memory on the write path does not
grow with the corpus. By default the generated vectors are **not** kept in the
result: `CognifyResult.embeddings` is empty unless
`CognifyConfig::with_retained_embeddings(true)` (field `retain_embeddings`,
default `false`) is set. The vectors are in the vector store either way, and
`CognifyResult.embedding_count` reports how many chunk, entity and summary
embeddings the run generated. Code that read `embeddings.len()` should read
`embedding_count` instead.

## Dates and Timestamp nodes

Dates come out of the default extraction, with no separate temporal pipeline
(the event-graph temporal cognify was removed; see
[Temporal Cognify](https://docs.cognee.ai/rust/guides/temporal-cognify.md)): a node the LLM types `Timestamp` whose name is an absolute time
(`1867`, `1969-07`, `23 March 1947`, `from 1803 to 1815`) becomes a shared
`Timestamp` graph node — one per normalized time string, carrying
`time_at`/`time_until` (ms since the epoch, UTC, half-open) and its
`precision` — that `*_at` edges point at and every chunk naming it `contains`.
Timestamps are not embedded. A name that does not parse stays an ordinary
entity. Year-less dates in the text ("27 April", "four weeks later") are
resolved against the last stated year earlier in the document and handed to the
default prompt as normalization hints; a custom prompt receives none.

## Entity ids

An extracted entity's id derives from its **name** (`Entity.id_for(name)`), so
the same name from two chunks or documents is one node; several nodes
sharing a name inside one chunk's graph get deterministic ordinal ids instead.
This is a breaking change: earlier Rust releases keyed entities on the LLM's
node id, so re-cognifying a dataset built before it adds the name-based entities
beside the old ones. Rebuild such a dataset from scratch to avoid the duplicates.
An extracted node with an empty type gets no `EntityType` and no `is_a` edge.
`COGNEE_COGNIFY_EXTRACTION_STRATEGY=cascade` swaps the single extraction call
for a multi-round cascade (see
[configuration](https://docs.cognee.ai/rust/configuration/cognify.md#chunking--tokenizer)).

## When a cognify run fails

A chunk or a file the pipeline cannot process is not a bare error: the chunking,
graph-extraction and summarization stages *collect* their failures, and the run
result carries a report naming which file, which chunk, which stage and what
went wrong. `CognifyResult.failures` carries it on success;
`CognifyError::RunFailed` carries the same report when the run failed.

What happens to what the run already wrote is decided at the end, from two
independent settings — *when to stop* (`FailFast`, the default, or `RunToEnd`)
and *what to sweep* (`WholeRun`, the default, `FailedItems`, or `Nothing`). The
full matrix is in
[Cognify failure handling](https://docs.cognee.ai/rust/configuration/cognify.md#cognify-failure-handling); the end
states are:

* **Default (`FailFast` + `WholeRun`).** The run stops at the first failed
  *batch* — not the first failed file, see below — everything it created is
  removed, the run is recorded `ERRORED` and the call returns `Err`. The store
  converges to its pre-run state — what earlier runs completed is untouched,
  because the sweep selects only rows naming *this* run.
* **`FailedItems` below the failure ratio.** The run *completes*. Only the
  failed files' contributions are removed; the files that finished are kept,
  indexed and marked complete; and the call returns `Ok` with the failed-file
  list in the report, so the next run knows exactly what to redo.
* **`FailedItems` over the ratio** escalates to a whole-run sweep and an errored
  run — a large enough share of failed chunks means something systemic, not one
  bad file.
* **`Nothing`** removes nothing. The escape hatch (code only — not selectable
  from the environment).

Under `FailFast` a run partitions its files three ways at the abort point:
*complete* (every chunk extracted — persisted and marked), *failed* (at least
one chunk failed) and *unreached* (never attempted). Only complete files are
persisted, which is what keeps a data item all-or-nothing; failed and unreached
files are indistinguishable to the next run and are simply redone.

The abort point is a *batch* boundary, not a file boundary, and how much work
`FailFast` actually saves depends on how big that batch is. Graph extraction
stops after the batch that contained the failure — `chunks_per_batch`, default
2000, so a dataset under 2000 chunks is one batch and every extraction call is
dispatched before the failure is noticed. Summarization has no batch loop: it stops
dispatching as soon as a fatal failure comes back, but the calls already in
flight (up to `max_parallel_extractions`) are neither cancelled nor discarded.
`FailFast` is an upper bound on wasted spend, not a promise of none.

A sweep deliberately keeps anything still claimed from outside its scope: an
entity a surviving file also produced, an artifact another run or another
dataset also names, or a row written before ownership tracking existed. The
worst case is a surplus artifact that keeps an owner, never an artifact with no
owner at all.

That claim is recorded even when the artifact is not written. Graph extraction
filters out edges the graph already holds, so a file cognified in a later run
adds no second copy of an edge an earlier file created — but it still records
ownership of it. Without that row the earlier file would be the edge's only
owner, and deleting it would take the edge's vectors away from a file that still
references it. Ownership is per (dataset, data item): an edge shared across two
*datasets* is still governed by each dataset's own exclusivity check, the same
limitation entity ownership has.

One caveat. **Cancellation is swept** — though there is no
caller-facing cancel handle on `cognify()` today, and simply dropping the future
runs no sweep at all.

For a run that was *killed* (not failed in-process), see
[Serialization and recovery of pipeline runs](https://docs.cognee.ai/rust/configuration/storage.md#serialization-and-recovery-of-pipeline-runs)
and `cognee-cli edge-reindex` in the [CLI Reference](https://docs.cognee.ai/rust/tools/cli.md).

## Re-cognifying is incremental

`incremental_loading` is on by default and now does what it says. A successful
run marks each data item it finished, in Python's own `pipeline_status` format,
and the next run skips the marked items before it builds a pipeline — no
classification, no chunking, no LLM calls.

> [!WARNING]
>
> **Behaviour change: re-cognifying an already-complete dataset is a no-op**,
> returning `Ok` with `already_completed = true`. Deployments that relied on
> cognify re-processing everything on every call will see it "stop doing
> anything"; set `with_incremental_loading(false)` to restore the previous
> behaviour.

A failed run marks nothing, and a sweep clears the markers of whatever it rolled
back, so the next run redoes exactly the work that was lost.

The markers live under Python's one `cognify_pipeline` key. See
[Completion markers and incremental loading](https://docs.cognee.ai/rust/configuration/cognify.md#completion-markers-and-incremental-loading).

## Related

> [card] [Add](https://docs.cognee.ai/rust/core-concepts/main-operations/legacy-operations/add.md)
>
> The step before cognify

> [card] [Search](https://docs.cognee.ai/rust/core-concepts/main-operations/legacy-operations/search.md)
>
> Query the graph

> [card] [Cognify configuration](https://docs.cognee.ai/rust/configuration/cognify.md)
>
> Chunking, extraction and failure settings


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.