Skip to main content

What are DataPoints

DataPoints are the units that make up memory. Every graph node is a DataPoint, and the vector store indexes them, so they are what cognify writes and what search reads.
  • Atomic: each DataPoint holds one concept or unit of information: a document, a chunk, a summary, an entity, an entity type.
  • Structured: each kind is a plain Rust struct that embeds a shared DataPoint base and is serialized with serde.
  • Contextual: the base carries provenance, versioning and indexing hints, so every later step knows where a node came from and how to use it.
The same id names a DataPoint in all three stores: the graph node, its vector rows and its ownership rows in the relational database.

Core Structure

DataPoint is a struct with a fixed set of standard fields. Typed nodes embed it as their base field.
DataPoint::new(data_type, dataset_id) and DataPoint::with_metadata(data_type, dataset_id, metadata) build one with a random UUID v4, both timestamps set to now, and the defaults above. A Some(dataset_id) is recorded in belongs_to_set as the dataset id’s string.Key fields:
  • id: the identifier shared by the graph node, its vector rows and its ownership rows. The built-in types replace the random default with a deterministic id (see Deduplication).
  • created_at, updated_at: timestamps in milliseconds since the epoch. touch() refreshes updated_at.
  • version: starts at 1. update_version() increments it.
  • topological_rank: the 1-based position of the pipeline task that produced the DataPoint. 0 means unset.
  • metadata: free-form map. The built-in types store their index_fields here.
  • data_type: the type discriminator, serialized as type ("Entity", "DocumentChunk", "TextDocument", …).
  • belongs_to_set: dataset and node-set tags. See Node sets.
  • ontology_valid: set when the node was matched against an ontology.
Some fields are filled in by cognee. Leave them at their defaults:
  • source_pipeline, source_task, source_user, source_node_set, source_content_hash, topological_rank: the pipeline executor stamps them on every Document, DocumentChunk, Entity, EntityType and EdgeType a task returns (bare or in a Vec), only where they are still unset. cognify stamps its TextSummary values itself. See Provenance stamping.
  • importance_weight: copied from the source Data record onto its document, chunks, summaries and extracted entities. Triplet ranking reads it. 0.5 is neutral.
  • feedback_weight: adjusted by improve’s feedback stage and read by triplet ranking. See Feedback weights and Triplet ranking.
A typed node is a struct whose base is flattened into it, so the stored node is one flat object:

How DataPoints work in Rust

A DataPoint is the base storage-layer unit: a structured record that carries a stable UUID, timestamps, a type discriminator, free-form metadata, and provenance fields (source_pipeline, source_task, source_node_set, source_content_hash). Typed graph nodes — Entity, EntityType, EdgeType, DocumentChunk, etc. — embed a DataPoint as their base, exposed through the HasDataPoint trait so provenance stamping can walk any node uniformly. When a DataPoint is indexed, its vector-store payload has the IndexSchema shape (index_schema_payload()): the indexed text, a few reference scalars (document_id, chunk_index, source_chunk_id, …) and the base fields at their IndexSchema defaults, plus the Rust-only dataset_id that dataset-scoped search filters on. LanceDB stores it without the metadata key; pgvector keeps it. (This replaces vector_metadata(); rows written by earlier versions stay readable.) Serialization and versioning helpers are to_json_string() / from_json(), to_dict() / from_dict() and update_version() (bumps version and refreshes updated_at). Rust: DataPoint / HasDataPoint in cognee-models.

Indexing & Embeddings

Each DataPoint type has one or more index fields. Their values are embedded and written to the vector store, which is what semantic search runs against.
  • Each indexed field gets its own vector collection named Type_field: an entity’s name goes to Entity_name, a chunk’s text to DocumentChunk_text.
  • Only the index field’s value is embedded. An entity’s description, for example, is not.
  • Non-indexed fields stay as properties of the graph node.
For the built-in types the indexed fields are fixed by cognify’s add-data-points stage, which writes each collection explicitly. The types also declare them as an INDEX_FIELDS constant (Entity::INDEX_FIELDS == ["name"]) and under metadata["index_fields"]. For a custom type, the index_fields you declare in the schema choose the collections. The vector row is not a copy of the DataPoint. It holds the embedded text, the shared id, a few reference fields and the dataset_id, as described in How DataPoints work in Rust. A write replaces any existing row with the same id.
Cross-store retrieval: a vector hit carries the DataPoint’s id, which is also its graph node id. Search reads the node’s full properties from the graph store, since the vector row holds only the indexed text.
Vectors are written in windows, and are not kept in the cognify result unless you ask for them. See Writing vectors.

From DataPoints to the Graph

cognify writes DataPoints in two stages (both in crates/cognify/src/tasks.rs): graph extraction (extract_graph_from_data) writes the extracted entities and the edges between them, and the add-data-points stage (add_data_points) writes the documents, chunks, entity types and summaries, their structural edges, and every vector collection. Together they:
  • Record in the relational database which data item owns each node and edge, before anything reaches the graph or vector store, so a failed write can be swept and forget can remove them with their document.
  • Write each DataPoint as a graph node, with its fields as node properties.
  • Turn its relationship fields into graph edges.
  • Embed the index fields and write the vector collections.
A relationship field holds the target’s Uuid, not the target struct. Each built-in type declares its outgoing relationships through the GraphExtractable trait, and get_graph_from_model collects them into deduplicated edges named after the field: The relationships the LLM extracts between entities become edges named by their relationship_name. Edges can carry an edge_text sentence. The EdgeType_relationship_name collection embeds each edge’s non-blank edge_text, or its relationship_name when there is none. This is how one unit gives both semantic similarity (vector) and structural reasoning (graph).

Examples and details

belongs_to_set is a tag, not a relationship: it never becomes an edge.
EdgeType rows exist only in the vector store. cognify does not add them to the graph as nodes. Each one is keyed on an edge’s retrieval text: its non-blank edge_text, else its relationship_name.All of these implement HasDataPoint. All but TextSummary are in cognee::models (crates/models/).
The Ladybug and Postgres graph adapters upsert a node by id: a node written with an existing id replaces the stored node’s name, type and properties. The LanceDB and pgvector stores replace a row with the same id too. So when cognify writes an entity whose deterministic id already exists, it updates that node rather than adding a second one.To change a document you added, use update. It drops the document’s memory, re-adds the new content under the same data_id and re-cognifies the dataset.Versioning is manual. Nothing in cognify calls update_version(): a DataPoint you change keeps version at its old value until you call it yourself.
DataPoints are deleted with the data they came from, not one by one. forget removes a data item, a dataset or everything, across the relational, graph and vector stores and file storage, using the ownership rows add-data-points recorded. A node that several data items produced keeps all of them as owners and is removed only with the last one.See What you can forget for the targets, and delete for the lower-level service with a dry-run preview.
The add-data-points stage reads the dataset from its input: the dataset_id, user_id and tenant_id fields of the SummarizedData that the earlier cognify stages fill in for the dataset being processed. There is no separate dataset argument.The dataset is used in two places:
  • Ownership rows: each one records the tenant, user, dataset and pipeline run alongside the node or edge it owns. When no user is given, the rows are recorded under a default user.
  • Vector rows: every row carries dataset_id, which dataset-scoped search filters on. When a row with the same id is indexed again for a second dataset, the vector adapter keeps the union of both datasets in the row’s dataset_ids.
See Datasets for how datasets are named and resolved.
The built-in types do not use the random UUID v4 that DataPoint::new gives. Each one derives a deterministic id, so the same thing written twice lands on the same node:Entity::id_for and EntityType::id_for call data_point_id_for in cognee_utils, which computes uuid5(NAMESPACE_OID, "ClassName:value1|value2|...") with each value lower-cased, spaces turned into _ and apostrophes removed. The class name namespaces the id, so an Entity and an EntityType with the same name never collide.Because an extracted entity’s id comes from its name, the same name in two chunks or two documents is one node. When several nodes in one chunk’s extraction share a name, each gets a chunk-scoped id instead. See Entity ids.Checking whether a DataPoint already existsCompute the id and ask the graph store:
Here graph_db is any GraphDBTrait implementation, such as the Ladybug or Postgres adapter.For a custom type, list the fields that identify it in identity_fields. See the schema contract.
cognify stores only the built-in structs and the types described by a custom graph schema. A struct of your own that embeds DataPoint and implements HasDataPoint is not picked up by the add-data-points stage, and the pipeline executor does not stamp it.To have cognify extract and store your own node types, pass a JSON Schema as CognifyConfig.graph_schema (or a spec with --graph-spec). Each model in it becomes a DataPoint type at runtime, and its metadata default sets the same policy the built-in types have:
  • index_fields: the scalar fields embedded into Model_field collections.
  • identity_fields: the fields that make up the deterministic id.
  • transparent: the model is a container, not a stored node.
Fields that reference another model become edges named after the field. See Custom Graph Schema for the full contract, validation, entry points and the Rust-only FromIdentity keyword.
In a custom graph schema, a model whose metadata default sets "transparent": true exists only to hold other models, for example a root model that wraps a list of entities.Wherever a transparent node appears, it is replaced by its DataPoint children:
  • The container is never stored: no node, and never an edge endpoint.
  • Its children take its place, so a transparent root yields its children as top-level nodes, and a field that points at a transparent node gets edges to that node’s children.
  • Nesting works: a transparent child of a transparent container resolves too.
  • A scalar field on a transparent model has nowhere to be stored. When it holds a value, cognee logs a warning and drops it.
A transparent root needs at least one reachable non-transparent descendant; otherwise the custom graph stores no nodes. See the schema contract.

Tasks

What produces DataPoints

Pipelines

How tasks are chained

Custom graph schema

Define your own node types

Cognify

The pipeline that writes DataPoints

Forget

Delete data and the DataPoints built from it