What are DataPoints
DataPoints are the units that make up memory. Every graph node is a DataPoint, and the vector store indexes them, so they are what cognify writes and what search reads.- Atomic: each DataPoint holds one concept or unit of information: a document, a chunk, a summary, an entity, an entity type.
- Structured: each kind is a plain Rust struct that embeds a shared
DataPointbase and is serialized with serde. - Contextual: the base carries provenance, versioning and indexing hints, so every later step knows where a node came from and how to use it.
id names a DataPoint in all three stores: the graph node, its vector rows and its ownership rows in the relational database.
Core Structure
DataPoint is a struct with a fixed set of standard fields. Typed nodes embed it as their base field.
See the struct definition
See the struct definition
DataPoint::new(data_type, dataset_id) and DataPoint::with_metadata(data_type, dataset_id, metadata) build one with a random UUID v4, both timestamps set to now, and the defaults above. A Some(dataset_id) is recorded in belongs_to_set as the dataset id’s string.Key fields:id: the identifier shared by the graph node, its vector rows and its ownership rows. The built-in types replace the random default with a deterministic id (see Deduplication).created_at,updated_at: timestamps in milliseconds since the epoch.touch()refreshesupdated_at.version: starts at1.update_version()increments it.topological_rank: the 1-based position of the pipeline task that produced the DataPoint.0means unset.metadata: free-form map. The built-in types store theirindex_fieldshere.data_type: the type discriminator, serialized astype("Entity","DocumentChunk","TextDocument", …).belongs_to_set: dataset and node-set tags. See Node sets.ontology_valid: set when the node was matched against an ontology.
source_pipeline,source_task,source_user,source_node_set,source_content_hash,topological_rank: the pipeline executor stamps them on everyDocument,DocumentChunk,Entity,EntityTypeandEdgeTypea task returns (bare or in aVec), only where they are still unset. cognify stamps itsTextSummaryvalues itself. See Provenance stamping.importance_weight: copied from the sourceDatarecord onto its document, chunks, summaries and extracted entities. Triplet ranking reads it.0.5is neutral.feedback_weight: adjusted by improve’s feedback stage and read by triplet ranking. See Feedback weights and Triplet ranking.
base is flattened into it, so the stored node is one flat object:
How DataPoints work in Rust
A DataPoint is the base storage-layer unit: a structured record that carries a stable UUID, timestamps, atype discriminator, free-form metadata, and
provenance fields (source_pipeline, source_task, source_node_set,
source_content_hash). Typed graph nodes — Entity, EntityType, EdgeType,
DocumentChunk, etc. — embed a DataPoint as their base, exposed through the
HasDataPoint trait so provenance stamping can walk any node uniformly. When a
DataPoint is indexed, its vector-store payload has the IndexSchema
shape (index_schema_payload()): the indexed text, a few reference
scalars (document_id, chunk_index, source_chunk_id, …) and the base fields
at their IndexSchema defaults, plus the Rust-only dataset_id that
dataset-scoped search filters on. LanceDB stores it without the metadata key;
pgvector keeps it. (This replaces
vector_metadata(); rows written by earlier versions stay readable.)
Serialization and versioning helpers are to_json_string()
/ from_json(), to_dict() / from_dict() and update_version() (bumps
version and refreshes updated_at).
Rust: DataPoint / HasDataPoint in cognee-models.
Indexing & Embeddings
Each DataPoint type has one or more index fields. Their values are embedded and written to the vector store, which is what semantic search runs against.- Each indexed field gets its own vector collection named
Type_field: an entity’snamegoes toEntity_name, a chunk’stexttoDocumentChunk_text. - Only the index field’s value is embedded. An entity’s
description, for example, is not. - Non-indexed fields stay as properties of the graph node.
INDEX_FIELDS constant (Entity::INDEX_FIELDS == ["name"]) and under metadata["index_fields"]. For a custom type, the index_fields you declare in the schema choose the collections.
The vector row is not a copy of the DataPoint. It holds the embedded text, the shared id, a few reference fields and the dataset_id, as described in How DataPoints work in Rust. A write replaces any existing row with the same id.
Cross-store retrieval: a vector hit carries the DataPoint’s
id, which is also its graph node id. Search reads the node’s full properties from the graph store, since the vector row holds only the indexed text.From DataPoints to the Graph
cognify writes DataPoints in two stages (both incrates/cognify/src/tasks.rs): graph extraction (extract_graph_from_data) writes the extracted entities and the edges between them, and the add-data-points stage (add_data_points) writes the documents, chunks, entity types and summaries, their structural edges, and every vector collection. Together they:
- Record in the relational database which data item owns each node and edge, before anything reaches the graph or vector store, so a failed write can be swept and
forgetcan remove them with their document. - Write each DataPoint as a graph node, with its fields as node properties.
- Turn its relationship fields into graph edges.
- Embed the index fields and write the vector collections.
Uuid, not the target struct. Each built-in type declares its outgoing relationships through the GraphExtractable trait, and get_graph_from_model collects them into deduplicated edges named after the field:
The relationships the LLM extracts between entities become edges named by their
relationship_name. Edges can carry an edge_text sentence. The EdgeType_relationship_name collection embeds each edge’s non-blank edge_text, or its relationship_name when there is none.
This is how one unit gives both semantic similarity (vector) and structural reasoning (graph).
Examples and details
Example: relationship fields
Example: relationship fields
belongs_to_set is a tag, not a relationship: it never becomes an edge.Built-in DataPoint types
Built-in DataPoint types
EdgeType rows exist only in the vector store. cognify does not add them to the graph as nodes. Each one is keyed on an edge’s retrieval text: its non-blank edge_text, else its relationship_name.All of these implement HasDataPoint. All but TextSummary are in cognee::models (crates/models/).Updating DataPoints
Updating DataPoints
The Ladybug and Postgres graph adapters upsert a node by
id: a node written with an existing id replaces the stored node’s name, type and properties. The LanceDB and pgvector stores replace a row with the same id too. So when cognify writes an entity whose deterministic id already exists, it updates that node rather than adding a second one.To change a document you added, use update. It drops the document’s memory, re-adds the new content under the same data_id and re-cognifies the dataset.Versioning is manual. Nothing in cognify calls update_version(): a DataPoint you change keeps version at its old value until you call it yourself.Deleting DataPoints
Deleting DataPoints
DataPoints are deleted with the data they came from, not one by one.
forget removes a data item, a dataset or everything, across the relational, graph and vector stores and file storage, using the ownership rows add-data-points recorded. A node that several data items produced keeps all of them as owners and is removed only with the last one.See What you can forget for the targets, and delete for the lower-level service with a dry-run preview.Dataset routing: which dataset receives the DataPoints?
Dataset routing: which dataset receives the DataPoints?
The add-data-points stage reads the dataset from its input: the
dataset_id, user_id and tenant_id fields of the SummarizedData that the earlier cognify stages fill in for the dataset being processed. There is no separate dataset argument.The dataset is used in two places:- Ownership rows: each one records the tenant, user, dataset and pipeline run alongside the node or edge it owns. When no user is given, the rows are recorded under a default user.
- Vector rows: every row carries
dataset_id, which dataset-scoped search filters on. When a row with the sameidis indexed again for a second dataset, the vector adapter keeps the union of both datasets in the row’sdataset_ids.
Deduplication: stable ids
Deduplication: stable ids
The built-in types do not use the random UUID v4 that Here
DataPoint::new gives. Each one derives a deterministic id, so the same thing written twice lands on the same node:Entity::id_for and EntityType::id_for call data_point_id_for in cognee_utils, which computes uuid5(NAMESPACE_OID, "ClassName:value1|value2|...") with each value lower-cased, spaces turned into _ and apostrophes removed. The class name namespaces the id, so an Entity and an EntityType with the same name never collide.Because an extracted entity’s id comes from its name, the same name in two chunks or two documents is one node. When several nodes in one chunk’s extraction share a name, each gets a chunk-scoped id instead. See Entity ids.Checking whether a DataPoint already existsCompute the id and ask the graph store:graph_db is any GraphDBTrait implementation, such as the Ladybug or Postgres adapter.For a custom type, list the fields that identify it in identity_fields. See the schema contract.Custom DataPoint types
Custom DataPoint types
cognify stores only the built-in structs and the types described by a custom graph schema. A struct of your own that embeds
DataPoint and implements HasDataPoint is not picked up by the add-data-points stage, and the pipeline executor does not stamp it.To have cognify extract and store your own node types, pass a JSON Schema as CognifyConfig.graph_schema (or a spec with --graph-spec). Each model in it becomes a DataPoint type at runtime, and its metadata default sets the same policy the built-in types have:index_fields: the scalar fields embedded intoModel_fieldcollections.identity_fields: the fields that make up the deterministic id.transparent: the model is a container, not a stored node.
FromIdentity keyword.Transparent containers: nodes that group rather than describe
Transparent containers: nodes that group rather than describe
In a custom graph schema, a model whose
metadata default sets "transparent": true exists only to hold other models, for example a root model that wraps a list of entities.Wherever a transparent node appears, it is replaced by its DataPoint children:- The container is never stored: no node, and never an edge endpoint.
- Its children take its place, so a transparent root yields its children as top-level nodes, and a field that points at a transparent node gets edges to that node’s children.
- Nesting works: a transparent child of a transparent container resolves too.
- A scalar field on a transparent model has nowhere to be stored. When it holds a value, cognee logs a warning and drops it.
Related
Tasks
What produces DataPoints
Pipelines
How tasks are chained
Custom graph schema
Define your own node types
Cognify
The pipeline that writes DataPoints
Forget
Delete data and the DataPoints built from it