DataPoints: Atomic Units of Knowledge

DataPoints are the smallest building blocks in Cognee.
They represent atomic units of knowledge — carrying both your actual content and the context needed to process, index, and connect it. They’re the reason Cognee can turn raw documents into something that’s both searchable (via vectors) and connected (via graphs).

What are DataPoints

Atomic — each DataPoint represents one concept or unit of information.
Structured — implemented as Pydantic models for validation and serialization.
Contextual — carry provenance, versioning, and indexing hints so every step downstream knows where data came from and how to use it.

Core Structure

A DataPoint is just a Pydantic model with a set of standard fields.

See example class definition

class DataPoint(BaseModel):
    id: UUID = Field(default_factory=uuid4)
    created_at: int = ...
    updated_at: int = ...
    version: int = 1
    topological_rank: Optional[int] = 0
    valid_to: int | None = None
    metadata: MetaData = {"index_fields": []}
    type: str = "DataPoint"
    ontology_uri: str | None = None
    belongs_to_set: Optional[List["DataPoint"]] = None

Key fields:

id — unique identifier (shared across all three stores, linking vector, graph, and relational records for the same DataPoint)
created_at, updated_at — timestamps (ms since epoch)
version — for tracking changes and schema evolution
topological_rank — an integer indicating the DataPoint’s position in a dependency hierarchy. Lower ranks mean fewer dependencies. For example, an Entity that other DataPoints reference would have a lower rank than a TextSummary that depends on it. Defaults to 0.
valid_to — bi-temporal validity stamp (int | None, default None): the ms-epoch moment at which this fact was superseded. None means the fact is still current. You normally do not set it by hand — close_node() stamps it on the stored node when a fact is replaced, so the old node stays in the graph instead of being deleted. It has no effect on node identity (dedup is still driven by id/identity_fields), and it is distinct from the time_to on Interval nodes (the time range an Event points to via during), which records when an event occurred rather than when a fact stopped being true.
metadata.index_fields — critical: determines which fields are embedded for vector search
type — the Python class name of the DataPoint subclass (e.g., "Person", "Book")
ontology_uri — optional external ontology IRI (str | None, default None). When an entity or type is matched against an ontology — or ingested from RDF — Cognee preserves the stable external IRI here instead of flattening it to a local label, so the node can be linked out to other domains and exported back to RDF. It defaults to None for ungrounded nodes and has no effect on node identity (dedup is still driven by id/identity_fields). See Ontologies → RDF read/write surface.
belongs_to_set — groups related DataPoints

Indexing & Embeddings

The metadata.index_fields tells Cognee which fields to embed into the vector store. This is the mechanism behind semantic search.

Fields in index_fields → converted into embeddings
Each indexed field → its own vector collection named Class_field (e.g., a Person DataPoint with index_fields=["name"] creates a Person_name vector collection). The Class part comes from the Python class name of your DataPoint subclass.
Non-indexed fields → stay as regular properties in the graph and relational stores
Choosing what to index controls search granularity

When you declare several index fields, Cognee deep-copies the DataPoint once per field and narrows the copy’s index_fields to the single field that collection embeds — so each collection holds that field’s own embedding, and the list you declared on your instance is left untouched by indexing.

Cross-store retrieval: When a vector search finds a match, Cognee uses the shared id to retrieve the full DataPoint from the graph store, which holds all properties (not just the indexed field). This is how Cognee returns complete results from a semantic search.

For custom scalar properties such as external IDs, labels, and tags, see Custom Data Models.

From DataPoints to the Graph

When you call add_data_points(), Cognee automatically:

Embeds the indexed fields into vectors
Converts the object into nodes and edges in the knowledge graph
Stores provenance in the relational store

This is how Cognee creates both semantic similarity (vector) and structural reasoning (graph) from the same unit.

Examples and details

Example: indexing only one field

from typing import Annotated
from cognee.infrastructure.engine import DataPoint, Embeddable

class Person(DataPoint):
    name: Annotated[str, Embeddable()]
    age: int

Only "name" is semantically searchable

Example: Book → Author transformation

from typing import Annotated
from cognee.infrastructure.engine import DataPoint, Embeddable

class Book(DataPoint):
    title: Annotated[str, Embeddable()]
    author: Author

# Produces:
# `Node(Book)` with `{title, type, ...}`
# Node(Author) with {name, type, ...}
# Edge(Book → Author, type="author")

Relationship syntax options

# Simple relationship
`author: Author`  

# With edge metadata
`has_items: (Edge(weight=0.8), list[Item])`

# List relationship
`chapters: list[Chapter]`

# Typed edge between siblings — flat relationship rows, not nesting
`cites: list[Edge[Book, Book]]`

# Reference an existing node by its identity value instead of nesting it
`genre: Annotated[Genre, FromIdentity()] | None`

The first three forms nest one DataPoint inside another. Edge[Source, Target] and FromIdentity() instead address nodes by their identity value; both come from cognee.low_level and are Python-SDK-only. See Custom Graph Model.

Built-in DataPoint types

Cognee ships with several built-in DataPoint types:

Documents — wrappers for source files (Text, PDF, Audio, Image)
- Document (metadata.index_fields=["name"])
Chunks — segmented portions of documents
- DocumentChunk (metadata.index_fields=["text"])
Summaries — generated text or code summaries
- TextSummary / CodeSummary / GlobalContextSummary (metadata.index_fields=["text"])
- GlobalContextSummary powers the Global Context Index
Entities — named objects (people, places, concepts)
- Entity, EntityType (metadata.index_fields=["name"])
Edges — relationships between DataPoints
- Edge — links between DataPoints

Example: custom DataPoint with best practices

from typing import Annotated
from cognee.infrastructure.engine import DataPoint, Embeddable

class Product(DataPoint):
    name: Annotated[str, Embeddable()]
    description: Annotated[str, Embeddable()]
    price: float
    category: Category

Best Practices:

Keep it small — one concept per DataPoint
Index carefully — only fields that matter for semantic search
Use built-in types first — extend with custom subclasses when needed
Version deliberately — track changes with version
Group related points — with belongs_to_set

Updating DataPoints

To update a custom DataPoint, mutate its fields, call update_version() to record the change, then re-add it with add_data_points(). The upsert replaces the existing node in all three stores.

from cognee.infrastructure.engine import DataPoint
from cognee.tasks.storage import add_data_points

product = Product(name="Widget", price=9.99, description="Original")
await add_data_points([product])

# Later — update the description
product.description = "Improved description"
product.update_version()   # version → 2, updated_at refreshed
await add_data_points([product])

For documents and files remembered via cognee.remember(), use cognee.update() instead — it replaces the existing item in the target dataset under the same data_id, re-extracting only the chunks the edit touched and falling back to a full delete, re-add and graph rebuild for that dataset when the chunk-level preconditions are not met.How versioning worksChanging a field on a DataPoint does not automatically create a new revision or persist anything by itself. In other words, versioning is manual:

Edit the DataPoint fields
Call update_version() if this should count as a new revision
Re-add the DataPoint with add_data_points() to persist the updated state

By calling update_version(), you mark your in-memory object as a new revision before writing it back with add_data_points(). It does two things:

Increments version by 1. New DataPoints start at version=1.
Sets updated_at to the current UTC timestamp in milliseconds.

Deleting DataPoints

Use the cognee.datasets API:

import cognee

# Soft-delete one item by ID (default — marks as deleted)
await cognee.datasets.delete_data(dataset_id=ds.id, data_id=item.id)

# Hard-delete to remove from all stores permanently
await cognee.datasets.delete_data(dataset_id=ds.id, data_id=item.id, mode="hard")

# Remove all items from a dataset (keeps the dataset itself)
await cognee.datasets.empty_dataset(dataset_id=ds.id)

# Delete all datasets for the current user
await cognee.datasets.delete_all()

Dataset routing: which dataset receives add_data_points output?

add_data_points() does not accept a dataset parameter directly. Dataset assignment is carried by the PipelineContext (ctx) that Cognee injects automatically when your task runs inside run_pipeline.

async for _ in run_pipeline(
    tasks=[my_task],          # task calls add_data_points internally
    data=my_data,
    datasets=["my_dataset"],  # <-- this sets ctx.dataset
):
    pass

Inside the task, ctx.dataset holds the resolved dataset object. add_data_points uses it to write provenance records (user, dataset, data item) to the relational store.If you call add_data_points outside a pipeline (without a ctx), nodes and edges are still written to the graph and vector stores, but no dataset-level provenance is recorded — the data is not associated with any named dataset.

embed_triplets: graph-structure embeddings

add_data_points accepts an embed_triplets: bool = False parameter. When set to True, Cognee derives (subject → predicate → object) triplets from the graph edges and indexes each one as a Triplet DataPoint embedding.

await add_data_points([product], embed_triplets=True)

Each triplet is embedded as a single text string in the form:

<subject text> -› <predicate> -› <object text>

These triplets are derived from the graph you just wrote, but they are not added back as extra graph nodes or edges.This allows vector search to match not just individual nodes, but relationship patterns across the graph.Use embed_triplets=True when:

Your queries describe relationships (e.g., “products made by company X”)
You want to retrieve graph edges via semantic similarity, not just individual nodes

Leave it False (the default) for standard node-level retrieval.For the same idea applied after graph creation, see Triplet Embeddings.

Deduplication: preventing duplicate entities

By default, each DataPoint receives a random UUID4 on instantiation. To make identical entities share the same node — and avoid duplicates — mark one or more fields as the deduplication key.

Option 1: Dedup()
Option 2: identity_fields

from typing import Annotated
from cognee.infrastructure.engine import DataPoint, Embeddable, Dedup

class Product(DataPoint):
    sku: Annotated[str, Dedup()]
    name: Annotated[str, Embeddable()]
    price: float

Cognee automatically populates identity_fields from the Dedup() annotations. No explicit metadata declaration is needed.

from cognee.infrastructure.engine import DataPoint
from cognee.infrastructure.engine.models.DataPoint import MetaData

class Product(DataPoint):
    sku: str
    name: str
    price: float
    metadata: MetaData = {"index_fields": ["name"], "identity_fields": ["sku"]}

How it worksWhen one or more identity fields are defined and all of them resolve on the instance, Cognee generates a deterministic UUID5 from the class name and the field values instead of a random UUID4. Concretely, the id is uuid5(NAMESPACE_OID, "ClassName:value1|value2|..."), with each value lower-cased and its spaces normalized to _ — so the class name namespaces the id and two different node types never collide on the same input. Two Product instances with the same sku produce the same id, so add_data_points upserts them onto the same graph node rather than creating a duplicate. If an identity field is missing and has no default, Cognee falls back to a random UUID4.You can recompute this id at any time without building an instance via Product.id_for(sku_value) — it returns the same UUID5 the instance would get, which is useful for lookups.Using an external system’s IDIf your records already have a stable identifier from an external system (for example a CMS entry UUID), pass it directly as id. An explicit id always takes precedence over Dedup() / identity_fields generation, so that external identifier is what sits on the graph node:

from uuid import UUID

# external_uuid comes from your source system (must be a valid UUID)
product = Product(id=UUID(external_uuid), sku="ABC-123", name="Widget", price=9.99)
await add_data_points([product])

Re-adding an object with the same id upserts onto the existing node, so syncing an update from the source system overwrites the node in place instead of creating a duplicate. (id must be a UUID — if your external key is not already a UUID, either normalize it into one or feed it through Dedup() / identity_fields so Cognee derives a deterministic UUID5 from it.)Checking whether a DataPoint already existsBecause the ID is deterministic, you can check existence by constructing the instance (which generates the identity ID) and querying the graph engine:

from cognee.infrastructure.databases.graph import get_graph_engine

candidate = Product(sku="ABC-123", name="Widget", price=9.99)

graph = await get_graph_engine()
existing = await graph.get_node(str(candidate.id))
if existing:
    print("Already in the graph:", existing)
else:
    print("New entity — id:", candidate.id)

Typing the metadata field (avoiding type-checker warnings)

The base metadata field is typed as MetaData, a TypedDict — not a plain dict. Annotating a subclass’s metadata as dict (e.g. metadata: dict = {...}) narrows the type incompatibly, which makes type checkers like Pylance warn that the override conflicts with the base field. Annotate it with MetaData instead:

from cognee.infrastructure.engine import DataPoint
from cognee.infrastructure.engine.models.DataPoint import MetaData

class Product(DataPoint):
    name: str
    description: str
    metadata: MetaData = {"index_fields": ["name", "description"]}

MetaData is defined in cognee.infrastructure.engine.models.DataPoint and must be imported from that path — it is not re-exported from cognee.low_level or cognee.infrastructure.engine, so from cognee.low_level import MetaData fails.The TypedDict accepts four keys:

index_fields: list[str] — required; fields embedded for vector search
type: str — optional
identity_fields: list[str] — optional; the deduplication key
transparent: bool — optional; marks the class as a container that is never stored

Prefer the Embeddable() / Dedup() annotations when you can — Cognee derives metadata from them automatically, so you never declare the field (or its type) by hand.

Defining a DataPoint type at runtime (instead of writing a class)

A DataPoint type is always a Pydantic class, and its fields must be declared. add_data_points rejects anything that is not a DataPoint instance, so a plain dict of values raises InvalidDataPointsInAddDataPointsError, and a value passed for a field the class does not declare is silently dropped — Cognee reads only declared fields when it converts an object into nodes and edges, so that value never reaches the stores.You do not have to write that class by hand, though. Because DataPoint is an ordinary Pydantic model, you can build the type at runtime with Pydantic’s create_model and __base__=DataPoint:

from typing import Annotated
from pydantic import create_model
from cognee.infrastructure.engine import DataPoint, Dedup, Embeddable
from cognee.tasks.storage import add_data_points

fields = {
    "sku": (Annotated[str, Dedup()], ...),
    "name": (Annotated[str, Embeddable()], ...),
    "price": (float, 0.0),
}
Product = create_model("Product", __base__=DataPoint, **fields)

await add_data_points([Product(sku="ABC-123", name="Widget", price=9.99)])

Such a class behaves exactly like a written one: Embeddable() and Dedup() are picked up, Product.id_for("ABC-123") recomputes the same deterministic id, and a field typed as another DataPoint becomes an edge.Two things to watch:

The name you pass is the class name. It plays that role everywhere this page mentions it — the node type, the Class_field vector collections, and the deduplication id namespace. Build each shape once and reuse the class — a new name means new collections and new ids for the same records.
index_fields must be a class-level default. If you declare metadata explicitly instead of using the annotations, pass it as a field definition ("metadata": (MetaData, {"index_fields": ["name"]}), importing MetaData as described above). Setting metadata on an instance does not reach storage: Cognee rebuilds the node from the class field defaults, so instance-level index fields are lost and nothing is embedded.

Transparent containers: nodes that group rather than describe

Some DataPoint classes exist only to hold other DataPoints — a wrapper you pass as graph_model so the LLM returns a list of entities, for example. Set metadata["transparent"] = True to declare that the class is structure rather than content:

from typing import List
from cognee.infrastructure.engine import DataPoint
from cognee.infrastructure.engine.models.DataPoint import MetaData

class Person(DataPoint):
    name: str
    metadata: MetaData = {"index_fields": ["name"], "identity_fields": ["name"]}

class PeopleGraph(DataPoint):
    people: List[Person]
    metadata: MetaData = {"index_fields": [], "transparent": True}

Wherever a transparent node appears, it is replaced by its DataPoint children:

The wrapper itself is never stored — no node, and never an edge endpoint. Without transparent, PeopleGraph becomes a node with a people edge to each Person.
Its DataPoint children are promoted to top-level roots, so one extraction can produce several top-level nodes, or none at all if the wrapper came back empty.
Nesting works: a transparent child of a transparent wrapper resolves too, and a field pointing at a transparent node gets edges to that node’s children instead.
belongs_to_set is not inherited by the children — a wrapper’s NodeSets are not its content.

Non-DataPoint fields on a transparent class are dropped. Once the wrapper is gone, a plain scalar declared on it has nowhere to live. Rather than disappearing silently, each one logs a warning the first time it carries a value (once per class field):

PeopleGraph is marked transparent but carries data in 'title'; a transparent node is never
stored, so that value is dropped. If the field is worth searching for, the class is not a
container - remove metadata['transparent'].

Fields inherited from DataPoint itself and empty values are exempt, so an unset optional relationship never warns.

Transparency is a property of the model. How widely a chunk links into the graph extracted from it is a property of the run — see cognify(chunk_attachment=...). Under "direct" (the default), a chunk whose extracted root is transparent links to the children that replaced it.

Tasks

Learn how DataPoints are created and processed

Pipelines

See how DataPoints flow through processing workflows

Main Operations

Understand how DataPoints are used in Remember, Improve, and Recall

Getting Started

Core Concepts

Setup Configuration

Guides

Examples

CLI

OSS

DataPoints

DataPoints: Atomic Units of Knowledge

What are DataPoints

Core Structure

Indexing & Embeddings

From DataPoints to the Graph

Examples and details

Tasks

Pipelines

Main Operations

​DataPoints: Atomic Units of Knowledge

​What are DataPoints

​Core Structure

​Indexing & Embeddings

​From DataPoints to the Graph

​Examples and details

Tasks

Pipelines

Main Operations

DataPoints: Atomic Units of Knowledge

What are DataPoints

Core Structure

Indexing & Embeddings

From DataPoints to the Graph

Examples and details