Entity Deduplication

A minimal guide to merging near-duplicate Entity nodes. When the same real-world thing is extracted under several names (New York City and NYC), it becomes several Entity nodes and its edges are split across all of them — the consolidate_entities Memify pipeline detects those near-duplicates and collapses each group into a single canonical node.

This pipeline is destructive. Duplicate Entity nodes are deleted from the graph and their name embeddings are deleted from the vector store. Run it with dry_run=True first and back up your graph and vector stores before running it for real.

Before You Start

Complete Quickstart to understand basic operations
Ensure you have LLM Providers configured — the script below builds a graph with remember()
Ensure you have Embedding Providers configured — detection embeds every entity name at run time
Have an existing knowledge graph with Entity nodes, or let the script create one
The acting user needs write access to the target dataset. See Permissions

Code in Action

import asyncio
import cognee

from os import path
from cognee.api.v1.visualize.visualize import visualize_graph
from cognee.memify_pipelines.consolidate_entities import consolidate_entities_pipeline

custom_prompt = """
Extract every place mentioned in the text as an entity, keeping the exact
surface form used in the text (so "NYC" stays "NYC").
Connect people to places with the relationship "visited".
Ignore all other entities.
"""


async def main():
    # Prune data and system metadata before running, only if we want "fresh" state.
    await cognee.forget(everything=True)
    # Ingest the two texts separately: extracted together, the LLM resolves the
    # abbreviation and emits a single entity, leaving nothing to merge.
    await cognee.remember(
        "Sara visited New York City last spring.",
        custom_prompt=custom_prompt,
        self_improvement=False,
    )
    await cognee.remember(
        "Bob thinks NYC has the best bagels.",
        custom_prompt=custom_prompt,
        self_improvement=False,
    )

    await visualize_graph(
        path.join(path.dirname(__file__), ".artifacts", "before_entity_deduplication.html")
    )

    # Preview the merge plan in the logs without touching the graph.
    await consolidate_entities_pipeline(similarity_threshold=0.6, dry_run=True)

    # Apply the merge for real.
    await consolidate_entities_pipeline(similarity_threshold=0.6)

    await visualize_graph(
        path.join(path.dirname(__file__), ".artifacts", "after_entity_deduplication.html")
    )


if __name__ == "__main__":
    asyncio.run(main())

What Just Happened

Step 1: Build a Graph with Duplicates

await cognee.forget(everything=True)
await cognee.remember(
    "Sara visited New York City last spring.",
    custom_prompt=custom_prompt,
    self_improvement=False,
)
await cognee.remember(
    "Bob thinks NYC has the best bagels.",
    custom_prompt=custom_prompt,
    self_improvement=False,
)

await visualize_graph(
    path.join(path.dirname(__file__), ".artifacts", "before_entity_deduplication.html")
)

Start from a clean state, then ingest two texts that mention the same city under different names — in two separate remember() calls. That separation matters: when the texts are ingested together, the LLM sees both mentions at once, resolves the abbreviation during extraction, and emits a single entity, leaving nothing to merge. Ingested independently, the graph ends up with separate New York City and NYC entities — visible in the before visualization — each holding only its own edges.

Step 2: Preview the Merge Plan

await consolidate_entities_pipeline(similarity_threshold=0.6, dry_run=True)

dry_run=True runs detection and computes the full merge plan, then logs it — which clusters were found, which node becomes canonical, and how many edges would be re-pointed — without touching the graph. The threshold is lowered from the 0.85 default because an abbreviation like NYC is not embedding-close enough to New York City to clear it.

Step 3: Apply the Merge

await consolidate_entities_pipeline(similarity_threshold=0.6)

await visualize_graph(
    path.join(path.dirname(__file__), ".artifacts", "after_entity_deduplication.html")
)

Once the plan looks right, drop dry_run to execute it. The after visualization shows a single canonical city node carrying the edges from both duplicates.

Options

consolidate_entities_pipeline() takes the following arguments:

Argument	Type	Default	Description
`similarity_threshold`	`float`	`0.85`	Minimum cosine similarity between two entity-name embeddings for them to be treated as the same entity. Must be in the range `(0, 1]`.
`dry_run`	`bool`	`False`	When `True`, compute and log the merge plan but perform zero mutations.
`protect_node_types`	`Optional[List[str]]`	`None`	`EntityType` names that must never be merged. Must be a list of non-empty strings.
`name_match`	`bool`	`True`	Also merge entities whose normalized names are identical, where normalizing means lower-casing and stripping every non-alphanumeric character (so `U.S.A.` matches `USA`). This pass is not bounded by `top_k`.
`top_k`	`int`	`10`	Max neighbors considered per entity during similarity clustering. Must be a positive integer.
`allow_cross_type`	`bool`	`False`	When `False` (the default, conservative choice), only entities sharing the same `EntityType` are merged together. Set `True` to allow merging across types.
`user`	`Optional[User]`	`None`	Acting user. The default user is used when omitted.
`dataset`	`str`	`"main_dataset"`	Dataset name or id whose graph to consolidate.
`run_in_background`	`bool`	`False`	Forwarded to `memify`.

Arguments are validated before anything runs. A CogneeValidationError is raised when similarity_threshold is outside (0, 1], when top_k is not a positive integer, when protect_node_types is not a list of non-empty strings, or when the user has no write access to the requested dataset.

What the Merge Changes

Each detected cluster collapses into one canonical Entity node; the rest are deleted. The canonical is the most connected member of the cluster, with ties broken by oldest created_at and then by normalized name.
The canonical keeps its existing graph id, so anything already pointing at it stays valid.
Every edge on a duplicate is re-pointed onto the canonical with its direction preserved. Edges that would become a self-loop on the canonical are dropped.
The canonical’s description becomes the union of the distinct, non-empty descriptions across the cluster.
The canonical’s belongs_to_set becomes the union of every member’s tags, so a merge never narrows a node’s dataset / node-set scoping.
Duplicate nodes are detach-deleted (which removes their old edges) and their name embeddings are purged from the Entity_name vector collection.
A merge report listing each canonical_id, canonical_name, and the merged_from duplicates is written to the logs on every run, including dry runs.

Additional Information

Runnable guide script available on our GitHub
Pipeline implementation: consolidate_entities.py

Tuning the matching

Raise similarity_threshold to merge only very close names, exclude entity types you never want collapsed, and widen top_k if dense clusters of similar names are being missed.

await consolidate_entities_pipeline(
    similarity_threshold=0.9,
    protect_node_types=["Person"],
    top_k=25,
    dataset="my_dataset",
)

Under the hood

Two tasks run in sequence, both exported from cognee.tasks.memify:

detect_entity_duplicates (extraction) — loads every Entity node from the graph, embeds its name, and clusters near-duplicates by cosine similarity and, when name_match is on, by normalized-name equality. Similarities are scanned in row-blocks and only each node’s top_k nearest neighbors are inspected, so the full N×N similarity matrix is never materialized.
merge_entity_duplicates (enrichment) — picks the canonical for each cluster, re-points the duplicates’ edges onto it in one batched call, writes back the canonical with unioned description and belongs_to_set, then deletes the duplicate nodes and their vectors.

The merge is backend-agnostic: it uses only get_graph_data, add_edges, add_nodes, delete_nodes, and the vector engine’s delete_data_points. No per-edge delete primitive is required.

Troubleshooting

Nothing was merged — the graph must contain at least two candidate Entity nodes. Check the logs for the detected-cluster count, then lower similarity_threshold or raise top_k.
Entities that should match are not merging — by default only entities sharing the same EntityType are merged. Set allow_cross_type=True if the duplicates were typed differently.
Too much was merged — restore from backup, then raise similarity_threshold and add the affected types to protect_node_types.
Validation errors — similarity_threshold must be in (0, 1], top_k must be a positive integer, and protect_node_types must be a list of non-empty strings.
Permission errors — the user must have write access to the target dataset. See Permissions.

Entity Consolidation

Rewrite fragmented entity descriptions with the LLM

Improve

Understand the current improvement workflow

Embedding Providers

Configure the embedder that duplicate detection relies on

Getting Started

Core Concepts

Setup Configuration

Guides

Examples

CLI

Rust SDK

TypeScript SDK

OSS

Before You Start

Code in Action

What Just Happened

Step 1: Build a Graph with Duplicates

Step 2: Preview the Merge Plan

Step 3: Apply the Merge

Options

What the Merge Changes

Additional Information

Entity Consolidation

Improve

Embedding Providers

​Before You Start

​Code in Action

​What Just Happened

​Step 1: Build a Graph with Duplicates

​Step 2: Preview the Merge Plan

​Step 3: Apply the Merge

​Options

​What the Merge Changes

​Additional Information

Entity Consolidation

Improve

Embedding Providers

Before You Start

Code in Action

What Just Happened

Step 1: Build a Graph with Duplicates

Step 2: Preview the Merge Plan

Step 3: Apply the Merge

Options

What the Merge Changes

Additional Information