> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognee.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Entity Deduplication
> Detect near-duplicate entity nodes and merge each group into a single canonical node
A minimal guide to merging near-duplicate `Entity` nodes. When the same real-world thing is extracted under several names (`New York City` and `NYC`), it becomes several `Entity` nodes and its edges are split across all of them — the `consolidate_entities` Memify pipeline detects those near-duplicates and collapses each group into a single canonical node.
This pipeline is **destructive**. Duplicate `Entity` nodes are deleted from the graph and their name embeddings are deleted from the vector store. Run it with `dry_run=True` first and back up your graph and vector stores before running it for real.
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — the script below builds a graph with `remember()`
* Ensure you have [Embedding Providers](/setup-configuration/embedding-providers) configured — detection embeds every entity name at run time
* Have an existing knowledge graph with `Entity` nodes, or let the script create one
* The acting user needs **write** access to the target dataset. See [Permissions](/core-concepts/multi-user-mode/permissions-system/datasets)
## Code in Action
```python theme={null}
import asyncio
import cognee
from os import path
from cognee.api.v1.visualize.visualize import visualize_graph
from cognee.memify_pipelines.consolidate_entities import consolidate_entities_pipeline
custom_prompt = """
Extract every place mentioned in the text as an entity, keeping the exact
surface form used in the text (so "NYC" stays "NYC").
Connect people to places with the relationship "visited".
Ignore all other entities.
"""
async def main():
# Prune data and system metadata before running, only if we want "fresh" state.
await cognee.forget(everything=True)
# Ingest the two texts separately: extracted together, the LLM resolves the
# abbreviation and emits a single entity, leaving nothing to merge.
await cognee.remember(
"Sara visited New York City last spring.",
custom_prompt=custom_prompt,
self_improvement=False,
)
await cognee.remember(
"Bob thinks NYC has the best bagels.",
custom_prompt=custom_prompt,
self_improvement=False,
)
await visualize_graph(
path.join(path.dirname(__file__), ".artifacts", "before_entity_deduplication.html")
)
# Preview the merge plan in the logs without touching the graph.
await consolidate_entities_pipeline(similarity_threshold=0.6, dry_run=True)
# Apply the merge for real.
await consolidate_entities_pipeline(similarity_threshold=0.6)
await visualize_graph(
path.join(path.dirname(__file__), ".artifacts", "after_entity_deduplication.html")
)
if __name__ == "__main__":
asyncio.run(main())
```
## What Just Happened
### Step 1: Build a Graph with Duplicates
```python theme={null}
await cognee.forget(everything=True)
await cognee.remember(
"Sara visited New York City last spring.",
custom_prompt=custom_prompt,
self_improvement=False,
)
await cognee.remember(
"Bob thinks NYC has the best bagels.",
custom_prompt=custom_prompt,
self_improvement=False,
)
await visualize_graph(
path.join(path.dirname(__file__), ".artifacts", "before_entity_deduplication.html")
)
```
Start from a clean state, then ingest two texts that mention the same city under different names — in two separate `remember()` calls. That separation matters: when the texts are ingested together, the LLM sees both mentions at once, resolves the abbreviation during extraction, and emits a single entity, leaving nothing to merge. Ingested independently, the graph ends up with separate `New York City` and `NYC` entities — visible in the before visualization — each holding only its own edges.
### Step 2: Preview the Merge Plan
```python theme={null}
await consolidate_entities_pipeline(similarity_threshold=0.6, dry_run=True)
```
`dry_run=True` runs detection and computes the full merge plan, then logs it — which clusters were found, which node becomes canonical, and how many edges would be re-pointed — without touching the graph. The threshold is lowered from the `0.85` default because an abbreviation like `NYC` is not embedding-close enough to `New York City` to clear it.
### Step 3: Apply the Merge
```python theme={null}
await consolidate_entities_pipeline(similarity_threshold=0.6)
await visualize_graph(
path.join(path.dirname(__file__), ".artifacts", "after_entity_deduplication.html")
)
```
Once the plan looks right, drop `dry_run` to execute it. The after visualization shows a single canonical city node carrying the edges from both duplicates.
## Options
`consolidate_entities_pipeline()` takes the following arguments:
| Argument | Type | Default | Description |
| ---------------------- | --------------------- | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `similarity_threshold` | `float` | `0.85` | Minimum cosine similarity between two entity-name embeddings for them to be treated as the same entity. Must be in the range `(0, 1]`. |
| `dry_run` | `bool` | `False` | When `True`, compute and log the merge plan but perform zero mutations. |
| `protect_node_types` | `Optional[List[str]]` | `None` | `EntityType` names that must never be merged. Must be a list of non-empty strings. |
| `name_match` | `bool` | `True` | Also merge entities whose normalized names are identical, where normalizing means lower-casing and stripping every non-alphanumeric character (so `U.S.A.` matches `USA`). This pass is not bounded by `top_k`. |
| `top_k` | `int` | `10` | Max neighbors considered per entity during similarity clustering. Must be a positive integer. |
| `allow_cross_type` | `bool` | `False` | When `False` (the default, conservative choice), only entities sharing the same `EntityType` are merged together. Set `True` to allow merging across types. |
| `user` | `Optional[User]` | `None` | Acting user. The default user is used when omitted. |
| `dataset` | `str` | `"main_dataset"` | Dataset name or id whose graph to consolidate. |
| `run_in_background` | `bool` | `False` | Forwarded to `memify`. |
Arguments are validated before anything runs. A `CogneeValidationError` is raised when `similarity_threshold` is outside `(0, 1]`, when `top_k` is not a positive integer, when `protect_node_types` is not a list of non-empty strings, or when the user has no write access to the requested dataset.
## What the Merge Changes
* Each detected cluster collapses into **one canonical `Entity` node**; the rest are deleted. The canonical is the most connected member of the cluster, with ties broken by oldest `created_at` and then by normalized name.
* The canonical **keeps its existing graph id**, so anything already pointing at it stays valid.
* Every edge on a duplicate is re-pointed onto the canonical with its direction preserved. Edges that would become a self-loop on the canonical are dropped.
* The canonical's `description` becomes the union of the distinct, non-empty descriptions across the cluster.
* The canonical's `belongs_to_set` becomes the union of every member's tags, so a merge never narrows a node's dataset / node-set scoping.
* Duplicate nodes are detach-deleted (which removes their old edges) and their name embeddings are purged from the `Entity_name` vector collection.
* A merge report listing each `canonical_id`, `canonical_name`, and the `merged_from` duplicates is written to the logs on every run, including dry runs.
## Additional Information
* Runnable guide script available on our [GitHub](https://github.com/topoteretes/cognee/blob/dev/examples/guides/entity_deduplication.py)
* Pipeline implementation: [consolidate\_entities.py](https://github.com/topoteretes/cognee/blob/dev/cognee/memify_pipelines/consolidate_entities.py)
Raise `similarity_threshold` to merge only very close names, exclude entity types you never want collapsed, and widen `top_k` if dense clusters of similar names are being missed.
```python theme={null}
await consolidate_entities_pipeline(
similarity_threshold=0.9,
protect_node_types=["Person"],
top_k=25,
dataset="my_dataset",
)
```
Two tasks run in sequence, both exported from `cognee.tasks.memify`:
1. **`detect_entity_duplicates`** (extraction) — loads every `Entity` node from the graph, embeds its name, and clusters near-duplicates by cosine similarity and, when `name_match` is on, by normalized-name equality. Similarities are scanned in row-blocks and only each node's `top_k` nearest neighbors are inspected, so the full N×N similarity matrix is never materialized.
2. **`merge_entity_duplicates`** (enrichment) — picks the canonical for each cluster, re-points the duplicates' edges onto it in one batched call, writes back the canonical with unioned `description` and `belongs_to_set`, then deletes the duplicate nodes and their vectors.
The merge is backend-agnostic: it uses only `get_graph_data`, `add_edges`, `add_nodes`, `delete_nodes`, and the vector engine's `delete_data_points`. No per-edge delete primitive is required.
* **Nothing was merged** — the graph must contain at least two candidate `Entity` nodes. Check the logs for the detected-cluster count, then lower `similarity_threshold` or raise `top_k`.
* **Entities that should match are not merging** — by default only entities sharing the same `EntityType` are merged. Set `allow_cross_type=True` if the duplicates were typed differently.
* **Too much was merged** — restore from backup, then raise `similarity_threshold` and add the affected types to `protect_node_types`.
* **Validation errors** — `similarity_threshold` must be in `(0, 1]`, `top_k` must be a positive integer, and `protect_node_types` must be a list of non-empty strings.
* **Permission errors** — the user must have write access to the target dataset. See [Permissions](/core-concepts/multi-user-mode/permissions-system/datasets).
Rewrite fragmented entity descriptions with the LLM
Understand the current improvement workflow
Configure the embedder that duplicate detection relies on