` in the system temp directory, with the skill's slug as a subfolder), which keeps the source directory fixed per dataset and skill name, so re-ingesting the same skill name (the `skill_name` field on the inline path; the uploaded `SKILL.md`'s parent-folder name on the upload path) into the same dataset upserts the same node with refreshed content and embedding. The same name in a *different* dataset still resolves to a distinct id, so attaching one skill to several datasets keeps working, and path-based (folder) ingestion is unchanged. The new `DELETE /api/v1/skills/{skill_id}` takes a required `dataset_id` query parameter and requires `delete` permission on that dataset — a separate grant from the `write` that ingestion needs and the `read` the list/fetch routes need — returning `200` with `{"status": "deleted", ...}`, `403` for a dataset you cannot delete in, `404` for an unknown skill id, and `409` when the deletion fails. It is a hard delete of the graph node, its edges and its `Skill_search_text` embedding (the embedding cleanup is best-effort — a vector-store failure is logged without failing the delete) rather than an `is_active=False` soft delete, because a hidden leftover node would be silently resurrected by a later re-ingest now that ids are stable. **Impact:** no migration runs, so Skill nodes duplicated by the old behavior stay in the graph until you remove them — delete the extra copies with the new endpoint (a subsequent re-ingest will then keep updating one node). Deletion is not recoverable; re-ingest the `SKILL.md` to restore a skill (PR #4290).
* Applies the session cache's sliding TTL lazily on the SQL backends (`sqlite` and `postgres`), removing a per-write rewrite of the whole session. The sliding TTL — Redis `EXPIRE`-on-write parity, which pushes a session's expiry forward on every write — was translated to SQL as an `UPDATE` over *all* of that session's rows at every write path, so the cost of a write grew with the length of the session and total write cost grew quadratically; on the default SQLite backend this produced extreme WAL write amplification (a reported 64.2 GB `cache.db-wal` against a 200 MB `cache.db`, growing 5–39 GB/hour under steady agent traffic). `log_usage` had the same shape one scope wider, re-stamping every usage-log row for the user on each logged call across the 12 decorated API routes (including `POST /api/v1/recall` and `POST /api/v1/search`) and the MCP tools that share the decorator. The `UPDATE` now skips rows whose recorded expiry lags the new target by less than 5% of the TTL, so each row is rewritten at most once per slack window and a write costs roughly its own bytes. **Behavior change:** session entries and usage logs now expire between 0.95 × `SESSION_TTL_SECONDS` and 1.0 × `SESSION_TTL_SECONDS` after the last write instead of exactly at the TTL — with the 7-day default, up to \~8.4 hours earlier — so treat the TTL as a lower bound with a small slack window; rows written while the TTL was disabled are still stamped on the next write, and Redis keeps exact `EXPIRE` semantics. Setting `SESSION_TTL_SECONDS=0` disables expiry entirely and skips the sliding-TTL writes as well, which is the lightest-I/O setting for long-lived sessions on SQLite. No public API signature, configuration option, or environment variable changed, and no migration is required (COG-6106, PR #4405).
* **Breaking: removes multiprocess and distributed (Modal) execution support.** The `COGNEE_DISTRIBUTED` environment variable, the `cognee[distributed]` install extra, and the Modal execution path are no longer supported, and running multiple Cognee processes against the same stores is not a supported configuration — the embedded defaults (Ladybug/Kuzu graph, SQLite, LanceDB) are file-based with process-local locks, so a second process opening the same files can see stale or empty data. Cognee runs as a single process; when more than one process or agent needs the same memory, route all access through a single Cognee service backed by external stores (Neo4j for the graph, Postgres for the relational store, PGVector for vectors). The distributed-execution guide and Modal deployment page have been removed, and the deployment, caching, and configuration docs now consistently describe single-process operation (COG-6050).
* Fixes Graphiti temporal-awareness indexing embedding the wrong text into the `GraphitiNode_name` and `GraphitiNode_summary` vector collections, so similarity search against those fields matched on the node's `content` instead of on the field the collection is named for. `index_and_transform_graphiti_nodes_and_edges()` builds one indexable point per entry in `GraphitiNode.metadata["index_fields"]` (`name`, `summary`, `content`) by calling `model_copy()` and then narrowing the copy's `metadata["index_fields"]` to the single field being indexed. Pydantic v2's `model_copy()` is a shallow copy and `DataPoint` does not override it, so every copy taken from a given node shared that node's one `metadata` dict: each field's assignment overwrote the previous ones, and by the time the points were flushed all of that node's copies carried the *last* indexed field — `content` where set, otherwise `summary`. The vector adapters then resolve the text to embed from `metadata["index_fields"][0]` rather than from the `index_property_name` they were called with — directly in `LanceDBAdapter.index_data_points`, and via `DataPoint.get_embeddable_data` on the PGVector and Turso adapters — so that overwritten field name decided what was actually embedded. Each copy now gets its own `metadata` dict before the assignment. The contamination was bounded to copies of a single node within one indexing pass: Pydantic v2 gives each instance its own copy of a mutable field default, so the class-level `GraphitiNode.metadata` default was never corrupted and later nodes and other pipelines were unaffected. `EdgeType` declares exactly one index field, so edge indexing made a single copy per edge type and was never affected. **Operator impact:** if you indexed a Graphiti graph before this fix, re-run `index_and_transform_graphiti_nodes_and_edges()` to correct the affected collections — nodes whose only non-`None` indexable field was the one being indexed were already correct. On the default LanceDB backend the re-run is enough, because its `merge_insert` upsert rewrites the stored vector along with the payload; on PGVector and Turso the `ON CONFLICT (id)` clause updates only the row's `payload` and leaves the existing vector untouched, so delete the stale rows from `GraphitiNode_name` and `GraphitiNode_summary` first (`delete_data_points`) or the wrong embeddings will survive the re-run. No schema change, no migration, and no public API signature, configuration option, or environment variable changed (fixes #3292, PR #3580).
* Bumps the pinned `enola` release used by code-graph extraction from `0.1.34` to `0.3.13` and ingests the explainer findings that 0.3.x writes to `insights.json` as a new fact kind. Each finding becomes a synthetic fact of kind `insight`, mapped to a new `CodeInsight` DataPoint whose `name` is the finding's title, whose `description` is the explainer's own prose (used verbatim instead of the generic `kind: k=v` property summary other fact kinds get), and whose `fact_properties` carry `source` (the explainer that produced it), `confidence`, `description`, and `suggested_actions`, each when the finding provides it. Each piece of evidence the finding cites — a symbol, fact, or file — becomes an `evidences` edge from the insight to that node when the target resolves to a fact in the snapshot, so a finding is linked to the code it is about. Findings come from enola's deterministic, LLM-free explainers — hotspots, god-class, dependency-depth, cycles, layers, exported-surface, complexity-outliers, and others. `SearchType.CODE` picks these up with no API change: `insight` is now a valid `kind`/`kinds` value, `CodeInsight` a valid `node_types` value, and `evidences` a usable `relation_types` value. Installation also gains a `darwin-amd64` (Intel macOS) build alongside the existing `darwin-arm64`, `linux-amd64`, `linux-arm64`, and `windows-amd64` ones, and archive extraction handles the 0.3.x tarball layout, which ships `LICENSE` and `NOTICE` next to the binary — the extractor still refuses any archive that does not contain exactly one top-level `enola*` file, or whose members contain a path separator or start with a dot. **Upgrade impact:** the `facts.jsonl` relation shape is unchanged, so existing consumers of the code graph keep working, and a snapshot without `insights.json` is skipped silently while one that cannot be parsed is logged and ignored, rather than failing extraction, so 0.1.x snapshots still extract; re-extracting a repo grows its graph by the number of findings enola reports (129 on the cognee repo itself). The auto-installed binary is version-scoped by filename, so an existing `enola-0.1.34-*` install is not reused — the first extraction after upgrading downloads `0.3.13` unless `ENOLA_PATH` points at your own binary, which still wins over the auto-install (COG-6113, PR #4404).
* Fixes the Amazon Neptune graph adapter (`NeptuneGraphDB`, used by `GRAPH_DATABASE_PROVIDER="neptune"`) never releasing its AWS client when the graph engine is dropped from Cognee's engine cache. Graph engines are created through `_create_graph_engine`, which is wrapped in `closing_lru_cache`; that cache closes each entry once it has left the cache and the last caller handle has been released. Its close step starts by checking whether the cached value has a `close` attribute and returns immediately when it does not — and `NeptuneGraphDB` implemented no `close()`, nor does `GraphDBInterface` declare one — so for Neptune the close was silently skipped and the `langchain_aws` `NeptuneAnalyticsGraph` together with its underlying boto3 client was discarded without being closed, leaving the client's pooled connections to be reclaimed non-deterministically by the interpreter instead of released at eviction. The adapter now implements `async close()`, which closes the wrapped boto3 client (`self._client.client`) when it exposes a `close()` method and then clears `self._client`; the client and attribute checks are defensive, so the call is a safe no-op when the client was never initialized and when close runs more than once, as the cache's idempotency requirement expects. The evictions this affects are the ones any provider sees — capacity eviction once more distinct graph configurations are in play than `DATABASE_MAX_LRU_CACHE_SIZE` (default `6`), explicit eviction when a dataset is deleted, and the `cache_clear` behind prune — so the benefit is to long-running services that cycle graph configurations, not to a short script that creates one engine and exits; Neptune is not the per-dataset handler (`GRAPH_DATASET_DATABASE_HANDLER` defaults to `ladybug`), so this is not about per-dataset database isolation. Eviction-time close remains deferred until the last engine handle drops and a failing close is still logged and swallowed rather than raised. The separate `neptune_analytics` provider is served by a different hybrid adapter and is unchanged. No public API signature, configuration option, or environment variable changed and no migration is required; callers continue to obtain engines through `get_graph_engine()` and are not expected to construct or close adapters themselves (PR #3244).
* Adds an extraction-oriented image transcription prompt and an optional local OCR pass to `ImageLoader` (`cognee/infrastructure/loaders/core/image_loader.py`). **Default behavior changes:** images are now transcribed with a new prompt template (`transcribe_image_prompt.txt`) that asks for the entities shown and their attributes, the relationships between them, all visible text/numbers/dates/labels transcribed verbatim, and structured content (tables as rows, charts as series and data points, diagrams as element connections), under a 1024-token completion cap — where previously every image got the hardcoded `"What's in this image?"` caption prompt and a 300-token cap. Images therefore produce longer, denser text and cost more tokens per image than before; set `IMAGE_EXTRACTION_ENABLED="false"` to restore the previous caption prompt and 300-token cap. Five environment variables are new: `IMAGE_EXTRACTION_ENABLED` (default `"true"`), `IMAGE_TRANSCRIPTION_PROMPT_PATH` (default `"transcribe_image_prompt.txt"`; a file name resolves inside `cognee/infrastructure/llm/prompts`, an absolute path is loaded from its own directory), `IMAGE_TRANSCRIPTION_MAX_COMPLETION_TOKENS` (default `1024`), `IMAGE_TRANSCRIPTION_REASONING_EFFORT` (default `"low"`; `minimal`/`low`/`medium`/`high`, dropped for models without reasoning support), and `IMAGE_OCR_ENABLED` (default `"false"`). With `IMAGE_OCR_ENABLED="true"` and the new `cognee[rapidocr]` extra installed (`rapidocr-onnxruntime`, pip-only — no system binary), a local OCR pass runs off the event loop and its recognized text is appended to the transcription under an `[OCR extracted text]` heading, truncated at 8000 characters; an OCR failure is logged and the vision transcription is kept rather than failing ingestion. Because `IMAGE_TRANSCRIPTION_MAX_COMPLETION_TOKENS` also caps reasoning tokens, a small value on a reasoning model (including the default `openai/gpt-5-mini`) can return empty content — the loader logs a warning suggesting a higher cap and continues with empty text for that image. `transcribe_image` gained optional `prompt`, `max_completion_tokens`, and `reasoning_effort` keyword parameters on `LLMGateway` and the LLM interface; existing calls keep working unchanged, and images still reduce to `chunk.text` and feed the existing graph extractor unchanged (partially addresses #3637, PR #3956).
* Fixes the two connected-components metrics being transposed in `get_graph_metrics()` on the Neptune graph backends (`GRAPH_DATABASE_PROVIDER=neptune` and Neptune Analytics). `NeptuneGraphDB.get_graph_metrics` unpacked its internal `_get_connected_components_stat()` helper as `num_cluster, list_clsuter_size`, but that helper returns `(sizes, count)` — so `num_connected_components` came back as the descending list of per-component sizes and `sizes_of_connected_components` as the integer number of components, the exact inverse of what the keys name. The unpack is corrected, so `num_connected_components` is now an `int` and `sizes_of_connected_components` a `list[int]` of per-component sizes in descending order, matching the `int`/`list[int]` shape the Ladybug, Neo4j, Postgres, and Turso adapters already returned — Neptune was the only backend that disagreed. **Impact is limited to Neptune deployments:** anything reading these two keys off a Neptune graph received the wrong type for each, so dashboards, alerts, and scripts written against the old shape — for example treating `num_connected_components` as a list, or `sizes_of_connected_components` as a scalar — must be switched back to the documented types, and a consumer that assumed the correct types was getting a type error or nonsense value rather than a wrong-but-plausible number. The same values feed the `graph_metrics` row written by `get_pipeline_run_metrics`, whose columns are `Integer` for `num_connected_components` and `JSON` for `sizes_of_connected_components`, so rows recorded from a Neptune graph before this fix cannot be trusted for these two fields; re-run metrics collection if you rely on their history. No public API signature, configuration option, environment variable, or migration changed — deploy the fix to pick it up (PR #3171).
* Separates ontology-aware from ontology-free graph construction in the Cognify extraction path, and bases persisted entity identity on entity names instead of the graph-local ids the LLM assigns. `cognee.modules.graph.utils` no longer exports `expand_with_nodes_and_edges` or `retrieve_existing_edges`: the first is replaced by `construct_data_points_and_edges` plus `attach_new_edges_to_data_points`, and the second by `find_existing_edge_identities`, which takes a collection of `EdgeIdentity` values and returns the subset already in graph storage rather than the previous `{edge_key: True}` mapping. Ontology enrichment moves out to `cognee/modules/ontology/construct_data_points_and_edges_with_ontology.py` and now runs as a canonicalize-first pre-pass that rewrites the extracted graph before any nodes are constructed from it, rather than validating nodes one by one as they are built: nodes matching the same ontology individual collapse into a single node with the collapsed nodes' edges rewired onto the survivor, and an edge from a matched ontology subgraph is attached only when **both** of its endpoints are part of that subgraph instead of minting a node for the missing endpoint. A run with no ontology configured no longer passes through the ontology code path at all — the new `get_configured_ontology_resolver(config)`, which `cognify()` and `get_default_tasks()` both now call in place of their duplicated branching, returns `None` when neither an explicit `config["ontology_config"]["ontology_resolver"]` nor an `ONTOLOGY_FILE_PATH` environment setting is present, where the previous code fell back to instantiating an empty `RDFLibOntologyResolver`. **Entity ids change:** they are now derived from `Entity.id_for(node.name)` rather than `Entity.id_for(node.id)`, so two chunks mentioning the same entity name converge on one node, while several distinct nodes sharing a name inside a single extracted graph keep deterministic chunk-scoped ids instead of collapsing together; an extracted edge whose endpoint is not among that chunk's extracted nodes is now dropped rather than pointing at an id no node was created for. Entity nodes written by earlier runs keep their old id-derived ids, so re-cognifying data that is already in the graph can create a second node for an entity that is already there — re-run the affected datasets from scratch if you need ids to line up. **Edge properties change:** persisted edges no longer carry an `ontology_valid` property — grounding was never applied to relationship names, so the flag is now node-only; filter on the endpoints instead. The `cognify()` signature, configuration options, environment variables, and migrations are unchanged (SDK-160, PR #4262).
* Makes enola code-graph ingestion incremental, so re-running it against an unchanged repository is near-free and a changed repository no longer accumulates facts that were deleted upstream. Previously every run re-loaded every fact and deleted nothing, so nodes for removed classes, files, and routes and edges for removed dependencies stayed in the graph indefinitely; the pipeline's generic incremental mode could not help, because it keys on the data item's content hash and the code-graph data item is a repository *path*, which does not change when the repository does. `extract_code_graph` now derives a snapshot identity for each run through the new `snapshot_identity()` helper — enola's `receipt.json` `snapshot_id` when present, otherwise a `sha256:` digest of `facts.jsonl`, and no identity at all when neither is readable, in which case the run always loads fully. That identity is compared against `last_snapshot_id`, a new field on the `CodeRepository` node; the marker is stored on the node in the graph rather than in the relational metastore because this pipeline persists no `Data` row to key relational state on, which also means it cannot outlive the graph it describes. On a match, `extract_code_graph` returns an empty list and both `add_code_graph_data_points` and `add_code_graph_edges` short-circuit, so an unchanged repository costs only the enola scan. When the identity differs, the load becomes a delta write followed by a sweep: `CodeGraphEntity` gained a `fact_hash` field fingerprinting each fact's derived fields, only facts whose hash is new or changed are written, only edges not already present are added, and then nodes and edges that earlier ingestions derived but the current snapshot no longer does are removed via `delete_nodes` and `delete_edge_triples` (chunked, so no single statement outruns the engine's per-call deadline). The sweep is deliberately narrow: it considers only code-graph node types belonging to repositories the current snapshot covers, and removes edges only between surviving code nodes, so other datasets, other repositories in the same graph, and edges to non-code nodes such as `belongs_to_set` → `NodeSet` are never touched. `last_snapshot_id` and a new `last_delta` record — added/updated/unchanged/removed counts, capped name samples, the snapshot id, and a load timestamp — are stamped on the repository node only after the load and the sweep both succeed, so a crashed run cannot later be mistaken for an up-to-date one. `SearchType.CODE`'s `code_query` gains a matching `delta` operation alongside `query_facts`, `explore`, `traverse`, `find_path`, and `impact_analysis`; it reads those records back and reports what the last ingestion changed per repository, returning `delta: null` for repositories loaded before this change. Same-named facts of the same kind now collapse to the first occurrence rather than the last, so a node's stored content and `fact_hash` stay stable across ingestions instead of flip-flopping and reading as "updated" on every run. **Operator impact:** repeated code-graph runs over an unchanged repository drop from minutes to roughly the \~3s enola scan, and orphans left behind by earlier ingestions are cleared on the first changed run after upgrading, so expect a one-off drop in code-graph node and edge counts. Because the marker is a property of the `CodeRepository` node, anything that drops the graph itself — prune, or deleting the graph database files — drops the marker with it, and the next run rebuilds from scratch. `forget(memory_only=True)` does not: it deletes by provenance, and the code-graph pipeline's payload is a repository path with no `Data` row to record provenance against, so its nodes and their marker survive. One known gap: on `index_vectors=True` runs, vector index entries for swept nodes are not yet removed; the default graph-only path (`index_vectors=False`) is fully handled. No environment variable, configuration option, or public function signature changed, and no migration is required (COG-6115, PR #4407).
* Fixes `POST /api/v1/cognify` and `POST /api/v1/memify` hiding the real failure inside their `500` response body. Both routers built the body's `detail` as `getattr(run, "error", None) or str(run)`, but `PipelineRunErrored` has no `error` attribute — the failing task's error is recorded on `payload`, which the pipeline runner sets to `repr(error)` — so the `getattr` was always `None` and `detail` fell back to the model's repr, e.g. `status='PipelineRunErrored' pipeline_run_id=UUID('...') dataset_name='ds' payload="ValueError('LLM_API_KEY is missing')" ...` instead of the actionable message it wraps. Both routers now read `payload` when it is a string — the pattern `POST /api/v1/add` already used — so `detail` is the task's own error, e.g. `ValueError('LLM_API_KEY is missing')`, and falls back to the run repr only when the run carries no error string. The `500` status code, the `{"error": "Pipeline run errored", "detail": ...}` body shape, and the separate `{"error": "Internal server error", "detail": ...}` body those endpoints return for unexpected exceptions are all unchanged. **Client impact:** only the text inside `detail` changed, so log parsing or alerting rules that pattern-matched the old `status='PipelineRunErrored' ...` repr no longer match and should read the plain message instead. No public API signature, configuration option, environment variable, or migration changed (PR #3265).
* Replaces the non-standard `418 I'm a teapot` status code with `500 Internal Server Error` in the four places it was used. `GET /api/v1/datasets` and `POST /api/v1/datasets` wrap any unexpected failure in an `HTTPException` — retrieving datasets and creating a dataset respectively — and both now raise `500`; their endpoint docstrings and the generated HTTP API reference list `500 Internal Server Error` accordingly. The global `CogneeApiError` handler in `cognee/api/client.py` also returns `500` for its fallback branch, which fires when a raised Cognee exception is missing a message, name, or status code and the handler substitutes `{"detail": "An unexpected error occurred."}`. The `CogneeApiError` base class default `status_code` moves from `418` to `500` as well; every direct subclass sets its own status code (for example `422` for `CogneeValidationError`, `503` for `CogneeTransientError`), so this default applies only to a direct `CogneeApiError(...)` raise that omits `status_code`. **Client impact:** integrations that branch on `418` for these responses must switch to `500` — response bodies, status codes for every other error, endpoint paths, and request shapes are unchanged. No configuration option or environment variable changed (issue #3742, PR #3860).
* Removes the stale `tree-sitter` and `tree-sitter-python` dependencies from the `cognee[codegraph]` extra, which now installs only `fastembed` and `transformers`. Neither package was imported anywhere in Cognee: they were left behind by the removed Python AST-based code-graph parser, and the current code-graph implementation delegates extraction to the external `enola` binary (`cognee/tasks/code_graph/extract_code_graph.py`). Because `codegraph` pinned `tree-sitter>=0.24.0,<0.25` while `docling-full` needs `tree-sitter>=0.25` through `docling-core[chunking]`, `pyproject.toml` also declared the two extras mutually exclusive under `[tool.uv] conflicts`, so the contributor setup command documented in `AGENTS.md` — `uv sync --dev --all-extras --reinstall` — failed to resolve at all with `error: Extras codegraph and docling-full are incompatible with the declared conflicts`. With the unused pins gone the conflict declaration is dropped too, so all extras install together and both `codegraph` and `docling-full` can be used in the same environment; `uv.lock` was regenerated to drop the obsolete conflict markers. **Impact:** an `ImportError` mentioning `tree_sitter` is no longer resolved by installing `cognee[codegraph]` — no Cognee code path imports it, so the error comes from something else in your environment. Code-graph extraction behavior is unchanged (it needs the `enola` binary, not a Python parser), and no public API signature, configuration option, or environment variable changed; no migration is required (PR #4432).
* **Security: restricts `POST /api/v1/settings` to superusers.** Writing the system LLM configuration (provider, model, API key) and vector-database configuration (provider, URL, API key) over HTTP only required an authenticated user, so any account that could log in — an ordinary tenant member, an integration or agent account, a signed-in UI session — could repoint the deployment's LLM or vector store and read back a masked preview of the stored keys (first ten characters) through `GET /api/v1/settings`. The `save_settings` handler now checks `user.is_superuser` before touching `save_llm_config` / `save_vector_db_config` and answers a non-superuser with `403 Forbidden` and the body `{"error": "Superuser privileges required to modify settings"}`; the request payload, the empty `200` on success, and the `400` / `500` error codes are unchanged. This re-adds a guard that shipped in #3115 and was later reverted, and it fixes **CVE-2026-58473 / GHSA-49f7-whx5-4256**; a regression test (`cognee/tests/api/test_settings_authorization.py`) now pins both the allowed and the forbidden case. **`GET /api/v1/settings` is unchanged** — reading the settings still requires only an authenticated user, so this is not a lockdown of the read path. **Who is affected:** only callers that are not superusers. `create_user(...)` defaults to `is_superuser=False`, so integrations, automation scripts, and UI flows that let non-admin accounts adjust LLM or vector settings now break with `403` and must either authenticate as a superuser or drop the write. The auto-created default user (`default_user@example.com`) *is* a superuser, and with authentication disabled every request falls back to that user, so single-user deployments, local development, and the `cognee.start_ui()` / `cognee-cli -ui` "Add your API key" modal keep working exactly as before. No configuration option, environment variable, or migration ships with the fix — deploy it to pick it up (PR #4434, mirroring contributor PR #4252).
* Fixes the eval-framework HTML dashboards interpolating arbitrary benchmark text into their markup without escaping it. Both dashboard modules — `cognee/eval_framework/metrics_dashboard.py`, the one the eval runner uses, and its analysis twin `cognee/eval_framework/analysis/dashboard_generator.py` behind `create_dashboard()` — built the report with f-strings: `generate_details_html` emitted each per-item field straight into a `| `, and `get_dashboard_html_template` dropped the `benchmark` name into `` and `<h1>`. Any `<`, `&`, or tag-like content in that text corrupted the rendered page (a `</td>` or `<script>` in a golden answer escaped its cell and broke the table), and made the generated report an HTML-injection sink when opened in a browser, since the text originates from benchmark data and model output rather than from Cognee. Both modules now apply `html.escape` at every interpolation point: the details-table cell values, the derived column headers (`metrics_dashboard.py`, which titles them from the item keys), the per-metric `<h3>` section heading, and the `benchmark` name in the page template. The fields this covers are the ones each module already rendered — `question`, `answer`, `golden_answer`, `reason`, and `score` in `dashboard_generator.py`, and whatever keys the metric items carry in `metrics_dashboard.py` (`question`, `answer`, `golden_answer` by default, `question` and `retrieval_context` for `contextual_relevancy`, and those two plus `golden_context` for `context_coverage` — each with `reason` and `score`). Plotly figure HTML from `create_distribution_plots()` / `create_ci_plot()` and the already-assembled `details_html` list are deliberately left unescaped as trusted, pre-rendered markup, so charts render exactly as before. **Impact:** benchmark fields that contained raw HTML previously rendered as markup and now display as literal text — there is no flag to restore the old behavior. No public API signature, configuration option, or environment variable changed, and no migration is required; regenerate a dashboard to pick up the fix (PR #4429).
* **Breaking: `cognee.validate()` now rejects a dataset it cannot read instead of falling back to the default stores.** `validate()` resolved its `dataset` argument through `get_authorized_existing_datasets(..., "read", user)` and read the first result, but when *none* of the requested names resolved it left the resolved dataset as `None` and entered the database context unscoped — so with backend access control disabled a typo, an unknown name, or a dataset the caller had no `read` permission on produced a `ValidationReport` about the single shared graph and vector stores, indistinguishable from a real report about the requested dataset. Every requested name must now resolve to a dataset the caller can read; otherwise `DatasetNotFoundError` (`"Dataset not found or not readable."`, from `cognee.modules.data.exceptions`, `status_code=404`) is raised *before* any graph or vector adapter is opened, so no store is touched on the rejected path. The check compares counts, so a partially authorized list is rejected as a whole — `dataset=["mine", "not-mine"]` raises rather than silently validating `mine` — and it does not branch on `ENABLE_BACKEND_ACCESS_CONTROL`, so the rejection is identical with access control on or off. Selection is otherwise unchanged: when all names are authorized, the first still determines which graph and vector store are read, and one graph is still checked per call. **Caller impact:** the default `dataset="main_dataset"` is subject to the same rule, so a bare `await cognee.validate()` against an installation where `main_dataset` does not exist yet now raises `DatasetNotFoundError` — where previously it reported on the shared stores with access control disabled, and failed with a generic `CogneeValidationError` ("A dataset must be provided...") with it enabled; and because dataset names resolve only within datasets the caller *owns*, a dataset merely shared with the caller is rejected when requested by name. Passing `dataset=None` or an empty list still skips resolution entirely, which remains the only path that enters no dataset context — with access control disabled that reads the unscoped shared stores, while with it enabled the call still fails, since a dataset is required to resolve the per-dataset databases. **Over HTTP the rejection surfaces as a `500`, not a `404`:** `GET /api/v1/validate` wraps every exception from the route in `{"status": "error", "reason": "validation failed: ..."}` with status `500`, so the `DatasetNotFoundError` never reaches the global `CogneeApiError` handler that would map it to `404` — expect additional `500`s whose `reason` is `validation failed: DatasetNotFoundError: Dataset not found or not readable. (Status code: 404)` — the `404` in the string is the exception's own string form, the HTTP status stays `500` — rather than a rise in `4xx`, and match the `reason` for `Dataset not found or not readable.` if you alert on this. Remediation is to request a dataset the caller is authorized to read, or to catch `DatasetNotFoundError`. No configuration option, environment variable, or migration ships with the fix, and `validate()`'s signature is unchanged (fixes #4414, PR #4455, mirroring contributor PR #4416).
* Adds `TELEMETRY_ORIGIN`, an environment variable that labels where a telemetry event comes from, so events can be segmented by origin. `send_telemetry` reads it with `os.getenv("TELEMETRY_ORIGIN", "sdk")` and stamps the value onto every event's `properties` payload as `telemetry_origin`; the Cognee-managed cloud sets `TELEMETRY_ORIGIN=cloud`, and everything else — local SDK usage, self-hosted servers — reports the default `sdk`. It is read directly from the environment on each event, matching how `send_telemetry` already reads `TELEMETRY_DISABLED` and `ENV`, so there is no config-schema change and it cannot be set through `cognee.config.set(...)`. The property is written before the `**additional_properties` spread, so a caller that passes its own `telemetry_origin` in `additional_properties` still overrides it per call. **Impact:** non-breaking and no action is required — telemetry payloads gain one property, and `TELEMETRY_DISABLED=true` still suppresses events entirely. No public API signature or migration changed (PR #4433).
* Fixes the Redis session-cache adapter (`CACHE_BACKEND="redis"`) returning `None` instead of `[]` from `get_latest_qa_entries` for an empty or unknown session. `RedisAdapter.get_latest_qa_entries` takes a `lindex` fast path when `last_n == 1` — the hot path, since loading the previous turn of a session reads with `last_n=1` — and that branch ended in `if data else None`, contradicting the method's declared `-> list[SessionQAEntry]` return type and diverging from the SQL and FS adapters; the general `lrange` path used for every other `last_n` already returned `[]`. It now returns `[]` on the fast path as well, and the `lindex` optimization is kept. **Impact:** `cognee.session.get_session()` was not affected, because `SessionManager.get_session` carries an `if entries is None` guard that returns `""` or `[]`; the `TypeError: 'NoneType' object is not iterable` this could raise on Redis (while the same call worked on SQLite and FS) was reachable only from code calling the cache adapter directly with `last_n=1`, including through the backward-compatible `get_latest_qa` shim. No public API signature, configuration option, environment variable, or migration changed, and no action is required on upgrade (fixes #3930, PR #3931).
* Fixes the Ladybug and Amazon Neptune graph adapters reporting a *failed* batch edge-existence check as an *empty* one, which could let a cognify run finish reporting success while persisting nothing. `has_edges(edges)` takes a list of `(source_id, target_id, relationship_name)` triples and returns the subset that already exists in the graph; `LadybugAdapter.has_edges` (`GRAPH_DATABASE_PROVIDER="ladybug"`, the default backend) and `NeptuneGraphDB.has_edges` (`GRAPH_DATABASE_PROVIDER="neptune"`) both wrapped the query in `except Exception`, logged the error, and returned `[]`. That value is indistinguishable from a genuine "none of these edges exist yet" answer, and the adapter method's only production caller is the cognify dedup step — `find_existing_edge_identities` in `cognee/modules/graph/utils/retrieve_existing_edges.py`, called from `extract_graph_from_data` to subtract already-stored edges before writing the rest. So when the store was unavailable or corrupt — a corrupt write-ahead log after an unclean shutdown, for example — the check answered "nothing exists", every extracted edge was treated as new and written, those writes failed against the same broken store, and the run completed without raising while the graph stayed empty. Both adapters now log and then re-raise, which is what the other backends already did: `neo4j` logs and re-raises `Neo4jError`, and `postgres` and `turso` never caught the error in the first place. Ladybug's empty-**input** short-circuit is unchanged — `has_edges([])` still returns `[]` without touching the store — and Neptune, which has no such short-circuit, is unchanged in that respect too. Only the batch `has_edges` path was touched; the single-edge `has_edge` and every other adapter method behave as before. **Impact:** no public API signature, configuration option, environment variable, or migration changed, and a run against a healthy graph store behaves exactly as it did. What changes is the failure mode: an ingest that previously "succeeded" against a broken store now fails with the backend error propagated to the caller, so a job that was silently a no-op may begin surfacing as a failure in logs and CI after upgrading. That failure reports a problem that was already present rather than introducing a new one, so the remedy is to fix or restore the graph store, not to suppress the error. This covers the silent-data-loss existence check in the linked issue; the startup write-ahead-log recovery and `SIGTERM` handling also requested there are a separate change and are not included. Unit tests in `cognee/tests/unit/infrastructure/databases/test_has_edges_error_propagation.py` cover the three cases — a successful check returning the existing-edge tuples, a query failure raising instead of returning `[]`, and empty input short-circuiting without reaching the store (fixes #4348, PR #4430).
* Gives each relational [dlt](/integrations/dlt-integration) source a stable identity, so re-ingesting one updates it in place instead of piling up copies. A source is now stored as a single record keyed on `dlt_source:{dataset_name}:{source_name}` — an identity that does not include the data — with change detection carried separately by a content hash over the source's tables and rows. A plain re-run of `add()` / `remember()` on an already-ingested source is idempotent and skips it without reprocessing, whether or not the data changed; to pick up upstream changes, re-ingest explicitly with `add(..., incremental_loading=False, data_cache=False)` (the completed-skip runs whenever either flag is on) or `update()` with the record's UUID. The record then updates in place under the same identity (so the source is never missing from the store between runs), and the next `cognify()` purges that source's previously derived graph nodes, edges, and vectors before re-emitting the current rows, so upstream deletions and edits no longer leave stale rows behind. Two further changes ship alongside it: dlt row embeddings move out of the shared `DocumentChunk_text` collection into their own `DltRow_text` collection — graph-completion retrieval reads both, but chunk search is now documents-only — and the `DLT_MAX_ROWS_PER_TABLE` default changes from `50` to `0`, which means no cap, so a source that was previously truncated to 50 rows per table is now ingested in full unless you set a positive value (via the env var or the `max_rows_per_table` kwarg). **Impact:** three things to plan for on upgrade. First, reprocessing a *changed* dlt source now requires `delete` permission on the dataset, because the purge is re-authorized as a delete; a run without it fails loudly rather than quietly serving stale rows, so check the permissions of any automation that re-syncs dlt sources. Second, dlt data ingested before this change was stored as pre-manifest per-row records, which are no longer supported — `cognify()` raises on them with a message naming the record, and re-adding the source ingests it once as a manifest (and sweeps the legacy records away), so expect a one-off re-ingest of existing dlt datasets. Third, renaming a dlt source or its dataset changes the identity and is therefore a remove + add: the new name ingests fresh and the old name's records remain until you delete them. No public API signature changed, and `primary_key`, `write_disposition`, `query`, and `max_rows_per_table` are accepted exactly as before (COG-2222, PR #4278).
* Fixes background `remember()` work being destroyed by Python's garbage collector mid-flight, so the call reported success while the knowledge graph was never updated. Both fire-and-forget paths were affected: the `run_in_background=True` run that performs `add()` followed by `cognify()`, and the session-to-graph bridge that `remember(session_id=...)` starts when `self_improvement` is on — it defaults to `True`, so every `session_id` call took that path. Each created its task with a bare `asyncio.create_task(...)` and kept the only application-level reference on the returned `RememberResult`. The event loop holds just a weak reference to a running task, so once the caller dropped that result — as `POST /api/v1/remember` does, serializing the result and discarding the object — the surviving task/closure/result references formed a closed cycle with nothing outside it, and the cycle collector could collect the pending task, leaving `Task was destroyed but it is pending!` in the logs as the only trace. Both tasks are now held in a module-level anchor set for their whole lifetime and remove themselves on completion via `add_done_callback`, the same pattern already used for background sync (`cognee/api/v1/sync/sync.py`) and for background pipeline runs (`cognee/modules/pipelines/layers/pipeline_execution_mode.py`). **Who is affected:** callers that start background remember work and do not keep the returned `RememberResult` alive — the HTTP endpoint, and SDK code that calls `remember(..., run_in_background=True)` or `remember(session_id=...)` without holding the result. A caller that kept the result and awaited it was never at risk, and blocking `remember()` calls were never affected. The loss depended on a collection cycle landing at the wrong moment, so an affected deployment saw it intermittently rather than on every request; runs that vanished this way were never recorded as failures, so re-run any background ingestion whose data is missing from the graph. The documented behavior of `remember()` is unchanged — background mode still returns immediately and the result can still be awaited later — it is simply now reliable, and no public API signature, configuration option, environment variable, or migration changed, so deploying the fix is the whole action (fixes #4312, PR #4456).
* Makes [`update()`](/python-api/update) keep a document's `data_id`, and refuses any call that is not a one-document replacement. `update()` deleted the old row and handed the raw content to `add()`, which minted a **new id derived from that content** — so every update silently invalidated whatever id the caller was holding. The incoming id is now resolved once, exact first and then the recorded pre-fork `legacy_id` left by the [dataset-scoping upgrade](/python-api/run-migrations#dataset-scoping-upgrade), and the re-add is pinned to the resolved id, so the document keeps it across updates. Two inputs that used to do something surprising are rejected outright. A `data_id` that matches no document in the dataset raises `UpdateTargetNotFoundError` (**HTTP 404**): previously `delete_data` reported success on an unknown id and `add()` went on to create a *second* document, so a stale or mistyped id looked like a successful update and duplicated the data instead. And a list of more than one item raises `IngestionError` rather than replacing one document with several new ones. `update()` never creates documents — use [`add()`](/python-api/add) for that (PR #4435).
* Fixes `GET /api/v1/datasets/{dataset_id}/data/{data_id}/raw` rejecting a caller who holds a dataset-level `read` grant but does not own the data item. The handler already authorized the dataset with `get_authorized_existing_datasets([dataset_id], "read", user)`, then resolved the item with `get_data(user.id, data_id, dataset[0].id)` — and `get_data` re-checks `data.owner_id == user_id`, raising `UnauthorizedDataAccessError` (HTTP **401**) when they differ. Since a document's `owner_id` is the principal that ingested it, anyone reading a dataset shared with them through an ACL could list its items but got a 401 on every raw download, even though the dataset permission check had already passed. The handler now resolves the item entirely within the authorized dataset: `resolve_data_id(dataset[0].id, data_id)` maps the caller-supplied id to the canonical row id (falling back to the recorded pre-fork `legacy_id`, so ids issued before the dataset-scoping upgrade keep resolving, exactly as before), and the row is then taken from `get_dataset_data(dataset[0].id)`, which is scoped by dataset membership and does not consult `owner_id`. **Impact:** dataset `read` is now sufficient to download raw files — a request that previously returned 401 for a non-owning reader returns the file. Nothing else about the endpoint changed: it is still gated on `read` for the containing dataset, an id that is unknown in that dataset still returns 404, and the owner's own downloads, the S3 streaming path, the local-file path, and the 501 response for unsupported storage schemes all behave as before. One adjacent change: when the dataset itself is not found or not readable, the 404 body's key is now `message` instead of `detail`, so a client reading `detail` off that specific response needs updating (fixes #4162, COG-5923, PR #4468, mirroring contributor PR #4200).
* Fixes file-extension detection being case-sensitive, so files named `data.CSV`, `NOTES.MD`, `config.YAML`, `payload.JSON`, or `feed.XML` were ingested as plain text. `guess_file_type` (`cognee/infrastructure/files/utils/guess_file_type.py`) took the extension from the file name and compared it against literal lowercase lists — `.txt`/`.text`, `.csv`, `.md`/`.markdown`, `.json`, `.xml`, `.yaml`/`.yml` — but `Path("data.CSV").suffix` is `".CSV"`, which matches none of them. Those are exactly the formats that carry no magic-number signature, so an unmatched uppercase extension fell through to `filetype.guess`, which returned `None`, and the file was labeled `text/plain` with extension `txt`. The extension is now lowercased before the lookup, so case no longer affects the result. **Impact:** two things downstream change for files whose extension was not already lowercase. `get_file_metadata` stores the guessed mime type and extension, and `classify_documents` selects the document class from that extension — so a `.CSV` file is now a `CsvDocument`, whose reader expands each row into `key: value` text, instead of a plain `TextDocument` chunked as prose; `.MD`, `.JSON`, `.XML`, and `.YAML` already mapped to `TextDocument` and keep that class, but the `mime_type` and `extension` recorded for them are now the real ones (`text/markdown`/`md` rather than `text/plain`/`txt`). Loader selection also shifts: `LoaderEngine.get_loader` tries the raw path extension first and does *not* lowercase it, so the corrected content-detected extension is what takes effect on the fallback branch, and `data.CSV` now reaches the CSV loader path — `CsvLoader`, or the DLT CSV loader when the `dlt` extra is installed — rather than `TextLoader`. Magic-number formats (PDF, images, audio, video) were already case-independent and are unaffected, and an uppercase `.TXT` still resolves to `text/plain` exactly as before. No public API signature, configuration option, environment variable, or migration changed. Data already ingested keeps the type it was stored with — re-ingest affected files to pick up the corrected type. Unit tests in `cognee/tests/unit/infrastructure/files/utils/test_guess_file_type.py` assert correct detection for lower- and uppercase csv/md/json extensions and uppercase XML/YAML, plus the unchanged `.TXT` case (PR #4453).
* Adds an opt-in, audit-grade provenance ledger: an append-only, tamper-evident record of the document → chunk → entity → relationship lineage each ingestion produced. This is the migration `b8c1d3e5f7a9` listed above — it creates the single `provenance_entries` table (SQLite and Postgres) that backs the feature, and it is idempotent, so it is a no-op where the table already exists. **Default off** — with `PROVENANCE_TRACKING` unset, the cognify task list and behavior are identical to before, nothing is written to the new table, and no public API signature changed. Enabling it (`PROVENANCE_TRACKING=true` on `CognifyConfig`, `cognee/modules/cognify/config.py`) splices a `record_provenance` task (`cognee/tasks/provenance/record_provenance.py`) into the default pipeline right after `add_data_points` — where node ids are persisted and stable — and before the contradiction-detection spread, committing all of one invocation's entries as a single chained transaction. The task returns its input unchanged and swallows all of its own errors, logging a warning rather than failing the run, so provenance can never break ingestion; DataPoints from custom `graph_model` schemas that lack the default `made_from`/`is_part_of`/`contains` shape are covered by a generic walk using the same traversal `add_data_points` uses. Every entry carries a SHA-256 checksum over its canonical JSON plus the previous entry's checksum, linked by a unique `sequence_id`, so tampering — deletion, reordering, or a single-field edit — is detectable. Ledger keys are prefixed with the dataset id, because cognee entity ids are deterministic and the ledger lives in the shared relational database, so two datasets mentioning the same entity name keep separate version chains. A new `ProvenanceManager` (`cognee.modules.provenance.get_provenance_manager()`) is the programmatic surface: `track_entity`, `track_chunk`, `track_relationship`, `get_provenance`, `get_lineage`, `trace_lineage`, `revision_history`, `invalidate`, `verify_chain`, `check`, and `get_statistics`. All entries share one `sequence_id` sequence and each commit takes a ledger-wide write lock (a Postgres advisory lock, so it serializes across processes), which is why one task invocation commits as one transaction — concurrent runs queue behind each other for their ledger writes. A batch that fails rolls back entirely, so its entries never claim sequence numbers and the chain still verifies as intact: `verify_chain()` proves the stored entries were not tampered with, not that everything an ingestion produced was recorded. **Operator note:** enabling this adds relational-DB writes to every `cognify()` run, so plan for disk and backup sizing and for the cost of periodic verification; `verify_chain()` streams the ledger by keyset pagination instead of materializing it client-side, and `get_statistics()` aggregates DB-side, so both are safe to run against a large table — `check()`, the referential-integrity pass, also streams its rows but first loads every entry id and activity id into memory, so its footprint grows with the ledger. See [Provenance ledger](/python-api/cognify#provenance-ledger) (COG-6172, PR #4476).
***
## v1.4.2
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.2)**
Release that bumps the package version from `1.4.1` to `1.4.2`. No new Alembic revision ships in this cut, so no migration is required on upgrade. The entries below are the work promoted from the development branch into this release; work logged under the v1.4.1.dev0 and v1.4.1.dev1 pre-release sections below — including the JSON visualization endpoints (PR #4331) and the cancelled-request session cleanup (PR #4250) — also ships in this release.
### Highlights
* Names raw-text uploads made over a `serve()` connection by content hash. The remote client sent every raw string passed to `remember()` or `add()` — including each string inside a list — under the fixed filename `data.txt`, so all text uploads for a tenant collided on a single remote object and concurrent adds raced the server's content-hash read-back, failing with `FileContentHashingError` 409s. The client now derives the name from the text itself as `text_<md5_hash>.txt`, reusing the same `TextData` namer local ingestion applies to nameless text, so a given string gets the same object name whether it is ingested locally or remotely. File-like uploads are unchanged and still use their own `name` (falling back to `upload`), and the uploaded text content is unchanged. **Operator impact:** tooling or tests that assert the uploaded basename is `data.txt` need updating to expect `text_<md5_hash>.txt`; the client exposes no way to supply your own filename. No public API signature, configuration option, or environment variable changed (PR #4366).
* Clarifies that the Postgres graph store is a demo feature and states where to reach us about production use. The `postgres` graph backend's module and adapter docstrings, and the `graph_database_provider == "postgres"` branch in `get_graph_engine.py`, now say explicitly that it is not production-ready and that production workloads should use a graph-native backend such as Kuzu or Neo4j; the production-ready adapter is a licensed product, and the contact route is now `social@cognee.ai` (or a call with the sales team via [cognee.ai](https://www.cognee.ai)) rather than the previously documented `social@topoteretes.com`. This is a documentation and positioning change only — the adapter's behavior, its `GRAPH_DATABASE_*` configuration and fallback to the relational `DB_*` settings, and its existing limitation that `SearchType.CYPHER` and `SearchType.NATURAL_LANGUAGE` raise `SearchTypeNotSupported` are all unchanged (PR #4366).
* Fixes a connection-pool deadlock in the API-key authentication path that could take the whole API down under concurrent load — surfacing first as intermittent `401`s and gateway timeouts, then as a full outage, with Postgres backends stranded `idle in transaction`. `UserManager.get_by_token()` in `cognee/modules/users/get_user_manager.py` looked up the `UserApiKey` row in its own session and then resolved the user with `self.get()` **while that session was still open**; `self.get()` borrows the request-scoped session behind the FastAPI `get_user_db` dependency, which is held until the response is finished, so it checked out a *second* pooled connection and kept it open for the entire request — across the slow completion call. Every in-flight authenticated request therefore pinned two pooled connections instead of one: a pool of N slots could safely serve only about N/2 concurrent authenticated requests, and at N concurrent requests every slot was held by an API-key lookup waiting for a second connection that could never free — a circular wait that deadlocked the pool (with the relational defaults of `pool_size=5` plus `max_overflow=35`, that ceiling is 40). `get_by_token()` now resolves both rows with two `select()` calls inside a single short-lived session and returns from inside it, never calling `self.get()`, so authentication holds one connection and releases it as soon as the lookup finishes. This is a **separate cause from the cancelled-request cleanup fix logged under v1.4.1.dev0** (PR #4250): the deadlock here reproduces with plain concurrency and no cancellation involved, which is why that earlier change did not resolve #4197. Two workarounds operators may have reached for did not address it and can be reconsidered — enlarging `POOL_ARGS` only raised the concurrency at which the deadlock hit, and `idle_in_transaction_session_timeout` never reclaimed these slots, because they belong to live, in-flight requests rather than abandoned ones. No public API signature, configuration option, or environment variable changed and no migration is required; redeploy to apply (fixes #4197, PR #4354).
* Keeps idle subprocess-backed database engines (embedded Ladybug/Kuzu graph, LanceDB vector) alive for a configurable idle TTL instead of closing them every time a dataset context exits. Previously, the last holder of a dataset's queue slot evicted and force-closed the cached engine on release, so a follow-up request for the same dataset paid for a worker close plus a fresh spawn. The release path now refreshes the engine's idle timestamp, and a background daemon thread (`subprocess-idle-reaper`, started lazily on the first kept-alive release) sweeps the engine caches every `max(5, min(60, TTL / 4))` seconds and force-closes only the engines that have gone a full TTL without use. The sweep skips datasets holding an active queue slot — the same pin that protects them from capacity eviction — so an operation running longer than the TTL cannot lose its engine mid-flight, and it skips engines that are not subprocess-backed, so remote stores (Neo4j, Postgres, PGVector) are unaffected and keep plain LRU behavior. The TTL is set with the new `SUBPROCESS_IDLE_TTL_SECONDS` environment variable, default `600` seconds; negative values are clamped to `0` and fractional values are accepted. **Operator impact:** a kept-warm worker holds its PID, its memory, and — for the file-based graph store — its database file lock for up to the TTL after its last use, so worker PIDs are now stable across requests and a data directory can stay locked longer than before (relevant when another tool or backup job needs to open the same files). The number of retained idle workers is still bounded by the engine cache capacity (`DATABASE_MAX_LRU_CACHE_SIZE`). Set `SUBPROCESS_IDLE_TTL_SECONDS=0` to restore the previous close-at-release behavior; the keep-alive also has no effect when the dataset queue is disabled (`DATASET_QUEUE_ENABLED=false`), because the release path never runs (PR #4384).
* Fixes the remaining code paths that held one pooled relational connection while acquiring a second, which could deadlock a bounded Postgres or PGVector pool. This is a different mechanism from the cancellation leak fixed in #4250: nothing was abandoned mid-transaction, the connections simply *overlapped* — a method opened a session (or an `engine.begin()` connection) and then, inside it, called a helper that checks out a connection of its own, so each such call pinned two at once. Once concurrency reached the pool's ceiling, every waiting task held one connection and waited for another that could not come free, and the stuck backends showed up as `idle in transaction`. The default embedded SQLite store has no bounded shared pool, so this only affected Postgres/PGVector deployments. All affected paths were reordered to resolve the lookup first and open the working session second: in `SQLAlchemyAdapter` (`cognee/infrastructure/databases/relational/sqlalchemy/SqlAlchemyAdapter.py`), `insert_data`, `delete_entity_by_id`, `get_all_data_from_table`, `get_table_names`, and `extract_schema` now resolve `get_table()` / `get_table_names()` / `get_schema_list()` before opening their own connection, since each of those helpers checks one out; `create_role`, `create_tenant`, `add_user_to_tenant`, `select_tenant`, `get_default_user`, `get_deletion_counts`, and `get_document_ids_for_user` resolve their user, tenant, and per-dataset lookups outside the session that does the work; and `get_pipeline_run_metrics` computes its token count on the session it already has open instead of calling `fetch_token_count`, which would have opened a second one. `delete_role`, `remove_user_from_role`, and `get_users_in_role` — where the permission check depends on data read from the first session — split into two sequential sessions with the check running on its own connection in between; the entity lookup still precedes the permission check, so `EntityNotFoundError` / `UserNotFoundError` / `RoleNotFoundError` still take precedence over a permission failure exactly as before, and `get_default_user`'s deferred `create_default_user()` call stays inside the same error handling, so a missing schema still surfaces as `DatabaseNotCreatedError`. Every method keeps its arguments, return value, and exceptions. No public API signature, configuration option, environment variable, or migration changed, and no pool resizing is needed — deploying the release is the whole action (PR #4392, follow-up to #4197).
* Fixes the SQL session-cache engine crashing at startup when `POOL_ARGS` sets `"poolclass": "nullpool"`. `POOL_ARGS` has two consumers: the relational adapter already normalized the string `"nullpool"` to SQLAlchemy's `NullPool` class before building its engine, while `SqlCacheAdapter` read the same relational pool arguments and passed them to `create_async_engine` unchanged — SQLAlchemy inspects `poolclass` as a class, so the string raised `CacheConnectionError: Failed to initialize SQL cache engine for …: 'str' object has no attribute '__dict__'` for both `CACHE_BACKEND="sqlite"` and `CACHE_BACKEND="postgres"`. The cache adapter now applies the same normalization, so one `POOL_ARGS` value works for both consumers — which matters behind an external pooler (for example Neon's `-pooler` pgbouncer endpoints), where disabling client-side pooling is the deliberate choice and the cache engine connects to that same endpoint. **Operator impact:** none beyond upgrading — no environment variable was added or renamed, no configuration change is required, and no migration ships with the fix; deployments that leave `POOL_ARGS` unset or omit `poolclass` are unaffected, and a deployment that had to drop `poolclass` to work around the crash can restore it (PR #4376).
* Fixes an explicitly configured relational `POOL_ARGS` being ignored for per-dataset PGVector engines under `ENABLE_BACKEND_ACCESS_CONTROL="true"`. `PGVectorAdapter` resolved pool arguments as `VECTOR_POOL_ARGS` → built-in access-control default → relational `POOL_ARGS`, so the built-in default (`{"pool_size": 2, "max_overflow": 20}`, which exists to curb connection fan-out when every dataset gets its own engine) beat the operator's own sizing and `POOL_ARGS` silently did nothing in multi-user mode. Precedence is now `VECTOR_POOL_ARGS` → relational `POOL_ARGS` → the access-control default (used only when neither is set, and only while access control is on; an empty pool config otherwise), so explicit configuration outranks the built-in default. **Operator impact:** a deployment that sets `POOL_ARGS` while running with backend access control enabled will see its per-dataset PGVector pools resized to that value after upgrading — larger or smaller than the previous 2/20 depending on what is configured, and multiplied by the number of active datasets — so check the total against the server's `max_connections` and set `VECTOR_POOL_ARGS` to keep PGVector on its own sizing if you want the two to differ. No environment variable was renamed or added and no migration is required; restart or redeploy to pick up the fix (PR #4351).
* Scopes role visibility to membership, so a tenant member can see the roles they belong to and who else is in them without holding tenant-wide user-management permission. `GET /api/v1/permissions/tenants/{tenant_id}/roles` no longer raises `PermissionDeniedError` for callers without that permission: when the tenant exists it now returns `200` (a nonexistent `tenant_id` still returns `404`), with owners and user-management holders (for example, an Admin role) seeing every role in the tenant and everyone else seeing only the roles they are a member of. **Client impact:** the `403` is gone from this endpoint — a caller who previously got `403` now gets a filtered list, and a caller who belongs to no role in the requested tenant (including a `tenant_id` for a tenant they are not part of) gets an empty list, so clients that branched on `403` to detect "not allowed" need to branch on the returned list instead. `GET /api/v1/permissions/tenants/{tenant_id}/roles/{role_id}/users` now returns `200` to members of the role itself; non-members without user-management permission still receive `403`. That endpoint also fixes a cross-tenant scoping hole: the role is now resolved by `(role_id, tenant_id)` rather than by id alone, so a role id from another tenant returns `404` instead of that tenant's member list. No new endpoint, request or response field, permission type, environment variable, or migration ships with this change (COG-6064, PR #4336).
* Fixes two code paths that held one pooled relational connection open while checking out a second, which could deadlock a bounded Postgres/PGVector pool under concurrency. Each in-flight call pinned two connections at once, so once concurrent calls reached the pool's ceiling the waiters formed a circular wait — requests hung rather than erroring, and the stranded backends showed up as `idle in transaction` and then as pool exhaustion. Two sites are fixed: `get_or_create_dataset_database` — the function that creates per-dataset databases under `ENABLE_BACKEND_ACCESS_CONTROL="true"` — wrapped `create_authorized_dataset(...)` in an `async with db_engine.get_async_session()` block that never used `session`, holding that connection idle while the callee opened its own session plus another for the permission grant; the dead wrapper is removed, so the call now runs with no outer session. That branch fired only when a dataset was passed to `get_or_create_dataset_database` by name rather than id — a live path from `add()` and `cognify()` in v1.4.0 and earlier, while since v1.4.1 the database context resolves dataset names to ids before entry, making the removed wrapper a latent hazard rather than a reachable deadlock on the code of the day (the name branch itself was removed in v1.5.4). And `PGVectorAdapter.delete_data_points` called `await self.get_table(collection_name)` *inside* its write session, but `get_table()` opens its own `engine.begin()` connection, so the table lookup is now resolved before `get_async_session()` — exactly as the sibling `retrieve()` and `search()` methods already did. When access control is off, PGVector borrows the relational engine, so the `delete_data_points` overlap contended the pool shared with every relational query; under access control it runs on the per-dataset PGVector engines, which are smaller (`pool_size=2`, `max_overflow=20` by default) and reach the two-connections-per-call ceiling soonest. This produces the same `idle in transaction` signature as the cancellation leak fixed in #4250 and the auth-path deadlock of #4197 (fixed in #4354) but by a different mechanism; the defect class is the same one swept across the relational adapter in #4392. No public API signature, configuration option, environment variable, or database migration changed, and no pool resizing is needed — restart or redeploy to pick up the fix (PR #4389).
* Fixes extraction schemas derived from a custom `graph_model` dropping domain fields that the model inherits from your own `DataPoint` subclasses. `datapoint_model_to_basemodel` — the conversion behind `graph_model_to_graph_schema` and the structured-output schema Cognee hands the LLM — selected fields by reading each class's own `__annotations__`, so only the fields annotated on the leaf class survived: given `Animal(DataPoint)` with `species: str` and `Dog(Animal)` adding `breed: str`, the schema for `Dog` contained `breed` alone and the LLM was never asked to extract `species`. Fields are now taken from the model's merged `model_fields` minus the names defined on `DataPoint` itself, so inherited domain fields are preserved — including required fields, fields with defaults or default factories, nested `DataPoint` fields (converted through the same shared cache, which also terminates cyclic `A → B → A` model graphs), and fields inherited through several levels of subclassing. `DataPoint`'s own infrastructure fields (`id`, `version`, `type`, `created_at`, `metadata`, and the rest) stay excluded, and because the exclusion is by field name a subclass that overrides `metadata` keeps it out of the schema too. **Impact:** the JSON schema returned by `graph_model_to_graph_schema` gains properties for inheritance-based custom models, and extraction for those models now populates the inherited fields, so downstream validators or snapshot tests that pinned the previous leaf-only schema need updating. Flat custom models — every field annotated on the class you pass, as in the guide examples — produce the same extraction schema as before; the one visible difference is in `graph_model_to_graph_schema` output, where a `metadata` override annotated on the class previously leaked into the schema and is now excluded like the rest of the infrastructure fields. No public API signature, configuration option, or environment variable changed, and no migration is required (SDK-161, PR #4373).
* Takes subprocess-engine teardown off the response path for datasets served by subprocess-mode graph or vector databases with the dataset queue enabled. Previously, the last task to release a dataset's queue slot fetched the cached engine and awaited its `close()` inline, so an interactive search or recall waited for the worker process to shut down and drop its file lock before returning. Teardown now routes through the engine cache: the eviction still runs synchronously while the slot is held, so no caller can fetch a dying engine, but the adapter's `close()` runs on the cache's dedicated close threads. File-lock safety is preserved by the cache's pending-close latch instead of by making the caller wait — the next creation of the same engine waits until the previous worker has exited and released its lock, bounded, after which it falls back to the existing `SUBPROCESS_OPEN_LOCK_RETRIES` / `SUBPROCESS_OPEN_LOCK_BACKOFF` open retries. **Operator impact:** lower interactive search and recall latency for subprocess-mode datasets, with unchanged lock safety; a failing engine `close()` now surfaces in the logs with its traceback rather than propagating out of the dataset-context exit, so monitor logs for teardown warnings instead of relying on request errors to reveal them. No configuration option or environment variable changed and no migration is required (PR #4358).
***
## v1.4.1.dev1
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.1.dev1)**
Development pre-release that bumps the package version from `1.4.1.dev0` to `1.4.1.dev1` and updates `uv.lock` to match. The lockfile change records the new `cognee` version only — no dependency versions moved, so no re-lock or reinstall is required for the bump itself. No new Alembic revision ships in this cut; the entries below are the work merged to the development branch since v1.4.1.dev0.
### Highlights
* Adds JSON siblings to the visualization endpoints, so an external dashboard can consume the same graph the built-in HTML page renders. `GET /api/v1/visualize/json` returns the full preprocessed payload — nodes, links, color maps, schema graph and schema data, pipeline stages, edge classes, bundles, provenance index, and memory map — plus `search_events`. `GET /api/v1/visualize/semantic` returns semantic positions and clusters for the same subgraph as a separate call, so a client that never opens the semantic view never pays for the embedding fetch and PCA behind it. `GET /api/v1/visualize/brains` returns every dataset the caller may read as a small `{dataset_id: {"name", "nodes", "links", "node_set_colors"}}` preview, with `max_nodes` (default `500`, maximum `5000`) applied independently to each dataset rather than as one larger shared cap. `GET /api/v1/visualize/live-events` returns search and improve events newer than a `since` cursor, for polling a timeline without re-fetching the whole payload; the filter is strict, so passing the previous response's `cursor` straight back never delivers an event twice. `GET /api/v1/schema/provenance/json` does the same for the memory-provenance graph. The matching Python entry points — `visualize_graph_json`, `visualize_semantic_json`, `build_brains_payload`, `get_live_events`, and `get_memory_provenance_payload` — are exported from `cognee.api.v1.visualize`. Authorization is unchanged and shared: every JSON route runs the same read-permission check as its HTML counterpart, and the HTML and JSON paths were refactored onto one `fetch_visualization_data` / `fetch_dataset_graph_data` pair so the two cannot drift on which subgraph they return. One scoping caveat on `live-events`: `dataset_id` gates who may call the endpoint, but the events themselves are collected per user rather than filtered to that dataset — the same events `/visualize/json` already embeds for that dataset. The existing `GET /api/v1/visualize` and `POST /api/v1/visualize/multi` endpoints are unchanged, and no configuration option or environment variable was added (CLO-401, PR #4331).
* Fixes an explicit relational `POOL_ARGS` being silently ignored by PGVector when backend access control is enabled. `PGVectorAdapter` resolved pool arguments as `VECTOR_POOL_ARGS` → built-in access-control default → relational `POOL_ARGS`, so in multi-user mode the built-in `{"pool_size": 2, "max_overflow": 20}` — a deliberately small default, because each dataset gets its own engine and a large pool fans out as N datasets × `pool_size` — outranked an operator's explicit sizing, which therefore had no effect. Precedence is now `VECTOR_POOL_ARGS` → relational `POOL_ARGS` → the access-control default, with that default applying only when neither is set and multi-user mode is on (and an empty dict otherwise), so explicit configuration beats the built-in. If you set `POOL_ARGS` while running with access control enabled, PGVector now honors it — check that the resulting size is what you want before deploying, since per-dataset connection use can rise above the previous fixed default. No public API signature, configuration option, or environment variable changed (PR #4351).
***
## v1.4.1.dev0
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.1.dev0)**
Development pre-release that bumps the package version from `1.4.0.dev4` to `1.4.1.dev0` and updates `uv.lock` to match. The lockfile change records the new `cognee` version only — no dependency versions moved, so no re-lock or reinstall is required for the bump itself. The release cut introduces no functional code, public API, configuration, or environment-variable changes; the entries below are the work merged to the development branch since v1.4.0.dev4. Most of them require no action, but two entries add Alembic revisions that existing deployments must run when upgrading — `c3d5e7f9a1b2` from the session-context deduplication fix and `d4e6f8a0b2c3` from the connector-credential `workspace_id` addition — see those entries for details.
### Highlights
* Fixes batched triplet reads skipping triplets on the Neo4j and Ladybug graph backends, which could leave Memify's triplet-embedding pass with incomplete coverage. `get_triplets_batch(offset, limit)` is the paginated read behind `create_triplet_embeddings`, and `get_triplet_datapoints` walks the entire graph with one offset loop, advancing the offset by each batch's size until a batch comes back short or empty — which is only exhaustive if every call slices the same ordering. Both Cypher adapters placed `SKIP $offset LIMIT $limit` after `RETURN` on a `MATCH` with no `ORDER BY`, so consecutive pages were cut from an undefined result order and a multi-batch run could miss triplets (and index others twice), leaving the `Triplet_text` collection incomplete while still reporting success — most likely on large graphs, where the default `triplets_batch_size=100` means many pages. Both adapters now sort before paginating, with `ORDER BY` moved into a `WITH` clause ahead of `SKIP`/`LIMIT` so the skip applies to an already-ordered stream: `start_node.id, end_node.id, type(relationship)` on Neo4j and `start_node.id, end_node.id, relationship.relationship_name` on Ladybug. This aligns them with the SQL-backed Postgres and Turso adapters, which already ordered by `(source_id, target_id, relationship_name)` and are unchanged. No migration or user action is required beyond re-running `create_triplet_embeddings` to pick up triplets a previous run missed; the contents of any single batch are unchanged, but which rows land in which page now differs, so callers that paginate `get_triplets_batch` directly and relied on the previous (undefined) order should not assume the old grouping. No public API signature, configuration option, or environment variable changed (PR #4334).
* Adds `opencode` as an agent connection type, accepted by the Python `register()` API and the `POST /api/v1/agents/register` endpoint's `type` field alongside the existing `sdk`/`api`/`mcp`/`claude_code`/`workflow`/`unknown` values, with idempotent registration, listing, and unregistration. The Cognee Cloud dashboard's **Get started** section gains an **OpenCode** connection card that walks through installing the Cognee plugin with `npx @cognee/cognee-opencode setup`; like the Claude Code and Codex cards, it auto-detects a new session — OpenCode sessions are recognized by the `opencode_` prefix the plugin puts on the session id. No existing connection type, API signature, or configuration option changed (PR #4333).
* Extends the opt-in OpenTelemetry layer from spans-only to spans, metrics, and logs for the core memory operations, aligned to memory-semconv v0.1.0. `COGNEE_TRACING_ENABLED=true` (or `enable_tracing()`) now also configures a `MeterProvider` and attaches an OTel log bridge, reusing an externally configured provider when one exists; failures setting up either are swallowed, so a partial OpenTelemetry install degrades to traces-only. `add()`, `cognify()`, `search()`, and `forget()` emit spans named `memory.store`, `memory.process`, `memory.retrieve`, and `memory.delete` carrying `memory.system`, `memory.operation`, and — depending on the operation — `memory.collection`, `memory.query.text` (first 500 characters only, to cap cardinality and bound how much user input reaches your backend), `memory.query.type`, and `memory.result.count`; `add()` is instrumented for the first time. **This renames the existing `cognee.api.search`, `cognee.api.cognify`, and `cognee.api.forget` spans, so dashboards, saved queries, and alerts matching the old names must be updated** — span attributes stay backward compatible, since the `cognee.*` keys are still set alongside the new `memory.*` keys. Six metric instruments are recorded: `memory.operation.duration` (histogram, `ms`; on `forget()` only the delete-everything path records metrics), `memory.items.stored`, `memory.items.retrieved`, `memory.items.deleted`, `memory.query.result.count`, and `memory.vector.searches`; four more (`memory.data.bytes.stored`, `memory.graph.nodes.added`, `memory.graph.edges.added`, `memory.operation.errors`) are registered for custom instrumentation but not yet recorded by Cognee. Metrics and logs are only collected when `OTEL_EXPORTER_OTLP_ENDPOINT` is set (or `console_output=True`) — unlike spans, they have no in-memory buffer — and their endpoints are derived from the traces endpoint by rewriting `/v1/traces` to `/v1/metrics` and `/v1/logs`, overridable with the standard `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` and `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`. Separately, OTLP span export moves from `SimpleSpanProcessor` to `BatchSpanProcessor` (call `disable_tracing()` before exit to flush), and the HTTP-only exporter special case — previously just Langfuse's `/api/public/otel` — now also covers Dynatrace (`dynatrace.com`, `/api/v2/otlp`), port `:4318`, and endpoints with an explicit `:443` port followed by a path (`:443/`), which fixes Dynatrace export silently dropping traces under the gRPC exporter; note that an endpoint spelling out `:443` and including a path now exports over HTTP where it previously used gRPC. Everything remains fully opt-in and a no-op when disabled or when OpenTelemetry is absent, and no new environment variable was introduced (PR #4323).
* Lets Cognee's own errors reach HTTP callers on `POST /api/v1/search`, `POST /api/v1/recall`, `POST /api/v1/remember` (including its COGX archive-import and `/entry` variants), and `POST /api/v1/improve` with their real status code and message. The four routers previously caught Cognee errors per type and rewrapped them into ad-hoc bodies such as `{"error": "...", "detail": "..."}` or `{"error": "...", "hint": "..."}`, sometimes replacing the actual message with unrelated advice — a recall permission failure, for example, returned `403` with a "Recall prerequisites not met" hint telling the caller to ingest and cognify first. The routers now re-raise every `CogneeApiError` subclass so the global handler in `cognee/api/client.py` answers with the error's own status code and the body `{"detail": "<message> [<ErrorName>]"}`. **Status codes are unchanged** (`402`/`403`/`404`/`409`/`422` all come from the exceptions themselves); only the response body shape changed, so clients parsing the legacy `error`/`hint` fields on these endpoints should switch to reading the HTTP status code and `detail` — see [Error Handling](/api-reference/introduction#error-handling). Two adjacent gaps were fixed in the same pass: recall's unreachable `except PermissionDeniedError: return []` arm was removed as dead code (it was always shadowed by an earlier catch, so the `200`-with-empty-list behavior it suggested never shipped), and a permission denial during a remember COGX archive import no longer collapses into a generic `409` "error occurred during COGX archive import". The generic fallbacks for unexpected, non-Cognee errors are unchanged (`500` with `{"error": "Internal server error", "detail": "..."}` for search; `409` with `{"error": "..."}` for recall, remember, and improve), and `POST /api/v1/cognify` and the LLM endpoints still return the legacy `{"error": "Token budget exhausted", "detail": "..."}` body on `402`. No public API signature, configuration option, or environment variable changed (SDK-255, PR #4285).
* Shrinks the `cognee-mcp` server's advertised tool surface so agents no longer load the whole catalog into context. By default, `tools/list` now advertises 8 of the 10 registered tools — the memory API (`remember`, `recall`, `forget`), the three workspace UI entry points (`visualize_graph_ui`, `upload_file_ui`, `open_cognee_workspace`), plus FastMCP's synthetic `search_tools` and `call_tool` — while the structured-JSON workspace helpers are found by calling `search_tools(query=...)` (BM25 ranking, up to 10 results) and invoked by name or through the `call_tool` proxy. Hiding a tool never makes it unreachable: unadvertised tools stay directly callable, which is what keeps the workspace UI (which calls its internals by name) working. The new `COGNEE_MCP_TOOL_MODE` environment variable — also settable per-process with `--tool-mode` — selects `default` (the surface above), `minimal` (only the three memory tools plus the search pair), or `all` (the previous flat catalog with no search transform); unknown values log a warning and fall back to `default`. One sharp edge: `search_tools` matching is purely lexical with no stemming, and zero-scoring tools are dropped, so a terse query like `dataset` misses `list_datasets_json` (token `datasets`) — multi-word natural-language queries are the reliable form. In the same rework, the `cognify_file` tool was removed and folded into `remember`, which now accepts `filename` + `content_base64` (up to 10 MB) to ingest an uploaded file alongside the existing `data` text form; session-cache writes (with `session_id`) remain text-only (PRs #4283 and #4258).
* Makes `cognee-cli config` a working, persistent interface. `cognee-cli config get <key>` and bare `config get` previously printed "not implemented" — they were gated on `hasattr` checks for methods that never existed — and now return real values through the new `cognee.config.get(key)` and `cognee.config.get_all()` APIs, masking secret values (`llm_api_key`, `embedding_api_key`, `vector_db_key`) by default; pass `--show-secrets` on the CLI or `reveal_secrets=True` in Python to print them in plaintext. `config set` and `config unset` previously mutated only the in-process `@lru_cache`d config singletons, so a value vanished the moment the CLI process exited; the CLI now calls `cognee.config.set(key, value, persist=True)`, which additionally writes the value under its resolved environment-variable name to the `.env` file in the current working directory (creating the file if needed) — the same file every config class already reads — so values survive across CLI invocations and are picked up by the next process started from that directory. `unset` resets a key to its default through the same persisted path. The `persist` parameter is new and defaults to `False`, so SDK callers of `config.set()` keep the previous in-memory-only behavior; unknown keys now raise `InvalidConfigAttributeError` on both paths (COG-5970, PR #4287).
* Gives graph nodes a notion of fact validity over time. Every `DataPoint` gains a `valid_to` field (milliseconds-epoch integer, default `None` = still current) — distinct from the temporal stack's Event/Interval `time_to`, which records when an event happened, not whether the fact still holds. The new `close_node(node_id, at_ms=None)` helper (`cognee/tasks/storage/close_node.py`) stamps `valid_to` on an existing node when a fact is superseded — mark "Alice works at X" closed and write the replacement, instead of deleting the old node — and `is_valid(node, at_ms=None)` reports whether a node is still current (`valid_to` is `None` or in the future), accepting both `DataPoint` instances and plain graph-node dicts. Persistence goes through a new optional `update_node(node_id, values)` extension on `GraphDBInterface`, currently implemented only on the default Ladybug (Kuzu) adapter with a read-modify-write that leaves fields the caller did not name untouched; on backends without it, `close_node` logs a warning and returns `False` rather than failing silently (`True` means the node existed and was updated). Closing is last-write-wins — re-closing overwrites the earlier stamp, so guard with `is_valid` first where idempotence matters. No existing API signature, configuration option, or environment variable changed (SDK-200, PR #4105).
* Turns the client-side LLM RPM limiter on automatically when the provider shows evidence it cannot keep up. Previously the limiter was purely opt-in: unless you set `LLM_RATE_LIMIT_ENABLED=true`, Cognee dispatched unbounded and a busy `cognify` run could keep hammering a provider that was already rejecting requests. A new overload policy (`cognee/infrastructure/llm/overload_policy.py`) watches the dispatch seam in `cognee/shared/rate_limiting.py`; when a failed dispatch carries overload evidence anywhere in its exception cause chain — a rate-limit error, a timeout (how overwhelmed local servers surface, since they never send rate limits), or HTTP `429`, `503`, or `529` — it logs one warning naming the cause and the budget, and paces every subsequent dispatch with `LLM_RATE_LIMIT_REQUESTS` / `LLM_RATE_LIMIT_INTERVAL` for a 900-second cooldown. Further evidence inside that window extends it silently; once the window lapses quietly, behavior returns to the configured state, and a fresh episode warns again. Because adapters enter the limiter inside their retry loop, retried attempts are paced too. The new `AUTO_RATE_LIMIT` environment variable (default `true`) controls this; set it to `false` to keep the old fully unbounded behavior, or keep using `LLM_RATE_LIMIT_ENABLED=true` to pace from the first request regardless. Response latency alone is not treated as overload evidence. Relatedly, when `LLM_RATE_LIMIT_REQUESTS` is not set explicitly, serial local inference servers (Ollama, llama.cpp by provider; LM Studio by the `lm_studio/` model prefix) now default to `10` requests per interval instead of `60`, since the cloud default would still flood them; vLLM is deliberately classed as a regular provider and keeps `60`. An explicitly configured `LLM_RATE_LIMIT_REQUESTS` always wins. User impact: runs against a strained provider now slow down instead of failing, at the cost of taking longer; requests already dispatched when the limiter engages cannot be un-queued (PR #4240).
* Raises the default `chunks_per_batch` for the standard Cognify pipeline from `100` to `2000`. `get_default_tasks` in `cognee/api/v1/cognify/cognify.py` uses this value as the `batch_size` for the graph extraction/summarization task and for `add_data_points`, so larger batches mean fewer, bigger chunk-level task batches per run. An explicit `chunks_per_batch` argument and the `chunks_per_batch` value from `CognifyConfig` both still take precedence, and the temporal Cognify pipeline default is unchanged at `10` (PR #4240).
* Scopes the default session to the dataset, so omitting `session_id` no longer funnels every dataset's turns into one shared conversation. `SessionManager.resolve_session_id` (now public, pure, and synchronous — it performs no database lookups) returns an explicit `session_id` unchanged, but when none is given and the manager knows its dataset — from the `dataset_id` constructor argument or the `current_dataset_id` context variable that dataset-scoped `recall()`/`search()` calls enter — it derives `default_session_<dataset_id>` instead of the single global `"default_session"`. With no dataset known the plain global default is still used, exactly as before. Both the read and write sides route through the same function, so an omitted `session_id` resolves identically in either direction within a dataset context. **Behavior change:** history that previously accumulated under the shared `default_session` is now written per dataset; existing `default_session` entries are not migrated and stay readable by passing the literal `session_id="default_session"`. `cognee.session.get_session`'s `session_id` default changed from `"default_session"` to `None` to defer resolution to the manager; a bare call outside any dataset context rebuilds its manager scoped to the caller's existing `main_dataset` (a read-only lookup — no dataset is created) so dataset-scoped writes are readable back without stating a dataset, and raises a `SessionPreconditionError` (a `CogneeValidationError`) when no `main_dataset` exists rather than silently returning the unscoped global session. Relatedly, `set_database_global_context_variables` now accepts exactly `Optional[UUID]` and rejects dataset names and UUID *strings* with a `CogneeValidationError` (`SessionManager` likewise rejects a non-UUID `dataset_id` with `SessionParameterValidationError` instead of degrading to an unscoped session); every production call site already passed a UUID object, but automation or tests that entered the context by dataset name must now resolve the dataset and pass its `.id`. In the CLI, the per-user `scoped_session_id` scheme (`"<user_id>:default"`) is removed from `search`, `recall`, `feedback`, and `sessions` — the dataset-scoped default replaces it, so `cognee search` now passes `session_id=None` and the explicit `--session-id` values you pass to the other commands are used verbatim rather than being prefixed with your user id. No configuration option or environment variable changed (SDK-255, PR #4286).
* Fixes duplicate session-context rows in the SQL session-cache backends (`CACHE_BACKEND=sqlite`, the default, or `postgres`) permanently breaking a session with a 503. `create_session_context_entry` appended a row on every call, so racing update-then-create writers (for example the session persist watermark) accumulated several rows for the same `(user_id, session_id, entry_id)`; `update_session_context_entry` then resolved that key with `scalar_one_or_none()` and raised `MultipleResultsFound`, which surfaced as a per-session 503 that recurred on every subsequent turn of that session. Creates now issue a dialect-specific `ON CONFLICT (user_id, session_id, entry_id) DO UPDATE` upsert (last writer wins, refreshing `payload` and `expires_at`), so those flows converge on a single row, and the update path resolves to the newest row (`ORDER BY seq DESC LIMIT 1`) instead of raising while its `UPDATE` still rewrites every straggler duplicate with the merged payload. The `cache_session_context` table gains a `uq_cache_session_context_entry` unique index on that key — the index the upsert targets. Cache tables are created on init rather than managed by Alembic, so **fresh** databases get the index from the table definition and need no action, but an **existing** deployment must run migrations (`cognee.run_migrations()`, or `alembic upgrade head`) before the upsert has an index to conflict on: new Alembic revision `c3d5e7f9a1b2` deletes all but the newest (`MAX(seq)`) row per key and creates the index, covering both the Alembic-connected database and the standalone SQLite `cache.db` that the default backend keeps next to the relational database. Because the migration mutates data and builds a unique index, it can take time and hold locks on a large `cache_session_context` table — back up the database and run it in a maintenance window. The migration has deliberately no runtime fallback: if `CACHE_DB_URL` points at a separate non-SQLite database it cannot reach, it raises `RuntimeError` naming that URL and printing the `DELETE` / `CREATE UNIQUE INDEX` statements to apply there manually before re-running, rather than letting a deployment proceed against a cache database the upsert would break on. No public API signature, configuration option, or environment variable changed (fixes #4226, PR #4232).
* Fixes cancelled requests leaking Postgres connections stuck in `idle in transaction`, which could exhaust the connection pool until every query failed. `SQLAlchemyAdapter.get_async_session()` in `cognee/infrastructure/databases/relational/sqlalchemy/SqlAlchemyAdapter.py` ended in `finally: await session.close()`, which runs before the `async_sessionmaker` context manager's own exit; when the surrounding task was being cancelled — a client disconnect, a timeout, or any `asyncio` cancellation — that `await` was interrupted before the `ROLLBACK` reached Postgres, so the connection went back to the pool still inside an open transaction. Enough of those piled up and the pool hit its ceiling, at which point requests began failing across the board (authentication included, since the API-key lookup is itself a DB query). The explicit `close()` is removed and replaced with an `except asyncio.CancelledError` branch that calls `await asyncio.shield(self._discard_cancelled_session(session))` and then re-raises; the new `_discard_cancelled_session()` static helper calls `await session.invalidate()`, which drops the DBAPI connection from the pool without needing the `ROLLBACK` round-trip that cancellation keeps interrupting, and logs (rather than raises) if the invalidation itself fails. The `asyncio.shield` keeps the cleanup from being cut short by the same cancellation. Successful requests and ordinary exceptions are unaffected — the sessionmaker's own context manager already closes and rolls back on those paths. No public API signature, configuration option, environment variable, or database migration changed; no pool-sizing changes are needed beyond deploying the fix (fixes #4197, PR #4250).
* Fixes Zep/Graphiti imports silently dropping scope from every entity node and every fact when the export spells the scope key `session_id` rather than `group_id`. In `ZepSource` (and its `GraphitiSource` alias), the episode branch already resolved the record's scope as `group_id or session_id`, but the entity and fact branches built `COGXScope(session_id=...)` from `group_id` alone — so a single export using the `session_id` spelling imported with scope intact on its episodes and empty on all of its nodes and edges, with nothing raised. Both branches now apply the same fallback (`node.get("group_id") or node.get("session_id")`, `edge.get("group_id") or edge.get("session_id")`), and `group_id` keeps precedence where a record carries both, matching the episode branch. User impact: exports already keyed on `group_id` are unaffected and import exactly as before; if you imported a `session_id`-keyed Zep or Graphiti export — most consequentially into a multi-tenant graph, where entities and facts lost their tenant attribution — re-run that import to restore scope on the affected nodes and edges. No public API signature, configuration option, environment variable, or migration changed (PR #4293).
* Fixes Mem0 and LangMem imports landing zero memories when the payload carries an accepted wrapper alias that is present but empty ahead of a populated one. `Mem0Source._load_raw` and `LangMemSource._load_raw` unwrapped on the *first* accepted alias that happened to be a list — `results`, `memories`, `items` for Mem0, and `memories`, `results`, `items`, `data` for LangMem — so a payload like `{"results": [], "memories": [...]}` unwrapped to the empty `results` list and never read the populated alias, and the import still reported success with nothing imported. Both sources now scan their alias order and return the records under the first alias that actually carries any, filtering to dict items before deciding an alias is populated — the same rule `ZepSource`'s `_first_list` helper already applied. Error behavior is preserved: a dict where none of the accepted aliases maps to a list, and a payload that is not a list, both still raise `ValueError`, while a recognized wrapper whose aliases are all empty still yields zero records without raising. User impact: payloads whose first accepted alias was already the populated one are unaffected and import exactly as before; if a Mem0 or LangMem import reported success but landed zero memories, re-run that import. No public API signature, configuration option, environment variable, or migration changed (PR #4314).
* Fixes `improve()` silently retargeting the caller's default dataset when the `dataset` argument could not be resolved. Previously an internal `_resolve_dataset_name` helper returned `"main_dataset"` for any unresolved reference, so a mistyped, nonexistent, or unauthorized dataset UUID quietly enriched the caller's own default dataset instead of failing, and individual stages re-resolved the dataset independently. The target dataset is now resolved and authorized once, up front, through the same `resolve_authorized_user_datasets` layer that `remember()` and `memify()` use: a UUID the caller does not hold `write` permission on raises `PermissionDeniedError` — read permission is not enough, since every improvement stage writes — and a UUID that does not exist raises the identical error, so dataset existence is never confirmed to an unauthorized caller. Dataset **names** keep their owner-scoped semantics: a name is resolved — or created — in the caller's own scope, so improving a name another user happens to own creates the caller's own dataset with that name rather than touching theirs; a dataset owned by someone else must be targeted by UUID. The session-bridging stages now receive the resolved dataset UUID instead of re-resolving a name. User impact: if your integration passed `improve()` a bad or unauthorized dataset reference and relied on the silent `main_dataset` fallback, the call now raises `PermissionDeniedError` — catch it and pass a UUID you hold `write` permission on. The `improve()` signature, configuration options, environment variables, and migrations are unchanged (SDK-255, PR #4294).
* Adds an optional `workspace_id` owner dimension to the third-party connector credential store, so several users can share one connection instead of each connection belonging to exactly one user. `upsert_credential` in `cognee/modules/integrations/credentials.py` gains a keyword-only `workspace_id: Optional[UUID] = None`; because the `integration_credentials` table is keyed on the external account (`UNIQUE(provider, provider_account_id)`), a reconnect by a *different* owner while the current connection is still active raises `CrossUserConflictError` rather than silently taking the connection over, and `workspace_id` now decides who that owner is. When it is passed, the conflict comparison runs on `workspace_id` — two different users in the same workspace can reconnect the same external account, while a different workspace is still refused — and `user_id` continues to record which user actually connected it. Omitted (the default), the ownership and conflict rules are exactly the previous single-user contract, compared on `user_id`. A companion `get_active_credential_for_workspace(workspace_id, provider)` mirrors the existing `get_active_credential_for_user(user_id, provider)` for the new dimension, filtering on `status == "active"` and ordering newest-first; use it for connections upserted with a `workspace_id`, since `user_id` on those rows is only who connected them, not the owner. The `IntegrationCredential` model gains a matching `workspace_id` column — `nullable`, indexed, and deliberately **not** a foreign key (a plain opaque id, same as `user_id`, so there are no cascade semantics) — backed by new Alembic revision `d4e6f8a0b2c3` (on top of `c3d5e7f9a1b2`), which adds the column and the `ix_integration_credentials_workspace_id` index. An **existing** deployment must run migrations (`cognee.run_migrations()`, or `alembic upgrade head`) to pick up the column; the revision is additive and both its `upgrade()` and `downgrade()` inspect the live schema first, so re-running either is a no-op. Existing rows keep `workspace_id` as `NULL` and behave as before. This module is internal plumbing rather than part of the public SDK: no public API signature, configuration option, or environment variable changed (CLO-406, PR #4318).
***
## v1.4.0.dev4
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.0.dev4)**
Development pre-release that bumps the package version from `1.4.0.dev3` to `1.4.0.dev4`; the accompanying `uv.lock` change records the new version only. No new Alembic revision ships in this cut. Relative to v1.4.0.dev3 this build also restores the v1.4.0.dev2 changes (including the OAuth integrations framework and its `b2c4d6e8f0a1` migration) that the dev3 release branch did not carry.
### Highlights
* Adds an `embedding_max_concurrent_data_points` setting (default `150`, env var `EMBEDDING_MAX_CONCURRENT_DATA_POINTS`, also settable via `cognee.config.set_embedding_config({"embedding_max_concurrent_data_points": ...})`) controlling how many data points may be in flight to the embedding engine during indexing. `index_data_points` in `cognee/tasks/storage/index_data_points.py` previously capped concurrency with a hardcoded `asyncio.Semaphore(4)` — four concurrent batches regardless of batch size; the semaphore is now sized `max(1, embedding_max_concurrent_data_points // batch_size)`. The default changes effective throughput: with the LiteLLM engine's default `batch_size=100`, indexing now runs one batch at a time (about 100 points in flight) instead of four (about 400) — raise `EMBEDDING_MAX_CONCURRENT_DATA_POINTS` to restore or exceed the old parallelism, or lower it to stay under provider rate limits. `embedding_batch_size` and the embedding rate-limit settings are unchanged, and no public API signature changed (SDK-285, PR #4144).
* Fixes the `dry_run=True` cost estimator applying the `ACCEPT_LOCAL_FILE_PATH` gate inconsistently across platforms. In `cognee/modules/cognify/estimator.py`, `_path_candidate` decided whether an existing file path was *absolute* — and therefore rejected when the flag is disabled — with a bare `value.startswith("/")` check. That check does not describe absolute paths on Windows, so a genuine Windows path like `C:\data\notes.txt` was not recognized as one: with `ACCEPT_LOCAL_FILE_PATH=False` it slipped past the gate and the path string itself was priced as raw text instead of raising, while conversely a `/`-prefixed string — which Windows resolves to a *drive-relative*, unusable path — was rejected even though a real run ingests it as text. Detection now matches `save_data_item_to_storage` exactly: the string must look absolute for the current platform (`/`-prefixed, or drive-lettered when `os.name == "nt"`) **and** satisfy `Path(os.path.normpath(value)).is_absolute()`. On Windows, `C:\...` paths that exist now raise `Local files are not accepted` when the flag is disabled, and drive-relative `/`-prefixed strings fall through to raw text; POSIX behavior is unchanged. Existing relative paths and strings that are not existing files remain raw text on every platform, and enabling the flag (the default) is unaffected. The rest of the PR is unit-test repair with no runtime effect — a missing `SAMPLE_ARGUMENTS` entry for `ProviderNotDeducibleError` in the error-contract test, and an autouse fixture that snapshots and restores patched module attributes so mocks stop leaking between test modules. No public API signature, configuration option, environment variable, or migration changed (PR #4257).
* Fixes Letta imports silently dropping every message — and the whole conversation episode — when a message serializes `content` as `null` and carries its text under the `text` alias. In `LettaSource`, `_message_text` resolved the alias with `message.get("content", message.get("text"))`, but `dict.get`'s default only fires on a *missing* key: a Letta serializer that writes unset fields as `null` instead of omitting them produced `content: None`, so the `text` fallback never ran and the message resolved to `""`. The caller skips any message with no text, and because every message in one agent file shares the same serialization, the agent's turn list ended up empty and the `if turns:` gate emitted no `COGXEpisode` at all — the import still reported success. The fallback is now explicit (`content = message.get("content")`, then `if content is None: content = message.get("text")`), so those messages and their episode are imported. Files that were already importing correctly are unaffected, and an explicitly empty `content` (`""`) is still treated as a message with no text rather than falling through to `text`. User impact: if you imported Letta agent files and found conversation history missing, re-run those imports to recover the episodes. No public API signature, configuration option, environment variable, or migration changed (PR #4261).
* Fixes `SearchType.CYPHER` and `SearchType.NATURAL_LANGUAGE` failing at query time on installations that do not have the optional `postgres` extra. Both retrievers began by checking whether the active graph engine is one of the Postgres graph backends, which they reject with `SearchTypeNotSupported` because those backends cannot run Cypher. That check imported the Postgres adapters (`PostgresAdapter`, `PostgresHybridAdapter`), and those adapter modules import `asyncpg` at module scope — a dependency that ships only with `cognee[postgres]` — so on an installation without the extra the check itself raised `ImportError`/`ModuleNotFoundError` rather than the backend it was trying to detect. In `NaturalLanguageRetriever.get_retrieved_objects` the import was unguarded, so the error propagated to the caller; in `CypherSearchRetriever.get_retrieved_objects` it sat inside the block whose handler wraps unexpected exceptions, so it surfaced as a misleading `CypherSearchError`. The check no longer imports backend adapters at all: Cypher capability is now declared on the adapter class itself — `GraphDBInterface` defines `supports_cypher_queries = True`, the Postgres, Postgres hybrid, and Turso adapters override it to `False` — and both retrievers read the flag off the active engine, raising `SearchTypeNotSupported` (naming the rejected adapter class) when it is false. User impact: on a Cypher-capable backend such as Kuzu or Neo4j, both search types now work without installing `cognee[postgres]`; installations that do have the extra are unaffected, and Postgres graph backends still raise `SearchTypeNotSupported` as before. The Turso graph backend — whose `query()` also runs SQL rather than Cypher — is newly covered by the same check, so it now raises `SearchTypeNotSupported` up front instead of failing when the generated Cypher reaches it. No public API signature, configuration option, environment variable, or migration changed (fixes #4123, PRs #4124 and #4274).
***
## v1.4.0.dev3
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.0.dev3)**
Development pre-release that bumps the package version from `1.4.0.dev2` to `1.4.0.dev3` and re-resolves `uv.lock`. Unlike the other pre-releases in this line it was cut from a release branch rather than the tip of the development branch, so it carries the deployment and Ollama-validation work below while omitting some changes already shipped in v1.4.0.dev2 — most notably the OAuth integrations framework and its `b2c4d6e8f0a1` migration are absent from this build; both are included again in v1.4.0.dev4.
### Highlights
* Adds an advisory Ollama model-support warning and ships S3 support in the default Docker image. When `LLM_PROVIDER="ollama"`, a new `cognee/infrastructure/llm/ollama_support.py` helper classifies the configured `LLM_MODEL` against a built-in support matrix as the LLM configuration is validated and logs a warning for models Cognee has not validated for structured graph extraction: `llama3` through `llama3.3` and `qwen2.5` tags of 14B or larger are recommended (nothing logged); `mistral`, `phi3`, `phi3.5`, and `qwen2.5` below 14B or with no parseable size warn about known limitations (schema validation errors, silent drops); every other model warns that it is unvalidated and extraction quality may vary. The check is advisory only — nothing is blocked, no exception is raised, and each distinct `LLM_MODEL` value warns at most once per process. Matching ignores an `ollama/` prefix and everything after the `:` in a tag (except `qwen2.5`, whose tag is parsed for a parameter count), and the message points at the new `docs/ollama_models.md` in the repo; see [Model Support Warning](/setup-configuration/llm-providers#model-support-warning) on the LLM Providers page for the full matrix. Separately, the root `Dockerfile` now includes the `aws` extra (`s3fs[boto3]`) in both `uv sync` steps, so the source-built API image supports [S3 file storage](/guides/s3-storage) out of the box without passing `COGNEE_EXTRAS="aws"`. The access-control handler compatibility check also now accepts the `ladybug`/`kuzu` provider aliases interchangeably — both handler names register the same embedded Ladybug (formerly Kuzu) handler and both provider names satisfy its check — so mixing the current and legacy names no longer raises `EnvironmentError` at startup under backend access control. The branch also adds runnable examples: `examples/demos/local_ollama_example.py` (fully local add → cognify → search with Ollama; since moved to `examples/guides/local_ollama_example.py`) and `examples/tutorials/migrate_from_mem0_tutorial.py` with a bundled sample export (since moved to `examples/demos/ingestion_and_migration/migrate_from_mem0/migrate_from_mem0.py`). No public API signature, configuration option, or environment variable changed (PR #4272).
***
## v1.4.0.dev2
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.0.dev2)**
Development pre-release that bumps the package version from `1.4.0.dev1` to `1.4.0.dev2`; the `uv.lock` change records the new version only. One entry requires action: the new OAuth integrations framework ships Alembic revision `b2c4d6e8f0a1`, which existing deployments must run when upgrading (`cognee.run_migrations()`, or `alembic upgrade head`) — see that entry for details.
### Highlights
* Adds an opt-in `memify` pipeline, `consolidate_entities`, that merges near-duplicate `Entity` nodes left behind by repeated `cognify` runs or multi-source ingestion (e.g. "New York City" vs "NYC", whose name-derived ids never collapse on their own). `consolidate_entities_pipeline` (in `cognee/memify_pipelines/consolidate_entities.py`) runs two new tasks exported from `cognee.tasks.memify`: `detect_entity_duplicates` embeds each entity name and clusters candidates by cosine similarity (`similarity_threshold=0.85` default, `top_k=10` neighbors) plus normalized-name equality (`name_match=True`), and `merge_entity_duplicates` picks the canonical node (most connected, ties broken by oldest then name), re-points every edge onto it with direction preserved, unions descriptions, records a `merged_from` report, then deletes the duplicate nodes and their `Entity_name` vector embeddings. Merging stays within one `EntityType` unless `allow_cross_type=True`; `protect_node_types` excludes types entirely, and `dry_run=True` logs the plan without mutating anything. The pipeline only runs when invoked — existing pipelines, public APIs, configuration options, and environment variables are unchanged (PR #3534).
* Fixes `cache_root_directory` in `BaseConfig` (`cognee/base_config.py`) silently accepting relative paths. The sibling root directories — `data_root_directory`, `system_root_directory`, and `logs_root_directory` — were already normalized through `ensure_absolute_path()`, which resolves absolute paths, passes `s3://` URLs through untouched, and raises `ValueError` on a relative path; `cache_root_directory` alone skipped that check, so a relative cache path would resolve differently depending on the current working directory, a hard-to-debug misconfiguration. The one-line change adds the same `ensure_absolute_path()` call for `cache_root_directory`, placed after the S3 auto-config logic so S3 cache paths keep working. Action needed only if you currently set a relative `cache_root_directory`: it now raises `ValueError` at configuration time and must be an absolute path (or an `s3://` URL). No new configuration option, environment variable, public API signature, or migration is involved (PR #3581).
* Adds a `COGNEE_PROVENANCE_MODE` environment variable (also readable from `.env`) controlling provenance stamping on DataPoints during pipeline runs, backed by the new `cognee/modules/pipelines/provenance_config.py` (`ProvenanceConfig`, cached via `get_provenance_config()`). Valid values are `lightweight` (default), `deep`, and `disabled`; values are lowercased, and an unknown value logs a warning and falls back to `lightweight`. In the shipped gating, `disabled` skips the `_stamp_provenance` call in `run_tasks_base.py` entirely as a zero-overhead escape hatch, while `lightweight` and `deep` both run the existing stamping (the config exposes `is_lightweight()`/`is_deep()` helpers, but no code path yet branches on them). PR #3775 as merged left `run_tasks_base.py` half-edited — a broken two-argument `_stamp_provenance(result_data, pipe_name)` call raised `TypeError: missing 1 required positional argument: 'task_name'` on every pipeline run alongside the surviving ungated call — and PR #4215 (SDK-317) repaired this before release with a single full-argument call gated on `get_provenance_config().is_disabled()`. Default behavior is unchanged; no action needed unless you want `disabled` (PRs #3775 and #4215).
* Fixes `LangchainChunker` (`cognee/modules/chunking/LangchainChunker.py`), which could not be instantiated at all after the chunker API unification even though it is user-facing — documented in the `cognify()` docstring and offered among the CLI chunker choices — so selecting it crashed mid-pipeline. Three stacked breaks are repaired, re-landing the alignment from PR #2966 that was reverted in #3167: the constructor parameter is renamed from `max_chunk_tokens` to `max_chunk_size` (every `Document.read()` call passes `max_chunk_size=`, which previously raised `TypeError`), `super().__init__` now passes the three arguments the `Chunker` base actually accepts, and `read()` checks `self.max_chunk_size` and emits `DocumentChunk` with `chunk_size=token_count` plus `importance_weight` propagation — matching `TextChunker` — instead of the undefined `word_count`/`token_count` fields while omitting the required `chunk_size`. New regression tests are guarded with `pytest.importorskip("langchain_text_splitters")` so environments without the `langchain` extra skip cleanly. `TextChunker` behavior, configuration options, and environment variables are unchanged (fixes #3888, PR #3893).
* Changes LLM configuration to reduce required setup: `LLM_PROVIDER` is now optional and inferred from the `llm_model` prefix (e.g. `anthropic/claude-...` implies `anthropic`) via a new validator in `cognee/infrastructure/llm/config.py`; an explicit provider — kwarg or env var — always wins, and a prefix outside `KNOWN_LLM_PROVIDERS` raises `ProviderNotDeducibleError` with a message naming the supported providers and pointing litellm-routed prefixes (e.g. `openrouter/`, `groq/`, `deepseek/`) at `LLM_PROVIDER="custom"`. Behavioral change requiring action: such prefixes previously defaulted to `openai` passthrough and now fail fast until you set `LLM_PROVIDER="custom"`. Additionally, a central `instructor_modes.py` table replaces the `default_instructor_mode` values duplicated across nine adapters; the `embedding_rate_limit_*` fields move from `LLMConfig` to `EmbeddingConfig` (field and env-var names unchanged); quote-stripping now covers all declared string fields instead of a 14-field allow-list; and the Ollama validator no longer checks embedding env vars, leaving that to `EmbeddingConfig` (SDK-142, part of #3382, PR #3994).
* Adds the catalog contract underpinning the planned Integrations Hub and Use-Case Gallery: a new top-level `catalog/` directory in the cognee repo with a draft-07 JSON Schema (`catalog/schema.json`), a validating loader (`catalog/loader.py`, runnable as `python -m catalog.loader`) that applies three passes — schema validation, naming rules, local path resolution — with aggregated errors, ten seed YAML entries under `catalog/entries/` spanning all three kinds (integrations such as `claude-code` and `langgraph`, packages such as `qdrant` and `weaviate`, and use-cases such as `agent-memory` and `temporal-reasoning`), an advisory cross-repo drift check (`catalog/inventory_sync.py`) against the `cognee-integrations` repository's `inventory.yml`, a `Catalog` CI workflow (`.github/workflows/catalog.yml`), and a contributor guide (`docs/contributing/add-catalog-entry.md`). This is chunk 1 of issue #3603; Hub/Gallery rendering and full cross-repo aggregation are deferred to follow-ups. The catalog tooling is not shipped in the wheel, so the installed `cognee` package, public APIs, configuration options, and environment variables are unchanged (SDK-151, PR #4023).
* Adds opt-in temporal contradiction resolution for functional (single-valued) relationships to the cognify pipeline. When a relationship like `ceo_of` should hold only one target per subject but the graph accumulates several — e.g. one document names Alice as CEO and a later one names Bob — cognee previously kept both as current facts. Passing the new keyword `cognify(functional_relationships={"ceo_of"})` appends a `resolve_temporal_contradictions` task (in `cognee/tasks/graph/resolve_temporal_contradictions.py`) that runs last, after the graph is written: for each declared relationship it groups stored edges by subject, keeps the most recent assertion (by `updated_at`) as current, and tags the older ones with `superseded=True`, `superseded_by` (the winner's `edge_object_id`), and `supersession_reason` — nothing is deleted and provenance is preserved. The core, `tag_superseded_edges` in `cognee/modules/graph/utils/temporal_conflict_resolver.py`, is deterministic with no LLM call, resolution runs against the stored graph so a fact ingested today supersedes one from last month, and re-runs are idempotent. The parameter defaults to `None`, so the default pipeline, `add_data_points`, and all existing behavior are unchanged; no environment variable or migration is involved (SDK-184, PR #4084).
* Adds a Graph Insight Report that describes a built knowledge graph instead of only drawing it: a Markdown report covering hub nodes ("god nodes", ranked by degree plus `networkx.pagerank` over the entity/entity-type layer, with a degree-only fallback), surprising cross-set connections (entity pairs whose endpoints belong to different `node_set`s, resolved through `belongs_to_set` edges), edge provenance (EXTRACTED entity-to-entity relationships vs DERIVED chunk/document scaffolding), and LLM-suggested follow-up questions ready to pipe into `search()` — the one section that costs an LLM call. Exposed three ways: a new public `cognee.report(datasets="main_dataset", output_path="graph_report.md", top_n=10)` API in `cognee/api/v1/report/report.py`, a new `SearchType.GRAPH_REPORT` backed by `GraphReportRetriever` in `cognee/modules/retrieval/graph_report_retriever.py`, and a `cognee-cli report` command. The feature is additive-only: it reads solely via `get_graph_data()`, so there are no schema, storage, or migration changes, and no existing API signature, configuration option, or environment variable changed (SDK-188, PR #4101).
* Adds opt-in contradiction detection between newly ingested facts and facts already stored in the graph. When enabled, cognify appends a `detect_contradictions` task (in `cognee/tasks/graph/detect_contradictions.py`) that runs last: it collects the entities the current ingestion touched, fetches their 1-hop stored neighbourhood via `get_neighborhood`, asks the LLM which fact pairs conflict, and records each contradiction as a queryable `contradicts` edge carrying both facts, the reason, and a confidence score, alongside a logged warning — nothing is deleted or overwritten, and the task swallows its own errors so detection can never break ingestion. It is enabled through configuration rather than a new function argument: `CONTRADICTION_DETECTION=true` on `CognifyConfig` (`cognee/modules/cognify/config.py`), tuned by `CONTRADICTION_CONFIDENCE_THRESHOLD` (default 0.5) and `CONTRADICTION_MAX_FACTS` (default 500); it also applies to `remember()`, which builds its graph through `cognify()`. Default off — with the flag unset the pipeline task list and behavior are identical to before, and no public API signature changed (SDK-199, PR #4104).
* Fixes three `CogneeError` subclasses that overrode `__init__` without calling `super().__init__()`, leaving `Exception.args` empty — which broke `repr()`, exception chaining (`raise ... from cause`), and the centralized logging in `CogneeApiError.__init__`. The offenders were `EntityNotFoundError` and `NodesetFilterNotSupportedError` in `cognee/infrastructure/databases/exceptions/exceptions.py` and `WrongTaskTypeError` in `cognee/modules/pipelines/exceptions/tasks.py`; an AST sweep confirmed these were the only three in the repo. All now call `super().__init__(message, name, status_code)`; the two database exceptions pass `log=False` because they are raised in routine control flow (user resolution, migrations, pruning, graph projection), so `args` and chaining are restored without flooding logs with ERROR lines for expected conditions, while `WrongTaskTypeError` — a genuine programming error — keeps default logging. A new regression guard, `cognee/tests/unit/exceptions/test_cognee_error_contract.py`, enforces the contract via both a static AST sweep and a runtime check of every importable subclass. Constructor signatures, exception types, and raising sites are unchanged, so no caller needs action (SDK-240, fixes #3749, PR #4151).
* Fixes ingestion turning a `/`-prefixed string that is not an actual file — e.g. a text note like `/remember to call Bob about the meeting` — into a broken `file://` URI pointing at a non-existent path, which the loader later failed on. The absolute-path branch of `save_data_item_to_storage` (in `cognee/tasks/ingestion/save_data_item_to_storage.py`) converted unconditionally; it now also requires `abs_path.is_file()`, mirroring the relative-path branch, so an absolute-looking string becomes a `file://` URI only when it points at an existing file and otherwise falls through to text ingestion on every platform. Existing absolute file paths still convert as before, and `ACCEPT_LOCAL_FILE_PATH=false` still rejects existing local files with `IngestionError`. The dry-run estimator's `_path_candidate` (`cognee/modules/cognify/estimator.py`), which mirrors this routing, is updated to price a missing absolute path as text instead of raising "file does not exist". Side effect of keying on "existing file": directory paths, broken symlinks, and pathological paths also route to text rather than a broken URI. Follow-up to #3892; no public API signature, configuration default, or migration changed (SDK-234, fixes #3887, PR #4155).
* Adds a reusable OAuth integrations framework to the cognee server, with Slack as the first provider. The framework (`OAuthIntegration` ABC and registry in `cognee/modules/integrations/`, a generic per-provider connect/callback/disconnect router in `cognee/api/v1/integrations/routers/get_integrations_router.py`) stores third-party credentials AES-256-GCM-encrypted in a new `integration_credentials` table, with key-rotation support via the `INTEGRATION_CREDENTIALS_KEYS` / `INTEGRATION_CREDENTIALS_ACTIVE_KEY_ID` env vars (legacy single-key `INTEGRATION_CREDENTIALS_KEY` still works). The Slack integration adds workspace connect/disconnect with token refresh and revoke, a `/cognee-ask` slash command answering via `HYBRID_COMPLETION` search through Slack's `response_url`, a per-channel allowlist, an App Home tab, a "Remember this" message shortcut that saves messages via `cognee.remember()`, request verification via `SLACK_SIGNING_SECRET` (plus `SLACK_`-prefixed app credentials), and a matching frontend Integrations page. The table is created by new Alembic revision `b2c4d6e8f0a1`, so existing deployments must run the migration (`cognee.run_migrations()`, or `alembic upgrade head`) to create it; the revision is guarded and skips if the table already exists. The core SDK pipeline (`add`/`cognify`/`search`) is unchanged (PR #4175).
* Fixes `cognee-cli --api-url <cloud-url>` commands failing against reachable cloud tenants with a bogus "Cannot connect to Cognee API… Is the server running?" error. `api_dispatch.dispatch()` ran a fatal pre-flight `client.health()` probe before every command, using an unauthenticated one-off `httpx` client against the DB-backed `/health` endpoint and mapping any exception — including 503, 404, and 401 responses — to that message. The probe is removed: the real command runs immediately, and only genuine transport failures (classified by the new `is_connection_error()` helper, matching `httpx.TransportError`) are translated into a friendly message that includes the attempted URL; real HTTP errors such as 401 now surface their actual detail. Dataset collection calls (`datasets_list`, `datasets_create`, `datasets_delete_all`) now use the canonical trailing slash on `/api/v1/datasets/` and the shared client sets `follow_redirects=True`, so both cloud and local OSS deployments work; `health()` remains available as an explicit check, reuses the pooled authed client, and returns the body on 503 instead of raising. No CLI flags, config options, or environment variables changed (CLO-321, PR #4189).
* Adds an opt-in, server-side default synthesis prompt for the MCP `recall` tool. Previously the per-call `system_prompt` argument (added in PR #4122) was the only way to override the terse `RecallPayloadDTO` default, so a shared cognee deployment could not define a default synthesis policy — every MCP client had to resend the full prompt on each call. `CogneeClient.recall` in `cognee-mcp/src/cognee_client.py` now resolves a default when the caller omits `system_prompt`: `COGNEE_MCP_RECALL_SYSTEM_PROMPT` supplies inline prompt text, and `COGNEE_MCP_RECALL_SYSTEM_PROMPT_FILE` points to a file holding it (inline takes priority; an unreadable file logs a warning and is skipped). Precedence is explicit caller argument, then env default, then backend default, and the resolution happens before the API/SDK branches so both modes are covered. With neither variable set, behavior is unchanged — no `system_prompt` is added to the payload — and the `search` tool is deliberately unaffected (PR #4205).
* Fixes `forget(everything=True)` appearing to hang for 15+ minutes when deleting thousands of vector rows from LanceDB, the default vector backend. `LanceDBAdapter.delete_data_points` issued one `collection.delete` per id; each delete is a LanceDB commit that appends a table version, and manifest listing degrades as versions accumulate, so a 13,207-id wipe became 13k increasingly slow commits (\~7 deletes/sec and falling) that no per-call timeout or worker-death check could catch. Deletes now run as `id IN (...)` predicates in batches of 1,000, controlled by the new `DELETE_PREDICATE_BATCH_SIZE` class attribute; per-id single-quote escaping is preserved, sequential batches from one caller cannot commit-conflict, and missing ids or collections remain no-ops. The measured 13,207-id delete drops from 30+ minutes to about 0.1 seconds. Other providers (PGVector, Turso, hybrid Postgres, Neptune) already batched and are unchanged; no public API signature, configuration option, or environment variable changed (PR #4235).
* Fixes Ollama model name handling that persisted after PR #3994's provider inference: when `LLM_MODEL` is set to a litellm-style prefixed name such as `ollama/llama3.1:8b`, the provider was correctly inferred as Ollama, but `OllamaAPIAdapter` stored the model name verbatim and sent the full prefixed string to Ollama's OpenAI-compatible endpoint, where no model by that name exists. The adapter's `__init__` (in `cognee/infrastructure/llm/structured_output_framework/litellm_instructor/llm/ollama/adapter.py`) now strips the prefix with `model.removeprefix("ollama/")` when the name starts with `ollama/`, so both `llama3.1:8b` and `ollama/llama3.1:8b` resolve to the bare model name `llama3.1:8b`. Unprefixed model names behave exactly as before, and no public API signature, configuration option, or environment variable changed; new unit tests cover both spellings and the inferred-provider path end to end (PR #4243).
* Fixes a `TypeError` when the shared storage `JSONEncoder` encountered a plain `datetime.date` value. JSON Schema fields declared with `format: "date"` are materialized as `datetime.date` objects, but `JSONEncoder.default` in `cognee/modules/storage/utils/__init__.py` only handled `datetime`, `UUID`, and `Decimal`, so dates fell through to the standard library encoder and raised. A new `isinstance(obj, date)` branch now serializes plain dates with `date.isoformat()` as ISO-8601 strings; it is placed after the existing `datetime` check (which matches first, since `datetime` is a `date` subclass), so datetime serialization is byte-for-byte unchanged, as are UUID and Decimal handling, and unsupported objects still raise `TypeError`. No public API signature, configuration option, or environment variable changed (fixes #4239, PR #4244).
* Fixes `parse_timestamp` in `cognee/modules/migration/cogx.py` — the helper feeding all five migration source adapters (Mem0, Zep, Graphiti, Letta, LangMem) and `export.py` — returning a mix of timezone-aware and naive datetimes: epoch values and ISO strings with a `Z` or offset came back UTC-aware, while offset-less ISO strings, bare dates, and naive `datetime` passthrough stayed naive. `LettaSource` can see both shapes in one conversation (Letta serializes `created_at` with or without an offset depending on version), which made `render_episode`'s `.timestamp()` sort apply the importing machine's local timezone — reordering transcript turns on any machine not set to UTC — and made direct naive/aware comparisons raise `TypeError`. `parse_timestamp` now always returns timezone-aware UTC datetimes (or `None`): offset-less inputs get `tzinfo=timezone.utc`, matching what the exporting systems store, while values with an explicit offset are preserved. `external_created_at`/`external_updated_at` metadata and rendered episode timestamps now carry `+00:00` where they were previously naive; no function signature or configuration changed (PR #4248).
* Fixes the Letta and Zep migration adapters silently importing nothing when an export carries an empty key alias ahead of the populated one. The `_first_list(container, *keys)` helper, duplicated in `cognee/modules/migration/sources/letta.py` and `zep.py`, resolves the different spellings exports use for the same collection (e.g. `messages`/`in_context_messages`/`message_history` in Letta, `facts`/`edges`/`entity_edges` in Zep), but it returned on the first key whose value was a list — and since `[]` is a list, an alias that was present but empty short-circuited the scan. A Letta file with `messages: []` and the real history under `in_context_messages` therefore imported nothing, with `records()` yielding an empty stream and the import reporting success; six call sites across the two adapters were affected. The helper now keeps scanning until an alias actually yields dict records. Behavior changes only for inputs that previously imported nothing: a populated alias still wins in the same priority order, and all-empty inputs still yield nothing (PR #4253).
* Fixes `recall()` auto-routing sending natural-language questions to coding-rules retrieval because they happened to contain a code token. In the rule-based router (`cognee/api/v1/recall/query_router.py`), the incidental code-token pattern (`def `, `return `, `async `, `await `, `import `, `class X(`, `.py`, `function x(`) scored `3.0` toward `CODING_RULES` — enough to clear the router's `2.0` default threshold and win outright, so questions like "Describe the import process for customer records" or "What does the return policy say?" were routed to coding rules, and the token could also outrank stronger intent cues in the same query. That weight is now `1.0`, which sits below the default threshold: an incidental code token can no longer select a search mode by itself, and it loses to any stronger cue. Explicit coding vocabulary (`coding rules`, `code review`, `best practice`, `lint`, `refactor`) keeps its `5.0` weight and still routes to `CODING_RULES`. User impact: affected queries now resolve to `GRAPH_COMPLETION` or to whichever cue actually dominates, so the `search_type`/`kind` reported on their results can differ from before; pass `query_type` explicitly to bypass the router entirely. No public API signature, configuration option, environment variable, or migration changed (PR #4207).
* Fixes edge-filtered neighborhood queries crashing on the default Ladybug (Kuzu) graph backend. `LadybugAdapter.get_neighborhood(edge_types=[...])` built its Cypher with an `ALL(rel IN r WHERE rel.relationship_name IN $edge_types)` predicate over the variable-length relationship binding `r` from `-[r*1..depth]-`; in Kuzu `r` is a `RECURSIVE_REL` rather than a `LIST`, so `ALL(...)` is a binder-type mismatch that, combined with the parameter reference, drove the engine into a failed internal assertion. Because Ladybug is the default backend, any edge-filtered neighborhood query failed out of the box (Neo4j and Postgres were unaffected); the built-in search and visualization flows never pass `edge_types`, so the crash surfaced for programmatic callers of the public `get_neighborhood` primitive — custom retrievers and SDK code supplying an edge-type allow-list. The `ALL()` predicate is now dropped; when `edge_types` is supplied the adapter fetches the paths unfiltered and post-filters in Python, keeping a neighbor iff some path reaching it within `1..depth` hops has every edge type in the allowed set (undirected) — the same semantic Neo4j and Postgres already enforce. The behavioral tradeoff: the filtered branch enumerates every path up to `depth` before post-filtering, so its cost grows combinatorially with node degree × depth on dense graphs; it is best suited to shallow, targeted neighborhoods. The fast `edge_types=None`/`[]` path is unchanged. No public API signature, configuration option, or environment variable changed (fixes #3585, carries community fix #3591 by @ly-wang19, PR #4156).
* Surfaces dataset and data-item **names** in the text channel of the `cognee-mcp` JSON tools `list_datasets_json` and `list_dataset_data_json`. Previously both tools put names only in `structuredContent` and emitted just a count in `content[0].text` (e.g. `8 dataset(s).`), so text-only MCP clients — such as agents in Cursor that never see `structuredContent` — could not tell what existed and fell back to raw HTTP. A new shared `_format_named_items` helper in `cognee-mcp/src/server.py` now renders one `- name (id)` line per item into the text content (falling back to `- name` when an id is absent and `(unnamed)` when a name is absent), prefixed by a header line (`8 datasets:`, or `1 dataset:` in the singular). The text is capped at 50 items, with a trailing `… and N more (see structuredContent).` note when the list is longer; an empty list reads `No datasets found.` / `No data items found.`. The change is backward-compatible and text-only: `structuredContent` (`{datasets: [...]}` / `{data: [...]}`), both tool schemas, and the Cognee workspace UI are unchanged, and no configuration option or environment variable was added (CLO-319, PR #4186).
* Fixes the `cognee-mcp` client failing or hanging when listing datasets or checking status against Cognee Cloud, and makes API-mode `cognify` submit a background run instead of blocking. In API mode the client's `list_datasets` requested `/api/v1/datasets` (no trailing slash) and relied on the server's 307 redirect to the canonical `/api/v1/datasets/`; the client does not follow redirects, so the call failed with an HTTP error on the redirect response, and the redirect `Location` could additionally downgrade to `http://` against the HTTPS-only edge (see the server-side fix, CLO-320). `list_datasets` now calls the canonical trailing-slash route `/api/v1/datasets/` directly, so no redirect is involved. Separately, the client applied its single client-wide `timeout=300.0` to every request, so a hung or black-holed read-only GET froze the caller for a full five minutes; a per-request `READ_TIMEOUT_SECONDS = 30.0` is now applied to the dataset **list** and **status** GETs, while the 300s client-wide default remains for other requests. `READ_TIMEOUT_SECONDS` is a hardcoded module constant in `cognee-mcp/src/cognee_client.py`, not an environment variable or documented setting; the tradeoff is that a valid but very slow (>30s) GET will now time out. Finally, the client's `cognify` POST now sends `run_in_background: true`, so the request submits the pipeline run on the server and returns immediately instead of holding the HTTP request open for the whole run; the MCP `cognify` tool already returned immediately and directs callers to poll dataset status, which now reflects the server-side background run. No public MCP tool signature, configuration option, or environment variable changed (CLO-322, PR #4184).
* Fixes OpenRouter embedding requests failing with a `400 invalid_value` error on `encoding_format`. Older LiteLLM releases serialize an *omitted* `encoding_format` as JSON `null`; OpenAI tolerates it, but OpenRouter rejects it (it accepts only `"float"` or `"base64"`), so an OpenRouter embedding config (`EMBEDDING_PROVIDER="custom"`, `EMBEDDING_MODEL="openrouter/openai/text-embedding-3-small"`) could 400 on every embedding call. `LiteLLMEmbeddingEngine.embed_text` now sets `encoding_format="float"` whenever it detects an OpenRouter route — a model id beginning with `openrouter/`, an explicit `openrouter` provider, or an `openrouter.ai` endpoint host (all matched case-insensitively). Cognee always consumes float vectors, so the value is safe to make explicit. The guard is scoped narrowly to OpenRouter on purpose: Cognee does not enable `litellm.drop_params`, and `encoding_format="float"` would raise `UnsupportedParamsError` for providers such as gemini/bedrock/vertex\_ai, so it is not applied unconditionally. On current LiteLLM versions the guard is a no-op for `openrouter/`-prefixed models (litellm's dedicated OpenRouter branch drops the omitted parameter), but endpoint-based configs — an unprefixed model pointed at an `openrouter.ai` endpoint — route through litellm's OpenAI handler, which injects the `null` even on current versions, so the guard is what fixes those. The change requires no user action, and no public API signature, configuration option, or environment variable changed (SDK-311, fixes #3660, PR #4195).
* Adds a `COGNEE_EXTRAS` Docker build argument to the root `Dockerfile`, so optional-dependency groups can be baked into a source-built API image without editing the `Dockerfile`. The argument takes a space-separated list of extra names — `docker build --build-arg COGNEE_EXTRAS="docs langchain" -t cognee-custom .` — which the build expands into `--extra <name>` flags on top of the existing default set (which has since grown to include the `aws` extra, so `aws` no longer needs to be passed). It is applied to **both** `uv sync` steps: the second sync is exact and would otherwise drop extras installed only in the dependency-cache layer, so applying it twice is what makes the packages reach the final runtime stage. The argument defaults to an empty string, making it a no-op, so existing builds are unaffected. Both `uv sync` invocations keep `--frozen`, so the argument only selects extras already resolved in `uv.lock` rather than resolving new dependencies — builds stay deterministic and no lockfile change is required. Note that the `cognee` service's `build:` block in `docker-compose.yml` has no `args:` entry, so `docker compose up --build cognee` does not forward the value until you add one. The argument is declared only in the root API `Dockerfile`; the MCP and frontend images do not accept it. No public API signature, configuration option, or environment variable changed (SDK-314, PR #4211).
* Stamps messages remembered through the Slack integration with a `slack` node set, so Slack-sourced data carries a structured origin marker into the graph. Slack's "Remember this" shortcut called `remember()` with only `dataset_name="slack"` and no `node_set`, so the resulting document reached the graph with no Slack marker at all — no `NodeSet` node, no `belongs_to_set` edge, no `source_node_set` property — and the only trace of where the content came from was the relational dataset name and the English prefix baked into the message text (`In #channel, <@user> said: …`). `remember_message` now passes `node_set=SLACK_NODE_SET` (a new module constant equal to `["slack"]` in `cognee/modules/integrations/slack/remember_message.py`), which materializes a `slack` [NodeSet](/core-concepts/further-concepts/node-sets) node with `belongs_to_set` edges that propagate from the document down to its chunks and extracted entities, and sets the `source_node_set` property the pipeline carries across task boundaries. The practical effect is that Slack items gain the same source dimension every other ingestion path already had: they are grouped and colored by origin in graph visualization, and retrieval can be scoped to them with `recall(..., node_name=["slack"])`. The change is non-breaking and requires no user action — Slack data already in the graph is untouched and is only marked if those messages are remembered again — and no public API signature, configuration option, or environment variable changed (SDK-318, PR #4216).
* Reduces CPU spent on openai-python's per-response type introspection in Cognify and other LLM-heavy workloads. On every API response the OpenAI SDK rebuilds its `ChatCompletion` response tree, repeatedly calling `get_origin`, `get_args`, `is_annotated_type`, and `is_literal_type` against the same static types; at cognify scale that introspection dominated CPU. A new internal module `cognee/infrastructure/llm/openai_type_cache.py` wraps each of those four helpers in a `functools.lru_cache(maxsize=4096)` and rebinds the cached versions at every known openai import site — patching the source module alone is insufficient, because most call sites use `from ._compat import get_origin` and capture the name at import time. The install runs once and is idempotent, triggered when `cognee.infrastructure.llm` is imported, which any normal Cognee usage already does before an OpenAI SDK call is made; no user action or configuration is required. In an end-to-end 200-document Cognify run, CPU time dropped from \~93.85s to \~67.70s (≈28%). Failure modes degrade safely rather than break: the rebind targets are openai-python **private** modules (`openai._models`, `openai._utils._compat`, `openai._utils._typing`, `openai._response`, `openai._legacy_response`, `openai._base_client`) that are not covered by its compatibility guarantees, so if a future openai release moves them the import fails, `install()` becomes a no-op, and Cognee keeps running against the uncached originals; likewise an unhashable argument falls back to the original uncached helper, and any call site that is not rebound simply stays uncached. No public API signature, configuration option, or environment variable changed (COG-5963, PR #2876).
* Adds `LangMemSource`, a memory-migration source for importing LangMem memories, so LangMem joins Mem0, Letta, Zep/Graphiti, and COGX archives as a system `cognee.remember()` can import from. Construct it with `LangMemSource(data, mode="re-derive")` — exported from `cognee.modules.migration.sources` — where `data` is a path to a LangMem JSON export, an already-parsed list (for live-API use, e.g. a response fetched with the LangMem client), or a dict wrapping the list under `memories`, `results`, `items`, or `data`; any other shape raises `ValueError`. Each item becomes a COGX **memory** record: content is the first string present among `content`, `text`, `memory`, `data`, and `message` (items with none of those are skipped), `user_id` falls back to `namespace` for the record's scope (which also carries `agent_id`, `session_id`, and `run_id` when present), `categories` accepts a single string or a list, `created_at`/`createdAt`/`timestamp` and `updated_at`/`updatedAt` supply timestamps, `id` falls back to a positional `langmem-<index>` identifier, and any `metadata` is carried over nested under `langmem_metadata`. The mode default is the base-class `re-derive`, which suits LangMem's free-form text (it carries no derived graph of its own to preserve). The change is additive: no existing source, `remember()` signature, configuration option, or environment variable changed. See [Migrate Memory Systems with COGX](/examples/migrate-memory-systems) (PR #4208).
* Fixes ingested document names being recorded percent-encoded, and platform-foreign paths being recorded whole. `get_file_metadata` derived the document name (`FileMetadata["name"]`, persisted as `Data.name`) with `Path(file_path).stem`, where `file_path` is the opened stream's `.name`. In the ingestion pipeline that value is a percent-encoded `file://` URI — `LocalFileStorage` wraps every opened file in a `FileBufferedReader` named `Path(full_path).as_uri()` — so escapes leaked into the stored name on every platform: a file named `Annual Report.pdf` was recorded as `Annual%20Report`. `Path` is also OS-specific, so a raw backslash path from another caller (`C:\Users\me\report.pdf`) yielded the whole path minus its extension as the "stem" on POSIX (the reverse direction was never broken: Windows path handling accepts `/` as a separator). A new `_derive_basename` helper now percent-decodes `file://` URIs (`unquote(urlparse(...).path)`) and resolves the basename with `PureWindowsPath`, which treats both `/` and `\` as separators on every host OS, so those inputs are recorded as `Annual Report` and `report`; a degenerate input that yields an empty stem becomes `None`, letting the caller fall back to an explicitly supplied filename. The prior extension-less stem semantics are preserved: only the last suffix is stripped (`archive.tar.gz` → `archive.tar`), dotfiles such as `.gitignore` stay intact, and the extension continues to be stored separately in `FileMetadata["extension"]` — persisted for the original file as the `Data.original_extension` column, while `Data.extension` describes the stored text representation (typically `txt`). Separately, `classify()`'s fallback for `BufferedReader` / `SpooledTemporaryFile` inputs passed without an explicit `filename` now derives the basename with `str(data.name).replace("\\", "/").split("/")[-1]` instead of splitting on `/` alone (the same normalization as `_normalize_filename` in `cognee/tasks/ingestion/utils.py`), so a Windows-style stream name resolves to `report.pdf` rather than the entire path; this is a defensive path, because in the ingestion pipeline `classify()` only ever sees a `file://` URI. The change is backward-compatible and requires no action: POSIX paths and unencoded names resolve exactly as before, deduplication remains content-hash based so no record changes identity, and no public API signature, configuration option, environment variable, or database migration changed. Names already stored before the fix are not rewritten automatically, though re-adding a file refreshes its stored name through the re-ingest update path (SDK-237, PR #4157).
* Fixes `cognify` failing on dlt-backed sources when the schema and foreign-key edges are registered in the relational rollback ledger. `extract_dlt_fk_edges` built its edge tuples with stringified node ids in the source/destination slots (`str(source_table_id)`, `str(relationship.id)`, and the `doc_id`/`target_data_id` strings it carries in its document map), but the same tuple list is handed both to `graph_engine.add_edges` and to the ledger's `upsert_edges`, which declares `List[Tuple[UUID, UUID, str, Dict]]` and binds slots 0/1 straight into the `edges` table's `source_node_id`/`destination_node_id` columns — both typed `UUID(as_uuid=True)`. SQLAlchemy's UUID bind processor reads `.hex` off the bound value, so the ledger insert raised `AttributeError: 'str' object has no attribute 'hex'` and failed the run. The failure was not limited to schemas that declare foreign keys: the per-row `is_row_of` edge linking each dlt row document to its `SchemaTable` node is emitted for every dlt row, so CSV, SQL connection-string, Gmail, and Slack-export ingestion were all affected. Slots 0/1 now carry `UUID` objects (including `UUID(doc_id)` and `UUID(target_data_id)` for the row-level edges), matching the tuple contract `upsert_edges` and the standard `get_graph_from_model` path already rely on, while each edge's JSON attribute dict keeps its string `source_node_id`/`target_node_id` copies, so stored edge properties are unchanged. Runs whose provenance is stamped in-graph skip the ledger write and were never affected. No public API signature, configuration option, environment variable, or database migration changed (SDK-183, PR #4081).
* Doubles the remote request timeouts used by the `cognee.serve()` client, so long-running server-side work is no longer cut off mid-flight. `CloudClient` bounds ordinary remote operations (`remember`, `recall`, `improve`, `add`, `cognify`, `search`, `forget`) with a client-wide `aiohttp` total timeout, and applies a separate per-request timeout to archive uploads (`content_type="cogx-archive"`, the `cognee.push()` path) whose total is uncapped in favour of a per-read inactivity bound. Both limits were 300 seconds, which a blocking `cognify()` over a large dataset — or an archive upload plus its synchronous server-side import — could legitimately exceed, aborting the call client-side even though the server was still making progress. `DEFAULT_TIMEOUT` is now `total=600` and `UPLOAD_TIMEOUT` is now `sock_read=600`; the `sock_connect=30` connect bound and the uncapped upload total are unchanged, as is the two-tier structure itself (only the two 300s values moved). Both are hardcoded module constants in `cognee/api/v1/serve/cloud_client.py`, not environment variables or documented settings, so there is no way to tune them per deployment; the tradeoff is that a genuinely hung request now blocks for up to ten minutes rather than five. The change is backward-compatible, and no public API signature, configuration option, or environment variable changed (SDK-284, PR #4145).
***
## v1.4.0.dev1
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.0.dev1)**
Development pre-release that bumps the package version from `1.4.0.dev0` to `1.4.0.dev1` (the tag sits on the SQL session-cache fix merge rather than a separate release-cut commit). No new Alembic revision ships in this cut.
### Highlights
* Gates delete operations behind the same per-dataset lock that serializes pipeline runs. The in-process asyncio lock registry moves from `cognee/modules/pipelines/operations/pipeline.py` into the new `cognee/infrastructure/locks/dataset_lock.py`, exporting `dataset_lock`, `get_dataset_lock`, and `held_datasets`; pipeline runs and deletes now acquire from one shared registry. `datasets.delete_dataset` and `datasets.delete_data` in `cognee/api/v1/datasets/datasets.py`, plus `cognee.forget`'s dataset- and data-level memory clearing in `cognee/api/v1/forget/forget.py`, now run inside `async with dataset_lock(dataset_id)`, so a delete waits for any in-flight `add`/`cognify`/`memify` run on the same dataset (and vice versa) and two deletes on one dataset are serialized, while different datasets still proceed in parallel. The lock is re-entrant per execution context via the `held_datasets` `ContextVar` and remains process-local (asyncio) — it does not protect against multiple processes or workers touching the same dataset. No public API signature, configuration option, or environment variable changed (PR #4042).
* Changes `cognee.improve()` so session-persistence failures are no longer swallowed. `_bridge_sessions` in `cognee/api/v1/improve/improve.py` wrapped its `persist_sessions_in_knowledge_graph_pipeline(...)` call in a blanket `try/except Exception` that logged `improve: session persistence failed (non-fatal)` as a warning and let the remaining improve stages (trace persistence, agent-context extraction, session distillation) run as if bridging had succeeded. That handler is removed: exceptions now propagate out of `improve()` (a `finally` still releases the per-session improve lock), and error handling is delegated to the pipeline system, which already records pipeline-run failures where suppression is appropriate. Callers that relied on `improve()` never raising during session bridging will now see those exceptions and can react to real failures instead of getting silent partial results. No public API signature, configuration option, or environment variable changed (PR #4056).
* Adds a first-class `dataset_id` keyword parameter to `cognee.remember()` and routes it through improve operations. Previously `dataset_id` was only an untyped pass-through kwarg (listed in `RememberKwargs`/`_ADD_ONLY` and forwarded to `add()`), and the `self_improvement=True` paths always invoked `improve()` with `dataset_name` — so improve could not target a dataset addressed by UUID. `remember()` in `cognee/api/v1/remember/remember.py` now declares `dataset_id: Optional[UUID] = None` (takes precedence over `dataset_name`), resolves or creates the dataset before the session branch, and both the permanent and session self-improvement paths call `improve(dataset=dataset_id or dataset_name, ...)`. `RememberResult` fills `dataset_name` from the pipeline run when only an id is supplied, session-mode results now include `dataset_id`, and `CloudClient.remember` forwards the id as a `datasetId` form field. Passing `dataset_id` with `MemorySource` imports or typed `MemoryEntry` payloads raises `ValueError`; those stay dataset-name based. No env vars or migrations involved (PR #4158).
* Fixes local-mode tenant context in the Cognee UI and batches related frontend/backend changes. `TenantContext`/`useTenant()` now expose `tenantReady`, `podUnreachable`, `isOwner`, `availableTenants`, and `releaseLoader`, and `LocalProvider` supplies the same shape, giving self-hosted (local mode) sessions the identical context contract as cloud tenants; tenant pod domain resolution is centralized in `getTenantApiDomain.ts` (explicit `NEXT_PUBLIC_TENANT_API_DOMAIN`, else parsed from `NEXT_PUBLIC_MANAGEMENT_API_URL`). Every `createHttpClient()` instance now attaches an `X-Request-Id` correlation header (the opt-in `setup.ts` registration is removed). Uploads in `rememberData.ts` pass `timeoutMs` to the shared HTTP client instead of racing a local `AbortController` against the client's 30-second default, restoring the intended 5-minute upload window, and `DatasetsPage` reports upload failures and knowledge-graph build failures separately while enforcing `MAX_FILES_PER_UPLOAD`. On the backend, a new `GET /api/v1/datasets/graph-summary` endpoint returns per-dataset node/edge counts cached in `GraphMetrics` per latest cognify `pipeline_run_id`, far cheaper for status polling than the full graph endpoint (PR #4179).
* Fixes the Mistral transcription adapter sending an entire Windows path as the API `file_name`. `MistralAdapter.create_transcript` derived the file name with `input.split("/")[-1]`; on Windows the audio path uses backslash separators (e.g. `C:\audio\clip.mp3`) and contains no forward slashes, so the split left the value unchanged and the whole path — rather than the basename `clip.mp3` — was sent to the Mistral transcription API. The basename is now derived with `str(input).replace("\\", "/").split("/")[-1]`, normalizing both `\` and `/` separators (the same handling used by `_normalize_filename` in `cognee/tasks/ingestion/utils.py`), so Windows and POSIX paths both send just the file name. The change is backward-compatible: POSIX paths resolve to the same basename as before, and no public API signature, configuration option, or environment variable changed (fixes #3587, PR #3588).
* Fixes writes failing on Postgres-backed graph and vector stores when node/edge fields or vector payloads contain NUL bytes (`\u0000`). Postgres text columns and JSONB reject the `\u0000` escape, and while the vector store's `json` column accepts it on insert, the `payload::jsonb` casts used by search/merge queries later reject it — so ingesting content with an embedded NUL byte could error. A shared `sanitize_relational_payload` helper now strips NUL bytes from strings and recurses through nested containers (dicts, lists, tuples), decoding `bytes`/`bytearray` values as UTF-8 with replacement so invalid byte sequences do not break persistence. It is applied in the Postgres graph adapter to the `id`, `name`, `type`, and `properties` of nodes and to the `source_id`, `target_id`, `relationship_name`, and `properties` of edges (ids sanitized identically on both sides so references stay consistent), and in `PGVectorAdapter` to each data point's serialized payload. This is an internal serialization fix for the Postgres graph and PGVector adapters; no public API signature, configuration option, environment variable, or database migration changed (PR #4153).
* Fixes ingestion crashing on Windows when a string starts with `/` or is drive-relative. In `save_data_item_to_storage`, the absolute-path branch treated any string beginning with `/` (or, on Windows, one whose second character is `:`) as a local file path and called `Path(...).as_uri()` on it. On Windows, `os.path.normpath("/etc/hosts")` yields a *drive-relative* path and a drive-relative input like `C:notes.txt` stays drive-relative, so `as_uri()` raised `ValueError` (relative paths cannot be expressed as `file:` URIs) — any POSIX-style path string or plain text note starting with `/` crashed `add()`. The branch is now additionally guarded by `Path(os.path.normpath(data_item)).is_absolute()`, so on the current platform only genuinely absolute paths convert to a `file:` URI; non-absolute `/`-prefixed or drive-relative strings fall through to the existing relative-path/text handling and are ingested as text (saved to Cognee's data storage as a text file). POSIX behavior is unchanged (`/...` paths still convert to `file:` URIs) and genuine Windows absolute paths (`C:\...`) still convert as before. No public API signature, configuration option, or environment variable changed, and `accept_local_file_path` continues to govern acceptance of true absolute paths (fixes #3887, PR #3892).
* Fixes S3 ingestion failing on Windows with `PermissionError` (WinError 32). In `data_item_to_text_file`, the S3 branch downloaded the object into a `tempfile.NamedTemporaryFile` created with the default `delete=True` and then passed `temp_file.name` to the loader, which reopens the file by name while Cognee's handle is still open. On Windows that reopen raises `PermissionError [WinError 32]`, so every S3 ingestion failed. The temp file is now created with `delete=False`, its handle is flushed and closed before the loader reopens it, and it is removed with `os.unlink` in a `finally` block (guarded against `OSError`) so no temp file is leaked — mirroring the `delete=False` pattern already used by the SQLAlchemy and ladybug S3 temp-file paths. POSIX behavior (Linux/macOS) is unchanged and temporary files are still cleaned up after use. No public API signature, configuration option, or environment variable changed (fixes #3339, PR #3340).
* Fixes two failures in the SQL session-cache backend (`CACHE_BACKEND=postgres` / `sqlite`) when ids are UUID-like or when many writers target the same session concurrently. First, cache key columns (`user_id`, `session_id`, `qa_id`, `entry_id`, `log_key`, and the KV `key`) now use a `StringKey` `TypeDecorator` that coerces stringable ids such as `uuid.UUID` to `str` in the bind processor: the asyncpg dialect renders explicit bind casts, so an id bound as a `uuid.UUID` made Postgres parse `text = uuid` and raise `42883` ("operator does not exist"), while SQLite rejected the non-str bind outright — passing a `uuid.UUID` id (rather than a string) to the adapter could therefore fail to read or write. The decorator normalizes every read and write to the same string regardless of the caller's type; because DDL is delegated to the underlying `Text` impl, the emitted column stays plain TEXT and existing tables need no migration. Second, on Postgres each same-session write transaction now takes a transaction-scoped `pg_advisory_xact_lock` keyed by `(table, user_id, session_id)` before writing, so concurrent writers to one session queue instead of deadlocking on the sliding-TTL UPDATE (SQLSTATE `40P01`); the lock auto-releases at COMMIT/ROLLBACK, `delete_session` acquires the per-table locks in a fixed order so a multi-table writer can't cycle with single-table writers, and the whole mechanism is a no-op on SQLite (which serializes writers with its own single-writer lock). The tradeoff is that concurrent writes to the *same* session may serialize slightly; writes across different sessions are unaffected. No public API signature, configuration option, environment variable, or database migration changed (PR #4182).
***
## v1.4.0.dev0
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.0.dev0)**
First development pre-release after v1.4.0, bumping the package version from `1.4.0` to `1.4.0.dev0` and re-resolving `uv.lock`. Note that cognee cuts its `.devN` pre-releases after the corresponding stable release, so despite what PEP 440 version ordering suggests, `1.4.0.dev0` is newer than `1.4.0`. No new Alembic revision ships in this cut.
### Highlights
* Fixes two crash/debuggability issues around telemetry and CLI user resolution. `send_telemetry()` in `cognee/shared/utils.py` called `asyncio.get_running_loop()` unconditionally, so in a sync context or after event-loop shutdown it raised `RuntimeError` and crashed the caller mid-pipeline; the `get_running_loop()`/`create_task()` preamble is now wrapped in `try/except RuntimeError`, making telemetry genuinely best-effort — the event is dropped instead of taking down the pipeline. Separately, `resolve_cli_user()` in `cognee/cli/user_resolution.py` caught bare `except Exception:`, so database connectivity or ORM failures were silently treated as "user not found"; the handler is narrowed to `EntityNotFoundError` (what `get_user()` raises for an unknown UUID), so genuine infrastructure errors now propagate with their original traceback. No public API signature, configuration option, or environment variable changed, and no action is needed (PRs #3276 and #3327).
* Fixes three gaps in the recall API's result normalization that left agents and MCP clients with incomplete metadata. `AGENTIC_COMPLETION` results, previously normalized as `kind: "unknown"`, now map to `SearchResultKind.GRAPH_COMPLETION` in `cognee/modules/recall/methods/normalize_search_payload.py`. `FEELING_LUCKY` searches are resolved via `select_search_type()` in `cognee/modules/search/methods/get_retriever_output.py` before the retriever is instantiated, so the payload (and tracing span) carries the effective search type — e.g. `CHUNKS` — instead of the unresolved sentinel that also normalized to `"unknown"`. And when a completion entry is a Pydantic `BaseModel` (a `response_model` search), the item now gets `kind: "structured"` with the `structured` field populated from `model_dump()` while `text` stays renderable, where `structured` was previously always `null`. Normalization-layer only: no retriever ranking, public API signature, configuration option, or environment variable changed (fixes #3820, PR #3822).
* Adds optional TLS to the Redis cache adapter, which previously had no SSL path and would hang against managed Redis endpoints with in-transit encryption (AWS ElastiCache, GCP Memorystore, Azure Cache for Redis) until `socket_timeout` and then raise `CacheConnectionError`. Two new `CacheConfig` settings — `cache_ssl` (bool) and `cache_ssl_cert_reqs` (`"required"`/`"optional"`/`"none"`), settable via the `CACHE_SSL` and `CACHE_SSL_CERT_REQS` environment variables — are threaded through `get_cache_engine`/`create_cache_engine` into `RedisAdapter`, which forwards `ssl` and `ssl_cert_reqs` to both its sync and async `redis.Redis` clients. This brings the Redis cache to parity with the Postgres and pgvector adapters, which already read SSL settings from `DATABASE_CONNECT_ARGS`. Defaults are off and `"required"`, so existing deployments connect exactly as before; to enable TLS, set `CACHE_SSL=true` and adjust `CACHE_SSL_CERT_REQS` for self-signed certificates (fixes #3850, PR #3851).
* Changes the four session-context methods on `SessionManager` (`create_session_context_entry`, `get_session_context_entries`, `update_session_context_entry`, `delete_session_context` in `cognee/infrastructure/session/session_manager.py`) to let `SessionParameterValidationError` propagate on an empty or whitespace `user_id`/`session_id`, matching `add_qa` and every other method in the class. Previously each wrapped `_validate_session_params` in `try/except Exception: return False` (or `[]`), so a caller bug like `user_id=""` was indistinguishable from an unavailable cache — and unlike the cache path, without even a `logger.warning`. The fail-open behavior for infrastructure errors is unchanged: when the cache is unavailable or the cache operation fails at runtime, the methods still return `False`/`[]` with a log message rather than raising, and behavior for valid inputs is identical. Callers passing invalid IDs will now see an exception instead of a silent `False`; no public API signature, configuration option, or environment variable changed (PRs #3881 and #3911).
* Adds a pre-flight dry-run token/cost estimator and makes LLM quota/billing exhaustion fail fast instead of retrying. `remember(dry_run=True)`, `cognify(dry_run=True)`, and the CLI `--dry-run` flag on both commands return a stage-level estimate (structured graph extraction plus chunk summarization) of LLM tokens and rough USD cost with no LLM calls, no ingestion, and no graph writes; the estimator (`cognee/modules/cognify/estimator.py`) reuses the real pipeline's document classification, chunker, and prompt templates, resolves datasets read-only, and rejects unsupported inputs (remote URLs, directories, binary formats, session memory, remote `serve()` mode) loudly rather than mis-estimating. Separately, a shared `llm_retry_condition` in `cognee/infrastructure/llm/retry_config.py` now governs every structured-output adapter and BAML, treating quota/billing wordings like `insufficient_quota` as terminal — `LLMGateway.acreate_structured_output` converts them into an actionable `LLMQuotaExceededError` — while transient rate limits (including Gemini free tier's recoverable "exceeded your current quota") still retry. `dry_run` defaults to `False`, so existing calls are unaffected (SDK-136, fixes #3643, PR #3974).
* Adds an optional `LLM_PROVIDER=mcp-sampling` backend that lets cognee, when running as an MCP server (`cognee-mcp`) inside a host such as Claude Code or Cursor, delegate all LLM completions to the host's own model via MCP's `sampling/createMessage` — so no `LLM_API_KEY` is needed (the provider is excluded from the API-key-required set; `LLM_MODEL` is only a hint, the host picks the model). The new `MCPSamplingAdapter` reads the host session from the MCP SDK's per-request context via `get_sampling_session()`, so `cognee-mcp/server.py` needed no changes; structured output is produced by embedding the response model's JSON Schema in the prompt with a bounded validate/repair loop, since sampling returns free text only. It fails closed with an actionable `MCPSamplingUnavailableError` when cognee is not running as an MCP server or the host did not grant the `sampling` capability. Completions only — embeddings, audio, and vision are not covered, so configure an embedding provider for vector search; non-MCP usage and all existing providers are unchanged, and `mcp` remains an optional dependency (SDK-139, closes #3644, PR #3982).
* Adds video ingestion: a new `video_loader` (`cognee/infrastructure/loaders/core/video_loader.py`) transcribes a video's audio track through the existing `LLMGateway.create_transcript` path and feeds the text into the normal pipeline as a regular `TextDocument` — no new document type. Supported extensions are `mp4`, `m4v`, `mov`, `webm`, `mkv`, and `avi`; on OpenAI/Azure the transcript carries inline `[HH:MM:SS]` segment timestamps so timing survives chunking, while other providers fall back to a plain transcript. ffmpeg is optional: `.mp4`/`.webm` are sent straight to the transcription endpoint, other containers use a system ffmpeg (found via `shutil.which("ffmpeg")`) to extract the audio first and raise an actionable error when it is absent. The PR also fixes a real cross-provider bug: `create_transcript` forwards `**kwargs`, but the mistral/gemini/custom adapters rejected them with a `TypeError` inside the retry loop; the `**kwargs` contract is now uniform across `LLMInterface` and all adapters. No new configuration options, environment variables, or extras were added (SDK-141, related issue #3636, PR #3986).
* Adds Turso (libSQL) as a selectable backend for all three cognee stores. Vector: `VECTOR_DB_PROVIDER=turso` routes to the new `TursoVectorAdapter` (collections as libSQL tables with `F32_BLOB` vector columns, cosine similarity via `vector_distance_cos`), working embedded (local file) or against Turso cloud via `VECTOR_DB_URL`/`VECTOR_DB_KEY`, and requires the `cognee[turso]` extra (`libsql-experimental`). Relational: `DB_PROVIDER=turso` selects a `TursoAdapter` that is a thin subclass of `SQLAlchemyAdapter` — a libSQL file is a SQLite file, so it reuses the same `aiosqlite` driver, sqlite dialect, and Alembic migrations; remote mode uses Turso's embedded-replica sync via `DB_TURSO_URL`/`DB_TURSO_AUTH_TOKEN`. Graph: `GRAPH_DATABASE_PROVIDER=turso` stores the knowledge graph as `graph_node`/`graph_edge` tables with recursive-CTE traversals, local/embedded only, needs no extra dependency, and optionally takes a `GRAPH_DATABASE_URL` file path. Vector and graph each ship a per-dataset dataset-database handler, so multi-user isolation under `ENABLE_BACKEND_ACCESS_CONTROL` works unchanged. All Turso backends are opt-in; defaults and existing backends are untouched (SDK-152, SDK-175 and SDK-176; PRs #4027, #4077 and #4080).
* Adds deterministic, LLM-free code-graph extraction backed by the external `enola` binary (Apache-2.0, by Enola Labs). The new `cognee/tasks/code_graph` package runs `enola --generate` (discovered on `PATH` or via the `ENOLA_PATH` env var, with an `install_enola` helper pinned to a known version) and maps its `.enola/facts.jsonl` output — module/symbol/route/storage/dependency/service facts — to DataPoint models with deterministic `uuid5` ids plus typed edges (`calls`, `imports`, `implements`, `depends_on`, and others). User-facing entry points: `cognee.remember(repo_path_or_git_url, content_type="code")` indexes one or more local repos or git URLs (optional `index_vectors=True` enables embeddings; the default graph-only run needs no LLM key), a new `SearchType.CODE` queries the result through the new `CodeRetriever`, and `get_code_graph_tasks(repo_path)` plugs into `run_custom_pipeline`. Extraction covers Go, Python, TypeScript, Java, Kotlin, Swift, Ruby, C/C++, PHP, Vue, Svelte, OpenAPI, and gRPC. No enola code is vendored and no new Python dependencies are added; a missing binary raises an actionable `EnolaNotInstalledError` (COG-5837, PR #4037).
* Adds Langfuse tracing support by wiring Langfuse into cognee's existing OpenTelemetry pipeline as just another OTLP destination — no separate Langfuse SDK. Setting `LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY` (optionally `LANGFUSE_HOST`, with `LANGFUSE_BASE_URL` accepted as an alias) makes `base_config.py` derive the OTLP endpoint (`{host}/api/public/otel/v1/traces`) and the `Authorization: Basic` header and turn tracing on; both keys must be set together or a `ValueError` is raised, and an explicit `OTEL_EXPORTER_OTLP_ENDPOINT`/`OTEL_EXPORTER_OTLP_HEADERS` always wins. Generation spans in `get_observe.py` now emit the vendor-neutral `gen_ai.request.model` and `gen_ai.system` attributes plus `langfuse.observation.type` and `SpanKind.CLIENT`, so Langfuse and any other OTLP backend render LLM calls as generations, and `tracing.py` forces the HTTP exporter for Langfuse endpoints (detected by the `/api/public/otel` path) since Langfuse does not support gRPC. Fully opt-in and off by default; requires the `cognee[tracing]` extra; no existing configuration or public API changed (SDK-167, PR #4055).
* Adds an opt-in `litellm_native` structured-output framework that returns validated Pydantic objects without the `instructor` library, using LiteLLM's own `response_format`. Set `STRUCTURED_OUTPUT_FRAMEWORK="litellm_native"` and `LLMGateway` routes `acreate_structured_output` to a single universal `NativeLiteLLMAdapter` (`cognee/infrastructure/llm/structured_output_framework/litellm_native/`) covering every provider: when `litellm.supports_response_schema(model)` is true (OpenAI, Azure, Gemini, Mistral, Bedrock, …) the Pydantic model is passed straight through as `response_format` and validated with `model_validate_json`; otherwise (Ollama, llama.cpp, custom endpoints) it falls back to `response_format={"type": "json_object"}`, injects the JSON Schema into the prompt, and on validation failure feeds the error back for up to 3 self-correcting retries. Error handling matches the instructor adapters: auth and budget errors (`LLMPaymentRequiredError`) are terminal, rate limits retry with backoff, and content-policy violations fall back to the configured fallback model. The default remains `instructor` and the `baml` path is untouched; `create_transcript`/`transcribe_image` are unaffected (SDK-172, PR #4066).
* Adds a one-command evaluation runner and makes the eval harness a clean optional addon. `run_eval(config) -> EvalResult` (`cognee/eval_framework/runner.py`) chains corpus building, answering, evaluation, and dashboard generation for a single deterministic config and returns artifact paths plus aggregate metrics; it is exposed as both `cognee eval …` and `python -m cognee.eval_framework …`, mapping the same flags onto `EvalConfig`. The evaluator registry now resolves engines lazily by import path, so importing the registry or running the DirectLLM engine no longer pulls in `deepeval`; selecting DeepEval without the dependency raises an actionable error pointing at `pip install "cognee[eval]"`, a new umbrella extra that installs the dashboard (`plotly`), the DeepEval engine, and dataset-download deps — `--engine direct_llm --no-dashboard` runs without it, and the runner preflights dashboard imports before any paid pipeline work. A `seed` is now actually threaded into the benchmark adapters, and artifacts are namespaced under `<output-dir>/<benchmark>_<engine>/` with the resolved config saved alongside (SDK-174, PR #4073).
* Adds optional EXIF metadata extraction and perceptual-hash deduplication to `ImageLoader` (`cognee/infrastructure/loaders/core/image_loader.py`). With `IMAGE_EXIF_ENABLED=true`, `_extract_exif_metadata` pulls camera make/model, date taken, exposure, F-number, ISO, focal length, and GPS coordinates from the image's EXIF data and appends them as an `[EXIF Metadata]` block to the vision-LLM transcription. With `IMAGE_PERCEPTUAL_HASH_ENABLED=true`, a 64-bit difference hash (dHash) is computed per image and appended as a `[Perceptual Hash: …]` marker; an in-memory check against hashes seen in the same process flags visually similar re-ingestions with a duplicate note rather than skipping them. Both features are controlled by plain environment variables read at load time, are off by default, and use only PIL, which is already a cognee dependency — no new packages, and existing behavior is byte-identical unless a flag is enabled (partially addresses #3637, PR #4076).
* Fixes BAML structured output failing on Pydantic models with PEP 604 optional fields. With `STRUCTURED_OUTPUT_FRAMEWORK=baml`, any response model containing an `X | None` field raised `ValueError: Unsupported type for BAML mapping: str | None` — in practice making `GRAPH_COMPLETION` recall unusable with BAML (common on local/Ollama setups), since the completion response model uses `str | None` fields. The cause: `map_type()` in `create_dynamic_baml_type.py` only matched `origin is Union` (`typing.Union`), while PEP 604 unions report `get_origin() == types.UnionType`, so they fell through to the unsupported-type error even though the equivalent `typing.Optional[str]` worked. The condition is now `if origin is Union or origin is types.UnionType`, handling both spellings on Python >= 3.10 (the project minimum). One-line fix plus a comment; no public API, configuration option, or environment variable changed, and non-BAML frameworks are unaffected (PR #4121).
* Adds an optional `system_prompt` parameter to the MCP `recall` tool and forwards it end to end, so MCP clients can override the synthesis prompt for completion searches. Previously the tool signature had no such parameter, so custom recall system prompts were silently impossible over MCP. The parameter now flows through `CogneeClient.recall` (`cognee-mcp/src/cognee_client.py`) in both modes: in API mode it is included in the JSON payload POSTed to `/api/v1/recall`, and in direct/SDK mode it is passed as a kwarg to `cognee.recall`. When omitted, nothing is added to the payload or kwargs, so existing behavior is unchanged; regression tests cover both the API payload forwarding and the MCP tool forwarding. No other tool parameters, defaults, or server endpoints changed (fixes #4120, PR #4122).
* Fixes `cognee remember --dry-run` and `cognee cognify --dry-run` silently executing real remote operations when `--api-url` is supplied. The API dispatch path bypassed the local command implementations that honor `dry_run` and simply ignored the flag, so a "dry run" performed an ordinary remote remember/cognify. `dispatch()` in `cognee/cli/api_dispatch.py` now checks `args.dry_run` up front and raises a `RuntimeError` ("--dry-run is not supported in --api-url mode. Run without --api-url to estimate locally without remote side effects.") before constructing the API client, so no health check or operation request is ever sent. Users who want a dry-run estimate must run without `--api-url`; ordinary API-mode behavior without the flag, local dry runs, and server behavior are unchanged, and no API endpoint, dependency, or configuration option changed (closes #4125, PR #4126).
* Changes `visualize_graph()` and `GET /api/v1/visualize` to render a bounded, relevant subgraph by default instead of the whole graph. The renderer now selects a small set of seed nodes, expands their *k*-hop neighborhood, and caps the result at `max_nodes`, keeping renders fast and readable on large graphs. Seeds are resolved by priority — explicit `seed_node_ids` > a `recall()` or search result's graph provenance (`recall_result`, via `used_graph_element_ids`) > a `query` string's nearest (distance-ranked) vector hits > the graph's highest-degree nodes as a fallback — so a bare `visualize_graph()` call still shows a representative view, and "show me the subgraph behind this answer" and query-seeded views are deterministic and capped. New **keyword-only** parameters were added to `visualize_graph()`: `full`, `query`, `seed_node_ids`, `recall_result`, `neighborhood_depth` (default `2`), `neighborhood_seed_top_k` (default `10`), and `max_nodes` (default `500`); when a neighborhood exceeds `max_nodes`, nodes are kept by hop distance from the seeds and edges survive only when both endpoints do (no dangling edges). To restore the previous whole-graph render, pass `full=True` (or `?full=true` on the endpoint). `GET /api/v1/visualize` gains matching query params `full`, `query`, `seed_node_ids`, `neighborhood_depth`, `neighborhood_seed_top_k`, and `max_nodes` (`recall_result` is Python-only). The change is backward-compatible: the new parameters are keyword-only, so existing positional callers keep working; the underlying renderer and shared graph primitives are reused. See [Graph Visualization → Bounded subgraph by default](/guides/graph-visualization) (SDK-140, PR #3985).
* Fixes the stored file size not refreshing when a file is re-ingested. On the re-ingest update path, `ingest_data` assigned the new size to `data_point.file_size`, but the `Data` model defines the column as `data_size` (the name already used correctly on the new-record branch). SQLAlchemy silently ignored the nonexistent attribute, so the persisted `data_size` stayed at its original value after a file was re-added with new or changed content. The assignment now targets `data_point.data_size`, so re-ingestion records the current size. No public API signature, configuration option, environment variable, or database migration changed (fixes #3160, PR #3578).
* Fixes regex entity extraction config files failing to load on platforms whose default locale encoding is not UTF-8 (commonly Windows). `RegexEntityConfig._load_config` previously opened the config JSON with `open(path, "r")`, which relies on the platform's locale encoding; on a non-UTF-8 system a config containing non-ASCII characters (for example Unicode entity names, descriptions, or regex patterns) could raise a decode error and fail to load. The file is now opened with an explicit `encoding="utf-8"`, so configs load consistently across platforms. The change is backward-compatible: existing UTF-8/ASCII configs load exactly as before, no config edits or migration are required, and no public API signature, configuration option, or environment variable changed (fixes #3316, PR #3337).
* Fixes a `TypeError` when instantiating the Amazon Neptune Analytics graph adapter (`NeptuneGraphDB`). `GraphDBInterface` declares `is_empty()` as an abstract method, but the Neptune adapter never implemented it, so the class was abstract and any attempt to construct it (including test collection in `cognee/tests/test_neptune_analytics_graph.py`) raised `TypeError: Can't instantiate abstract class NeptuneGraphDB with abstract method is_empty`. The adapter now implements `async is_empty() -> bool`, which runs a small openCypher node-existence query (`MATCH (n) RETURN true LIMIT 1`) and returns `True` when the graph contains no nodes and `False` otherwise; it relies on Neptune's openCypher support. This is a non-breaking bug fix that only adds the required method — no public SDK function, configuration option, or environment variable changed (closes #3407, PR #3457).
* Fixes embedding retries wasting the full back-off window on deterministic "context window too small" failures. When an over-length embedding input is split down to a single string that still exceeds the model's context window but can no longer be divided, the engines now raise a new terminal `EmbeddingContextWindowTooSmallError` (a subclass of `EmbeddingException`, default message `Text is too short to split further but exceeds context window.`) and add it to their `retry_if_not_exception_type` set, so the failure returns immediately instead of consuming the \~128-second retry/back-off window. This applies to `LiteLLMEmbeddingEngine`, `FastembedEmbeddingEngine`, and `OpenAICompatibleEmbeddingEngine`; generic `EmbeddingException` failures remain retryable. The `OpenAICompatibleEmbeddingEngine` also now imports the shared `EmbeddingException`/`EmbeddingContextWindowTooSmallError` from `cognee.infrastructure.databases.exceptions` instead of defining a local `EmbeddingException`. No public API signature, configuration option, or environment variable changed; code that already catches `EmbeddingException` continues to catch the new subclass (fixes #3319, PR #3424).
* Fixes `.txt` prompt templates being HTML-escaped on the wire. `render_prompt` configured its Jinja2 environment with `autoescape=select_autoescape(["html", "xml", "txt"])`, and because every prompt template shipped with Cognee is a `.txt` file, every interpolated variable in every rendered LLM prompt was HTML-escaped for all providers — apostrophes became `'`, triplet arrows in retrieval context became `-->`, ampersands became `&`, and angle brackets became `<`/`>`, including the user's own question in completion prompts. Autoescape now covers only markup templates (`["html", "xml"]`), so `.txt` prompts render their content verbatim while `.html`/`.xml` templates remain escaped. The user-visible effect is restored prompt fidelity, reduced token waste, and fewer subtle parsing/extraction issues; the change is internal to prompt rendering and non-breaking — no public API signature, configuration option, or environment variable changed (SDK-203, PR #4115).
* Fixes DLT orphan cleanup leaving forgotten rows in the per-dataset graph and vector stores under `ENABLE_BACKEND_ACCESS_CONTROL`. When re-ingesting a DLT source (or a document source such as Notion/Slack/Google Drive) after rows were removed upstream, Cognee reconciles the corpus by deleting rows no longer present. Under access control the graph and vector engines are dataset-scoped, but the cleanup ran outside the dataset DB context — most visibly on the background-ingest path, where `orphan_cleanup` runs before any pipeline establishes that context — so `delete_data_nodes_and_edges` resolved the *default* engines and the graph + vector purge silently targeted the wrong database, leaving the forgotten row's chunks and entities in place and still retrievable (only the relational record was removed). The per-orphan deletion now runs inside `set_database_global_context_variables(dataset.id, dataset.owner_id)`, so the graph, vector, and relational stores are all purged for the correct dataset. Cleanup remains best-effort: partial failures are logged and retried on the next ingest rather than failing the add. No public API signature, configuration option, or environment variable changed (SDK-189, PR #4090).
***
## v1.4.0
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.4.0)**
Release that bumps the package version from `1.3.0` to `1.4.0` and refreshes `uv.lock`. The release cut itself introduces no functional code, public API, configuration, or environment-variable changes; the entries below are the accumulated work promoted from the development branch in this release (SDK-197). No migration or action is required for existing integrations.
### Highlights
* Fixes `RAG_COMPLETION` and `TRIPLET_COMPLETION` searches ignoring the `node_name` filter. The public `search()` API already accepted `node_name` (and `node_name_filter_operator`) to restrict results to specific node sets, but for these two search types the argument was silently dropped: `CompletionRetriever` and `TripletRetriever` never received it, so their vector lookups (`DocumentChunk_text` / `Triplet_text`) searched the whole collection and returned chunks/triplets from outside the requested node set(s). Both retrievers now accept `node_name` and `node_name_filter_operator`, the search-type factory (`get_search_type_retriever_instance`) forwards them, and they are passed through to the vector search so results are scoped to the given node set(s) using the chosen `AND`/`OR` operator. The change is backward-compatible: `node_name` defaults to `None` (no filtering), so calls that never set it behave exactly as before, and no public API signature, configuration option, or environment variable changed (COG-5868, PR #4053).
* Fixes two `cognee-mcp` bugs that surfaced on the first `remember` of a clean direct-mode (stdio) MCP session. First, the session-backed `remember()` flow now completes the session-to-graph bridge cleanly instead of tripping over dataset setup on the first write. Second, MCP startup migration output no longer pollutes the stdio JSON‑RPC channel, so clients do not misread migration chatter as protocol data. The `remember()` / `cognee.remember()` signatures are unchanged and no configuration option or environment variable changed; the only user-visible tradeoff is a brief one-time delay on the first `remember` while the session is initialized and bridged (SDK-192, PR #4091).
* Fixes `improve()` runs failing to persist agent-trace feedback on multi-tenant (`ENABLE_BACKEND_ACCESS_CONTROL=true`) deployments. The agent-trace-feedback persistence path now forwards the authenticated `user` into `cognee.add()` and `cognee.cognify()`: `cognify_agent_trace_feedback` accepts a `user` parameter and passes it to both calls, and `persist_agent_trace_feedbacks_in_knowledge_graph_pipeline` supplies the pipeline's `user` to that enrichment task. Previously these `add`/`cognify` calls ran as the default user, which has no write ACL on multi-tenant deployments, so trace persistence raised a `403 PermissionDeniedError` and the `improve()` run showed errored memify-pipeline stages while feedbacks were silently skipped. The `user` parameter these internal pipelines and tasks already accepted is unchanged, and no public `improve()`/`memify()` API signature, configuration option, or environment variable was modified (COG-5893, PR #4097).
***
## v1.3.0
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.3.0)**
Release that bumps the package version from `1.2.2` to `1.3.0` and regenerates the dependency lockfiles (`poetry.lock`, `uv.lock`, `cognee-mcp/uv.lock`). The release cut itself introduces no functional code changes; the entries below are the accumulated work promoted from the development branch in this release. Deployers upgrading should re-lock and reinstall to pick up the refreshed dependency graph.
### Highlights
* Fixes document classification for text-like and unknown file extensions. `classify_documents` previously looked up the document class with `EXTENSION_TO_DOCUMENT_CLASS[data_item.extension]`, which raised `KeyError` during ingestion for extensions the map didn't cover — including common text formats (`md`, `json`, `xml`, `yaml`) and uppercase variants such as `.PDF` or `.CSV`. The extension is now normalized to lowercase before lookup, `md`/`json`/`xml`/`yaml` are mapped to `TextDocument`, and any unrecognized extension falls back to `TextDocument` instead of crashing the pipeline. Files that previously failed to ingest are now classified and processed. The change is backward-compatible: already-mapped extensions classify exactly as before, and no public API signature, configuration option, or environment variable changed (fixes #3657, PR #3662).
* Fixes the CLI `cognify` command's `--ontology-file` flag, which previously had no effect. The command passed `ontology_file_path=` to `cognee.cognify()`, but `cognify()` accepts only a `config` object and silently swallowed the unsupported argument through `**kwargs`, so the ontology was never loaded. The command now translates `--ontology-file` into the canonical ontology `config` structure (an `rdflib` resolver with fuzzy matching, built with the same factory `cognify()` uses for its env-based fallback), validates up front that every referenced path exists and otherwise raises a clear `Ontology file not found: <paths>` error, and accepts multiple ontology files as a comma-separated list. Separately, a failed CLI command now always prints its error message: `cognee cognify` failures raise a `CliCommandException` whose `raiseable_exception` field is unset, and the entry point previously printed the message only when that field was set, so the command exited with code `1` but no explanation. Exit codes are unchanged (still `1` on failure), and no new flag, public API signature, or environment variable was introduced (PR #3997).
* Improves concurrency and throughput of LanceDB subprocess mode (`VECTOR_DB_SUBPROCESS_ENABLED=true`) by replacing the session-wide RPC lock with id-based routing. Each async RPC now carries a per-request id and a main-process reader thread routes responses to per-call futures, so concurrent `call_async` operations run in parallel instead of serializing behind a single lock. A new `SUBPROCESS_WORKER_MAX_INFLIGHT` environment variable (default `16`) bounds how many async operations a worker runs at once; it must be `> 0` or worker initialization raises `ValueError` rather than silently degrading. Failure semantics also change: a per-call timeout or cancellation now resolves only that call and no longer tears down the entire subprocess session — the session ends only on genuine crash/shutdown/respawn events, which propagate a `SubprocessTransportError` to any still-pending calls. Synchronous calls (such as the Kuzu graph backend) continue to run serially and are unchanged. In internal Locust benchmarks, `/api/v1/add` average latency dropped from \~1371ms to \~246ms and p95 from \~9300ms to \~520ms, with overall throughput up \~21% (PR #2826).
* Fixes propagation of the authenticated user into the memify session- and feedback-persistence pipelines, correcting multi-tenant attribution. The `persist_sessions_in_knowledge_graph_pipeline` and `persist_agent_trace_feedbacks_in_knowledge_graph_pipeline` functions now set the session user context (`set_session_user_context_variable(user)`) before running memify, so persisted sessions and agent-trace feedbacks are recorded against the intended authenticated user instead of a default/missing user. The `user` parameter these pipelines already accept is unchanged — no public API signature, parameter, or environment variable was modified, and no data migration is required. Logs and stored knowledge-graph entries may now show different (correct) user associations; custom memify pipeline hooks that relied on the previous missing-user behavior should be verified (PR #3950).
* Adds optional per-stage LLM model routing so the extraction, summarization, and query stages can each run on a different model or provider. Each stage reads an optional `LLM_<STAGE>_*` environment group — `LLM_EXTRACTION_*`, `LLM_SUMMARIZATION_*`, and `LLM_QUERY_*`, each accepting `MODEL`, `PROVIDER`, `ENDPOINT`, `API_KEY`, and `API_VERSION` — whose set fields override the base `LLM_*` values for that stage while any unset field falls back to `LLM_*`. Because extraction runs once per chunk and dominates token spend, a common setup routes a cheaper or local model for extraction while keeping a stronger model for summarization and query-time reasoning (for example `LLM_EXTRACTION_MODEL="ollama_chat/llama3.1"`, `LLM_EXTRACTION_PROVIDER="ollama"`, `LLM_EXTRACTION_ENDPOINT="http://localhost:11434"`). Under the hood `LLMConfig.stage_config(stage)` returns a copy of the base config with any stage overrides applied, and a `pipeline_stage(stage)` context manager sets the existing `llm_config` ContextVar to that merged config for the duration of the stage; the client cache key is already derived from the context config, so each stage transparently gets its own cached client. This is fully backward-compatible and requires no action for single-model setups: the stage fields default to empty, so with no `LLM_<STAGE>_*` variables set the effective config is identical to today, and no extraction, summarization, retrieval, or SDK call signature changed. See the "Per-Stage Model Routing" section of [LLM Providers](/setup-configuration/llm-providers) for the full field reference and examples (PR #3961).
* Fixes the missing exception type in error logs. When an exception is logged, `setup_logging()`'s structlog `exception_handler` processor records the exception class name in the `exception_type` field. That field guarded its assignment with `hasattr(exc_type, __name__)`, which used the module's own `__name__` (the string `"cognee.shared.logging_utils"`) rather than the literal `"__name__"`; since an exception class never has an attribute by that name the check was always false, so `exception_type` was never added to the log event. The guard now checks `hasattr(exc_type, "__name__")`, so logged exceptions record their type (e.g. `ValueError`) alongside the existing `exception_message`, identifying what failed rather than only that something failed. No public API, configuration option, or environment variable changed (PR #3998).
* Fixes the `exception_type` field being silently omitted from logged exceptions. The custom `exception_handler` structlog processor in `cognee.shared.logging_utils.setup_logging` checked `hasattr(exc_type, __name__)`, where the unquoted `__name__` resolved to the module's name rather than the literal attribute name — so the check almost never passed and `event_dict["exception_type"]` was never set. The argument is now the string `"__name__"`, so log records for exceptions correctly capture the exception class name (`exception_type = exc_type.__name__`). This only affects the metadata attached to logged exceptions; no public API signature, configuration option, or environment variable changed (fixes #3709, PR #3849).
* Restores a synchronous `get_vector_engine()` as a deprecated backward-compatibility shim and establishes `get_vector_engine_async()` as the canonical async accessor for the vector engine. Both are exported from `cognee.infrastructure.databases.vector`. `get_vector_engine()` is safe to call synchronously from any context — it does no async work when constructing the engine handle — but it now emits a `DeprecationWarning`, and the returned adapter's methods (`embed_text`, `search`, `get_connection`, ...) remain coroutines that must be awaited inside a running event loop. Released users who called `get_vector_engine()` without `await` are unaffected. Dev users who adopted the unreleased async form `await get_vector_engine()` should switch to `await get_vector_engine_async()`, which keeps a uniform "await the engine getter" contract alongside `await get_graph_engine()` (PR #3967).
* Speeds up graph extraction for inputs with many chunks by removing a quadratic scan in `extract_graph_from_data`. DLT row chunks (whose graph is built deterministically from schema metadata rather than by the LLM) are excluded from the extraction path; previously each chunk was matched against the DLT set with a repeated list-membership check that triggered a full Pydantic `__eq__` comparison per pair, so the filter cost scaled with `len(data_chunks) × len(dlt_chunks)`. The function now partitions `data_chunks` into DLT and non-DLT lists in a single pass and returns `integrated + dlt_chunks`, making the extraction hot path linear in the number of chunks (a micro-benchmark reports roughly a 9,000× speedup at 4,000 chunks). Outputs are identical and the change is internal to `extract_graph_from_data` — no public API signature, configuration option, or environment variable changed (fixes #4015, PR #4017).
* Preserves external ontology IRIs end-to-end and adds an RDF/SPARQL read surface plus RDF ingestion. `DataPoint` gains an optional `ontology_uri` field (defaults to `None`) that carries the external IRI a node is grounded in, threaded through `expand_with_nodes_and_edges` so persisted nodes keep their identifier instead of collapsing it to a local label. A new read surface (`cognee.modules.graph.rdf`) exposes the memory graph as RDF: `graph_data_to_rdf`, `export_memory_graph_to_rdf`, `serialize_memory_graph`, and `query_memory_graph_sparql`. Ungrounded nodes receive minted IRIs under `https://cognee.ai/graph/...` so the RDF is well-formed, `is_a` maps to `rdf:type` (individual→class) or `rdfs:subClassOf` (class→class), and other relations become predicate IRIs. RDF ingestion (`cognee.modules.ontology.rdf_xml.rdf_ingest` — `ingest_rdf`, `load_rdf_graph`, `build_datapoints_from_rdf`) parses TBox/ABox into `EntityType`/`Entity` datapoints that preserve verbatim IRIs, with identity derived from the IRI so re-ingesting the same RDF is idempotent. The change is backward-compatible: `ontology_uri` defaults to `None`, no DB migration is required, and the RDF surface rides on the existing `rdflib` ontology dependency (ensure `rdflib` and any parser backends are available in your runtime to use RDF export/ingest). See [Ontologies → RDF read/write surface](/core-concepts/further-concepts/ontologies#rdf-readwrite-surface) (PR #3928).
***
## v1.2.2
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.2.2)**
Patch release that bumps the package version from `1.2.1` to `1.2.2` and refreshes `uv.lock`. This release introduces truth-subspace retrieval improvements, opt-in feedback weighting, and reliability fixes for S3-backed LanceDB setups.
### Highlights
* Adds the truth subspace builder, which compiles accepted session learnings into centroids and slots that can be used to align and rerank retrieval results.
* Adds opt-in truth-subspace reranking and learned feedback weighting for graph search. The default influence remains `0.0`; enable it with `DEFAULT_FEEDBACK_INFLUENCE` or per-call `feedback_influence` values.
* Adds `build_truth_subspace` to the Improve API so truth-subspace indexes can be rebuilt as part of the enrichment flow.
* Tracks the active dataset through request-local context so retrieval and background tasks can keep dataset-scoped truth state aligned.
* Fixes LanceDB dataset provisioning for S3-backed system roots by avoiding direct local directory creation for S3 paths.
* Adds demos and tests for truth-subspace building, reranking, feedback influence, and graph truth-state persistence.
### Other fixes
* Removes the Sentry and Langfuse third-party observability integrations while keeping the OpenTelemetry tracing layer intact. The `Observer.LANGFUSE` enum value, the Langfuse branch in `get_observe()`, and the `LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY` / `LANGFUSE_HOST` configuration fields and environment variables are gone, and Sentry initialization is dropped from the API. The `@observe` decorator now maps only to OpenTelemetry — it emits an OTEL span when tracing is enabled (`COGNEE_TRACING_ENABLED=true`) and is a no-op otherwise — so existing `@observe` usage keeps working unchanged with no call-site edits. **Packaging (breaking):** the `monitoring` extra is removed in favor of the existing `tracing` extra, which installs only the OpenTelemetry API/SDK and OTLP exporters. Migrate with `pip install cognee[tracing]` in place of `pip install cognee[monitoring]`. Users who relied on Sentry or Langfuse should switch to an OTLP-compatible backend; configure it via `OTEL_EXPORTER_OTLP_ENDPOINT` and related `OTEL_*` variables (see [OpenTelemetry Tracing](/integrations/opentelemetry-tracing)). Lockfiles (`uv.lock` / `poetry.lock`) that still reference the removed `sentry-sdk` / `langfuse` packages should be regenerated.
* Fixes a crash when running a pipeline in the background (`run_in_background=True`) with no explicit `datasets`. The background runner (`run_pipeline_as_background_process`) now reads the effective `user` from the run's `params` first and only falls back to the default user when none was supplied, then resolves the run across all datasets that user has write access to. Previously `user` was bound only on the fallback path, so the usual case — a user passed in `params` — left `user` unassigned and raised `UnboundLocalError: cannot access local variable 'user'` before the run started. No API or CLI changes are required.
* Fixes a lock-starvation bug in single-session `improve()`. When `improve()` is called with one `session_id`, it holds a per-session lock so concurrent auto-improve, idle-watcher, and `SessionEnd` runs serialize instead of duplicating work. Previously the lock was released only after a successful run (and on early stage 1–2 failures), so an exception in a later stage — default enrichment (`memify`), the global context index, or the graph-to-session sync — left the lock held permanently, and every subsequent `improve()` for that session silently returned `{}` until the process restarted. All stages are now wrapped in a single `try/finally`, so the session lock is always released on exit regardless of which stage fails. No public signature, parameter, return type, configuration option, or environment variable changed. The fix prevents the issue from recurring; sessions already stuck from before the upgrade still need a process restart to clear the held lock (closes #3313, PR #3317).
* Fixes a CLI startup crash on a fresh, uninitialized database (for example a new Postgres) when no `--user-id` is passed. When `resolve_cli_user()` resolves the default user, it now catches `DatabaseNotCreatedError`, runs the database migrations to create the schema, and retries — so the command proceeds with the default user instead of failing on first run. The recovery is automatic when resolving the default user, including omitted `--user-id` and non-strict fallback-to-default paths; normal calls against an already-initialized database are unaffected and incur no extra overhead. No new flag, configuration option, or environment variable is introduced (fixes #3267, PR #3308).
* Allows one automatic retry on structured-output (instructor) calls in the generic LLM API adapter (`LLM_PROVIDER="custom"` and other generic OpenAI-compatible providers). The adapter's `acreate_structured_output` now passes `max_retries=2` to instructor on both the primary and the content-policy fallback request, where it previously allowed no retry. When the model returns output that fails instructor's schema parsing or validation, instructor reissues the request once before surfacing an `InstructorRetryException`. The user-visible effect is fewer transient structured-output failures, at the cost of a slight latency increase on the rare request that is retried. No public API signature, configuration option, or environment variable changed, and the existing retry-warning logs are unchanged (PR #3413).
***
## v1.2.1
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.2.1)**
Patch release that bumps the package version from `1.2.0` to `1.2.1` and refreshes `uv.lock`. This release follows `v1.2.0` with targeted reliability fixes for dataset-scoped ingestion, background task lifetime, and dataset helper authorization.
### Highlights
* Fixes `remember(..., dataset_id=...)` so it now forwards `dataset_id` to `add()`. Previously `dataset_id` was used only to build the `cognify()` target while `add()` silently ingested raw data into the default `main_dataset`, so `cognify()` ran on the intended (but empty) dataset and produced no new graph. Ingestion and graph building now target the same dataset. No API or migration changes are required; callers who passed `dataset_id` and saw missing results should upgrade.
* Anchors fire-and-forget background tasks so Python's garbage collector can no longer abort them mid-run. Background syncs (`cognee.api.v1.sync.sync.sync`) and background pipeline runs (`run_pipeline_as_background_process`) now hold a strong reference to their in-flight `asyncio.Task` in a module-level set (`_BACKGROUND_SYNC_TASKS` / `_BACKGROUND_PIPELINE_TASKS`) until the task finishes, with a done-callback that discards the reference on completion. Previously the event loop kept only a weak reference, so the GC could collect a still-running task and silently abort a background sync or pipeline run. This fixes those intermittent silent aborts; the only side effect is a small, transient increase in retained memory while tasks run (released as each task completes). No public API signature, request/response schema, configuration option, or environment variable changed.
* Fixes `cognee.datasets.has_data()` raising `AttributeError`. The method now forwards the full `User` object to its internal authorization helper instead of `user.id`, so calls succeed and return the expected `bool`. No signature, parameter, or behavioral change beyond the method no longer crashing.
***
## v1.2.0
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.2.0)**
Release that promotes accumulated `dev` work after the `v1.2.0` development builds, bumping the package version to `1.2.0` and refreshing the lockfile (`uv.lock`). Highlights include ChromaDB search enhancements, BM25 lexical chunk search, search-answer reference evidence, session-context guidance enabled by default, and a range of Postgres/Neptune adapter, visualization, logging, and memory-stability fixes.
### Highlights
* Adds ChromaDB vector search support for `include_payload=False`, so callers can omit metadata payloads from returned `ScoredResult` values when they only need ids and scores.
* Adds ChromaDB `node_name` filtering for `search()` and `batch_search()`, including `OR` and `AND` semantics through `node_name_filter_operator`.
* Prevents Entity and EntityType node id collisions by namespacing generated ids by node category.
* Excludes internal `EntityType` taxonomy nodes and their `is_a` edges from the schema inventory output (`get_schema_inventory` and the visualize schema inventory endpoint). Consumers no longer receive a separate `EntityType` type group or `is_a` relationship aggregates; entity instances are still grouped under their resolved semantic type.
* Improves ontology parsing for file-like inputs with filename/content-type detection, RDFLib fallback formats, and clearer initialization errors when parsing fails.
* Reaps subprocess database workers deterministically at interpreter exit. The `cognee_db_workers` harness now registers an `atexit` handler that force-terminates any still-live LanceDB/Kuzu worker processes on shutdown, instead of relying on garbage-collector and `__del__` ordering that is not guaranteed to run at interpreter exit (notably for Windows `spawn` daemon workers). This helps avoid leftover worker processes and shutdown hangs when running with `graph_database_subprocess_enabled=true` or `vector_db_subprocess_enabled=true`.
* Offloads the Ollama adapter's blocking client calls off the asyncio event loop. The `LLM_PROVIDER="ollama"` adapter wraps a synchronous OpenAI-compatible client, so its chat-completion, audio-transcription, and image/vision calls previously ran inline and blocked the running event loop for the full duration of each Ollama request, serializing concurrent async callers (for example the per-chunk extraction that `cognify()` fans out). These calls are now dispatched through `asyncio.to_thread`, so they execute in worker threads and no longer stall the loop. Public async signatures are unchanged and no configuration or migration changes are required; because requests now run in worker threads, any objects shared with the Ollama client should be thread-safe.
* Serializes concurrent decodes on the shared llama.cpp local in-process model. The `LLM_PROVIDER="llama_cpp"` local (in-process) adapter now guards calls into its single `llama_cpp.Llama` instance with a lock, so the per-chunk extraction that `cognify()` fans out via `asyncio.gather`/`asyncio.to_thread` no longer decodes on the same non-thread-safe instance concurrently. This helps avoid native `GGML_ASSERT` crashes from corrupted KV-cache/logits state during local llama.cpp runs; in-process requests are now processed one at a time (use server mode for parallel decoding).
* Simplifies the structured-output schema sent to the LLM during graph extraction when a custom `graph_model` (a `DataPoint` subclass) is used. `extract_content_graph` now converts the model to a plain `BaseModel` that keeps only the fields you declare on each subclass — DataPoint infrastructure fields (such as `id`, `created_at`, `version`, `type`, `belongs_to_set`) and the `metadata` field are dropped from the schema the LLM is asked to fill — and then rehydrates the LLM result back into your original `DataPoint` model via `model_validate`. The LLM extracts only your domain fields, while declared `metadata` defaults (for example `{"index_fields": ["name"]}`) are preserved on the rehydrated objects, so indexing behavior is unchanged. This is a no-action change for callers of the high-level extraction, `cognify`, and `remember` APIs.
* Fixes Neptune (`GRAPH_DATABASE_PROVIDER="neptune"`) edge writes for relationship types that contain spaces, hyphens, or openCypher reserved words. The adapter now backtick-quotes (and escapes embedded backticks in) the relationship type when interpolating it into the generated openCypher `MERGE` statements for both single-edge and batched (`UNWIND`) edge upserts, preventing query syntax errors and unsafe interpolation. Also fixes the batched-edge fallback path so that when a batch insert fails, the per-edge retry iterates the edges for that relationship instead of the relationship grouping map. No configuration or migration changes are required, but generated/logged openCypher will now show backtick-quoted relationship-type names.
* Reuses a single `aiohttp.ClientSession` across anonymous telemetry requests instead of opening a new session per call. This avoids a repeated DNS + TCP + TLS handshake to the telemetry endpoint on every event, helping lower latency and connection churn for telemetry. The shared session is created lazily inside the running event loop and rebuilt transparently when the loop changes (for example across tests or `asyncio.run` boundaries) or after it is closed; telemetry stays best-effort and never raises. No new configuration is required, and telemetry can still be turned off with `TELEMETRY_DISABLED=true`.
* Tolerates a missing `dataset_database` table on PostgreSQL during startup migrations and pruning. The `run_startup_migrations()` vector step and the graph/vector prune routines now also catch the asyncpg `ProgrammingError` / `UndefinedTableError`, in addition to the SQLite `OperationalError` already handled. Running against a fresh PostgreSQL/pgvector database (for example the pgvector example) now skips the step with a warning instead of crashing with an undefined-table error.
* Reduces peak memory use of the Postgres graph adapter (`GRAPH_DATABASE_PROVIDER="postgres"`) for graph node and edge relational upserts. `add_nodes`/`add_edges` now stream each batch to Postgres in fixed-size chunks (1000 rows per `INSERT ... ON CONFLICT` statement) instead of compiling one large multi-thousand-row statement, and JSONB property columns are serialized once at execute time via an engine-level `json_serializer` (the UUID/datetime-aware `JSONEncoder`) rather than a per-row `json.loads(json.dumps(...))` round-trip. This helps avoid the transient allocation churn and memory spikes seen on large single-batch writes (the commit reports roughly a 20x reduction). The number of rows written, the upsert/conflict semantics, and the data stored are unchanged; no configuration or migration changes are required.
* Switches `CHUNKS_LEXICAL` search to BM25 ranking. Lexical chunk searches now rank exact-term matches with BM25 instead of the previous Jaccard-style scorer, and the retriever filters default stop words unless explicitly configured otherwise. API signatures stay the same, but result ordering can change for `SearchType.CHUNKS_LEXICAL`.
* Adds lightweight references (Evidence) to completion-style search answers via a new `include_references` flag (default `true`) on `search()`, `recall()`, and the `POST /api/v1/search` and `POST /api/v1/recall` request bodies. When enabled, a deterministic `Evidence:` block is appended to the answer text, assembled in-process (no extra LLM call) from the retrieved chunk payloads, falling back to entity → chunk → document graph traversal when chunk metadata is missing. The response schema and return types are unchanged — Evidence is added to the answer text only. Because this changes default answer text, snapshot and evaluation baselines will diff; set `include_references=False` to restore the exact prior output. Older indexes lacking the new `document_id`/`document_name` chunk fields use the graph fallback where available or omit the Evidence block silently.
* Disables local-variable rendering in logged exception tracebacks. `setup_logging()` now configures the console renderer with `RichTracebackFormatter(show_locals=False)`, so when an exception is logged the traceback no longer expands each frame's local variables. In the retrieval/search path those locals can hold graph objects carrying embedding vectors and deep node/edge references, and rendering them recursively spiked memory to multiple GB and OOM-killed the process (notably in CI) whenever an exception was logged mid-search. Tracebacks themselves are still logged; only the per-frame locals dump is omitted. No configuration changes are required.
* Restores Cognee's safe uncaught-exception hook. `setup_logging()` now installs `sys.excepthook` so non-`KeyboardInterrupt` exceptions are logged through structlog before Python's default traceback is printed, and falls back to plain traceback output if rich rendering itself fails. No configuration changes are required.
* Propagates relational `DATABASE_CONNECT_ARGS` SSL settings to the Postgres maintenance, PGVector, and graph Postgres engines, so connections to managed Postgres that enforce SSL (for example Neon, RDS/Aurora, Azure Database for PostgreSQL) succeed. Previously only the main relational engine received these args, so CREATE/DROP DATABASE maintenance, per-dataset PGVector engines, and the `GRAPH_DATABASE_PROVIDER="postgres"` graph engine could fail with missing-SSL errors. The maintenance engine also maps the libpq `sslmode` key to the asyncpg `ssl` key, and rewrites a Neon `-pooler.` host to its direct endpoint because CREATE/DROP DATABASE cannot run through Neon's PgBouncer pooler. No configuration schema change is required — supply asyncpg SSL options via the existing `DATABASE_CONNECT_ARGS` and they are now honored across Cognee's Postgres engines; the env unset stays a no-op for in-cluster Postgres. Deployments on managed Postgres with enforced SSL should upgrade.
* Fixes two interaction glitches in the graph visualization (`visualize_graph`) story view. Clicking a node no longer displaces it: the click-vs-drag threshold is raised from 3px to 6px so trackpad jitter on a plain click is no longer treated as a drag that reheated the force simulation and sent the clicked node (and, in the **Force** and **Flow** layouts, the whole layout) flying off the canvas. In the **Story** layout, dragging now drives the node position directly instead of reheating the pinned grid, and a released node snaps back cleanly to its lane. Separately, the pipeline stage-header pills (shown in the **Story** and **Flow** layouts) are now drawn in a final pass after edges, nodes, and labels, so a dense graph panned toward the top of the viewport can no longer paint over them. Generated visualization HTML changes only; no API, configuration, or migration changes are required.
* Retries the Kuzu/Ladybug JSON extension load on the live connection when it is missing at runtime. In the subprocess graph worker (`graph_database_subprocess_enabled=true`), if `LOAD EXTENSION` fails with a "not been installed" error, the worker now runs `INSTALL` on the active connection and retries the load once; if that `INSTALL` fails it raises with the real underlying cause instead of the generic load error. This recovers from cases where the best-effort warm-up install on the throwaway database did not complete (for example a transient network error while downloading the extension on a fresh machine). The warm-up install path also now logs its failure cause to stderr (`[ladybug worker] warm-up INSTALL JSON failed: ...`) instead of swallowing it silently, so these conditions are diagnosable from worker/CI logs. The retry may perform an extension install on the live connection and add a small startup delay; no configuration or migration changes are required.
* Relaxes the bundled Ladybug graph-store dependency from the `ladybug==0.16.0` pin to `ladybug>=0.16.0,<0.18`, so installs can pick up the `0.17.x` line. The database migration worker's storage-version table now maps the `0.17` on-disk format (catalog code `41`) to `0.17.1`, so an existing `0.16.x`/`0.17.x` graph database is recognized as current and is not flagged for legacy migration (migration still targets only pre-`0.15.0` databases). No manual database schema changes are required; deployers upgrading should re-lock dependencies (refresh `uv.lock`) and redeploy database workers to pick up the new range.
* Adds a session-context guidance layer and turns it on by default. The cache `AUTO_FEEDBACK` setting now defaults to `true` (previously `false`), so when `CACHING` is enabled, session-capable completion searches run one additional structured-output LLM call per answered turn under the resolved session (`default_session` when `session_id` is omitted) to analyze the current turn against the previous one. The analysis can rewrite the turn into an effective query used for retrieval, accumulate durable per-session guidance grouped into `goals`, `rules`, `preferences`, and `lessons_learned` that can be injected into later answers, and **gate** a follow-up turn — returning a short acknowledgement (the analysis reply, or `"Got it."`) instead of running retrieval and completion. The step fails open to answering the original query when analysis errors or no session is available. Because guidance and the effective query can change retrieval inputs, session answers and turn gating may differ from history-only sessions, and per-turn latency and token usage increase. Set `AUTO_FEEDBACK=false` to disable and restore plain conversation-history replay. See [Sessions and Caching](/core-concepts/sessions-and-caching#session-context-guidance-auto-feedback).
* Bypasses the Instructor structured-output pipeline when `acreate_structured_output` is called with `response_model=str` on the default OpenAI, generic, and Ollama LLM adapters. Plain-text requests are now sent directly to the provider and the model's raw string content is returned, instead of being wrapped in Instructor's JSON/tool-call schema. This avoids repeated parse failures and retry storms on local llama.cpp-compatible servers that don't honor those schemas, and can lower latency for string completions. Rate limiting still applies to these direct calls. Passing a Pydantic model is unchanged — it still returns a validated model instance — so this is a no-action change for callers.
* Fixes the condition that gates name-to-UUID resolution of the `datasets` argument in `search()`. The check was wrapped in a single-element list (`[all(...)]`), which is always truthy, so the name-resolution path ran for any non-`None` `datasets` value. It now runs only when every entry in `datasets` is a string. Passing dataset names (the documented usage) is unaffected; the only behavior change is that non-string entries supplied through `datasets` (for example already-resolved UUIDs) are no longer forced through name-based authorization lookup — pass UUIDs via `dataset_ids` as before. No API signature, default, or migration change is required.
* Corrects two `.env.template` knob names that the config loader was ignoring. The template previously listed `LLM_MAX_TOKENS` and `EMBEDDING_MAX_TOKENS`, but Cognee's settings classes read these values from `LLM_MAX_COMPLETION_TOKENS` (default `16384`) and `EMBEDDING_MAX_COMPLETION_TOKENS` (default `8191`). Anyone who copied the old template and set the chunk-sizing limits under the previous names had them silently ignored, so chunk sizing fell back to the defaults. If you relied on those entries, rename them to `LLM_MAX_COMPLETION_TOKENS` / `EMBEDDING_MAX_COMPLETION_TOKENS` in your `.env`. No code or schema changes are required. The same `.env.template` update also documents additional already-supported settings (LLM/embedding tuning and rate limiting, chunking, session cache, graph/vector connection and subprocess tuning, Langfuse monitoring, llama.cpp, and the auth-token secrets) as commented examples with their defaults.
* Fixes an `UnboundLocalError` in file metadata extraction (`get_file_metadata`) when the underlying file-like object cannot seek. `content_hash` is now initialized before the seek/hash attempt, so when `file.seek(0)` raises `io.UnsupportedOperation` (the error is still logged), metadata is returned with an empty `content_hash` instead of crashing. This makes ingestion more robust for non-seekable file-like inputs; no configuration or API changes are required.
* Preserves structured search completions in the result payload. `SearchResultPayload.completion` now models a single `dict`, a Pydantic `BaseModel`, and a list of models in addition to the previous `str` / list-of-string / list-of-dict shapes, and a custom serializer dumps model instances to their `dict` representation. This fixes searches that pass a non-string `response_model` (typed LLM output via `retriever_specific_config`): the structured object is kept as-is instead of being dropped or coerced into an empty model. The default string-answer path is unchanged; no configuration or migration changes are required.
* Caps LLM retries and recovers from over-length embedding input. The structured-output and transcription adapters (anthropic, azure\_openai, gemini, generic\_llm\_api, llama\_cpp, mistral, ollama, openai) now stop on a fixed number of attempts (`stop_after_attempt`) instead of the previous time-based `stop_after_delay(128)` window, and the instructor retry counts were lowered (for example structured-output generation tops out at 3–4 attempts, while Bedrock's and the other adapters' inner instructor `max_retries` drop to 1–2). This makes transient failures fail faster and at lower cost, with a small reduction in resilience to intermittent errors — watch your LLM error/latency/cost metrics after upgrading. Separately, `LiteLLMEmbeddingEngine` now recovers from over-length embedding input: a context-window error or a `400 BadRequestError` matching `maximum input length` triggers recursive split-and-pool (splitting the batch, or splitting a single string into overlapping halves and averaging the resulting vectors) instead of failing, while other 400 errors still fail fast. No API or configuration changes are required. See [Embedding Providers → Timeout and Retry Behavior](/setup-configuration/embedding-providers).
* Bounds the input-data preview persisted in the `pipeline_runs.run_info` column so a single run cannot grow the table without limit. On pipeline run start, error, and completion, the audit-only `run_info` data is summarized: a list of `Data` records is still reduced to their IDs and empty input is still recorded as `"None"`, but any other payload is now stringified and truncated to a 512-character preview ending with `... [truncated, <N> chars total]` instead of being stored verbatim. `run_info` is never read back during processing; persist large raw inputs (for example text passed to `add()`/`cognify()`) elsewhere if you need the full payload. No configuration or migration changes are required.
* Fixes `forget(everything=True)` under multi-tenant per-dataset database isolation (`ENABLE_BACKEND_ACCESS_CONTROL=true`). The `everything` branch no longer runs inside a single-dataset database context; the per-dataset context is now established per dataset inside the underlying delete-all flow. Previously, entering a single-dataset context with no dataset reference could try to create a `dataset_database` row for a non-existent dataset and fail the operation. Single-dataset, single-item, and `memory_only` modes are unchanged, and the public `forget()` signature, return shapes, and error messages are unchanged.
* Forwards `FALLBACK_ENDPOINT` to the OpenAI adapter's content-policy fallback request (`LLM_PROVIDER="openai"`). Previously this `api_base` override was not applied, so the fallback completion always went to the default OpenAI endpoint even when `FALLBACK_ENDPOINT` was set; now the fallback request is routed to the configured base URL. Deployments that set `FALLBACK_ENDPOINT` to an OpenAI-compatible proxy or alternate endpoint will see their fallback traffic go there. `FALLBACK_ENDPOINT` remains optional for `openai` — when unset, the fallback still uses the default OpenAI endpoint.
* Caps the `instructor` dependency at `<1.15.3` (previously `<2.0.0`) and lowers the `litellm` minimum to `>=1.83.7` (previously `>=1.84.0`). This pins structured-output extraction to a known-good `instructor` range and widens the compatible `litellm` window; lockfiles (`poetry.lock`, `uv.lock`) are refreshed to match. No API or behavioral changes — callers using the high-level `cognify`/`search` APIs are unaffected. Developers and deployers should re-lock and reinstall dependencies to pick up the new constraints.
* Detects Markdown, JSON, XML, and YAML files by extension during file-type guessing. `guess_file_type` now returns deterministic types for `.md`/`.markdown` (`text/markdown`), `.json` (`application/json`), `.xml` (`application/xml`), and `.yaml`/`.yml` (`application/yaml`) instead of relying on content-based detection, which has no magic-number signature for these formats and fell back to `text/plain`/`txt`. The recorded file metadata (`mime_type` and `extension`) for these files now reflects their actual format. Loader selection is unchanged — `TextLoader` already handled these extensions — so no action or migration is required.
* Adds a `GET /api/v1/proposals/{proposal_id}` endpoint for reviewing a stored skill-improvement proposal before applying it. The endpoint takes a required `dataset_id` query parameter and returns the proposal's `status` (`proposed`/`applied`), `confidence`, `rationale`, `model_name`, and before/after procedures (`old_procedure`/`proposed_procedure`); it is read-only and never mutates the graph (applying still goes through `POST /api/v1/remember/entry` with `skill_improvement`). It returns `403` when the caller is not authorized for the dataset and `404` when the proposal is not found.
* Adds no-code, inline skill ingestion. `POST /api/v1/remember` (with `content_type=skills`) now accepts `skills_text` (a `SKILL.md` markdown body as a string) and `skill_name` (the skill name/slug, defaults to `skill`) form fields, so a skill can be ingested without uploading a file — when `skills_text` is set and no files are uploaded, it is written to a temporary `SKILL.md` and ingested through the existing skills pipeline. A new `POST /api/v1/skills` endpoint exposes the same inline ingestion via a JSON body (`skills_text`, optional `skill_name`, and one of `dataset_name`/`dataset_id`).
### Notes
* Includes a behavior-preserving cleanup of the LiteLLM embedding engine (`LiteLLMEmbeddingEngine`): no public `__init__` signature, env-var (`MOCK_EMBEDDING`, `EMBEDDING_ENDPOINT`), or default changes, and embedding behavior is unchanged.
* Deployers upgrading should re-lock dependencies (refresh `uv.lock`) and reinstall, then rebuild/redeploy to pick up the updated dependency set.
* The ontology parser update improves file-like parsing behavior; upload endpoint format restrictions should be documented separately if they change.
***
## v1.1.3
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.1.3)**
Patch release focused on API-mode robustness and dependency safety. It enables remote pipeline status checks for MCP/API deployments, improves vector retrieval behavior for empty input, and tightens the `instructor` dependency range.
### Highlights
* Enables `cognify_status` in API mode. The MCP can resolve dataset IDs remotely and read pipeline status from `GET /api/v1/datasets/status`, so self-hosted API deployments can check background pipeline status without local database access.
* Adds API-mode support to `CogneeClient.get_pipeline_status`, which now queries the server's `/api/v1/datasets/status` endpoint instead of raising `NotImplementedError`.
* Makes LanceDB retrieval return an empty list when called with an empty id list, preventing avoidable errors for callers that sometimes have no vector ids to fetch.
* Pins `instructor` below `1.15.3` and refreshes lock metadata. Deployers with exact dependency pins should re-lock or reinstall against the updated constraints.
* Refreshes the README with clearer Cognee positioning, branding, and a research paper link.
***
## v1.1.2
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.1.2)**
Patch release with a refreshed public frontend, improved Cloud UI workflows, and a Postgres graph adapter compatibility fix for asyncpg/PostgreSQL 16.
### Highlights
* Syncs the public frontend with the SaaS application, bringing updated dashboard, search, dataset, connection, onboarding, knowledge graph, and graph model editor experiences.
* Adds conversation-based search history and refreshed multi-dataset search flows in the frontend.
* Improves connection and onboarding flows with a connection modal, step-by-step agent setup guidance, new quickstart assets, and updated loading visuals.
* Adds memory customization UI support for datasets, including graph models, custom prompts, and ontology-related configuration.
* Fixes Postgres graph neighborhood expansion under asyncpg/PostgreSQL 16 by casting recursive CTE seed parameters to `text[]`.
### Notable Changes
* Bumps the package version from `1.1.1` to `1.1.2` and refreshes lockfiles.
* Aligns frontend API routes and local development behavior with the OSS backend.
* Updates API key, tenant, configuration, dataset, ingestion, ontology, search-history, session, analytics, and user frontend modules.
* Adds frontend assets for quickstarts, agent integrations, loading states, and graph previews.
* Adds regression coverage for Postgres graph neighborhood seed array typing and retries a flaky usage-logger e2e path in CI.
### Fixes and Improvements
* **Postgres neighborhood query parameter typing**: The Postgres graph adapter's `get_neighborhood` query now casts the seed parameter to `text[]` (`unnest(CAST(:seeds AS text[]))`) in its recursive CTE seed row. Deployments using `GRAPH_DATABASE_PROVIDER=postgres` with asyncpg/PostgreSQL 16 should no longer hit parameter type inference errors when expanding neighbors from seed node ids.
* **Cloud UI refresh**: Dashboard, dataset, dataset detail, connections, search, onboarding, knowledge graph, and graph model editor screens were refreshed and aligned with current Cloud workflows.
* **Search and dataset workflows**: Search now supports conversation history and multi-dataset recall flows, while dataset pages add improved status polling, graph access, and memory customization entry points.
* **Connect Agent flow**: The frontend adds clearer connection setup prompts, modal-based setup guidance, and integration visual assets.
* **Frontend resilience**: Error handling, loading states, analytics logging, tenant context, user configuration, and local fetch behavior were updated across the public frontend.
***
## v1.1.1
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.1.1)**
Patch release that promotes accumulated `dev` work after `v1.1.1.dev0`, with agent-management APIs, graph visualization updates, custom graph-model support in `remember`, and backend stability fixes.
### Highlights
* Adds agent management and connection endpoints for listing, creating, inspecting, registering, unregistering, and deleting agents and their active connections.
* Reworks graph visualization with a pipeline-aware Story layout, Schema view, improved labels, legends, and modular visualization components.
* Adds `graph_model` support to the `remember` REST endpoint, letting API callers pass a JSON-serialized graph schema into ingestion.
* Expands graph and retrieval behavior with local Neo4j dataset handling, global context graph bucketing, improved edge text, and `node_name` filtering for chunk retrieval.
* Improves LLM, PGVector, remember/session, prune, forget, and graph-projection error handling.
### Notable Changes
* Bumps the package version from `1.1.0` to `1.1.1` and refreshes the release lockfiles.
* Splits agent lifecycle and connection handling into dedicated modules and API routes, including persisted agent connection state and agent-session names.
* Adds SDK/API support for retrieving specific agent configuration and for inspecting current agent connections.
* Adds local Neo4j dataset database handling and updates graph database selection to recognize that handler.
* Reworks global context index internals with graph bucketing, scoring, build, update, load, summarize, and persistence flows.
* Improves edge indexing and rendering by preserving natural edge descriptions, generating fallback edge text from metadata, and rendering relationship labels inside edge markup.
* Updates CI and test coverage across database adapters, agents, visualization, global context indexing, retrieval filters, and LLM configuration.
### Fixes and Improvements
* **Remember custom graph models**: The `remember` REST endpoint now accepts an optional `graph_model` form field, parses the JSON schema into a graph model, and forwards it into the ingestion flow.
* **Agent lifecycle and connections**: Agent endpoints now separate agent resources from agent connections, support agent-session names, persist connection metadata, mark unregistering agents inactive, and expose connection detail.
* **Graph visualization**: Story view spacing, column pinning, schema rendering, edge-label rendering, and fallback labeling were improved so generated graph views are easier to inspect.
* **Graph ingestion and retrieval**: Edges with unprojectable endpoints are skipped instead of failing graph projection, `KnowledgeGraph` subclasses follow the knowledge-graph integration path, chunk retrieval receives `node_name` filters, and `forget` can handle dataset values that are string UUIDs.
* **PGVector metadata consistency**: `create_collection` now reflects SQLAlchemy metadata only after the table-creation transaction commits, avoiding stale metadata entries when table creation rolls back.
* **LLM adapters**: Generic LLM API transcription and Ollama image transcription now raise clear `ValueError` messages for empty responses, Mistral guards against `None` messages before reading content, and OpenAI instructor mode is honored.
* **Session remember routing**: `remember(session_id=...)` now routes through the JSON `/entry` endpoint in API mode, and using `custom_prompt` with `session_id` raises a clear `ValueError`.
* **Operational stability**: Prune errors and dataset lookup issues are handled more defensively, brittle batch-query test settings were adjusted, and optional LLM configuration can be passed through CI.
***
## v1.1.0.dev1
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.1.0.dev1)**
Developer preview release on the way to `v1.1.0dev1`. This release includes API, retrieval, permissions, storage-runtime, and backend consistency changes.
### Highlights
* Adds database subprocess workers for LanceDB and Kuzu so native database work can run outside the main Cognee process. The wheel now includes the `cognee_db_workers` package.
* Exposes more ingestion controls through the public API and remote client paths, including chunk sizing and background execution options for `remember()` and `cognify()`.
* Adds `dataset_ids` support to `recall()`, making shared-dataset retrieval more reliable when dataset names are not owned by the calling user.
* Expands permission management with DELETE endpoints for dataset permissions, roles, and user-role membership.
* Improves session visibility so parent users can see sessions created by child-agent users where appropriate.
### Notable Changes
* Adds `graph_database_subprocess_enabled` and `vector_db_subprocess_enabled` configuration, plus Kuzu tuning variables for threads, buffer pool size, and max DB size.
* Keeps `belongs_to_set` metadata consistent across dataset deletion and shared-node/vector upserts in LanceDB, PGVector, and Neo4j paths.
* Adds `include_payload` behavior to Neptune Analytics vector search.
* Improves Postgres hybrid batching by respecting embedding-engine batch size.
* Improves infer-schema text sampling and prompting.
* Rewrites the examples README into a fuller index and adds performance-testing support with Locust.
* Deprecates `.env.example` as the canonical template in favor of `.env.template`.
* Bumps the package version from `1.0.9` to `1.1.0.dev1` and refreshes lockfiles.
***
## v1.0.3
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.0.3)**
Patch release with bug fixes and stability improvements on top of v1.0.2.
### Highlights
* Promotes accumulated `dev` work to `main` for the `v1.0.3` release
* Adds session lifecycle APIs, unified memory/session handling, and dashboard support
* Introduces dataset queueing for async context management and ingestion flows
* Ships new relational migrations, including session lifecycle tables and `parent_user_id`
* Expands recall/remember and cloud routing behavior, plus frontend onboarding and Connect Agent updates
### Notable Changes
* Added session endpoints, metrics, and supporting persistence work
* Added dataset queue infrastructure and follow-up fixes for background processing
* Added database migrations for new tables and user/dataset ownership handling
* Updated recall, remember, improve, and search-related API behavior
* Added frontend work for Connect Agent, dashboard/activity views, API keys, and onboarding
* Included guide updates, workflow/tooling changes, and dependency updates such as `litellm` and `onnxruntime`
### Bug Fixes
* **PostgreSQL null-byte compatibility**: Embedded null bytes (`\x00`) in node or edge string fields no longer cause errors when using PostgreSQL as the relational backend. Null bytes are now automatically stripped from all string values (including nested attributes) before writes to the relational store. This sanitization is transparent — affected strings are silently cleaned rather than rejected.
* Fixed duplicate graph nodes caused by `DataPoint.id` being regenerated during graph construction. The original `id` is now preserved when converting DataPoint instances into graph nodes, ensuring node identity is stable across graph extraction passes.
***
## v1.0.2
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.0.2)**
Patch release with bug fixes and stability improvements on top of v1.0.1.
### Bug Fixes
* **LanceDB schema migration**: "contained null values" errors (raised when old rows lack a field required by a newer DataPoint schema) are now treated as recoverable schema drift. The affected table is automatically rebuilt from the current schema instead of raising a hard failure.
* **cognee-mcp Docker image build**: Added missing `build-essential` and `libpq-dev` system packages to the builder stage so that `cognee[postgres]` can compile `psycopg2` from source on Linux.
### Dependency Updates
* Bumped `llama-index-core` requirement from `>=0.13.0,<0.14` to `>=0.14.20,<0.15` for the `llama-index` extra.
* Pinned `nltk>=3.9.3,<4` explicitly in the `docs` extra to satisfy `unstructured`'s dependency until `unstructured` v0.21.0.
***
## v1.0.1
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.0.1)**
Patch release with bug fixes on top of v1.0.0.
***
## v1.0.0
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v1.0.0)**
### Highlights
* **New high-level API**: `remember`, `recall`, `improve`, and `forget` cover the full memory lifecycle in four operations
* Session-aware memory via `session_id` — short-term context that can be promoted into the permanent graph
* Unified `recall` replaces the previous `search` call with automatic retrieval strategy selection
* Legacy operations (`add`, `cognify`, `search`, `memify`) remain available as lower-level building blocks
### New Features
* `cognee.remember(data, session_id=...)` — ingest and graph in one call; supports permanent or session memory
* `cognee.recall(query, session_id=...)` — query across both the permanent graph and session cache
* `cognee.improve(...)` — enrich an existing graph with feedback-based weighting and session promotion
* `cognee.forget(dataset=..., session_id=...)` — delete data, datasets, or full session memory
***
## v0.5.4.dev1
**Released:** March 5, 2026\
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v0.5.4.dev1)**
### Highlights
* Developer preview release focused on quality, performance, and developer ergonomics
* Faster ingestion and sync
* Improved search relevance and new filtering options
* Stability fixes for memory creation, deletion, and CLI workflows
* Internal refactoring and dependency upgrades
### New Features
* Bulk import CLI for faster batched ingestion
* Search filters for tags and date ranges
* Optional per-collection ingestion throttling
### Improvements
* Lower latency for ingestion and sync
* Better search ranking
* More robust deletion and duplicate handling
* Clearer CLI messages and debug logs
### Bug Fixes
* Fixed duplicate memories under concurrent ingestion
* Fixed partial state after deletion
* Fixed CLI export formatting issues
* Fixed intermittent retrieval failures under load
***
## v0.5.3
**Released:** February 27, 2026\
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v0.5.3)**
### Highlights
* New graph visualization improvements
* Expanded permissions and user management work
* SessionManager and cache/session persistence work
* Search and graph retrieval improvements
* Multiple stability and CI/CD fixes
### Notable Changes
* Added role-based permission checks and permission endpoints
* Added graph visualization updates, including note set coloring
* Added return type hints to API functions
* Added chunk associations for the memify pipeline
* Added vector filtering based on node sets
* Fixed delete flow bugs, health check issues, MCP issues, and several config/integration issues
***
## v0.5.3.dev1
**Released:** February 20, 2026\
**[View on GitHub](https://github.com/topoteretes/cognee/releases/tag/v0.5.3.dev1)**
### Highlights
* Added vector filtering based on node sets
* Added principal Cognee configuration
* Fixed health check issues
* Fixed FalkorDB adapter port bug
* Fixed Ollama image ingestion argument issue
### Notes
* Includes a small set of targeted fixes and feature work on top of `v0.5.3.dev0`
* Introduced one new contributor in this release
# Cognee CLI
Source: https://docs.cognee.ai/cognee-cli/overview
Command line interface for Cognee AI memory operations
The `cognee-cli` command lets you run Cognee from the terminal so you can remember data, enrich memory, and ask questions without opening a Python file. The commands are designed to be short, use friendly defaults, and are safe for people who are just starting out.
## Install the CLI
The `cognee-cli` command ships with the `cognee` package — installing `cognee` makes it available. To use it inside a project, install Cognee the usual way (see the [Installation Guide](/getting-started/installation)):
<CodeGroup>
```bash pip theme={null}
pip install cognee
```
```bash uv virtual environment theme={null}
uv pip install cognee
```
```bash uv project (pyproject.toml) theme={null}
uv add cognee
```
</CodeGroup>
To install the CLI **globally** so the `cognee-cli` command works from anywhere, use a tool that installs Python applications into isolated environments and puts their commands on your `PATH`:
```bash theme={null}
# With pipx
pipx install cognee
# Or with uv
uv tool install cognee
```
This exposes `cognee-cli` system-wide without polluting your project's virtual environment. To add an extra (for example Postgres support) to the global install, include it in the package spec, e.g. `pipx install "cognee[postgres]"` or `uv tool install "cognee[postgres]"`.
## Try the Demo Graph
Before configuring anything, `cognee-cli demo` gives you a working example with **no API key and no embedding provider**, on a machine with no network access:
```bash theme={null}
# Import the bundled graph and run the two built-in example queries
cognee-cli demo
# Ask your own question instead
cognee-cli demo --query "Where does Alice live?"
# Load it into a different dataset, and return more results per query
cognee-cli demo --dataset-name my_demo --top-k 5
```
A small pre-built [COGX archive](/core-concepts/further-concepts/cogx) ships inside the `cognee` package. The command restores it **graph-only** — no embeddings are computed — and then queries it with `CHUNKS_LEXICAL`, a keyword (BM25) search that needs neither an LLM nor an embedding provider. It prints how many nodes and edges landed, the matching text for each query, and the next steps to take. Nothing in the run reaches the network except Cognee's anonymous telemetry event, which is best-effort and switched off by [`TELEMETRY_DISABLED=true`](/setup-configuration/overview#observability--telemetry).
<Accordion title="Demo Command Options">
* `--dataset-name` (`-d`): Dataset to load the demo graph into (default: `demo`)
* `--query` (`-q`): Run this single query instead of the two built-in example queries (`"Who works at Anthropic?"` and `"What does cognee depend on?"`)
* `--top-k` (`-k`): Maximum number of results per query (default: `3`)
</Accordion>
When you're done, remove the demo dataset — matching the name you loaded it into:
```bash theme={null}
cognee-cli forget --dataset demo
```
<Note>
`demo` pins `AUTO_FEEDBACK=false` for its own process, so the per-turn analysis LLM call that normally follows an answered query cannot fire on a machine without an API key. This applies only to the `demo` process — it does not change your configuration.
LLM-backed answers over your own data (`GRAPH_COMPLETION`, `HYBRID_COMPLETION`, and the other completion types) still require `LLM_API_KEY`; continue with [Setup](#setup) below. `demo` is not among the commands forwarded in [`--api-url` mode](#talk-to-a-running-cognee-api) — running it with `--api-url` set fails with an error.
</Note>
## Setup
Before using the CLI, you need to configure your API key. The recommended approach is to store it in a `.env` file:
```bash theme={null}
# Create a .env file in your project root
echo "LLM_API_KEY=your_openai_api_key" > .env
```
Alternatively, you can export it in your terminal session:
```bash theme={null}
export LLM_API_KEY=your_openai_api_key
```
<Note>
`cognee-cli config set` writes to a `.env` file too — it saves the value into `.env` in the directory you run it from, so the setting survives across CLI invocations. See [Manage Configuration](#manage-configuration) for details.
</Note>
Once the `.env` is in place, run `cognee-cli doctor` to confirm the configuration and local services are usable before you ingest anything — see [Diagnose Your Setup](#diagnose-your-setup).
## Quick Tour of Commands
* `cognee-cli demo` loads a bundled example graph and searches it without an API key
* `cognee-cli remember <data>` ingests data and builds retrieval-ready memory in one step
* `cognee-cli recall "question"` retrieves answers from the graph or session memory
* `cognee-cli improve` enriches an existing dataset
* `cognee-cli datasets` lists datasets, inspects their contents, and reports processing status
* `cognee-cli forget` removes stored data when you no longer need it
* `cognee-cli config` reads and updates saved settings
* `cognee-cli doctor` checks your configuration and local services before you start
* `cognee-cli push` uploads a local dataset's knowledge graph to Cognee Cloud
* `cognee-cli report` writes a Graph Insight Report describing what a dataset's graph contains
* `cognee-cli -ui` launches the local web app
Add `--help` after any command (for example, `cognee-cli recall --help`) to see every option.
<Note>
The CLI still includes lower-level legacy commands such as `add`, `cognify`, `search`, and `delete`, but for new workflows the v1.0 `remember` / `recall` / `improve` / `forget` commands are the preferred interface.
</Note>
## Remember Data
Start by loading something the graph can learn from. You can remember files, folders, URLs, S3 paths, or even plain text.
```bash theme={null}
# Remember a single file into the default dataset
cognee-cli remember docs/company-handbook.pdf
# Pick a dataset name so you can separate topics later
cognee-cli remember docs/policies.docx --dataset-name onboarding
# Remember multiple files at once
cognee-cli remember docs/policies.docx docs/faq.md --dataset-name onboarding
# Remember an entire folder (walks all subdirectories recursively)
cognee-cli remember docs/ --dataset-name onboarding
# Remember a short text note (wrap the note in quotes)
cognee-cli remember "Kickoff call notes: customer wants faster onboarding" --dataset-name sales_calls
```
<Accordion title="Remember Command Options">
* `data`: One or more file paths, directory paths, URLs, S3 paths, or text strings. Mix and match as needed
* `--dataset-name` (`-d`): Defaults to `main_dataset`. Use clear names so the team remembers what each dataset holds
* `--chunk-size`: Token limit for each chunk. Leave blank to let Cognee choose
* `--chunker`: `TextChunker` (default), `CsvChunker`, or `LangchainChunker`
* `--background` (`-b`): Ingests data, then keeps graph-building running in the background
* `--chunks-per-batch`: Number of chunks to process per task batch
* `--dry-run`: Estimate LLM token usage and cost without ingesting data or making LLM calls. Prints a stage-level token/cost summary and exits
* `--presort`: Scan exactly one folder for junk, duplicates, versions, personal data, and proposed dataset groupings instead of ingesting it. Mutually exclusive with `--dry-run` and `--from-report`
* `--from-report`: Apply a saved presort report — ingest its proposed groups as datasets. Cannot be combined with data arguments
* `--allow-root`: Let presort read folders outside the default allowed roots, by extending `COGNEE_ALLOWED_LOCAL_FILE_ROOTS` for this process only
* `--apply`: Scan and ingest in one run — apply the report as soon as it is produced (the report JSON is still saved first)
* `--apply-graph`: Also write the report's relationship graph (files, groups, duplicates, PII tags) into its own dataset
The remaining presort flags tune the two phases: `--report-output` (`-o`), `--use-llm`, `--no-pii`, `--no-subdirectories`, `--no-check-existing`, `--dataset-prefix`, `--spec`, `--apply-group` (repeatable), `--exclude-pii`, `--graph-dataset`, and `--keep-duplicates`. See the "Pre-organize a messy folder" section below.
</Accordion>
<Accordion title="Estimate cost before running (--dry-run)">
Add `--dry-run` to `remember` (or the legacy `cognify` command) to print a stage-level estimate of LLM token usage and rough cost **without ingesting data, making LLM calls, or writing the graph**.
```bash theme={null}
# Estimate the cost of ingesting a note
cognee-cli remember "Kickoff call notes: customer wants faster onboarding" --dry-run
# Estimate the cost of (re-)processing an existing dataset
cognee-cli cognify --datasets onboarding --dry-run
```
Dataset resolution is read-only for a dry run, so a typo'd dataset name fails instead of creating an empty dataset. Only local text, local files, and `file://` URIs are supported for estimation — remote URLs, S3 paths, directories, and binary formats (PDF, images, audio) are rejected because a real run would fetch, walk, or transcribe them.
</Accordion>
<Accordion title="Pre-organize a messy folder (--presort)">
`--presort` scans a folder **without ingesting or modifying it** and prints what it found: junk files, exact-duplicate clusters, version candidates (`report_v2.pdf`), potential personal data, which files Cognee already knows, and a set of proposed dataset groupings. It takes exactly one folder path.
```bash theme={null}
# 1. Scan and save the report
cognee-cli remember ~/Downloads --presort --allow-root -o report.json
# 2. Review report.json, then ingest the proposed groups as datasets
cognee-cli remember --from-report report.json
# Or do both in one run
cognee-cli remember ~/Downloads --presort --apply --allow-root
```
The scan is deterministic — it needs no LLM. Add `--use-llm` for content classification, deeper personal-data detection, and semantic grouping. Without an API key the CLI warns and carries on: the scan still runs, and any apply step stages files with `add()` only, leaving the graph build for a later `cognify`.
Presort reads only from its permitted roots (the working directory, the temp directory, and Cognee's own storage, or `COGNEE_ALLOWED_LOCAL_FILE_ROOTS` when that is set). `--allow-root` adds the folder you named to that list for the duration of the command — see [Presort scan roots](/setup-configuration/security#presort-scan-roots).
Apply-time flags let you narrow what gets ingested: `--apply-group NAME` (repeatable) picks individual groups, `--exclude-pii` drops files with potential personal data, and `--keep-duplicates` ingests every copy instead of one per cluster. `--apply-graph` additionally writes the report's relationship graph into its own dataset (`--graph-dataset` names it).
Folder presort is opt-in. Setting `PRESORT_FOLDERS_ENABLED=true` makes a plain `cognee-cli remember <folder>` scan and apply automatically; with the variable unset, folder ingestion behaves exactly as before.
The [Folder Presort guide](/guides/presort-downloads) walks through the same two phases from Python, including how to review and edit a report before applying it.
</Accordion>
<Accordion title="Ingesting a Folder">
Pass a directory path directly to `remember` (or `add`) and Cognee walks it recursively, picking up every file in the folder and all of its subdirectories. There is no `--recursive` flag and no need to shell-expand with globs or `find`; the ingestion pipeline handles the traversal for you.
```bash theme={null}
# Ingest every file under docs/, at any depth
cognee-cli remember docs/ --dataset-name handbook
# Local paths and S3 prefixes work the same way
cognee-cli remember s3://my-bucket/docs/ --dataset-name handbook
```
To restrict ingestion to specific files, list them explicitly instead of pointing at the parent folder. For a messy folder — duplicates, old versions, junk, files you would rather not ingest — run `--presort` first to see what is in there and split it into datasets before ingesting.
</Accordion>
## Improve Memory
Use `improve` when you want to enrich an existing dataset after ingestion. This is especially useful for session-bridging or an explicit post-processing pass over memory you already stored.
```bash theme={null}
# Improve the default dataset
cognee-cli improve
# Improve a named dataset
cognee-cli improve --dataset-name onboarding
# Improve a dataset and bridge selected session histories
cognee-cli improve --dataset-name onboarding --session-ids chat_1 chat_2
# Kick off a long job and return immediately
cognee-cli improve --dataset-name onboarding --background
```
<Accordion title="Improve Command Options">
* `--dataset-name` (`-d`): Dataset to improve. Defaults to `main_dataset`
* `--dataset-id`: Dataset UUID (alternative to `--dataset-name`)
* `--node-name`: Narrow the improvement pass to specific named entities
* `--session-ids` (`-s`): Session IDs whose Q\&A and feedback should be bridged into the permanent graph
* `--feedback-alpha`: Learning rate for feedback-based weighting updates
* `--background` (`-b`): Handy for large datasets; the CLI exits while the job keeps running
</Accordion>
## Recall Memory
Once `remember` finishes, you can question the graph. Start with a simple natural-language question, then experiment with search types. The CLI exposes a **subset** of the available retrieval types; see [Recall](/core-concepts/main-operations/recall) for the memory-oriented workflow and [Search](/core-concepts/main-operations/legacy-operations/search) for the lower-level search type reference.
```bash theme={null}
# Default recall (HYBRID_COMPLETION)
cognee-cli recall "Who owns the rollout plan?"
# Limit the scope to one dataset
cognee-cli recall "What is the onboarding timeline?" --datasets onboarding
# Return three answers at most
cognee-cli recall "List the key risks" --top-k 3
# Save a JSON response for another tool
cognee-cli recall "Which documents mention security?" --output-format json
```
<Accordion title="Recall Types">
Try these quick examples to feel the differences:
```bash theme={null}
# Conversational answer with reasoning (default)
cognee-cli recall "Give me a summary of onboarding" --query-type GRAPH_COMPLETION
# Shorter answer based on chunks
cognee-cli recall "Show the onboarding steps" --query-type RAG_COMPLETION
# Raw text passages you can copy
cognee-cli recall "Find security requirements" --query-type CHUNKS --top-k 5
# Keyword (BM25) passages — no LLM and no embedding provider needed
cognee-cli recall "Find security requirements" --query-type CHUNKS_LEXICAL --top-k 5
# Summaries only (great for reviews)
cognee-cli recall "Summarise the onboarding handbooks" --query-type SUMMARIES
# Advanced graph query (requires Cypher skills)
cognee-cli recall "MATCH (n) RETURN COUNT(n)" --query-type CYPHER
```
`CHUNKS_LEXICAL` runs a keyword (BM25) search over stored chunks, so it needs
neither an LLM nor an embedding provider. That is what makes it usable on a
machine with no API key, and why [`cognee-cli demo`](#try-the-demo-graph) uses it.
<Note>
The CLI supports a **subset** of search types: `GRAPH_COMPLETION`, `HYBRID_COMPLETION`, `RAG_COMPLETION`, `CHUNKS`, `CHUNKS_LEXICAL`, `SUMMARIES`, `CODE`, `CYPHER`, `GRAPH_REPORT`, and `SKILLS`. Other search types (like `GRAPH_SUMMARY_COMPLETION`, `CODING_RULES`, and `TEMPORAL`) are available in the Python API.
`GRAPH_REPORT` ignores the question you pass and returns a whole-graph
[Graph Insight Report](#generate-a-graph-insight-report); use the dedicated
`cognee-cli report` command if you also want the Markdown written to a file.
</Note>
</Accordion>
<Accordion title="Recall Command Options">
* `--query-type`: Subset of search types (e.g. HYBRID\_COMPLETION, GRAPH\_COMPLETION, RAG\_COMPLETION, CHUNKS, CHUNKS\_LEXICAL, SUMMARIES, CYPHER). See [Search](/core-concepts/main-operations/legacy-operations/search) for the full list.
* `--datasets`: Limit search to specific datasets
* `--top-k`: Maximum number of results to return
* `--system-prompt`: Point to a custom prompt file for LLM-backed modes
* `--session-id` (`-s`): Search session memory directly when used by itself, or add session history to graph-backed recall
* `--output-format` (`-f`): `pretty` (friendly layout), `simple` (minimal text), or `json` (structured output for scripts)
</Accordion>
## Generate a Graph Insight Report
Not sure what actually ended up in your graph? `cognee-cli report` writes a Markdown
**Graph Insight Report** covering the graph's hub nodes, connections that cross
node-set boundaries, an edge-provenance breakdown, and a few LLM-suggested questions
to try with `recall`.
```bash theme={null}
# Report on the default dataset, writing ./graph_report.md
cognee-cli report
# Report on a named dataset
cognee-cli report --datasets onboarding
# Surface more hubs and connections, and choose the output file
cognee-cli report -d onboarding -n 25 -o reports/onboarding-graph.md
```
The command prints a confirmation, the output path, and the first 500 characters of
the report as a preview.
<Accordion title="Report Command Options">
* `--datasets` (`-d`): Dataset name(s) to analyse (default: `main_dataset`). Only the **first** dataset you have access to is analysed — run the command once per dataset
* `--output` (`-o`): Output file path for the Markdown report (default: `graph_report.md`)
* `--top-n` (`-n`): Number of hub nodes and surprising connections to surface (default: `10`)
</Accordion>
<Note>
The report is read-only — it computes everything from the existing graph and changes
nothing. It makes a single LLM call for the suggested-questions section and falls back
to a generic question if that call fails. `report` is not supported in `--api-url`
mode — run it without `--api-url` and it executes in-process against your local
databases. See the
[`cognee.report()` SDK reference](/python-api/report) for the programmatic equivalent
and a breakdown of each report section.
</Note>
## Inspect Datasets
Forgot what you already stored? `cognee-cli datasets` answers "which datasets do I have?" and "did processing finish?".
```bash theme={null}
# List every dataset you can access (ID, name, created date)
cognee-cli datasets list
# Create an empty dataset up front
cognee-cli datasets create onboarding
# List the data items in a dataset
cognee-cli datasets data 123e4567-e89b-12d3-a456-426614174000
# Check processing status for one or more datasets
cognee-cli datasets status 123e4567-e89b-12d3-a456-426614174000
# Check several pipelines at once
cognee-cli datasets status <dataset-uuid> --pipelines cognify_pipeline memify_pipeline
# Export a dataset's knowledge graph as JSON
cognee-cli datasets graph <dataset-uuid> -o graph.json
# Delete a dataset and everything in it
cognee-cli datasets delete <dataset-uuid>
```
`status` prints one line per dataset, for example `123e4567-…: PipelineRunStatus.DATASET_PROCESSING_COMPLETED` (in [`--api-url` mode](#talk-to-a-running-cognee-api) the status arrives as a plain string, without the `PipelineRunStatus.` prefix). Datasets with no recorded pipeline run are omitted (or shown as `<no pipeline runs found>` when you pass several `--pipelines`).
<Accordion title="Datasets Subcommands">
* `list`: List all datasets you have `read` access to, with ID, name, and creation date
* `create <name>`: Create an empty dataset and grant yourself `read`, `write`, `share`, and `delete` on it. Re-running with an existing name prints that dataset's ID instead of creating a duplicate
* `data <dataset_id>`: List the data items in a dataset (ID, name, MIME type, creation date)
* `status <dataset_ids...>`: Show pipeline status for one or more datasets. `--pipelines` selects which pipelines to check (default: `cognify_pipeline`)
* `graph <dataset_id>`: Export the dataset's knowledge graph as JSON. `-o`/`--output` writes to a file instead of stdout
* `delete <dataset_id>`: Delete the dataset and all of its data. `-f`/`--force` skips the confirmation prompt
</Accordion>
<Note>
Every subcommand except `create` takes a dataset **UUID**, not a dataset name — run `cognee-cli datasets list` first to look the ID up. Use the global `--user-id` flag to act as a specific user (`status` checks pipeline runs directly and ignores it). For the programmatic equivalent, see [`cognee.datasets`](/python-api/datasets); for the browser view, see the [Datasets page](/cognee-cloud/ui/datasets).
</Note>
## Forget Data
Clean up when a dataset is outdated or when you reset the environment.
```bash theme={null}
# Remove one dataset
cognee-cli forget --dataset onboarding
# Remove a single item from a dataset
cognee-cli forget --dataset onboarding --data-id 123e4567-e89b-12d3-a456-426614174000
# Wipe everything for the current user (--all is an alias for --everything)
cognee-cli forget --everything
# Clear graph and vector memory but keep the raw files and data records,
# so the dataset can be re-cognified with different settings
cognee-cli forget --dataset onboarding --memory-only
```
<Accordion title="Forget Command Options">
* `--dataset`: Dataset name or UUID to remove
* `--dataset-id`: Dataset UUID to remove
* `--data-id`: Remove a single item from the specified dataset (requires `--dataset` or `--dataset-id`)
* `--everything` (alias `--all`): Remove all datasets and data for the current user
* `--memory-only`: Delete only graph and vector memory, keeping raw files and data records so the dataset (or item) can be re-cognified later. Requires `--dataset` or `--dataset-id`; combining it with `--everything` is refused with an error, since `--everything` deletes all datasets and data outright. This is the CLI equivalent of [`forget(..., memory_only=True)`](/python-api/forget)
</Accordion>
<Note>
`forget` is the v1.0 deletion interface. If you still need the older `delete` flow, it remains available as a lower-level legacy command.
</Note>
## Manage Configuration
The CLI stores its settings so you do not have to repeat them. Configuration updates line up with the Python API.
```bash theme={null}
# See the list of supported keys
cognee-cli config list
# Check one value
cognee-cli config get llm_model
# Show every known setting (secrets masked)
cognee-cli config get
# Update your LLM provider and model
cognee-cli config set llm_provider openai
cognee-cli config set llm_model gpt-4o-mini
# Store an API key (quotes are optional)
cognee-cli config set llm_api_key sk-yourkey
# Reset a key back to its default value
cognee-cli config unset chunk_size
```
### Settings Persist to `.env`
`config set` and `config unset` write the resolved value into a `.env` file in the
**current working directory**, creating the file if it does not exist yet. This is the
same file Cognee reads at startup, so a value you set is picked up by the next CLI
invocation or script started from that directory:
```bash theme={null}
$ cognee-cli config set llm_model gpt-4o-mini
Success: Set llm_model = gpt-4o-mini
Note: Created new .env file at /home/you/project/.env
Note: Persisted LLM_MODEL to /home/you/project/.env
```
<Warning>
Because the `.env` file lands in whatever directory you happen to run the command from,
`cognee-cli config set llm_api_key ...` can drop a plaintext API key into a source tree.
Add `.env` to your `.gitignore`, restrict its permissions (`chmod 600 .env`), and run
`config set` from the directory you actually want the settings to apply to.
</Warning>
### Secrets Are Masked by Default
`config get` masks secret values — `llm_api_key`, `embedding_api_key`, and
`vector_db_key` — showing only the first three and last four characters (values of
eight characters or fewer are replaced entirely with `*`). Pass `--show-secrets` to
print them in plaintext:
```bash theme={null}
$ cognee-cli config get llm_api_key
llm_api_key: sk-...c3d4
$ cognee-cli config get llm_api_key --show-secrets
llm_api_key: sk-proj-9f2a7b41c3d4
```
<Accordion title="Config Command Options">
* `list`: Print the common keys
* `get [key]`: Show the saved value; omit the key to list every known setting. Secrets are masked unless you add `--show-secrets`
* `set <key> <value>`: Save a new value and persist it to `.env` in the current directory. JSON strings such as `{}` or `true` are parsed automatically
* `unset <key>`: Reset to the default and persist that default to `.env`. Add `--force` to skip confirmation
* `reset`: Placeholder for a future "reset everything" command
</Accordion>
<Accordion title="Useful Configuration Keys">
* Language model: `llm_provider`, `llm_model`, `llm_api_key`, `llm_endpoint`
* Storage: `graph_database_provider`, `vector_db_provider`, `vector_db_url`, `vector_db_key`
* Chunking: `chunk_size`, `chunk_overlap`
</Accordion>
## Diagnose Your Setup
`cognee-cli doctor` runs preflight diagnostics against your current configuration and
prints a pass/fail report. Use it right after writing a fresh `.env`, or when a
`remember` run fails with a provider error and you want to know whether the problem is
your configuration, a local database, or the provider itself.
```bash theme={null}
# Configuration and local service checks (no network calls)
cognee-cli doctor
# Also make live LLM and embedding round-trips
cognee-cli doctor --probe
```
The report has up to three sections:
1. **Configuration** — prints the resolved LLM and embedding provider, model, and whether
each API key is set, then runs the same zero-network consistency check that `add()` and
`remember()` run. It catches the only-LLM-or-only-embeddings trap, where the side you
did not configure silently defaults to OpenAI and breaks minutes into the first
ingestion. See [LLM/Embedding Configuration](/setup-configuration/overview#configuration-workflow)
for the exact conditions and fixes.
2. **Services** — relational, vector, and graph databases plus file storage, using the same
component checks as the `/health` endpoint. A missing local database is not necessarily a
problem: it is created on first use, so run any `cognee-cli remember` command (or check
your `DB_*` settings if you configured Postgres).
3. **Providers (live probes)** — only with `--probe`. Performs a real LLM round-trip and a
real embedding round-trip (reporting the vector dimensions it got back), so it makes
network calls and may incur token costs.
A clean run ends with `All checks passed. cognee is ready to use.`. Any failed check makes
the command exit non-zero, so you can gate CI jobs and setup scripts on it:
```bash theme={null}
cognee-cli doctor || exit 1
```
<Accordion title="Doctor Command Options">
* `--probe`: Also run live LLM and embedding connectivity probes (network calls; may cost tokens). Without it, `doctor` makes no provider calls at all
</Accordion>
<Note>
`doctor` is not among the commands forwarded in [`--api-url` mode](#talk-to-a-running-cognee-api) —
running it with `--api-url` set fails with an error. Run it without the flag to diagnose
your local configuration and databases.
</Note>
## Manage Agents
The `agents` command creates and manages agents and their connections. Each agent
is backed by its own agent user with a one-time API key, and every action is
scoped to the acting user (see the global `--user-id` flag below).
```bash theme={null}
# Create an agent and grant it read/write on one or more datasets
cognee-cli agents create my-agent --datasets onboarding sales_calls
# List the agents you own
cognee-cli agents list
# Show details for a single agent
cognee-cli agents get <agent-uuid>
# Delete an agent (use -f to skip the confirmation)
cognee-cli agents delete <agent-uuid>
# Register and unregister an agent connection (session)
cognee-cli agents register session-1 --dataset-names onboarding
cognee-cli agents unregister session-1
# List active agent connections and their memory sources
cognee-cli agents connections --range 7d --status active
```
<Warning>
`agents create` prints the agent's API key once. It is a one-time secret and
cannot be retrieved again — store it immediately.
</Warning>
<Accordion title="Agents Subcommands">
* `create <name>`: Create a new agent. `--datasets` accepts dataset names or UUIDs to grant the agent read/write access to
* `list`: List all agents you own
* `get <agent_id>`: Show an agent's id, email, and API key label
* `delete <agent_id>`: Delete an agent. `-f`/`--force` skips the confirmation prompt
* `register <agent_session_name>`: Register an agent connection. Options: `--type` (default `api`), `--memory-mode` (default `unknown`), `--session-id`, `--dataset-ids`, `--dataset-names`
* `unregister <agent_session_name>`: Unregister an agent connection and report the remaining active count
* `connections`: List active connections. Options: `--agent-id`, `--range` (default `30d`), `--status`, `--limit` (default `50`), `--offset` (default `0`)
</Accordion>
<Note>
The `agents` command resolves `--user-id` **strictly**: a valid-but-unknown UUID
is a hard error rather than a silent fallback to the default user, because falling
back would break the isolation the flag promises. Create the user first, or omit
`--user-id` to act as the default user.
</Note>
For the equivalent Python SDK, see [`cognee.agents`](/python-api/agents).
## Launch the UI
Prefer a browser view? Launch the UI with one flag.
```bash theme={null}
cognee-cli -ui
```
The CLI starts the backend on `http://localhost:8000` and the React app on `http://localhost:3000`. Leave the window open and press `Ctrl+C` to stop everything.
It also tries to launch the Cognee MCP server in Docker. If Docker is not reachable, the CLI skips MCP startup and leaves the UI and backend running.
The launcher runs the UI in **local mode** (it defaults `NEXT_PUBLIC_IS_CLOUD_ENVIRONMENT=false`). In this mode an LLM API key is optional at startup: if none is configured, you can enter one from the dashboard, and Cognee saves it to the running backend for the rest of the session — the equivalent of setting `LLM_API_KEY`. Because the key lives only in the running process, it is not persisted after you stop the UI; add it to your `.env` if you want it to survive a restart.
<Accordion title="MCP Docker Networking">
`cognee-cli -ui` supports Docker Desktop, Colima, or any OCI-compatible runtime with a working `docker` CLI. Before pulling the MCP image, it runs a `docker info` preflight check and logs setup guidance if the daemon is unavailable.
When the backend starts with the UI, the MCP container receives `API_URL=http://localhost:<backend-port>` (default backend port `8000`). The launch command does **not** add an explicit `--add-host host.docker.internal:host-gateway` mapping; instead, the MCP image rewrites `localhost` by trying Docker Desktop, Colima/Lima, and the container gateway fallback automatically.
If the container logs say host-address auto-detection failed, use the manual networking workarounds in the [MCP API mode notes](/cognee-mcp/mcp-quickstart#api-mode-shared-knowledge-graph). See [Docker & Colima Setup](https://github.com/topoteretes/cognee/blob/dev/docs/docker-colima-setup.md) for Docker setup and troubleshooting.
</Accordion>
## Talk to a Running Cognee API
Add `--api-url` to delegate any supported command to a running Cognee API server instead of executing it in-process. This is the recommended mode for multi-agent or concurrent usage with file-based databases (SQLite, Ladybug, LanceDB), because it lets a single server own all database connections.
```bash theme={null}
# Use a locally running API server
cognee-cli --api-url http://localhost:8000 remember docs/handbook.pdf
# Ask the same server for an answer
cognee-cli --api-url http://localhost:8000 recall "Who owns onboarding?"
# Improve a dataset on the remote server in the background
cognee-cli --api-url http://localhost:8000 improve --dataset-name onboarding --background
```
Commands supported in `--api-url` mode: `add`, `cognify`, `search`, `memify`, `datasets`, `delete`, `remember`, `recall`, `improve`, and `forget`. Pass `--api-url` only with commands from this list — any other command (`demo`, `doctor`, `report`, `config`, and the rest) fails with an error rather than falling back to a local run. Drop the flag to execute those against your local databases.
`--api-url` works against Cognee Cloud tenants as well as self-hosted servers: the CLI follows HTTP redirects and normalizes dataset endpoints so both cloud (slash-canonical) and local OSS installs resolve. The CLI runs your command directly against the endpoint rather than pinging `/health` first, so a reachable server is never mis-reported as offline.
<Accordion title="Connection and HTTP errors">
If the CLI can't reach the server (wrong URL, server down, DNS or timeout failure), it reports the attempted URL so you can spot a typo quickly:
```
Could not reach the Cognee API at https://your-api.example.com: <transport error>
Check the --api-url value and that the server is reachable
(local server: uvicorn cognee.api.client:app --port 8000).
```
If the server responds with an HTTP error — for example `401`/`403` (bad or missing credentials) or `404` (wrong path) — the CLI shows the status code and the server's actual response instead of masking it as a generic connection failure. Use this detail to distinguish an unreachable server from an authentication or routing problem.
</Accordion>
### Authenticate Against the API
If the target API requires authentication, supply credentials with one of the flags below (or their environment-variable fallbacks). The CLI sends the credentials only when `--api-url` is set.
```bash theme={null}
# Cognee Cloud or any backend that uses an API key
cognee-cli --api-url https://your-api.example.com \
--api-key sk-yourkey \
recall "What's in the handbook?"
# Self-hosted backend with a bearer token issued by /api/v1/auth/login
cognee-cli --api-url http://localhost:8000 \
--api-token your_bearer_token \
remember docs/handbook.pdf
```
<Accordion title="API Mode Options">
* `--api-url`: URL of the Cognee API server (for example `http://localhost:8000`). When set, supported commands are forwarded over HTTP
* `--api-key`: API key sent as the `X-Api-Key` header. Falls back to the `COGNEE_API_KEY` environment variable
* `--api-token`: Bearer token sent as `Authorization: Bearer <token>`. Falls back to the `COGNEE_API_TOKEN` environment variable. Ignored when `--api-key` is also provided
* `--user-id`: Optional UUID forwarded as the `X-User-Id` header for multi-agent isolation. The server must be configured to honour this header
</Accordion>
<Note>
In `--api-url` mode the server controls chunking and feedback weighting, so `--chunker` on `remember` and `--feedback-alpha` on `improve` are ignored.
</Note>
## Push to Cloud
Already built a knowledge graph locally and want it on Cognee Cloud? `cognee push`
exports the dataset's graph as a [COGX archive](/core-concepts/further-concepts/cogx)
and imports it on the remote instance, **preserving the entities and relationships
you extracted locally** instead of re-deriving them from the raw files.
```bash theme={null}
# Log in once (saves credentials for reuse)
cognee serve
# Push the default dataset (main_dataset)
cognee push
# Push a named dataset
cognee push my_dataset
# Push into a different dataset name on the remote instance
cognee push my_dataset --target-dataset prod_dataset
# Preserve the graph AND re-cognify the raw content remotely
cognee push my_dataset --mode hybrid
# Large graph: schedule the remote import and return after the upload
cognee push my_dataset --background
# Push to an explicit instance without a prior serve login
cognee push --url https://my.cognee.ai --api-key ck_...
```
<Accordion title="Push Command Options">
* `dataset`: Local dataset name to push (default: `main_dataset`)
* `--target-dataset`: Dataset name on the remote instance (default: same as local)
* `--mode`: Remote import mode (default: `preserve`)
* `preserve` — map exported entities/facts directly, zero LLM calls
* `hybrid` — preserve the graph and also cognify the raw content
* `re-derive` — ignore the exported graph and rebuild from raw content remotely
* `--url`: Remote instance URL. Falls back to the active `serve` connection, `COGNEE_SERVICE_URL`, or saved serve credentials
* `--api-key`: API key for the remote instance. Falls back to `COGNEE_API_KEY`
* `--background`, `-b`: Schedule the remote import in the background and return after the upload (recommended for large graphs); prints the pipeline run id
</Accordion>
<Note>
The dataset must already have a knowledge graph — run `cognee remember` (or `cognee cognify`) first.
Authentication reuses your `cognee serve` login; alternatively pass `--url`/`--api-key`
or set `COGNEE_SERVICE_URL` and `COGNEE_API_KEY`. This is the graph-preserving counterpart
to [syncing](/cognee-cloud/connections/syncing-local-instance), which instead ships raw data
for the remote instance to rebuild. See the [`cognee.push()` SDK reference](/python-api/push)
for the programmatic equivalent.
</Note>
## Next Steps
<CardGroup>
<Card title="Installation Guide" href="/getting-started/installation" icon="download">
**Set up your environment**
Install Cognee and configure your environment to start using the CLI.
</Card>
<Card title="Quickstart Tutorial" href="/getting-started/quickstart" icon="play">
**Run your first example**
Get started with Cognee by running your first knowledge graph example.
</Card>
</CardGroup>
# Claude Code with Cognee Cloud
Source: https://docs.cognee.ai/cognee-cloud/agent-integrations/claude-code
Connect Claude Code to your Cognee Cloud tenant with the memory plugin.
Add persistent memory to [Claude Code](https://www.anthropic.com/claude-code) backed by your Cognee Cloud tenant with the **Cognee memory plugin** — no code and no `pip install`. The plugin hooks into Claude Code's lifecycle: it captures your prompts, tool traces, and answers into session memory, injects relevant context on every prompt, and syncs the session into your knowledge graph on session end.
Sessions are disposable; your memory isn't.
## 1. Install the plugin
Install from the Claude Code marketplace **before** launching Claude Code, so the first `claude` launch is a clean session that runs the plugin bootstrap automatically:
```bash theme={null}
claude plugin marketplace add topoteretes/cognee-integrations
claude plugin install cognee-memory@cognee
```
<Info>
These CLI subcommands use the same plugin manager as the in-chat `/plugin` commands. If you instead install from inside the chat with `/plugin`, **restart Claude Code** (start a new session) before memory connects — `/reload-plugins` loads the skills but does not run `SessionStart`.
</Info>
<Note>
Install scope defaults to `user` (global). Pass `--scope project` or `--scope local` to confine the plugin to a single repo.
</Note>
On the first clean launch you'll see a **"Cognee Memory Connected"** message, and the status line shows `cognee: <dataset> · <mode>`.
## 2. Point it at Cognee Cloud
Write both variables **once** into `~/.cognee/.env`, using your tenant base URL and an [API key](/cognee-cloud/ui/api-keys). The file is created with a commented template on the first session start, is shared with the Codex plugin, and its values act like shell exports — except you only set them once, and they survive closing the terminal:
<Tabs>
<Tab title="macOS / Linux (bash, zsh)">
```bash theme={null}
mkdir -p ~/.cognee
cat >> ~/.cognee/.env <<'EOF'
COGNEE_BASE_URL="https://your-tenant.aws.cognee.ai"
COGNEE_API_KEY="<your-api-key>"
EOF
chmod 600 ~/.cognee/.env
```
</Tab>
<Tab title="Windows (PowerShell)">
```powershell theme={null}
New-Item -ItemType Directory -Force "$env:USERPROFILE\.cognee" | Out-Null
@'
COGNEE_BASE_URL="https://your-tenant.aws.cognee.ai"
COGNEE_API_KEY="<your-api-key>"
'@ | Add-Content "$env:USERPROFILE\.cognee\.env"
```
</Tab>
<Tab title="Windows (cmd.exe)">
```cmd theme={null}
if not exist "%USERPROFILE%\.cognee" mkdir "%USERPROFILE%\.cognee"
(echo COGNEE_BASE_URL=https://your-tenant.aws.cognee.ai)>>"%USERPROFILE%\.cognee\.env"
(echo COGNEE_API_KEY=your-api-key)>>"%USERPROFILE%\.cognee\.env"
```
</Tab>
</Tabs>
Re-running the block is safe — when a key appears more than once the last value wins, so pasting again updates the credentials instead of stacking duplicates. Changes apply on the next `claude` launch.
<Info>
When `COGNEE_BASE_URL` is set, the plugin runs as a pure thin HTTP client to your tenant — it does **not** install a local Cognee runtime. When `COGNEE_BASE_URL` is unset, the plugin instead bootstraps a local API at `http://localhost:8011`. Setting the Cloud URL and key is what routes memory to your tenant.
</Info>
Keeping a local-mode `LLM_API_KEY` in the same file is fine: cloud still wins, because a configured URL is what selects cloud. To send one terminal to local instead, `export COGNEE_BACKEND=local` before launching — `unset COGNEE_BASE_URL` does **not** work, because the env file re-injects the URL at the next launch. See [Which mode wins](/integrations/claude-code-integration#which-mode-wins-and-how-to-switch).
## 3. Choose a dataset
All writes and recall are scoped to a **single dataset**, selected with the `COGNEE_PLUGIN_DATASET` environment variable. By default both the Claude Code and Codex plugins use `agent_sessions`, so memory is shared across both integrations automatically.
Set a custom dataset at launch:
```bash theme={null}
export COGNEE_PLUGIN_DATASET="my-project-memory"
```
<Warning>
Recall searches only the active dataset. `COGNEE_PLUGIN_DATASET` seeds it at launch and is read only then, so changing the variable mid-session does nothing. To move the running session to another dataset, use `/cognee-memory:cognee-switch-datasets` — it syncs the current session into its dataset first, then registers a fresh session on the chosen one, and its choice beats `COGNEE_PLUGIN_DATASET` for the rest of the launch.
</Warning>
Data added to the same dataset outside Claude Code (via the SDK or the server) is visible in Claude Code through the plugin.
## 4. Pick a session (optional)
By default the `session_id` is derived from the Claude Code session, so a new conversation starts a new one and `claude --resume` continues the same one. Set `COGNEE_SESSION_ID` to pin a specific named session, or to deliberately share one live session across two terminals:
```bash theme={null}
export COGNEE_SESSION_ID="my-project"
```
## 5. Verify
Restart Claude Code so `SessionStart` runs with the new credentials, then open a fresh session and ask:
> What do you know from cognee?
Answering from a clean session confirms it's recalling from your Cloud memory. You can also invoke the skills explicitly:
| Skill | Purpose |
| --------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| `/cognee-memory:cognee-remember` | Store something in memory now |
| `/cognee-memory:cognee-search` | Query your memory |
| `/cognee-memory:cognee-sync` | Persist the current session into the graph |
| `/cognee-memory:cognee-forget` | Delete memory you ask to forget, along with its derived graph knowledge. Irreversible |
| `/cognee-memory:cognee-code` | Index a repository into the code graph and query it. Against a tenant this is **explicit only** — see below |
| `/cognee-memory:cognee-switch-datasets` | Move the running session to another dataset |
Two things about the code graph differ against a Cloud tenant. Repositories are **not** indexed automatically — the plugin only auto-indexes against a local server, so a private checkout is never shipped to a hosted tenant on its own initiative. Ask for it with `/cognee-memory:cognee-code` and give it a **git URL**: your tenant clones the repo rather than reading your disk, which is also why the graph reflects your last pushed commit rather than your working tree. A local path is not an option here — the server cannot see it. See [Code graph](/integrations/claude-code-integration#code-graph).
<Note>
Memory is captured and synced from your sessions over time, and writes build the graph in the background — so a brand-new setup may recall nothing until at least one session has synced. An empty first recall is expected.
</Note>
Share specific datasets with connected agents from the [Connections](/cognee-cloud/connections/managing-connections) page to grant scoped read access.
## Configuration reference
Precedence: environment variables → `~/.cognee/.env` → defaults. There is no `config.json`; older versions wrote one, and `SessionStart` now deletes a leftover file. The `COGNEE_BACKEND` / `COGNEE_CLAUDE_BACKEND` [mode switch](/integrations/claude-code-integration#which-mode-wins-and-how-to-switch) follows the same precedence. Wherever it is set, it pins that terminal's mode regardless of where the connection variables are defined.
| Env var | Default | Notes |
| ------------------------- | ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- |
| `COGNEE_BASE_URL` | unset | Your tenant URL; enables Cloud/managed-endpoint mode |
| `COGNEE_API_KEY` | unset | Cloud API key from the [API Keys](/cognee-cloud/ui/api-keys) page; auto-minted only in local mode |
| `COGNEE_BACKEND` | unset | `local` or `cloud` — pins this terminal's mode; `COGNEE_CLAUDE_BACKEND` does the same for this plugin only and beats it |
| `COGNEE_ENV_FILE` | `~/.cognee/.env` | Read the one-time setup file from somewhere else |
| `COGNEE_PLUGIN_DATASET` | `agent_sessions` | Dataset for writes and recall at launch; `/cognee-memory:cognee-switch-datasets` changes it mid-session |
| `COGNEE_SESSION_ID` | derived from the Claude Code session | Override to pin a named session, or share one across terminals |
| `COGNEE_SESSION_PREFIX` | `claude` | Prefix on auto-derived session IDs (`<prefix>_<claude-session-id>`) |
| `COGNEE_SESSION_STRATEGY` | `per-directory` | Currently inert — the plugin derives session IDs from the Claude Code session and never reads this |
| `COGNEE_PREFER_MEMORY` | `true` | Inject the `SessionStart` steer treating Cognee as authoritative memory |
| `COGNEE_STATUSLINE` | `true` | Auto-configure the `cognee: <dataset> · <mode>` status line |
See the [full integration guide](/integrations/claude-code-integration#session-distillation-self-improvement) for the session-distillation and idle-watcher variables; the advanced session-sync and update-notification knobs are covered in the plugin README.
***
<CardGroup>
<Card title="Full integration guide" icon="book" href="/integrations/claude-code-integration">
Hooks reference, session sync, and debugging
</Card>
<Card title="Codex" icon="terminal" href="/cognee-cloud/agent-integrations/codex">
The same memory plugin for the Codex CLI
</Card>
</CardGroup>
# Claude Desktop with Cognee Cloud
Source: https://docs.cognee.ai/cognee-cloud/agent-integrations/claude-desktop
Connect Claude Desktop to your Cognee Cloud tenant via MCP.
[Claude Desktop](https://claude.ai/download) is Anthropic's native app for macOS and Windows. It supports [MCP](https://modelcontextprotocol.io) servers through a config file you edit manually — register Cognee and Claude Desktop can `remember`, `recall`, and `forget` against your Cognee Cloud tenant.
## 1. Open the config file
Find `claude_desktop_config.json` for your platform (create it and any parent directories if it doesn't exist):
| Platform | Path |
| -------- | ----------------------------------------------------------------- |
| macOS | `~/Library/Application Support/Claude/claude_desktop_config.json` |
| Windows | `%APPDATA%\Claude\claude_desktop_config.json` |
## 2. Add the Cognee server
Point the entry at your tenant using your API Base URL and a key from the [API Keys](/cognee-cloud/ui/api-keys) page:
```json theme={null}
{
"mcpServers": {
"cognee": {
"command": "uvx",
"args": [
"cognee-mcp@latest",
"--api-url", "https://your-tenant.aws.cognee.ai",
"--api-token", "<your-api-key>"
]
}
}
}
```
<Info>
This is the stdio form the in-product [Integrations](/cognee-cloud/ui/integrations) page generates. Cognee runs via `uvx` (requires [uv](https://docs.astral.sh/uv/)) — no separate install of Cognee itself. Make sure `--api-url` has **no trailing slash** and the token has no surrounding quotes or whitespace.
</Info>
<Warning>
`uvx` requires **[uv](https://docs.astral.sh/uv/)** to be installed first — it is not bundled with Claude Desktop or Windows. If tools don't appear after setup, `uvx` is likely missing from `PATH`.
* **macOS/Linux:** `curl -LsSf https://astral.sh/uv/install.sh | sh`
* **Windows (PowerShell):** `powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"`
After installing, restart your terminal (and fully quit/reopen Claude Desktop) so the updated `PATH` takes effect.
</Warning>
## 3. Restart Claude Desktop
Fully quit and reopen the app. On macOS use **Cmd + Q** (not just closing the window) so it reloads the config.
## 4. Verify
Open a new conversation and ask:
> What Cognee tools do you have available?
Claude should list `remember`, `recall`, and `forget`. You can then prompt explicitly:
* "Remember this in Cognee: we use PostgreSQL for the main database"
* "Recall what you know about our database choices from Cognee"
<Tip>
If no Cognee tools appear, Claude usually can't find `uvx` on its `PATH` — run `which uvx` and use that absolute path as the `command` value in the config.
</Tip>
***
<CardGroup>
<Card title="Full MCP guide" icon="book" href="/cognee-mcp/integrations/claude-desktop">
Standalone, source, and remote-URL setups, plus troubleshooting
</Card>
<Card title="Cloud MCP" icon="plug" href="/cognee-cloud/connections/cloud-mcp">
How MCP clients connect to Cognee Cloud
</Card>
</CardGroup>
# Codex with Cognee Cloud
Source: https://docs.cognee.ai/cognee-cloud/agent-integrations/codex
Connect Codex to your Cognee Cloud tenant with the memory plugin.
Add persistent memory to the Codex CLI backed by your Cognee Cloud tenant with the **Cognee memory plugin** — no code and no `pip install`. The plugin captures your prompts, tool traces, and answers into session memory, injects relevant context on every prompt, and syncs the session into your knowledge graph on session end.
Sessions are disposable; your memory isn't.
## 1. Enable hooks and install
The plugin depends on Codex lifecycle hooks. Enable them first, then install from the Codex marketplace:
```bash theme={null}
codex features enable hooks
codex plugin marketplace add topoteretes/cognee-integrations --ref main
codex plugin add cognee@cognee
```
<Info>
You can enable hooks manually instead by adding a `[features]` section with `hooks = true` to `~/.codex/config.toml`. If Codex prompts you to review the hooks, approve the Cognee hooks so it can call the plugin on prompt submit, tool use, stop, compaction, and session end.
</Info>
On startup the status line shows `cognee: <dataset> · <mode>` to confirm the plugin is active.
## 2. Point it at Cognee Cloud
Write both variables **once** into `~/.cognee/.env`, using your tenant base URL and an [API key](/cognee-cloud/ui/api-keys). The file is created with a commented template on the first session start, is shared with the Claude Code plugin, and its values act like shell exports — except you only set them once, and they survive closing the terminal:
<Tabs>
<Tab title="macOS / Linux (bash, zsh)">
```bash theme={null}
mkdir -p ~/.cognee
cat >> ~/.cognee/.env <<'EOF'
COGNEE_BASE_URL="https://your-tenant.aws.cognee.ai"
COGNEE_API_KEY="<your-api-key>"
EOF
chmod 600 ~/.cognee/.env
```
</Tab>
<Tab title="Windows (PowerShell)">
```powershell theme={null}
New-Item -ItemType Directory -Force "$env:USERPROFILE\.cognee" | Out-Null
@'
COGNEE_BASE_URL="https://your-tenant.aws.cognee.ai"
COGNEE_API_KEY="<your-api-key>"
'@ | Add-Content "$env:USERPROFILE\.cognee\.env"
```
</Tab>
<Tab title="Windows (cmd.exe)">
```cmd theme={null}
if not exist "%USERPROFILE%\.cognee" mkdir "%USERPROFILE%\.cognee"
(echo COGNEE_BASE_URL=https://your-tenant.aws.cognee.ai)>>"%USERPROFILE%\.cognee\.env"
(echo COGNEE_API_KEY=your-api-key)>>"%USERPROFILE%\.cognee\.env"
```
</Tab>
</Tabs>
Re-running the block is safe — when a key appears more than once the last value wins, so pasting again updates the credentials instead of stacking duplicates. Changes apply on the next `codex` launch.
<Info>
When `COGNEE_BASE_URL` is set, the plugin runs as a pure thin HTTP client to your tenant — it does **not** install a local Cognee runtime. When `COGNEE_BASE_URL` is unset, the plugin instead bootstraps a local API at `http://localhost:8011`. Setting the Cloud URL and key is what routes memory to your tenant.
</Info>
Keeping a local-mode `LLM_API_KEY` in the same file is fine: cloud still wins, because a configured URL is what selects cloud. To send one terminal to local instead, `export COGNEE_BACKEND=local` before launching — `unset COGNEE_BASE_URL` does **not** work, because the env file re-injects the URL at the next launch. See [Which mode wins](/integrations/codex-integration#which-mode-wins-and-how-to-switch).
## 3. Choose a dataset
All writes and recall are scoped to a **single dataset**, selected with the `COGNEE_PLUGIN_DATASET` environment variable. By default both the Codex and Claude Code plugins use `agent_sessions`, so memory is shared across both integrations automatically.
Set a custom dataset at launch:
```bash theme={null}
export COGNEE_PLUGIN_DATASET="my-project-memory"
codex
```
<Warning>
Recall searches only the active dataset. `COGNEE_PLUGIN_DATASET` seeds it at launch and is read only then, so changing the variable mid-session does nothing. To move the running session to another dataset, ask Codex to switch datasets (the `cognee-switch-datasets` skill) — it syncs the current session into its dataset first, then registers a fresh session on the chosen one, and its choice beats `COGNEE_PLUGIN_DATASET` for the rest of the launch.
</Warning>
Data added to the same dataset outside Codex (via the SDK or the server) is visible in Codex through the plugin.
## 4. Pick a session (optional)
By default the `session_id` is derived from the Codex thread, so a new conversation starts a new one and `codex resume` continues the same one. Set `COGNEE_SESSION_ID` to pin a specific named session, or to deliberately share one live session across two terminals:
```bash theme={null}
export COGNEE_SESSION_ID="my-project"
codex
```
## 5. Verify
Quit Codex — the `SessionEnd` hook syncs the session into Cognee (an exit-watcher fallback covers a hard exit). Then start a fresh session and ask:
> What do you know from cognee?
Answering from a clean session confirms it's recalling from your Cloud memory. Alongside automatic capture, the plugin ships skills for explicit requests — ask for what you want and Codex picks the matching one:
| Skill | Purpose |
| ------------------------ | ----------------------------------------------------------------------------------------------------------- |
| `memory` | Remember something now, search or recall memory, and improve existing memory |
| `cognee-forget` | Delete memory you ask to forget, along with its derived graph knowledge. Irreversible |
| `codebase` | Index a repository into the code graph and query it. Against a tenant this is **explicit only** — see below |
| `cognee-switch-datasets` | Move the running session to another dataset — see [Choose a dataset](#3-choose-a-dataset) |
Two things about the code graph differ against a Cloud tenant. Repositories are **not** indexed automatically — the plugin only auto-indexes against a local server, so a private checkout is never shipped to a hosted tenant on its own initiative. Ask for it with the `codebase` skill and give it a **git URL**: your tenant clones the repo rather than reading your disk, which is also why the graph reflects your last pushed commit rather than your working tree. A local path is not an option here — the server cannot see it. See [Code graph](/integrations/codex-integration#code-graph).
<Note>
Memory is captured and synced from your sessions over time, and writes build the graph in the background — so a brand-new setup may recall nothing until at least one session has synced. An empty first recall is expected.
</Note>
Share specific datasets with connected agents from the [Connections](/cognee-cloud/connections/managing-connections) page to grant scoped read access.
## Configuration reference
Precedence: environment variables → `~/.cognee/.env` → defaults. There is no `config.json`; older versions wrote one, and `SessionStart` now deletes a leftover file. The `COGNEE_BACKEND` / `COGNEE_CODEX_BACKEND` [mode switch](/integrations/codex-integration#which-mode-wins-and-how-to-switch) follows the same precedence. Wherever it is set, it pins that terminal's mode regardless of where the connection variables are defined.
| Env var | Default | Notes |
| ------------------------- | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| `COGNEE_BASE_URL` | unset | Your tenant URL; enables Cloud/managed-endpoint mode |
| `COGNEE_API_KEY` | unset | Cloud API key from the [API Keys](/cognee-cloud/ui/api-keys) page; auto-minted only in local mode |
| `COGNEE_BACKEND` | unset | `local` or `cloud` — pins this terminal's mode; `COGNEE_CODEX_BACKEND` does the same for this plugin only and beats it |
| `COGNEE_ENV_FILE` | `~/.cognee/.env` | Read the one-time setup file from somewhere else |
| `COGNEE_PLUGIN_DATASET` | `agent_sessions` | Dataset for writes and recall at launch; switchable mid-session |
| `COGNEE_SESSION_ID` | derived from the Codex thread | Override to pin a named session, or share one across terminals |
| `COGNEE_SESSION_PREFIX` | `codex` | Prefix on auto-derived session IDs (`<prefix>_<codex-thread-id>`) |
| `COGNEE_SESSION_STRATEGY` | `per-directory` | Currently inert — the plugin derives session IDs from the Codex thread and never reads this |
Update the plugin with `codex plugin marketplace upgrade cognee` — the `cognee` marketplace tracks `main`, and updates are not automatic. See the [full integration guide](/integrations/codex-integration#session-distillation-self-improvement) for the session-distillation and idle-watcher variables; the advanced session-sync and update-notification knobs are covered in the plugin README.
***
<CardGroup>
<Card title="Full integration guide" icon="book" href="/integrations/codex-integration">
Manual config, hooks reference, and debugging
</Card>
<Card title="Claude Code" icon="bot" href="/cognee-cloud/agent-integrations/claude-code">
The same memory plugin for Claude Code
</Card>
</CardGroup>
# Cursor with Cognee Cloud
Source: https://docs.cognee.ai/cognee-cloud/agent-integrations/cursor
Connect the Cursor editor to your Cognee Cloud tenant via MCP.
[Cursor](https://cursor.com) is an AI-powered code editor with native support for the [Model Context Protocol](https://modelcontextprotocol.io). Register Cognee as an MCP tool provider and Cursor's Agent can `remember`, `recall`, and `forget` against your Cognee Cloud tenant.
## 1. Add Cognee to your MCP config
Cursor reads MCP servers from `~/.cursor/mcp.json`. Add a `cognee` entry that points at your tenant, using your API Base URL and a key from the [API Keys](/cognee-cloud/ui/api-keys) page:
```json theme={null}
{
"mcpServers": {
"cognee": {
"command": "uvx",
"args": ["cognee-mcp"],
"env": {
"COGNEE_BASE_URL": "https://your-tenant.aws.cognee.ai",
"COGNEE_API_KEY": "your-api-key"
}
}
}
}
```
<Info>
This is the stdio form the in-product [Integrations](/cognee-cloud/ui/integrations) page generates. Cognee runs via `uvx` (requires [uv](https://docs.astral.sh/uv/)) — no separate install of Cognee itself. You can also open the config from **Settings → Tools & MCP → + Add MCP Server**.
</Info>
<Warning>
`uvx` requires **[uv](https://docs.astral.sh/uv/)** to be installed first — it is not bundled with Cursor or Windows. A `'uvx' is not recognized...` error in the MCP logs means `uv` isn't on `PATH`.
* **macOS/Linux:** `curl -LsSf https://astral.sh/uv/install.sh | sh`
* **Windows (PowerShell):** `powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"`
Restart your terminal (and fully restart Cursor) after installing so the updated `PATH` takes effect.
</Warning>
<Note>
If you already use other MCP servers in Cursor, merge the `cognee` entry into your existing `mcpServers` block instead of replacing the file.
</Note>
## 2. Verify
In **Settings → Tools & MCP**, restart Cursor (or use the toggle to refresh the connection). You should see Cognee's tools (`remember`, `recall`, `forget`) listed, confirming the server is connected.
## 3. Use it
1. Open the **Chat** panel and make sure **Agent** mode is selected.
2. Issue prompts that use Cognee tools:
* "Remember this file in Cognee"
* "Recall authentication logic from Cognee"
* "What do you know from cognee?"
<Note>
Prefer a dedicated in-editor memory UI with citations? Cursor is built on VS Code, so the [Cognee VS Code extension](/cognee-cloud/agent-integrations/vscode) also runs in Cursor.
</Note>
***
<CardGroup>
<Card title="Full MCP guide" icon="book" href="/cognee-mcp/integrations/cursor">
Local and Docker transports, and troubleshooting
</Card>
<Card title="Cloud MCP" icon="plug" href="/cognee-cloud/connections/cloud-mcp">
How MCP clients connect to Cognee Cloud
</Card>
<Card title="Sessions" icon="timeline" href="/cognee-cloud/ui/sessions">
Inspect what Cursor stored in your live Sessions
</Card>
</CardGroup>
# Hermes Agent with Cognee Cloud
Source: https://docs.cognee.ai/cognee-cloud/agent-integrations/hermes
Connect Hermes Agent to Cognee Cloud with a drop-in memory provider.
Give [Hermes Agent](https://github.com/NousResearch/hermes-agent) persistent memory backed by your Cognee Cloud tenant with a drop-in memory provider plugin. Each completed turn is stored in Cognee's session cache and promoted into the permanent knowledge graph at session end — no code required.
## 1. Install the plugin
The `cognee-integration-hermes-agent` package is not yet published on PyPI. Install it locally from the [cognee-integrations](https://github.com/topoteretes/cognee-integrations) repository:
```bash theme={null}
git clone https://github.com/topoteretes/cognee-integrations.git
cd cognee-integrations
mkdir -p ~/.hermes/plugins/cognee
cp -R integrations/hermes-agent/. ~/.hermes/plugins/cognee/
hermes memory setup # select "cognee" in the memory provider picker
```
<Info>
Requires Python 3.10+. The plugin declares `cognee>=1.0.0,<2.0.0` as a pip dependency.
</Info>
## 2. Connect to Cognee Cloud
Hermes connects to Cloud in **remote mode**: set `COGNEE_BASE_URL` (and `COGNEE_API_KEY`) to your tenant. There are no silent fallbacks — if the configured mode fails, the failure surfaces.
```bash theme={null}
export COGNEE_BASE_URL="https://your-tenant.aws.cognee.ai"
export COGNEE_API_KEY="<your-api-key>"
```
No LLM key is needed in remote mode — your tenant does the knowledge extraction server-side. Non-secret settings are also saved to `$HERMES_HOME/cognee.json`; secrets go to `$HERMES_HOME/.env`.
## 3. Run and verify
Run Hermes as usual — memory is captured and recalled automatically:
```bash theme={null}
hermes # start a session
```
During a session, just talk to the agent:
```
You: Remember that Alice works in engineering.
(Cognee stores this turn in the session cache)
You: What does Alice do?
(Hermes recalls from Cognee Cloud memory)
```
The provider exposes three memory tools to the agent:
* `cognee_recall` — session-first recall with graph fallback
* `cognee_remember` — persist content into the knowledge graph
* `cognee_forget` — delete memory
At session end, if `COGNEE_IMPROVE_ON_END=true` (the default), `cognee.improve(...)` promotes the session cache into your permanent graph.
## Configuration
| Variable | Default | Description |
| ----------------------- | -------- | ----------------------------------------------- |
| `COGNEE_BASE_URL` | — | Cognee API base URL (enables remote/cloud mode) |
| `COGNEE_API_KEY` | — | Cognee service API key |
| `COGNEE_DATASET` | `hermes` | Dataset name for stored memory |
| `COGNEE_TOP_K` | `5` | Max results per recall |
| `COGNEE_IMPROVE_ON_END` | `true` | Run `improve()` at session end |
<Info>
Manage the plugin with `hermes cognee status`, `hermes cognee setup`, `hermes cognee config`, and `hermes cognee install`.
</Info>
***
<CardGroup>
<Card title="Full integration guide" icon="book" href="/integrations/hermes-agent-integration">
Tools, connection modes, and how it works
</Card>
<Card title="Hermes Agent" icon="book" href="https://github.com/NousResearch/hermes-agent">
Learn about Hermes Agent
</Card>
</CardGroup>
# Agent Integrations
Source: https://docs.cognee.ai/cognee-cloud/agent-integrations/index
Connect your coding and terminal agents to Cognee Cloud for persistent, shared memory.
Give your agents persistent memory backed by your Cognee Cloud tenant. Once connected, an agent recalls relevant context before it answers and writes new knowledge back into your knowledge graph automatically — memory survives across sessions, terminals, and even across different agents that share a dataset.
This section walks through connecting each supported agent to **Cognee Cloud**. For local-only setups and full configuration references, each page links out to the dedicated [integration guide](/integrations/index).
## How agents connect
Agents read and write Cognee memory in one of a few ways. Pick the one your agent supports:
| Method | Used by | How it works |
| --------------------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Lifecycle plugin** | Claude Code, Codex | A plugin hooks into the agent's session lifecycle to capture prompts, tool traces, and answers, and to inject recalled context automatically. No code. |
| **MCP tool provider** | Cursor, Claude Desktop, VS Code Copilot | Cognee registers as an MCP server so the client's agent can call `remember` / `recall` / `forget` as tools. |
| **Editor extension** | VS Code (Cognee extension) | A dedicated extension gives the editor a citable project memory with in-editor commands. |
| **Memory provider** | OpenClaw, Hermes Agent | Cognee registers as the agent's memory backend, indexing memory files and recalling across scopes/sessions. |
| **SDK / REST** | Any framework | Call `remember` / `recall` from your own code or the REST API. See [Connecting an Agent](/cognee-cloud/connections/connecting-an-agent). |
## Before you start
Every Cloud connection needs two values from your tenant:
<Steps>
<Step title="Copy your tenant base URL">
Your Cloud base URL looks like `https://your-tenant.aws.cognee.ai`. It's shown in the dashboard and in the [Integrations](/cognee-cloud/ui/integrations) page, which prefills it into every copy-on-click snippet.
</Step>
<Step title="Create an API key">
Generate a key on the [API Keys](/cognee-cloud/ui/api-keys) page. Store it securely — treat it like a password.
</Step>
</Steps>
Most plugins read these from environment variables:
```bash theme={null}
export COGNEE_BASE_URL="https://your-tenant.aws.cognee.ai"
export COGNEE_API_KEY="<your-api-key>"
```
Sharing one `COGNEE_API_KEY` across several agents keeps working — Claude Code, Codex, and OpenCode installs that authenticate this way stay connected and are still recognized from the prefixes they put on their session IDs (`cc_`, `codex_`, `opencode_`), until that plugin gets an identity of its own. Alternatively, supported plugins (`claude-code`, `desktop`, `codex`, `opencode`, `openclaw`, `mcp`, `api`) can be given their own agent identity and their own labeled key, so each one can be tracked and revoked separately. See [per-plugin agent identities](/cognee-cloud/connections/managing-connections#per-plugin-agent-identities).
<Tip>
The in-product [Integrations](/cognee-cloud/ui/integrations) page (under **CONNECT** in the sidebar) has quickstart cards for each agent that inject your live base URL and API key into ready-to-paste snippets — the fastest path to a working connection.
</Tip>
## Choose your agent
<CardGroup>
<Card title="Claude Code" href="/cognee-cloud/agent-integrations/claude-code" icon="bot">
Install the memory plugin and point it at your tenant — memory across every session.
</Card>
<Card title="Codex" href="/cognee-cloud/agent-integrations/codex" icon="terminal">
Enable hooks, install the plugin, and connect to Cloud. Shares a dataset with Claude Code.
</Card>
<Card title="Cursor" href="/cognee-cloud/agent-integrations/cursor" icon="code">
Register Cognee as an MCP tool provider in Cursor's Agent.
</Card>
<Card title="VS Code (Cognee extension)" href="/cognee-cloud/agent-integrations/vscode" icon="code">
A citable project memory extension — remember, ask, and recall in-editor.
</Card>
<Card title="Claude Desktop" href="/cognee-cloud/agent-integrations/claude-desktop" icon="laptop">
Add Cognee as an MCP server in Claude Desktop's config.
</Card>
<Card title="OpenClaw" href="/cognee-cloud/agent-integrations/openclaw" icon="bot">
Register Cognee as OpenClaw's memory provider pointed at your Cloud tenant, with multi-scope recall.
</Card>
<Card title="Hermes Agent" href="/cognee-cloud/agent-integrations/hermes" icon="bot">
Drop-in memory provider in remote mode — turns are captured and promoted to the graph.
</Card>
</CardGroup>
<Note>
Don't see your agent? Any framework can connect through the SDK, an MCP server, or the REST API — see [Connecting an Agent](/cognee-cloud/connections/connecting-an-agent) and the full [integration catalog](/integrations/index).
</Note>
# OpenClaw with Cognee Cloud
Source: https://docs.cognee.ai/cognee-cloud/agent-integrations/openclaw
Register Cognee Cloud as OpenClaw's memory provider with multi-scope recall.
Give your [OpenClaw](https://github.com/openclaw/openclaw) agents Cognee-backed memory sourced from your Cognee Cloud tenant. The plugin indexes your Markdown memory files, recalls relevant context before each run, and searches across sessions with natural language — with optional multi-scope routing (company / user / agent).
## 1. Install the plugin
```bash theme={null}
openclaw plugins install @cognee/cognee-openclaw@2026.6.11
```
Or install locally for development:
```bash theme={null}
cd integrations/openclaw
npm install && npm run build
openclaw plugins install --link .
```
## 2. Run setup
```bash theme={null}
openclaw cognee setup # Cognee only (replaces built-in memory)
openclaw cognee setup --hybrid # keep built-in memory enabled in config
```
## 3. Connect to Cognee Cloud
Cognee Cloud tenants serve the same `/api/v1/*` API as a self-hosted server, so connecting is just the default config pointed at your tenant URL with an API key. Edit `~/.openclaw/openclaw.json`:
```json theme={null}
{
"plugins": {
"entries": {
"cognee-openclaw": {
"enabled": true,
"hooks": {
"allowPromptInjection": true
},
"config": {
"baseUrl": "https://your-tenant.aws.cognee.ai",
"apiKey": "${COGNEE_API_KEY}",
"datasetName": "my-project"
}
}
}
}
}
```
`COGNEE_API_KEY` is **mandatory** for a cloud tenant — remote servers expose no login route, so there's nothing to auto-mint. Set it in the environment the **gateway process** starts from (a daemonized gateway doesn't see `export`s from your current shell), using a key from the [API Keys](/cognee-cloud/ui/api-keys) page:
```bash theme={null}
export COGNEE_API_KEY="<your-api-key>"
```
<Warning>
`hooks.allowPromptInjection: true` is required. Without it, OpenClaw blocks the plugin from reading the prompt in the `before_prompt_build` hook and recall is silently skipped. This key was named `allowConversationAccess` before OpenClaw 2026.4.2; the old key is silently rejected. Restart the gateway after changing it: `openclaw gateway stop && openclaw gateway start`.
</Warning>
<Info>
Do **not** set `mode: "cloud"` — leave it at the default. That mode targets a legacy path scheme that current tenants don't serve (it will 404). All operations — file sync, updates, recall, session capture, and improve — work against Cloud through the standard `/api/v1/*` paths.
</Info>
## 4. Verify
```bash theme={null}
openclaw cognee health # verify Cognee API connectivity
openclaw cognee status # files indexed, dataset info, per-scope breakdown
```
A healthy status shows the dataset ID, indexed file count, and a recent last-sync timestamp.
## Multi-scope memory
Enable multi-scope mode by setting any scope-specific dataset name; memory files are routed to the right dataset by path:
| Scope | Purpose | Example files |
| ----------- | ------------------------------------------- | ------------------------------ |
| **Company** | Shared knowledge across all users/agents | `memory/company/policies.md` |
| **User** | Per-user preferences, feedback, corrections | `memory/user/preferences.md` |
| **Agent** | Per-agent learned behaviors, tool outputs | `memory/tools.md`, `MEMORY.md` |
See the full integration guide for the complete config, search types, and CLI reference.
***
<CardGroup>
<Card title="Full integration guide" icon="book" href="/integrations/openclaw-integration">
Multi-scope config, search types, and troubleshooting
</Card>
<Card title="OpenClaw Docs" icon="book" href="https://docs.openclaw.ai/concepts/memory">
Learn about OpenClaw's memory system
</Card>
</CardGroup>
# VS Code
Source: https://docs.cognee.ai/cognee-cloud/agent-integrations/vscode
Give VS Code a citable project memory backed by your Cognee Cloud tenant.
The **Cognee — Project Memory** extension gives your editor a persistent, **citable** memory of the project. Remember and recall knowledge without leaving VS Code, and *ask your project memory* — get answers grounded in what Cognee knows about the current repository, with links back to the exact source files.
It works against a local Cognee server or, as covered here, your **Cognee Cloud** tenant. The extension needs no environment variables — it's configured entirely through VS Code settings and secret storage.
## 1. Install the extension
Install **Cognee — Project Memory** (extension ID `cognee.cognee-vscode`) from the VS Code Marketplace — search for it in the Extensions view. If it isn't available in your Marketplace yet, install the bundled VSIX with **Install from VSIX…**, or the CLI:
```bash theme={null}
code --install-extension cognee-vscode-0.1.0.vsix
```
<Info>
Requires VS Code `1.85.0` or newer.
</Info>
## 2. Connect to Cognee Cloud
Run **`Cognee: Set Up`** from the command palette (`Cmd/Ctrl + Shift + P`):
<Steps>
<Step title="Enter your endpoint">
Your Cloud tenant URL, e.g. `https://your-tenant.aws.cognee.ai`.
</Step>
<Step title="Enter your API key">
A key from the [API Keys](/cognee-cloud/ui/api-keys) page. It's stored securely in the OS keychain (secret storage) — **not** in `settings.json`.
</Step>
<Step title="Health check">
The command runs a health check so you know the connection works before you start.
</Step>
</Steps>
## 3. Use it
<Steps>
<Step title="Remember">
Open a file, select some code, and run **`Cognee: Remember Selection`**.
</Step>
<Step title="Ask">
Run **`Cognee: Ask My Project Memory`** and ask a question — the answer appears in a panel with ranked, clickable citations that open the exact source file.
</Step>
</Steps>
### Commands
| Command | What it does |
| ------------------------------- | --------------------------------------------------------- |
| `Cognee: Ask My Project Memory` | Open the panel and query the repo's memory with citations |
| `Cognee: Recall` | One-off query from the palette |
| `Cognee: Remember Selection` | Store the current selection (or the whole file) |
| `Cognee: Remember File` | Store a file (also on the explorer context menu) |
| `Cognee: Remember Note` | Store a free-form note typed into an input box |
| `Cognee: Index Workspace` | Bulk-ingest eligible files after a preflight confirmation |
| `Cognee: Forget Project Memory` | Clear the graph (keep files) or delete the dataset |
| `Cognee: Set Up` | Configure endpoint + key and run a health check |
## Per-repository datasets
Each repository maps to a **stable `vscode_<hash>` dataset**, derived from the git remote (or the workspace path as a fallback), so memory stays isolated per project. Set `cognee.datasetOverride` to pin a fixed dataset name instead.
## Settings
| Setting | Default | Description |
| ----------------------------------- | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cognee.endpoint` | `http://localhost:8011` | Cognee backend base URL (set to your tenant URL for Cloud) |
| `cognee.apiKey` | `""` | API key, sent as the `X-Api-Key` header. Prefer `Cognee: Set Up`, which uses secret storage |
| `cognee.datasetOverride` | `""` | Fixed dataset name; empty derives `vscode_<hash>` per repo |
| `cognee.searchType` | `auto` | Recall strategy; `auto` lets Cognee route the query, or force one of `GRAPH_COMPLETION`, `GRAPH_COMPLETION_COT`, `RAG_COMPLETION`, `CHUNKS`, `SUMMARIES` |
| `cognee.topK` | `15` | Max recall results |
| `cognee.includeReferences` | `true` | Attach source citations to answers |
| `cognee.ingestion.respectGitignore` | `true` | Skip `.gitignore`/`.cogneeignore` files when indexing (dependency/build dirs always skipped) |
| `cognee.ingestion.maxFileSizeKb` | `512` | Skip files larger than this when indexing |
| `cognee.requestTimeoutMs` | `300000` | HTTP request timeout |
## How it works
The extension talks to Cognee over the HTTP API: **Ask / Recall** → `POST /api/v1/recall` (with `include_references: true`, scoped to the workspace dataset); **Remember / Index** → `POST /api/v1/remember` (ingest + graph build in one call); **Forget** → `POST /api/v1/forget`. Everything it ingests carries a `Source: <path>` header, so a citation resolves straight to the exact file — even when several files share a name.
<Note>
Prefer GitHub Copilot's agent mode instead of the extension? VS Code can also use Cognee as an **MCP tool provider** via `.vscode/mcp.json` — see [Claude Desktop](/cognee-cloud/agent-integrations/claude-desktop) for the shared Cloud MCP config and [Cloud MCP](/cognee-cloud/connections/cloud-mcp) for the details.
</Note>
***
<CardGroup>
<Card title="GitHub Repository" icon="github" href="https://github.com/topoteretes/cognee-integrations/tree/main/integrations/vscode">
Source, settings, and development guide
</Card>
<Card title="Cursor" icon="code" href="/cognee-cloud/agent-integrations/cursor">
Connect the Cursor editor via MCP
</Card>
</CardGroup>
# Cognee Cloud Architecture
Source: https://docs.cognee.ai/cognee-cloud/cognee-cloud-architecture
Understanding Cognee's managed infrastructure and how components work together
Cognee Cloud layers orchestration and managed services on top of the open-source Cognee storage model. This document explains, at a high level, how the main components fit together.
<Info>
Behind the scenes, Cognee Cloud runs on managed Kubernetes (AWS EKS), where each tenant gets its own isolated deployment backed by a dedicated managed PostgreSQL database.
</Info>
## System Overview
Cognee Cloud's architecture centers around two main layers that work together to provide a managed knowledge processing platform:
### Managed Compute (Kubernetes)
Cognee Cloud runs as a managed service on AWS EKS:
* **API Services**: Each tenant runs its own Cognee application, which serves the REST endpoints and handles authentication (see [Cloud SDK](/cognee-cloud/connections/cloud-sdk))
* **Per-tenant isolation**: Every tenant is provisioned as its own dedicated deployment, so tenants don't share application compute
This keeps execution reliable and scalable while all infrastructure is managed by Cognee Cloud—users never provision servers, clusters, or credentials.
### Storage Services (Managed by Cognee Cloud)
All data persistence is handled through Cognee Cloud's managed storage infrastructure:
* **Dedicated tenant database** – Each tenant has its own managed PostgreSQL database holding all of that tenant's durable state: relational data, the vector embeddings generated during the [cognify process](/core-concepts/main-operations/legacy-operations/cognify), and the knowledge graph
* **Platform database** – A separate PostgreSQL database, isolated from all tenant data, stores users, quotas, and billing records
Within a tenant, each dataset maintains separate storage namespaces for its vector and graph data.
## Key Architectural Principles
* **Tenant & Dataset Isolation**: Each tenant runs as its own deployment with a dedicated database, and processing happens at the dataset level with separate storage namespaces (see [permissions & access control](/cognee-cloud/functionality/permissions-and-access-control) for details)
* **Managed Infrastructure**: Users don't configure compute, storage, or database credentials—everything is managed by Cognee Cloud
* **Compatibility**: Storage schemas remain compatible with [self-hosted Cognee](/getting-started/installation) for easy [migration](/cognee-cloud/local-mode-and-sync)
## Continue exploring
<CardGroup>
<Card title="Permissions & access control" href="/cognee-cloud/functionality/permissions-and-access-control" icon="shield">
See how tenant isolation and RBAC layer onto the storage services.
</Card>
<Card title="Security & data protection" href="/cognee-cloud/functionality/data-and-security" icon="lock">
Tenant isolation, encryption, data durability, and GDPR.
</Card>
</CardGroup>
# Cloud MCP
Source: https://docs.cognee.ai/cognee-cloud/connections/cloud-mcp
Connect MCP-compatible clients to Cognee Cloud
Cognee runs as an [MCP (Model Context Protocol)](https://modelcontextprotocol.io) server. Any MCP-compatible client (Claude Desktop, Cursor, VS Code Copilot) can connect to it as a tool provider. The MCP server can run locally or connect to your Cognee Cloud tenant.
## Step 1 — Start the MCP server
Connect the MCP server to Cognee Cloud using your API Base URL and API key from the [API Keys](/cognee-cloud/ui/api-keys) page:
```bash theme={null}
cognee-mcp --transport sse --port 8001 \
--serve-url https://your-tenant.aws.cognee.ai \
--serve-api-key your-api-key
```
<Note>
For **local mode** (no Cloud connection), omit the `--serve-url` and `--serve-api-key` flags. The server will manage its own local knowledge graph. Local mode requires an `LLM_API_KEY` environment variable.
</Note>
## Step 2 — Add to your MCP client config
Add Cognee as a tool server in your MCP client's configuration. For an SSE connection to a running server, point the client at the server URL:
```json theme={null}
{
"mcpServers": {
"cognee": {
"url": "http://localhost:8001/sse"
}
}
}
```
Alternatively, you can run the server over **stdio** — the client launches `cognee-mcp` itself. Install it with `pip install cognee-mcp`, then use this config (this is the form the in-product **Integrations** page generates):
```json theme={null}
{
"mcpServers": {
"cognee": {
"command": "cognee-mcp",
"env": {
"COGNEE_BASE_URL": "https://your-tenant.aws.cognee.ai",
"COGNEE_API_KEY": "your-api-key"
}
}
}
}
```
For per-client config file locations (Claude Desktop, Cursor, Hermes, VS Code, Gemini CLI, Cline), see [Integrations](/cognee-cloud/ui/integrations).
## Step 3 — Available tools
Once connected, your MCP client gets the Cognee API v1 memory tools:
| Tool | Description |
| ---------- | ------------------------------------------------ |
| `remember` | Store data in memory (add + cognify in one step) |
| `recall` | Search memory with auto-routing |
| `forget` | Delete data from memory |
For detailed setup per client, see the [MCP integration guides](/cognee-mcp/integrations/claude-code).
## Next steps
<CardGroup>
<Card title="Cloud SDK" href="/cognee-cloud/connections/cloud-sdk" icon="terminal">
Connect to Cognee Cloud programmatically using the Python SDK.
</Card>
<Card title="Cloud functionality" href="/cognee-cloud/functionality/data-ingestion" icon="cloud">
Explore the full API surface available in Cognee Cloud.
</Card>
</CardGroup>
# Cloud SDK
Source: https://docs.cognee.ai/cognee-cloud/connections/cloud-sdk
Connect to Cognee Cloud programmatically using the Cognee Python SDK
The Cognee Python SDK connects to Cognee Cloud via `cognee.serve()`. Once connected, all operations (`remember`, `recall`, `forget`, `improve`) route to your cloud tenant.
<Note>
The same `cognee` package is used for both local and cloud workflows. The only difference is calling `cognee.serve()` to connect to a remote instance.
</Note>
## Install
```bash theme={null}
pip install cognee
```
## Complete example
```python theme={null}
import asyncio
import cognee
async def main():
# Connect to Cognee Cloud
await cognee.serve(
url="https://your-tenant.aws.cognee.ai",
api_key="your-api-key"
)
# Store content in memory — ingests, builds knowledge graph, enriches
await cognee.remember(
"Cognee Cloud automates knowledge graph creation in the cloud.",
dataset_name="default_dataset",
)
# Retrieve from memory
results = await cognee.recall(
query_text="What does Cognee Cloud automate?",
)
for result in results:
print(result)
# Disconnect when done
await cognee.disconnect()
asyncio.run(main())
```
## What just happened
### Connecting to cloud
Pass your tenant URL and API key to `cognee.serve()`:
```python theme={null}
await cognee.serve(
url="https://your-tenant.aws.cognee.ai",
api_key="your-api-key"
)
```
Create an API key from the [API Keys](/cognee-cloud/ui/api-keys) page in the Cognee Cloud console. You can find your tenant URL there as well.
To verify the remote connection, check the tenant health endpoint and then confirm authentication against the datasets API:
```bash theme={null}
curl https://your-tenant.aws.cognee.ai/health
curl https://your-tenant.aws.cognee.ai/api/v1/datasets/ \
-H "X-Api-Key: your-api-key"
```
A `200 OK` from `/health` confirms the service is up. For `GET /api/v1/datasets/`, a `200` means the URL and key are working, `401` means the key is wrong, and `404` or a `5xx` usually means the URL or service is unavailable.
### Storing data
```python theme={null}
await cognee.remember(
"Cognee Cloud automates knowledge graph creation in the cloud.",
dataset_name="default_dataset",
)
```
[`remember`](/core-concepts/main-operations/remember) ingests data and builds the knowledge graph in a single call. Data is organized by [dataset](/core-concepts/further-concepts/datasets) for isolation and permissions.
For more control, use the lower-level [`add`](/core-concepts/main-operations/legacy-operations/add) and [`cognify`](/core-concepts/main-operations/legacy-operations/cognify) operations separately.
### Retrieving data
```python theme={null}
results = await cognee.recall(
query_text="What does Cognee Cloud automate?",
)
```
[`recall`](/core-concepts/main-operations/recall) auto-routes the query to the best retrieval strategy. For direct control over search types, see [Search Basics](/guides/search-basics).
### Disconnecting
```python theme={null}
await cognee.disconnect()
```
Closes the connection to the cloud tenant. Credentials remain cached for the next session.
## Next steps
<CardGroup>
<Card title="Cloud functionality" href="/cognee-cloud/functionality/data-ingestion" icon="cloud">
Explore the full API surface available in Cognee Cloud.
</Card>
<Card title="Cloud MCP" href="/cognee-cloud/connections/cloud-mcp" icon="plug">
Connect MCP-compatible clients to Cognee Cloud.
</Card>
</CardGroup>
# Managing Connections
Source: https://docs.cognee.ai/cognee-cloud/connections/managing-connections
View connected agents, manage datasets, and share data across your workspace
To connect an agent to your workspace, open the **Integrations** page under **CONNECT** in the sidebar (route `/integrations`). It walks you through the connect flow for each supported framework. See [Integrations](/cognee-cloud/ui/integrations) for the available options and [Sessions](/cognee-cloud/ui/sessions) to review agent activity.
This page covers the connection concepts that apply once an agent is connected: its identity, status, and how to share datasets with it.
## Connected agents
Each agent user connected to your tenant has:
* **Agent type** — The framework or integration name (e.g., LangGraph, CrewAI).
* **Status** — `LIVE`, `INACTIVE`, or `NEVER_CONNECTED`. Status is computed from the agent's most recent activity, not from the presence of an API key.
* `LIVE` — the agent has produced activity (data added, data accessed, or a search query) within the last 30 minutes.
* `INACTIVE` — the agent has activity on record, but the most recent event is older than 30 minutes.
* `NEVER_CONNECTED` — the agent user exists (and may even have an API key) but has no recorded activity yet.
* **Datasets** — Number of datasets the agent has access to.
* **Last active** — Timestamp of the agent's most recent activity, derived from the latest data ingestion, data access, or search query. Empty for agents in `NEVER_CONNECTED` state.
### Connection identity
Each connection is identified by an `agent_session_name` chosen by the agent. The server combines that name with the authenticated user's ID to derive the underlying connection ID, so the same `agent_session_name` always resolves to the same connection for a given user. The connection API response includes this `agent_session_name` so clients can match rows and details back to the session name they registered.
To inspect connections for the currently authenticated user without knowing the agent ID, call **`GET /agents/connections/me`** (optionally filtered by `?agent_session_name=<name>`). See the [Cloud SDK](/cognee-cloud/connections/cloud-sdk) reference for request/response examples.
### Per-plugin agent identities
Instead of sharing one tenant API key across every plugin, a supported plugin can be provisioned its own agent sub-user and its own labeled API key. Each plugin then shows up as a distinct connection, and its keys can be rotated or revoked without touching the others.
These plugin keys are accepted; any other value returns `404`:
| Plugin key | Label |
| ------------- | -------------- |
| `claude-code` | Claude Code |
| `desktop` | Cognee Desktop |
| `codex` | Codex |
| `opencode` | OpenCode |
| `openclaw` | Openclaw |
| `mcp` | MCP |
| `api` | API/SDK |
#### Provision an identity
```bash theme={null}
curl -X POST https://your-tenant.aws.cognee.ai/api/v1/integrations/plugins/claude-code/provision \
-H "X-Api-Key: <your-user-api-key>"
```
```json theme={null}
{
"pluginKey": "claude-code",
"agentId": "0f1c2d3e-4a5b-6c7d-8e9f-0a1b2c3d4e5f",
"apiKey": "<the-plugin-api-key>",
"created": true
}
```
The call is authenticated and idempotent as a get-or-create:
* The first call creates the agent sub-user for this user and plugin and returns `created: true`.
* Every later call returns the **same** `agentId` with `created: false` and a **rotated** key — minting a new key revokes all of the plugin agent's previous keys. Re-provisioning is the rotation flow.
* `apiKey` is returned once and cannot be retrieved again. Store it before you close the response.
* An unknown plugin key returns `404`. A `409` means an agent for that plugin already exists but could not be resolved as one of your agents.
Provisioning also registers the plugin in the agent-connection registry, under the parent account's tenant, with the connection name `plugin:<plugin_key>` and a connection type of `claude_code`, `opencode`, `mcp`, or `api` (other plugin keys register as a generic `sdk` connection). Because registration happens at provision time rather than on first traffic, the plugin appears in the connections list immediately, in `NEVER_CONNECTED` state until it produces its first activity.
**`GET /api/v1/integrations/status`** returns the current state of every known plugin (`key`, `connected`, `agentId`, `provisionedAt`, `lastActiveAt`, `sessionCount`, `source`) alongside OAuth integration status.
#### Disconnect a plugin
```bash theme={null}
curl -X DELETE https://your-tenant.aws.cognee.ai/api/v1/integrations/plugins/claude-code \
-H "X-Api-Key: <your-user-api-key>"
```
```json theme={null}
{ "disconnected": true }
```
This revokes every API key held by the plugin agent and deactivates its connection. The agent user and everything it wrote are kept — disconnecting is not deleting. The response is `{"disconnected": false}` when the plugin was never provisioned.
Re-provisioning later revives the same identity with a fresh key. To remove the agent entirely, use **`DELETE /api/v1/agents/{agent_id}`**.
### Share a dataset with an agent
You can give a connected agent read access to a dataset you own. This grants access via the [dataset permissions](/cognee-cloud/functionality/permissions-and-access-control#dataset-permissions) endpoint, after which the dataset appears in the agent's dataset list.
<Info>
To connect a new agent, open the [Integrations](/cognee-cloud/ui/integrations) page under **CONNECT**.
</Info>
# Syncing a Local Instance
Source: https://docs.cognee.ai/cognee-cloud/connections/syncing-local-instance
Connect your local Cognee SDK to Cognee Cloud
The Cognee Python SDK can connect to a remote Cognee instance using `cognee.serve()`. Once connected, all operations (`remember`, `recall`, `forget`, `improve`) route to the remote instance instead of running locally.
## Connect to Cognee Cloud
```bash theme={null}
pip install cognee
export COGNEE_AUTH0_DEVICE_CLIENT_ID="your-device-client-id" # required for the interactive login
```
```python theme={null}
import cognee
# Opens the device login in your browser; discovers your cloud tenant automatically
await cognee.serve()
```
After login, the SDK stores credentials at `~/.cognee/cloud_credentials.json`. Subsequent calls to `cognee.serve()` first health-check the saved instance URL and probe an authenticated endpoint with the cached API key, and reconnect without re-authenticating whenever that instance responds and accepts the key — even if the stored Auth0 access token has expired. Re-authentication (a token refresh when possible, otherwise the device login) runs only when no usable credentials are cached, the saved instance does not respond, or it rejects the cached key. The device login itself needs `COGNEE_AUTH0_DEVICE_CLIENT_ID`; without it, `serve()` raises `CogneeConfigurationError` rather than attempting to log in.
<Note>
Because reuse is gated on the instance accepting your key rather than on token lifetime, an expired token by itself does not block startup: a healthy instance keeps accepting your cached credentials through an Auth0 outage.
</Note>
## Connect to any instance
For self-hosted or staging environments, pass the URL and API key directly:
```python theme={null}
await cognee.serve(
url="https://your-tenant.aws.cognee.ai",
api_key="your-api-key"
)
```
Or use environment variables:
```bash theme={null}
export COGNEE_SERVICE_URL="https://your-tenant.aws.cognee.ai"
export COGNEE_API_KEY="your-api-key"
```
```python theme={null}
await cognee.serve() # Reads from environment
```
## Local-to-local connections
Connect to a Cognee backend on the same machine or local network:
```python theme={null}
# Same machine
await cognee.serve(url="http://localhost:8000", api_key="your-api-key")
# Local network
await cognee.serve(url="http://192.168.1.50:8000", api_key="your-api-key")
```
Start the local server with `cognee serve` before connecting. A server with its default posture requires authentication, and `serve()` raises when the key is missing or rejected; omit `api_key` only when the server runs with `ENABLE_BACKEND_ACCESS_CONTROL=false`.
## Usage after connection
Once connected, all SDK operations execute on the remote instance:
```python theme={null}
# Ingested into the remote tenant
await cognee.remember("Einstein developed general relativity in 1915.")
# Queries the remote knowledge graph
results = await cognee.recall("What did Einstein develop?")
# Disconnect when done
await cognee.disconnect()
```
<Note>
The local UI in Cognee Cloud automatically detects running local instances on `localhost:8000` and shows their connection status.
</Note>
## Push a prebuilt graph instead of raw data
Connecting with `serve()` and re-running `remember()` makes the **remote** instance
rebuild the knowledge graph. If you have already built the graph locally and want to
avoid re-deriving it (and the LLM cost that comes with it), use
[`cognee.push()`](/python-api/push) instead. It exports the dataset's graph as a
[COGX archive](/core-concepts/further-concepts/cogx) and imports it on the remote
instance, preserving your local entities and relationships:
```python theme={null}
# Build locally, then upload the graph itself
await cognee.push("my_dataset")
```
The same operation is available from the terminal as
[`cognee push`](/cognee-cli/overview#push-to-cloud).
# Account & Billing
Source: https://docs.cognee.ai/cognee-cloud/functionality/account-and-billing
Usage-based pricing, prepaid token credits, and API keys
These endpoints are available on the platform API (`api.aws.cognee.ai`) and manage account-level resources.
## Pricing
Cognee Cloud is usage-based — you start for free and pay only for what you use, on any plan. Pricing has two parts:
* **Token usage** — **\$1.00 per 1M tokens** processed.
* **Workspaces** — your first workspace is free; each additional workspace is **\$5 / month**.
Your free workspace includes **10M tokens**, **unlimited users**, and **unlimited API calls**, and supports the agentic integrations (Claude Code, Codex, MCP). It is created automatically when you sign up — no Stripe checkout required. You only start paying when you process tokens beyond your prepaid balance or create additional workspaces. The per-workspace charge keeps the free general workspace open to unlimited users while gating the creation of extra workspaces.
Storing data is not billed on top of that — there is no per-seat, per-document, or per-byte charge — but the volume each member can store is capped, and processing what you put in it spends tokens. The cap is the same in every workspace, free or paid: paying buys tokens, additional workspaces, and support, not additional storage. See [Storage and document limits](/cognee-cloud/functionality/data-ingestion#storage-and-document-limits).
### Credit codes
If you have a credit code, you can redeem it for prepaid token credits on the **Billing** page — see [Redeem a credit code](#redeem-a-credit-code) below.
## Plans
The plan chooser lives at `/setup` (and `/plan` redirects there). It is headed **Choose your plan** — *Subscribe to start building with Cognee. Cancel anytime.* — or **Manage your plan** once you are on one, with a line naming your current plan. The token rate is the same on every plan; the plans differ in how many workspaces and how much support you get.
| Plan | Price | Card description | What's included |
| -------------- | -------------------------------------- | ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| **Hobby** | **\$0** / month · badge *Free forever* | *The perfect starting place for your first agent with memory. Free forever.* | 10M tokens included, unlimited users, unlimited API calls, 1 tenant, agentic integrations (Claude Code, Codex, MCP) |
| **Growth** | **\$5** / tenant / month | *Everything you need to ship multi-tenant memory to production.* | All Hobby features, plus unlimited tenants at \$5 each per month, data source integrations (Slack, Notion, Google Drive), in-app support |
| **Enterprise** | **\$2,916** / month | *Dedicated support, your own cloud, and SLAs for memory at scale.* | All Growth features, plus a dedicated Slack channel, a dedicated support engineer, BYO cloud, and a support SLA |
<Note>
The cards are not headed by these plan names — each is identified on screen by its description, price, and badge, in the left-to-right order above. The names surface elsewhere: the second and third cards head their feature lists **All Hobby features, plus:** and **All Growth features, plus:**, and the line above the cards names the plan you are on.
</Note>
Each card also carries a one-line positioning note under its price — *No fixed costs — pay only for the tokens you process*, *Scale tenant by tenant — no platform fee*, and *Flat fee — predictable costs at any scale* — and **+ \$1.00 per 1M tokens**, since token usage is billed the same way on all three. Your current plan is badged **Current plan**, and a plan covered by a higher one you already hold reads **Included in Enterprise**.
A **Have a voucher code?** field sits under the cards — enter a code and press **Enter** to redeem it.
## Billing model
The **Billing** page (`/billing`) is where you manage prepaid token credits. It is owner-only — non-owners see *"Only the owner of this workspace can manage billing and buy credits."*
<Steps>
<Step title="Buy prepaid token credits">
Purchase credits through a Stripe Checkout session. Use the quick-pick amounts (**$10**, **$25**, **$50**, **$100**) or enter a custom amount (minimum **\$0.50**). The estimated number of tokens the amount buys is shown before you check out.
</Step>
<Step title="Track your balance">
An **Available credits** card shows your **Remaining balance** (*of \$X purchased*) and a spend meter. A **By workspace** breakdown appears only once you own more than one workspace. A status line under the meter reports whether [auto recharge](#auto-recharge) is on, off, or failing.
</Step>
<Step title="Turn on auto recharge">
Let Cognee top the workspace up automatically with your saved card when the balance runs low — see [Auto recharge](#auto-recharge) below.
</Step>
<Step title="Redeem a credit code">
Click the **Have a code?** link below the top-up options to reveal the code field and add prepaid credits to the workspace — see [Redeem a credit code](#redeem-a-credit-code) below.
</Step>
<Step title="Review purchases">
Your purchase history lists every credit purchase made on the workspace, including automatic recharges. Rows backed by a Stripe invoice carry a **View invoice** link that opens the hosted invoice in a new tab.
</Step>
</Steps>
### Redeem a credit code
A credit code adds a fixed amount of prepaid token credits to a workspace. Codes are redeemed on the **Billing** page, in the **Have a code?** field below the top-up options:
<Steps>
<Step title="Select the workspace">
Credits are added to the workspace you are currently viewing, so switch to the right workspace first. If no workspace is selected, redemption fails with *"No workspace selected."*
</Step>
<Step title="Enter the code">
Click **Have a code?** to reveal the field, then type your code into it (placeholder **Enter code**). The input is automatically uppercased as you type. Press **Enter** or click **Redeem** to continue.
</Step>
<Step title="Confirm in Stripe">
Redeeming opens a Stripe page to confirm the credit. When it is done, you are returned to the Billing page with the message *"Code redeemed. Your credits are added once Stripe confirms — this can take a few seconds."* If you cancel, you see *"Redemption cancelled — no credits were added."* and nothing is charged or credited.
</Step>
</Steps>
<Note>
Each code can be redeemed once per user — a code you have already redeemed is blocked. An invalid or unredeemable code shows *"Could not redeem code."* Credits appear in your **Remaining balance** a few seconds after Stripe confirms.
</Note>
## Auto recharge
Auto recharge keeps a workspace topped up automatically: when the balance drops to a threshold you choose, Cognee charges your saved card for a fixed amount so work in flight is not interrupted by a `402`. It is configured in the **Auto recharge** panel on the **Billing** page and is **off** until you turn it on.
| Setting | Meaning | Default |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------ | --------- |
| **When balance drops to** | The balance that triggers a recharge. | `$5` |
| **Recharge amount** | How much is charged each time. Minimum **\$5**. | `$20` |
| **Monthly recharge limit** | Auto-recharging pauses for the rest of the month once this total is reached. Must be at least the recharge amount. | `$200` |
| **Notify me when balance reaches** | Optional balance at which you want to be notified. Leave empty to skip. | *(empty)* |
<Steps>
<Step title="Set your amounts">
Fill in the fields above. If the recharge amount is under \**$5**, or the monthly limit is below the recharge amount, the panel shows *"Recharge amount must be at least $5.00 and the monthly limit must be at least the recharge amount."* and refuses to save.
</Step>
<Step title="Turn the switch on">
The **Auto recharge** switch takes effect immediately — you do not have to press **Save** first. Turning it on records your authorization for off-session charges: *"By turning on auto recharge, you authorize Cognee to automatically charge your saved card `{amount}` whenever your balance drops to `{threshold}`, up to `{monthly limit}` per month. You can turn this off at any time."* You can turn it off again whenever you want.
</Step>
<Step title="Save edits separately">
**Save** works independently of the switch, so you can adjust thresholds while auto recharge is off — or edit them later without re-triggering a charge. **Save** is only enabled while the form differs from what is stored, and a green **Saved** confirms the write.
</Step>
</Steps>
Once configured, the panel collapses to a one-line summary — *"On · charges $20.00 when balance drops to $5.00 · $40.00 of $200.00 this month"*, *"Off"*, or *"Not set up yet"* — with a chevron and a **Manage** link to reopen it. While auto recharge is on, the expanded panel shows a progress bar of how much of the monthly limit has already been used.
### Where its state shows up
* **Balance card** — a status line next to the spend meter reads **Auto recharge: on · $X of $Y this month**, **Auto recharge: off**, or **Auto recharge: charge failed — see above**.
* **Failed-charge banner** — if the last automatic charge failed, a red banner appears at the top of the Billing page with the reason: *"Your last automatic recharge failed: `{reason}` Buy credits below to update your card — auto recharge resumes after the next successful charge."*
* **Announcement** — the first time it is available to you, a **Never run out of credits again** dialog introduces the feature with a **Set up auto recharge** button that takes you to Billing. **Maybe later** (or the ✕) dismisses it for good.
<Note>
Auto recharge needs a saved card and a backend that supports it. On an environment that predates the feature, saving returns *"Auto recharge isn't available on this environment yet."*
</Note>
## When credits run out
Every operation that spends credits — uploading files (`remember`), graph processing (`cognify` / `improve`), and search or recall — is checked against your workspace balance before it runs. When the balance is too low, the request is rejected with **HTTP 402** and the UI tells you which operation failed, instead of the request failing silently.
### Insufficient-credits dialog
A **Not enough credits to run ** dialog (for example, *"Not enough credits to run upload"*) appears wherever you are in the app — the check is wired into the shared workspace API client, so uploads, processing, search, and recall all surface it. It shows your **Current balance** and offers **Dismiss** or **Go to billing**.
Its explanation depends on the workspace's [auto recharge](#auto-recharge) state:
| State | Message |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| Off (or never configured) | *"Your workspace doesn't have enough credits left. Add credits on the billing page to continue."* |
| On | *"Auto recharge is on — a top-up may take a couple of minutes. Retry shortly, or add credits on the billing page to continue right away."* |
| On, last charge failed | *"Auto recharge is on but the last automatic charge failed: `{reason}` Add credits manually or update your card on the billing page."* |
| Still being looked up | *"Checking your workspace's credit balance…"* — the dialog never claims auto recharge is on or off before it knows. |
Because the dialog already explains the failure, pages no longer add their own error text for it: on the [Search](/cognee-cloud/ui/search) page the pending answer bubble is removed rather than replaced with a duplicate inline error.
### Persistent notice
If you never saw the dialog — the tab was closed or reloaded before the response arrived — a red bar appears under the top bar on your next visit: **Your last failed — your workspace balance is too low.** with a **Top up credits →** link and a dismiss button.
* The notice is recorded against the workspace that actually failed, so switching workspaces while a request is in flight still attributes it correctly.
* It is dropped automatically after **24 hours**, and dismissing the live dialog also clears it.
### Pre-flight warning on upload
Before an upload is sent, Cognee estimates its cost from the selected files and compares it with the workspace balance. If the estimate reaches or exceeds your balance, a **This upload might use more credits than you have** dialog shows **Could cost** (an *"Up to \~\$X.XX"* ceiling) next to your **Current balance**, with **Cancel** and **Top up first**. There is no *upload anyway* path — the dialog reads *"Top up to continue with this upload."* and both exits abandon the upload, so top up (or turn on [auto recharge](#auto-recharge)) and try again.
The figure is deliberately a ceiling rather than a prediction, and the real cost is usually lower. For PDF and DOCX files the text is extracted and tokenized in the browser first, so the upload button relabels itself **Estimating cost...** and a progress line reads *Estimating cost…* while a large batch is analyzed. When a selection contains a format with no cost model, no warning is shown.
### Low-balance banner
While the balance is above zero but under **$5**, the [Overview](/cognee-cloud/ui/dashboard) carries a red banner naming the actual figure: **Your workspace credit balance is $0.42. Agent requests may fail.** At zero or below, the wording hardens to **Agent requests will fail.** Owners get a **Top up credits →** link; everyone else sees **Ask the workspace owner to top up.** The banner can be dismissed.
<Note>
The threshold was **\$1** before the September 2026 release, and a separate banner used to appear at ≥90% of credits spent. That percentage banner is gone — a percentage of a prepaid balance said less about whether the next request would fail than the balance itself does.
</Note>
### Failures during processing
A run that starts successfully and exhausts the balance part-way through never produces a client-visible 402. Instead the dataset reports it as **Failed — insufficient credits** in its status dot, status pill, and a dedicated banner — see [Datasets](/cognee-cloud/ui/datasets#status-dots). Your uploaded files are kept; top up before retrying, since a retry on an empty balance fails the same way.
After you top up, retrying the build starts a new run that skips documents already processed before the failure — you don't pay again for the work that completed. Only the unprocessed remainder (including the document that was mid-processing when the balance ran out) is run and charged.
This dedicated treatment exists for dataset builds only. An automatic [improve](/cognee-cloud/functionality/session-distillation) run that runs out of credits mid-run is reported on the [Sessions page](/cognee-cloud/ui/sessions) as a regular failed **Improve** entry with its failure reason shown on the entry — there is no credit-specific label there.
## Workspaces
Your first workspace is the **Personal Workspace** — it is free, cannot be deleted, and supports unlimited users.
Each additional workspace you own costs **\$5 / month**:
* Creating an extra workspace charges immediately, with proration applied (`proration_behavior: always_invoice`).
* Removing a workspace deletes it and its data **immediately** — the confirmation reads *This will permanently delete the workspace and all its data. This cannot be undone.* There is no scheduled removal and nothing to cancel. Its paid seat is released from your subscription right after the deletion succeeds.
### Creating a workspace
**Create new workspace** in the workspace picker opens a **Create a new workspace** dialog — *Name your workspace. You can switch between workspaces from the top bar.* It states which case you are in before you commit, and the button matches it:
| Case | What the dialog says | Button |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------- |
| You own no workspace yet | *Your first workspace is **free**. Workspaces you were invited to don't count, so this one costs nothing.* | **Create workspace** |
| You already own one | *A new workspace costs **\$5/month**. You'll be taken to Stripe to confirm payment — the workspace is created once payment succeeds.* | **Continue to payment · \$5/month** |
A workspace the dialog called free is never sent to Stripe, and workspaces you were only *invited* to are not counted against your first free one. **Cancel** genuinely cancels — it closes the dialog and creates nothing — and abandoning Stripe checkout returns you to a usable app rather than a state that needs a reload to recover.
## Payment method
Your free workspace can be created without adding a payment method. A payment method is required for additional workspaces and for billable usage beyond your prepaid balance. When a customer who already has a default payment method on file is charged for a workspace, the subscription is created directly against that card; Stripe Checkout is used as a fallback when card verification (such as 3D Secure) is required.
## Delete your account
Account deletion is self-service, from **Settings → Danger zone**. The **Delete account** action there *"Permanently deletes your account: every workspace you own, all their data, your subscription, and your login."*
<Steps>
<Step title="Open the danger zone">
Go to **Settings** (`/settings`) and scroll to the **Danger zone** card — *"Irreversible account actions"*. Click **Delete account**.
</Step>
<Step title="Confirm with your email address">
Type your own email address into the confirmation field. **Permanently delete my account** stays disabled until it matches (case-insensitively); **Cancel** backs out without deleting anything.
</Step>
<Step title="You are signed out">
Cognee deletes the account server-side (`DELETE /users/me`) — its workspaces, Stripe customer, and login — then routes you through the normal sign-out flow so cookies, local storage, and your Auth0 session are all cleared. If the request fails, the reason is shown inline and you stay signed in.
</Step>
</Steps>
<Warning>
This cannot be undone. Data in the workspaces you own is destroyed along with the account — see [Deletion & erasure](/cognee-cloud/functionality/data-and-security#6-data-protection--gdpr). Deleting a single workspace instead of the whole account is done from **Settings → Workspace** (*Manage your workspace*), where **Workspace settings** offers the deletion and **Yes, delete workspace** confirms it. Two workspaces cannot be deleted there: your personal workspace (*Your personal workspace can't be deleted.*) and one you do not own (*Only the workspace owner can delete this workspace.*).
</Warning>
## API keys
| Endpoint | Description |
| --------------------------------- | ---------------------------------------------------------- |
| `GET /api/v1/api-keys` | List all API keys for the authenticated user |
| `POST /api/v1/api-keys` | Generate a new API key |
| `DELETE /api/v1/api-keys` | Delete an API key |
| `POST /api/v1/api-keys/check` | Validate whether an API key is active |
| `GET /api/v1/api-keys/my-user-id` | Get the user ID associated with the current authentication |
API keys are also managed through the [API Keys UI](/cognee-cloud/ui/api-keys).
## Health
**`GET /health`** — Basic availability probe for the Cognee Cloud service.
Returns an empty `200 OK` response when the service is running.
# Configuration & Ontologies
Source: https://docs.cognee.ai/cognee-cloud/functionality/configuration-and-ontologies
Endpoints for user configuration and ontology management
## User configuration
Store and retrieve per-user configuration for your Cognee Cloud tenant.
**`POST /api/v1/configuration/store_user_configuration`** — Store a configuration object.
**`GET /api/v1/configuration/get_user_configuration/{config_id}`** — Retrieve a specific configuration by ID.
**`GET /api/v1/configuration/get_user_configuration/`** — Retrieve all configurations for the authenticated user.
## Ontologies
Ontologies define the structural schema for knowledge graph extraction. They specify which entity types and relationships Cognee should look for during [cognify](/core-concepts/main-operations/legacy-operations/cognify). See [Ontologies](/core-concepts/further-concepts/ontologies) for the underlying concept.
**`GET /api/v1/ontologies`** — List uploaded ontologies.
**`POST /api/v1/ontologies`** — Upload an ontology file.
```bash theme={null}
curl -X POST https://your-tenant.aws.cognee.ai/api/v1/ontologies \
-H "X-Api-Key: your-key" \
-F "ontology_key=my_domain_ontology" \
-F "ontology_file=@my_ontology.owl" \
-F "description=Domain ontology for entity extraction"
```
| Parameter | Type | Required | Description |
| --------------- | ------ | -------- | ----------------------------------------- |
| `ontology_key` | string | yes | User-defined identifier for this ontology |
| `ontology_file` | file | yes | Ontology file (OWL format) |
| `description` | string | no | Human-readable description |
**`DELETE /api/v1/ontologies/{ontology_key}`** — Delete an uploaded ontology by key.
```bash theme={null}
curl -X DELETE https://your-tenant.aws.cognee.ai/api/v1/ontologies/my_domain_ontology \
-H "X-Api-Key: your-key"
```
On success, returns:
```json theme={null}
{"status": "success", "ontology_key": "my_domain_ontology"}
```
| Status code | Meaning |
| ----------- | ----------------------------- |
| `200` | Ontology deleted successfully |
| `400` | `ontology_key` not found |
| `500` | Internal error (filesystem) |
<Info>
Ontologies are optional. Without one, Cognee uses its default extraction pipeline. Define an ontology when you need domain-specific entity types or relationships.
</Info>
## LLM helpers
The tenant exposes two LLM helper endpoints that power the **Graph Model** editor and the **Prompt** editor in the UI. They are also usable directly with an API key.
**`POST /api/v1/llm/infer-schema`** — Propose a JSON schema (entity types and relationships) from sample text and/or uploaded files. The request is `multipart/form-data`:
```bash theme={null}
curl -X POST https://your-tenant.aws.cognee.ai/api/v1/llm/infer-schema \
-H "X-Api-Key: your-key" \
-F "text=Einstein developed general relativity in 1915." \
-F "data=@sample.pdf" \
-F 'parameters={"temperature":0.1}'
```
| Form field | Type | Required | Description |
| ------------ | ----------- | -------------------- | -------------------------------------------------------------------------------- |
| `text` | string | one of `text`/`data` | Inline sample text to analyze |
| `data` | file(s) | one of `text`/`data` | One or more sample files (uses the same loader engine as ingestion) |
| `parameters` | JSON string | no | Forwarded to the LLM. Allowed keys: `temperature`, `max_tokens`, `top_p`, `seed` |
The response shape is `{ "graphSchema": { ... } }`, validated against the graph model utility before being returned. The endpoint returns `400` when neither `text` nor `data` is supplied, `422` when the LLM output is not valid JSON, and `500` for other inference errors.
**`POST /api/v1/llm/custom-prompt`** — Generate a starter cognify prompt for a given graph model. The request is JSON:
```bash theme={null}
curl -X POST https://your-tenant.aws.cognee.ai/api/v1/llm/custom-prompt \
-H "X-Api-Key: your-key" \
-H "Content-Type: application/json" \
-d '{"graph_model": { ... }, "parameters": {}}'
```
Returns `{ "customPrompt": "..." }`. The same `parameters` keys (`temperature`, `max_tokens`, `top_p`, `seed`) are accepted.
<Info>
Both endpoints live on the tenant API (the per-tenant API base URL shown on the [API Keys](/cognee-cloud/ui/api-keys) page), not the platform management API.
</Info>
# Security & Data Protection
Source: https://docs.cognee.ai/cognee-cloud/functionality/data-and-security
How Cognee Cloud isolates, encrypts, and protects your data — and why your data is safe with us.
Cognee Cloud is a managed, multi-tenant knowledge-graph platform. This page
describes the technical controls that protect your data today. We describe what
we actually do — not aspirations. Where we do **not** yet hold a formal
certification, we say so plainly.
***
## In one paragraph
Every customer runs in a **physically separate database and an isolated network
boundary** — not a shared table with a `tenant_id` column. Data is **encrypted
in transit and at rest**, secrets never live in source code, and each tenant's
workload holds only the credentials for its own data and nothing else. Your data
is stored on managed infrastructure with **automated backups and point-in-time
recovery**, so a bad deploy, an accidental deletion, or hardware failure does not
mean data loss. We are a German company (Topoteretes UG, Berlin) with a named
Data Protection Officer and a GDPR-aligned privacy program.
***
## 1. Tenant isolation — your data is physically separate
Cognee Cloud does **not** use shared-table multi-tenancy. Each tenant is isolated
at three independent layers:
| Layer | What it means for you |
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Dedicated database per tenant** | Each tenant is provisioned its own dedicated managed Postgres project — holding *all* of that tenant's durable state (relational data, vector embeddings, and the knowledge graph). Your data does not share a database, schema, or table with any other customer. |
| **Dedicated network namespace** | Each tenant runs in its own isolated Kubernetes namespace. A default-deny **network policy** blocks all inbound traffic except from within the tenant's own namespace and from our control plane. One tenant's workload cannot open a network connection to another tenant's workload. |
| **Least-privilege credentials** | A tenant's application pod holds **only** the password to its own database. It has no platform API keys, no cloud-provider credentials, and no cluster permissions (no Kubernetes RBAC). Even in the unlikely event a single tenant's workload were compromised, it has no path to reach another tenant's data or the platform control plane. |
**Blast radius by design.** Because isolation is enforced by separate databases
and network boundaries — not by application code correctly filtering a shared
table — a bug in application logic cannot leak one customer's data to another.
Database access can additionally be restricted by IP allow-list to our
infrastructure's egress addresses only.
***
## 2. Encryption
| | Control |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| **In transit (you → Cognee)** | All public endpoints are served over **HTTPS/TLS**. |
| **In transit (internal)** | Application-to-database connections **require TLS** (`ssl: require`); a plaintext database connection is refused. |
| **At rest** | Durable data is stored on managed Postgres infrastructure that **encrypts data at rest**. Backups are likewise encrypted. |
***
## 3. Authentication & access control
* **Authentication** is handled by **Auth0** (Okta). Every API request is
validated against a signed Auth0 JWT — there is no unauthenticated access to
tenant data.
* **API keys** are scoped per tenant and can be rotated.
* **Platform secrets** (database credentials, provider keys) are stored in **AWS
Secrets Manager** and synced into the cluster at runtime via the External
Secrets Operator. They are **never committed to source control** and never
exposed to tenant workloads.
* Provider/API keys are **scoped per environment**, so a credential used in a
development environment can never touch production data.
***
## 4. Why we won't lose your data
Data durability is engineered, not assumed:
* **Managed Postgres with automated backups.** Both the platform control-plane
database and every per-tenant database run on managed Postgres with
**continuous automated backups and point-in-time recovery (PITR)** — we can
restore to a specific moment before an accidental deletion or a bad change.
* **No-data-loss migrations.** When we move or upgrade critical databases, the
process is gated on **verified, row-count-checked copies**, the original is
kept fully intact until a bake period passes (instant rollback), and an
independent backup artifact is retained in object storage. Migrations that
cannot prove an exact copy do not proceed.
* **Infrastructure recovery.** Workloads run on Amazon EKS (Kubernetes) and are
automatically rescheduled on node or pod failure; durable state lives in the
managed database layer, not on ephemeral pod storage.
***
## 5. Where your data is processed (sub-processors)
Cognee is a knowledge-graph platform: to build a graph from your content, that
content is processed by a third-party large-language-model inference provider.
We rely on a small set of managed infrastructure and service providers to run
Cognee Cloud.
***
## 6. Data protection & GDPR
Cognee is operated by **Topoteretes UG**, Schönhauser Allee 163, 10435 Berlin,
Germany — so your data is handled under **EU / German data-protection law**.
* **Data Controller:** Topoteretes UG (Berlin, Germany).
* **Data Protection Officer:** heyData GmbH, Berlin.
* **Your GDPR rights** — access, rectification, erasure, restriction,
portability, and objection — are supported in line with the GDPR.
* **Deletion & erasure:** because each tenant's data lives in a dedicated
database, deleting a tenant deletes the entire database — there is no residue
in shared tables. Backup copies age out per the retention window.
***
## 7. Certifications — where we stand honestly
We do **not** currently hold SOC 2, ISO 27001, or an equivalent third-party
audit certification. We believe it is more useful to tell you exactly what we do
than to imply an attestation we don't have.
The controls described above — physical tenant isolation, encryption in transit
and at rest, least-privilege credentials, secrets management, and backed-up
durable storage — map directly to the technical safeguards those frameworks
require. If your organization needs a formal certification or a signed Data
Processing Agreement (DPA) before adopting Cognee Cloud, please reach out — we're
happy to discuss your requirements and share more detail under NDA.
***
*Last reviewed: 2026-07-22. This document describes the current architecture and
will be updated as our controls evolve.*
# Data Ingestion
Source: https://docs.cognee.ai/cognee-cloud/functionality/data-ingestion
Endpoints for uploading and updating data in Cognee Cloud
These endpoints add data to your Cognee Cloud tenant. All require authentication via API key.
For the underlying concepts, see [Remember](/core-concepts/main-operations/remember) and [Add](/core-concepts/main-operations/legacy-operations/add).
## Remember
**`POST /api/v1/remember`** — Ingest data and build the knowledge graph in a single call.
Combines the add and cognify steps. Equivalent to calling `cognee.remember()` in the Python SDK. Accepts multipart form data. For most workflows, `remember` is the simplest entry point; use the lower-level `add` + `cognify` operations separately when you need to upload multiple files before triggering processing.
```bash theme={null}
curl -X POST https://your-tenant.aws.cognee.ai/api/v1/remember \
-H "X-Api-Key: your-key" \
-F "data=@document.pdf" \
-F "datasetName=my_dataset"
```
| Parameter | Type | Required | Description |
| ------------------- | ------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `data` | file(s) | yes | One or more files to upload |
| `labels` | string (JSON) | no | JSON array of per-file labels, paired positionally with `data` — see [how per-file labels and metadata work](#how-per-file-labels-and-metadata-work). Rejected with `400` when combined with `session_id` or `content_type`. |
| `external_metadata` | string (JSON) | no | JSON array of per-file metadata objects, paired positionally with `data` — see [how per-file labels and metadata work](#how-per-file-labels-and-metadata-work). Rejected with `400` when combined with `session_id` or `content_type`. |
| `datasetName` | string | no | Target dataset name |
| `datasetId` | UUID | no | Existing dataset UUID |
| `session_id` | string | no | Session identifier for grouping operations |
| `node_set` | string\[] | no | Node identifiers |
| `run_in_background` | boolean | no | Run asynchronously (default: `false`) |
| `custom_prompt` | string | no | Custom extraction prompt |
| `chunks_per_batch` | integer | no | Chunks per batch (default: `10`) |
| `graph_model` | string | no | JSON-serialised graph model schema (same format as cognify), including a top-level `title` key. Leave empty to use the default `KnowledgeGraph` model. |
| `content_type` | string | no | Set to `skills` to ingest SKILL.md files as Skill nodes. Only `skills` is supported; leave empty for normal ingestion. |
On a completed run, the response includes `items_processed` — the count of successfully ingested items (entries whose pipeline run did not error).
### Error responses
| Status | When |
| ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `400` | Neither `datasetName` nor `datasetId` was provided, an unsupported `content_type` was sent (anything other than empty or `skills`), or `graph_model` was not valid JSON or could not be converted to a graph model schema. Also returned when `labels` is a JSON array containing non-string entries (a value that does not parse as a JSON array is read as comma-separated labels rather than rejected), when `external_metadata` is not a JSON array of objects or contains the reserved key `node_set`, when the number of `labels` or `external_metadata` entries does not match the number of uploaded files, or when either field is combined with `session_id` or `content_type`. |
| `402` | The configured LLM provider (or LiteLLM proxy) reported that its token budget is exhausted. The response body is `{"detail": "LLM provider requires payment or token budget is exhausted. [LLMPaymentRequiredError]"}`, or `{"detail": "LLM budget exhausted: <provider sentence> [LLMPaymentRequiredError]"}` when the provider's own budget sentence can be identified — see [402 Payment Required](/api-reference/introduction). Handle 402 by surfacing a top-up / billing flow rather than retrying. |
| `403` | You lack write access to the target dataset. The response body is `{"detail": "<message> [PermissionDeniedError]"}`. This also applies to COGX archive imports, where a permission denial previously collapsed into a generic `409`. |
| `409` | The remember run failed during processing, or a blocking (non-background) run finished in an `errored` state. The response body carries the underlying pipeline or validation error message. Cognee's own errors no longer land here — they keep their own status code. |
<Note>
A `graph_model` that is invalid JSON or cannot be converted to a schema is now rejected with `400` rather than being silently ignored. Empty optional fields (`content_type`, `graph_model`, `session_id`, `node_set` entries) are treated as omitted.
</Note>
### How per-file labels and metadata work
`POST /api/v1/remember` and [`POST /api/v1/add`](#add) both accept two optional form fields that attach a label and arbitrary metadata to each uploaded file. They are the HTTP equivalent of the Python SDK's [`DataItem(label=..., external_metadata=...)`](/core-concepts/main-operations/legacy-operations/add).
Each field is sent as a **single JSON part**, not one part per file, because multipart clients cannot reliably repeat an array form field. Entries pair **positionally** with the uploaded files: the Nth entry applies to the Nth file.
| Field | Format | Skipping a file |
| ------------------- | ----------------------------------------------------------------- | --------------- |
| `labels` | JSON array of strings — `["finance", "people", ""]` | Empty string |
| `external_metadata` | JSON array of objects — `[{"source": "crm", "ticket": 42}, null]` | `null` or `{}` |
```bash theme={null}
curl -X POST https://your-tenant.aws.cognee.ai/api/v1/remember \
-H "X-Api-Key: your-key" \
-F "data=@q4-earnings.pdf" \
-F "data=@team-roster.csv" \
-F 'labels=["q4-earnings", "people"]' \
-F 'external_metadata=[{"source": "crm", "ticket": 42}, null]' \
-F "datasetName=my_dataset"
```
When either field carries at least one non-empty entry, it must have exactly one entry per uploaded file — a partial list is ambiguous and returns `400`. `external_metadata` is merged into the file's stored metadata, with your keys taking precedence over loader-derived ones; `node_set` is reserved and rejected, so use the `node_set` form field instead.
Saved values are returned by [`GET /api/v1/datasets/{dataset_id}/data`](/cognee-cloud/functionality/dataset-management#dataset-data) as `label` and `externalMetadata`.
<Note>
**Swagger UI caveat.** Typing a JSON array of strings into the `labels` field in Swagger UI sends it as a comma-joined string (`finance,people,`). The endpoint accepts that form equivalently, so try-it-out works — but it means a label cannot contain a comma unless your client sends real JSON. `external_metadata` has no comma-separated fallback: it must always be valid JSON.
</Note>
## Lower-level operations
The following endpoints provide more granular control over data ingestion. Most users should prefer `remember` above.
### Add
**`POST /api/v1/add`** — Upload files to a dataset without processing.
Accepts multipart form data. Files are stored in the dataset but not yet processed into the knowledge graph — call [cognify](/cognee-cloud/functionality/knowledge-processing#cognify) separately.
```bash theme={null}
curl -X POST https://your-tenant.aws.cognee.ai/api/v1/add \
-H "X-Api-Key: your-key" \
-F "data=@document.pdf" \
-F "datasetName=my_dataset"
```
| Parameter | Type | Required | Description |
| ------------------- | ------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `data` | file(s) | yes | One or more files to upload |
| `labels` | string (JSON) | no | JSON array of per-file labels, paired positionally with `data` — see [how per-file labels and metadata work](#how-per-file-labels-and-metadata-work) |
| `external_metadata` | string (JSON) | no | JSON array of per-file metadata objects, paired positionally with `data` — see [how per-file labels and metadata work](#how-per-file-labels-and-metadata-work) |
| `datasetName` | string | no | Target dataset name (created if it does not exist). Required unless `datasetId` is given. |
| `datasetId` | UUID | no | Existing dataset UUID |
| `node_set` | string\[] | no | Node identifiers |
| `run_in_background` | boolean | no | Run asynchronously (default: `false`) |
Unlike `remember`, `add` has no `session_id` or `content_type`, so `labels` and `external_metadata` are always available here.
Supported file types:
* **Documents** — PDF, TXT, Markdown, CSV, JSON, DOCX, PPTX
* **Images** — PNG, JPG, JPEG, GIF, WEBP, TIFF, BMP, and more, extracted via the tenant's configured vision model
* **Audio** — MP3, WAV, M4A, OGG, FLAC, and more, transcribed to text
Images and audio are converted to text using the tenant's configured LLM before the knowledge graph is built, so they are ingested the same way as text documents. The same file types apply to `POST /api/v1/remember`.
### Update
**`PATCH /api/v1/update`** — Replace an existing document in a dataset.
Accepts multipart form data. Requires both the data item ID and dataset ID as query parameters.
```bash theme={null}
curl -X PATCH "https://your-tenant.aws.cognee.ai/api/v1/update?data_id=<uuid>&dataset_id=<uuid>" \
-H "X-Api-Key: your-key" \
-F "data=@updated_document.pdf"
```
| Parameter | Location | Type | Required | Description |
| ------------ | -------- | --------- | -------- | ----------------------------------------- |
| `data_id` | query | UUID | yes | ID of the document to replace |
| `dataset_id` | query | UUID | yes | ID of the dataset containing the document |
| `data` | form | file(s) | yes | Replacement file(s) |
| `node_set` | form | string\[] | no | Node identifiers |
## Storage and document limits
Keeping data in a workspace is not itself billed — there is no per-document or per-byte charge, only the token cost of [processing](/cognee-cloud/functionality/knowledge-processing) it. Stored volume is capped instead:
| Limit (per member, per workspace) | Value |
| --------------------------------- | ------------------------------ |
| Stored data | **1 GB** (1,074,000,000 bytes) |
| Documents | **50,000** |
Both are checked by [`POST /api/v1/add`](#add) — and by `POST /api/v1/add_text` — before anything is stored, against what you already hold plus the incoming payload. A call that would cross either limit is rejected with **`413 Request Entity Too Large`**, and none of its files are stored:
```json theme={null}
{"detail": "Storage quota exceeded. Used: 1073500000 bytes, incoming: 2000000 bytes, limit: 1074000000 bytes."}
```
```json theme={null}
{"detail": "Document count quota exceeded. Current: 49998, incoming: 5, limit: 50000."}
```
The allowance is yours, not the workspace's: usage covers the data **you** own across every dataset in the workspace you are calling, so each member of a shared workspace gets their own 1 GB and 50,000 documents, and workspaces are counted separately from one another. **`GET /api/v1/quotas/usage`** reports where you stand, as `storageUsedInBytes` against `storageLimitInBytes`.
`POST /api/v1/remember` carries the token-credit guard but not this check, so data ingested through it is not counted against either limit at request time. The check is also per request rather than atomic, so uploads running concurrently can overshoot slightly before the next one is rejected.
<Note>
Uploads from the Cloud UI go through `add` in batches of up to 10 files (200 files per selection), so a selection that crosses a limit part-way keeps the batches that already landed and fails the one that crossed it. Free the space up by deleting data you no longer need — see [Delete](/cognee-cloud/functionality/dataset-management#delete) — and retry the remainder.
</Note>
# Dataset Management
Source: https://docs.cognee.ai/cognee-cloud/functionality/dataset-management
Endpoints for creating, listing, and managing datasets
Datasets are the organizational unit for all data in Cognee Cloud. Each dataset maintains its own knowledge graph and vector store. See [Datasets](/core-concepts/further-concepts/datasets) for the underlying concept.
## List datasets
**`GET /api/v1/datasets/`** — List all datasets accessible to the authenticated user.
```bash theme={null}
curl https://your-tenant.aws.cognee.ai/api/v1/datasets/ \
-H "X-Api-Key: your-key"
```
## Create a dataset
**`POST /api/v1/datasets/`** — Create a new dataset or return the existing one if the name already exists.
```bash theme={null}
curl -X POST https://your-tenant.aws.cognee.ai/api/v1/datasets/ \
-H "X-Api-Key: your-key" \
-H "Content-Type: application/json" \
-d '{"name": "my_dataset"}'
```
<Note>
Datasets are also created implicitly when you call `add` or `remember` with a `dataset_name` that does not yet exist.
</Note>
## Dataset status
**`GET /api/v1/datasets/status`** — Get the processing status of all datasets.
Returns the pipeline state for each dataset: whether cognify is pending, running, or completed.
### In-flight progress
**`GET /api/v1/datasets/status/progress`** — Get the same statuses, plus how far each running pipeline has got.
Takes the same selection parameters as `/status`: repeat `dataset` for each dataset UUID (omit it to cover every dataset you can read) and repeat `pipeline` to pick pipelines (omit it to default to `cognify_pipeline`).
```bash theme={null}
curl "https://your-tenant.aws.cognee.ai/api/v1/datasets/status/progress?dataset=b8a7c3de-4f5a-4b6c-8d9e-0f1a2b3c4d5e" \
-H "X-Api-Key: your-key"
```
Every value is an object `{status, progress}` rather than a bare status. The flat-versus-nested rule matches `/status` — flat for zero or one `pipeline`, nested per dataset and pipeline for more than one:
```json theme={null}
{
"b8a7c3de-4f5a-4b6c-8d9e-0f1a2b3c4d5e": {
"status": "DATASET_PROCESSING_STARTED",
"progress": {
"completed_items": 3,
"total_items": 10,
"current_stage": "add_data_points"
}
}
}
```
```json theme={null}
{
"b8a7c3de-4f5a-4b6c-8d9e-0f1a2b3c4d5e": {
"add_pipeline": { "status": "DATASET_PROCESSING_COMPLETED", "progress": null },
"cognify_pipeline": {
"status": "DATASET_PROCESSING_STARTED",
"progress": { "completed_items": 3, "total_items": 10, "current_stage": "add_data_points" }
}
}
}
```
* **completed\_items** / **total\_items**: files finished out of files in the run. An item that errored still counts as finished, so `completed_items` always reaches `total_items`.
* **current\_stage**: the task most recently entered in an item's task chain. Results stream through the whole chain before they surface, so in practice this names the chain's **final** task (`add_data_points` on the default cognify pipeline) once the first result lands, and is `null` before that. Treat it as "the run is moving", not as a stage-by-stage position — `completed_items`/`total_items` are the fields to drive a progress bar from.
`progress` is `null` when there is nothing in flight to report — before the run's first progress tick, and once the run reaches a terminal state, since the completed or errored record carries no progress snapshot. Treat `null` as "no progress information", not as an error.
Progress ticks are throttled to roughly 20 database writes per run (the first and last file always persist), so on a large batch the numbers advance in steps rather than one file at a time.
<Note>
`/status` is unchanged — it still returns bare status values in the same shape. Use `/status/progress` when you want the N-of-M numbers, and the [cognify WebSocket](/python-api/cognify) when you want live in-flight updates while a run is executing; polling this endpoint is how a client recovers granular progress after a page refresh or a dropped subscription.
</Note>
A `409` is returned if the progress cannot be retrieved — including when you ask for a dataset you do not have read permission on.
## Graph summary
**`GET /api/v1/datasets/graph-summary`** — Get node and edge counts for each dataset.
Counts are computed once per dataset's latest cognify run and cached, so this is much cheaper than the full graph endpoint when polling dataset sizes repeatedly.
```bash theme={null}
curl "https://your-tenant.aws.cognee.ai/api/v1/datasets/graph-summary" \
-H "X-Api-Key: your-key"
```
Pass one or more `dataset_ids` query parameters to summarize specific datasets. Omit it to summarize every dataset you have read access to.
```bash theme={null}
curl "https://your-tenant.aws.cognee.ai/api/v1/datasets/graph-summary?dataset_ids=b8a7c3de-4f5a-4b6c-8d9e-0f1a2b3c4d5e" \
-H "X-Api-Key: your-key"
```
Returns a list of summaries, one per dataset, each containing:
* **datasetId**: The dataset's UUID
* **pipelineRunId**: The dataset's latest cognify run, or `null` if it has never been cognified
* **numNodes** / **numEdges**: Graph size for that run
* **computedAt**: When the counts were cached, or `null` if they were not cached on this call. A `null` here has two causes, and they mean opposite things: either the last count attempt degraded (for example, the graph store was unavailable), in which case the counts are `0` placeholders and the next call retries; or a concurrent caller cached the same run first, in which case the counts you were served are exact. Read the counts themselves rather than treating every `null` as a failure.
A `409` is returned when the summary could not be built. This means the relational read failed — a single unreadable graph store does not cause it, since that dataset simply comes back with zero counts.
## Live dataset updates
**`WS /api/v1/visualize/subscribe/{dataset_id}`** — Follow one dataset's activity over a WebSocket instead of polling.
This is the push replacement for repeatedly calling [`GET /api/v1/visualize/live-events`](/python-api/visualize#over-http) and refetching the graph payload just to notice that a cognify run finished. Authentication accepts the same credentials as every other endpoint — API key header, bearer header, or auth cookie, sent with the handshake — plus one WebSocket-only fallback: a browser cannot set headers when opening a WebSocket, so the same API key or bearer token can be passed as a `?token=` query parameter instead. Prefer the header or cookie where you can send one — a handshake is an HTTP request, and its full URL (query string included) lands in the default access logs of common reverse proxies such as nginx or an AWS ALB, so deployments terminating WebSocket traffic behind one should redact the `token` parameter there. Cognee redacts it from uvicorn's own logs. Nothing is ever read from client frames.
Every frame is a JSON object discriminated by `kind`:
* **`ready`** — `{"kind": "ready", "dataset_id": str, "cursor": str | null}`, sent once the connection is authorized, echoing the cursor it starts from
* **`live_events`** — `{"kind": "live_events", "events": [...], "cursor": str}`, roughly every 2s and only when the delta is non-empty. The events are exactly what `GET /api/v1/visualize/live-events` returns
* **`graph_grew`** — `{"kind": "graph_grew", "pipeline_run_id": str}`, within about 5s of a cognify run for this dataset completing. A run already complete when you connected is the baseline and is not announced
* **`heartbeat`** — `{"kind": "heartbeat", "time": str}`, every 15s or so, so a quiet stream is distinguishable from a dead one
Pass the `cursor` from the last `live_events` frame back as the `since` query parameter when reconnecting; omit it to start from every available event.
Close codes:
* **1008**: not authenticated, or no read permission on this dataset. A retry replays the same rejection, so clients should stop. Permission is re-checked on every poll, so access revoked mid-stream closes the connection this way too
* **1011**: the stream failed server-side; reconnecting is reasonable
A malformed `since`, or a `dataset_id` that is not a UUID, fails request validation *before* the connection is accepted. Depending on the ASGI server the client sees either a 1008 close or a plain HTTP rejection of the handshake, so treat a handshake that never opened as a client-side bug rather than a stream error.
## Dataset data
**`GET /api/v1/datasets/{dataset_id}/data`** — List all data items in a dataset.
Each item also carries the label and metadata attached at upload time:
* **label**: The label given to this file via the `labels` field on [add or remember](/cognee-cloud/functionality/data-ingestion#how-per-file-labels-and-metadata-work), or `null` if none was set
* **externalMetadata**: The stored metadata object — the `external_metadata` entry you sent merged over loader-derived keys, plus a `node_set` key when one was passed at ingest. A file uploaded without metadata stores an empty object rather than `null`.
**`GET /api/v1/datasets/{dataset_id}/data/{data_id}/raw`** — Download the original file for a specific data item. Requires `read` permission on the containing dataset; ownership of the data item itself is not required, so a dataset shared with you is fully downloadable.
## Delete
**`DELETE /api/v1/datasets/{dataset_id}`** — Delete a dataset and all its contents.
**`DELETE /api/v1/datasets/{dataset_id}/data/{data_id}`** — Delete a specific data item from a dataset.
<Warning>
Deleting a dataset removes all associated documents, knowledge graph data, and embeddings. This cannot be undone.
</Warning>
# Knowledge Processing
Source: https://docs.cognee.ai/cognee-cloud/functionality/knowledge-processing
Endpoints for building and removing knowledge graphs
These endpoints transform raw data into structured knowledge graphs and remove data from them.
## Cognify
**`POST /api/v1/cognify`** — Transform datasets into structured knowledge graphs.
Takes uploaded data and runs entity extraction, relationship detection, and embedding generation. This is the same pipeline described in [Cognify](/core-concepts/main-operations/legacy-operations/cognify).
```bash theme={null}
curl -X POST https://your-tenant.aws.cognee.ai/api/v1/cognify \
-H "X-Api-Key: your-key" \
-H "Content-Type: application/json" \
-d '{"datasets": ["my_dataset"]}'
```
By default the request blocks until graph building finishes. Set `run_in_background=true` in the JSON body to return immediately and track progress with the [dataset status](/cognee-cloud/functionality/dataset-management#dataset-status) endpoint.
<Note>
If you used [`remember`](/cognee-cloud/functionality/data-ingestion#remember) to ingest data, cognify was already executed automatically. You only need to call cognify separately when using the `add` endpoint.
</Note>
### Error responses
| Status | Body | Meaning |
| ------ | ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `400` | `{"error": "No datasets or dataset_ids provided"}` | Neither `datasets` nor `dataset_ids` was provided. |
| `402` | `{"error": "Token budget exhausted", "detail": "..."}` | The configured LLM provider (or LiteLLM proxy) reported that its token budget is exhausted. Treat as terminal — see the note below. |
| `409` | `{"error": "..."}` | A referenced `ontology_key` does not exist. |
| `500` | `{"error": "Pipeline run errored", "detail": "ValueError('LLM_API_KEY is missing')"}` | The pipeline run failed server-side (e.g. missing LLM API key, database connection failure, or a dataset that does not exist). `detail` carries the failing task's own error, as the pipeline recorded it (the exception's `repr()`); it falls back to the full `PipelineRunErrored` run repr only when the run carries no error string. |
<Warning>
`POST /api/v1/cognify` returns **`402 Payment Required`** with body `{"error": "Token budget exhausted", "detail": "..."}` when the configured LLM provider or LiteLLM proxy reports token-budget exhaustion. This response is **terminal** — Cognee does not retry budget-exhaustion errors, so clients should not reattempt the request automatically. Handle `402` by surfacing a top-up / billing flow rather than retrying.
</Warning>
## Forget
**`POST /api/v1/forget`** — Remove data from the knowledge graph.
Deletes content from the graph, vector store, and associated metadata. See [Forget](/core-concepts/main-operations/forget) for the underlying operation.
Identify the target with the **`dataset`** field (the dataset name) — not `dataset_name`. There is no `data` field; to remove a single item, pass its `data_id` together with a dataset.
```bash theme={null}
curl -X POST https://your-tenant.aws.cognee.ai/api/v1/forget \
-H "X-Api-Key: your-key" \
-H "Content-Type: application/json" \
-d '{"dataset": "my_dataset"}'
```
### Request body
| Parameter | Type | Description |
| ------------ | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `dataset` | string | Dataset **name** to delete (or clear with `memoryOnly`). Use this or `datasetId`, not both. |
| `datasetId` | UUID | Dataset UUID, alternative to `dataset`. |
| `dataId` | UUID | UUID of a single data item to remove. Requires `dataset` or `datasetId` to also be set. |
| `memoryOnly` | boolean | When `true` with a dataset, delete only graph nodes/edges and vector embeddings and reset pipeline status — raw files are preserved so the dataset can be re-cognified. Default `false`. |
| `everything` | boolean | **DANGER:** when `true`, permanently deletes **all** datasets and data you own. Ignores `dataId`/`dataset`/`datasetId`. Default `false`. |
<Note>
Field names are shown camelCased in the schema; snake\_case aliases (`data_id`, `dataset_id`, `memory_only`) are accepted too.
</Note>
### Forget behavior
* **`dataset` (or `datasetId`) alone** → delete the entire dataset and its graph/vector data.
* **`dataset` + `dataId`** → delete a single item, leaving the dataset intact.
* **`dataset` + `memoryOnly: true`** → clear the dataset's memory but keep the raw files, so you can re-run [cognify](#cognify).
* **`everything: true`** → wipe everything you own.
### Error responses
| Status | When |
| ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `403` | `{"detail": "Request owner does not have necessary permission: [delete] for all datasets requested. [PermissionDeniedError]"}` — the `datasetId` resolves, but you hold no `delete` grant on it. |
| `404` | `{"detail": "Dataset '<name>' not found or not accessible. [DatasetNotFoundError]"}` — the `dataset` name matched nothing among the datasets you can delete. A name that does not exist and one you simply cannot delete give the same answer on purpose, so the response never reveals which dataset names exist. Confirm the exact name with [`GET /api/v1/datasets/`](/cognee-cloud/functionality/dataset-management#list-datasets), then retry. |
| `422` | Invalid parameter combination — e.g. both `dataset` and `datasetId`, `dataId` without a dataset, or `memoryOnly` without a dataset. |
| `500` | `{"error": "An error occurred during deletion."}` — the deletion failed server-side. Deployments predating this typed-error change answered the `403` and `404` cases above with this response instead. |
<Accordion title="Calling forget (and other JSON endpoints) with curl on Windows">
The single quotes around the `-d` body in these examples are a Unix shell convention. `cmd.exe` and PowerShell do not strip them, so the body is sent with literal quotes and the request fails to parse.
**Windows `cmd.exe`** — wrap the body in double quotes and escape the inner double quotes with `\`:
```bat theme={null}
curl -X POST https://your-tenant.aws.cognee.ai/api/v1/forget ^
-H "X-Api-Key: your-key" ^
-H "Content-Type: application/json" ^
-d "{\"dataset\": \"my_dataset\"}"
```
**PowerShell** — escaping is fragile; passing the JSON from a file with `-d "@body.json"` avoids it entirely (this also works on macOS/Linux):
```powershell theme={null}
'{"dataset": "my_dataset"}' | Out-File -Encoding ascii body.json
curl -X POST https://your-tenant.aws.cognee.ai/api/v1/forget `
-H "X-Api-Key: your-key" `
-H "Content-Type: application/json" `
-d "@body.json"
```
</Accordion>
# Permissions & Access Control
Source: https://docs.cognee.ai/cognee-cloud/functionality/permissions-and-access-control
Dataset isolation, roles, tenants, and user management in Cognee Cloud
Cognee Cloud enforces access control at the dataset level. Each dataset gets its own Kuzu graph database and LanceDB vector store, ensuring complete data isolation.
For the full permissions system documentation, see [Cognee Permissions System](/core-concepts/multi-user-mode/permissions-system/overview).
## Dataset isolation
* Each dataset maintains separate storage namespaces.
* Search queries only return results from datasets the user has access to.
* Scoped search (single dataset) and combined search (across accessible datasets) are both supported.
## Managing members from the UI
The **Members** page (open the **Profile** menu in the top-right corner and select **Members**) lists everyone in the current tenant. From this page the tenant **owner** can:
* Invite new members by email — either one at a time or multiple addresses at once.
* Remove existing members from the tenant.
Non-owner members see the same list of tenant members but cannot invite or remove anyone — the invite card and the per-row remove control are only shown to the owner.
### Switching tenants
If your account belongs to more than one tenant, use the tenant switcher in the top bar to move between them. The selected tenant is remembered in your browser and used as the active workspace for datasets, search, and ingestion until you switch again.
## Tenant management
Tenants group users and resources. Each Cognee Cloud workspace operates as a tenant.
**Platform API** (`api.aws.cognee.ai`):
| Endpoint | Description |
| ----------------------------------------- | ----------------------------------- |
| `POST /api/v1/tenants` | Create a new tenant |
| `DELETE /api/v1/tenants` | Remove a tenant |
| `GET /api/v1/tenants/current` | Get current tenant details |
| `GET /api/v1/tenants/current/service-url` | Get the API base URL for the tenant |
| `POST /api/v1/tenants/users` | Assign a user to a tenant |
| `DELETE /api/v1/tenants/users` | Remove a user from a tenant |
## Tenant selection and membership
| Endpoint | Description |
| --------------------------------------------------- | ---------------------------------------------- |
| `POST /api/v1/permissions/tenants/select` | Set the active tenant |
| `GET /api/v1/permissions/tenants/me` | List tenants the authenticated user belongs to |
| `GET /api/v1/permissions/tenants/{tenant_id}/users` | List users in a tenant |
These endpoints back the in-app workspace switcher and the [Members](/cognee-cloud/ui/members) page. Owners use the Members page to invite and remove teammates; non-owners see the roster but not the invite or remove controls. Users who belong to multiple tenants can switch between them without signing out — the selected tenant is persisted in the `cognee_selected_tenant` cookie.
## Roles
Roles define what actions a user can perform within a tenant.
| Endpoint | Description |
| ------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| `POST /api/v1/permissions/roles` | Create a new role |
| `POST /api/v1/permissions/users/{user_id}/roles` | Assign a role to a user |
| `GET /api/v1/permissions/tenants/{tenant_id}/roles` | List roles in a tenant |
| `GET /api/v1/permissions/tenants/{tenant_id}/roles/{role_id}/users` | List users with a specific role |
| `GET /api/v1/permissions/tenants/{tenant_id}/roles/users/{user_id}` | Get roles for a specific user |
| `GET /api/v1/permissions/principals/{principal_id}/datasets` | List the datasets a principal (user, role, or tenant) holds a permission on |
## Dataset permissions
Grant access to datasets for specific users or agents.
**`POST /api/v1/permissions/datasets/{principal_id}`** — Grant dataset permissions to a principal.
The `principal_id` is the UUID of any entity that can hold permissions — this includes both **users** and **agents**. The same endpoint is used regardless of whether you are granting access to a human user or to an agent service account.
The `permission_name` query parameter controls the access level: `read`, `write`, or `delete`.
```bash theme={null}
# Grant read access to a user
curl -X POST "https://your-tenant.aws.cognee.ai/api/v1/permissions/datasets/{user_id}?permission_name=read" \
-H "X-Api-Key: your-key" \
-H "Content-Type: application/json" \
-d '["dataset-uuid-1", "dataset-uuid-2"]'
# Grant read access to an agent (same endpoint, different principal_id)
curl -X POST "https://your-tenant.aws.cognee.ai/api/v1/permissions/datasets/{agent_id}?permission_name=read" \
-H "X-Api-Key: your-key" \
-H "Content-Type: application/json" \
-d '["dataset-uuid-1"]'
```
This is the same mechanism used by the [Connections UI](/cognee-cloud/connections/managing-connections) when you share a dataset with an agent.
### Roles vs. direct permissions
Roles and direct dataset permissions work together:
* **Roles** define a reusable set of capabilities within a tenant (e.g., "viewer", "editor"). Assign a role to a user via `POST /api/v1/permissions/users/{user_id}/roles`, and that user inherits the role's permissions across the tenant.
* **Direct dataset permissions** grant access to specific datasets for a specific principal. Use the `datasets/{principal_id}` endpoint above to give a user or agent access to individual datasets, independent of their role.
Both mechanisms can be combined: a user can have a tenant-level role and additional per-dataset grants.
<Info>
For a complete walkthrough of permission patterns, see [Permission Snippets](/guides/permission-snippets) and the [Cognee Permissions System](/core-concepts/multi-user-mode/permissions-system/overview).
</Info>
# Search & Recall
Source: https://docs.cognee.ai/cognee-cloud/functionality/search-and-recall
Endpoints for querying knowledge graphs and retrieving data
These endpoints query your knowledge graphs. For the full parameter reference, see [Search Basics](/guides/search-basics).
## Recall
**`POST /api/v1/recall`** — Retrieve information from the knowledge graph.
Auto-routes the query to the best retrieval strategy. This is the primary search endpoint.
```bash theme={null}
curl -X POST https://your-tenant.aws.cognee.ai/api/v1/recall \
-H "X-Api-Key: your-key" \
-H "Content-Type: application/json" \
-d '{"query": "What entities are in my data?"}'
```
`query` is a **required** body field: a request that omits it is rejected with `400` and a `detail` array naming the missing field, rather than being answered — see [Requests without a `query`](#requests-without-a-query) for the error shape and the history of the removed default.
The request body accepts an `include_references` boolean (default `true`). When enabled, completion-style answers get a deterministic `Evidence:` block appended to the answer text, citing the source chunks or graph context. The response schema is unchanged. Set `include_references` to `false` to restore the exact prior answer text.
```bash theme={null}
curl -X POST https://your-tenant.aws.cognee.ai/api/v1/recall \
-H "X-Api-Key: your-key" \
-H "Content-Type: application/json" \
-d '{"query": "What entities are in my data?", "include_references": false}'
```
**`GET /api/v1/recall`** — Retrieve recall history for the authenticated user.
### Recall and search history
Recall records the questions it answers. A `POST /api/v1/recall` that runs graph retrieval writes its question and answer into the same history that `POST /api/v1/search` uses, so recall traffic — including questions from agents and the [Search UI](/cognee-cloud/ui/search), which both call recall — appears in both `GET /api/v1/recall` and `GET /api/v1/search`.
What a recall records:
* **The search type that ran.** If you omit `search_type`, recall auto-routes the question, and the history row stores the type the router chose, such as `GRAPH_COMPLETION`.
* **One question-and-answer entry per dataset that answered.** A recall spanning several datasets records a separate pair for each, attributed to that dataset, rather than one combined entry. Recalls whose results carry no dataset — which is what happens when [access control](/cognee-cloud/functionality/permissions-and-access-control) is disabled — are recorded without dataset attribution.
* **Unanswered questions too.** A recall that matched nothing still records one entry, unattributed, with empty answer text.
History is written after retrieval finishes, so the recorded search type and dataset reflect what actually ran. The write is not best-effort: if it fails, the request fails rather than returning an answer that was never recorded. Set `COGNEE_LOG_SEARCH_HISTORY` to `false` to stop recording history altogether.
### Recall prerequisites
Recall reads from an existing knowledge graph — it does not create one. Before recall (or search) returns anything, the dataset must already be ingested **and** processed:
1. [`POST /api/v1/remember`](/cognee-cloud/functionality/data-ingestion#remember), **or**
2. [`POST /api/v1/add`](/cognee-cloud/functionality/data-ingestion#add) followed by [`POST /api/v1/cognify`](/cognee-cloud/functionality/knowledge-processing#cognify).
If you recall before the graph exists, the endpoint returns:
| Status | Body | Meaning |
| ------ | -------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `422` | `{"detail": "Recall prerequisites not met: no database/default user found. Initialize Cognee before recalling by: ... [RecallPreconditionError]"}` | No graph has been built yet for this user/dataset — ingest and cognify first. The `detail` string carries the full remediation steps. |
| `402` | `{"detail": "LLM provider requires payment or token budget is exhausted. [LLMPaymentRequiredError]"}` | The configured LLM provider (or LiteLLM proxy) reported that its token budget is exhausted. When the provider's own budget sentence can be identified it replaces the generic wording, giving `{"detail": "LLM budget exhausted: <provider sentence> [LLMPaymentRequiredError]"}` — see [402 Payment Required](/api-reference/introduction). Handle 402 by surfacing a top-up / billing flow rather than retrying. |
| `403` | `{"detail": "Request owner does not have necessary permission: [read] for all datasets requested. [PermissionDeniedError]"}` | At least one dataset you named is not readable by you. When no dataset is named and you can read none at all, the message reads `Request owner does not have permission: [read] for any dataset.` instead. |
| `409` | `{"error": "An error occurred during recall."}` | An unexpected, non-Cognee error interrupted the request server-side. |
| `422` | `{"detail": "response_schema: unsupported JSON Schema keyword: allOf [CogneeValidationError]"}` | The supplied [`response_schema`](#structured-output-with-response_schema) falls outside the supported subset, or exceeds the depth/property budgets. Every message from this path is prefixed with `response_schema:` — for example `recursive reference '#/$defs/Node'`, `nesting deeper than 10 levels`, or `more than 200 properties in total`. The schema is rebuilt before retrieval starts, so a rejected schema costs no retrieval or LLM work. |
<Note>
Errors raised inside Cognee reach the caller with their own status code and a single `detail` field of the form `"<message> [<ErrorName>]"` — see [Error Handling](/api-reference/introduction#error-handling).
</Note>
### Structured output with `response_schema`
The request body accepts an optional `response_schema` object: a JSON Schema describing the shape you want the completion to conform to, typically produced client-side with `MyModel.model_json_schema()`. The server rebuilds a Pydantic model from it and validates the completion against that model, so each result carries the validated payload in its `structured` field. Only completion-style search types support it.
```python theme={null}
from pydantic import BaseModel
class NLPFacts(BaseModel):
field_name: str
parent_disciplines: list[str]
NLPFacts.model_json_schema()
# {'properties': {'field_name': {'title': 'Field Name', 'type': 'string'},
# 'parent_disciplines': {'items': {'type': 'string'}, ...}},
# 'required': ['field_name', 'parent_disciplines'],
# 'title': 'NLPFacts', 'type': 'object'}
```
```bash theme={null}
curl -X POST https://your-tenant.aws.cognee.ai/api/v1/recall \
-H "X-Api-Key: your-key" \
-H "Content-Type: application/json" \
-d '{
"query": "What is NLP and which disciplines does it belong to?",
"response_schema": {
"type": "object",
"title": "NLPFacts",
"properties": {
"field_name": {"type": "string"},
"parent_disciplines": {"type": "array", "items": {"type": "string"}}
},
"required": ["field_name", "parent_disciplines"]
}
}'
```
The Python SDK sends this field for you: when [`recall(response_model=...)`](/python-api/recall#structured-output-with-response_model) runs against a remote server, the client forwards `response_model.model_json_schema()` as `response_schema`.
#### Supported schema subset
The server accepts only the structural subset that Pydantic itself emits:
| Supported | Rejected with `422` |
| ------------------------------------------------------------------ | ------------------------------------------------------------- |
| Root `"type": "object"` with non-empty `properties` | Any other root type, or an object with no declared properties |
| Primitives: `string`, `integer`, `number`, `boolean`, `null` | Missing or unrecognized `type` on a node |
| `array` with an `items` schema | `array` without `items` |
| `enum` of non-empty strings, integers, or booleans | Empty enums or enums of other value types |
| `anyOf` unions and optionals, and list-valued `type` | `allOf`, `oneOf`, `not`, `patternProperties` |
| Nested objects via `#/$defs/...` or `#/definitions/...` references | Recursive, unresolvable, or otherwise-prefixed `$ref` values |
Two budgets guard the service against abusive schemas: nesting may not exceed **10 levels**, and the schema may not declare more than **200 properties in total**.
<Warning>
Value constraints such as `minLength`, `minimum`, or `pattern` are **not** enforced server-side, and Python-side custom validators do not travel with the schema — only the structure is reconstructed. To run your full validation logic, rehydrate the result against your own class on the client: `NLPFacts.model_validate(result["structured"])`.
</Warning>
## Search
**`POST /api/v1/search`** — Search for nodes in the graph database.
Provides direct control over the retrieval strategy. Accepts a `search_type` parameter to select a specific search mode.
```bash theme={null}
curl -X POST https://your-tenant.aws.cognee.ai/api/v1/search \
-H "X-Api-Key: your-key" \
-H "Content-Type: application/json" \
-d '{
"query": "What did Einstein develop?",
"search_type": "GRAPH_COMPLETION",
"datasets": ["physics_data"]
}'
```
Available search types are documented in [Search Types](/core-concepts/main-operations/legacy-operations/search).
The request body also accepts an `include_references` boolean (default `true`), which behaves the same as on `POST /api/v1/recall`: it appends an `Evidence:` block to completion-style answer text. Set it to `false` to disable.
`query` is **required** here as well, on the same terms as on `POST /api/v1/recall` — see [Requests without a `query`](#requests-without-a-query).
**`GET /api/v1/search`** — Retrieve search history for the authenticated user.
Searches are recorded per dataset. A `POST /api/v1/search` spanning several datasets records one question-and-answer entry for each dataset that answered, rather than a single combined entry, so history grows in proportion to the datasets a search touches. See [Recall and search history](#recall-and-search-history) for the full recording rules, which are shared by both endpoints.
### Requests without a `query`
`query` is a required body field on both `POST /api/v1/recall` and `POST /api/v1/search`. A body that omits it fails request validation before any retrieval runs, and Cognee returns `400` — not the `422` that FastAPI emits by default — with the validation errors in `detail`:
```json theme={null}
{
"detail": [
{
"type": "missing",
"loc": ["body", "query"],
"msg": "Field required",
"input": {}
}
],
"body": {}
}
```
The fix is to send an explicit `query` string. This matters most for integrations written against earlier releases, where both endpoints declared `query` with a default of `"What is in the document?"`: a `{}` body returned `200` with an answer to that placeholder question, searched across every dataset the caller could read. Those calls now fail loudly instead of returning an answer to a question nobody asked. Requests that already pass a real `query` — including every example on this page — are unaffected.
The placeholder string is still the schema example for the field, so the interactive reference and Swagger "Try it out" keep prefilling it in the request body. It is an example you can edit or replace, not a value the server substitutes when you leave `query` out.
The Python SDK is unaffected: [`recall()`](/python-api/recall) and `search()` already take the query as a required positional argument.
## Visualize
**`GET /api/v1/visualize`** — Generate an HTML visualization of a dataset's knowledge graph.
Requires a `dataset_id` query parameter (UUID). Returns a self-contained HTML page with an interactive graph.
```bash theme={null}
curl "https://your-tenant.aws.cognee.ai/api/v1/visualize?dataset_id=<uuid>" \
-H "X-Api-Key: your-key"
```
See also the [Knowledge Graph UI](/cognee-cloud/ui/knowledge-graph) for the built-in visualization.
**`POST /api/v1/visualize/multi`** — Generate a combined visualization from multiple users' datasets.
<Info>
`recall` is recommended for most use cases. Use `search` when you need to specify a particular retrieval strategy.
</Info>
# Session Distillation in Cognee Cloud
Source: https://docs.cognee.ai/cognee-cloud/functionality/session-distillation
Endpoint and automatic triggers for bridging agent sessions into permanent memory
**Session distillation** bridges a conversation from the short-lived session cache into your permanent knowledge graph: it weights existing memories by the session's feedback, persists the Q\&A and agent traces, distills durable guidance into reusable lessons (`session_learnings`), and enriches the graph so future sessions recall from it. Surfaced in the UI as **Self-improvement**.
This page covers the Cloud endpoint and how it fires automatically for connected agents. For the concept and SDK walkthroughs, see [Session Distillation (guide)](/guides/session-distillation), the [Self-Improvement Quickstart](/guides/self-improvement-quickstart), and the [`improve`](/core-concepts/main-operations/improve) operation.
## Improve
**`POST /api/v1/improve`** — Enrich the graph and, when `session_ids` is passed, bridge those sessions into it.
```bash theme={null}
curl -X POST https://your-tenant.aws.cognee.ai/api/v1/improve \
-H "X-Api-Key: your-key" \
-H "Content-Type: application/json" \
-d '{"dataset_name": "project_memory", "session_ids": ["conversation_1"], "run_in_background": true}'
```
Without `session_ids`, the call runs default graph enrichment only. With `session_ids`, it runs the full session-to-graph pipeline described above for each session.
### Request body
| Parameter | Type | Description |
| ---------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `dataset_name` | string | Target dataset name. Provide this **or** `dataset_id`. |
| `dataset_id` | UUID | Target dataset UUID, alternative to `dataset_name`. |
| `session_ids` | string\[] | Sessions to bridge into the permanent graph. Omit to run enrichment only. |
| `run_in_background` | boolean | Run the cognify-heavy pipelines asynchronously. Default `false`. With `session_ids`, the distillation stages still run inside the request, so the call can take minutes — use a generous read timeout. |
| `build_global_context_index` | boolean | Build the global context index after enrichment. Default `false`. **Skipped in background mode** (ordered background chaining is not supported). |
| `node_name` | string\[] | Restrict enrichment to specific named entities. |
<Note>
Either `dataset_name` or `dataset_id` is required. `extraction_tasks`, `enrichment_tasks`, and `data` are power-user overrides most callers never set.
</Note>
### Error responses
| Status | Meaning |
| ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `400` | Neither `dataset_name` nor `dataset_id` was provided. |
| `402` | Insufficient credits to run improve — the tenant's prepaid budget cannot cover the estimated cost. Raised pre-flight, before any work starts. Add credits and retry. |
| `403` | `{"detail": "<message> [PermissionDeniedError]"}` — you lack write access to the target dataset. |
| `404` | `{"detail": "<message> [DatasetNotFoundError]"}` — the requested dataset does not exist or resolved to nothing. |
| `409` | `{"error": "An error occurred during graph improvement."}` — an unexpected, non-Cognee server error interrupted the improve run. Cognee's own errors no longer collapse into this status; they keep their own code and message. |
| `420` | The pipeline run itself completed with an error; the response body carries the failed run's details. |
## Automatic distillation for agent sessions
You rarely call this endpoint directly. When you connect a coding agent through the [Agent Integrations](/cognee-cloud/agent-integrations/index), the plugin fires it for you.
The Cognee coding-agent plugins (Claude Code, Codex) run session distillation for you — you never call `improve()` by hand. A distillation pass fires on three triggers:
| Trigger | When it fires | How |
| ---------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Session end** | You quit the agent | The `SessionEnd` hook drains buffered turns into the server session cache, then fires `POST /api/v1/improve` for the session id in the background. A launch-exit watcher covers exits where the hook never fires — a hard exit, or Codex CLI shutdowns that skip `SessionEnd`. |
| **Every N tool calls** | Long sessions, incrementally | A per-session counter fires an improve every `COGNEE_AUTO_IMPROVE_EVERY` stored tool calls/stops (default **150**), so a long session bridges into the graph without waiting for the end. |
| **On idle** | The session goes quiet | A background idle watcher polls every `COGNEE_IDLE_POLL` seconds and fires an improve after `COGNEE_IDLE_THRESHOLD` seconds of inactivity, then waits at least `COGNEE_IMPROVE_COOLDOWN` seconds before the next idle run. |
<Note>
Overlapping triggers are safe. A per-session improve **lock** on the server serializes concurrent runs, and unchanged session content **dedups server-side by content hash** — so a repeat improve over content that hasn't changed is a cheap no-op, not duplicated work.
</Note>
### Configuration
All triggers are tuned through environment variables read by the plugin. The defaults are chosen so distillation stays out of your way; you rarely need to change them.
| Variable | Default | What it controls |
| ------------------------------- | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `COGNEE_AUTO_IMPROVE_EVERY` | `150` | Stored tool calls/stops between automatic mid-session improves |
| `COGNEE_IDLE_THRESHOLD` | `60` | Seconds of inactivity before an idle improve fires |
| `COGNEE_IMPROVE_COOLDOWN` | `600` | Minimum seconds between idle improve runs |
| `COGNEE_IDLE_POLL` | `10` | How often the idle watcher checks for inactivity |
| `COGNEE_IMPROVE_SUBMIT_TIMEOUT` | `420` (Claude Code) / `180` (Codex) | Read timeout for the improve POST (distillation runs inside the request). The Claude Code plugin raises this to `420` at startup to clear cognee's LLM-retry floor; Codex uses the `180` fallback. |
| `COGNEE_IDLE_DISABLED` | *unset* | Set to `1` / `true` to turn off the idle-watcher trigger entirely |
<Tip>
The plugin READMEs document additional advanced knobs — timing (poll deadlines, busy-retry intervals for a held session lock), session-sync retries, and the update-notification variables (`COGNEE_UPDATE_CHECK`, `COGNEE_UPDATE_CHECK_INTERVAL`). You almost never need them — reach for the table above first.
</Tip>
### Turning it down or off
* **Stop idle-triggered improves:** set `COGNEE_IDLE_DISABLED=1` before launching the agent. Session-end and per-turn improves still run.
* **Reduce mid-session improves:** raise `COGNEE_AUTO_IMPROVE_EVERY` to a large value so the per-turn trigger effectively never fires within a session.
* **Session-end distillation always runs** when the plugin is active — it's how a finished session reaches permanent memory.
### Confirming it happened
* **Cloud UI:** the **Self-improvement** card at the top of a session on the [Sessions page](/cognee-cloud/ui/sessions#self-improvement) shows the status of the last graph enrichment and the dataset it wrote to.
* **Plugin hook log:** each automatic run emits an `improve_fired` event you can grep for when debugging (in local SDK mode, where the plugin calls the library directly instead of the HTTP endpoint, look for `auto_improve_fired` instead).
* **`improve-unsupported.json` marker:** if this file appears in the plugin's shared state directory (24h TTL), the server rejected the improve endpoint and the plugin fell back to the legacy `remember` bridge for that window — a signal the server predates session-aware improve.
# Run the UI Locally
Source: https://docs.cognee.ai/cognee-cloud/local-ui
Launch the Cognee Cloud UI on your own machine — no account required
The Cognee Cloud UI can run entirely on your local machine using `cognee.start_ui()`. This gives you the same interface as Cognee Cloud without needing an account or any cloud infrastructure.
**Before you start:**
* Complete the [Quickstart](/getting-started/quickstart) to make sure your environment is set up
* Have a valid LLM and embedding provider configured (see [Setup Configuration](/setup-configuration/overview)), including `LLM_API_KEY`. The local UI has no in-app field for it, so set it in your environment or `.env` before you start.
## Start the local UI
<Steps>
<Step title="Install Cognee">
```bash theme={null}
pip install cognee
```
</Step>
<Step title="Store some memory">
Store content with [`remember()`](/python-api/remember) before launching the UI — it ingests the data and builds the knowledge graph in one call, so the interface has something to show.
```python theme={null}
import asyncio
import cognee
async def main():
await cognee.remember(
[
"Natural language processing (NLP) is an interdisciplinary subfield of computer science.",
"Machine learning (ML) is a subset of artificial intelligence.",
]
)
asyncio.run(main())
```
</Step>
<Step title="Launch the UI server">
Call `cognee.start_ui()` to start the local frontend and backend servers. Setting `open_browser=True` opens the interface in your default browser automatically.
```python theme={null}
import os
import signal
import time
import cognee
child_pids = []
server = cognee.start_ui(
pid_callback=child_pids.append,
port=3000,
open_browser=True,
start_backend=True,
backend_port=8000,
)
if server:
print("UI available at http://localhost:3000")
try:
while server.poll() is None:
time.sleep(1)
except KeyboardInterrupt:
server.terminate()
server.wait()
for pid in child_pids:
if pid != server.pid:
os.kill(pid, signal.SIGTERM)
```
The UI is available at **[http://localhost:3000](http://localhost:3000)**, and the backend API is available at **[http://localhost:8000](http://localhost:8000)**. Press `Ctrl+C` to stop the server when you are done.
<Note>
`start_ui()` launches the frontend in local mode. Set `LLM_API_KEY` (and any other provider variables) in your environment or `.env` before starting it — Cognee cannot process uploads without one, and the UI has no in-app field for it.
</Note>
</Step>
</Steps>
### Run it with Docker instead
If you have the [cognee repository](https://github.com/topoteretes/cognee) checked out, the bundled Compose file ships a `ui` profile that starts the frontend and backend together — no `pip install` or `start_ui()` call needed:
```bash theme={null}
docker compose --profile ui up
```
The UI is served on **[http://localhost:3000](http://localhost:3000)** and the backend on **[http://localhost:8000](http://localhost:8000)**, the same ports `start_ui()` uses.
The `ui` profile runs the published `cognee/cognee-ui` image rather than building from your checkout, so you can also run the UI on its own — against a backend started any other way — with plain `docker run`:
```bash theme={null}
docker run -p 3000:3000 -e COGNEE_BACKEND_URL=https://cognee.example.com cognee/cognee-ui
```
Leave `COGNEE_BACKEND_URL` unset for the usual localhost setup. See [Connect the UI to the backend](#connect-the-ui-to-the-backend) for how it relates to `NEXT_PUBLIC_LOCAL_API_URL`, and [Docker Deployment](/how-to-guides/cognee-sdk/deployment/docker#additional-information) for the `ui-dev` profile, which builds the hot-reloading UI from source instead.
## Full example
The complete script below combines all three steps:
```python theme={null}
import asyncio
import os
import signal
import time
import cognee
async def main():
# Store sample data — ingests it and builds the knowledge graph
await cognee.remember(
[
"Natural language processing (NLP) is an interdisciplinary subfield of computer science and information retrieval.",
"Machine learning (ML) is a subset of artificial intelligence that focuses on algorithms and statistical models.",
]
)
# Start the UI and backend
child_pids = []
server = cognee.start_ui(
pid_callback=child_pids.append,
port=3000,
open_browser=True,
start_backend=True,
backend_port=8000,
)
if server:
print("UI available at http://localhost:3000")
print("Press Ctrl+C to stop...")
try:
while server.poll() is None:
time.sleep(1)
except KeyboardInterrupt:
server.terminate()
server.wait()
for pid in child_pids:
if pid != server.pid:
os.kill(pid, signal.SIGTERM)
else:
print("Failed to start the UI server.")
if __name__ == "__main__":
asyncio.run(main())
```
<Note>
`cognee.start_ui()` launches the same frontend that powers Cognee Cloud. Data stays on your machine — nothing is sent to any external service.
</Note>
## Connect the UI to the backend
The local UI and the Cognee backend API are separate servers. When you use `cognee.start_ui(..., start_backend=True)` or `cognee-cli -ui`, the UI runs on **port 3000** and the backend runs on **port 8000** by default. If you run either service on a different host or port, configure both the frontend's backend URL and the backend's allowed browser origins.
| Variable | Read by | Default | Purpose |
| --------------------------- | ------------------------------------ | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `COGNEE_BACKEND_URL` | Local UI server, **at run time** | Unset — the browser derives the address (see below) | Backend API base URL, read on every request rather than baked into the build. Set this to point a UI container at a backend; it is the only one of the two that a prebuilt image can honour. |
| `NEXT_PUBLIC_LOCAL_API_URL` | Local UI frontend, **at build time** | In the browser, the page's own protocol and hostname on port `8000`; server-side, `http://localhost:8000` | The older, build-time way to set the same thing. Still honoured, so existing setups keep working, but `NEXT_PUBLIC_*` values are inlined into the JavaScript bundle by `next build` and cannot be changed on an already-built image. |
| `UI_APP_URL` | Backend API | `http://localhost:3000` | Origin allowed through the backend's CORS policy. Set this to the UI's URL when you serve the UI on a different host or port. |
| `CORS_ALLOWED_ORIGINS` | Backend API | — | Comma-separated list of allowed origins. When set, it takes precedence over `UI_APP_URL` — use it to allow more than one origin. |
`COGNEE_BACKEND_URL` must be an absolute `http(s)` URL, for example `http://localhost:8000`. Anything else is a configuration error: in the [Docker image](/how-to-guides/cognee-sdk/deployment/docker#additional-information) the entrypoint reports it and refuses to start, rather than booting and failing every request. A trailing slash is stripped. Because the browser calls the backend directly, the value is the address **as seen from the browser** — a container name or an internal-only hostname will not work. Give either variable a scheme, host, and port only — the UI appends `/api/v1/...` itself.
When neither variable is set, the UI resolves the backend address in the browser from the page you are on, so API calls stay on the host you typed in the address bar. The resolution order is:
1. `COGNEE_BACKEND_URL`, if set. The UI server renders it into the page on each request, and the browser reads it from there — which is how one prebuilt image can serve any backend.
2. `NEXT_PUBLIC_LOCAL_API_URL`, if set — whatever value was inlined when the UI was built, in the browser and on the server.
3. In the browser — the current page's protocol and hostname, with the port pinned to `8000`. Loading the UI on `http://127.0.0.1:3000` targets `http://127.0.0.1:8000`; on `http://localhost:3000` it targets `http://localhost:8000`.
4. During server-side rendering, where there is no browser location — `http://localhost:8000`.
Only the protocol and hostname come from the browser; the port is always `8000` unless you override one of the variables. Set a backend URL when the backend lives on a different machine than the UI, or on a port other than `8000` — `COGNEE_BACKEND_URL` if you are running the published UI image, `NEXT_PUBLIC_LOCAL_API_URL` if you build the frontend yourself.
Deriving the host in the browser keeps the auth cookie on the host you are browsing, but it does not exempt you from the backend's CORS policy. `start_ui()` and `cognee-cli -ui` start the backend with the default allowed origin `http://localhost:3000`, so browsing the UI on any other host — `http://127.0.0.1:3000` included — needs a matching `UI_APP_URL` (or `CORS_ALLOWED_ORIGINS`) on the backend:
```dotenv theme={null}
# .env — browsing the UI on 127.0.0.1 instead of localhost
UI_APP_URL=http://127.0.0.1:3000
```
For example, if the UI and backend are served from different machines:
```dotenv theme={null}
# .env
NEXT_PUBLIC_LOCAL_API_URL=http://192.168.1.20:8000
UI_APP_URL=http://192.168.1.50:3000
```
Restart the UI and backend after changing these values so both processes read the updated environment.
<Note>
`UI_APP_URL` is unrelated to the MCP server's `API_URL` variable, which instead points the [Cognee MCP server](/cognee-mcp/mcp-quickstart#api-mode-shared-knowledge-graph) at a self-hosted backend. The two are easy to confuse because both wire a component to the Cognee API.
</Note>
## Cloud-only pages
The local UI is the same frontend as Cognee Cloud, but the pages that report on a hosted workspace's usage and spend are not part of the open-source build. In local mode they render a text-only notice — *Build your own dashboard from the API, or use the hosted one in Cognee Cloud* — with an **Open Cognee Cloud** link, instead of the real surface:
| Surface | Local mode |
| --------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| [Activity](/cognee-cloud/ui/activity) | **Activity is a Cognee Cloud feature** |
| [Analytics](/cognee-cloud/ui/analytics) | **Analytics is a Cognee Cloud feature** |
| [Memory Coverage](/cognee-cloud/ui/memory-coverage) | **Memory Coverage is a Cognee Cloud feature** |
| Overview → memory flow diagram | **Live memory graph is a Cognee Cloud feature**, over a blurred placeholder |
| Overview → **Cost Savings**, **Memory Coverage**, **Activity** panels | The same notice inside each panel frame |
Billing also does not apply locally, so the **Billing / Pricing** button and the sidebar footer that holds it are hidden, and the credit banners and pulse survey described on the [Overview](/cognee-cloud/ui/dashboard) page never appear.
Everything else works the same locally as it does in the cloud: [Datasets](/cognee-cloud/ui/datasets), [Search](/cognee-cloud/ui/search), [Skills](/cognee-cloud/ui/skills), [Sessions](/cognee-cloud/ui/sessions), the [Mindmap](/cognee-cloud/ui/knowledge-graph), and [Integrations](/cognee-cloud/ui/integrations).
## Troubleshooting the local UI
<AccordionGroup>
<Accordion title=""Cognee frontend is not available" / prompted to download">
In a pip-installed package, the frontend is not bundled in the runtime environment. On first launch, `start_ui()` looks for a local `cognee-frontend` directory and, if it can't find one, prints:
> The cognee frontend is not available on your system.
It then asks `Would you like to download the frontend now? (y/N)`. Answer `y` to download the frontend that matches your installed version from GitHub releases and cache it in `~/.cognee/ui-cache/` (a one-time setup per cognee version, reused offline afterwards).
To skip the prompt and download automatically, pass `auto_download=True` to `cognee.start_ui()`. The `cognee-cli -ui` command already sets this, so it never prompts.
If the download fails with a `404`, the release for your version does not exist on GitHub yet or the installed version is a development/mismatched build. Install a stable release of cognee (`pip install -U cognee`) and try again.
</Accordion>
<Accordion title="Signing in bounces straight back to the login page, or fails on 127.0.0.1">
The auth cookie is host-scoped, and `localhost` and `127.0.0.1` are distinct hosts to the browser even though they resolve to the same machine. If the page and the API are not on the same host, the cookie never applies to the requests that check it: the `GET /api/v1/users/me` check comes back `401` and the UI sends you back to `/local-login` on every attempt.
The UI now derives the API host from the page you loaded, so the cookie is always set on the host you are browsing. What it does not do is widen the backend's CORS policy. If you sign in on a host other than `localhost`, work through these in order:
* **Allow the origin you browse to.** `start_ui()` and `cognee-cli -ui` start the backend allowing only `http://localhost:3000`, so signing in on `http://127.0.0.1:3000` is refused before the cookie is ever set — the login form reports `Cannot connect to local backend at http://127.0.0.1:8000. Is it running?`. Set `UI_APP_URL` (or `CORS_ALLOWED_ORIGINS`) to the exact origin in your address bar and restart the backend — see [Connect the UI to the backend](#connect-the-ui-to-the-backend).
* **Check `COGNEE_BACKEND_URL` and `NEXT_PUBLIC_LOCAL_API_URL`.** Either explicit value wins over the browser-derived host — `COGNEE_BACKEND_URL` first. If one points at a different hostname than the one in your address bar, either unset it or make the two match.
* **Clear stale cookies** for the old host after changing any of this, then sign in again.
The simplest fix is to browse the UI on `http://localhost:3000`, which every default already covers.
</Accordion>
<Accordion title="localhost refuses to connect (ERR_CONNECTION_REFUSED)">
If the browser can't reach `http://localhost:3000`, the frontend server isn't running. Check these in order:
* **Node.js and npm are installed.** The UI runs on Next.js and needs Node.js. If either tool is missing, `start_ui()` first tries to install `nvm` and Node.js automatically on supported platforms; if that fails, it logs `Cannot start UI` and you should install Node.js from [nodejs.org](https://nodejs.org/) before relaunching.
* **The port is free.** `start_ui()` returns `None` and logs `ports already in use` if port `3000` (frontend) or `8000` (backend) is taken. Stop the conflicting process, or pass a different `port` / `backend_port`.
* **Give Next.js time to compile.** After launch the server prints `The UI will be available once Next.js finishes compiling`. The first compile takes a few seconds — reload once you see the `[FRONTEND]` logs report it's ready.
* **Watch the `[FRONTEND]` logs.** If the process exits early, `start_ui()` logs `Frontend server failed to start` — the streamed `[FRONTEND]` output above it shows the underlying error (for example a failed `npm install`).
</Accordion>
</AccordionGroup>
## Next steps
<CardGroup>
<Card title="Cognee Cloud" href="/cognee-cloud/overview" icon="cloud">
Move to the hosted version for managed infrastructure and collaboration features.
</Card>
<Card title="Core Concepts" href="/core-concepts/overview" icon="brain">
Learn about remember, recall, improve, and forget operations.
</Card>
</CardGroup>
# Cognee Cloud
Source: https://docs.cognee.ai/cognee-cloud/overview
Use the Cognee UI and pipelines hosted in the cloud, or run everything locally for free — no account required
Cognee Cloud gives you the full Cognee platform — a web UI, managed pipelines, and storage — as a hosted service. If you prefer to stay local, you can also **run the UI and all pipelines entirely on your own machine for free**, with no account required, using `cognee.start_ui()`.
## What Cognee Cloud provides
* **Managed infrastructure** — Preconfigured compute, storage ([PostgreSQL](https://postgresql.org/), [LanceDB](https://lancedb.com/), [Kuzu](https://github.com/kuzudb/kuzu)), and pipeline execution. No local installation required.
* **Pipeline execution** — Run [remember](/core-concepts/main-operations/remember), [recall](/core-concepts/main-operations/recall), [forget](/core-concepts/main-operations/forget), and the lower-level [add](/core-concepts/main-operations/legacy-operations/add), [cognify](/core-concepts/main-operations/legacy-operations/cognify), [search](/core-concepts/main-operations/legacy-operations/search) operations from the UI or API.
* **Web UI** — Upload data, explore knowledge graphs, run searches, and manage datasets through the browser.
* **Python SDK** — The [`cognee` package](/cognee-cloud/connections/cloud-sdk) connects to Cognee Cloud via `cognee.serve()`, using the same API as local Cognee.
* **Agent connections** — Connect agent frameworks, local instances, or external data sources. See [Integrations](/cognee-cloud/ui/integrations).
* **Multi-tenancy** — Dataset-level isolation with role-based access control. See [Permissions](/cognee-cloud/functionality/permissions-and-access-control).
## Hosted cloud vs. free local UI
| | Hosted Cloud | Local UI (free) |
| ------------------- | ------------ | --------------- |
| Account required | Yes | No |
| Infrastructure | Managed | Your machine |
| Collaboration | Yes | No |
| Same UI & pipelines | Yes | Yes |
* **[Hosted Cognee Cloud](/cognee-cloud/sign-up)** — Managed infrastructure, storage, and compute. Sign up at [platform.cognee.ai](https://platform.cognee.ai/).
* **[Local UI (free)](/cognee-cloud/local-ui)** — Run the full UI on your own machine with `pip install cognee` and one function call. No account, no cloud.
## Workspace sleep and wake
A hosted workspace that has been quiet for a while scales to zero — it goes to sleep. Nothing is lost while it sleeps, and you do not have to do anything to bring it back: opening it wakes it.
While it starts, the app shows a **Waking ** screen in place of the [Overview](/cognee-cloud/ui/dashboard): *It went to sleep after a quiet spell. Starting it back up usually takes under a minute, and nothing was lost.* A **RESTORING · m:ss** counter under the message tracks how long it has been going, and keeps counting across a reload or a route change rather than restarting. After about **75 seconds** the message changes to *Still waking up. This one is taking longer than usual, but your data is safe.*
A few details worth knowing:
* **Every way in wakes it.** Switching workspaces in the picker, reloading the page, and opening a direct URL all go through the same wake, and the wake survives a reload rather than treating a still-cold workspace as an ordinary reconnect.
* **A workspace that falls asleep mid-session is caught.** If it goes to sleep while your tab is open, the next request notices, wakes it, and retries once it is serving.
* **The wait is bounded.** A workspace that never comes back lands on an error within minutes rather than spinning forever: ** didn't wake up** — *It was asleep and we asked it to start, but it hasn't come online. Try again, or sign out and back in.*
* **The workspace badge says which state it is in.** A dashed, hollow badge means not running; its accessible label reads *running*, *asleep*, *waking up*, or *no active subscription*. In the workspace picker each row spells it out under the name — **Asleep · wakes when you open it**, **Waking up…**, **Didn't wake up**, or **No active subscription**.
* **Not every page takes over the screen.** The full **Waking ** screen replaces the content of the Overview and of every page that reads from the workspace: [Datasets](/cognee-cloud/ui/datasets), [Search](/cognee-cloud/ui/search), [Skills](/cognee-cloud/ui/skills), [Mindmap](/cognee-cloud/ui/knowledge-graph), [Sessions](/cognee-cloud/ui/sessions), [Analytics](/cognee-cloud/ui/analytics), [Activity](/cognee-cloud/ui/activity), and the [Graph Model editor](/cognee-cloud/ui/datasets#graph-model-editor). Pages that do not touch the workspace stay usable while it starts — [Memory Coverage](/cognee-cloud/ui/memory-coverage) is one of them, and shows only the provisioning banner.
A fetch that fails just after a wake names that as the likely cause rather than reporting a flat error: [Activity](/cognee-cloud/ui/activity) and [Analytics](/cognee-cloud/ui/analytics) both read *Couldn't load activity. If the workspace was asleep it may still be starting up; otherwise the activity endpoint is not answering.*
<Note>
Sleep applies to hosted workspaces only. The [local UI](/cognee-cloud/local-ui) runs for as long as your own process does.
</Note>
## Relationship to Cognee OSS
Cognee Cloud uses the same concepts, operations, and API patterns as [open-source Cognee](/core-concepts/overview). The difference is deployment:
* **Cognee Cloud** — Hosted persistence, managed compute, collaboration features.
* **Open-source Cognee** — Local development, custom infrastructure, air-gapped environments.
You can connect a local Cognee instance to Cognee Cloud using [`cognee.serve()`](/cognee-cloud/connections/syncing-local-instance).
## Get started
<CardGroup>
<Card title="Try the UI for free" icon="monitor" href="/cognee-cloud/local-ui">
Run the Cognee Cloud UI on your own machine — no account required.
</Card>
<Card title="Create an account" icon="key" href="/cognee-cloud/sign-up">
Sign up and generate an API key.
</Card>
<Card title="Quick start" icon="play" href="/cognee-cloud/quickstart">
Upload data, build a knowledge graph, and run your first search.
</Card>
<Card title="Navigating the UI" icon="mouse-pointer" href="/cognee-cloud/ui/dashboard">
Learn what each page in the Cognee Cloud console does.
</Card>
<Card title="Cloud functionality" icon="cloud" href="/cognee-cloud/functionality/data-ingestion">
Explore the API endpoints available in Cognee Cloud.
</Card>
<Card title="Connections" icon="plug" href="/cognee-cloud/ui/integrations">
Connect agents, databases, or a local instance.
</Card>
<Card title="Architecture" icon="building" href="/cognee-cloud/cognee-cloud-architecture">
How managed compute, storage services, and datasets fit together.
</Card>
</CardGroup>
# Quick Start
Source: https://docs.cognee.ai/cognee-cloud/quickstart
Upload data, build a knowledge graph, and search — in under five minutes
After [creating your account](/cognee-cloud/sign-up), brand-new users land on a **Welcome to Cognee Cloud** splash — *Your AI memory layer.* Click **Let's go →** to begin onboarding. You can re-enter onboarding at any time from the **?** help menu in the top bar — choose **Onboarding**.
<Note>
On your first sign-in a **Personal Workspace** is created for you automatically — there is no naming step. The welcome screen appears right away while the workspace finishes provisioning in the background. You can create additional workspaces later and give them a name. Uploaded data lands in your **`default_dataset`**.
</Note>
## Choose how to start
Onboarding opens with **How do you want to start?** and offers three paths:
<CardGroup>
<Card title="Claude Code" icon="terminal">
Connect your Claude Code agent to Cognee Cloud as a memory layer.
</Card>
<Card title="Codex" icon="terminal">
Connect your Codex agent to Cognee Cloud as a memory layer.
</Card>
<Card title="Company Dataset" icon="building">
Upload your own data and build a searchable knowledge graph.
</Card>
</CardGroup>
## Company Dataset flow
If you choose **Company Dataset**, onboarding guides you through three steps.
<Steps>
<Step title="Connect your data">
Drag files into the drop zone, click to browse, or paste text content directly. Supported formats: PDF, CSV, TXT, Markdown, and JSON.
</Step>
<Step title="Building your memory">
Cognee processes your data automatically: **Setting up workspace** → **Uploading files** → **Building knowledge graph**. This runs the same pipeline as the [`cognify`](/core-concepts/main-operations/legacy-operations/cognify) operation. Processing time depends on the volume of data.
</Step>
<Step title="Ask cognee anything">
Once your knowledge graph is ready, enter a natural language query and Cognee searches the graph and returns contextual results — the same mechanism as [`recall`](/core-concepts/main-operations/recall) and [`search`](/core-concepts/main-operations/legacy-operations/search). When you're ready, click **Connect my agent now →**.
</Step>
</Steps>
## Agent flow (Claude Code / Codex)
If you choose **Claude Code** or **Codex**, onboarding helps you wire your agent into Cognee Cloud.
<Steps>
<Step title="Open your terminal">
The panel opens with **Connect Claude Code** (or **Codex**) and the reminder "Run this setup in your Terminal — not inside Claude Code." Everything here is run from your own shell, not from inside the agent.
</Step>
<Step title="Copy and run">
Onboarding gives you a single block to paste: it writes your `COGNEE_BASE_URL` and `COGNEE_API_KEY` to `~/.cognee/.env` — the file both the Claude Code and the Codex plugin read at session start — and registers and installs the plugin in the same paste. Because the credentials live in a file rather than the shell's environment, they survive closing the terminal, and re-running the block replaces the old values instead of stacking duplicates.
Use the **Mac / Windows** toggle at the top right of the panel to pick your shell — on **Windows** the block comes out as PowerShell (`New-Item` + `Set-Content`, with `icacls` locking the file to your user) and the step above it tells you to open PowerShell rather than Terminal. The same toggle sits inside the dashboard's connect modals and the Integrations page's setup modals; see [Mac / Windows](/cognee-cloud/ui/integrations#mac--windows) for everything it rewrites. For cmd.exe, or to set the credentials by hand, see the [Claude Code](/cognee-cloud/agent-integrations/claude-code#2-point-it-at-cognee-cloud) or [Codex](/cognee-cloud/agent-integrations/codex#2-point-it-at-cognee-cloud) integration pages.
</Step>
<Step title="Start your agent">
Run `claude` (or `codex`). In Claude Code you should now see Cognee in the status bar; with Codex, memory connects automatically from here.
</Step>
<Step title="Continue to the dashboard">
Click **Continue**. It is always enabled — onboarding does not wait to detect agent activity before letting you through. **Back** returns to the agent picker, and **Set up later** skips the whole flow.
<Note>
Agent not recalling anything? Confirm `~/.cognee/.env` exists and contains both `COGNEE_BASE_URL` and `COGNEE_API_KEY`, and that you started the agent after writing it. Then see "Debugging & Resuming Sessions" in the full plugin guide: [Cognee Plugin for Claude Code](/integrations/claude-code-integration) · [Cognee Plugin for Codex](/integrations/codex-integration).
</Note>
</Step>
</Steps>
<Note>
You can skip the onboarding at any point and return to it later. All progress is preserved.
</Note>
## Next steps
<CardGroup>
<Card title="Workspace Overview" href="/cognee-cloud/ui/dashboard" icon="layout-dashboard">
Overview of your workspace, stats, and quick actions.
</Card>
<Card title="Open Datasets" href="/cognee-cloud/ui/datasets" icon="brain">
Manage your data and knowledge graphs.
</Card>
<Card title="Connect an agent" href="/cognee-cloud/ui/integrations" icon="plug">
Use Cognee as memory for your agent.
</Card>
<Card title="Cloud functionality" href="/cognee-cloud/functionality/data-ingestion" icon="cloud">
Explore the full API surface available in Cognee Cloud.
</Card>
</CardGroup>
# Create an Account
Source: https://docs.cognee.ai/cognee-cloud/sign-up
Sign up for Cognee Cloud and generate your first API key
You can sign up with a social provider (no password to manage) or with an email address and password.
<Info>
Cognee Cloud supports **Google** and **GitHub** sign-in, as well as **email + password**.
</Info>
## Sign up
<Steps>
<Step title="Open the console">
Go to [platform.cognee.ai](https://platform.cognee.ai/) and choose **Continue with Google**, **Continue with GitHub**, or enter an **email** and **password** (at least 8 characters, confirmed).
</Step>
<Step title="Authorize or verify">
For a social provider, approve the OAuth prompt — Cognee uses the provider for identity only. For email + password, check your inbox and click the verification link to confirm your address.
</Step>
<Step title="Account created">
Your account is created and ready to use.
</Step>
</Steps>
### Email verification
When you sign up with email and password, you must verify your email before signing in:
1. After registering, you are prompted to **verify your email** and a verification message is sent to your address.
2. Clicking the link confirms your address and takes you to an **Email verified!** confirmation page (*Your account is ready. Sign in to get started.*).
3. Select **Sign in** to return to the login page, where a **Your email has been verified. You can now sign in.** notice confirms the address is active.
### If sign-ups are capped
Cognee occasionally closes new sign-ups. When that happens a new account lands on **We're at capacity** — *Due to overwhelming demand, we are currently operating at full capacity. You have been added to our waitlist and we will notify you as soon as a spot opens up.* The only action on that page is **Sign out**; you are notified by email when a place frees up.
### Forgotten password
**Forgot Password** on the sign-in page asks for your email — *Enter your email and we'll send you instructions to reset your password.* — and **Send reset link** sends it, with **Back to sign in** to return. Repeated attempts are rate-limited (*Too many attempts. Please wait a moment and try again.*). Follow the emailed link to set a new password.
## Generate an API key
API keys authenticate requests from the [Cloud SDK](/cognee-cloud/connections/cloud-sdk), the [REST API](/api-reference/introduction), and [agent connections](/cognee-cloud/ui/integrations).
1. In the sidebar, open **API Keys**.
2. Click **Create new key**, then name the key in the **Create API key** modal.
3. Copy the key and store it securely. It is shown only once.
<Warning>
Treat API keys like passwords. Do not commit them to version control. Use environment variables (`COGNEE_API_KEY`) instead.
</Warning>
## Next steps
<CardGroup>
<Card title="Onboarding walkthrough" href="/cognee-cloud/quickstart" icon="play">
Upload your first file, build a knowledge graph, and run a search.
</Card>
<Card title="Cloud SDK" href="/cognee-cloud/connections/cloud-sdk" icon="terminal">
Interact with Cognee Cloud programmatically using the Python SDK.
</Card>
</CardGroup>
# Activity in the Cloud UI
Source: https://docs.cognee.ai/cognee-cloud/ui/activity
Every pipeline run, memory operation, and session in your workspace, filterable and exportable
The Activity page (route `/activity`) is the full log of what your workspace did: pipeline runs, memory operations, and sessions, with nested work indented under the operation that ran it. Reach it from **View full log →** on the [Overview](/cognee-cloud/ui/dashboard) Activity panel, or from any breakdown row on [Analytics](/cognee-cloud/ui/analytics).
The header states what the numbers mean: duration and tokens are recorded where available (older activity shows **—**), and cost is estimated. A **Refresh** button re-pulls the feed, a **last updated** readout beside it ticks (*just now*, *12s ago*, *4m ago*), and **Export CSV** writes every filtered row in the order shown — including the nested operations that **hide nested** is keeping off screen, so the file can hold more rows than the table does. It appends four columns the table has no room for — **Depth**, **Nested under**, **Counts toward total**, and **Session ID** — and the filename is stamped with the snapshot the rows came from (`cognee-activity-<timestamp>.csv`).
<Note>
Activity is a Cognee Cloud feature. In the local, open-source UI the route renders a notice — **Activity is a Cognee Cloud feature** — instead of the log. See [Local UI](/cognee-cloud/local-ui#cloud-only-pages).
</Note>
## Columns
| Column | What it shows |
| --------------- | ---------------------------------------------------------------------------------- |
| **Timestamp** | When the row happened. Nested rows are indented under their parent. |
| **Action** | `recall`, `remember`, `improve`, `forget`, or `unknown`, each with its own colour. |
| **Status** | Lifecycle state of the row — see below. |
| **Duration** | Elapsed time where the workspace recorded one. |
| **User** | The actor, with a stable coloured dot so distinct agents are easy to tell apart. |
| **Via** | The access channel the operation came through. |
| **Dataset** | The dataset the operation was scoped to. |
| **Run ID** | The run's identifier, searchable in the filter bar. |
| **Tokens** | Tokens recorded for the row. |
| **Cost (est.)** | Estimated spend in USD to four decimals. |
Rows carry inline chips where they add something the columns cannot: **bg** on the action of a background job (*accepted and started; this row is written once and never revisited*), **3 err** on the status of a session with errors recorded inside it, and **nested** on the timestamp of a row whose parent is not in the current view (*widen the range or clear a filter to see its parent*). A still-running row shows its elapsed time counting up rather than a fixed duration.
An **`unknown`** action is an operation name this UI has not been taught — a plugin installed on your pod can record its own operations. It deliberately reads as absent rather than being folded into one of the real actions.
### Statuses
Statuses are listed in lifecycle order, which is also the order the filter offers them and the order the **Status** column sorts by:
| Status | Colour | Meaning |
| ------------- | ------ | ---------------------------------------------------------------------------------------------------------- |
| **queued** | Slate | Accepted but not started. |
| **running** | Purple | In flight. |
| **accepted** | Cyan | Background work handed off — nothing updates this row again; the pipelines it launched carry the progress. |
| **completed** | Green | Finished. |
| **failed** | Red | Finished unsuccessfully. |
| **abandoned** | Amber | Computed as likely stuck rather than observed to have failed — a warning, not a verdict. |
| **unknown** | Grey | A state this UI has not been taught. |
## Filters
The filter bar sits above the table. Every filter is reflected in the URL, so a filtered view can be shared, bookmarked, and reloaded.
Always visible:
* **date** — condition-first: pick a comparison (**is**, **is between**, **is on**, **is before**, **is after**, **is on or before**, **is on or after**), then either a one-click preset (**Today**, **Yesterday**, **This week**, **Past week**, **This month**, **Past month**, **This year**, **Past year**) or a real date. **is between** takes two dates.
* **action**, **status**, **user** — multi-select lists, built from the values that actually appear in the loaded rows rather than from a fixed enum. Each can include or exclude: **is any of** / **is none of**.
* **hide nested** (on by default) — hides operations nested under a parent. Their tokens stay in the total, counted under that parent.
* **hide 0 tokens** (on by default) — hides rows with nothing to show: those that measured zero tokens, and queued runs that never started. Rows with *no* token data recorded stay, since those are unmeasured rather than free.
Behind **+ More filters**: **via** and **dataset** (both with the same include/exclude operators), **Run ID** (**contains** / **does not contain**), and range filters for **tokens** and **cost**. Setting either bound drops every row with no recorded value — the panel warns *Rows with no recorded tokens are excluded* — which is why the separate **hide 0 tokens** toggle exists for the measured-zero case. The extra row stays open on its own whenever one of those filters is active.
**Clear filters** appears whenever any filter is set, and **Reset sort** whenever the table is sorted away from its default. Click a column header to sort by it.
<Note>
Filters run in the browser over the rows already loaded. A date far enough back can therefore match nothing even where that activity exists — the empty state says so, and offers **Clear all filters**.
</Note>
## What the feed loads
The page polls every 30 seconds and loads the newest **200 sessions** and **500 runs and memory operations**. When a workspace holds more than that, an amber-ruled **Partial view.** notice above the table names exactly what was kept — *the newest 200 of 3,412 sessions and the newest 500 runs and memory operations* — and repeats that the filters run over what is loaded, so a filtered view can miss older matches. **Export CSV** carries the same caveat in its tooltip.
Rows are added to the DOM 150 at a time, so a workspace with hundreds of rows stays responsive without the count being hidden from you.
A totals row is pinned to the bottom of the table and always reachable. It reads **Total · 1,284 events** with the summed duration, tokens, and cost beneath their columns, and reports separately what those sums leave out — *· 96 unmeasured, excluded below* and the nested rows that are hidden and counted inside their parents.
## When the feed fails
Failure notices sit above the table rather than replacing the rows, so a failed refresh never blanks data you were reading:
| Notice | When |
| ------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------- |
| *Couldn't refresh activity — showing the last data loaded.* | A refetch failed but earlier rows are in hand |
| *Couldn't load activity. If the workspace was asleep it may still be starting up; otherwise the activity endpoint is not answering.* | Nothing loaded at all |
| *Sessions couldn't be loaded, so recall activity from before your workspace started recording operations is missing.* | Only the sessions half failed |
## Empty state
With no activity at all, the page reads **No activity recorded yet.** — *Activity appears here once an agent reads from or writes to your memory. Add data or connect an agent to get the first rows.* — with **Add data** and **Connect an agent** actions.
If the filters match nothing, it reads **No activity matches these filters.** instead, with the caveat above and a **Clear all filters** action.
## Opening a run
A row that belongs to a session is clickable — its timestamp is a button labelled *Open session for at * — and it opens that run in [Sessions](/cognee-cloud/ui/sessions). Rows with no session behind them are not clickable.
<Note>
Looking for one agent conversation rather than the whole workspace? [Sessions](/cognee-cloud/ui/sessions) shows a single run's transcript, tool calls, and feedback. Looking for where the tokens went rather than what ran? See [Analytics](/cognee-cloud/ui/analytics).
</Note>
# Analytics in the Cloud UI
Source: https://docs.cognee.ai/cognee-cloud/ui/analytics
Where your tokens went, broken down by dataset, access channel, action, and agent
The Analytics page (route `/analytics`) answers one question: where your tokens went. It breaks the workspace's memory operations down by dataset, access channel, action, and agent, and every breakdown row links through to the operations behind it on [Activity](/cognee-cloud/ui/activity).
Tokens are what the workspace actually recorded; cost is estimated from the gateway rate. Open it from **View breakdown →** on the [Overview](/cognee-cloud/ui/dashboard)'s Cost Savings panel, which passes its current range through.
<Note>
Analytics is a Cognee Cloud feature, and was called **Cost Breakdown** before the September 2026 release. In the local, open-source UI the route renders a notice instead of the breakdown — see [Local UI](/cognee-cloud/local-ui#cloud-only-pages).
</Note>
## Time range
A segmented control in the header scopes the whole page:
| Range | Window |
| ------- | ------------------- |
| **24h** | The last 24 hours |
| **7d** | The last 7 days |
| **30d** | The last 30 days |
| **All** | All loaded activity |
Beside it, a **last updated** readout, a **Refresh** button, and **Export CSV**, which writes every breakdown bucket shown, in the order shown.
## Headline figures
Three figures lead the page:
* **Tokens** — routed through memory over the range.
* **Estimated cost** — at the gateway rate.
* **Operations** — its hint reads *recalls, writes and pipeline runs*, replaced by *plus 412 nested inside them* when there is nested work.
A dash means *not measured*, which is not the same as zero: when no token usage was recorded for the operations in range, both **Tokens** and **Estimated cost** read **—** and their tooltips say *No token usage was recorded for these operations*. A measured zero renders as `0`.
Under the figures, measurement caveats spell out what the numbers do not cover. The token-data caveat appears only when *nothing* in range was measured (*None of these 412 operations report token usage, so no total can be given*); a breakdown bucket that is only partly measured is flagged with an asterisk and a footnote saying the figure is a floor. Rows with no usable date are reported separately.
## Trend
A **Tokens over time** panel — with a lighter *per hour* / *per day* / *per week* beside the heading — plots the range as bars on local calendar boundaries, so the busiest day does not move depending on when you look. **24h** buckets by hour, **7d** and **30d** by day, and **All** takes the finest unit that fits in 30 bars, stepping up through hour, day, week, month, and year.
The final bar is always the period the clock is currently inside. It is drawn with a diagonal hatch and reads *(so far)* on hover, because it is not comparable with the completed bars beside it. The same data is available to screen readers as a table with **Period**, **Tokens**, and **Operations** columns.
## Breakdowns
Four tables split the same totals four ways:
| Table | Groups by |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **By dataset** | The dataset each operation was scoped to |
| **By access channel** | How the operation reached Cognee |
| **By action** | `recall`, `remember`, `improve`, `forget`, and `unknown` for an operation name this UI has not been taught (see [Activity](/cognee-cloud/ui/activity#columns)) |
| **By agent** | The actor that ran it |
Each row carries the bucket name, its **Share**, **Tokens**, **Est. cost**, and **Ops**. The share percentage is the bucket's slice of the page total (shown as `<0.1%` below a tenth of a percent), while the bar beside it is scaled to the largest bucket in that table — so the bars compare rows within a table, and the percentages compare across the page. Rows are ordered by tokens, and each table shows only the eight largest buckets with a **Show all 23** / **Show fewer** control underneath. Clicking a row opens [Activity](/cognee-cloud/ui/activity) filtered to that bucket.
Two exceptions to that link:
* Under **By access channel**, operations that record no channel are grouped as unattributed. Activity has no filter option for "absent", so this one bucket does not link anywhere.
* A dimension whose names have not resolved yet is dropped rather than shown, since its links would select nothing.
## Notices and empty state
Notices sit *above* the figures rather than replacing them: a failed refetch leaves the previous numbers on screen with a warning, instead of blanking figures you were reading. Each notice says only what it knows — a load error, a sessions-only failure, or a feed capped at one page (in which case the breakdowns describe the newest slice, not the whole history).
When nothing failed and there is genuinely nothing to show, the page reads **No memory operations recorded in the last 7 days. Try a wider range.** — or, on **All**, *Connect an agent and its recall and ingestion will show up here.*
# Cognee API Keys
Source: https://docs.cognee.ai/cognee-cloud/ui/api-keys
Create and manage API keys for SDK and API access
The API Keys page lets you create and manage keys used to authenticate with the [Cloud SDK](/cognee-cloud/connections/cloud-sdk), the [REST API](/api-reference/introduction), and [agent connections](/cognee-cloud/ui/integrations).
## Create a key
<Steps>
<Step title="Open the create modal">
Click **Create new key**. A modal opens with a name input (placeholder **e.g. Production, CI/CD, Local Dev...**).
</Step>
<Step title="Name and create the key">
Enter a name and confirm. This calls **`POST /api-keys`** with `{name}`.
</Step>
<Step title="Copy the key">
The new key is shown in full exactly once. Copy it immediately — afterwards only a masked label is displayed.
</Step>
</Steps>
<Note>
If the workspace has no active subscription, a banner sits at the top of the page — **No active subscription** / *Subscribe to unlock data uploads, search, and all features.* — with an **Upgrade** button that opens the [plan chooser](/cognee-cloud/functionality/account-and-billing#plans). The same banner appears on [Search](/cognee-cloud/ui/search).
In cloud mode, the base URL shows your tenant URL once provisioned; while the workspace is still provisioning the field shows a loading placeholder and the copy button is hidden.
</Note>
## Connection Details
The **Connection Details** card surfaces the values your clients need, each copyable:
| Field | Description |
| ------------ | --------------------------------------------------------- |
| API Base URL | Your tenant URL, e.g. `https://your-tenant.aws.cognee.ai` |
| Tenant ID | Your tenant identifier |
| User ID | Your user identifier, from **`GET /api-keys/my-user-id`** |
It also shows an auth-header hint: `X-Api-Key: <your-api-key> • X-Tenant-Id: <tenant>`. Both the **X-Api-Key** and **X-Tenant-Id** headers are used by API clients and MCP.
## Manage keys
The page lists all active keys with their creation date (**`GET /api-keys`**). To revoke a key, click the delete icon next to it (**`DELETE /api-keys/{id}`**). Revoked keys stop working immediately.
Every key also records a creation time and a last-used time server-side. The last-used timestamp is refreshed when the key successfully authenticates a request, throttled to at most one write every five minutes, so a busy key's "last seen" can lag by a few minutes. Recording it never blocks authentication: if the write fails, the request still succeeds.
## Plugin-provisioned keys
Keys for a [per-plugin agent identity](/cognee-cloud/connections/managing-connections#per-plugin-agent-identities) are not created from the **Create new key** modal. They are minted by **`POST /api/v1/integrations/plugins/{plugin_key}/provision`**, named after the plugin key (`claude-code`, `codex`, …), and belong to the plugin's agent sub-user rather than to you — so they are listed under that agent, not on this page.
They follow the same show-once rule as keys created here: the provision response is the only place the full key appears. Calling provision again rotates the key and revokes the plugin agent's earlier keys, and disconnecting the plugin (**`DELETE /api/v1/integrations/plugins/{plugin_key}`**) revokes all of them.
## API reference
Two reference links are available on the page:
* **API Reference** — the shared Swagger docs at `api.aws.cognee.ai/docs`.
* **API Tenant Reference** — your tenant instance docs at `https://your-tenant.aws.cognee.ai/docs`.
<Warning>
Store API keys in environment variables or a secrets manager. Do not hardcode them in source files.
</Warning>
## Usage
Pass the API key in requests:
**Python SDK:**
```python theme={null}
import cognee
await cognee.serve(
url="https://your-tenant.aws.cognee.ai",
api_key="your-key"
)
```
**REST API:**
```bash theme={null}
curl -H "X-Api-Key: your-key" https://your-tenant.aws.cognee.ai/api/v1/datasets/
```
See [Cloud SDK](/cognee-cloud/connections/cloud-sdk) for the full setup guide.
# Workspace Overview
Source: https://docs.cognee.ai/cognee-cloud/ui/dashboard
Balance, spend, memory coverage, and recent activity for your workspace
The Overview is the landing page after login (route `/dashboard`, under the **DATA** section of the sidebar). It shows what your memory is doing right now: what it cost, how well it answers, and what ran most recently. A user who has never completed onboarding is sent to the [onboarding flow](/cognee-cloud/quickstart) first — it is shown once per user and never re-forced, whatever the workspace activity. While the workspace provisions, a skeleton is shown, and a **Your workspace is ready** notice appears once it is ready.
<Note>
The sidebar label for this page is **Overview**, while the route remains `/dashboard` and it appears as **Dashboard** in analytics.
</Note>
At the top you see a time-based greeting (**Good morning**, **Good afternoon**, or **Good evening**) with your name, and — when an agent is selected — a lavender chip naming its type.
## Get started
**Get started** is a collapsed strip under the greeting — below the [credit banner](#credit-banners) when one is showing — carrying the subtitle **Connect your AI agents to give them persistent memory** and a count of the agents that have ever connected — *3 connectors*, or *1 connector* in the singular. It counts lifetime registrations, not agents currently running. Use **Expand ▾** / **Collapse ▴** to open it.
Expanded, it presents the connection cards:
| Card | What it does |
| ------------------- | --------------------------- |
| **Claude Code** | Memory for every project |
| **Codex** | Wire Codex to your graph |
| **Openclaw** | Connect via `AGENTS.md` |
| **API / MCP** | Via REST API or MCP |
| **Company Dataset** | Upload docs to build memory |
Clicking an agent card opens a **Connect ** modal with the steps for that tool, and a **Mac** / **Windows** toggle that rewrites them — see [Mac / Windows](/cognee-cloud/ui/integrations#mac--windows). The **Claude Code** and **Codex** flows write your credentials to `~/.cognee/.env`; the **Openclaw** and **API / MCP** flows print environment-variable commands to run in the terminal you will start the agent from — `export …` on Mac, `$env:… = …` in PowerShell on Windows — and tell you to add them to your shell profile to keep them across terminals. The **Company Dataset** card opens the file picker instead, and uploads straight into a dataset without leaving the page.
Once files are processed, a **Knowledge graph built** modal offers next steps: **Search your data**, **Inspect the knowledge graph** (the dataset in [Datasets](/cognee-cloud/ui/datasets)), and **Explore the knowledge graph** (the [Mindmap](/cognee-cloud/ui/knowledge-graph)). Which dataset the files land in depends only on how many you have: with more than one, an **Upload to which dataset?** picker always appears first; with exactly one, they go straight into it; with none, a dataset named `default_dataset` is created for them. Accepted file types are `.pdf`, `.csv`, `.txt`, `.md`, `.json`, and `.docx`.
## Overview panels
Below the strip, an **Overview** heading (*Balance, spend, and usage across your workspace*) introduces four panels: the memory flow diagram, **Cost Savings**, **Memory Coverage**, and **Activity**.
### Memory flow
A framed diagram shows how memory moves through your workspace: **Data sources** on the left, the **COGNEE MEMORY** core in the middle, and **Agents** on the right, wired together with spokes. A live badge on the core's stroke reflects workspace health. Each element navigates:
| Click | Goes to |
| ---------------------- | --------------------------------------------- |
| The memory core | [Mindmap](/cognee-cloud/ui/knowledge-graph) |
| A source or agent node | [Integrations](/cognee-cloud/ui/integrations) |
A **Teams** row with team cards and a **+ Create a new team** button is built but not yet shown: the whole row is hidden until real team data is wired in, rather than showing an empty grid.
### Cost Savings
The **Cost Savings** panel leads with the tokens your memory saved: a large figure and the label **tokens saved**, with a sub-line reading *\$X less than the same work at Opus 5 list prices · estimated*. The estimate assumes the same work would consume a multiple of the tokens without Cognee's recall, priced at the comparison model's list rate, against what Cognee actually cost at the gateway rate. Tokens are the honest unit here — on usage-based billing they convert to cash, on a subscription they are limit headroom left unspent.
The panel header carries your **Balance** in USD and a range toggle (**24h**, **7d**, **30d**) that scopes only this panel. The chart plots one series, keyed **what cognee cost**. **View breakdown →** opens [Analytics](/cognee-cloud/ui/analytics) carrying the same range.
Three honesty details worth knowing:
* The headline reads **—**, not `0`, until the activity feed has measured tokens for the range. A genuine zero renders as `0`.
* When neither billing nor the feed has priced the range at all, the sub-line reads *No spend priced for this range yet* instead of a dollar figure.
* When the feed is capped at one page, an amber **at least** flag appears beside the headline: the token figure counts the newest slice only, while the dollar line comes from billing and is not capped the same way.
### Memory Coverage
The **Memory Coverage** panel is a glance view of [Memory Coverage](/cognee-cloud/ui/memory-coverage), which replays the questions your workspace was really asked and scores how well memory answered them. The panel shows **Real recall** as a percentage over a full-width meter, then **Coverage by dataset** as one lavender hairline per dataset with its score.
Each percentage is a dataset's coverage score out of 5 expressed as a percentage, and **Real recall** is the mean of those across datasets that actually have a score — a dataset nobody has scored is left out rather than counted as zero. These roll-ups exclude the **Other** topic, so a dataset's figure here can differ from the score on the Memory Coverage page itself; [The score](/cognee-cloud/ui/memory-coverage#the-score) explains why. Its header meta line reads **LLM-as-judge**, or **LLM-as-judge · loading**, **· couldn't load**, or **· not scored yet**.
At most four datasets are listed; with more, a line reads *Showing 4 of 11 datasets. Real recall averages every dataset that has a score.* — so the headline covers more datasets than the rows do. Click a dataset row to open Memory Coverage scoped to it, or use the footer's **View analysis →** for the full page and **Upload data** for the same file picker as the Company Dataset card.
The panel says which of three situations you are in: with no datasets at all, *Add data to see coverage by dataset*; with datasets but no scores, *Not scored yet. Run an analysis to see coverage per dataset.*; and on a failed fetch, *Couldn't load coverage.* with a retry in place rather than sending you to the other page to find out.
### Activity
The **Activity** panel is a glance over the same rows as the full [Activity](/cognee-cloud/ui/activity) log, newest first, with six columns: **Timestamp**, **Action**, **Status**, **Duration**, **User**, and **Cost (est.)**. Its header summarizes the feed in at most three facts — operation count, failures, runs in flight, and total estimated cost, for example *1,284 operations · 3 failed · \$0.4210*.
The glance lists the newest **eight** top-level operations, and clicking a row opens that run's session in [Sessions](/cognee-cloud/ui/sessions). Under the table, **View full log →** opens the Activity page, and a quiet line on the right says what the glance is *not* showing — how many more rows exist, how many nested steps are folded in, and how many rows the default filters dropped. When every loaded row measured zero tokens or is still queued, the panel says so outright rather than looking idle.
While the feed loads the panel header reads *loading…*; if it fails the header reads *couldn't load* in red and the table is replaced by an explanation rather than an empty state.
With no activity at all it reads **No activity recorded yet.** — *Rows appear here as soon as an agent reads from or writes to your memory* — with a **Connect an agent** action.
## Credit banners
A single red banner appears between the greeting and the **Get started** strip when your workspace credit balance falls below **$5**: **Your workspace credit balance is $0.42. Agent requests may fail.** At a balance of zero or less the wording hardens to **Agent requests will fail.** Workspace owners get a **Top up credits →** link to [Billing](/cognee-cloud/functionality/account-and-billing); everyone else sees **Ask the workspace owner to top up.** The banner can be dismissed with the **✕**.
<Note>
The old credit-usage banner at ≥90% spent is gone, and the promotional voucher banner no longer appears on the Overview — the low-balance banner is the only credit banner on this page. Both remain cloud-only; neither shows in the local, open-source UI.
</Note>
A run that actually fails for want of credits is reported separately from this banner — see [When credits run out](/cognee-cloud/functionality/account-and-billing#when-credits-run-out).
## While the workspace is unavailable
The Overview stands in for the workspace itself when the workspace is not ready:
* **Waking** — a workspace that scaled to zero shows a **Waking ** screen instead of the panels. See [Workspace sleep and wake](/cognee-cloud/overview#workspace-sleep-and-wake).
* **Unreachable** — when the pod cannot be reached, an amber card reads **We're having trouble reaching your workspace** (or ** didn't wake up** after a wake attempt), with a retry and the suggestion to sign out and back in.
* **Partial telemetry** — if workspace activity fails to load, a red line under the panels reads *Couldn't load workspace activity, so the panels above may be incomplete or empty. We'll keep trying in the background.* If only sessions fail, an amber line says that activity from before your workspace started recording operations is missing from these panels. After repeated failed polls a notification also appears once per episode — **Workspace is having trouble responding** — and a faint *refreshing telemetry…* line shows while a refetch is in flight. The panels keep their last-known figures rather than blanking.
## Sidebar
The sidebar is shared by every page in the app and can be collapsed to an icon-only rail on desktop. Hover over the sidebar and click the chevron handle on its right edge (**Collapse sidebar** / **Expand sidebar**) to toggle it. The sidebar's three sections are **DATA**, **EXPLORE**, and **CONNECT**, in that order. Collapsed, the nav labels become icons with hover tooltips, each section heading becomes a thin divider rule, and the **Billing / Pricing** button becomes a card icon.
The choice is stored in a cookie for a year and read on the server, so a collapsed sidebar renders collapsed on the first paint after a reload instead of animating shut. Collapsing applies to desktop only — on mobile the sidebar remains a full-width drawer with labels, opened from the hamburger button.
## Help menu
A **?** button in the top bar, on every page, opens a help menu:
| Item | What it does |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Onboarding** | Re-enters the [onboarding flow](/cognee-cloud/quickstart) |
| **Extraction Settings** | Opens a modal for **Chunk Size** (*Characters per document chunk during ingestion*), **Chunks Per Batch** (*Chunks processed in parallel during cognification*), **Top-K Results** (*Maximum results returned per search query*), and **Source references** |
| **Docs** | Opens this documentation |
| **Discord community** | Opens the Cognee Discord |
| **Keyboard shortcuts** | Display only — it shows a <kbd>⌘ /</kbd> hint, and nothing is bound to that key yet |
| **System status** | Display only — a label with a green dot; it does not open a status page |
| **What's new** | A short changelog of recent releases, each with a date |
## Give feedback
The sidebar footer holds three controls on every page, in order: **Give feedback**, **Book a call** (opens a Calendly booking page in a new tab), and **Billing / Pricing**. The whole footer is cloud-only — it is not part of the local, open-source UI.
Clicking **Give feedback** opens a short form; submitting sends your message straight to the Cognee team, tagged with your current workspace and the page you were on for context. You must be signed in to send feedback.
### Pulse survey
Cognee also asks for feedback on its own once, through a small **Quick question** card in the bottom-right corner of the app. It is not a modal — the rest of the page stays usable — and you can close it with the **✕** button or the <kbd>Esc</kbd> key (Escape is ignored while you are typing an answer).
The card is offered at most once per user, and only once your account has some tenure behind it: it is at least **15 days** old, or it predates onboarding entirely. Creating a dataset is only an occasion to run that check sooner — it is never a way around it, so a brand-new signup is not asked on day one. The card appears on whichever page you happen to be on. Like the credit banners above, the pulse survey appears only in Cognee Cloud — it is not part of the local, open-source UI.
<Steps>
<Step title="Score">
**How likely are you to recommend Cognee to a colleague?** — pick **0** (*Not likely*) to **10** (*Extremely likely*).
</Step>
<Step title="Follow-up">
One optional free-text question, chosen from your score: **What's the main reason for that score?** (0–6), **What would make this a 10 for you?** (7–8), or **What do you value most about Cognee?** (9–10). The 9–10 question also offers an **OK to quote this publicly (first name only)?** checkbox. Use **← Back** to change your score, **Skip** to send only the score, or **Send** to submit both. Going back and changing the score clears any answer and quote consent you had already given, so a response is never submitted against a question you did not see.
</Step>
<Step title="Thanks">
A short confirmation — **Your score and note are on their way to the team.** — closes itself after a couple of seconds.
</Step>
</Steps>
# Datasets
Source: https://docs.cognee.ai/cognee-cloud/ui/datasets
Upload documents to build searchable knowledge graphs.
The **Datasets** page (route `/datasets`, under the **DATA** section of the sidebar) lists all the datasets in your workspace. A dataset is a container for documents and all subsequent operations. See [Datasets](/core-concepts/further-concepts/datasets) for the underlying concept.
<Note>
This page and the sidebar item were called **Brain** until the September 2026 release; every surface now says **Dataset**. Analytics events are the one exception — they still record this page as `Brains`, so historical dashboards keep working.
</Note>
## Dataset list
The page uses a Finder-style two-column layout:
* The **left column** lists your datasets. Each dataset shows a **status dot**, its **document count**, and **Share** and **Delete** actions.
* The **right column** lists the documents in the currently selected dataset.
Select a dataset in the left column to load its documents on the right.
### Status dots
The status dot reflects the dataset's processing state:
| State | Color |
| ----------------------------- | -------- |
| Pending / Running | Amber |
| Completed | Green |
| Failed | Red |
| Failed — insufficient credits | Lavender |
| Empty / Loading | Gray |
A dataset whose graph is **outdated** (its config changed after files were processed) also shows an amber dot.
A build that ran out of workspace credits part-way through is reported separately from a generic failure: the dot is lavender and labelled **Failed — insufficient credits**, with the hint *"Your workspace ran out of credits mid-run — top up on the billing page"*. The same distinction appears on the dataset detail status pill, where the label links straight to [Billing](/cognee-cloud/functionality/account-and-billing). The [Mindmap](/cognee-cloud/ui/knowledge-graph) carries no status chip of its own — it draws whatever graph exists and says nothing about how the last build ended.
## Create a dataset
Click **New dataset** to open the **Create dataset** modal, enter a name, and click **Create dataset**. The field uses the placeholder **e.g. product-docs, sec-filings...** and explains itself under the input: *Spaces become hyphens as you type — periods aren't allowed. Saved as lowercase.*
Spaces are not rejected, they are converted: each one is replaced with a hyphen as you type. Periods are the only character blocked outright — type one and the input turns red with **Dataset name cannot contain periods.**, and **Create dataset** stays disabled until you remove it. The name is lowercased on create.
In a cloud (multi-tenant) workspace, newly created datasets are automatically granted tenant-level `read` and `write` permissions, so every member of the current tenant can see and write to them. To restrict a dataset to specific users, revoke the tenant grant and assign per-user permissions through the [permissions API](/cognee-cloud/functionality/permissions-and-access-control#dataset-permissions). The grant is best-effort: if it fails, the dataset is still created — it is just not shared with the workspace yet.
In [local self-hosted mode](/cognee-cloud/local-ui) that grant is skipped entirely. There the active tenant is the `local` sentinel rather than a real tenant UUID, and there are no other members to share with, so the UI makes no tenant-permission call at all. The dataset is created normally and you keep full owner access to it.
## Manage documents
The documents column header includes two actions:
* **Add files** — Browse and upload files. Supported formats: PDF, CSV, TXT, Markdown, JSON, DOCX.
* **Paste text** — Open a textarea modal and paste raw text. The text is saved as `pasted-text-{timestamp}.txt`.
You can also drag files anywhere onto the documents column — a **Drop to upload** overlay appears while dragging.
You can upload at most **100 files in a single batch**. Selecting more shows **You selected files. Please upload 100 or fewer at a time.** and the upload is not started.
If the estimated cost of the selected files reaches your workspace credit balance, a **This upload might use more credits than you have** dialog appears before the upload is sent, offering **Cancel** and **Top up first** — the upload does not proceed until the balance covers it. See [When credits run out](/cognee-cloud/functionality/account-and-billing#when-credits-run-out) for how the estimate is framed.
Each document row shows a **file-type badge** (PDF, DOC, MD, TXT, CSV, JSON), the file **size**, the upload **date**, and a **Delete** button (confirm dialog **Delete document**).
Uploaded files appear in the document list automatically — no manual page refresh is needed. Because the background ingestion step can save document records a few moments after the upload call returns, the page keeps polling for the new documents and updates the list as soon as they are available.
After upload, the [add](/core-concepts/main-operations/legacy-operations/add) and [cognify](/core-concepts/main-operations/legacy-operations/cognify) pipeline runs in the background. Once the files are saved, the upload is treated as successful — a later failure in status polling or the background graph build is reported separately and no longer surfaces as a false **Upload failed** error.
## Refresh and polling
Click **Refresh** to re-poll dataset statuses on demand. The page also polls automatically every 5 seconds while any dataset is in a **pending** or **running** state, updating the status dots as processing completes.
## Dataset detail
The dataset detail page (route `/datasets/[id]`) shows a single dataset. The header displays the dataset name (with a **Default** badge for `default_dataset`), the document count, and a **status pill**:
| Pill | Meaning |
| ----------------------------- | ------------------------------------------------------------------------------ |
| Processing | Pipeline is pending or running |
| Failed | Processing failed |
| Failed — insufficient credits | Processing stopped because the workspace ran out of credits; links to Billing |
| Outdated | Config changed after files were processed; click **Rebuild graph** to clear it |
| Ready | Graph built and current |
| Empty | No documents yet |
Header actions:
* **Sync** — Re-runs cognify. Appears only for connected (integration) sources.
* **Delete** — Removes the dataset. Hidden for `default_dataset`.
* **Share** — Manage access.
* **Upload files** — Add documents.
When a knowledge-graph build fails, the dataset shows **Building the knowledge graph failed. Your files are still here — you can retry the build.** with a **Retry build** action, so a failed build never loses your uploaded documents. The same retry is offered from the failed-dataset banner on the [Dataset list](#dataset-list).
When the build failed specifically because the workspace ran out of credits, the banner instead reads **Building the knowledge graph failed — your workspace ran out of credits mid-run. Your files are still here.** and offers **Go to billing** rather than a retry, because retrying on an empty balance fails the same way.
## Extraction settings
How Cognee extracts knowledge from a dataset is configured on its detail page, under the **Memory customization** heading: a **Graph Model**, a **Prompt**, and an **Ontology**, each a dropdown defaulting to **Automatic** and each carrying an info tooltip explaining what it controls. The settings are per dataset. For the API equivalents, see [Configuration & Ontologies](/cognee-cloud/functionality/configuration-and-ontologies).
<Warning>
Changing the model, prompt, or ontology once the dataset already has files marks it **Outdated**, and the page says why: *Knowledge graph is outdated. The graph model was changed since the last build.* Click **Rebuild graph** to re-run cognify with the current settings and clear the state. The same **Outdated** state appears on the [dataset list](#status-dots) and on the [status pill](#dataset-detail).
</Warning>
### Graph model
The **Graph Model** dropdown chooses which entity types and relationships Cognee extracts:
* **Automatic** (marked *Default*) — Cognee infers the structure from your data.
* A **saved model** — one you created and named. Each row carries a pencil that opens it in the editor.
**Create new** opens the **Create Graph Model** modal with two starting points:
* **Infer from data** — Cognee analyses the dataset's files and proposes entity types and relationships. The option reads *Analyze files to suggest a schema*, and is disabled with *No files in this dataset yet* on an empty dataset.
* **Start blank** — define the entity types and relationships yourself.
Either way the model is created as * Schema* and opens straight into the editor.
### Graph Model editor
The Graph Model editor is the visual node editor for a model. It has no sidebar entry and opens full-screen at `/graph-models/[id]`, reachable only from the **Graph Model** dropdown above. Visiting `/graph-models` directly bounces you back to [Datasets](/cognee-cloud/ui/datasets).
The canvas draws each entity type as a node. **Add entity type** sits at the top left, and a hint bar reads *Click a type to edit · Drag to reposition*. Selecting a node opens a right sidebar carrying that type's **Name**, **Description**, and **Fields ()**.
A field is one of:
| Type | What it holds |
| ---------- | --------------------------------------------------------------------------------- |
| String | Text value |
| Number | Numeric value |
| Boolean | True/false value |
| Date | Date value |
| Relation → | A link to another entity type; choose the target from **Select target entity...** |
A relation added here is always many-valued — the editor has no cardinality control. Single-valued relations come only from an inferred schema. The canvas labels each relation edge with the field name and its cardinality.
The header carries the model name (double-click to rename), a ** types, relationships** count, an amber **Unsaved** flag while edits are pending, and **Delete**, **Regenerate**, and **Save**.
**Regenerate** opens the **Regenerate Schema** modal — *Select a dataset and files to analyze. Cognee will infer entity types and relationships from the selected files.* Pick a dataset, choose which of its files to use (all are selected for you), and confirm with **Regenerate from files**. A **Schema regenerated** notification reports *Detected entity types from files.*
<Warning>
Regenerating replaces every entity type on the canvas rather than merging into what is there. The result is left unsaved, so review it before clicking **Save**.
</Warning>
### Prompt
The **Prompt** dropdown chooses the extraction prompt: **Automatic** (marked *Default*), or a named prompt of your own. **Create new** opens **Create Custom Prompt** with two starting points:
* **Generate from graph model** — seeds the prompt from the currently selected graph model. It stays disabled until one is selected, reading *Select a graph model first*.
* **Start blank** — write your own extraction prompt.
The pencil on a saved prompt opens the **Edit Prompt** modal, which holds its **Name** and **Prompt** text, a bin to delete it, and **Save prompt**.
### Ontology
The **Ontology** dropdown attaches an optional OWL ontology to guide extraction: **Automatic** for none, or one you have uploaded — those are listed by filename, each with a bin to delete it.
**Upload new** opens **Upload Ontology** — *Upload an OWL ontology file to guide how Cognee structures your knowledge graph.* It takes a **Key** (required, placeholder *e.g. biomedical-ontology*), an **OWL File** (required, `.owl` only), and an optional **Description**.
## Sync a dataset
For datasets connected to an external (integration) source, the dataset detail header includes a **Sync** button that re-runs the cognify pipeline against the current contents. Use this to rebuild the knowledge graph after source data changes without uploading new files again. Datasets without a connected source do not show this button.
## Share a dataset
Use the **Share** action (on a dataset in the left column or in the dataset detail header) to open the **Share dataset** modal. From here you can:
* **Share with your whole workspace** — Grant access to **Everyone in workspace**, which covers all current *and future* members. Choose **Can edit** (read and write — members can query the dataset and run cognify on it) or **Can view** (read-only). This grants the permission to the workspace (tenant) principal, so newly added members are included automatically without any per-member step.
* **Share with individual agents and users** — Grant **read-only** access to a specific agent or user from the list.
Sharing is applied immediately for the current session. The modal does not yet list shares granted previously; to manage grants in detail, use the [permissions API](/cognee-cloud/functionality/permissions-and-access-control#dataset-permissions).
## Dataset selector
The dataset selector is an in-page control, not part of the breadcrumb — the top bar carries only the workspace switcher and the page name. On [Search](/cognee-cloud/ui/search) it sits above the input and scopes the query to exactly one dataset — there is no **All datasets** option there, and if you have not picked one, the first dataset in the list is used.
Pages that scope to one dataset but keep their own switcher rather than the breadcrumb one:
* [Mindmap](/cognee-cloud/ui/knowledge-graph) — a **dataset** switcher above the canvas; it visualizes one dataset's graph at a time.
* [Memory Coverage](/cognee-cloud/ui/memory-coverage) — a dataset switcher under the score, since a coverage run is always scoped to a single dataset.
* [Activity](/cognee-cloud/ui/activity) — a **dataset** filter behind **+ More filters** rather than a selector, so you can compare several at once. [Analytics](/cognee-cloud/ui/analytics) has no dataset control at all; it always covers the whole workspace and splits it in the **By dataset** breakdown.
## Empty state
When you have no datasets yet, the page shows **No datasets yet** with a **Create dataset** call to action.
If the dataset list *fails to load* (for example, while a large upload is still processing and the server is briefly unreachable), the page shows a distinct error state — **Couldn't load your datasets** with **Your datasets are safe — we just couldn't reach the server** and a **Retry** button — rather than the empty **No datasets yet** state. A failed fetch is never mistaken for a genuinely empty account.
## Delete a dataset
Use the **Delete** action on a dataset (in the left column or the dataset detail header). The confirm dialog warns that this will **permanently remove the dataset and all its files. This action cannot be undone.**
The `default_dataset` dataset cannot be deleted: it has no **Delete** button and shows a **Default** badge on its detail page.
<Note>
Dataset operations are also available through the [REST API](/cognee-cloud/functionality/dataset-management) and the [Cloud SDK](/cognee-cloud/connections/cloud-sdk).
</Note>
# Integrations in the Cloud UI
Source: https://docs.cognee.ai/cognee-cloud/ui/integrations
Connect AI agents, coding tools, and automation platforms to Cognee memory
The Integrations page (route `/integrations`, under the **CONNECT** section of the sidebar alongside **API Keys**) is the in-product hub for wiring external tools into your Cognee memory. Each card opens a step-by-step modal that injects your tenant's live base URL and API key into copy-on-click code blocks, so the snippets are ready to paste without editing.
Everything in these flows runs locally on your machine — nothing is sent to Cognee. One step backs up what it touches: the OpenClaw flow copies an existing `AGENTS.md` to `AGENTS.md.bak` before writing, and tells you to merge it back by hand. The credentials step writes `~/.cognee/.env` without a backup — re-running it replaces the values rather than stacking duplicates.
Each agent card carries a connection pill: a green **Connected** once a session from that agent has reached your workspace, or a grey **Not connected yet** before that. Only **Claude Code** and **Codex** report a detection signal, so the other cards stay neutral rather than claiming a state they cannot observe.
The page has four sections, in order: **Agents** ("Connect your AI agents and coding tools to Cognee for persistent memory"), **Automation platforms** ("Give your automation workflows access to Cognee memory via MCP"), **Data sources** ("Connect the tools your team already uses to your datasets. One connection per workspace, shared with everyone."), and **More data sources** ("Not live yet — tell us which ones to prioritize.").
## Mac / Windows
A **Mac / Windows** toggle sits in the top-right corner of three surfaces, and defaults to **Mac** everywhere:
* The setup modals on this page, on the four OS-aware cards: **Claude Code**, **Codex**, **OpenClaw**, and **API / MCP**.
* The connect modal that opens from a **Claude Code**, **Codex**, **Openclaw**, or **API / MCP** card on the [dashboard](/cognee-cloud/ui/dashboard). The toggle is inside the modal, not on the card. The **Company Dataset** card opens the document upload flow instead, which has no toggle.
* The agent connection steps in [onboarding](/cognee-cloud/quickstart).
Flipping it to **Windows** rewrites the commands shown on screen, the commands copied to your clipboard, and the step text that says where to run them:
| What changes | Mac / Linux | Windows (PowerShell) |
| -------------------------------------------------------------------- | -------------------------------------------- | ---------------------------------------------------------------- |
| Where to run it (step title and copy) | "Open **Terminal**" | "Open **PowerShell**" |
| Plugin credentials — Claude Code and Codex write `~/.cognee/.env` | `mkdir -p` + `printf` + `chmod 600` | `New-Item -ItemType Directory -Force` + `Set-Content` + `icacls` |
| Shell credentials — Openclaw and API / MCP `export` into the session | `export COGNEE_BASE_URL="…"` | `$env:COGNEE_BASE_URL = "…"` |
| Skill file path (Openclaw) | `~/.openclaw/skills/cognee/SKILL.md` | `$env:USERPROFILE\.openclaw\skills\cognee\SKILL.md` |
| Writing a skill file | `mkdir -p` + heredoc | `New-Item -ItemType Directory -Force` + `Set-Content` |
| REST call (dashboard **API / MCP** card) | `curl` (multi-line, backslash continuations) | `curl.exe` (single line) |
Your credentials and the skill content itself are identical on both — only the shell syntax around them changes. The credentials file also ends up locked to your user on both: `chmod 600` on Mac, `icacls` stripping the inherited permissions on Windows.
<Note>
The toggle does not reach every step. On this page, the **OpenClaw** `AGENTS.md` step and the **API / MCP** REST and skill steps stay in bash form even with the toggle set to Windows, and the cards without a toggle at all — Claude Desktop, Cursor, VS Code, Gemini CLI, Cline, Hermes Agent — are written for macOS and Linux throughout. On Windows, translate those with the table above: the config-file paths are the ones that differ, and where a card tells you to run `which uvx`, run `Get-Command uvx` instead.
</Note>
## Agents
Agent cards come in three styles depending on how the tool reads memory.
### Plugin-based
Coding agents (CTA **Connect via plugin**) install the **cognee-memory plugin**, which hooks into the agent's lifecycle to capture and recall memory automatically — no code:
<Tabs>
<Tab title="Claude Code">
```bash theme={null}
claude plugin marketplace add topoteretes/cognee-integrations
claude plugin install cognee-memory@cognee
```
</Tab>
<Tab title="Codex">
```bash theme={null}
codex features enable hooks
codex plugin marketplace add topoteretes/cognee-integrations --ref main
codex plugin add cognee@cognee
```
</Tab>
</Tabs>
See the dedicated [Claude Code](/integrations/claude-code-integration) and [Codex](/integrations/codex-integration) integration pages for full configuration.
### Prompt-based
This card (CTA **Connect via prompts**) writes a memory instruction file into your workspace so the agent learns to use Cognee through its system prompt:
| Tool | File written |
| ------------ | --------------------------------- |
| **OpenClaw** | `~/.openclaw/workspace/AGENTS.md` |
Prompt-based flows read the `COGNEE_BASE_URL` environment variable. The instruction file teaches the agent to recall context before answering and to remember new facts afterward.
For Claude Code, the marketplace plugin is the recommended direct memory path. The full setup lives in [Cognee Plugin for Claude Code](/integrations/claude-code-integration); the key install commands are:
```bash theme={null}
claude plugin marketplace add topoteretes/cognee-integrations
claude plugin install cognee-memory@cognee
```
Use `COGNEE_BASE_URL` and `COGNEE_API_KEY` for Cognee Cloud or a remote server. If those are missing, the plugin uses local mode and needs only `LLM_API_KEY`.
### MCP-based
These cards (CTA **Connect via MCP**) register Cognee as an MCP server in the client's config. Cognee runs via `uvx` (requires [uv](https://docs.astral.sh/uv/)) — no separate install. The **Claude Desktop**, **Cursor**, and **Gemini CLI** cards are guided walkthroughs: they step you through installing `uv`, opening the client's MCP config file, merging in the Cognee server block (shown as an annotated preview so you can copy just the highlighted fragment into your existing file), and restarting the client to test.
Most clients read an `mcpServers` entry that passes credentials as environment variables:
```json theme={null}
{
"mcpServers": {
"cognee": {
"command": "uvx",
"args": ["cognee-mcp"],
"env": {
"COGNEE_BASE_URL": "https://your-tenant.aws.cognee.ai",
"COGNEE_API_KEY": "your-api-key"
}
}
}
}
```
Each client reads its config from a different location:
| Client | Config location |
| ------------------ | --------------------------------------------------------------------------------------- |
| **Claude Desktop** | Open **Settings → Developer → Edit Config** to create/open `claude_desktop_config.json` |
| **Cursor** | `~/.cursor/mcp.json` (use **Add Custom MCP** to scaffold it) |
| **Hermes Agent** | `~/.hermes/config.yaml` (YAML) |
| **Gemini CLI** | `~/.gemini/settings.json` |
| **Cline** | VS Code Cline sidebar → **MCP Servers** |
<Note>
The **Claude Desktop** flow passes credentials as command-line arguments rather than environment variables — it pins `cognee-mcp@latest` and appends `--api-url` and `--api-token`. Merge only the `"cognee"` entry into any `mcpServers` block you already have, then fully quit (⌘Q / quit from the tray on Windows) and reopen Claude Desktop. If Claude never calls a `cognee` tool, it usually can't find `uvx` on its `PATH` — run `which uvx` (`Get-Command uvx` on Windows) and use that absolute path as the `command` value.
</Note>
#### Gemini CLI
Gemini CLI has no UI for adding an MCP server, so its card walks you through editing `~/.gemini/settings.json` yourself and shows the complete file you should end up with — your tenant URL and API key included, and the `"cognee"` entry highlighted — so nothing is hidden inside a copied command. Like Claude Desktop, it passes credentials as `--api-url` and `--api-token` arguments to `cognee-mcp@latest` rather than as environment variables:
<Steps>
<Step title="Install uv">
`brew install uv`, or `curl -LsSf https://astral.sh/uv/install.sh | sh`. This is what provides `uvx`.
</Step>
<Step title="Open ~/.gemini/settings.json">
If the file or the `~/.gemini` folder doesn't exist yet, run `gemini` once to create them, or create the file yourself.
</Step>
<Step title="Add the Cognee server and save">
On a new or empty file, paste the whole block the card shows (the copy button copies all of it). If you already have an `mcpServers` block, add only the highlighted `"cognee"` entry inside it.
</Step>
<Step title="Confirm it connected">
Start `gemini`, then type `/mcp` in the session — **cognee** should be listed with its tools. Then ask *"What do you know from cognee?"*.
</Step>
</Steps>
<Note>
If `/mcp` doesn't list cognee, Gemini usually can't find `uvx` on its `PATH`. Run `which uvx` and use that absolute path as the `"command"` value in `settings.json`.
</Note>
### Extension-based
The **VS Code** card (CTA **Connect via extension**) uses the **"Cognee — Project Memory"** extension from the VS Code Marketplace instead of an MCP config file. After installing it:
1. Run **Cognee: Set Up** from the Command Palette and paste your endpoint, then your API key (stored in your OS keychain, not settings). A health check confirms the connection.
2. Use the core commands from the Command Palette under **Cognee**:
* **Cognee: Remember Selection** — store the selected code (or the whole file) in this repo's memory.
* **Cognee: Ask My Project Memory** — ask a question and get answers with clickable citations.
* **Cognee: Index Workspace** — bulk-ingest the whole repository at once.
### API / MCP
The **Connect via API or MCP** card is for any tool not covered above. It gives you a `curl` recall example and the option to install the generic Cognee skill:
```bash theme={null}
curl -X POST https://your-tenant.aws.cognee.ai/api/v1/recall \
-H "X-Api-Key: $COGNEE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"query": "What are the main entities?"}'
```
## Automation platforms
Bring Cognee memory into your workflow tools (CTA **Connect via node** / **Connect via plugin**):
| Platform | How |
| -------- | -------------------------------------------------------------------------------------------- |
| **n8n** | Install the community node package `n8n-nodes-cognee`, then add a **Cognee API** credential. |
| **Dify** | Install the Marketplace plugin by `topoteretes`. |
<Warning>
For **Dify**, the base URL **must** include the `/api` suffix (for example `https://your-tenant.aws.cognee.ai/api`).
</Warning>
## Credentials and environment variables
The flows export these variables:
| Variable | Used by |
| ----------------- | ------------------------------------------------------------------------------------------------------- |
| `COGNEE_BASE_URL` | Prompts, skills, the Claude Code plugin, and MCP servers. If missing, the client drops into local mode. |
| `COGNEE_API_KEY` | Cloud and remote clients, for authentication. The local Claude Code plugin can auto-mint one if absent. |
<Note>
Setups now use `COGNEE_BASE_URL` as the single endpoint variable — the older `COGNEE_SERVICE_URL` alias is no longer exported by these flows.
</Note>
Authentication uses the `X-Api-Key` header (not `Bearer`). To confirm a key and URL work, run a health ping against the datasets endpoint:
```bash theme={null}
curl https://your-tenant.aws.cognee.ai/api/v1/datasets/ \
-H "X-Api-Key: $COGNEE_API_KEY"
```
A `200` means the connection is good, `401` means the key is wrong, and `404` or a `5xx` means the URL is wrong or the service is unavailable.
## How memory works once connected
Connected tools read and write memory through two endpoints.
**Recall** (**`POST /api/v1/recall`**) retrieves context. The `search_type` parameter controls how results are assembled:
| `search_type` | Behavior |
| -------------------------- | --------------------------------------------- |
| `HYBRID_COMPLETION` | Default — combines graph and chunk retrieval. |
| `GRAPH_COMPLETION` | Answers from the knowledge graph. |
| `CHUNKS` | Returns raw matching text chunks. |
| `GRAPH_SUMMARY_COMPLETION` | Summarized graph-based answer. |
**Remember** (**`POST /api/v1/remember/entry`**) stores a fact. Tie related entries together with a shared `session_id`:
```json theme={null}
{
"entry": { "type": "qa", "question": "...", "answer": "..." },
"dataset_name": "default_dataset",
"session_id": "..."
}
```
The default dataset name is `default_dataset`.
<Note>
Use `/remember/entry` for inline text. The plain `/remember` endpoint requires a file upload and returns `422` for inline text.
</Note>
## Verify the connection
After running any flow, verify the integration by asking the connected agent:
> What do you know from cognee?
If memory is wired up correctly, the agent recalls context from your knowledge graph. The final step of each flow includes a **Go to Sessions →** button that takes you to your live [Sessions](/cognee-cloud/ui/sessions).
## Debug plugin hooks
When a plugin-based agent does not capture or recall memory, debug the layers separately.
| Layer | What to check |
| ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Hooks | The agent must call the Cognee plugin on prompt submit, tool use, stop, compaction, and session end. For Codex, run `codex features enable hooks` and trust the Cognee hooks in `/hooks` if prompted. For Claude Code, confirm the startup message says **"Cognee Memory Connected"**. |
| Backend | Cloud and remote mode need `COGNEE_BASE_URL` and `COGNEE_API_KEY`. Local mode needs `LLM_API_KEY` so the plugin-managed local Cognee API can start again after it was stopped. |
| Session | A resumed terminal only continues the same live session when the session id and dataset match. Set `COGNEE_SESSION_ID` and keep `COGNEE_PLUGIN_DATASET` unchanged when you want a restart to keep writing to the same session. |
Hooks do not replay events that happened while they were disabled, untrusted, or unable to reach Cognee. After changing hook trust, credentials, dataset, or session id, restart the agent so its startup hook can initialize the new state.
For the most reliable graph sync, exit the agent normally. Claude Code users can also run `/cognee-memory:cognee-sync` before closing. If the process is killed, recent session cache entries may exist, but the final session-to-graph sync may not have run yet.
## Data sources
The **Data sources** section lists connectors that pull the tools your team already uses into your datasets — one connection per workspace, shared with everyone. **Slack** is the first live connector: **Connect** on its card opens a dialog that connects your workspace, and a connected card shows the connection's status and channel/sync health, with **Manage** (or **Reconnect**, when the connection needs attention) opening the same dialog.
Authorizing Slack reads nothing on its own — **ingestion is opt-in per channel**. The Manage dialog leads with the channels the bot can see, and you tick the ones to ingest (with a search box and bulk actions for a long list); switching a channel on also ingests its existing history. Until you do, the card says so: **Not in any channel yet** when the bot is in none, or **No channels selected yet** when it is in some but none are routed. Once channels are routed the line reads *3 of 12 channels syncing*, or **Channel list unavailable** when the list cannot be read.
Only the workspace owner can connect, manage, disconnect, or choose channels — *Only the workspace owner can connect this integration.* Members see the card without an action.
### Linking your own Slack account
Connecting Slack authorizes the workspace, not you personally. To have `/cognee-recall` in Slack answer as *you* — from what your own Cognee account can see — run **`/cognee-link`** in Slack. It replies with a link to **Connect your Slack account** (route `/link-slack`), where **Connect my account** completes the pairing. Each link is valid for **10 minutes**; when it expires or arrives malformed the page says so and tells you to run `/cognee-link` again.
The page refuses to pair the wrong things, since it can only see which Slack member is asking:
| Situation | What it says |
| ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| The workspace has no Slack connection | * has no Slack connection, so there is nothing to link your account to here.* Switch to the workspace your team connected Slack in, then reopen the link. |
| The link came from a different Slack workspace | *This link came from a different Slack workspace than the one connected to .* Switch to the matching Cognee workspace at the top of the page, then reopen the link. |
| Done | *`/cognee-recall` in Slack now answers as you, from what your Cognee account can see*, with **Back to Integrations**. |
Below it, a **More data sources** section ("Not live yet — tell us which ones to prioritize.") lists twenty-two upcoming connectors: Notion, Google Drive (Docs, Sheets, and Slides), Confluence, GitHub, Gmail, Jira, Linear, Granola, Asana, monday.com, Figma, HubSpot, Intercom, Box, Canva, PostHog, Stripe, Vercel, MotherDuck, Xero, lemlist, and Workable. A **Search data sources** field beside that heading filters the cards by name or description as you type; if nothing matches, the section shows **No data sources match "".** followed by a **Request it** link — *and we'll consider it next* — which opens a pre-filled email to [support@cognee.ai](mailto:support@cognee.ai).
These upcoming connectors are not built yet — clicking one opens a **Coming soon** dialog with a **Get notified once live** button that registers your interest so we can prioritize what to build next. A footer link (**Let us know**) emails [support@cognee.ai](mailto:support@cognee.ai) if you want to request a source directly.
<Note>
Self-hosted Cognee deployments run their own, separate Slack app against their own backend — setup and usage are covered in the [Slack integration guide](/integrations/slack-integration). To bulk-load data from these tools in code today (for example Slack channel history via dlt's [`slack_source`](https://dlthub.com/docs/dlt-ecosystem/verified-sources/slack)), see the [dlt integration](/integrations/dlt-integration).
</Note>
## Related
<CardGroup>
<Card title="Sessions" icon="messages-square" href="/cognee-cloud/ui/sessions">
Inspect the live memory your connected tools read and write.
</Card>
<Card title="API Keys" icon="key" href="/cognee-cloud/ui/api-keys">
Create and manage the keys these integrations authenticate with.
</Card>
<Card title="Cloud SDK" icon="code" href="/cognee-cloud/connections/cloud-sdk">
Call recall and remember programmatically from your own code.
</Card>
<Card title="Cloud MCP" icon="plug" href="/cognee-cloud/connections/cloud-mcp">
Details on the Cognee MCP server and its tools.
</Card>
</CardGroup>
# Mindmap
Source: https://docs.cognee.ai/cognee-cloud/ui/knowledge-graph
Explore one dataset's knowledge graph on a live canvas — sources, records, and the answers they served
The Mindmap (route `/knowledge-graph`, under the **EXPLORE** section of the sidebar) draws one dataset's knowledge graph as a live, force-directed canvas: the records Cognee extracted, the sources they came from, the people and agents who can reach them, and the questions the graph has answered.
The same view is also served at `/business`, which is an alias for this page.
It is rendered in the browser from `GET /v1/visualize/json` (one dataset at a time, up to 1,000 nodes), not embedded as an image or an iframe, so everything on it is interactive and it keeps itself current — there is no **Refresh** button. The same data is available programmatically through the [visualize](/cognee-cloud/functionality/search-and-recall#visualize) endpoint, or you can generate visualizations in code — see [Graph Visualization](/guides/graph-visualization).
## The canvas
* **Pan** by dragging the background; **zoom** with the scroll wheel or a pinch.
* **Click a record** to focus its neighbourhood and open its panel.
* **Shift-click a second record** to trace the shortest path between the two.
* **Hover a record** for a tooltip with its type, the sources it came from, and how many places it was seen in.
Records cannot be dragged individually — the layout is force-directed and settles on its own.
### Legend
A legend explains the encoding:
| Mark | Meaning |
| ----------------- | --------------------------------- |
| Size | Importance |
| Colour | Which source it came from |
| Amber ring | Part of the live answer |
| Dashed ring | Answered questions before |
| Double ring | Spans sources / agent memory |
| Green ring | Shortest path between two records |
| Faint dashed grey | No connections yet |
| Drifting dots | Relationships at work |
## Choosing a dataset
A **dataset** switcher sits in the top-left corner. It groups datasets into **team** and **personal**, shows each one's sources, and hints when a dataset has not been cognified yet. Switching datasets clears the current selection and any focus lens, then re-narrates once the new graph lands.
## Sources rail
A **sources** rail runs down the left side, one card per source in the dataset. Click a card to enter a **focus lens**: everything not from that source dims, the camera flies to what remains, and the narration line reads *showing only — 412 entities · click again for everything*. Click it again to clear the lens.
A source with nothing extracted yet cannot be focused — the page says so instead of dimming the whole graph: * has no extracted entities yet — 8 items not shown as a graph*.
## Operators rail
An **operators** rail on the right lists the workspace's users and agents. Hovering one highlights what that principal can reach, so access reads as a highlight on the graph rather than as drawn edges.
## Record panel
Selecting a record opens a panel on the right with its name, type, sources, how many places it was seen in, and its connection count. It also offers:
* **shift+click another record to trace the path between them** — once traced, the panel shows *path to — 3 hops* with a dismiss control.
* **⚠ what breaks without this record?** — simulates removing it, so you can see what would become unreachable.
* **Used in N answers** — the questions this record actually served, newest first.
## Search
A search bar at the top does double duty: type a record name to jump straight to it, or ask a question and get an answer over the graph. Its placeholder is *find a record or ask anything…*.
## Live updates
The Mindmap keeps itself current while you watch it. It subscribes to the selected dataset's live channel and redraws as memory lands, so a `cognify` finishing or an agent writing shows up without any action from you. A **Cognify complete** notification names how many new entities joined the model and which source they came from.
Three separate channels feed it, and they are not interchangeable:
| Channel | What it carries | Cadence |
| ----------------------------- | ---------------------------------------- | -------------------------------------------------------------------- |
| WebSocket subscription | Graph-growth pushes and session events | Push |
| Session-event poll (fallback) | Session events only — never graph growth | \~1.5s, backing off to 10s on repeated failures |
| Graph re-fetch | The dataset's graph | \~8s while socket pushes are not arriving, backing off to \~1 minute |
The fallback poll is used when the subscription cannot be established at all — a local instance with no socket target, or a workspace whose pod does not serve the endpoint.
Each dataset is drawn with up to **1,000 nodes**. The cap is about payload size and canvas legibility rather than backend work, and it sits deliberately below the threshold at which the canvas starts capping to the viewport, so a single dataset's scene is never viewport-capped however dense it is. The graph read is given an extended timeout of **45 seconds**, well above the normal request budget, so a slow load on a large graph is not misreported as a failure.
### When the graph answers a question
When a live search event arrives, a chip appears at the bottom-left reading **this graph just answered a question** with the question quoted beneath it. **▶ see what it used** replays the reasoning: the records the answer drew on light up with an amber ring, and an answer card surfaces with the response. Records that have served an answer keep a dashed ring afterwards, so the graph accumulates a visible record of what it has been useful for.
## Bottom dock
A dock across the bottom carries:
* **A narration line** that reacts to what you do — switching datasets, focusing a source, growth landing, insights surfacing.
* **An altimeter** with four levels — **Business**, **Players**, **Connections**, **Records** — that changes how much of the graph is drawn. **Records** exposes the raw records layer (chunks, documents, and summaries behind the entities) and carries its own count.
* **A live indicator** — **● LIVE** when the subscription is connected, **○ live: reconnecting…** otherwise.
* **▶ tour** — a flythrough of the graph. Any click stops it (the button then reads **■ stop tour**).
## Empty and error states
| State | What you see |
| -------------------------------------- | ---------------------------------------------------------------------------------------------- |
| Loading | *weaving your business model…* |
| No dataset selected | *no dataset selected — create a dataset and upload documents to see your business model* |
| The dataset list failed | *couldn't load this workspace's datasets — check your connection and reload* |
| This dataset's graph failed | *couldn't load this dataset's graph — check your connection and try switching to it again* |
| Dataset has content, nothing extracted | *content ingested, nothing extracted into the graph yet. check back after processing finishes* |
| Dataset is empty | *no content in this dataset yet* |
<Note>
To rebuild a dataset's graph — after changing its extraction model, prompt, or ontology — use **Rebuild graph** on the [dataset detail page](/cognee-cloud/ui/datasets#dataset-detail), which is also where those settings live.
</Note>
# Members
Source: https://docs.cognee.ai/cognee-cloud/ui/members
Invite teammates, manage tenant members, and switch between tenants
The Members page lets tenant owners invite and remove team members. All tenant members can open the page to see who else is on the team. It lives at **/members** (the older **/access-management** route redirects there). Open it from the **profile menu** in the top bar — click your avatar and select **Members**.
The first **Personal Workspace** is free and cannot be deleted. Each additional workspace you own costs \$5/month.
## Invite members (owner only)
The **Invite Members** card is visible only to the tenant owner. Other members see the team roster but not the invite controls.
1. Enter one or more email addresses in the invite form. Use **Add another** to invite several people at once.
2. Click **Invite**. Each address is added to the tenant via **`POST /tenants/users?tenant_id=&email=`**.
3. Invited users gain access to all tenant-wide datasets and can sign in to the same workspace.
<Note>
The person you invite must have signed in to Cognee at least once before they can be added. They don't need a payment method of their own — workspaces support unlimited users.
</Note>
You can invite people who do not yet belong to the tenant. Invite emails are sent via a webhook, and the result is reported per outcome:
* Addresses that already belong to an account are added immediately (*"N existing members added"*).
* Addresses with no account yet receive an invite email and are recorded as pending (*"N invite emails sent — they join automatically when they sign up with that email"*). The invite link pre-fills the email field on the sign-up form, and the new user joins the tenant automatically when they sign up with that exact address.
## View members
The **Members** tab lists everyone who has joined the current tenant, fetched from **`GET /tenants/{tenantId}/users`**. Each row shows the member's avatar, email, and role (`Owner` or `Member`). Use the search box to filter by email.
<Note>
Roles are not yet populated in the UI, so every member currently shows as `Owner` or `Member`. The role filter and text filter exist, but the role filter is scaffolding for now.
</Note>
Service accounts whose email ends in `@cognee.agent` are filtered out of the roster.
## Pending invitations
Invited addresses that have not signed up yet appear under the **Pending Invitations** tab (with a count) rather than in the members roster. Each pending row shows a **Pending** badge and the note *"Joins automatically when they sign up with this email."* The tenant owner can revoke a pending invitation from the `⋮` menu on the row.
## Remove a member (owner only)
The remove (`⋮`) control next to each row is visible only to the tenant owner. Confirm the prompt to remove a member; the call goes to **`DELETE /tenants/users?tenant_id=&user_id=`** and also revokes the member's role and dataset permissions inside the tenant. The owner cannot be removed.
## Switch tenants
If you belong to more than one tenant, the workspace can be switched without signing out. Available tenants are fetched from `GET /api/v1/permissions/tenants/me`. Selecting another tenant from the workspace switcher stores the choice in a `cognee_selected_tenant` cookie (also mirrored to `localStorage`) and reloads the app against that tenant's API base URL.
The selected tenant is sticky across page loads. Server actions read the same cookie so management calls (members, billing) target the active tenant.
<Note>
Tenant membership and ownership are part of the broader [permissions model](/core-concepts/multi-user-mode/permissions-system/tenants). For the underlying API endpoints, see [Permissions & Access Control](/cognee-cloud/functionality/permissions-and-access-control).
</Note>
[Learn more about Team Members](/cognee-cloud/functionality/permissions-and-access-control)
# Memory Coverage in the Cloud UI
Source: https://docs.cognee.ai/cognee-cloud/ui/memory-coverage
Score every question your agents asked by how well memory answered it, and find the gaps
The Memory Coverage page (route `/memory-gap-analysis`) replays the questions your workspace was actually asked and scores each one by how well memory answered it. Low scores are your gaps — the page exists to point at the knowledge worth adding next.
Its header states that plainly: *Every question you asked, scored by how well memory answered it. Low scores are your gaps — add the missing knowledge to fix them.* Two actions sit beside it: **Add memory**, which opens [Datasets](/cognee-cloud/ui/datasets), and **Run analysis**.
Open it from the [Overview](/cognee-cloud/ui/dashboard)'s Memory Coverage panel — clicking a dataset row there opens this page already scoped to that dataset.
<Note>
Memory Coverage is a Cognee Cloud feature. In the local, open-source UI the route renders a notice instead of the analysis — see [Local UI](/cognee-cloud/local-ui#cloud-only-pages).
</Note>
## How coverage is measured
A coverage run is not a synthetic benchmark — it is built from the questions your workspace was really asked. A run does four things:
<Steps>
<Step title="Collect">
It gathers the [recall](/core-concepts/main-operations/recall) traffic against the selected dataset — every question your agents and your own searches put to it.
</Step>
<Step title="Dedupe">
Near-identical questions are clustered into one entry. The cluster's size becomes that question's **recall** count: observed demand, not a forecast. Asking the same thing forty times makes it one question with a count of forty, so a much-asked gap cannot hide behind a long tail of one-offs.
</Step>
<Step title="Replay">
Each deduped question is put back through recall against the graph **as it stands now**. That is the point of replaying rather than re-reading the original answers: the score describes the memory you have today, including everything ingested since the question was first asked.
</Step>
<Step title="Judge">
An LLM judge scores each replayed answer from **0 to 5** — **0–1** nothing usable was retrieved, **2–3** partially answered, **4–5** answered. A question whose replay or judge call fails reaches *no verdict* and is left out of every average rather than counted as a zero.
</Step>
</Steps>
Each run also assigns every question to a topic, so the score can be read per subject area rather than only in aggregate.
## Running an analysis
**Run analysis** replays every question the selected dataset was asked and scores the answers with an LLM judge. A run covers the whole corpus, so it takes a few minutes; the button reads **Starting…** and then **Running…** while it works.
Everything on the page stays fully readable and interactive during a run — you can filter, search, and export the previous run while the new one computes. A status line says so out loud: **Run in progress.** *Replaying every question takes a few minutes. Everything below is the previous run until it finishes.*
The page polls the run every four seconds until it settles.
## The score
A full-width meter carries the score for the selected dataset, out of **5.0**, with the number riding the fill edge in its band colour:
| Band | Score | Verdict |
| ----- | --------- | ----------- |
| Red | 0.0 – 1.9 | **Gap** |
| Amber | 2.0 – 3.9 | **Partial** |
| Green | 4.0 – 5.0 | **Covered** |
The number is an **unweighted mean of the topic averages**, and each topic average is the mean of that topic's *measured* questions. Topics therefore count equally: a topic holding four questions moves the score as much as one holding forty. That is deliberate — it keeps a single busy subject area from burying a small one that answers nothing.
<Note>
This page counts **Other** as a topic, so questions that matched no topic weigh on the score like any other bucket. The **Real recall** figure on the [Overview](/cognee-cloud/ui/dashboard) panel is the workspace roll-up and **excludes Other**, then averages across datasets that have a score. The two numbers can differ for the same dataset, and neither is wrong — they answer different questions.
</Note>
A run that produced no score says which of the several possible reasons applies, rather than showing `0.0`: **Scoring…** while pending or running, **No questions yet** for a completed run that replayed nothing, **Run failed**, or **No verdict reached** when questions *were* replayed but none reached a verdict — an outage in the judge or the pod rather than a dataset that answers badly.
### Sample size
Directly under the meter, a line says what the score actually covers — **Covers 38 of 50 questions.** A score of 4.6 over five measured questions of fifty is not the claim that 4.6 over fifty would be, so the count is always stated.
When a fifth or more of the corpus reached no verdict the line turns amber and adds why it matters: *12 reached no verdict, so this score does not describe the rest of the dataset.* Questions whose replay or judge call failed are excluded from the average rather than averaged in as zero.
## Datasets
A dataset switcher sits under the score. A coverage run is always scoped to one dataset, so switching datasets shows that dataset's latest run.
Each chip carries the dataset name and its own coverage score in the band colour — a neutral **—** when it has none, and **!** when that dataset's coverage could not be loaded, so one unreachable dataset never makes the others look unscored.
Hovering a chip adds a note about its run: *Loading…*, *Not scored yet*, *Couldn't load this dataset's coverage*, a sample caveat such as *Coverage 4.6 of 5*, or — when **20% or more** of the dataset's questions match no topic — *43% of questions match no topic*, a signal that the topics need work, since a large **Other** bucket makes the per-topic scores less useful. The sample caveat outranks the topic warning: a score that does not describe the corpus is the more important thing to say first.
## Questions
The questions panel holds the run's replayed questions, and its frame is labelled with the current topic (or **Questions** when no topic is selected).
* **Topic chips** filter the questions by topic and show how many questions the topic holds — `12`, or `5/12` when only some of them were scored, with the number turning amber once a fifth or more went unscored. They do not show a score. Topics belong to you and persist across runs; each run assigns its questions to them, and anything that matches none confidently lands in **Other**.
* **View toggle** switches between a card grid and a dense list; the list is the default.
* **Export** writes the questions currently in view to CSV, with the columns **Question**, **Coverage**, **Relevance**, **Topic**, **First asked**, **Reference**, and **Replayed answer** — so the answer memory actually gave, and the source it drew on, come out with the score even though the cards do not show them.
* **Search questions** is stemmed and prefix-tolerant, and searches four fields: the question text, the replayed answer, the topic label, and the reference.
* **Sort** offers **Score** (worst first — your gaps) and **Recalls** (most-asked first).
* A **Recall** readout on the right of the toolbar totals the recalls behind whatever is currently filtered, and is relabelled **Topic Recall** while a topic chip is selected.
Each question shows its score chip in the band colour, a lavender **recall** chip counting how often it was asked in the window (*the size of its dedup cluster, not a forecast*), and its topic. An unmeasured question shows a neutral dash with the tooltip *Not measured: this question's replay or judge call failed* — never a `0.0`, which would read as *memory answered nothing*.
### Which run you are looking at
A footnote under the questions panel identifies the run on screen — **Run** id, **Status**, and **Created** date — alongside the band legend (**Gap 0–1.9**, **Partial 2.0–3.9**, **Covered 4.0–5.0**). It is the quickest way to check whether the figures above came from a finished run.
### Deleting a topic
A topic chip can be deleted. The confirm dialog is explicit that nothing is lost: the questions move back to **Other**, and the next run may group them again.
<Warning>
A topic deletion is not saved yet. It applies only to the browser tab you are looking at — a reload, a workspace switch (which is a full page reload), or the next run brings the topic back. Switching datasets and back does not: the edit is held per dataset, so the deletion is still there when you return. While it holds, its questions count under **Other**, which also changes the score on this page, since **Other** counts as a topic here.
</Warning>
## Empty and error states
Each state names a different fact, because the next step differs:
| State | What you see |
| ----------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| No dataset in the workspace | *Add a dataset and ask it something — this page scores every question by how well memory answered it.* |
| Dataset never scored | * has not been scored yet. Run analysis replays every question it was asked and scores how well memory answered.* |
| The fetch failed | **Couldn't load coverage for ** — *The workspace didn't answer. Nothing is lost — any run it has already scored is still there.* with **Retry**. |
| A refresh failed, older run in hand | **Couldn't refresh .** *This is the last run we have, so it may be out of date.* with **Retry** — the previous run stays on screen. |
| **Run analysis** was rejected | **Couldn't start the run.** with the reason and **Try again**, above the run it failed to replace. |
<Note>
Coverage runs appear in [Activity](/cognee-cloud/ui/activity) as `recall` operations named `coverage_run`: replaying a question reads from memory and writes nothing.
</Note>
# Search in the Cloud UI
Source: https://docs.cognee.ai/cognee-cloud/ui/search
Ask questions about the knowledge graph built from your documents.
The Search page is a chat-style interface for querying your knowledge graphs. When a conversation is empty, the page greets you with **What are you looking for today?**
<Note>
Search consumes token credits under Cognee Cloud's [usage-based billing](/cognee-cloud/functionality/account-and-billing). If your workspace runs out of prepaid credits, the query is rejected and an insufficient-credits dialog explains the failure and links to Billing — see [When credits run out](/cognee-cloud/functionality/account-and-billing#when-credits-run-out).
</Note>
## Sending a query
1. Type a natural language query in the search bar (placeholder **Ask a question about your data...**) and press **Enter** to send; use **Shift+Enter** for a newline.
2. Results appear in the conversation area below.
Suggestion chips are shown when a conversation is empty:
* "What are the main entities?"
* "Summarize the uploaded documents"
* "What relationships exist in the data?"
Click any chip to send it immediately.
Past searches are loaded from the backend and shown in the history sidebar. History persists across sessions and is fetched via the [`GET /api/v1/search`](/cognee-cloud/functionality/search-and-recall#search) endpoint. Queries sent from this page are recall calls, and recall records the questions it answers, so your own UI searches appear in that history alongside searches made through the API — see [Recall and search history](/cognee-cloud/functionality/search-and-recall#recall-and-search-history).
## Dataset
The active dataset is chosen with the dataset selector in the input area, at the right just above the search bar — not from the top bar, which carries only the workspace switcher and the page name. A query is always scoped to exactly one dataset: there is no **All datasets** option here, and if you have not picked one, the first dataset in the list is used. See [Dataset selector](/cognee-cloud/ui/datasets#dataset-selector).
Every query searches the knowledge graph built from the documents you uploaded into that dataset. There is no memory scope to choose either: the former **Company Brain** / **Agent Memory** pills were removed, and the page no longer searches agent sessions. To look through what your connected agents stored, open [Sessions](/cognee-cloud/ui/sessions) instead.
## Conversation history
The sidebar on the left lists past conversations grouped by age — **Today**, **Yesterday**, **This week**, **This month**, **Older**. Click any conversation to reopen it, use **+** to start a new one, or use the sidebar toggle to collapse and expand the history panel.
Past searches are loaded from the backend via [`GET /api/v1/search`](/cognee-cloud/functionality/search-and-recall#search). New conversations created during the current page session appear alongside that history, deduped by id, so a restored conversation you have continued is listed once.
The sidebar shows a skeleton while history loads, rather than asserting there are no conversations before it knows. Each conversation is titled with the workspace's own title for the session, which is computed from the full transcript — the detail endpoint returns only the last 20 turns, so on a long conversation the first visible question is not necessarily the one that was asked first. A conversation whose visible tail is blank still keeps its place in the list, and a single malformed row costs that one row rather than the whole page.
<Note>
Conversations are backed by backend sessions whose ids are prefixed with `search-ui-`, so your history survives a page refresh. Legacy single-Q\&A entries are read-only; continuing one forks it into a new session.
</Note>
## How search works
The UI search calls the `POST /api/v1/recall` endpoint with graph scope — a standard graph retrieval over the active dataset. For details on available search types and parameters, see [Search Basics](/guides/search-basics).
<Note>
The Overview used to carry an inline search terminal. It no longer does — the redesigned [Overview](/cognee-cloud/ui/dashboard) reports on your memory instead, and this page is where you query it.
</Note>
# Sessions in the Cloud UI
Source: https://docs.cognee.ai/cognee-cloud/ui/sessions
Monitor agent runs that read and wrote to your Cognee memory
The Sessions page (route `/sessions`, under the **DATA** section of the sidebar) lists agent runs that read or wrote to your tenant's Cognee memory. Each session captures the observations, tool calls, tokens, and cost for one agent conversation. For the workspace-wide log of individual runs and memory operations rather than whole conversations, see [Activity](/cognee-cloud/ui/activity).
The header shows a one-line summary of the selected time range — for example, *Agent runs that wrote to your memory · 142 sessions · 96% success · \$3.21*. Aggregate stats (total sessions, success rate, and total spend in USD to two decimals) are pulled from **`GET /v1/sessions/stats?range=`**, and the list itself comes from **`GET /v1/sessions`** (paged).
## Time range
A segmented control in the header filters sessions and stats by time range:
| Range | Window |
| ------- | ---------------------- |
| **24h** | Last 24 hours |
| **7d** | Last 7 days |
| **30d** | Last 30 days (default) |
| **all** | All sessions |
Click **Refresh** to re-pull the list, the aggregate stats, and the currently open session detail.
## Sessions list
The page uses a two-pane layout. The left pane lists up to 100 sessions; the right pane shows the detail for the selected session.
Each row in the list shows:
* A colored **status dot** indicating the run state.
* A short **session id**.
* The **last-activity** or started date.
* The **model** used.
Status values and their colors:
| Status | Color |
| ----------- | ------ |
| `completed` | Green |
| `running` | Purple |
| `failed` | Red |
| `abandoned` | Gray |
## Session detail
Selecting a row loads the session from **`GET /v1/sessions/{session_id}`** into the right pane, which opens with a set of stat cards:
* **Observations** — Message count for the session.
* **Tool calls** — Number of tool invocations.
* **Tokens** — Tokens in plus tokens out.
* **Cost** — Spend in USD to four decimals (e.g. `$0.0123`).
* **Duration** — Total run time.
## Transcript
Below the stat cards, the transcript lists the session's question-and-answer turns. Each turn is tagged either **Recall** (purple) or **Remember** (green):
* A turn is **Recall** when its question matches a recall trace — the agent read from memory.
* Otherwise the turn is a **Remember** — the agent wrote to memory.
Click a turn to expand it and reveal the answer, and use the copy-answer button to copy it. Each turn has feedback thumbs (positive or negative) and an optional feedback text field. When a session has more than two turns, an in-transcript search appears so you can filter turns.
**Improve** (graph-enrichment) runs are interleaved into the transcript in chronological order, tagged **Improve** (purple), with a truthful status: **success**, **failed**, or **in progress**. A failed run reads *Graph enrichment did not complete — no memory was bridged this run* and shows its failure reason when one is available. That includes a run that failed because the workspace ran out of credits mid-run — it shows the same failed status with its reason; the credit-specific **Failed — insufficient credits** label exists only on [datasets](/cognee-cloud/ui/datasets#status-dots). Improve entries are hidden while you are searching the transcript.
## Self-improvement
A **Self-improvement** card at the top of the session detail surfaces the session-to-graph bridge — the visible signal of Cognee's [improve()](/core-concepts/main-operations/improve) operation. As a session accumulates turns or goes idle, Cognee automatically runs `improve()`: the session's questions, answers, and feedback are bridged into the permanent knowledge graph — weighting existing memories by feedback, persisting the conversation, distilling reusable lessons, and enriching the graph so future sessions recall from it.
The card shows the status of the **last graph enrichment** (with a colored status dot, plus **in progress** or **failed** labels). Before any run has happened it reads *No graph enrichment yet. Improve runs automatically once this session accumulates turns or goes idle.*
## Tool invocations and recent activity
A **Tool invocations** bar chart counts traces by function, so you can see which tools the agent used most. A **Recent activity** feed lists the latest traces, each with a green or red status dot, the function name, and either feedback or an error message.
## How sessions get populated
Sessions appear when an agent calls **`POST /v1/recall`** (reads) and **`POST /v1/remember/entry`** (writes) with a consistent `session_id`.
<Tip>
Use a session id of the form *agent name + a unix timestamp*, and reuse it across the whole conversation so every recall and remember call lands in the same session.
</Tip>
```python theme={null}
session_id = f"support-agent-{int(time.time())}"
# read from memory
recall(session_id=session_id, query="What did the customer order?")
# write to memory
remember_entry(session_id=session_id, content="Customer requested a refund.")
```
Chat sessions created from the in-app [Search](/cognee-cloud/ui/search) page carry the prefix `search-ui-`, so they are easy to distinguish from agent sessions in the list.
## Empty state
When no sessions exist for the selected range, the page shows:
> When an agent connects to Cognee and reads or writes memory, its session will appear here with tools used, observations, and cost.
To start sending data, connect an agent through the [Integrations](/cognee-cloud/ui/integrations) flow, which deep-links back here with a **Go to Sessions →** button.
<Note>
Looking for chat conversations instead of agent runs? See the [Search](/cognee-cloud/ui/search) page, whose sessions carry the `search-ui-` prefix.
</Note>
# Settings
Source: https://docs.cognee.ai/cognee-cloud/ui/settings
Your profile, workspace deletion, and account deletion
The Settings page (route `/settings`) holds your own account details and the two destructive actions Cognee Cloud offers. It has no sidebar entry — open it from the avatar menu in the top bar, where the entry is labelled **Profile**. That menu also links to [Members](/cognee-cloud/ui/members), to [Billing](/cognee-cloud/functionality/account-and-billing) for workspace owners, and to **Log out**.
`/account` is an alias for this page and redirects here.
The page is three cards.
## Profile
**Profile** — *Your personal account information* — shows your **Name** and **Email**. Both are read-only here; they come from the account you signed up or signed in with.
## Workspace
**Workspace** — *Manage your workspace* — carries **Workspace settings**, from which you can delete the workspace you are currently in. The confirmation states the consequence plainly: *This will permanently delete the workspace and all its data. This cannot be undone.* **Yes, delete workspace** carries it out.
Deletion is immediate and irreversible — there is no scheduled removal and nothing to cancel. Once it succeeds, the workspace's paid seat is released from your subscription. See [Workspaces](/cognee-cloud/functionality/account-and-billing#workspaces).
Two workspaces cannot be deleted here:
* Your personal workspace — *Your personal workspace can't be deleted.*
* A workspace you do not own — *Only the workspace owner can delete this workspace.*
## Danger zone
**Danger zone** — *Irreversible account actions* — holds **Delete account**, which *permanently deletes your account: every workspace you own, all their data, your subscription, and your login. This cannot be undone.*
To arm it you must type your own email address to confirm; **Permanently delete my account** then completes it. See [Delete your account](/cognee-cloud/functionality/account-and-billing#delete-your-account).
<Warning>
Deleting your account removes every workspace you own along with all of their data. Export anything you need first — the [Activity](/cognee-cloud/ui/activity), [Analytics](/cognee-cloud/ui/analytics), and [Memory Coverage](/cognee-cloud/ui/memory-coverage) pages all offer CSV export.
</Warning>
# Skills
Source: https://docs.cognee.ai/cognee-cloud/ui/skills
Browse, upload, and share Skills — procedural playbooks that give your agents reusable, dataset-scoped instructions — from the Skills page in Cognee Cloud.
**Skills** are procedural playbooks (`SKILL.md` files) loaded by the agent to give it reusable, named instructions for a task. The **Skills** page lets you browse every skill registered in your workspace, upload new ones, and share an existing skill with more datasets. Open it from the sidebar.
Skills are **scoped to datasets** — a skill is attached to one or more datasets, and the same skill can be copied into additional datasets.
## Browse skills
The page is a two-column browser:
* **Datasets** (left) — every dataset that has at least one skill registered. Datasets without skills are hidden. Select a dataset to see its skills.
* **Skills** (right) — the skills registered in the selected dataset, grouped by their maintainer.
Use the search box to filter by skill **name**, **maintainer**, **description**, or **tag**. A green dot marks an active skill; a grey dot marks an inactive one.
### Skill detail
Click a skill row to expand it. The detail view shows:
* **Description** and metadata — maintainer (linked to its homepage or repository when a URL is provided), version, license, source repository, and the number of datasets the skill is attached to.
* **Declared tools** — the tools the skill is allowed to invoke.
* **Procedure** — the full skill instruction body, fetched on demand when you expand the row.
## Add a skill
Click **Add skill** (enabled once you have at least one dataset) to open the upload dialog:
1. Enter a **name** for the skill. The name is slugified into the ingestion path (for example, *Weather lookup* becomes `weather-lookup/SKILL.md`).
2. Provide the **content** — either upload a `.md` file or paste the skill markdown directly.
3. Select one or more **datasets** to attach the skill to.
4. Submit. The result is reported per dataset, so you can see which datasets the skill was added to successfully.
Adding a skill is **idempotent per dataset and name**: if you add a skill with a name that already exists in the selected dataset, the existing skill is updated with the new content instead of a second copy appearing in the list. That makes the dialog the way to edit a skill — re-add it with the same name and the revised markdown. Adding the same name to a *different* dataset still creates a separate skill, which is what the share flow below relies on.
## Share a skill with more datasets
Each skill row has a **share** control ("Add this skill to more datasets"). It opens a dialog listing the datasets the skill is **not** yet attached to. Select one or more destination datasets to copy the skill into them; the outcome is reported per dataset.
<Note>
Skills are ingested through the same path as the `content_type: "skills"` option of the ingestion API — see [Data Ingestion](/cognee-cloud/functionality/data-ingestion).
</Note>
# Claude Code with Cognee MCP
Source: https://docs.cognee.ai/cognee-mcp/integrations/claude-code
Connect Claude Code to Cognee with the memory plugin or MCP tools.
Claude Code is Anthropic's command-line AI assistant. Cognee can connect to it in two ways:
* **Cognee memory plugin**: recommended for Claude Code memory across sessions. The plugin captures prompts, tool traces, and assistant responses into session memory, injects relevant context on each prompt, and syncs session memory into the permanent knowledge graph at session end.
* **MCP server**: use this when you want Claude Code to call Cognee MCP tools from an existing Cognee MCP server.
<Tip>
For most users, the [**Cognee memory plugin**](/integrations/claude-code-integration) is the simpler path — it adds persistent memory with two commands and no separate MCP server to run. The MCP setup below is for connecting Cognee's full MCP tool set.
</Tip>
## Prerequisites
* Node.js and npm installed
* For plugin local mode: `LLM_API_KEY`
* For plugin Cloud or remote mode: `COGNEE_BASE_URL` and `COGNEE_API_KEY`
* For MCP mode: Cognee MCP server running (see [Quickstart](/cognee-mcp/mcp-quickstart) or [Local Setup](/cognee-mcp/mcp-local-setup))
## Plugin Setup
<Steps>
<Step title="Install Claude Code">
```bash theme={null}
npm install -g @anthropic-ai/claude-code
```
</Step>
<Step title="Install the Cognee memory plugin">
Install from your shell before launching Claude Code, so the first `claude` session can bootstrap memory cleanly:
```bash theme={null}
claude plugin marketplace add topoteretes/cognee-integrations
claude plugin install cognee-memory@cognee
```
</Step>
<Step title="Choose local or Cloud mode">
<Tabs>
<Tab title="Local">
Local mode is the default. The plugin starts a local Cognee API at `http://localhost:8011`. Only `LLM_API_KEY` is required; the Cognee API key is auto-minted if absent.
```bash theme={null}
export LLM_API_KEY="sk-..."
claude
```
</Tab>
<Tab title="Cloud or remote">
Set both variables before launching Claude Code:
```bash theme={null}
export COGNEE_BASE_URL="https://your-tenant.aws.cognee.ai"
export COGNEE_API_KEY="ck_..."
claude
```
</Tab>
</Tabs>
</Step>
<Step title="Confirm the connection">
On startup, Claude Code should show a **Cognee Memory Connected** system message.
</Step>
</Steps>
The plugin hooks into Claude Code's lifecycle: `SessionStart` selects mode and sets up identity, `UserPromptSubmit` injects dataset-scoped context, `PostToolUse` captures tool traces, `Stop` writes the assistant's answer, `PreCompact` preserves memory across context resets, and `SessionEnd` syncs the session into the permanent graph.
See the [plugin README](https://github.com/topoteretes/cognee-integrations/tree/main/integrations/claude-code) for sessions, datasets, and full configuration.
## MCP Setup
Use MCP if you already run Cognee MCP and want Claude Code to call Cognee tools through the Model Context Protocol.
<Steps>
<Step title="Navigate to your project">
```bash theme={null}
cd /path/to/your/project
```
</Step>
<Step title="Add the Cognee MCP server">
Choose the command that matches how you started Cognee MCP:
<Tabs>
<Tab title="Docker (HTTP)">
Use this if you started Cognee MCP with Docker:
```bash theme={null}
claude mcp add --transport http cognee http://localhost:8000/mcp -s project
```
This creates a configuration in your project's `.mcp.json` file that connects to the HTTP endpoint.
**Options:**
* `-s project`: Stores configuration in the project and requires approval per project
* Omit `-s project`: Stores configuration at user level
</Tab>
<Tab title="Local (stdio)">
Use this if you cloned the repository and run Cognee MCP from source:
```bash theme={null}
claude mcp add cognee \
-s project \
-e LLM_API_KEY="your-openai-key" \
-- uv --directory /absolute/path/to/cognee-mcp run cognee-mcp
```
Replace:
* `your-openai-key` with your OpenAI API key
* `/absolute/path/to/cognee-mcp` with the path to your local `cognee-mcp` directory
</Tab>
</Tabs>
</Step>
<Step title="Start Claude Code and approve MCP">
```bash theme={null}
claude
```
On first run in this project, Claude Code asks whether to approve project MCP servers. Select **Enable** or press Enter.
</Step>
<Step title="Use Cognee tools">
Claude Code will use Cognee tools when relevant to your requests. You can explicitly ask:
* "Remember this design note in Cognee"
* "Recall authentication patterns from Cognee"
* "Improve the project memory in Cognee"
</Step>
</Steps>
## Where MCP Configuration Lives
The MCP configuration is stored in `.mcp.json` in your project directory when you pass `-s project`, or in your user settings when you omit it. Claude Code reads this file when starting a session in that directory.
## Need Help?
<Card title="Join Our Community" icon="discord" href="https://discord.gg/m63hxKsp4p">
Get support and connect with other developers using Cognee.
</Card>
# Claude Desktop with Cognee MCP
Source: https://docs.cognee.ai/cognee-mcp/integrations/claude-desktop
Connect Claude Desktop to Cognee MCP on macOS or Windows.
Claude Desktop is Anthropic's native app for macOS and Windows. It supports MCP servers through a config file that you edit manually — no command-line setup required after initial configuration.
## Prerequisites
* [Claude Desktop](https://claude.ai/download) installed
* Cognee MCP server available (see [Quickstart](/cognee-mcp/mcp-quickstart) or [Local Setup](/cognee-mcp/mcp-local-setup))
* OpenAI API key (for standalone mode) **or** Cognee Cloud credentials (for Cloud-connected mode)
## Config file location
Find the `claude_desktop_config.json` file for your platform:
| Platform | Path |
| -------- | ----------------------------------------------------------------- |
| macOS | `~/Library/Application Support/Claude/claude_desktop_config.json` |
| Windows | `%APPDATA%\Claude\claude_desktop_config.json` |
If the file does not exist yet, create it (and any missing parent directories).
## Setup Steps
<Steps>
<Step title="Open the config file">
<Tabs>
<Tab title="macOS">
```bash theme={null}
open ~/Library/Application\ Support/Claude/
```
Open `claude_desktop_config.json` in your editor of choice.
</Tab>
<Tab title="Windows">
Press `Win + R`, type `%APPDATA%\Claude` and press Enter.
Open `claude_desktop_config.json` in Notepad or your editor.
</Tab>
</Tabs>
</Step>
<Step title="Add the Cognee server">
Choose the mode that matches your setup:
<Tabs>
<Tab title="Cloud-connected">
Point the MCP server at your Cognee Cloud tenant with `--api-url` and `--api-token`, using your API Base URL and API key from the [API Keys](/cognee-cloud/ui/api-keys) page. These are the flags the MCP tools use to reach your tenant. This runs via `uvx` (requires [uv](https://docs.astral.sh/uv/) — see [Troubleshooting](#troubleshooting) if `uvx` isn't found) and is the same config the in-product [Integrations](/cognee-cloud/ui/integrations) page generates:
```json theme={null}
{
"mcpServers": {
"cognee": {
"command": "uvx",
"args": [
"cognee-mcp@latest",
"--api-url", "https://your-tenant.aws.cognee.ai",
"--api-token", "your-cognee-cloud-api-key"
]
}
}
}
```
</Tab>
<Tab title="Standalone (local)">
Run the MCP server locally with its own knowledge graph. Requires an LLM API key:
```json theme={null}
{
"mcpServers": {
"cognee": {
"command": "cognee-mcp",
"args": ["--transport", "stdio"],
"env": {
"LLM_API_KEY": "your-openai-api-key"
}
}
}
}
```
If you installed from source with `uv`, use the full `uv run` invocation:
```json theme={null}
{
"mcpServers": {
"cognee": {
"command": "uv",
"args": [
"--directory", "/absolute/path/to/cognee-mcp",
"run", "cognee-mcp",
"--transport", "stdio"
],
"env": {
"LLM_API_KEY": "your-openai-api-key"
}
}
}
}
```
Replace `/absolute/path/to/cognee-mcp` with the actual path to your cloned `cognee-mcp` directory.
</Tab>
<Tab title="Remote / self-hosted URL">
Connect to a Cognee MCP server that is **already running** and exposed on a public domain — for example, a self-hosted backend where you started the server with `cognee-mcp --transport http --host 0.0.0.0` behind a reverse proxy. Claude Desktop only launches `stdio` servers directly, so use the [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) bridge to reach an HTTP/SSE URL. This requires [Node.js](https://nodejs.org) (for `npx`).
```json theme={null}
{
"mcpServers": {
"cognee": {
"command": "npx",
"args": [
"-y", "mcp-remote",
"https://cognee.example.com/mcp"
]
}
}
}
```
Use the `/mcp` path for an HTTP-transport server and `/sse` for an SSE-transport server. If the endpoint is protected (for example, a bearer token enforced at your reverse proxy), pass it as a header:
```json theme={null}
{
"mcpServers": {
"cognee": {
"command": "npx",
"args": [
"-y", "mcp-remote",
"https://cognee.example.com/mcp",
"--header", "Authorization: Bearer your-token"
]
}
}
}
```
On the server, whitelist the public hostname with `MCP_ALLOWED_HOSTS` (and browser origins with `MCP_CORS_ALLOW_ORIGINS`), or the transport's DNS-rebinding protection rejects the requests. See [Transport Security](/cognee-mcp/mcp-local-setup).
</Tab>
</Tabs>
</Step>
<Step title="Restart Claude Desktop">
Fully quit and reopen Claude Desktop. On macOS, use **Cmd + Q** (not just closing the window) to ensure the app reloads the config.
</Step>
<Step title="Verify the connection">
Open a new conversation in Claude Desktop and ask:
> "What Cognee tools do you have available?"
Claude should list tools such as `remember`, `recall`, and `forget`. If the tools don't appear, check [Troubleshooting](#troubleshooting) below.
</Step>
<Step title="Use Cognee tools">
Claude Desktop will use Cognee tools automatically when relevant. You can also prompt explicitly:
* "Remember this in Cognee: we use PostgreSQL for the main database"
* "Recall what you know about our database choices from Cognee"
</Step>
</Steps>
## Troubleshooting
<AccordionGroup>
<Accordion title="Tools don't appear after restart">
* Confirm the config file is valid JSON (no trailing commas, correct bracket nesting).
* On macOS, make sure you fully quit the app with **Cmd + Q**.
* For the `uvx` (Cloud-connected) config: confirm [uv](https://docs.astral.sh/uv/) is installed — run `uvx --version` in a terminal. On Windows, a missing `uv` install is the most common cause (`'uvx' is not recognized...`); install it with `powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"` and restart both your terminal and Claude Desktop.
* For the standalone `cognee-mcp` command form: check that it's on your `PATH` — run `which cognee-mcp` (macOS/Linux) or `where cognee-mcp` (Windows). If it isn't found, use the full path in the `command` field.
* For source installs, verify the `--directory` path points to the folder that contains `pyproject.toml`.
</Accordion>
<Accordion title="Authentication errors (Cloud mode)">
* Confirm the `--api-url` value matches exactly the API Base URL on your [API Keys](/cognee-cloud/ui/api-keys) page — no trailing slash.
* Regenerate your API key if it may have expired and update the config.
* Verify the key value has no surrounding quotes or extra whitespace.
</Accordion>
<Accordion title="Can't connect to a remote MCP URL">
* Confirm the URL path matches the server transport: `/mcp` for HTTP, `/sse` for SSE.
* Make sure `npx` (Node.js) is installed and on your `PATH` — run `npx --version`.
* If the connection is rejected, whitelist your public hostname on the server with `MCP_ALLOWED_HOSTS` and set `MCP_CORS_ALLOW_ORIGINS`. See [Transport Security](/cognee-mcp/mcp-local-setup).
* Open the URL's health endpoint in a browser (e.g. `https://cognee.example.com/health`) to confirm the server is reachable from outside your network.
</Accordion>
<Accordion title="Server crashes on startup">
Run `cognee-mcp --transport stdio` directly in a terminal to see error output. Common causes are a missing `LLM_API_KEY` in standalone mode or an unreachable `--api-url` endpoint.
</Accordion>
</AccordionGroup>
## Need Help?
<Card title="Join Our Community" icon="discord" href="https://discord.gg/m63hxKsp4p">
Get support and connect with other developers using Cognee MCP.
</Card>
# Cline
Source: https://docs.cognee.ai/cognee-mcp/integrations/cline
Connect the Cline VS Code extension to Cognee MCP.
# Cline Integration
Cline is a VS Code extension that provides AI assistance with support for MCP servers. It enables natural language interactions with external tools directly in your development environment.
## Prerequisites
* Visual Studio Code installed
* Cline extension installed
* Cognee MCP server running (see [Quickstart](/cognee-mcp/mcp-quickstart) or [Local Setup](/cognee-mcp/mcp-local-setup))
* OpenAI API key
## Setup Steps
<Steps>
<Step title="Install Cline">
1. Open Visual Studio Code
2. Go to the Extensions panel
3. Search for "Cline" or visit the [Marketplace page](https://marketplace.visualstudio.com/items?itemName=saoudrizwan.claude-dev)
4. Click Install
</Step>
<Step title="Open Cline MCP Settings">
Follow the [Cline MCP configuration guide](https://docs.cline.bot/mcp/configuring-mcp-servers) to access settings:
1. Click the "MCP Servers" icon in the top navigation bar of the Cline extension
2. Select the "Configure" tab
3. Click the "Configure MCP Servers" button at the bottom of the pane
Cline will open the `cline_mcp_settings.json` file with the base structure:
```json theme={null}
{
"mcpServers": {
}
}
```
</Step>
<Step title="Add Cognee Server Configuration">
Add the Cognee server inside the `mcpServers` object. Choose the configuration that matches how you started the Cognee MCP server:
<Tabs>
<Tab title="Docker (SSE)">
Use this if you want Cline to connect to a Docker-hosted Cognee MCP server over SSE:
```bash theme={null}
docker run -e TRANSPORT_MODE=sse --env-file ./.env -p 8000:8000 --rm -it cognee/cognee-mcp:main
```
```json theme={null}
{
"mcpServers": {
"cognee": {
"url": "http://localhost:8000/sse",
"disabled": false
}
}
}
```
This configuration tells Cline to connect to the SSE endpoint exposed by the Docker container.
</Tab>
<Tab title="Local (stdio)">
Use this if you cloned the repository and run from source:
```json theme={null}
{
"mcpServers": {
"cognee": {
"command": "uv",
"args": [
"--directory",
"/absolute/path/to/cognee-mcp",
"run",
"cognee-mcp"
],
"env": {
"LLM_API_KEY": "your-openai-key"
},
"disabled": false
}
}
}
```
Replace:
* `/absolute/path/to/cognee-mcp` with the full path to your cognee-mcp directory
* `your-openai-key` with your OpenAI API key
</Tab>
</Tabs>
Save the file after adding your configuration. The Cognee server will appear in the MCP Servers panel. You can use the toggle to enable/disable it or click Restart if needed.
</Step>
<Step title="Confirm the Tools Are Enabled">
In the Cline MCP Servers panel:
1. Make sure the Cognee server toggle is enabled
2. Click restart if the server shows an old status
3. Expand the server entry and verify that tools such as `remember`, `recall`, and `forget` are listed
</Step>
<Step title="Use Cognee Tools">
Open the Cline interface.
Example commands:
* "Remember this repository note in Cognee" - Store new memory
* "Recall authentication logic from Cognee" - Query graph or session memory
* "Improve the onboarding dataset in Cognee" - Run an enrichment pass
</Step>
</Steps>
## Where to Use This Configuration
The `cline_mcp_settings.json` file is in your VS Code global storage directory. Cline reads this file when the extension starts and applies the configuration to all projects.
You can manage servers through the Cline UI - click the "MCP Servers" icon to enable/disable servers, restart them, or adjust settings without editing JSON directly. See the [Cline MCP documentation](https://docs.cline.bot/mcp/configuring-mcp-servers) for details.
## Need Help?
<Card title="Join Our Community" icon="discord" href="https://discord.gg/m63hxKsp4p">
Get support and connect with other developers using Cognee MCP.
</Card>
# Codex with Cognee MCP
Source: https://docs.cognee.ai/cognee-mcp/integrations/codex
Register Cognee MCP with the OpenAI Codex CLI agent.
Codex is OpenAI's coding agent with built-in MCP support. You can register Cognee MCP once, then use Cognee memory and retrieval tools directly in your Codex sessions.
<Tip>
For most users, the [**Cognee memory plugin**](/integrations/codex-integration) is the simpler path — it adds persistent memory with a few commands and no separate MCP server to run. The MCP setup below is for connecting Cognee's full MCP tool set.
</Tip>
## Prerequisites
* Codex CLI installed and authenticated
* Cognee MCP server running (see [Quickstart](/cognee-mcp/mcp-quickstart) or [Local Setup](/cognee-mcp/mcp-local-setup))
* OpenAI API key (for Cognee's LLM operations)
## Setup Steps
<Steps>
<Step title="Add Cognee MCP Server">
Choose the command that matches how you started the Cognee MCP server:
<Tabs>
<Tab title="Docker (HTTP)">
Use this if you started the server with Docker:
```bash theme={null}
docker run -e TRANSPORT_MODE=http --env-file ./.env -p 8000:8000 --rm -it cognee/cognee-mcp:main
codex mcp add cognee --url http://localhost:8000/mcp
```
</Tab>
<Tab title="Local (stdio)">
Use this if you cloned the repository and run from source:
```bash theme={null}
codex mcp add cognee \
--env LLM_API_KEY="your-openai-key" \
-- uv --directory /absolute/path/to/cognee-mcp run cognee-mcp
```
Replace:
* `your-openai-key` with your OpenAI API key
* `/absolute/path/to/cognee-mcp` with the path to your local `cognee-mcp` directory
</Tab>
</Tabs>
If a `cognee` server entry already exists, remove it first:
```bash theme={null}
codex mcp remove cognee
```
</Step>
<Step title="Verify Server Registration">
```bash theme={null}
codex mcp list
```
You should see `cognee` in the MCP server list.
</Step>
<Step title="Start Codex and Use Cognee Tools">
```bash theme={null}
codex
```
Codex can now call Cognee tools automatically when relevant.
Example prompts:
* "Remember this repository note in Cognee: the billing API uses Stripe webhooks for subscription updates"
* "Recall authentication logic from Cognee before editing this file"
* "Improve the current Cognee dataset with the architecture notes in this repository"
* "Use Cognee to find prior decisions about database migrations"
</Step>
</Steps>
## Where to Use This Configuration
Codex stores MCP server settings in `~/.codex/config.toml` under the `mcp_servers` section. Once added, the Cognee server is available to Codex sessions unless removed or disabled.
You can also configure the mcp server directly in the `config.toml` file.
## Need Help?
<Card title="Join Our Community" icon="discord" href="https://discord.gg/m63hxKsp4p">
Get support and connect with other developers using Cognee MCP.
</Card>
# Continue
Source: https://docs.cognee.ai/cognee-mcp/integrations/continue
Configure Continue to use Cognee MCP in VS Code or JetBrains.
# Continue Integration
Continue is an open-source AI coding assistant for VS Code and JetBrains IDEs. It supports MCP servers through YAML configuration files in your workspace.
## Prerequisites
* VS Code or JetBrains IDE installed
* Continue extension installed
* Cognee MCP server running (see [Quickstart](/cognee-mcp/mcp-quickstart) or [Local Setup](/cognee-mcp/mcp-local-setup))
* OpenAI API key
## Setup Steps
<Steps>
<Step title="Install Continue">
1. Open your IDE
2. Install the Continue extension from the marketplace ([documentation](https://www.continue.dev))
</Step>
<Step title="Create MCP Configuration Directory">
Create a folder called `.continue/mcpServers` at the top level of your workspace:
```bash theme={null}
mkdir -p .continue/mcpServers
```
</Step>
<Step title="Add Cognee MCP Configuration">
Choose the configuration that matches how you started the Cognee MCP server:
<Tabs>
<Tab title="Docker (SSE)">
Start Cognee MCP with SSE transport:
```bash theme={null}
docker run -e TRANSPORT_MODE=sse --env-file ./.env -p 8000:8000 --rm -it cognee/cognee-mcp:main
```
Create a file `.continue/mcpServers/cognee.yaml` with:
```yaml theme={null}
name: Cognee MCP Server
version: 0.0.1
schema: v1
mcpServers:
- name: Cognee
type: sse
url: http://localhost:8000/sse
```
This connects to the SSE endpoint exposed by the Docker container.
</Tab>
<Tab title="Local (stdio)">
Create a file `.continue/mcpServers/cognee.yaml` with:
```yaml theme={null}
name: Cognee MCP Server
version: 0.0.1
schema: v1
mcpServers:
- name: Cognee
type: stdio
command: uv
args:
- --directory
- /absolute/path/to/cognee-mcp
- run
- cognee-mcp
env:
LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
```
Replace `/absolute/path/to/cognee-mcp` with your local path.
</Tab>
</Tabs>
</Step>
<Step title="Use Cognee Tools in Agent Mode">
Open Continue and switch to **Agent mode** (MCP only works in agent mode).
Example prompts:
* "Remember this architecture note in Cognee"
* "Recall authentication logic from Cognee"
* "Improve the project dataset in Cognee"
</Step>
</Steps>
## Where to Use This Configuration
The `.continue/mcpServers/` directory is at the workspace level. Each workspace can have its own MCP server configurations. Continue automatically detects YAML files in this directory.
## Alternative: Using JSON Format
If you have JSON MCP configuration from another tool, you can copy it directly:
```bash theme={null}
# Copy from Cursor, Claude, or Cline
cp ~/.cursor/mcp.json .continue/mcpServers/cognee.json
```
Continue automatically picks up both YAML and JSON configurations.
## Need Help?
<Card title="Join Our Community" icon="discord" href="https://discord.gg/m63hxKsp4p">
Get support and connect with other developers using Cognee MCP.
</Card>
# Cursor with Cognee MCP
Source: https://docs.cognee.ai/cognee-mcp/integrations/cursor
Connect Cursor to Cognee MCP through its Tools & MCP settings.
Cursor is an AI-powered code editor built on VS Code with native support for the Model Context Protocol. It provides AI assistance through its Composer interface and chat panel.
## Prerequisites
* Cursor installed on your machine
* Cognee MCP server running (see [Quickstart](/cognee-mcp/mcp-quickstart) or [Local Setup](/cognee-mcp/mcp-local-setup))
* OpenAI API key
## Setup Steps
<Steps>
<Step title="Open MCP Settings">
1. Launch Cursor
2. Click the gear icon to open Settings
3. Navigate to **Tools & MCP**
4. Click **+ Add MCP Server**
This opens the `mcp.json` configuration file.
</Step>
<Step title="Add Cognee Server Configuration">
Choose the configuration that matches how you started the Cognee MCP server:
<Tabs>
<Tab title="Docker (HTTP)">
Use this if you started the server with Docker:
```json theme={null}
{
"mcpServers": {
"cognee": {
"url": "http://localhost:8000/mcp"
}
}
}
```
This configuration tells Cursor to connect to the HTTP endpoint exposed by the Docker container.
</Tab>
<Tab title="Local (stdio)">
Use this if you cloned the repository and run from source:
```json theme={null}
{
"mcpServers": {
"cognee": {
"command": "uv",
"args": [
"--directory",
"/Users/yourname/path/to/cognee-mcp",
"run",
"cognee-mcp"
]
}
}
}
```
Replace `/Users/yourname/path/to/cognee-mcp` with the absolute path to your local cognee-mcp directory.
</Tab>
</Tabs>
Save the file after adding your configuration.
</Step>
<Step title="Verify Connection">
Use the toggle in MCP Tools to refresh the connection. You should see a list of available tools from Cognee, confirming the server is connected.
</Step>
<Step title="Use Cognee Tools">
1. Open the Chat panel in Cursor
2. Make sure **Agent** mode is selected
3. Issue prompts that use Cognee tools
Example prompts:
* "Remember this file in Cognee"
* "Recall authentication logic from Cognee"
* "Improve the current Cognee dataset"
</Step>
</Steps>
## Where to Use This Configuration
The `mcp.json` file is located in your Cursor settings directory. Cursor reads this file at startup and when you refresh the MCP connection. The configuration applies to all projects you open in Cursor.
## Need Help?
<Card title="Join Our Community" icon="discord" href="https://discord.gg/m63hxKsp4p">
Get support and connect with other developers using Cognee MCP.
</Card>
# Python Agent
Source: https://docs.cognee.ai/cognee-mcp/integrations/python-agent
Call Cognee MCP tools from your own Python agent with the MCP SDK.
Connect your own Python LLM agent to Cognee MCP to give it persistent knowledge graph memory. The `mcp` Python SDK lets you call all [Cognee MCP tools](/cognee-mcp/mcp-tools) programmatically, without an IDE or chat client.
<Info>
The MCP server exposes the v1.0 memory tools (`remember`, `recall`, `forget`) plus `cognify_status`. For lower-level control — explicit search types beyond `recall`'s options, custom graph models, dataset listing and creation — use the Python SDK or REST API.
</Info>
## Prerequisites
* Python 3.10+
* `uv` installed
* `LLM_API_KEY` environment variable set (OpenAI key or equivalent)
* `mcp` package installed in your agent environment:
```bash theme={null}
uv pip install "mcp>=1.12.0"
```
## Connection Options
Choose the transport that matches how you want your Python code to connect to Cognee MCP. Each option below creates the same kind of initialized `ClientSession`; the tool-calling code is shared in the next section.
<Tabs>
<Tab title="stdio">
Use stdio when your Python process should launch Cognee MCP as a subprocess and communicate over stdin/stdout.
```bash theme={null}
git clone https://github.com/topoteretes/cognee.git
cd cognee/cognee-mcp
uv sync --dev --all-extras
```
```python theme={null}
import os
from contextlib import asynccontextmanager
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
@asynccontextmanager
async def connect_to_cognee():
server_params = StdioServerParameters(
command="uv",
args=[
"--directory",
"/absolute/path/to/cognee/cognee-mcp",
"run",
"cognee-mcp",
],
env={**os.environ, "LLM_API_KEY": os.environ["LLM_API_KEY"]},
)
async with stdio_client(server_params) as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
yield session
```
Replace `/absolute/path/to/cognee/cognee-mcp` with the absolute path to the `cognee-mcp` directory in your cloned repository.
</Tab>
<Tab title="HTTP">
Use HTTP when Cognee MCP is already running as a local or remote server with Streamable HTTP enabled.
```bash theme={null}
docker run -e TRANSPORT_MODE=http --env-file ./.env -p 8000:8000 --rm -it cognee/cognee-mcp:main
```
```python theme={null}
from contextlib import asynccontextmanager
from mcp import ClientSession
from mcp.client.streamable_http import streamable_http_client
@asynccontextmanager
async def connect_to_cognee():
async with streamable_http_client("http://localhost:8000/mcp") as (read, write, _):
async with ClientSession(read, write) as session:
await session.initialize()
yield session
```
</Tab>
<Tab title="SSE">
Use SSE when your MCP server or client requires the older Server-Sent Events transport.
```bash theme={null}
docker run -e TRANSPORT_MODE=sse --env-file ./.env -p 8000:8000 --rm -it cognee/cognee-mcp:main
```
```python theme={null}
from contextlib import asynccontextmanager
from mcp import ClientSession
from mcp.client.sse import sse_client
@asynccontextmanager
async def connect_to_cognee():
async with sse_client("http://localhost:8000/sse") as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
yield session
```
</Tab>
</Tabs>
## Send Requests
After you define one of the `connect_to_cognee()` functions above, the rest of your agent code is transport-agnostic. This example stores a fact with `remember`, then retrieves it with `recall`.
```python theme={null}
import asyncio
async def main():
async with connect_to_cognee() as session:
await session.call_tool(
"remember",
arguments={
"data": "Acme Corp signed a $1.2M healthcare contract in Q1 2025.",
"dataset_name": "sales",
},
)
result = await session.call_tool(
"recall",
arguments={
"query": "healthcare contracts",
"search_type": "GRAPH_COMPLETION",
},
)
print(result.content[0].text)
asyncio.run(main())
```
## Inject context into your LLM calls
Once you have the retrieved context string, pass it to your LLM as part of the system or user prompt:
```python theme={null}
import openai
client = openai.AsyncOpenAI()
async def answer_with_memory(session, question: str) -> str:
result = await session.call_tool(
"recall",
arguments={
"query": question,
"search_type": "GRAPH_COMPLETION",
},
)
context = result.content[0].text
response = await client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": f"Use this context to answer:\n\n{context}"},
{"role": "user", "content": question},
],
)
return response.choices[0].message.content
```
## Key tools for agent context
| Tool | Purpose |
| ---------------- | ------------------------------------------------------------------- |
| `remember` | v1.0 API - store data with optional session scoping |
| `recall` | v1.0 API - smart retrieval with session awareness |
| `forget` | v1.0 API - delete a single data item, a dataset, or wipe everything |
| `cognify_status` | Poll background ingestion started with `remember(background=True)` |
See the [Tools Reference](/cognee-mcp/mcp-tools) for all available tools and parameters.
## Need Help?
<Card title="Join Our Community" icon="discord" href="https://discord.gg/m63hxKsp4p">
Get support and connect with other developers using Cognee MCP.
</Card>
# Cognee Cloud & MCP
Source: https://docs.cognee.ai/cognee-mcp/mcp-cloud-connection
Connect Cognee MCP to your Cognee Cloud tenant with an API base URL and API key.
## Connecting MCP to Cognee Cloud
The Cognee MCP server can connect to your Cognee Cloud tenant using the `--api-url` and `--api-token` flags, with the API Base URL and API key from your [API Keys](/cognee-cloud/ui/api-keys) page. This lets any MCP-compatible client (Claude Desktop, Cursor, VS Code Copilot) work with your cloud-hosted knowledge graph.
| | Cognee MCP | Cognee Cloud |
| ------------------ | ----------------------------------------------------------------------------- | ----------------------------------------------------- |
| **Where it runs** | Locally on your machine | Hosted by Cognee |
| **Access method** | MCP protocol | REST API via `--api-url`, or Cloud UI |
| **Authentication** | Bearer token by default, or `X-Api-Key` with `--api-auth-scheme` | `X-Api-Key` header |
| **API endpoints** | Self-hosted backend endpoints such as `/api/v1/remember` and `/api/v1/recall` | The same REST endpoints on your tenant's API Base URL |
| **Use case** | AI IDE tools (Cursor, Claude Code, etc.) | Cloud-managed knowledge graphs |
### How API mode connects to a backend
The MCP server's `--api-url` / `--api-token` options put it in **API mode**: instead of running pipelines locally, the MCP tools send their requests to the Cognee backend at that URL. API mode works with both self-hosted backends and Cognee Cloud tenants:
* Self-hosted backends authenticate with `Authorization: Bearer <token>` by default
* Cognee Cloud tenant URLs are detected automatically and authenticate with the `X-Api-Key` header, plus an `X-Tenant-Id` header carrying the tenant ID
Detection keys on the URL itself: the MCP server looks for a `tenant-<uuid>` label in `--api-url` and reads the tenant ID from it. A real tenant host looks like `https://tenant-0b7c1f2e-3a4d-5e6f-8a9b-0c1d2e3f4a5b.aws.cognee.ai` — the `your-tenant` placeholder used in examples throughout these docs stands in for that label. Copy the API Base URL from your [API Keys](/cognee-cloud/ui/api-keys) page verbatim so the label is preserved — an `--api-url` without it is treated as a self-hosted backend and falls back to `Authorization: Bearer <token>`, which a Cloud tenant rejects.
URL detection is only the default. `--api-auth-scheme x-api-key` (or `COGNEE_API_AUTH_SCHEME=x-api-key`) forces the `X-Api-Key` header on any backend — which is what a self-hosted instance needs when you authenticate with a server-issued API key rather than a login JWT. `--api-auth-scheme bearer` forces `Authorization: Bearer` even on a tenant URL, so avoid setting `COGNEE_API_AUTH_SCHEME=bearer` in an environment that also talks to Cloud.
<Note>
The `--serve-url` / `--serve-api-key` flags are a different mechanism: they call `cognee.serve()` to route SDK operations, and they do not configure the API client the MCP tools use — so a Cloud connection configured only with `--serve-url` leaves some tools working against local storage. For Cloud connections, use `--api-url` / `--api-token`.
</Note>
## End-to-end walkthrough
<Steps>
<Step title="Get your API credentials">
Open the [API Keys](/cognee-cloud/ui/api-keys) page in the Cognee Cloud console. Copy:
* **API Base URL** — looks like `https://your-tenant.aws.cognee.ai`
* **API Key** — a long token used to authenticate requests
</Step>
<Step title="Start the MCP server">
Pass your credentials via CLI flags:
```bash theme={null}
cognee-mcp --transport sse --port 8001 \
--api-url https://your-tenant.aws.cognee.ai \
--api-token your-api-key
```
The server is ready when you see output like:
```
INFO: Started server process
INFO: Waiting for connections on http://127.0.0.1:8001
```
</Step>
<Step title="Add Cognee to your MCP client">
Add the server to your client's MCP configuration. The config file location varies by client — see the [integrations](/cognee-mcp/integrations) section for your specific tool.
```json theme={null}
{
"mcpServers": {
"cognee": {
"url": "http://localhost:8001/sse"
}
}
}
```
</Step>
<Step title="Verify the connection">
Test that memory operations reach your Cloud tenant. In your MCP client, ask:
> "Use Cognee to remember: cloud connection test successful"
Then retrieve it:
> "Use Cognee to recall the cloud connection test"
If Cognee returns the stored value, the end-to-end connection is working. You can also confirm the data appeared in the [Cognee Cloud UI](/cognee-cloud/ui/datasets).
</Step>
</Steps>
## Available tools
Once connected, your MCP client gets the Cognee API v1 memory tools:
| Tool | Description |
| ---------- | ------------------------------------------------ |
| `remember` | Store data in memory (add + cognify in one step) |
| `recall` | Search memory with auto-routing |
| `forget` | Delete data from memory |
## Other connection options
<AccordionGroup>
<Accordion title="Use Cognee MCP for local AI memory (standalone)">
Run the MCP server in **standalone mode** (no Cloud connection). The server manages its own local knowledge graph.
```bash theme={null}
# Standalone mode — requires LLM_API_KEY
LLM_API_KEY=sk-... cognee-mcp --transport sse --port 8001
```
This is the simplest way to add persistent memory to Cursor, Claude Code, Cline, and other MCP-compatible tools.
</Accordion>
<Accordion title="Use the Python SDK instead">
Access Cognee Cloud programmatically using the [`cognee` SDK](/cognee-cloud/connections/cloud-sdk) connected through `serve()`, which handles authentication and communication with the hosted service.
```bash theme={null}
export COGNEE_SERVICE_URL="https://your-tenant.aws.cognee.ai"
export COGNEE_API_KEY="your-cognee-cloud-api-key"
```
```python theme={null}
import asyncio
import cognee
async def main():
await cognee.serve() # Reads COGNEE_SERVICE_URL and COGNEE_API_KEY
await cognee.remember("...", dataset_name="my_dataset")
results = await cognee.recall("...", datasets=["my_dataset"])
print(results)
asyncio.run(main())
```
If you prefer not to use environment variables, pass both values directly:
```python theme={null}
await cognee.serve(
url="https://your-tenant.aws.cognee.ai",
api_key="your-api-key",
)
```
See the [Cloud SDK guide](/cognee-cloud/connections/cloud-sdk) for a complete walkthrough.
</Accordion>
<Accordion title="Connect MCP to a self-hosted Cognee backend">
If you want multiple AI clients to share a single knowledge graph, run a self-hosted Cognee backend and point the MCP server at it using `API_URL` and `API_TOKEN`:
```bash theme={null}
# 1. Start a self-hosted Cognee backend
docker run -e LLM_API_KEY=your_key -p 8080:8000 --rm -it cognee/cognee:main
# 2. Start MCP in API mode pointing to your backend
docker run \
-e TRANSPORT_MODE=sse \
-e API_URL=http://localhost:8080 \
-e API_TOKEN=your_backend_token \
-p 8000:8000 --rm -it cognee/cognee-mcp:main
```
See the [MCP Quickstart](/cognee-mcp/mcp-quickstart#api-mode-shared-knowledge-graph) for full details on this pattern.
</Accordion>
</AccordionGroup>
# Local Setup
Source: https://docs.cognee.ai/cognee-mcp/mcp-local-setup
Build and run Cognee MCP from source with stdio, SSE, or HTTP transport.
Build and run Cognee MCP from source to access advanced customization, multiple transport options, and the latest development features, including the current memory-oriented MCP tools.
## Advantages of Local Setup
* **Full Control**: Customize server configuration, add providers, and modify behavior
* **Latest Features**: Access development features before they reach Docker releases
* **Multiple Transports**: Choose stdio, SSE, or HTTP transport modes
* **Current Tool Surface**: Use `remember`, `recall`, and `forget` alongside compatibility tools
* **Development Ready**: Debug, modify, and contribute to the codebase
## Setup Steps
<Steps>
<Step title="Clone Repository">
```bash theme={null}
git clone https://github.com/topoteretes/cognee.git
cd cognee
```
</Step>
<Step title="Create Environment File">
Create a `.env` file with your configuration:
```bash theme={null}
LLM_API_KEY="your-openai-api-key"
```
<Note>
**No API key?** If your MCP host grants the `sampling` capability, set `LLM_PROVIDER="mcp-sampling"` and omit `LLM_API_KEY` — Cognee delegates completions to the host's own model. You still need an embedding provider for vector search, and host support varies. See [MCP Sampling](/setup-configuration/llm-providers#provider-setup-guides) on the LLM Providers page.
</Note>
</Step>
<Step title="Install Dependencies">
First, install the [uv](https://docs.astral.sh/uv/getting-started/installation/) package manager for your operating system:
<Tabs>
<Tab title="macOS / Linux">
```bash theme={null}
# via Homebrew
brew install uv
# or via the standalone installer
curl -LsSf https://astral.sh/uv/install.sh | sh
```
</Tab>
<Tab title="Windows">
```powershell theme={null}
# in PowerShell
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
```
</Tab>
<Tab title="Any (pip)">
```bash theme={null}
pip install uv
```
</Tab>
</Tabs>
Then install the project dependencies. These commands are the same on every platform:
```bash theme={null}
cd cognee-mcp
uv sync --dev --all-extras --reinstall
```
<Note>
**When to re-run `uv sync`.** Re-run it after pulling new commits, switching branches, or updating Cognee — anytime `pyproject.toml` or `uv.lock` may have changed. For routine refreshes, use `uv sync --dev --all-extras`; keep `--reinstall` for the first install or when your environment looks stale or broken.
</Note>
</Step>
<Step title="Activate and Run">
```bash theme={null}
# Run with default stdio transport
uv run cognee-mcp
```
</Step>
</Steps>
## Running in API Mode
To connect the MCP server to an existing Cognee backend instead of running standalone:
```bash theme={null}
# Start MCP in HTTP or SSE mode pointing to the backend
uv run cognee-mcp --transport http --api-url http://localhost:8080
# Optional: add an auth token if the backend requires it
uv run cognee-mcp --transport http --api-url http://localhost:8080 --api-token your_backend_token
```
When `--api-url` is provided, the MCP server acts as an interface to the centralized backend. This allows multiple MCP instances and clients to share the same knowledge graph.
`--api-token` is read once at startup and sent with every backend request, so the backend resolves all traffic from one MCP process to a single Cognee user. The tools carry no caller identity of their own — `remember`, `recall`, and `forget` take a dataset and an optional `session_id`, never a user. Dataset names therefore organize that one user's memory rather than isolating it: any client connected to the process can name any dataset. To separate tenants, run one MCP process per tenant, each started with a token for its own backend user, so the [permissions system](/core-concepts/multi-user-mode/permissions-system/overview) enforces the boundary. See [Tenant isolation](/how-to-guides/cognee-sdk/deployment/deployment-options#tenant-isolation) for how the logical, backend, and infrastructure levels compare.
You can also pass these as command-line arguments:
```bash theme={null}
uv run cognee-mcp --transport http --api-url http://localhost:8080 --api-token your_token
```
<Note>
The source-run `uv run cognee-mcp` entrypoint reads the `--api-url`, `--api-token` and `--api-auth-scheme` flags, each falling back to an environment variable when the flag is omitted: `COGNEE_BASE_URL`, `COGNEE_API_KEY` and `COGNEE_API_AUTH_SCHEME` respectively. The differently named `API_URL` and `API_TOKEN` variables are read by the Docker entrypoint wrapper, which turns them into flags; that wrapper builds no `--api-auth-scheme` argument, so in Docker set `COGNEE_API_AUTH_SCHEME` as a container environment variable instead.
</Note>
**Use cases:**
* Team collaboration with shared memory
* Multiple AI clients accessing consistent data
* Centralized knowledge graph management
## Further details
<AccordionGroup>
<Accordion title="Transport Modes">
Choose the transport mode based on your client requirements:
<Tabs>
<Tab title="stdio (default)">
Default mode for most MCP clients. The client starts the server as a subprocess and communicates through standard input/output.
```bash theme={null}
uv run cognee-mcp
# or equivalently:
uv run cognee-mcp --transport stdio
```
Configure your MCP client to launch the server directly:
```json theme={null}
{
"mcpServers": {
"cognee": {
"command": "uv",
"args": [
"--directory", "/absolute/path/to/cognee-mcp",
"run", "cognee-mcp"
]
}
}
}
```
Replace `/absolute/path/to/cognee-mcp` with the actual path to your cloned `cognee-mcp` directory.
</Tab>
<Tab title="HTTP">
HTTP transport mode. The server starts on `http://127.0.0.1:8000` and exposes a Streamable HTTP endpoint at `/mcp`.
```bash theme={null}
uv run cognee-mcp --transport http
# Custom host/port:
uv run cognee-mcp --transport http --host 0.0.0.0 --port 9000
```
Configure your MCP client to connect to the HTTP endpoint:
```json theme={null}
{
"mcpServers": {
"cognee": {
"url": "http://localhost:8000/mcp"
}
}
}
```
Verify the server is running with the health check endpoint:
```bash theme={null}
curl http://localhost:8000/health
```
</Tab>
<Tab title="SSE">
Server-Sent Events transport mode. The server starts on `http://127.0.0.1:8000` and exposes an SSE endpoint at `/sse`.
```bash theme={null}
uv run cognee-mcp --transport sse
# Custom host/port:
uv run cognee-mcp --transport sse --host 0.0.0.0 --port 9000
```
Configure your MCP client to connect to the SSE endpoint:
```json theme={null}
{
"mcpServers": {
"cognee": {
"url": "http://localhost:8000/sse"
}
}
}
```
</Tab>
</Tabs>
<Tip>
If you encounter errors on first run, reset your MCP configuration and restart.
</Tip>
</Accordion>
<Accordion title="Default databases (and how to change them)">
In standalone mode the MCP server runs the full Cognee stack in-process, so it uses Cognee's standard defaults — [SQLite](/setup-configuration/relational-databases) for relational metadata, [LanceDB](/setup-configuration/vector-stores) for embeddings, and the embedded [Ladybug (Kuzu)](/setup-configuration/graph-stores) engine for the knowledge graph. All three live as files under `SYSTEM_ROOT_DIRECTORY`, so no external database service is required, and the same defaults apply to the Docker image from the [Quickstart](/cognee-mcp/mcp-quickstart). Each linked page documents what that database stores, its default file location, and the env vars to switch providers; the [Setup Configuration overview](/setup-configuration/overview) covers `SYSTEM_ROOT_DIRECTORY` handling and the `.env` workflow.
In [API mode](#running-in-api-mode) these settings are irrelevant to the MCP process — the backend it points at owns the databases, so configure them there instead.
Two MCP-specific notes:
* The embedded Ladybug/Kuzu graph store uses file-based locking, so several MCP server processes sharing one `SYSTEM_ROOT_DIRECTORY` — for example separate editor windows each spawning their own stdio server — contend for the same graph file. [Neo4j](/setup-configuration/graph-stores) is the recommended upgrade path for that setup.
* The `cognee-mcp` package already depends on `cognee[postgres-binary,docs,neo4j]`, so the Neo4j and Postgres/PGVector drivers ship with it — switching providers needs only the environment variables, not the extra installs the setup pages mention.
Switching providers does not migrate existing memory — re-run your ingestion (`remember`) against the new backend to repopulate it.
</Accordion>
<Accordion title="Token lifetime for long-running MCP servers">
Against a self-hosted backend, `--api-token` is sent as `Authorization: Bearer <token>` by default. (The MCP server switches to the `X-Api-Key` header when `--api-url` carries a `tenant-<uuid>` label, and whenever you pass `--api-auth-scheme x-api-key` — see [Cognee Cloud & MCP](/cognee-mcp/mcp-cloud-connection#how-api-mode-connects-to-a-backend).) When the backend has `REQUIRE_AUTHENTICATION=true`, the token returned by `POST /api/v1/auth/login` is a JWT whose lifetime is controlled by `JWT_LIFETIME_SECONDS` on the backend (default: `3600` — one hour). Once it expires, the MCP server's requests start failing with `401 Unauthorized` until you restart it with a fresh token. The MCP server does **not** automatically re-authenticate.
For deployments where the MCP server runs longer than the JWT lifetime (e.g., MCP and the API hosted as separate services on Railway, Fly.io, ECS, or similar), use one of these patterns:
* **Authenticate with a server-issued API key instead of a login JWT.** Create a key on the backend (`POST /api/v1/auth/api-keys`) and start the MCP server with `--api-auth-scheme x-api-key`, which sends `--api-token` as `X-Api-Key: <token>`. API keys are looked up directly rather than decoded as JWTs, so `JWT_LIFETIME_SECONDS` does not apply to them and there is nothing to expire. Without the flag the key goes out as a Bearer token and every call fails with `401` immediately.
* **Extend the JWT lifetime on the backend.** Set `JWT_LIFETIME_SECONDS` to a value that comfortably exceeds your MCP uptime and restart the backend. See [Security — JWT token settings](/setup-configuration/security#jwt-token-settings) for the full settings.
* **Run the MCP server unauthenticated against a backend that does not require auth.** If the MCP server and the API are co-located on a private network (same VPC, internal Railway/Fly project network, sidecar container), set `REQUIRE_AUTHENTICATION=false` (and `ENABLE_BACKEND_ACCESS_CONTROL=false`) on the backend and omit `--api-token`. Do **not** expose the backend publicly in this mode.
* **Restart the MCP server on a schedule** with a freshly minted token. Re-run `POST /api/v1/auth/login`, capture `access_token`, and pass it on the next `uv run cognee-mcp` (or container restart). This is a workaround, not a substitute for a longer `JWT_LIFETIME_SECONDS`.
The MCP server has no built-in token refresh, so changing `JWT_LIFETIME_SECONDS` on the backend (or removing the auth requirement entirely on a private network) is the recommended fix for "the MCP server stops working after an hour".
</Accordion>
<Accordion title="Server Arguments Reference">
All available arguments for `uv run cognee-mcp`:
| Argument | Default | Description |
| ------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--transport` | `stdio` | Transport protocol: `stdio`, `http`, or `sse` |
| `--host` | `127.0.0.1` | Host to bind the server to (HTTP/SSE only) |
| `--port` | `8000` | Port to bind the server to (HTTP/SSE only) |
| `--path` | `/mcp` | URL path for the HTTP endpoint |
| `--log-level` | `info` | Log verbosity: `debug`, `info`, `warning`, or `error` |
| `--no-migration` | off | Skip database migrations on startup |
| `--tool-mode` | `default` | How many tools appear in `tools/list`: `default`, `minimal`, or `all`. Overrides `COGNEE_MCP_TOOL_MODE` |
| `--api-url` | — | URL of a running Cognee backend (enables API mode). Overrides `COGNEE_BASE_URL` |
| `--api-token` | — | Auth token for the backend API (if required). Overrides `COGNEE_API_KEY` |
| `--api-auth-scheme` | — | API-mode auth header: `bearer` or `x-api-key`. Defaults to `x-api-key` for Cloud tenant URLs and `bearer` otherwise. Overrides `COGNEE_API_AUTH_SCHEME` |
| `--serve-url` | — | Cognee Cloud or remote instance URL used with `cognee.serve()` |
| `--serve-api-key` | — | API key for the `--serve-url` instance |
**Example with all options:**
```bash theme={null}
uv run cognee-mcp \
--transport http \
--host 0.0.0.0 \
--port 8000 \
--api-url http://localhost:8080 \
--api-token your_token
```
</Accordion>
<Accordion title="Transport Security (DNS Rebinding & CORS)">
The HTTP and SSE transports validate the `Host` and `Origin` headers on every request to protect against DNS rebinding attacks. By default, only loopback addresses (`127.0.0.1`, `localhost`, `[::1]`) are accepted. If you bind the server to `0.0.0.0` or a LAN IP — for example, to access it from another machine over SSH or a private network — you must whitelist the hosts you connect with, or requests will be rejected.
| Variable | Default | Description |
| -------------------------------------- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `MCP_ALLOWED_HOSTS` | — | Comma-separated extra `Host` header patterns to accept (e.g. `192.168.1.50:*,myserver.local:*`). Each entry must include the `:*` port glob. Matching `Origin` values (`http://<host>`) are added automatically. |
| `MCP_DISABLE_DNS_REBINDING_PROTECTION` | `false` | Set to `true` to disable all `Host`/`Origin` validation. Use only in trusted networks (LAN, Docker, VPN) where you control all clients. |
| `MCP_CORS_ALLOW_ORIGINS` | `http://localhost:3000` | Comma-separated origins allowed by the CORS middleware on the HTTP and SSE endpoints. |
**Local development (default):** No configuration needed. The server binds to `127.0.0.1` and accepts requests from the loopback defaults.
**Remote deployment — strict (recommended):** Run the MCP server on a remote box and connect from a local AI client. Whitelist the hostname your client uses to reach the MCP server, and set CORS to the browser or app origins that will call it.
```bash theme={null}
# On the remote machine
export MCP_ALLOWED_HOSTS="myserver.local:*,192.168.1.50:*"
export MCP_CORS_ALLOW_ORIGINS="http://localhost:3000,https://chat.example.com"
uv run cognee-mcp --transport http --host 0.0.0.0 --port 8000
```
In this example, `MCP_ALLOWED_HOSTS` covers the MCP server hostnames clients connect to, while `MCP_CORS_ALLOW_ORIGINS` covers the calling page or app origins.
**Remote deployment — permissive:** Disable protection entirely on a trusted private network. This skips both `Host` and `Origin` validation.
```bash theme={null}
export MCP_DISABLE_DNS_REBINDING_PROTECTION=true
uv run cognee-mcp --transport http --host 0.0.0.0 --port 8000
```
<Warning>
Only disable DNS rebinding protection on networks you fully control. With protection off, any website your browser visits could issue requests against the MCP server.
</Warning>
These variables can also be set in your `.env` file alongside `LLM_API_KEY` and other configuration. They apply equally to source runs (`uv run cognee-mcp`) and the Docker image used in the [Quickstart](/cognee-mcp/mcp-quickstart).
</Accordion>
<Accordion title="Tool modes">
The server does not put its whole tool catalog in `tools/list`. By default it lists a small set and makes the rest discoverable through the `search_tools` / `call_tool` pair, which keeps the per-turn tool payload small for connected agents. Tools left out of the list stay registered and callable directly by name.
| Variable | Default | Description |
| ---------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `COGNEE_MCP_TOOL_MODE` | `default` | Which tools appear in `tools/list`. `default` and `minimal` both list `remember`, `recall`, `forget`; `all` lists every registered tool (those three plus `cognify_status`) and installs no search transform. An unrecognized value logs a warning and falls back to `default`. |
```bash theme={null}
# List only the memory API; everything else via search_tools
export COGNEE_MCP_TOOL_MODE=minimal
# Restore the previous flat tool listing
export COGNEE_MCP_TOOL_MODE=all
```
The same choice can be made per process with the `--tool-mode` argument, which takes precedence over the environment variable:
```bash theme={null}
uv run cognee-mcp --tool-mode all
```
<Note>
The mode is resolved when the MCP server process starts, so changes only take effect after you restart the server (or redeploy the container). If an integration or script depends on the full flat `tools/list` response, use `all`. See the [Tools Reference](/cognee-mcp/mcp-tools#tool-modes) for the per-mode tool lists and for how an agent finds and calls an unlisted tool.
</Note>
</Accordion>
<Accordion title="Default recall synthesis prompt">
By default, when a caller invokes the [`recall`](/cognee-mcp/mcp-tools) tool without a `system_prompt`, the request uses Cognee's built-in synthesis prompt. You can set an opt-in, server-side default so that every `recall` that omits `system_prompt` uses a synthesis policy you define once — useful when a single Cognee instance is shared as long-term memory across several MCP clients.
Provide the default with exactly one of these environment variables:
| Variable | Default | Description |
| -------------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------- |
| `COGNEE_MCP_RECALL_SYSTEM_PROMPT` | — | Inline prompt text used as the default synthesis prompt for `recall`. |
| `COGNEE_MCP_RECALL_SYSTEM_PROMPT_FILE` | — | Path to a UTF-8 file whose contents are used as the default synthesis prompt. Read once per `recall` call. |
`COGNEE_MCP_RECALL_SYSTEM_PROMPT` takes precedence over `COGNEE_MCP_RECALL_SYSTEM_PROMPT_FILE` when both are set. If the file cannot be read, the server logs a warning and falls back as if the default were not configured.
Precedence for the prompt used by `recall` is: **explicit caller `system_prompt` > server-side env/file default > backend default.** When neither variable is set (or both resolve to empty), behavior is unchanged and no default `system_prompt` is added.
```bash theme={null}
# Inline
export COGNEE_MCP_RECALL_SYSTEM_PROMPT="Answer concisely and cite the source memory."
# Or from a file
export COGNEE_MCP_RECALL_SYSTEM_PROMPT_FILE=/etc/cognee/recall_prompt.txt
```
<Note>
The default is resolved from the environment inside the MCP server process, so changes only take effect after you restart the server (or redeploy the container). Treat the prompt content as configuration — if it contains sensitive instructions, manage it with your existing secrets practices rather than committing it to source control.
</Note>
</Accordion>
</AccordionGroup>
<Note>
The preferred MCP workflow is to use `remember`, `recall`, and `forget`. For the full tool catalog and how much of it appears in `tools/list`, see the [Tools Reference](/cognee-mcp/mcp-tools).
</Note>
## Next Steps
After starting the server, configure your AI client to connect to it. See the [integrations](/cognee-mcp/integrations) section for client-specific setup instructions.
## Need Help?
<Card title="Join Our Community" icon="discord" href="https://discord.gg/m63hxKsp4p">
Get support and connect with other developers using Cognee MCP.
</Card>
# Cognee MCP Overview
Source: https://docs.cognee.ai/cognee-mcp/mcp-overview
Connect Cognee to MCP-compatible AI tools like Claude, Cursor, and Cline.
Cognee MCP brings persistent AI memory to your workflow through the Model Context Protocol.
## What is MCP?
The [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) is a standard for adding specialized tools to AI assistants. It allows AI tools like Claude or Cursor to work with external systems such as databases, APIs, and AI platforms.
Without MCP, each AI assistant needs custom integrations for every external system. This creates duplication and inconsistency across tools.
MCP provides a single method for extending AI assistants with:
* **Standardized connections** between AI tools and external systems
* **Secure data access** with built-in authentication and permissions
* **Tool interoperability** so you can switch between AI providers
* **Persistent memory** that survives across conversations and sessions
## How Cognee MCP Works
Cognee MCP exposes specialized tools through the MCP protocol. These tools handle memory management, code intelligence, and data operations. You access them through MCP-compatible AI assistants like Cursor, Claude Desktop, Continue, Cline, and Codex.
To keep the tool payload small, the server lists only a subset of its catalog in `tools/list` by default and makes the rest discoverable through tool search. The remaining tools stay callable by name. See [Tool modes](/cognee-mcp/mcp-tools#tool-modes) for the modes and how to restore the full listing.
The tools enable your AI assistant to:
* Store and retrieve knowledge from previous conversations
* Build persistent understanding of your codebase and projects
* Access structured memories across different sessions
For new integrations, prefer the v1.0 memory tools (`remember`, `recall`, `forget`). The legacy tools remain available when you need lower-level control.
See the [Tools Reference](/cognee-mcp/mcp-tools) for all available operations.
## Architecture Modes
Cognee MCP can run in two modes:
**Standalone Mode**: The MCP server manages its own database and processing. Each MCP instance maintains separate data. Use this for personal development or when clients need isolated environments. Out of the box it uses SQLite, LanceDB, and the embedded Ladybug (Kuzu) graph store — see [Default databases](/cognee-mcp/mcp-local-setup#further-details) for details and how to switch to Neo4j or Postgres.
**API Mode**: The MCP server connects to a centralized Cognee backend via API. Multiple MCP instances can share the same knowledge graph. Use this when you want team members to access shared memory or when running multiple AI clients that need consistent data. Each instance authenticates with a single token, so everything it writes belongs to one Cognee user — see [Running in API Mode](/cognee-mcp/mcp-local-setup#running-in-api-mode) for what that means when several tenants share one deployment.
<Info>
**Do I need to install Cognee separately?** No. The `cognee-mcp` package bundles the full Cognee library, so following any setup guide ([Docker](/cognee-mcp/mcp-quickstart) or [Local Setup](/cognee-mcp/mcp-local-setup)) installs the complete Cognee stack together with the MCP tools — you get "everything plus MCP." In **Standalone Mode** the server runs the entire Cognee pipeline (ingestion, graph building, and search) locally with its own database; no extra Cognee install or running backend is required. You only need a separate [Cognee REST API](/guides/deploy-rest-api-server) when you switch to **API Mode** to share one knowledge graph across clients.
</Info>
## Setup Options
Choose your deployment method:
<CardGroup>
<Card title="Docker Quickstart" href="/cognee-mcp/mcp-quickstart" icon="docker">
**Recommended for most users**
Get running in minutes with a pre-built container.
</Card>
<Card title="API Mode (Shared)" href="/cognee-mcp/mcp-quickstart#api-mode-shared-knowledge-graph" icon="network">
**For teams**
Connect multiple clients to a shared knowledge graph.
</Card>
<Card title="Local Setup" href="/cognee-mcp/mcp-local-setup" icon="code">
**For development**
Build from source for full control and latest features.
</Card>
</CardGroup>
<Info>
**Using Cognee Cloud?** Cognee MCP and Cognee Cloud are separate systems with different APIs and authentication schemes. See [Cognee Cloud & MCP](/cognee-mcp/mcp-cloud-connection) to understand how they relate and when to use each.
</Info>
## Next Steps
<CardGroup>
<Card title="Tools Reference" href="/cognee-mcp/mcp-tools" icon="wrench">
See all available MCP tools and operations
</Card>
<Card title="Client Integrations" href="/cognee-mcp/integrations" icon="code">
Connect with Cursor, Claude, Continue, and more
</Card>
</CardGroup>
# Cognee MCP Quickstart
Source: https://docs.cognee.ai/cognee-mcp/mcp-quickstart
Start the Cognee MCP server with Docker and test it from any MCP client.
Start the Cognee MCP server using Docker to quickly test AI memory integration.
## Prerequisites
* Docker installed and running
* OpenAI API key
## Setup Steps
<Steps>
<Step title="Set Your API Key">
```bash theme={null}
export LLM_API_KEY=your_api_key_here
```
</Step>
<Step title="Create Environment File">
```bash theme={null}
echo "LLM_API_KEY=your_api_key_here" > .env
```
</Step>
<Step title="Start the Server">
```bash theme={null}
docker run -e TRANSPORT_MODE=http --env-file ./.env -p 8000:8000 --rm -it cognee/cognee-mcp:main
```
The server starts on port 8000 with HTTP transport mode.
</Step>
<Step title="Verify the Server">
```bash theme={null}
curl http://localhost:8000/health
```
You should see a healthy response from the server.
</Step>
</Steps>
## Persist Data
By default, the container removes its local data when it stops. Use a bind mount or named Docker volume if you want memory to survive restarts.
The image stores everything under `/cognee-storage`, which is where you must mount:
```dockerfile theme={null}
ENV SYSTEM_ROOT_DIRECTORY=/cognee-storage/system
ENV DATA_ROOT_DIRECTORY=/cognee-storage/data
```
`system` holds the embedded graph and vector files; `data` holds ingested files and loader outputs. Mount both:
```bash theme={null}
docker run -e TRANSPORT_MODE=http --env-file ./.env -p 8000:8000 \
-v cognee_system:/cognee-storage/system \
-v cognee_data:/cognee-storage/data \
--rm -it cognee/cognee-mcp:main
```
Each source can be either:
* A named Docker volume, such as `cognee_data:/cognee-storage/data`
* A local directory path, such as `./cognee_data:/cognee-storage/data`
<Note>
The container runs as the non-root user `cognee` (uid/gid 1000). A fresh **named volume** picks up that ownership from the image and is writable straight away; a **host directory** keeps its own ownership, so run `chown -R 1000:1000 ./cognee_data` before mounting it, or ingestion fails with `PermissionError: [Errno 13] Permission denied`.
The `cognee/cognee` backend image defaults to the same paths and the same uid, so pointing both at one pair of volumes gives the API server and the MCP server a shared memory store.
</Note>
## Kuzu/Ladybug JSON Extension
The default Ladybug (Kuzu) graph backend needs its JSON extension for graph queries that rely on JSON — including `recall` and temporal search. The `cognee/cognee-mcp` image pre-installs this extension at build time, so it is baked into the image even when the container has no network access at runtime. Pulling the latest image (or rebuilding it) ensures the extension is present.
If you run the MCP server in a network-restricted container that does not already contain the extension, the adapter attempts a one-time runtime `INSTALL JSON; LOAD JSON;` on startup. When it cannot install or load the extension, it logs a warning explaining that JSON-dependent queries (such as `recall` and temporal search) will otherwise fail with `Extension: json ... has not been installed`. To resolve it, give the process network access at startup, use an image that already bundles the extension, or run `INSTALL json; LOAD json;` once against the database.
## API Mode (Shared Knowledge Graph)
To connect multiple clients to a shared knowledge graph, run MCP in API mode pointing to a centralized Cognee backend:
<Steps>
<Step title="Start Cognee Backend">
First, start a Cognee backend instance:
```bash theme={null}
docker run -e LLM_API_KEY=your_api_key_here -p 8080:8000 --rm -it cognee/cognee:main
```
</Step>
<Step title="Start MCP in API Mode">
Start the MCP server and point it to the backend:
```bash theme={null}
docker run -e TRANSPORT_MODE=http -e API_URL=http://localhost:8080 -p 8000:8000 --rm -it cognee/cognee-mcp:main
```
The container rewrites `localhost` / `127.0.0.1` in `API_URL` to a host-reachable address so the MCP container can reach a backend running on your host machine. The entrypoint auto-detects the address using the following fallback order: `host.docker.internal` (Docker Desktop), then `host.lima.internal` (Colima / Lima), then the container's default gateway IP (plain Linux Docker, typically `172.17.0.1`). If none resolve it keeps `host.docker.internal` and prints guidance. This means Docker Desktop, Colima, and plain Linux Docker generally work without manual configuration. The MCP server now acts as an interface to the shared backend.
</Step>
<Step title="Connect Additional Clients (Optional)">
If you need to support multiple clients, start additional MCP instances on different ports:
```bash theme={null}
docker run -e TRANSPORT_MODE=http -e API_URL=http://localhost:8080 -p 8001:8000 --rm -it cognee/cognee-mcp:main
```
Each client connects to its own MCP instance, but all share the same knowledge graph through the backend.
</Step>
</Steps>
<Note>
* The API mode requires SSE or HTTP transport
* If `API_URL` uses `localhost` / `127.0.0.1`, the container rewrites it to the first host address it can resolve: `host.docker.internal` (Docker Desktop), then `host.lima.internal` (Colima / Lima), then the default gateway IP (plain Linux Docker, typically `172.17.0.1`)
* If auto-detection still fails, use `--network host` (Linux) or set `API_URL` to a bridge address such as `http://172.17.0.1:8000` directly; on Colima, start the VM with `colima start --network-address`
* Add `-e API_TOKEN=your_token` if your backend requires authentication
* The container's startup log prints `calling cognee-mcp … --api-token <redacted>`, so the token is not readable from `docker logs` (it is still visible via `docker inspect` and the container's process arguments)
* For backend authentication setup and how to obtain a Bearer token, see [Deploy REST API Server](/guides/deploy-rest-api-server#authentication)
</Note>
## Docker Compose (Production Setup)
For production deployments, use Docker Compose to run the Cognee backend and MCP server together. This avoids `localhost` mapping issues and uses Docker's internal DNS for service discovery.
```yaml docker-compose.yml theme={null}
services:
cognee-backend:
image: cognee/cognee:main
container_name: cognee-backend
restart: unless-stopped
ports:
- "8080:8000"
environment:
LLM_API_KEY: "${LLM_API_KEY}"
LLM_PROVIDER: "${LLM_PROVIDER:-openai}"
LLM_MODEL: "${LLM_MODEL:-openai/gpt-5-mini}"
volumes:
# The image's baked-in SYSTEM_ROOT_DIRECTORY / DATA_ROOT_DIRECTORY
- cognee_system:/cognee-storage/system
- cognee_data:/cognee-storage/data
networks:
- cognee_internal
cognee-mcp:
image: cognee/cognee-mcp:main
container_name: cognee-mcp
restart: unless-stopped
ports:
- "8000:8000"
environment:
TRANSPORT_MODE: "http"
API_URL: "http://cognee-backend:8000"
LLM_API_KEY: "${LLM_API_KEY}"
LLM_PROVIDER: "${LLM_PROVIDER:-openai}"
LLM_MODEL: "${LLM_MODEL:-openai/gpt-5-mini}"
depends_on:
- cognee-backend
networks:
- cognee_internal
volumes:
cognee_system:
cognee_data:
networks:
cognee_internal:
driver: bridge
```
<Info>
**Networking notes:**
* Use the **service name** (`cognee-backend`) as the hostname in `API_URL` — Docker resolves it automatically within the same network.
* Use the **internal port** (`8000`) in `API_URL`, not the host-mapped port (`8080`).
* If you place a reverse proxy (Nginx, Caddy) in front, you do **not** need to set a `Host: localhost` header — the backend accepts requests on any host.
* Add `-e API_TOKEN=your_token` to the MCP service if your backend requires authentication.
</Info>
## Troubleshooting
<AccordionGroup>
<Accordion title="Graph queries fail with 'Extension: json ... has not been installed'">
The default Ladybug graph store (Kuzu) downloads a JSON extension at startup, which some graph queries (such as recall and temporal search) rely on. In a network-restricted container, that download can fail and queries surface a cryptic `Extension: json ... has not been installed` Binder error.
The published `cognee/cognee-mcp` image now bakes this extension in at build time, so pulling and running the latest image is enough. If you build the MCP image yourself, **rebuild it** after updating to pick up this change.
When the extension still cannot be installed (for example, an offline container built without network access), the server now logs a clear warning at startup and attempts to install and load the extension directly on its database connection. To remediate, either give the process network access at startup, pre-install the extension in your image, or run `INSTALL json; LOAD json;` once against the database.
</Accordion>
<Accordion title="Client gets repeated 404s on /sse (or /mcp)">
The `TRANSPORT_MODE` environment variable sets which endpoint the container exposes, and it must match the URL your MCP client connects to. A 404 on `/sse` almost always means the client is pointed at the SSE endpoint while the server was started with `TRANSPORT_MODE=http` (or vice versa).
| `TRANSPORT_MODE` | Endpoint exposed | Client URL |
| -------------------------- | ----------------------------------- | --------------------------------------------------------- |
| `http` (Streamable HTTP) | `/mcp` | `http://localhost:8000/mcp` |
| `sse` (Server-Sent Events) | `/sse` | `http://localhost:8000/sse` |
| `stdio` (default) | none (subprocess over stdin/stdout) | not applicable — the client launches the process directly |
`stdio` is the default when `TRANSPORT_MODE` is unset, so a network client (`http`/`sse`) needs the variable set explicitly, as in the Quickstart. To fix the 404, either change your client's URL to match the running mode, or restart the container with the `TRANSPORT_MODE` your client expects:
```bash theme={null}
# Client expects /sse → start the server in SSE mode
docker run -e TRANSPORT_MODE=sse --env-file ./.env -p 8000:8000 --rm -it cognee/cognee-mcp:main
```
The `/health` check answers on any of the network modes, so a healthy `/health` but a 404 on `/sse` or `/mcp` is a transport mismatch, not a server that failed to start. See [Local Setup — Transport Modes](/cognee-mcp/mcp-local-setup) for the full list of transports and their client configuration.
</Accordion>
<Accordion title="Why is the image so large and why does it install NVIDIA/CUDA packages?">
The `cognee/cognee-mcp` image is several GB because it bundles Cognee's document-processing stack. The MCP package installs `cognee[postgres-binary,docs,neo4j]`, and the `docs` extra pulls `unstructured[pdf]` for parsing PDFs, Office files, and other documents. That chain brings in heavy ML libraries — `torch`, `torchvision`, `timm`, `onnxruntime`, and `opencv-python` — used by `unstructured-inference` for layout/OCR models.
The `nvidia-*-cu12` (CUDA) packages you see during the build come from the **default PyPI `torch` wheel**, which bundles the CUDA runtime as hard dependencies. They are installed regardless of whether a GPU is present, and account for most of the image size. No GPU or NVIDIA driver is required to run the server — these libraries simply ship with the standard `torch` wheel and fall back to CPU.
There is currently no published slim / CPU-only image variant, but a CPU-only image is on the way. If you don't need document parsing and want a smaller build today, remove the `docs` extra from `cognee-mcp/pyproject.toml`, regenerate `cognee-mcp/uv.lock`, and build a custom image from the [source Dockerfile](https://github.com/topoteretes/cognee/blob/main/cognee-mcp/Dockerfile). The Dockerfile uses `uv sync --frozen`, so the lockfile must match the edited dependency set. Removing `docs` drops the `unstructured`, `torch`, and CUDA dependencies. See [Local Setup](/cognee-mcp/mcp-local-setup) for building from source.
</Accordion>
</AccordionGroup>
## Connect to AI Clients
After starting the server, connect it to your AI development tool:
<CardGroup>
<Card title="Cursor" href="/cognee-mcp/integrations/cursor" icon="code">
AI-powered code editor with native MCP support
</Card>
<Card title="Claude Code" href="/cognee-mcp/integrations/claude-code" icon="bot">
Command-line AI assistant from Anthropic
</Card>
<Card title="Codex" href="/cognee-mcp/integrations/codex" icon="terminal">
OpenAI coding agent with built-in MCP support
</Card>
</CardGroup>
<CardGroup>
<Card title="Cline" href="/cognee-mcp/integrations/cline" icon="terminal">
VS Code extension for AI-assisted development
</Card>
<Card title="Continue" href="/cognee-mcp/integrations/continue" icon="play">
Open-source AI coding assistant
</Card>
<Card title="Python Agent" href="/cognee-mcp/integrations/python-agent" icon="python">
Build your own MCP client
</Card>
</CardGroup>
## Next Steps
<CardGroup>
<Card title="Tools Reference" href="/cognee-mcp/mcp-tools" icon="wrench">
See all available MCP tools and operations
</Card>
<Card title="Local Setup" href="/cognee-mcp/mcp-local-setup" icon="code">
Run from source for customization and development
</Card>
</CardGroup>
# Tools Reference
Source: https://docs.cognee.ai/cognee-mcp/mcp-tools
Reference for Cognee MCP tools, parameters, defaults, and usage notes.
Cognee MCP exposes the memory API — `remember`, `recall`, and `forget` — plus `cognify_status` for tracking background ingestion. Every registered tool is callable by name, but only some of them appear in the server's `tools/list` response by default — see [Tool modes](#tool-modes).
Unless noted otherwise, parameter names and defaults below reflect the current MCP server implementation.
<Note>
Some MCP parameters use compact transport-friendly encodings:
* `datasets` is a comma-separated string, not a JSON array.
* `top_k` must be between `1` and `100`.
With backend access control enabled, dataset names are resolved against datasets owned by the current user. Shared datasets that the user can access but did not create may not be targetable by name through MCP retrieval tools.
</Note>
## Available Tools
<AccordionGroup>
<Accordion title="Memory Tools">
The core memory API. These tools map to Cognee's main operations and appear in `tools/list` in every tool mode.
<AccordionGroup>
<Accordion title="`remember`">
Store content as permanent graph memory or session memory in one call. Accepts either text or a base64 file upload. See [Remember](/core-concepts/main-operations/remember).
| Parameter | Type | Default | Notes |
| ---------------- | ------ | ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `data` | `str` | `None` | Text to store. Mutually exclusive with `filename` / `content_base64`. |
| `filename` | `str` | `None` | Original filename of a file upload. Used to derive the stored document's name. When omitted, the file is stored as `upload.txt`. |
| `content_base64` | `str` | `None` | Base64-encoded file content to ingest, up to 10 MB. |
| `dataset_name` | `str` | agent-scoped | Target dataset for permanent memory. Defaults to the MCP client's agent-scoped dataset (e.g. `cursor_vscode_memory`), or `main_dataset` when no client identity is detected. |
| `session_id` | `str` | `None` | When set, stores in session cache instead of permanent graph memory. |
| `custom_prompt` | `str` | `None` | Custom extraction prompt for permanent-memory mode. |
| `background` | `bool` | `False` | Queue permanent ingestion as a background task and return immediately instead of waiting for the pipeline. Ignored when `session_id` is set. |
Provide either `data` or `content_base64` (optionally with `filename`). Passing both, or neither, returns an error. With `background=true` the call returns before ingestion finishes, so failures cannot surface in the return value — check progress and any captured errors with [`cognify_status`](#available-tools).
<Note>
File uploads are permanent-memory only: combining `content_base64` with `session_id` returns an error, and content that is not valid base64 or exceeds 10 MB is rejected. The stored document keeps the file's basename (directory components stripped, `.txt` appended when there is no suffix).
</Note>
</Accordion>
<Accordion title="`recall`">
Retrieve memory with auto-routing and session-aware behavior. See [Recall](/core-concepts/main-operations/recall).
| Parameter | Type | Default | Notes |
| --------------- | ----- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `query` | `str` | required | Natural-language query. |
| `search_type` | `str` | `None` | Optional override for auto-routing. |
| `datasets` | `str` | `None` | Comma-separated dataset names. Name lookup is owner-scoped. |
| `session_id` | `str` | `None` | Session-first retrieval scope. |
| `system_prompt` | `str` | `None` | Override the synthesis prompt for completion searches. When omitted, falls back to a server-side default if one is configured (see note below). |
| `top_k` | `int` | `15` | Must be between `1` and `100`. |
<Note>
When `system_prompt` is omitted, `recall` falls back to a server-side default synthesis prompt if the server sets `COGNEE_MCP_RECALL_SYSTEM_PROMPT` or `COGNEE_MCP_RECALL_SYSTEM_PROMPT_FILE`. An explicit `system_prompt` always takes precedence over the server-side default. If neither environment variable is set, behavior is unchanged. See [Local Setup](/cognee-mcp/mcp-local-setup) for configuration.
</Note>
<Warning>
`recall` currently accepts dataset names, not `dataset_ids`. If Bob is querying Alice's shared dataset, `datasets="shared_dataset"` can fail even when Bob has permission to use it. In that case, either omit `datasets` to search across all accessible datasets or use the Python SDK / REST API where `dataset_ids` are supported.
</Warning>
</Accordion>
<Accordion title="`forget`">
Delete a single data item, a dataset, or all memory owned by the current user. See [Forget](/core-concepts/main-operations/forget).
| Parameter | Type | Default | Notes |
| ------------ | ------ | ------- | ------------------------------------------------------------------------------------------------------------------------- |
| `dataset` | `str` | `None` | Dataset name to delete entirely. |
| `everything` | `bool` | `False` | Set to `true` to delete all user-owned memory. |
| `data_id` | `str` | `None` | UUID of a single data item to delete. Must be paired with `dataset` or `dataset_id` so the owning dataset is unambiguous. |
| `dataset_id` | `str` | `None` | UUID of the dataset to delete entirely, or to scope `data_id`. |
At least one of `dataset`, `dataset_id`, `data_id`, or `everything=true` must be provided; passing `data_id` alone returns an error. Malformed UUIDs are rejected with a message rather than a traceback.
</Accordion>
</AccordionGroup>
</Accordion>
<Accordion title="Discovery Tools">
These are tools for finding other tools. `search_tools` searches the server's own tool catalog by natural-language query, and `call_tool` invokes a tool found that way — together they let an agent reach every registered tool without all of them being in `tools/list`.
The pair only exists in `default` and `minimal` mode. In `all` mode the server does not create them: every tool is already in `tools/list`, so there is nothing to search for (see [Tool modes](#tool-modes)).
<AccordionGroup>
<Accordion title="`search_tools`">
Find registered tools by natural-language query. Returns a JSON array of matching tools, each with its `name`, `description`, full `inputSchema`, and any UI metadata — enough to call the tool without another round trip. Returns empty content when nothing matches.
| Parameter | Type | Default | Notes |
| --------- | ----- | -------- | ------------------------------------------------------------------------- |
| `query` | `str` | required | Natural-language description of what you need, e.g. `"list my datasets"`. |
At most 10 tools are returned, and tools already in `tools/list` are never among them. Matching is lexical and does not stem words, so multi-word, natural phrasings work best — for example `"is my background ingestion finished?"` returns `cognify_status`.
</Accordion>
<Accordion title="`call_tool`">
Invoke a tool by name without it being in `tools/list` — typically one just found through `search_tools`. Calling the found tool directly by name works just as well; the proxy exists for clients that only invoke listed tools.
| Parameter | Type | Default | Notes |
| ----------- | -------- | -------- | -------------------------------------------------------------------------------------------- |
| `name` | `str` | required | Name of the tool to invoke. |
| `arguments` | `object` | — | Arguments for that tool, matching its `inputSchema`. Pass `{}` for tools without parameters. |
Returns the target tool's result unchanged. Refuses to invoke `search_tools` or `call_tool` themselves.
</Accordion>
</AccordionGroup>
</Accordion>
<Accordion title="Status Tools">
<AccordionGroup>
<Accordion title="`cognify_status`">
Check the progress of background ingestion started by `remember(background=True)`. Reports active and completed pipeline jobs for a dataset, including failures that a backgrounded call could not return inline.
| Parameter | Type | Default | Notes |
| -------------- | ----------- | ---------------------- | ----------------------------------------------------------------------------------------------------------- |
| `dataset_name` | `str` | agent-scoped | Dataset to report on. Defaults to the MCP client's agent-scoped dataset, so each agent sees its own status. |
| `pipelines` | `list[str]` | `["cognify_pipeline"]` | Restrict the report to specific pipeline names. |
This tool is registered but not advertised: it stays out of `tools/list` in `default` and `minimal` mode, while remaining discoverable through `search_tools` and callable directly by name. It is listed in `all` mode (see [Tool modes](#tool-modes)).
</Accordion>
</AccordionGroup>
</Accordion>
</AccordionGroup>
<Note>
Earlier versions of the server also registered `cognify`, `search`, `prune`, `improve`, `save_interaction`, `get_document`, `get_chunk_neighbors`, `list_data`, `delete`, and `delete_dataset` as MCP tools, along with the workspace UI entry points (`visualize_graph_ui`, `upload_file_ui`, `open_cognee_workspace`) and the structured JSON tools the UI used (`list_datasets_json`, `list_dataset_data_json`, `get_client_info_json`, `create_dataset_json`). The workspace UI and all of these tools have been removed in every mode — use `remember` / `recall` / `forget` instead, and reach for the Python SDK or REST API when you need lower-level control such as explicit search types, or dataset listing and creation.
</Note>
## Tool Modes
An MCP client learns which tools a server offers from the server's `tools/list` response. By default, Cognee MCP keeps that list short — the memory tools above — and exposes the rest through two discovery tools: `search_tools`, which finds a tool by natural-language query, and `call_tool`, which invokes it. A shorter list costs a connected agent less context on every turn.
**Tools left out of `tools/list` stay registered and remain callable directly by name**, so clients that invoke a tool without listing it first are unaffected.
Choose how much of the catalog is listed with the `COGNEE_MCP_TOOL_MODE` environment variable (or the `--tool-mode` server argument):
| Mode | Listed in `tools/list` |
| --------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `default` (used when unset) | `remember`, `recall`, `forget`, plus `search_tools` and `call_tool` |
| `minimal` | `remember`, `recall`, `forget`, plus `search_tools` and `call_tool` |
| `all` | Every registered tool — the three memory tools and `cognify_status` — with no `search_tools` / `call_tool` |
An unrecognized value logs a warning and falls back to `default`.
<Note>
`default` and `minimal` currently advertise the same set: the memory tools carry both the `default` and `memory` tags, so pinning either tag yields `remember`, `recall`, and `forget`. The two modes stay distinct because they pin by different tags — a tool added with only the `default` tag would appear in `default` but not in `minimal`.
</Note>
### Finding and calling an unlisted tool
1. Call `search_tools` with a natural-language `query`. Matches are returned with their full `inputSchema`, so no extra round trip is needed before invoking one.
2. Call the tool you found, either directly by name or through the proxy: `call_tool(name="cognify_status", arguments={...})`.
Both tools are documented under **Discovery Tools** in [Available Tools](#available-tools).
<Note>
If you have an integration or script that depends on the full flat `tools/list` response, set `COGNEE_MCP_TOOL_MODE=all` to restore the previous behavior. The mode is read when the server process starts, so changes require a restart. See [Local Setup](/cognee-mcp/mcp-local-setup#further-details).
</Note>
## Usage Notes
* Start with `remember` to store data and `recall` to retrieve it; use `forget` to remove a single item, a dataset, or all memory owned by the current user.
* When a `remember` call would outlast your client's request deadline, pass `background=true` and poll `cognify_status` — find it with `search_tools` if your client only calls listed tools.
* For lower-level control — explicit search types, custom graph models, dataset listing and creation — use the Python SDK or REST API.
* In shared-dataset setups, prefer the Python SDK or REST API when you need UUID-based dataset scoping for a dataset the current user did not create.
## Next Steps
<Card title="Client Integrations" href="/cognee-mcp/integrations" icon="code">
Learn how to use these tools with your AI development environment
</Card>
# Contributing
Source: https://docs.cognee.ai/contributing/contributing-overview
Contribute to the cognee project
We welcome contributions from the community! Your input helps make Cognee better for everyone.
This page outlines instructions and best practices for contributing to Cognee, ensuring your contributions are integrated into the project efficiently.
## How To Contribute
There are many ways in which you can contribute to this project include: submitting bug reports and feature requests via GitHub [issues](https://github.com/topoteretes/cognee/issues),
opening PRs with features, fixes, or tests, reviewing others’ PRs, and collaborating by commenting or answering questions in the [Discord community](https://discord.com/invite/m63hxKsp4p).
### Development Setup
Firstly, you will need to set up your local copy of the cognee code repository. Keep in mind that we have two different
repositories: our [core repo](https://github.com/topoteretes/cognee), which contains core cognee functionalities,
and our [community repo](https://github.com/topoteretes/cognee-community), which contains community-maintained add-ons and custom packages.
Once you have chosen the repo you are going to work on, you have to fork it, and clone it to your machine:
```bash theme={null}
git clone https://github.com/<your-github-username>/cognee.git
cd cognee
# OR
git clone https://github.com/<your-github-username>/cognee-community.git
cd cognee-community
```
After this, just create a branch for your work, and add your code there:
```bash theme={null}
git checkout -b feature/your-feature-name
```
### Development Guidelines
While working on new features and fixes, make sure to keep these guidelines in mind:
* **Code Style** - Make sure to follow the [PEP8](https://peps.python.org/pep-0008/) style guide, and also the style of the project codebase.
Reuse as much of the code as possible, and follow the existing package and file hierarchy.
* **Tests** - Make sure to add tests for new features, and that the tests pass before opening a PR
* **Commits** - Write clear and concise commit messages, following the commit and PR title style below
#### Commit & PR Title Style
The same style applies to commit messages and PR titles. If you use a coding agent (e.g. Claude Code)
in the repository, it picks up these rules from
[`CLAUDE.md`](https://github.com/topoteretes/cognee/blob/main/CLAUDE.md) automatically.
**Subject line (required):**
* Use the format `(type): (short summary)`
* Write the summary as if it is giving an instruction — "Fix bug", not "Fixed bug"
* Keep it to 50 characters or less
* Capitalize the first character of the summary
* Do **not** end it with a period
**Body (optional):**
* Explain the motivation behind the change, what problem it solves, and any relevant background —
the *what* and *why*, not the *how*; the code itself should make the "how" clear
* Separate the subject line from the body with a blank line. Generally, all commits should have separate
subject and body.
#### Linting, Formatting, and Type Checking
Before creating a Pull Request, you need to make sure that your code is linted and formatted correctly.
To do this, you first have to install [ruff](https://docs.astral.sh/ruff/) on your system.
Ruff is part of the project's dev dependencies, so you can install it with
[uv](https://docs.astral.sh/uv/) in the root of the core repo, or on its own through `pip`.
```bash theme={null}
uv sync
# OR
pip install ruff
```
Then run the linter and the formatter from the root of the project:
```bash theme={null}
ruff check . # report lint errors
ruff format . # rewrite files to the project format
```
If you only want to verify formatting without rewriting any files — which is what CI does — use the
`--check` variant instead:
```bash theme={null}
ruff format --check .
```
Both checks also run as [pre-commit](https://pre-commit.com) hooks (`ruff` and `ruff-format`) and again
directly in CI's Code Quality job. Install the hooks once so they run automatically on every commit:
```bash theme={null}
pre-commit install
```
The hooks are pinned to a specific ruff version in
[`.pre-commit-config.yaml`](https://github.com/topoteretes/cognee/blob/main/.pre-commit-config.yaml),
while the `ruff` dependency in `pyproject.toml` allows a range of versions. If a locally installed ruff
disagrees with CI, run the pinned version through the hooks instead:
```bash theme={null}
pre-commit run --all-files
```
Ruff's configuration lives in `pyproject.toml` under `[tool.ruff]` — most notably a line length of 100
and a list of excluded paths — and the rules the project deliberately switches off live under
`[tool.ruff.lint]` in the `ignore` list, most with a comment explaining why. The `ASYNC220`, `ASYNC221`,
`ASYNC230`, and `ASYNC251` rules (blocking file, subprocess, and `sleep` calls inside `async` functions)
are in that list: the document loaders and the local storage layer do their file IO synchronously today,
and the parser holding the handle — not the `open()` call — is the blocking work, so the fix is moving
each loader's parse off the event loop rather than wrapping the `open()` calls. That work is tracked
separately, and the rules are meant to be re-enabled when it lands. Don't add new blocking IO to `async`
code on the strength of those rules being off.
Ruff is not the last gate, though. The Code Quality job runs these steps, in this order:
```bash theme={null}
uv lock --check # the lockfile matches pyproject.toml
pre-commit run --all-files # the pinned hooks
uv run ruff check . # lint
uv run ruff check . --select BLE001,S110,S112,G201,TRY201,TRY002 --ignore-noqa # exception handling, noqa not honoured
uv run ruff format --check . # formatting, without rewriting files
uv run ty check . # type check
```
If `uv lock --check` fails, run `uv lock` and commit the updated `uv.lock`.
The second ruff pass re-runs the exception-handling rules with `--ignore-noqa`, so a `# noqa` comment on
any of them counts as a finding rather than a waiver. A blind `except Exception`, a `try`/`except` that
only passes or continues, or a bare re-raise has to be fixed — log with the traceback, re-raise, or narrow
the exception type — not suppressed.
The last step is [ty](https://github.com/astral-sh/ty), the type checker. It ships in the project's `dev`
extra, which is also how CI installs its environment, so run it the same way:
```bash theme={null}
uv sync --extra dev
uv run ty check .
```
Two things worth knowing about how the type check is scoped:
* **It does not cover the whole repository yet.** `[tool.ty.src]` in `pyproject.toml` lists the directories
that are checked — currently around fifteen paths under `cognee/`, such as `cognee/infrastructure/llm`,
`cognee/infrastructure/loaders`, `cognee/memory`, and `cognee/shared` — and the generated BAML client is
explicitly excluded. The list is meant to grow, so a clean `ty check .` means the checked directories are
clean, not that every module in your PR was inspected. If you touch a directory that is not on the list yet,
adding it is welcome but is a change of its own.
* **Optional dependencies are suppressed inline.** Imports that only resolve when an extra is installed —
Pillow and `rapidocr_onnxruntime` in the image loader, for example — carry a
`# ty: ignore[unresolved-import]` comment, because CI type-checks with only the `dev` extra installed. Use
that marker for the same situation, and keep it narrow: prefer fixing the underlying problem where a public
API exists (for instance `Image.Resampling.LANCZOS` rather than suppressing `Image.LANCZOS`) instead of
silencing the diagnostic.
#### Submitting a Pull Request
After successfully linting and formatting your code, you can push your changes:
```bash theme={null}
git add .
git commit -s -m "feat: Add my new feature"
git push origin feature/your-feature-name
```
And now, create a Pull Request so your contribution can be reviewed, and eventually merged to the project repository:
* Go to the repository you made changes for (i.e. the core repo or the community repo)
* Click **Compare & Pull Request** and open a PR, being careful against **which branch**
you open it (`dev` for core repo, `main` for community)
* Fill in the PR template with details about your changes
After opening the PR, the right reviewers will be notified automatically — Cognee uses a [CODEOWNERS](https://github.com/topoteretes/cognee/blob/main/.github/CODEOWNERS) file to request reviews based on the directories your PR touches. No manual ping required. We will make sure to review it, and eventually your contribution will be a part of the Cognee project!
#### Changelog Entries
If maintainers ask for a changelog entry, add it under the `Unreleased` section of `CHANGELOG.md`.
* Use `Added` for new capabilities
* Use `Changed` for behavior or documentation updates
* Use `Fixed` for bug fixes
Example entry:
```markdown theme={null}
## Unreleased
### Fixed
- Clarify the minimal Docker Compose setup for first-time contributors.
```
<CardGroup>
<Card title="Contributing Guide Details" href="https://github.com/topoteretes/cognee/blob/main/CONTRIBUTING.md" icon="book">
More details about the contributing process.
</Card>
<Card title="Community Guidelines" href="https://github.com/topoteretes/cognee/blob/main/CODE_OF_CONDUCT.md" icon="book">
Be respectful and follow our code of conduct. Help others learn and grow, and provide constructive feedback.
</Card>
<Card title="Join our Discord Community" href="https://discord.gg/m63hxKsp4p" icon="discord">
Join the community for real-time discussions with us and other users!
</Card>
</CardGroup>
# Architecture
Source: https://docs.cognee.ai/core-concepts/architecture
Learn how Cognee combines relational, vector, and graph storage.
# Cognee Architecture
<img alt="Cognee architecture: agents, apps, and data sources connect to session memory and permanent memory through three data pipelines (session learning, self-improvement, and ingestion), all sharing the same Task and DataPoint pipeline structure" />
## Why multiple stores
No single database can handle all aspects of memory. Cognee combines three complementary storage systems. Each one plays a different role, and together they make your data both **searchable** and **connected**.
* **Relational store** — Tracks your documents, their chunks, and provenance
(i.e. where each piece of data came from and how it's linked to the source).
* **Vector store** — Holds embeddings for semantic similarity
(i.e. numerical representations that let Cognee find conceptually related text, even if the wording is different).
* **Graph store** — Captures entities and relationships in a knowledge graph
(i.e. nodes and edges that let Cognee understand structure and navigate connections between concepts).
Cognee ships with lightweight defaults that run locally and sets everything up for you — tables, entities, and schemas are created automatically, so you never define them yourself. You can swap in production-ready backends when needed (see [Setup](/getting-started/installation)).
## What is stored where
Roughly speaking:
* The **relational store** handles document-level metadata and provenance.
* The **vector store** contains semantic fingerprints of chunks and [DataPoints](./building-blocks/datapoints).
* The **graph store** captures higher-level structure in the form of entities and relationships.
There is some overlap: for efficiency, parts of the same information may be indexed in more than one store.
## How they are used
The stores play different roles depending on the phase:
* The **relational store** matters most during permanent-memory ingestion, keeping track of documents, chunks, and where each piece of information comes from.
* The **vector** and **graph** stores come into play during *recall and retrieval*:
* **Semantic searches** (vector): find conceptually related passages based on embeddings
* **Structural searches** (graph): explore entities and relationships using Cypher directly
* **Hybrid searches** (vector + graph): combine both perspectives to surface results that are contextually rich and structurally precise.
To see how data moves between these stores — the three pipelines that carry session learning, self-improvement, and ingestion — see [Data Flows](/core-concepts/data-flows).
<Columns>
<Card title="Main Operations" icon="play" href="/core-concepts/main-operations/remember">
See how Remember, Improve, and Recall use the storage systems
</Card>
<Card title="Building Blocks" icon="puzzle" href="/core-concepts/building-blocks/datapoints">
Learn about DataPoints, Tasks, and Pipelines that feed into storage
</Card>
<Card title="Recall" icon="search" href="/core-concepts/main-operations/recall">
Explore the default retrieval flow built on top of the architecture
</Card>
</Columns>
# DataPoints
Source: https://docs.cognee.ai/core-concepts/building-blocks/datapoints
Atomic units of knowledge in Cognee.
# DataPoints: Atomic Units of Knowledge
DataPoints are the smallest building blocks in Cognee.\
They represent **atomic units of knowledge** — carrying both your actual content and the context needed to process, index, and connect it.
They're the reason Cognee can turn raw documents into something that's both **searchable** (via vectors) and **connected** (via graphs).
## What are DataPoints
* **Atomic** — each DataPoint represents one concept or unit of information.
* **Structured** — implemented as [Pydantic](https://docs.pydantic.dev/) models for validation and serialization.
* **Contextual** — carry provenance, versioning, and indexing hints so every step downstream knows where data came from and how to use it.
## Core Structure
A DataPoint is just a Pydantic model with a set of standard fields.
<Accordion title="See example class definition">
```python theme={null}
class DataPoint(BaseModel):
id: UUID = Field(default_factory=uuid4)
created_at: int = ...
updated_at: int = ...
version: int = 1
topological_rank: Optional[int] = 0
valid_to: int | None = None
metadata: MetaData = {"index_fields": []}
type: str = "DataPoint"
ontology_uri: str | None = None
belongs_to_set: Optional[List["DataPoint"]] = None
```
Key fields:
* `id` — unique identifier (shared across all three stores, linking vector, graph, and relational records for the same DataPoint)
* `created_at`, `updated_at` — timestamps (ms since epoch)
* `version` — for tracking changes and schema evolution
* `topological_rank` — an integer indicating the DataPoint's position in a dependency hierarchy. Lower ranks mean fewer dependencies. For example, an `Entity` that other DataPoints reference would have a lower rank than a `TextSummary` that depends on it. Defaults to `0`.
* `valid_to` — bi-temporal validity stamp (`int | None`, default `None`): the ms-epoch moment at which this fact was **superseded**. `None` means the fact is still current. You normally do not set it by hand — [`close_node()`](/guides/fact-validity) stamps it on the stored node when a fact is replaced, so the old node stays in the graph instead of being deleted. It has **no effect on node identity** (dedup is still driven by `id`/`identity_fields`), and it is distinct from the `time_to` on `Interval` nodes (the time range an `Event` points to via `during`), which records when an event *occurred* rather than when a fact stopped being true.
* `metadata.index_fields` — critical: determines which fields are embedded for vector search
* `type` — the Python class name of the DataPoint subclass (e.g., `"Person"`, `"Book"`)
* `ontology_uri` — optional external ontology IRI (`str | None`, default `None`). When an entity or type is matched against an ontology — or ingested from RDF — Cognee preserves the stable external IRI here instead of flattening it to a local label, so the node can be linked out to other domains and exported back to RDF. It defaults to `None` for ungrounded nodes and has **no effect on node identity** (dedup is still driven by `id`/`identity_fields`). See [Ontologies → RDF read/write surface](/core-concepts/further-concepts/ontologies#rdf-readwrite-surface).
* `belongs_to_set` — groups related DataPoints
</Accordion>
## Indexing & Embeddings
The `metadata.index_fields` tells Cognee which fields to embed into the vector store.
This is the mechanism behind semantic search.
* Fields in `index_fields` → converted into embeddings
* Each indexed field → its own vector collection named `Class_field` (e.g., a `Person` DataPoint with `index_fields=["name"]` creates a `Person_name` vector collection). The `Class` part comes from the Python class name of your DataPoint subclass.
* Non-indexed fields → stay as regular properties in the graph and relational stores
* Choosing what to index controls search granularity
When you declare several index fields, Cognee deep-copies the DataPoint once per field and narrows the copy's `index_fields` to the single field that collection embeds — so each collection holds that field's own embedding, and the list you declared on your instance is left untouched by indexing.
<Info>
**Cross-store retrieval:** When a vector search finds a match, Cognee uses the shared `id` to retrieve the full DataPoint from the graph store, which holds all properties (not just the indexed field). This is how Cognee returns complete results from a semantic search.
</Info>
For custom scalar properties such as external IDs, labels, and tags, see [Custom Data Models](/guides/custom-data-models#custom-fields-and-read-back).
## From DataPoints to the Graph
When you call `add_data_points()`, Cognee automatically:
* Embeds the indexed fields into vectors
* Converts the object into **nodes** and **edges** in the knowledge graph
* Stores provenance in the relational store
This is how Cognee creates both **semantic similarity** (vector) and **structural reasoning** (graph) from the same unit.
## Examples and details
<Accordion title="Example: indexing only one field">
```python theme={null}
from typing import Annotated
from cognee.infrastructure.engine import DataPoint, Embeddable
class Person(DataPoint):
name: Annotated[str, Embeddable()]
age: int
```
Only `"name"` is semantically searchable
</Accordion>
<Accordion title="Example: Book → Author transformation">
```python theme={null}
from typing import Annotated
from cognee.infrastructure.engine import DataPoint, Embeddable
class Book(DataPoint):
title: Annotated[str, Embeddable()]
author: Author
# Produces:
# `Node(Book)` with `{title, type, ...}`
# Node(Author) with {name, type, ...}
# Edge(Book → Author, type="author")
```
</Accordion>
<Accordion title="Relationship syntax options">
```python theme={null}
# Simple relationship
`author: Author`
# With edge metadata
`has_items: (Edge(weight=0.8), list[Item])`
# List relationship
`chapters: list[Chapter]`
# Typed edge between siblings — flat relationship rows, not nesting
`cites: list[Edge[Book, Book]]`
# Reference an existing node by its identity value instead of nesting it
`genre: Annotated[Genre, FromIdentity()] | None`
```
The first three forms nest one DataPoint inside another. `Edge[Source, Target]` and `FromIdentity()` instead address nodes by their identity value; both come from `cognee.low_level` and are Python-SDK-only. See [Custom Graph Model](/guides/custom-graph-model#step-1-define-your-entity-classes-and-relationships).
</Accordion>
<Accordion title="Built-in DataPoint types">
Cognee ships with several built-in DataPoint types:
* **Documents** — wrappers for source files (Text, PDF, Audio, Image)
* `Document` (`metadata.index_fields=["name"]`)
* **Chunks** — segmented portions of documents
* `DocumentChunk` (`metadata.index_fields=["text"]`)
* **Summaries** — generated text or code summaries
* `TextSummary` / `CodeSummary` / `GlobalContextSummary` (`metadata.index_fields=["text"]`)
* `GlobalContextSummary` powers the [Global Context Index](/core-concepts/further-concepts/global-context-index)
* **Entities** — named objects (people, places, concepts)
* `Entity`, `EntityType` (`metadata.index_fields=["name"]`)
* **Edges** — relationships between DataPoints
* `Edge` — links between DataPoints
</Accordion>
<Accordion title="Example: custom DataPoint with best practices">
```python theme={null}
from typing import Annotated
from cognee.infrastructure.engine import DataPoint, Embeddable
class Product(DataPoint):
name: Annotated[str, Embeddable()]
description: Annotated[str, Embeddable()]
price: float
category: Category
```
**Best Practices:**
* **Keep it small** — one concept per DataPoint
* **Index carefully** — only fields that matter for semantic search
* **Use built-in types first** — extend with custom subclasses when needed
* **Version deliberately** — track changes with `version`
* **Group related points** — with `belongs_to_set`
</Accordion>
<Accordion title="Updating DataPoints">
To update a custom DataPoint, mutate its fields, call `update_version()` to record the change, then re-add it with `add_data_points()`. The upsert replaces the existing node in all three stores.
```python theme={null}
from cognee.infrastructure.engine import DataPoint
from cognee.tasks.storage import add_data_points
product = Product(name="Widget", price=9.99, description="Original")
await add_data_points([product])
# Later — update the description
product.description = "Improved description"
product.update_version() # version → 2, updated_at refreshed
await add_data_points([product])
```
For documents and files remembered via `cognee.remember()`, use [`cognee.update()`](/python-api/update) instead — it replaces the existing item in the target dataset under the same `data_id`, re-extracting only the chunks the edit touched and falling back to a full delete, re-add and graph rebuild for that dataset when the [chunk-level preconditions](/python-api/update#how-it-works) are not met.
**How versioning works**
Changing a field on a DataPoint does **not** automatically create a new revision or persist anything by itself.
In other words, versioning is manual:
* Edit the DataPoint fields
* Call `update_version()` if this should count as a new revision
* Re-add the DataPoint with `add_data_points()` to persist the updated state
By calling `update_version()`, you mark your in-memory object as a new revision before writing it back with `add_data_points()`. It does two things:
* Increments `version` by 1. New DataPoints start at `version=1`.
* Sets `updated_at` to the current UTC timestamp in milliseconds.
</Accordion>
<Accordion title="Deleting DataPoints">
Use the [`cognee.datasets`](/python-api/datasets) API:
```python theme={null}
import cognee
# Soft-delete one item by ID (default — marks as deleted)
await cognee.datasets.delete_data(dataset_id=ds.id, data_id=item.id)
# Hard-delete to remove from all stores permanently
await cognee.datasets.delete_data(dataset_id=ds.id, data_id=item.id, mode="hard")
# Remove all items from a dataset (keeps the dataset itself)
await cognee.datasets.empty_dataset(dataset_id=ds.id)
# Delete all datasets for the current user
await cognee.datasets.delete_all()
```
</Accordion>
<Accordion title="Dataset routing: which dataset receives add_data_points output?">
`add_data_points()` does not accept a `dataset` parameter directly. Dataset assignment is carried by the [`PipelineContext`](/core-concepts/building-blocks/pipelines) (`ctx`) that Cognee injects automatically when your task runs inside `run_pipeline`.
```python theme={null}
async for _ in run_pipeline(
tasks=[my_task], # task calls add_data_points internally
data=my_data,
datasets=["my_dataset"], # <-- this sets ctx.dataset
):
pass
```
Inside the task, `ctx.dataset` holds the resolved dataset object. `add_data_points` uses it to write provenance records (user, dataset, data item) to the relational store.
If you call `add_data_points` **outside** a pipeline (without a `ctx`), nodes and edges are still written to the graph and vector stores, but no dataset-level provenance is recorded — the data is not associated with any named dataset.
</Accordion>
<Accordion title="embed_triplets: graph-structure embeddings">
`add_data_points` accepts an `embed_triplets: bool = False` parameter. When set to `True`, Cognee derives `(subject → predicate → object)` triplets from the graph edges and indexes each one as a `Triplet` DataPoint embedding.
```python theme={null}
await add_data_points([product], embed_triplets=True)
```
Each triplet is embedded as a single text string in the form:
```
<subject text> -› <predicate> -› <object text>
```
These triplets are derived from the graph you just wrote, but they are not added back as extra graph nodes or edges.
This allows vector search to match not just individual nodes, but **relationship patterns** across the graph.
Use `embed_triplets=True` when:
* Your queries describe relationships (e.g., "products made by company X")
* You want to retrieve graph edges via semantic similarity, not just individual nodes
Leave it `False` (the default) for standard node-level retrieval.
For the same idea applied after graph creation, see [Triplet Embeddings](/guides/memify-triplet-embeddings).
</Accordion>
<Accordion title="Deduplication: preventing duplicate entities">
By default, each DataPoint receives a random UUID4 on instantiation. To make identical entities share the same node — and avoid duplicates — mark one or more fields as the **deduplication key**.
<Tabs>
<Tab title="Option 1: `Dedup()`">
```python theme={null}
from typing import Annotated
from cognee.infrastructure.engine import DataPoint, Embeddable, Dedup
class Product(DataPoint):
sku: Annotated[str, Dedup()]
name: Annotated[str, Embeddable()]
price: float
```
Cognee automatically populates `identity_fields` from the `Dedup()` annotations. No explicit `metadata` declaration is needed.
</Tab>
<Tab title="Option 2: `identity_fields`">
```python theme={null}
from cognee.infrastructure.engine import DataPoint
from cognee.infrastructure.engine.models.DataPoint import MetaData
class Product(DataPoint):
sku: str
name: str
price: float
metadata: MetaData = {"index_fields": ["name"], "identity_fields": ["sku"]}
```
</Tab>
</Tabs>
**How it works**
When one or more identity fields are defined and all of them resolve on the instance, Cognee generates a deterministic UUID5 from the class name and the field values instead of a random UUID4. Concretely, the id is `uuid5(NAMESPACE_OID, "ClassName:value1|value2|...")`, with each value lower-cased and its spaces normalized to `_` — so the class name namespaces the id and two different node types never collide on the same input. Two `Product` instances with the same `sku` produce the same `id`, so `add_data_points` upserts them onto the same graph node rather than creating a duplicate. If an identity field is missing and has no default, Cognee falls back to a random UUID4.
You can recompute this id at any time without building an instance via `Product.id_for(sku_value)` — it returns the same UUID5 the instance would get, which is useful for lookups.
**Using an external system's ID**
If your records already have a stable identifier from an external system (for example a CMS entry UUID), pass it directly as `id`. An explicit `id` always takes precedence over `Dedup()` / `identity_fields` generation, so that external identifier is what sits on the graph node:
```python theme={null}
from uuid import UUID
# external_uuid comes from your source system (must be a valid UUID)
product = Product(id=UUID(external_uuid), sku="ABC-123", name="Widget", price=9.99)
await add_data_points([product])
```
Re-adding an object with the same `id` upserts onto the existing node, so syncing an update from the source system overwrites the node in place instead of creating a duplicate. (`id` must be a UUID — if your external key is not already a UUID, either normalize it into one or feed it through `Dedup()` / `identity_fields` so Cognee derives a deterministic UUID5 from it.)
**Checking whether a DataPoint already exists**
Because the ID is deterministic, you can check existence by constructing the instance (which generates the identity ID) and querying the graph engine:
```python theme={null}
from cognee.infrastructure.databases.graph import get_graph_engine
candidate = Product(sku="ABC-123", name="Widget", price=9.99)
graph = await get_graph_engine()
existing = await graph.get_node(str(candidate.id))
if existing:
print("Already in the graph:", existing)
else:
print("New entity — id:", candidate.id)
```
</Accordion>
<Accordion title="Typing the metadata field (avoiding type-checker warnings)">
The base `metadata` field is typed as `MetaData`, a `TypedDict` — not a plain `dict`. Annotating a subclass's `metadata` as `dict` (e.g. `metadata: dict = {...}`) narrows the type incompatibly, which makes type checkers like Pylance warn that the override conflicts with the base field. Annotate it with `MetaData` instead:
```python theme={null}
from cognee.infrastructure.engine import DataPoint
from cognee.infrastructure.engine.models.DataPoint import MetaData
class Product(DataPoint):
name: str
description: str
metadata: MetaData = {"index_fields": ["name", "description"]}
```
`MetaData` is defined in `cognee.infrastructure.engine.models.DataPoint` and must be imported from that path — it is **not** re-exported from `cognee.low_level` or `cognee.infrastructure.engine`, so `from cognee.low_level import MetaData` fails.
The `TypedDict` accepts four keys:
* `index_fields: list[str]` — required; fields embedded for vector search
* `type: str` — optional
* `identity_fields: list[str]` — optional; the [deduplication key](#deduplication-preventing-duplicate-entities)
* `transparent: bool` — optional; marks the class as a [container that is never stored](#transparent-containers-nodes-that-group-rather-than-describe)
Prefer the [`Embeddable()` / `Dedup()` annotations](#deduplication-preventing-duplicate-entities) when you can — Cognee derives `metadata` from them automatically, so you never declare the field (or its type) by hand.
</Accordion>
<Accordion title="Defining a DataPoint type at runtime (instead of writing a class)">
A DataPoint type is always a Pydantic class, and its fields must be **declared**. `add_data_points` rejects anything that is not a DataPoint instance, so a plain dict of values raises `InvalidDataPointsInAddDataPointsError`, and a value passed for a field the class does not declare is silently dropped — Cognee reads only declared fields when it converts an object into nodes and edges, so that value never reaches the stores.
You do not have to write that class by hand, though. Because `DataPoint` is an ordinary Pydantic model, you can build the type at runtime with Pydantic's [`create_model`](https://docs.pydantic.dev/latest/concepts/models/#dynamic-model-creation) and `__base__=DataPoint`:
```python theme={null}
from typing import Annotated
from pydantic import create_model
from cognee.infrastructure.engine import DataPoint, Dedup, Embeddable
from cognee.tasks.storage import add_data_points
fields = {
"sku": (Annotated[str, Dedup()], ...),
"name": (Annotated[str, Embeddable()], ...),
"price": (float, 0.0),
}
Product = create_model("Product", __base__=DataPoint, **fields)
await add_data_points([Product(sku="ABC-123", name="Widget", price=9.99)])
```
Such a class behaves exactly like a written one: `Embeddable()` and `Dedup()` are picked up, `Product.id_for("ABC-123")` recomputes the same deterministic id, and a field typed as another DataPoint becomes an edge.
Two things to watch:
* **The name you pass is the class name.** It plays that role everywhere this page mentions it — the node `type`, the [`Class_field` vector collections](#indexing--embeddings), and the [deduplication id namespace](#deduplication-preventing-duplicate-entities). Build each shape once and reuse the class — a new name means new collections and new ids for the same records.
* **`index_fields` must be a class-level default.** If you declare `metadata` explicitly instead of using the annotations, pass it as a field definition (`"metadata": (MetaData, {"index_fields": ["name"]})`, importing [`MetaData`](#typing-the-metadata-field-avoiding-type-checker-warnings) as described above). Setting `metadata` on an instance does not reach storage: Cognee rebuilds the node from the class field defaults, so instance-level index fields are lost and nothing is embedded.
</Accordion>
<Accordion title="Transparent containers: nodes that group rather than describe">
Some DataPoint classes exist only to hold other DataPoints — a wrapper you pass as `graph_model` so the LLM returns a *list* of entities, for example. Set `metadata["transparent"] = True` to declare that the class is structure rather than content:
```python theme={null}
from typing import List
from cognee.infrastructure.engine import DataPoint
from cognee.infrastructure.engine.models.DataPoint import MetaData
class Person(DataPoint):
name: str
metadata: MetaData = {"index_fields": ["name"], "identity_fields": ["name"]}
class PeopleGraph(DataPoint):
people: List[Person]
metadata: MetaData = {"index_fields": [], "transparent": True}
```
Wherever a transparent node appears, it is **replaced by its DataPoint children**:
* The wrapper itself is **never stored** — no node, and never an edge endpoint. Without `transparent`, `PeopleGraph` becomes a node with a `people` edge to each `Person`.
* Its DataPoint children are **promoted to top-level roots**, so one extraction can produce several top-level nodes, or none at all if the wrapper came back empty.
* Nesting works: a transparent child of a transparent wrapper resolves too, and a field pointing at a transparent node gets edges to that node's children instead.
* `belongs_to_set` is not inherited by the children — a wrapper's NodeSets are not its content.
**Non-DataPoint fields on a transparent class are dropped.** Once the wrapper is gone, a plain scalar declared on it has nowhere to live. Rather than disappearing silently, each one logs a warning the first time it carries a value (once per class field):
```text theme={null}
PeopleGraph is marked transparent but carries data in 'title'; a transparent node is never
stored, so that value is dropped. If the field is worth searching for, the class is not a
container - remove metadata['transparent'].
```
Fields inherited from `DataPoint` itself and empty values are exempt, so an unset optional relationship never warns.
<Note>
Transparency is a property of the model. How widely a chunk links into the graph extracted from it is a property of the run — see [`cognify(chunk_attachment=...)`](/python-api/cognify#chunk-attachment). Under `"direct"` (the default), a chunk whose extracted root is transparent links to the children that replaced it.
</Note>
</Accordion>
<Columns>
<Card title="Tasks" icon="square-check" href="/core-concepts/building-blocks/tasks">
Learn how DataPoints are created and processed
</Card>
<Card title="Pipelines" icon="git-merge" href="/core-concepts/building-blocks/pipelines">
See how DataPoints flow through processing workflows
</Card>
<Card title="Main Operations" icon="play" href="/core-concepts/main-operations/remember">
Understand how DataPoints are used in Remember, Improve, and Recall
</Card>
</Columns>
# PipelineContext
Source: https://docs.cognee.ai/core-concepts/building-blocks/pipeline-context
Typed runtime context automatically injected into pipeline task functions.
# PipelineContext: Runtime Context for Tasks
`PipelineContext` is a typed dataclass that the pipeline framework automatically builds and injects into any [task](/core-concepts/building-blocks/tasks) that declares a `ctx` parameter. It carries the user, dataset, and per-item context for the current pipeline run, and provides an `extras` dict for custom state.
## Fields
| Field | Type | Description |
| --------------- | ---------------- | ------------------------------------------------------------------------------------------------ |
| `user` | `Any` | The user that triggered the pipeline. Used for access control and provenance. |
| `dataset` | `Any` | The resolved dataset object for the current run. |
| `data_item` | `Any` | The individual data item being processed in this pipeline execution. |
| `pipeline_name` | `Optional[str]` | The name passed to `run_pipeline` or `run_tasks`. |
| `extras` | `Dict[str, Any]` | Arbitrary key/value state you can pass into the pipeline and read in any task. Defaults to `{}`. |
## How injection works
The framework inspects every task function's signature at construction time. If it finds a parameter named **`ctx`**, it passes the current `PipelineContext` as that argument when the task runs. Matching is by **parameter name**, not by type annotation.
```python theme={null}
from cognee.modules.pipelines.models.PipelineContext import PipelineContext
# ctx is injected automatically — no manual wiring needed
async def my_task(data, ctx: PipelineContext = None):
if ctx:
print(ctx.user)
print(ctx.dataset)
print(ctx.pipeline_name)
```
Tasks that do not declare `ctx` simply receive no context and are unaffected.
## Using `extras` for custom pipeline state
Pass a dict as the `context` argument to `run_pipeline` (or `extras` to `run_tasks`). Every task in the pipeline can read those values from `ctx.extras`.
```python theme={null}
from cognee.modules.pipelines import Task, run_pipeline
from cognee.modules.pipelines.models.PipelineContext import PipelineContext
async def score_items(data, ctx: PipelineContext = None):
multiplier = ctx.extras.get("score_multiplier", 1) if ctx else 1
return [item * multiplier for item in data]
async for _ in run_pipeline(
tasks=[Task(score_items)],
data=[1, 2, 3],
datasets=["my_dataset"],
pipeline_name="scoring_pipeline",
context={"score_multiplier": 10}, # becomes ctx.extras
):
pass
```
`ctx.extras` is always a plain dict — it is never `None`.
## Examples and details
<Accordion title="Accessing user and dataset in a task">
The `user` and `dataset` fields are most useful when you need to write provenance records or apply per-tenant logic:
```python theme={null}
async def store_result(data_points, ctx: PipelineContext = None):
user = ctx.user if ctx else None
dataset = ctx.dataset if ctx else None
data_item = ctx.data_item if ctx else None
for dp in data_points:
await write_to_store(dp, user_id=user.id, dataset_id=dataset.id)
return data_points
```
The built-in `add_data_points` task already does this automatically, so you typically only need to read these fields when writing your own storage tasks.
</Accordion>
<Accordion title="Extras persist across chained tasks">
All tasks in the same pipeline run share the same `PipelineContext` object, so values written during construction remain available in every downstream task:
```python theme={null}
async def filter_task(data, ctx: PipelineContext = None):
threshold = ctx.extras.get("min_threshold", 0) if ctx else 0
return [x for x in data if x >= threshold]
async def label_task(data, ctx: PipelineContext = None):
prefix = ctx.extras.get("label_prefix", "") if ctx else ""
return [f"{prefix}{x}" for x in data]
# Both tasks receive the same extras
async for _ in run_pipeline(
tasks=[Task(filter_task), Task(label_task)],
data=[1, 5, 10],
datasets=["demo"],
context={"min_threshold": 4, "label_prefix": "item_"},
):
pass
```
</Accordion>
<Accordion title="Making ctx optional">
Always default `ctx` to `None` so the task can also be called directly in tests or scripts without a running pipeline:
```python theme={null}
async def my_task(data, ctx: PipelineContext = None):
name = ctx.pipeline_name if ctx else "standalone"
...
```
</Accordion>
<Accordion title="Dataset routing: which dataset does add_data_points write to?">
The built-in `add_data_points` discovers which dataset to attribute nodes and edges to entirely from `ctx.dataset`:
```python theme={null}
async def add_data_points(data_points, ..., ctx: PipelineContext = None):
user = ctx.user if ctx else None
data_item = ctx.data_item if ctx else None
dataset = ctx.dataset if ctx else None
if user and dataset and data_item:
# writes dataset-level provenance (dataset_id, data_id)
# alongside the graph and vector writes
```
When you call `run_pipeline(..., datasets=["my_dataset"])`, the pipeline resolves each name (or UUID) into a `Dataset` object and places it on `ctx.dataset`, so every task — including `add_data_points` — receives the resolved dataset automatically. You never pass the dataset to the task itself.
**Called outside a pipeline** (`await add_data_points(points)` with `ctx=None`), `user`, `dataset`, and `data_item` are all `None`, so the provenance block is skipped: the nodes and edges are still written to the graph and vector stores, but **no dataset-level provenance is recorded**. Run the task through `run_pipeline`/`run_tasks` with a `datasets=[...]` argument whenever you need per-[dataset](/core-concepts/further-concepts/datasets) attribution.
</Accordion>
<Columns>
<Card title="Tasks" icon="square-check" href="/core-concepts/building-blocks/tasks">
Learn how tasks are defined and composed
</Card>
<Card title="Pipelines" icon="git-merge" href="/core-concepts/building-blocks/pipelines">
See how tasks and context flow through pipeline runs
</Card>
<Card title="Custom Tasks & Pipelines" icon="workflow" href="/guides/custom-tasks-pipelines">
Step-by-step guide to building your own pipeline
</Card>
</Columns>
# Pipelines
Source: https://docs.cognee.ai/core-concepts/building-blocks/pipelines
Orchestrate tasks into coordinated data processing workflows.
## What pipelines are
Pipelines coordinate ordered [Tasks](../building-blocks/tasks) into a reproducible workflow. Default Cognee operations like [Remember](../main-operations/remember) run on top of the same execution layer. You typically do not call low-level functions directly; you trigger pipelines through the higher-level operations unless you need staged control.
## Prerequisites
* **Dataset**: a container (name or UUID) where your data is stored and processed. Every document remembered by Cognee belongs to a dataset.
* **User**: the identity for ownership and access control. A default user is created and used if none is provided.
* More details are available below
## How pipelines run
Somewhat unsurprisingly, the function used to run pipelines is called `run_pipeline`.
Cognee uses a **layered execution model**: a single call to `run_pipeline` orchestrates **multi-dataset processing** by running **per-file pipelines** through the sequence of tasks.
* **Statuses** are yielded as the pipeline runs and written to **databases** where appropriate
* **User access** to datasets and files is carefully verified at each layer
* **Pipeline run information** includes dataset IDs, completion status, and error handling
* **Background execution** uses queues to manage status updates and avoid database conflicts
<Accordion title="Pipeline Names and Caching">
Every `run_pipeline` call takes a `pipeline_name` parameter (default: `"custom_pipeline"`) and a `use_pipeline_cache` flag (default: `False`). These two values together control whether a pipeline re-processes a dataset that was already handled.
### Reserved pipeline names
Two pipeline names are used internally and carry special meaning:
| Name | Used by | Behavior |
| ------------------ | ------------------ | ----------------------------------------------------------------------------------------------------------------------- |
| `cognify_pipeline` | `cognee.cognify()` | Runs with `use_pipeline_cache=False`; starts a new dataset-level run instead of skipping because of a prior dataset run |
| `add_pipeline` | `cognee.add()` | Runs with `use_pipeline_cache=False`; starts a new dataset-level run instead of skipping because of a prior dataset run |
Both built-in operations run with `use_pipeline_cache=False`, so they do **not** short-circuit based on a dataset-level `DATASET_PROCESSING_COMPLETED` or `DATASET_PROCESSING_STARTED` record. They start a new dataset-level run, while per-document pipeline status can still skip data items that already completed for that pipeline. Concurrent runs on the **same** dataset are kept safe by a per-dataset lock (see the "Per-dataset serialization" section below) rather than by the cache check.
The lower-level `cognee.add()` step, which is also used inside `remember()`, always resets the stored status for **both** `add_pipeline` and `cognify_pipeline` before running, so that new data can be re-processed by the downstream `cognify()` step on the next call.
<Warning>
Do not use `cognify_pipeline` or `add_pipeline` as `pipeline_name` values in your own `run_pipeline` calls. Reusing these names causes your pipeline to read and write the same status records as the built-in operations, which can lead to unexpected skipping or incorrect state resets.
</Warning>
### How `use_pipeline_cache` works
When `use_pipeline_cache=True`, Cognee checks the relational database for the most recent run of `pipeline_name` on the target dataset before executing:
* If the stored status is **`DATASET_PROCESSING_COMPLETED`** → the pipeline yields the cached result and returns immediately without re-running the tasks.
* If the stored status is **`DATASET_PROCESSING_STARTED`** → the pipeline yields the in-progress status and returns, preventing duplicate concurrent runs.
* If there is **no prior record** (new dataset or new pipeline name) → the pipeline runs normally.
When `use_pipeline_cache=False` (the default for custom pipelines, and the mode used by `cognee.add()` and `cognee.cognify()`), the dataset-level qualification check is skipped entirely — the prior dataset run status is not read — and the pipeline starts a new dataset-level run regardless of any prior dataset completion status. Per-document pipeline status is checked later during task execution, so individual data items that already completed for that pipeline can still be skipped. Safety against concurrent runs on the same dataset is provided by the per-dataset lock described below instead of by this check.
</Accordion>
<Accordion title="Per-dataset serialization">
Pipeline runs are serialized **per dataset**. Before a run starts, `run_pipeline_per_dataset` acquires a lock keyed on the dataset ID, so two runs that target the **same** dataset execute one after another — the second waits until the first finishes — while runs on **different** datasets still proceed in parallel. This protects each dataset from concurrent writers (for example, two `cognify()` calls on the same dataset) without globally serializing all pipeline activity.
Delete operations share the **same** per-dataset lock. Deleting a dataset or a single data item (including the memory-clearing `forget()` paths) acquires the lock keyed on that dataset ID before it mutates anything, so a delete waits for an in-flight `add()`, `cognify()`, or `memify()` run on the same dataset to finish — and a pipeline run started while a delete is in progress waits for the delete. Two deletes targeting the same dataset serialize the same way. Deletes on **different** datasets still proceed in parallel.
<Warning>
The lock is **process-local** — it is an in-memory `asyncio.Lock`. It only serializes runs and deletes within a single process/event loop and does **not** guard against multiple processes or workers operating on the same dataset at once. Cognee is designed to run as a single process; do not point multiple processes or workers at the same stores.
</Warning>
### Nested (re-entrant) runs
A pipeline task may legitimately start another pipeline on the same dataset — for example, a session-driven run calling `add()` or `cognify()` on the dataset it is already processing. Because the per-dataset lock is not re-entrant, re-acquiring it from within the same execution would self-deadlock. Cognee detects that the current execution already holds the dataset's lock and lets the nested run proceed **without re-locking**; external runs (and deletes) on that dataset stay queued behind the lock the ancestor run holds. The same re-entrancy applies to deletes, since they acquire the lock from the same registry.
</Accordion>
<Accordion title="Custom pipeline naming">
For your own pipelines, choose a unique `pipeline_name` that does not conflict with `cognify_pipeline` or `add_pipeline`. Using a unique name means:
* State tracking is isolated to your pipeline — a completed run of the built-in `cognify()` will not affect your pipeline's qualification check.
* If you enable `use_pipeline_cache=True` for your custom pipeline, you must reset its status manually (via `reset_dataset_pipeline_run_status`) when you want to re-process a dataset.
```python theme={null}
# Custom pipeline with a unique name — safe to use alongside the built-in memory workflows
async for run_info in run_pipeline(
tasks=tasks,
data=text,
datasets=["my_dataset"],
pipeline_name="my_enrichment_pipeline", # unique name, no conflict
use_pipeline_cache=False, # default: always re-runs
):
pass
```
</Accordion>
<Accordion title="Crash recovery and stuck pipelines">
How an interrupted run ends depends on **how** it was interrupted:
* A **hard kill** — `SIGKILL`, a container OOM, power loss — stops the process before any cleanup code can run, so the pipeline run record in the relational database is left with a `DATASET_PROCESSING_STARTED` status.
* A **cancellation** — a graceful server shutdown or restart, or any other `asyncio.CancelledError` delivered to the task driving the run — is cooperative, so the run's terminal handler gets to run. The run is finalized as `DATASET_PROCESSING_ERRORED` rather than being left in progress, and the cancellation is then re-raised so it still propagates to the caller exactly as before.
<Note>
Marking cancelled runs as errored is new. Previously the terminal handler caught only `Exception`, and `asyncio.CancelledError` is a `BaseException`, so a cancelled run never reached the error-logging step and its record stayed at `DATASET_PROCESSING_STARTED` indefinitely.
</Note>
On the cancellation path the run is now finalized like any other failed run: the rollback handler runs first, then the terminal `pipeline_runs` row is written with `outcome` `FAILED`, `error_class` `CancelledError`, and a scrubbed `error_message`, and a `PipelineRunErrored` event is yielded before the `CancelledError` is re-raised. Behavior for ordinary exceptions is unchanged.
Because `cognify()` and `add()` run with `use_pipeline_cache=False`, they do not consult the dataset-level status at all — the next call starts a new dataset-level run, serialized by the per-dataset lock. A stuck `DATASET_PROCESSING_STARTED` record therefore does not block the built-in operations either way, although completed data items can still be skipped by their per-document pipeline status.
| How the run was interrupted | Pipeline run status left behind | What `cognify()` does on retry | Outcome |
| ----------------------------------------------------------- | ------------------------------- | ------------------------------------------------------------------------------ | -------------------------------------------------------- |
| Hard kill (process killed, OOM, power loss) | `DATASET_PROCESSING_STARTED` | Dataset-level cache check skipped; starts a new run under the per-dataset lock | Runs normally; completed data items may still be skipped |
| Cancelled (graceful shutdown or restart, task cancellation) | `DATASET_PROCESSING_ERRORED` | Dataset-level cache check skipped; starts a new run under the per-dataset lock | Runs normally; completed data items may still be skipped |
| Raised an error | `DATASET_PROCESSING_ERRORED` | Dataset-level cache check skipped; starts a new run under the per-dataset lock | Runs normally; completed data items may still be skipped |
### Automatic recovery
Two automatic layers clean up failed or abandoned cognify runs, so a re-run can resume instead of redoing or half-skipping work:
* **Rollback on error** — when a cognify run fails (including when it is cancelled), an error handler rolls back the partial graph, vector, and relational artifacts written by that run and clears the per-document `cognify_pipeline` status of the data items the failed run touched. Data items completed in earlier successful runs keep their status and are still skipped, so the next run resumes at the first unprocessed document and reprocesses the rolled-back ones cleanly.
* **Startup recovery for stale runs** — when the API server starts, it finds datasets whose latest cognify run is still `DATASET_PROCESSING_STARTED`, rolls back those older than `COGNEE_STALE_RUN_RECOVERY_MIN_AGE_SECONDS` (env var, default `3600` seconds), and resets their status to `DATASET_PROCESSING_INITIATED` so the dataset is no longer reported as "already being processed" and can be cognified again. Younger runs are left alone because they may still be executing in another live worker or replica. This runs only during API server startup — library-only usage does not trigger it. Because it keys on `DATASET_PROCESSING_STARTED`, it covers hard kills; a cancelled run already reaches the terminal `DATASET_PROCESSING_ERRORED` status on its own, so there is nothing for the reaper to pick up and no wait for the minimum-age threshold.
By default a per-document processing error aborts the whole run; set the `RAISE_INCREMENTAL_LOADING_ERRORS` env var to `false` to log the error and continue with the remaining data items instead.
### Manual reset
The `reset_dataset_pipeline_run_status` helper below is still useful for **custom pipelines that opt into `use_pipeline_cache=True`**, where a stuck `DATASET_PROCESSING_STARTED` record would otherwise cause the cache check to report the dataset as "already being processed" and skip a re-run. That now applies to hard kills only: the cache check lets a `DATASET_PROCESSING_ERRORED` record through, so a run ended by cancellation re-runs on its own without a manual reset.
<AccordionGroup>
<Accordion title="Unblock a stuck pipeline">
To unblock a stuck pipeline that uses `use_pipeline_cache=True`, call `reset_dataset_pipeline_run_status`. It writes a new `DATASET_PROCESSING_INITIATED` record, which clears the stuck status so the next cached run is no longer skipped.
```python theme={null}
from uuid import UUID
from cognee.modules.pipelines.layers.reset_dataset_pipeline_run_status import (
reset_dataset_pipeline_run_status,
)
# Reset all pipelines on a dataset
await reset_dataset_pipeline_run_status(dataset_id=my_dataset.id, user=current_user)
# Or reset only specific pipelines by name
await reset_dataset_pipeline_run_status(
dataset_id=my_dataset.id,
user=current_user,
pipeline_names=["cognify_pipeline"],
)
```
**Parameters**
| Parameter | Type | Required | Description |
| ---------------- | ------------------- | -------- | ---------------------------------------------------------------------------- |
| `dataset_id` | `UUID` | Yes | The ID of the dataset whose pipeline runs should be reset |
| `user` | `User` | Yes | The user object used for ownership lookup |
| `pipeline_names` | `list[str] \| None` | No | If provided, only runs for these pipeline names are reset; omit to reset all |
</Accordion>
<Accordion title="What happens after reset">
Once reset, calling `cognify()` again is safe:
* Documents that **fully completed** before the crash (their per-document `pipeline_status` entry is `DATA_ITEM_PROCESSING_COMPLETED`) are skipped — no duplicate graph nodes or embeddings are written.
* Documents that were **mid-processing** when the crash occurred will be reprocessed from the beginning. These items will be re-chunked, re-extracted, and re-embedded.
</Accordion>
</AccordionGroup>
<Note>
`reset_dataset_pipeline_run_status` resets the dataset-level run status only. It does not clear per-document status. Documents that completed before the crash remain marked as completed and are not reprocessed.
</Note>
</Accordion>
<Accordion title="What gets stored in a pipeline run record">
Pipeline runs are persisted in the relational `pipeline_runs` table. Alongside the status, IDs, and pipeline name, the record keeps a `run_info` column with an **audit-only preview** of the input the run was started with. This preview is bounded so a single run cannot grow the table without limit:
* If the input is a list of Cognee `Data` records, only their IDs are stored.
* Any other input is stringified and, if longer than **512 characters**, truncated to a preview that ends with `... [truncated, <N> chars total]`.
* Empty or missing input is recorded as `"None"`.
This preview is intended for inspection and debugging only — Cognee never reads it back during processing. If you need the full input payload (for example, large raw text passed to `add()` or `cognify()`), persist it yourself in object storage or a linked record rather than relying on `run_info` to retain it verbatim.
`run_info` also holds a second key, `progress`, which *is* read back — by `GET /api/v1/datasets/status/progress`. It is an in-flight snapshot of `{completed_items, total_items, current_stage}` written as items finish, and it is the one exception to the append-only rule below: a progress tick **updates the run's existing `DATASET_PROCESSING_STARTED` row in place** rather than inserting. Inserting per tick would grow `pipeline_runs` without bound as batches accumulate, so progress is stored as metadata inside the started state — `PipelineRunStatus` gains no new member for it. Three consequences worth knowing:
* Ticks are throttled to roughly 20 database writes per run (`max(1, total_items // 20)`, with the first and last item always persisted), so the snapshot is coarse by design — that keeps write pressure off the default SQLite backend, which serializes writers behind a file lock.
* Concurrent ticks for the same run race on a read-modify-write with no locking: last write wins. For a display-only signal that only risks a slightly stale snapshot between ticks.
* A late tick can never make a finished run read as running again. Status readers pick the newest row, so a tick that lands after the terminal row merely updates the older `STARTED` row in place, invisibly; and the defensive path for a `STARTED` row that has gone missing drops the tick outright rather than inserting a new `STARTED` row with a later `created_at` than the terminal one.
**The table is otherwise not one row per pipeline run.** It is append-only, and it holds two kinds of row:
* A pipeline run writes **several** rows that share one `pipeline_run_id` (initiated → started → terminal). Only the terminal row carries the run's `outcome` and `tokens_in` / `tokens_out`.
* Non-pipeline operations (`search`, `recall`, `remember`, `forget`, `delete`, `prune`) write exactly **one** row each, with `status`, `pipeline_name` and `pipeline_id` left `NULL` — which is what keeps them invisible to readers that look up a dataset's latest pipeline status.
Two rules follow when you aggregate over the table (for example over [`GET /api/v1/activity/pipeline-runs`](/guides/deploy-rest-api-server)):
1. Deduplicate by `pipeline_run_id` before summing, or a single run is counted once per row it wrote.
2. `parent_operation_id` links a child operation to its parent's `pipeline_run_id`, forming a tree — but token counts already chain up into the parent. Sum one level only; summing across levels double-counts.
Because the table now grows with every operation rather than with every pipeline run, it carries a composite `(created_at, id)` index matching the newest-first, id-tiebroken order that the activity feed and the time-bucketed usage queries read it in. Existing databases pick it up from the usual Alembic upgrade, so there is no manual step unless you run with `ENABLE_AUTO_MIGRATIONS=false` — then apply it with `cognee-cli upgrade` as you would any other revision.
</Accordion>
<Accordion title="pipeline_runs as the record of every operation">
`pipeline_runs` is no longer pipeline-only: it is Cognee's **local activity record** — an audit table of every operation run on the deployment, kept in the deployment's own relational database (SQLite or Postgres, whichever is configured). The rows never leave your instance, are readable by whoever can query that database, and are unrelated to Cognee's anonymous product telemetry.
Operations that never run a pipeline — `search()`, `recall()`, `remember()`, `improve()`, `forget()`, `datasets.delete_data()`, and both prune paths — each write **exactly one** self-contained row when they finish, so a single indexed SQL query answers "what did this user run, did it work, how long did it take, and what did it cost in tokens" without parsing JSON.
These operation rows are written with `status`, `pipeline_name`, and `pipeline_id` left `NULL`. That is deliberate: every status reader filters on `pipeline_name`/`status`, so operation rows are invisible to the caching, crash-recovery, and status-reporting logic described above. Each row still gets its own `pipeline_run_id` — the operation's id, which children reference as `parent_operation_id`. The append-only pattern and the `PipelineRunStatus` enum are unchanged.
### Operation-record columns
All of these columns are nullable and are populated on both operation rows and the terminal (completed/errored) row of a pipeline run:
| Column | Meaning |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `user_id`, `tenant_id` | The identity that triggered the operation |
| `operation_name` | The operation (`"search"`, `"recall"`, `"remember"`, `"improve"`, `"forget"`, `"delete"`, `"prune_data"`, `"prune_system"`) — or the pipeline name on pipeline rows |
| `started_at`, `ended_at` | Timezone-aware start and finish timestamps; subtract for duration |
| `outcome` | `"succeeded"` or `"failed"` (plain string, from the `OperationOutcome` enum) |
| `error_class` | Exception class name on failure, e.g. `"DatasetNotFoundError"` |
| `error_message` | Failure message, PII-scrubbed (emails, secrets, home-directory user names, long digit runs) and truncated to 512 characters |
| `tokens_in`, `tokens_out` | LLM tokens spent inside the operation. `NULL` means not measured; `0` means measured as zero |
| `origin` | The surface that initiated the work: `"sdk"` (default), `"api"`, `"cli"`, `"mcp"`, or `"background"` |
| `session_id` | The session-cache id, when the operation ran in a session |
| `parent_operation_id` | The enclosing run's `pipeline_run_id`, making nesting such as `remember` → `add`/`cognify`/`improve` a queryable tree |
| `background` | `True` when the call only *launched* background work — `outcome="succeeded"` then means "accepted and started", not "the background work finished" |
<Warning>
Token counts **chain to parents**: a parent row's `tokens_in`/`tokens_out` already include everything its children spent. Filter by `parent_operation_id IS NULL`, or read one level at a time — never `SUM` token columns across nesting levels, or you double-count.
</Warning>
### Recording is fail-open
The recorder can never break the operation it records. The wrapped operation's exceptions always propagate unchanged, and a failure to persist the row is logged and swallowed. One known consequence: `prune_system(metadata=True)` drops the relational database including `pipeline_runs`, so its own record is self-erasing by design.
### Cost and accuracy
Recording is a single `INSERT` off the hot path — roughly **2.5 ms per operation**, with no additional LLM or network calls. The recorded token counts are the provider-billed `response.usage` figures when the provider reports usage (falling back to a character-based estimate when it does not), so they include hidden reasoning tokens; usage you read out of this table may therefore be higher than counts derived from visible output alone. Nothing about billing changes — only the accuracy of what Cognee reports.
### Upgrading
The columns are added by a single idempotent Alembic migration (`a7f3c9e1b5d2`). Old rows keep `NULL` in the new columns — there is no backfill, and existing readers are unaffected. The API server applies migrations at startup; for standalone scripts, CI, or self-managed database lifecycles, run [`run_migrations`](/python-api/run-migrations) after upgrading the `cognee` package.
</Accordion>
<Accordion title="PipelineContext and ctx injection">
`PipelineContext` is the runtime context object that Cognee automatically builds and injects into any task that declares a `ctx` parameter. It carries the user, dataset, and per-item context for the current pipeline run, and provides an `extras` dict for custom state.
| Field | Type | Description |
| --------------- | ---------------- | ------------------------------------------------------------------------------------------------ |
| `user` | `Any` | The user that triggered the pipeline. Used for access control and provenance. |
| `dataset` | `Any` | The resolved dataset object for the current run. |
| `data_item` | `Any` | The individual data item being processed in this pipeline execution. |
| `pipeline_name` | `Optional[str]` | The name passed to `run_pipeline` or `run_tasks`. |
| `extras` | `Dict[str, Any]` | Arbitrary key/value state you can pass into the pipeline and read in any task. Defaults to `{}`. |
The framework inspects each task function's signature. If it finds a parameter named `ctx`, it passes the current `PipelineContext` when the task runs. Matching is by parameter name, not by type annotation.
Tasks that do not declare `ctx` simply receive no context and are unaffected.
<AccordionGroup>
<Accordion title="Using extras for custom pipeline state">
Pass a dict as the `context` argument to `run_pipeline` or `extras` to `run_tasks`. Every task in the pipeline can read those values from `ctx.extras`.
```python theme={null}
from cognee.modules.pipelines import Task, run_pipeline
from cognee.modules.pipelines.models.PipelineContext import PipelineContext
async def score_items(data, ctx: PipelineContext = None):
multiplier = ctx.extras.get("score_multiplier", 1) if ctx else 1
return [item * multiplier for item in data]
async for _ in run_pipeline(
tasks=[Task(score_items)],
data=[1, 2, 3],
datasets=["my_dataset"],
pipeline_name="scoring_pipeline",
context={"score_multiplier": 10},
):
pass
```
`ctx.extras` is always a plain dict, never `None`.
</Accordion>
<Accordion title="Accessing user and dataset in a task">
The `user` and `dataset` fields are most useful when you need to write provenance records or apply per-tenant logic:
```python theme={null}
async def store_result(data_points, ctx: PipelineContext = None):
user = ctx.user if ctx else None
dataset = ctx.dataset if ctx else None
data_item = ctx.data_item if ctx else None
for dp in data_points:
await write_to_store(dp, user_id=user.id, dataset_id=dataset.id)
return data_points
```
The built-in `add_data_points` task already does this automatically, so you typically only need to read these fields when writing your own storage tasks.
The built-in `ingest_data` task reads `ctx.dataset` too, but conditionally, and the condition matters if you compose it into your own pipeline. `run_pipeline` resolves and write-checks the run's dataset once, so `ingest_data` reuses `ctx.dataset` instead of re-resolving it per item — but only when that dataset demonstrably is the one the call targets: matching the `dataset_id` argument, or matching the `dataset_name` argument for a dataset the calling user owns in the same tenant. Point `ingest_data` at any other dataset and it ignores `ctx.dataset`, resolving the dataset itself and running the usual write-permission check, so the reuse is a saved lookup and never a widened permission.
</Accordion>
<Accordion title="Making ctx optional">
Always default `ctx` to `None` so the task can also be called directly in tests or scripts without a running pipeline:
```python theme={null}
async def my_task(data, ctx: PipelineContext = None):
name = ctx.pipeline_name if ctx else "standalone"
...
```
</Accordion>
</AccordionGroup>
</Accordion>
<Accordion title="Error handling and exception propagation">
When a task raises while processing a data item, the pipeline logs the error, yields a `PipelineRunErrored` status, and then re-raises the original exception to the caller. A failing data item therefore both surfaces a `PipelineRunErrored` event **and** propagates the underlying exception out of the pipeline run, rather than being silently collapsed into an error status only.
Because of this, wrap pipeline runs in `try`/`except` when you iterate them, so you can react to the propagated exception:
```python theme={null}
try:
async for run_info in run_pipeline(
tasks=tasks,
data=text,
datasets=["my_dataset"],
pipeline_name="my_pipeline",
):
...
except Exception as error:
# the original task exception propagates here after the
# PipelineRunErrored status has been yielded
handle(error)
```
**What a propagated Cognee exception looks like.** Cognee's own exception types derive from `CogneeApiError`, which carries three attributes you can read in the handler: `message`, `name`, and `status_code`. Its `__str__` formats them as `"{name}: {message} (Status code: {status_code})"` — for example, passing a list containing something other than `Task` instances to a custom pipeline raises `"WrongTaskTypeError: tasks argument must be a list of Task class instances, got str in the list. (Status code: 400)"`. The base constructor also populates `Exception.args` with `(message, name)`, so `repr()` and `raise ... from error` chaining behave the way they do for any standard Python exception:
```python theme={null}
from cognee.exceptions import CogneeApiError
try:
...
except CogneeApiError as error:
logger.warning("%s failed with %s", error.name, error.status_code)
raise MyAppError(error.message) from error # error stays reachable as __cause__
```
Not every propagated error is a `CogneeApiError`, though — plain built-in exceptions and errors from underlying database or LLM libraries surface too, so keep the broad `except Exception` above as the safety net.
`CogneeApiError.__init__` also logs the exception centrally, at `ERROR` by default. Some exceptions that represent expected control flow rather than failures — such as an adapter reporting an unsupported capability — opt out with `log=False`, so a missing `ERROR` log line for one of those does not mean the exception was swallowed. It still propagates to your `except` block exactly as above.
</Accordion>
<Accordion title="Layered execution">
* Innermost layer: individual task execution with telemetry and recursive task running in batches
* Middle layer: per-dataset pipeline management and task orchestration
* Outermost layer: multi-dataset orchestration and overall pipeline execution
* Execution modes: blocking (wait for completion) or background (return immediately with "started" status)
* In background mode with no `datasets` passed, the run resolves to every dataset the run's user has write access to — the user supplied in the run's params, or the default user when none is given
</Accordion>
<Accordion title="Customization approaches and tips">
* Use [Remember](../main-operations/remember) for the default ingestion path
* Modify transformation steps without touching low-level functions, avoid going below `run_pipeline`
* Custom tasks let you extend or replace default behavior
</Accordion>
<Accordion title="Users">
* Identity: represents who owns and acts on data. If omitted, a default user is used
* Ownership: every ingested item is tied to a user; content is deduplicated per owner
* Permissions: enforced per dataset (read/write/delete/share) during processing and API access
</Accordion>
<Accordion title="Datasets">
* Container: a named or UUID-scoped collection of related data and derived knowledge
* Scoping: `remember()` writes into a specific dataset, and dataset-scoped pipelines process the dataset(s) you pass
* Lifecycle: new names create datasets and grant the calling user permissions; UUIDs let you target existing datasets (given permission)
</Accordion>
<Columns>
<Card title="Tasks" icon="square-check" href="/core-concepts/building-blocks/tasks">
Learn about the individual processing units that make up pipelines
</Card>
<Card title="DataPoints" icon="circle" href="/core-concepts/building-blocks/datapoints">
Understand the structured outputs that pipelines produce
</Card>
<Card title="Main Operations" icon="play" href="/core-concepts/main-operations/remember">
See how pipelines are used in Remember and lower-level ingestion workflows
</Card>
</Columns>
# Tasks
Source: https://docs.cognee.ai/core-concepts/building-blocks/tasks
Building blocks that transform data in Cognee pipelines.
# Tasks: Smallest Executable Units
Tasks are Cognee's **smallest executable units** — they wrap any Python callable and give it a uniform interface for batching, error handling, and logging. While they can work with anything, Tasks are most powerful when creating or enriching [DataPoints](../building-blocks/datapoints). These same task primitives power the default `remember()` / `improve()` workflows.
## What are Tasks
Tasks are Cognee's **smallest executable units**.
* They wrap any Python callable (function, coroutine, generator, async generator).
* Give a **uniform interface** for batching, error handling, and logging.
* Can work with anything, but are **most powerful when creating or enriching [DataPoints](../building-blocks/datapoints)**.
## Why Tasks Exist
* Normalize different kinds of Python functions so they behave consistently.
* Enable **stream-based processing**: outputs flow directly into the next step.
* Provide **batching controls** for efficiency, especially with LLM or I/O-heavy operations.
* Form the **building blocks** of higher-level [Pipelines](../building-blocks/pipelines).
## Core Concepts
* **Execution**: run functions in a consistent way, regardless of sync/async/gen.
* **Batching**: configurable with `task_config`.
* **Composition**: Tasks can be chained — one Task's output is the next Task's input.
* **Flexibility**: Tasks don't need to handle DataPoints, but Cognee's defaults encourage it.
## Dependencies & Ordering
Tasks often assume a certain **input type** and produce an expected **output type**.
Example flow (educational, not exhaustive):
* Raw data → Documents
* Documents → Chunks
* Chunks → Entities and relationships
* Entities/Chunks → Summaries
* Any DataPoint → Storage
## Built-in Tasks
* **Ingestion**: `resolve_data_directories`, `ingest_data`
* **Classification**: `classify_documents`
* **Access control**: `check_permissions_on_dataset`
* **Chunking**: `extract_chunks_from_documents`
* **Graph extraction**: `extract_graph_from_data`
* **Summarization**: `summarize_text`, `summarize_code`
* **Persistence**: `add_data_points`
## Examples and details
<Accordion title="Task API & Constructor">
```python theme={null}
Task(executable, *args, task_config={...}, **kwargs)
```
**Key parameters:**
* `executable`: Any Python callable (function, coroutine, generator, async generator)
* `task_config`: Configuration for batching, error handling, and logging
* `default_params`: Parameters that are always passed to the executable
</Accordion>
<Accordion title="Supported Task Types">
Cognee automatically detects and handles different Python function types:
* **Functions**: Standard synchronous functions
* **Coroutines**: Async functions using `async def`
* **Generators**: Functions that yield multiple values
* **Async Generators**: Async functions that yield multiple values
Each type is executed appropriately within Cognee's task system.
</Accordion>
<Accordion title="Writing a Custom Task">
```python theme={null}
def my_custom_task(data_chunk):
# Process the data chunk
processed_data = process_chunk(data_chunk)
# Create or enrich DataPoints
datapoint = DataPoint(
content=processed_data,
metadata={"source": "custom_task"}
)
return datapoint
# Wrap it in a Task
my_task = Task(my_custom_task)
```
**Why idempotent, DataPoint-focused functions are easiest to compose:**
* Predictable inputs and outputs
* Easy to chain together
* Clear data flow between steps
</Accordion>
<Accordion title="Execution Flow">
Tasks execute in sequence within [Pipelines](../building-blocks/pipelines), with each Task's output becoming the next Task's input. This creates a data transformation pipeline that builds up to the final knowledge graph used by `remember()`, `improve()`, and the lower-level staged APIs.
</Accordion>
<Columns>
<Card title="DataPoints" icon="circle" href="/core-concepts/building-blocks/datapoints">
The structured units that Tasks create and process
</Card>
<Card title="Pipelines" icon="git-merge" href="/core-concepts/building-blocks/pipelines">
How Tasks are orchestrated into workflows
</Card>
<Card title="Main Operations" icon="play" href="/core-concepts/main-operations/remember">
See Tasks in action during memory ingestion and enrichment
</Card>
</Columns>
# Data Flows
Source: https://docs.cognee.ai/core-concepts/data-flows
Follow the three data pipelines that move memory through Cognee.
The [Architecture](/core-concepts/architecture) page explains where memory lives. This page explains how memory **moves**: three separate Cognee pipelines carry data between your agents, short-term session memory, and the permanent knowledge graph — and all three are built from the same Task-based structure you can use for your own pipelines.
<img alt="Cognee memory data flows: Data Pipeline 1 (session learning), Data Pipeline 2 (self-improvement), and Data Pipeline 3 (ingestion) connect agents, session memory, and permanent memory; all three share the same Task and DataPoint pipeline structure" />
## Data Pipeline 1 — Session learning
Agents — Claude Code coding with codebase memory, Codex reviews that remember decisions, an OpenClaw agent on a long-running autonomous task — write conversation turns, feedback, traces, and guidance into [session memory](/core-concepts/sessions-and-caching) with `remember(data, session_id=...)`. These writes are raw and fast — no chunking, no graph extraction — which is exactly why they don't reach the permanent graph on their own. In the diagram they are the hollow nodes in the session graph: learned this session, not yet synced.
The session-learning pipeline is what bridges them. [`improve(session_ids=[...])`](/core-concepts/main-operations/improve) distills gated session guidance into curated lesson documents, persists session Q\&A and agent traces into the graph, and applies feedback weights so graph elements that helped produce well-rated answers become more influential in later retrieval. With `self_improvement=True` (the default for session writes), this pipeline starts automatically in the background; with `self_improvement=False`, session content stays in the cache until you run `improve()` yourself.
## Data Pipeline 2 — Self-improvement
Once data is in the permanent graph, the self-improvement pipeline enriches it in place. Running [`improve()`](/core-concepts/main-operations/improve) on a dataset adds derived retrieval structures on top of the existing graph — for example triplet indexes, and optionally dataset-level bucket and root summaries via the [global context index](/core-concepts/further-concepts/global-context-index) — so later recall works better without re-ingesting anything.
This pipeline is also how permanent memory feeds back into session learning: after enrichment, new graph relationships can be synced back into the session cache as readable context — the filled nodes in the session graph above are exactly this recalled core — making future session recall faster and better grounded.
## Data Pipeline 3 — Ingestion
Calling [`remember(data)`](/core-concepts/main-operations/remember) without a `session_id` — on anything from Google Drive documents to Slack threads to Postgres records — writes straight to permanent memory. Under the hood this runs the [Add](/core-concepts/main-operations/legacy-operations/add) + [Cognify](/core-concepts/main-operations/legacy-operations/cognify) pipeline: documents are loaded and chunked, entities and relationships are extracted into the knowledge graph, embeddings are indexed in the vector store, and provenance is tracked in the relational store.
## How the two memories meet at read time
Retrieval ties the flows together. [`recall(query, session_id=...)`](/core-concepts/main-operations/recall) checks the session cache first; on a cache miss it falls through to the permanent knowledge graph, and results are tagged with the `_source` they came from. Combined with the sync-back from Data Pipeline 2, the session and the graph continuously exchange context in both directions.
## Anatomy of a memory pipeline
All three flows above — and any pipeline you build yourself — share one structure: a [Pipeline](/core-concepts/building-blocks/pipelines) is an ordered sequence of [Tasks](/core-concepts/building-blocks/tasks), where each Task transforms data and passes [DataPoints](/core-concepts/building-blocks/datapoints) to the next until results land in the stores.
In practice a Task is just a Python function — plain function, coroutine, or generator — wrapped in `Task(...)` so Cognee can batch it, handle its errors, and stream its output into the next step. Even something as trivial as `def task(data): return 2 + 2` is a valid Task; the power comes from chaining them over the same flowing object. The `task1 → … → taskN` chain in the diagram is exactly how the built-in operations run internally, with tasks like `extract_chunks_from_documents`, `extract_graph_from_data`, and `add_data_points`.
Every run is **owned**: it executes on behalf of a user against a dataset, and user access to datasets and files is verified at each layer. That means the built-in pipelines and your custom ones follow the same rules — swap in your own Tasks, give the pipeline a unique name, and it runs with the same ownership, status tracking, and per-dataset serialization as the defaults.
<Columns>
<Card title="Pipelines" icon="git-merge" href="/core-concepts/building-blocks/pipelines">
How pipeline runs, caching, and per-dataset locking work
</Card>
<Card title="Improve" icon="sparkles" href="/core-concepts/main-operations/improve">
The operation behind session learning and self-improvement
</Card>
<Card title="Sessions and Caching" icon="message-square" href="/core-concepts/sessions-and-caching">
How short-term session memory works and expires
</Card>
</Columns>
# Agent Memory Decorator
Source: https://docs.cognee.ai/core-concepts/further-concepts/agent-memory-decorator
Attach Cognee memory retrieval to an async agent function.
`cognee.agent_memory` is a decorator that adds two capabilities to an async agent function: it searches Cognee memory before the function runs and makes the retrieved context available to model calls inside it, and it records the function's output as a session-backed trace that future calls can retrieve as memory.
## How It Works
In agent systems, agents and subagents are often just async functions — they receive input, call an LLM, and return a result. The `cognee.agent_memory` decorator is designed for exactly this pattern. Before the function executes, the decorator searches the configured dataset and provides the result as context inside the function. The function uses that context in its model call — typically by including it in the system prompt. After the function returns, if `save_session_traces=True`, the inputs, outputs, and retrieved context are saved into the session-backed trace store for that agent flow. This means the agent's own execution history becomes searchable memory for future calls.
## Making the Model Call
The decorator does not call the LLM — the function does. When the function calls [`LLMGateway.acreate_structured_output()`](/guides/low-level-llm), the gateway automatically prepends the retrieved memory to the text input. No extra code is needed inside the function — calling LLMGateway normally is sufficient. See [Agent Memory Quickstart](/guides/agent-memory-quickstart) for a working example.
## When to Use It
The `cognee.agent_memory` decorator does two things at the function boundary:
* **Retrieval** (`with_memory=True`): before the function runs, Cognee fetches relevant context from the dataset and makes it available to the model call.
* **Trace persistence** (`save_session_traces=True`): after the function returns, Cognee stores the inputs, outputs, and memory context in the session-backed trace store. Later calls can retrieve those traces as memory, so the agent accumulates context from its own execution history.
Apply it to an agent or subagent function when that function should consistently run with memory, record its executions, or both — without adding that logic inside the function body each time.
## Memory Sources
There are two ways to populate memory that the decorator can read from:
**External knowledge** — process documents ahead of time:
1. Store source data with [`cognee.remember(...)`](/core-concepts/main-operations/remember).
2. Optionally deepen that dataset later with [`cognee.improve(...)`](/core-concepts/main-operations/improve).
**Agent self-memory from traces** — enable `save_session_traces=True`:
* After each call, the decorator writes a trace step into session-backed storage for that agent flow. No separate `remember()` call is needed for that trace persistence step.
* Subsequent calls can retrieve those prior traces as memory, so the agent builds context from its own execution history.
These two memory sources can be combined: external knowledge comes from the dataset, while agent trace memory comes from the session-backed trace store.
If you enable dataset-backed retrieval with `with_memory=True`, `dataset_name` must match the dataset where memory exists. When [access control](/core-concepts/multi-user-mode/multi-user-mode-overview) is enabled, the user needs both read and write access to that dataset. Session-backed trace memory, by contrast, is scoped by `user` and `session_id`.
## Choosing a Retrieval Query
When `with_memory=True`, set one of two retrieval query options:
* **`memory_query_fixed`** — one stable query used on every call. Use this when the function always retrieves the same kind of context.
* **`memory_query_from_method`** — takes the query from a named function parameter. Use this when retrieval should change with input.
If you set neither, the decorator falls back to the first string argument the function was called with (ignoring `user`, `dataset_name`, and `session_id`); if it finds none, it skips retrieval for that call. Set one of the options explicitly whenever the query matters.
<Tabs>
<Tab title="Fixed Query">
```python theme={null}
@cognee.agent_memory(
memory_query_fixed="What animal does the internal codename refer to?",
dataset_name="agent_memory_demo",
)
async def answer_with_memory() -> str:
return await ask_llm("What animal does the internal codename refer to?")
```
</Tab>
<Tab title="Query From Method">
```python theme={null}
@cognee.agent_memory(
memory_query_from_method="question",
dataset_name="agent_memory_demo",
)
async def answer(question: str) -> str:
return await ask_llm(question)
```
</Tab>
</Tabs>
<Accordion title="Parameters">
* **`agent_session_name`** (str, optional): Stable name used to identify the agent connection that the decorator registers on each call. The connection ID is derived from this name combined with the authenticated user's ID (a sanitized prefix of the name plus a short hash of `agent_session_name|user_id`), so the same `agent_session_name` always resolves to the same connection for a given user — and is visible in the Cognee Cloud [Connections](/cognee-cloud/connections/managing-connections) view. If omitted, a fresh UUID is generated per call, so every invocation registers a new, unique connection. The decorator registers the connection before the wrapped function runs and deactivates it after the function returns (or raises).
* **`with_memory`** (bool, default `True`): Enables or disables retrieval.
* **`with_session_memory`** (bool, default `False`): Includes recent session-trace feedback from the session-backed trace store in the injected memory context.
* **`save_session_traces`** (bool, default `False`): Persists execution traces for decorated calls.
* **`memory_query_fixed`** (str, optional): Fixed retrieval query used on every call.
* **`memory_query_from_method`** (str, optional): Name of the function parameter to use as the retrieval query.
* **`memory_system_prompt`** (str, optional): Shapes how retrieved memory is interpreted by the downstream model call.
* **`memory_top_k`** (int, default `5`): Number of memory results to retrieve.
* **`memory_only_context`** (bool, default `False`): Injects raw retrieved context instead of an LLM-generated completion, skipping the retrieval-time answer call.
* **`session_memory_last_n`** (int, default `5`): Number of recent session-trace feedback entries to include when `with_session_memory=True`.
* **`session_id`** (str, optional): Session key used for session-backed trace retrieval and trace persistence. Prefer a dedicated session per decorated function — reusing one `session_id` across entrypoints mixes unrelated trace history.
* **`dataset_name`** (str, optional): Dataset to retrieve memory from when `with_memory=True`. Defaults to `main_dataset`.
* **`user`** (User, optional): User context for retrieval. Required for multi-user or background-job flows.
* **`session_trace_summary`** (bool, default `True`): Generates an LLM summary for each saved trace step. Set to `False` to store the step without that call.
* **`persist_session_trace_after`** (int, optional): Bridges session traces into the permanent graph after every Nth trace step in the session. Requires `save_session_traces=True`.
* **`persist_session_trace_raw_content`** (bool, default `False`): Persists raw return values instead of trace summaries when bridging into the graph.
* **`persist_session_trace_node_set_name`** (str, optional): [Node set](/core-concepts/further-concepts/node-sets) for the persisted traces. Defaults to `agent_trace_feedbacks`.
</Accordion>
<Note>
The decorated function must be `async`. `memory_query_fixed` and `memory_query_from_method` are mutually exclusive — set at most one. `with_session_memory`, `save_session_traces`, and `persist_session_trace_after` require [caching](/core-concepts/sessions-and-caching) (`CACHING=true`, the default); otherwise the decorator raises at definition time.
</Note>
<Columns>
<Card title="Agent Memory Quickstart" icon="bot" href="/guides/agent-memory-quickstart">
End-to-end working example
</Card>
<Card title="Datasets" icon="database" href="/core-concepts/further-concepts/datasets">
Scope memory retrieval to the right dataset
</Card>
<Card title="Search Basics" icon="search" href="/guides/search-basics">
Direct retrieval without a decorator
</Card>
<Card title="Low-Level LLM" icon="cpu" href="/guides/low-level-llm">
`LLMGateway` for direct model calls
</Card>
</Columns>
# Chunkers
Source: https://docs.cognee.ai/core-concepts/further-concepts/chunkers
Learn how Cognee splits documents into smaller pieces.
Chunkers are responsible for splitting large documents into smaller, manageable pieces called chunks. This is a crucial step before embedding and graph extraction, as most embedding models have a limit on the amount of text they can process at once.
## Token-Based Sizing
Cognee uses token-based sizing for chunks, rather than character counts. This means that `chunk_size` refers to the maximum number of tokens allowed in a chunk, which is directly tied to the tokenizer used by your embedding model. This ensures that chunks are always within the model's context window.
The token-counting tokenizer is auto-selected to match your embedding model, so chunk sizes (and the `--dry-run` token estimate) reflect how the model actually tokenizes text. If a matching tokenizer cannot be loaded, Cognee logs a non-fatal advisory warning and falls back to the TikToken tokenizer — token counts become approximate but ingestion continues. To restore exact counts and silence the warning, set `HUGGINGFACE_TOKENIZER` to a tokenizer that matches your embedding model. See [Embedding Providers](/setup-configuration/embedding-providers) for the per-provider tokenizer mapping and override details.
The `chunk_size` ceiling is respected wherever the text gives Cognee a boundary to split on; a run with no boundary at all is emitted whole as a single oversized chunk rather than cut mid-run — see [Unsplittable Text and Oversized Chunks](#unsplittable-text-and-oversized-chunks) below.
## Available Chunkers
Cognee provides several built-in chunkers to handle different types of content:
* **TextChunker**: The default chunker. It splits text by paragraphs while respecting the token limit. It tries to keep paragraphs together but will split them if they exceed the `chunk_size`.
* **CsvChunker**: Designed specifically for CSV data. It splits by rows, ensuring that each chunk contains complete rows and does not break data in the middle of a record. Each row is chunked independently — row-level state (text, size, and chunk index) is reset at each row boundary, so one row's fields are never accumulated onto the next row's chunk.
* **LangchainChunker**: Wraps LangChain's `RecursiveCharacterTextSplitter`. It splits text recursively by characters (e.g., `\n\n`, `\n`, ` `) and supports `chunk_overlap` (in words; default: `10`). Requires `pip install cognee[langchain]`.
* **TextChunkerWithOverlap**: A paragraph-based chunker that supports overlap via a `chunk_overlap_ratio` (a fraction of `chunk_size`, e.g. `0.2` = 20% overlap). Useful for maintaining context across chunk boundaries.
<Note>
**Overlap only works with `LangchainChunker` and `TextChunkerWithOverlap`.** The default `TextChunker` splits strictly at paragraph boundaries and does not use overlap. Calling [`cognee.config.set_chunk_overlap()`](/python-api/config#chunking-configuration) has no effect when using `TextChunker`.
</Note>
## Additional Information
<AccordionGroup>
<Accordion title="Using a Specific Chunker">
```python theme={null}
from cognee.modules.chunking.TextChunker import TextChunker
await cognee.remember(
data="my document text",
dataset_name="my_dataset",
chunker=TextChunker, # or CsvChunker, LangchainChunker
chunk_size=1024, # Maximum tokens per chunk
)
```
</Accordion>
<Accordion title="Using Chunk Overlap">
Chunk overlap causes consecutive chunks to share a portion of text, which helps preserve context at chunk boundaries and can improve entity extraction quality — at the cost of more LLM calls and a slightly larger graph.
When standard document readers instantiate your chunker, they call it as
`chunker_cls(document, get_text=..., max_chunk_size=...)`. The `max_chunk_size`
value is the `chunk_size` you pass to `remember()` or `cognify()`. Extra
chunking parameters such as overlap are **not** forwarded, so set them by
subclassing the chunker and hard-wiring them inside `__init__`.
| Name | Where it appears | Meaning |
| --------------------- | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| `chunk_size` | `remember(..., chunk_size=...)` / `cognify(..., chunk_size=...)` | Outer token ceiling passed into the pipeline. |
| `max_chunk_size` | `TextChunker`, `TextChunkerWithOverlap`, and `LangchainChunker` constructors | Same outer token ceiling, after the pipeline passes it to the chunker. |
| `chunk_size` | `LangchainChunker` constructor | Splitter target size measured with `len(text.split())`, so it behaves like a word-count target. |
| `chunk_overlap_ratio` | `TextChunkerWithOverlap` constructor | Fraction of `max_chunk_size` to repeat between chunks. |
| `chunk_overlap` | `LangchainChunker` constructor | Number of words to repeat between chunks. |
<Tabs>
<Tab title="TextChunkerWithOverlap">
`TextChunkerWithOverlap` takes a `chunk_overlap_ratio` between `0.0` and `1.0` (fraction of `chunk_size`). Because this value cannot be passed through `cognee.config.set_chunk_overlap()` at runtime, configure it by subclassing:
```python theme={null}
import asyncio
import cognee
from cognee.modules.chunking.text_chunker_with_overlap import TextChunkerWithOverlap
class OverlappingChunker(TextChunkerWithOverlap):
def __init__(self, document, get_text, max_chunk_size):
# 20% of chunk_size tokens will overlap with the next chunk
super().__init__(document, get_text, max_chunk_size, chunk_overlap_ratio=0.2)
async def main():
await cognee.remember(
"my_document.txt",
dataset_name="my_dataset",
chunker=OverlappingChunker,
chunk_size=1024, # ~205 tokens will repeat in the next chunk
)
asyncio.run(main())
```
</Tab>
<Tab title="LangchainChunker">
`LangchainChunker` uses the same subclassing pattern and takes the token ceiling as `max_chunk_size`, exactly like the other chunkers, so you can forward the pipeline's value straight through. On top of that it takes its own `chunk_size` (word-count splitter target, default `1024`) and `chunk_overlap` (words shared between chunks, default `10`), which you set yourself. It requires `pip install cognee[langchain]`.
```python theme={null}
import asyncio
import cognee
from cognee.modules.chunking.LangchainChunker import LangchainChunker
class OverlappingLangchainChunker(LangchainChunker):
def __init__(self, document, get_text, max_chunk_size):
super().__init__(
document,
get_text,
max_chunk_size=max_chunk_size, # token ceiling per chunk
chunk_size=512, # word-count splitter target
chunk_overlap=50, # words shared between chunks
)
async def main():
await cognee.remember(
"my_document.txt",
dataset_name="my_dataset",
chunker=OverlappingLangchainChunker,
chunk_size=1024,
)
asyncio.run(main())
```
</Tab>
</Tabs>
</Accordion>
<Accordion title="Custom Chunkers">
You can create a custom chunker by inheriting from the `Chunker` base class and implementing the `read` method. Your chunker must yield `DocumentChunk` objects.
```python theme={null}
from cognee.modules.chunking.Chunker import Chunker
from cognee.modules.chunking.models.DocumentChunk import DocumentChunk
class MyCustomChunker(Chunker):
async def read(self):
async for text in self.get_text():
# Your logic to split text into chunks
yield DocumentChunk(
text="chunk content",
chunk_size=100,
# ... other required fields
)
```
</Accordion>
<Accordion title="Configuring chunk size via Docker or REST API">
The `/cognify` REST API endpoint does not accept a `chunk_size` parameter directly. When you use Cognee through Docker or the HTTP API, set the chunk size via environment variables instead:
```bash theme={null}
# In your .env file (or passed as Docker env vars)
chunk_size=1500 # Max tokens per chunk (default: 1500)
chunk_overlap=10 # Word overlap between chunks (default: 10, only applies to LangchainChunker)
```
With Docker Compose, add these to your `.env` before starting:
```bash theme={null}
chunk_size=2048
```
Then start the server:
```bash theme={null}
docker compose up --build cognee
```
With `docker run`, pass them inline:
```bash theme={null}
docker run -e chunk_size=2048 -e LLM_API_KEY=... cognee/cognee:main
```
<Note>
The `chunk_size` environment variable is read once at startup. Restart the container after changing it.
</Note>
</Accordion>
<Accordion title="Chunk Size and Graph Quality">
The `chunk_size` passed to `remember()` directly affects how the knowledge graph is built:
| | Smaller chunks | Larger chunks |
| ---------------------- | ----------------------------------------------- | --------------------------------------- |
| **Entity granularity** | Fine-grained — each entity fits in fewer chunks | Coarser — entities may span fewer nodes |
| **Context per chunk** | Less surrounding context for the LLM | More surrounding context for the LLM |
| **LLM calls** | More calls (higher cost, slower) | Fewer calls (lower cost, faster) |
| **Best for** | Dense, technical text (code, legal, science) | Narrative or long-form prose |
If `chunk_size` is not set, Cognee auto-calculates it as the minimum of your embedding model's context window and half of your LLM's context window.
</Accordion>
<Accordion title="Unsplittable Text and Oversized Chunks">
`chunk_size` is a ceiling Cognee respects wherever the text gives it a boundary to respect. A run with no sentence or paragraph boundary in it — a protein or DNA sequence, a base64 blob, a very long URL, a private-use glyph run left by PDF extraction — offers none, so it is emitted whole as a **single oversized chunk** rather than cut mid-run. This holds wherever the run sits in the document, including when it is the last thing in the document; a trailing run previously raised `ValueError: Input word … longer than chunking size …`, which failed the entire pipeline run rather than just that document.
For a trailing run, Cognee logs a non-fatal advisory warning naming the token count and the limit; a mid-document oversized run is emitted silently, so this warning is the only log signal an oversized chunk ever produces, and only in the trailing case:
```text theme={null}
Trailing run of 4096 tokens exceeds the chunk size of 1024; emitting it as a single oversized chunk.
```
It is worth watching, because such a chunk can exceed your embedding model's input limit downstream. If you see it and the source is splittable in a way Cognee can't infer, pre-split the content before ingesting or write a custom chunker that understands your format (see the *Custom Chunkers* accordion above).
</Accordion>
<Accordion title="Preserving structured or procedural data (rules, XML, JSON)">
For structured data, the goal is to make chunk boundaries follow the shape of the source. `TextChunker` (and the underlying `chunk_by_paragraph`) preserves sentence and paragraph boundaries where possible, batching complete sentence-sized units up to `chunk_size` before starting a new chunk. It does not understand a "rule", XML element, or JSON record by itself, so treat `chunk_size` as a guardrail rather than a semantic guarantee.
If you have procedural rules, configuration, or other structured content where each unit must stay intact, choose the smallest source unit you would be comfortable retrieving on its own:
1. **Raise `chunk_size` so the whole document fits in one chunk.** Because `chunk_size` is a token budget, any content that fits under it is kept together. Set it large enough to hold your entire ruleset and no splitting occurs:
```python theme={null}
await cognee.remember(
data=my_rules_text,
dataset_name="rules",
chunk_size=8000, # large enough to hold the full ruleset
)
```
2. **Add each rule as its own data item** so every rule becomes an independent document and is processed on its own:
```python theme={null}
await cognee.remember(
data=["Rule 1: ...", "Rule 2: ...", "Rule 3: ..."],
dataset_name="rules",
)
```
3. **Write a custom chunker** (see the *Custom Chunkers* section above) that splits on your own boundaries (e.g. one chunk per rule, per XML element, or per JSON record) when the natural unit isn't a paragraph.
<Note>
Cognee is strongest when the data has concepts, entities, and relationships to reason over. For workflows that must reproduce deterministic API calls or fixed rules verbatim, keep the canonical procedure in a regular file or source system, then use Cognee for the parts that benefit from semantic search and graph reasoning.
</Note>
</Accordion>
</AccordionGroup>
# COGX Exchange Format
Source: https://docs.cognee.ai/core-concepts/further-concepts/cogx
The portable format Cognee uses to import memories from other tools and to export, back up, and move its own.
**COGX (the Cognee eXchange format) is the common shape that memory data takes
when it moves into or out of Cognee.** It is the single format that every
importer translates *into* and every exporter writes *out of* — so adding a new
memory tool, backing up a dataset, or moving data between Cognee instances all
use the same machinery.
You rarely touch COGX directly. It's the layer underneath
[`cognee.remember()`](/core-concepts/main-operations/remember) (for import) and
`cognee.export()` (for export). This page explains what it is so the
[migration tools](/examples/migrate-memory-systems) make sense.
## Why a common format
Every memory tool stores data differently: Mem0 has short memory strings, Letta
has agent memory blocks and message history, Zep and Graphiti have entity
graphs with time-stamped facts. Without a shared format, importing from *N*
tools and exporting to *M* tools would need *N × M* one-off converters.
COGX collapses that into *N + M*. Each tool only has to translate to or from
COGX once:
```
Mem0 ─┐ ┌─► back up to disk
Letta ─┤ ├─► move to another Cognee instance
Zep ─┼─► COGX records ─► Cognee graph ─► COGX records ─┤
Graphiti ┘ (import hub) (export hub) └─► translate onward
```
* **On import**, a *source* adapter reads one tool's export and emits COGX
records. A single loader ingests those records — identical no matter where
they came from. Sources ship for Mem0, LangMem, Letta, Zep, and Graphiti,
and you can [write your own](/examples/migrate-memory-systems#write-your-own-source)
for any other tool.
* **On export**, Cognee dumps a dataset's graph into a COGX archive that you
can store, restore, or hand to another system.
## What's in COGX
COGX represents memory as a stream of typed **records**. Each record describes
one kind of thing a memory system can hold:
| Record | What it represents | Typical source |
| ---------------- | ------------------------------------------------------------------------------ | ------------------------------------- |
| **Document** | Raw source content — a file, an archival passage, or standalone text. | Letta archival memory, uploaded files |
| **Episode** | A conversation: ordered turns, each with a role and timestamp. | Letta message history, Zep episodes |
| **Entity** | An extracted entity, with an optional type, aliases, and description. | Zep / Graphiti nodes |
| **Fact** | A triplet (subject → predicate → object), optionally with a validity window. | Zep / Graphiti edges |
| **Memory** | A single short derived fact ("atomic memory"). | Mem0 memories, LangMem memories |
| **Memory block** | A named, size-bounded core-memory block. | Letta core memory |
| **Raw node** | Any graph node stored verbatim when no typed mapping fits, so nothing is lost. | Cognee → Cognee exports |
Typed records also carry a **scope** (`user_id`, `agent_id`, `session_id`,
`run_id`) and timestamps, so ownership and time information survive the move.
Raw nodes preserve their original graph properties verbatim.
## Archive layout
An exported COGX archive is just a directory you can inspect:
```
my_cogx_archive/
├── manifest.json # format version, source system, record counts
├── documents.jsonl # one JSON record per line, one file per record kind
├── episodes.jsonl
├── entities.jsonl
├── facts.jsonl
├── memories.jsonl
├── memory_blocks.jsonl
├── nodes.jsonl # raw graph nodes that do not map to a typed record
└── ...
```
`manifest.json` records the COGX version the archive was written with (the
current version is `0.1`, exposed as `cognee.migration.COGX_VERSION`). A reader
refuses to load an archive written by a **newer major version** than it
understands, so an out-of-date Cognee won't silently misread a future archive.
When an archive has to travel as a single file — pushing a dataset to Cognee
Cloud, or uploading to the remember API endpoint — the directory is packed as a
`.cogx.tar.gz` tarball. On disk you'll normally work with the plain directory.
## Import modes
When records enter Cognee, a `mode` controls how much processing they get. This
is the main knob you'll actually set:
| Mode | What it does | Cost |
| ----------- | -------------------------------------------------------------------------------------------------------------------------- | ------------------------- |
| `re-derive` | Ingest the raw content and run Cognee's own extraction. The source's own graph is ignored. | LLM calls; richest result |
| `preserve` | Map the source's already-extracted entities and facts straight into the graph. Raw content is stored but not re-processed. | Zero LLM calls |
| `hybrid` | Keep the source's graph **and** re-cognify the raw content. | LLM calls; most complete |
Each source picks a sensible default (Mem0, LangMem, and Letta default to
`re-derive`, Zep/Graphiti to `hybrid`, COGX archives to `preserve`), and you can
override it.
<Note>
A Cognee-origin `cogx` archive writes each document chunk twice — as a
**Document** record *and* as a **Raw node** carrying the chunk's original
properties. A `preserve` restore rehydrates the raw node, so facts that
reference the chunk keep their graph topology instead of dangling, and the
chunk's content survives the round-trip. As a result, CHUNK and hybrid search
work against the dataset after a Cognee → Cognee import.
</Note>
## Where to go next
<CardGroup>
<Card title="Migrate Memory Systems" icon="upload" href="/examples/migrate-memory-systems">
Import from Mem0, LangMem, Letta, Zep, or Graphiti — with runnable examples
</Card>
<Card title="remember()" icon="brain" href="/core-concepts/main-operations/remember">
The operation that ingests COGX records into the graph
</Card>
</CardGroup>
# Datasets
Source: https://docs.cognee.ai/core-concepts/further-concepts/datasets
Organize documents, permissions, and processing with datasets.
## What is a dataset in Cognee?
A dataset is a named container that groups documents and their metadata. It is the main boundary for:
* Organizing content
* Running pipelines
* Applying permissions
Operations that write to a dataset — such as `remember`, `improve`, and `memify` — fall back to a dataset named `main_dataset` when you don't name one. Your first write creates it, so you can start without deciding on a dataset layout up front.
<Warning>
**Dataset isolation** requires specific configuration. See [permissions system](../multi-user-mode/multi-user-mode-overview) for details on access control requirements and supported database setups.
</Warning>
* **[Remember](../main-operations/remember)**:
* Direct new content into a specific dataset (by name or ID)
* If it doesn’t exist, Cognee creates it and associates your permissions
* Items ingested are linked to that dataset and deduplicated within it
* **[Improve](../main-operations/improve)**:
* Runs enrichment against a chosen dataset
* Loads the dataset’s existing graph, checks rights, and runs the improvement pipeline in dataset scope
* Lets you deepen or bridge memory without re-ingesting the source data
* **[Recall](../main-operations/recall)**:
* Queries can be scoped by dataset
* Results and metrics remain separated by dataset
* **[Forget](../main-operations/forget)**:
* Removes memory at item, dataset, or full-user scope
* Uses dataset permissions to decide what the current user can remove
## Dataset name vs dataset id
Every dataset carries two identifiers, and they are not interchangeable:
* **`dataset_name`** — the human-readable string you choose (`"finance"`, `"main_dataset"`). Names cannot contain spaces or dots.
* **`dataset_id`** — the dataset's `UUID`, stored as its primary key.
The id is **derived from the name, not random**: it is a `uuid5` of the dataset name combined with the owner's user id and tenant id. So the same name, used by the same user, always resolves to the same dataset — but the same name used by a *different* user or tenant resolves to a *different* UUID. Datasets are shared across users by id only, never by name: to reach a dataset [shared with you](../multi-user-mode/permissions-system/datasets), pass its `dataset_id`, because passing the name would resolve to (or create) a separate dataset of your own.
On write paths, an unknown **name** creates a new dataset; an unknown **id** raises `DatasetNotFoundError`.
### Which identifier each operation accepts
| Operation | By name | By id |
| --------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------ |
| [`remember()`](../main-operations/remember) / [`add()`](../main-operations/legacy-operations/add) | `dataset_name="finance"` (defaults to `main_dataset`) | `dataset_id=UUID(...)` — takes precedence over `dataset_name` |
| [`cognify()`](../main-operations/legacy-operations/cognify) | `datasets="finance"` or `datasets=["finance"]` | `datasets=[UUID(...)]` — same parameter, don't mix names and UUIDs in one list |
| [`recall()`](../main-operations/recall) / [`search()`](../main-operations/legacy-operations/search) | `datasets=["finance"]` | `dataset_ids=[UUID(...)]` |
| [`improve()`](../main-operations/improve) | `dataset="finance"` | `dataset=UUID(...)` — same parameter |
| [`forget()`](../main-operations/forget) | `dataset="finance"` | `dataset_id=UUID(...)` |
| [`cognee.datasets.*`](/python-api/datasets) | — | `dataset_id` only; these helpers require UUIDs |
To go from a name to an id, list your datasets:
```python theme={null}
datasets = await cognee.datasets.list_datasets()
dataset_id = next(ds.id for ds in datasets if ds.name == "finance")
```
## Access control
* Permissions (read, write, share, delete) are enforced at the dataset level
* Share one dataset with a team, keep another private
* Independently manage who can modify or distribute content
## Incremental processing
* Processing status is tracked per dataset
* After you remember more data, the underlying cognify step focuses on new or changed items
* Skips what’s already completed for that dataset
## Datasets vs NodeSets
**Datasets** scope storage, permissions, and pipeline execution; **[NodeSets](../further-concepts/node-sets)** are semantic tags within a dataset.
* During `remember()`, you can label items with one or more NodeSet names (e.g., "AI", "FinTech")
* The underlying graph-building step propagates those labels into the graph by creating `NodeSet` nodes and linking derived chunks and entities via `belongs_to_set` relationships
* This lets you slice a single dataset’s graph by topic or team without creating new datasets, while dataset-level permissions still control overall access
<Columns>
<Card title="Remember" icon="plus" href="/core-concepts/main-operations/remember">
Direct content into a dataset
</Card>
<Card title="Improve" icon="brain-cog" href="/core-concepts/main-operations/improve">
Enrich memory within a dataset
</Card>
<Card title="Recall" icon="search" href="/core-concepts/main-operations/recall">
Scope queries by dataset
</Card>
<Card title="Forget" icon="trash" href="/core-concepts/main-operations/forget">
Remove datasets and data
</Card>
</Columns>
# Global Context Index
Source: https://docs.cognee.ai/core-concepts/further-concepts/global-context-index
Build dataset-level summaries to improve graph completion retrieval.
The **global context index** is an optional summary layer that helps Cognee answer questions that depend on the broader shape of a dataset, not only the closest graph facts.
Normal graph retrieval is local: Cognee searches for graph edges, chunks, summaries, and entities that match the query. That works well when the answer is near a few specific facts. The global context index adds a higher-level map: semantic buckets of `TextSummary` nodes and a root summary of the dataset. A **bucket** is a generated summary that groups a handful of `TextSummary` nodes — or, at higher levels, other buckets — into one combined summary. The **root** is the single summary at the very top of that structure, covering the whole dataset.
## Why use it
Use the global context index when answers often depend on document-wide or dataset-wide context:
* long documents where important details are spread across chapters or sections
* evolving conversations where the final state depends on earlier updates
* project memory where the answer needs the overall plan, risks, and current status
* policy or research corpora where local facts need broader framing
It is most useful when you want retrieval to include both:
* **local evidence** from the graph
* **global orientation** from compact dataset summaries
## How it works
During normal ingestion and enrichment, Cognee creates `DocumentChunk` and `TextSummary` datapoints. The global context index adds the `GlobalContextSummary` layers shown inside the dashed lines:
```text theme={null}
--------------------------------------------------
root GlobalContextSummary
-> optional higher-level GlobalContextSummary bucket
-> GlobalContextSummary bucket
--------------------------------------------------
-> TextSummary
-> DocumentChunk
```
The build process is bottom-up, starting from `TextSummary` nodes. The retrieval hierarchy is top-down, starting from the root summary.
<Note>
The index groups `TextSummary` nodes, not raw `DocumentChunk` nodes directly.
</Note>
## What retrieval adds
Two search types read the index: [`GRAPH_COMPLETION`](/python-api/search-type) and [`HYBRID_COMPLETION`](/python-api/search-type). Other search types ignore `include_global_context_index`, including the graph completion variants such as `GRAPH_COMPLETION_COT` and `GRAPH_COMPLETION_DECOMPOSITION`.
When enabled, Cognee prepends a global context prelude before the usual retrieved context:
```text theme={null}
World summary:
...
Relevant areas:
...
<normal graph context follows>
```
The **World summary** comes from the root `GlobalContextSummary`. The **Relevant areas** are the top matching non-root `GlobalContextSummary` bucket texts for the query.
This gives the model a compact map before it reads the local graph facts. `HYBRID_COMPLETION` places the same prelude under a `## Global context` heading at the top of its context block.
## Build the index
The index is opt-in. Build it after memory has been created:
```python theme={null}
await cognee.improve(
dataset="product_docs",
build_global_context_index=True,
)
```
`improve()` first runs the normal enrichment pass, then builds the global context index. It groups up to 4 children per bucket and uses entity-overlap (graph) bucketing for the bottom level, adding levels until the topmost level fits under the root.
Later `improve()` runs update the index incrementally: only newly added `TextSummary` nodes — ones not yet assigned to a bucket — are placed into existing buckets, and the root is regenerated whenever new placements occur below it.
<Warning>
`build_global_context_index=True` is skipped when `run_in_background=True`, because ordered background pipeline chaining is not currently supported for this step.
</Warning>
<Note>
If the build itself fails, `improve()` logs a warning and still returns successfully — the index is simply absent, and search falls back to normal graph context.
</Note>
## Use it during search
Enable it through `retriever_specific_config` on graph completion search:
```python theme={null}
from cognee import SearchType
results = await cognee.recall(
query_text="What is the current state of the rollout plan?",
query_type=SearchType.GRAPH_COMPLETION,
datasets=["product_docs"],
retriever_specific_config={
"include_global_context_index": True,
"global_context_index_top_k": 3,
},
)
```
To inspect exactly what will be sent as context, use `only_context=True`:
```python theme={null}
context = await cognee.recall(
query_text="What changed after the second meeting?",
query_type=SearchType.GRAPH_COMPLETION,
datasets=["product_docs"],
only_context=True,
retriever_specific_config={
"include_global_context_index": True,
"global_context_index_top_k": 3,
},
)
```
## Configuration
| Option | Default | Where it is used | What it does |
| ------------------------------ | ------: | --------------------------- | ------------------------------------------------------------------------------------ |
| `build_global_context_index` | `False` | `cognee.improve()` | Builds the bucket and root summaries after enrichment. |
| `include_global_context_index` | `False` | `retriever_specific_config` | Prepends global context during `GRAPH_COMPLETION` and `HYBRID_COMPLETION` retrieval. |
| `global_context_index_top_k` | `3` | `retriever_specific_config` | Number of non-root bucket summaries to include as relevant areas. |
## Benefits and tradeoffs
The main benefit is better long-range coherence. The model can see a compact summary of the dataset before it reasons over the retrieved graph context. This can reduce failures where local retrieval finds a relevant fragment but misses the broader story.
The tradeoff is that the index is lossy. A bucket summary is an orientation aid, not a replacement for source chunks or graph facts. The most reliable answers still come from combining global context with precise retrieved evidence.
Building the index also adds work. Cognee makes one structured LLM call per bucket it creates or updates, at every level, plus one more for the root summary, and embeds each `GlobalContextSummary` into its own vector collection. The first build is the expensive one; because later runs only place newly added `TextSummary` nodes, incremental updates cost far less.
At search time the cost is small and fixed: one lookup for the root summary and one vector search for the top `global_context_index_top_k` buckets, plus the extra prompt tokens the prelude adds.
## When not to use it
You may not need the global context index when:
* your dataset is small enough that normal retrieval already has enough context
* queries are mostly simple fact lookups
* you need the fastest possible enrichment pass
* you want retrieval context to contain only direct local graph evidence
For small datasets, start without it. Add it when you see questions that need broader orientation or multi-part memory.
<Columns>
<Card title="Building the Global Context Index" icon="globe" href="/guides/global-context-index">
A runnable guide: build the index, inspect its root/bucket structure, and see it update incrementally as new data arrives.
</Card>
<Card title="Reading the Global Context Index" icon="search" href="/guides/global-context-index-recall">
A runnable guide: see exactly what include\_global\_context\_index adds to GRAPH\_COMPLETION retrieval, and what stays the same.
</Card>
</Columns>
# Loaders
Source: https://docs.cognee.ai/core-concepts/further-concepts/loaders
Learn how Cognee handles different file formats.
Loaders are responsible for reading files from your disk or cloud storage and converting them into plain text that Cognee can process. When you run `remember()`, Cognee automatically selects the most appropriate loader for each file based on its extension and content type.
## Loader Selection
Cognee uses a priority system to decide which loader to use. It tries to match a loader in the following order:
1. **CodeLoader**: For source-code files (`.py`, `.ts`, `.go`, `.rs`, etc.). It sits ahead of `TextLoader` because code content-sniffs as plain text, so `TextLoader` would otherwise claim it first.
2. **TextLoader**: For plain text files (`.txt`, `.md`, `.json`, `.xml`, etc.).
3. **PyPdfLoader**: For PDF files (requires `pypdf`).
4. **ImageLoader**: For images (uses vision models to transcribe content, with an optional local OCR pass).
5. **AudioLoader**: For audio files (uses transcription models).
6. **VideoLoader**: For video files (transcribes the audio track with inline `[HH:MM:SS]` timestamps).
7. **DltCsvLoader**: For CSV files, when `cognee[dlt]` is installed (routes rows through dlt structured ingestion instead of text).
8. **CsvLoader**: For CSV files (converts rows to text).
9. **UnstructuredLoader**: For complex formats like `.docx`, `.pptx`, `.epub` (requires `unstructured`).
10. **AdvancedPdfLoader**: For layout-aware PDF extraction (requires `unstructured`).
11. **DoclingLoader**: If no other loader can ingest the file type, [Docling](https://github.com/docling-project/docling) is used for conversion (if the type is supported).
If you want to force a specific loader or provide custom configuration, you can use the `preferred_loaders` parameter in `remember()`.
## Available Loaders
### Core Loaders
These are available by default in every Cognee installation:
* **CodeLoader**: Claims source-code files by file-name extension and stores them verbatim under their original extension. Files it claims take the deterministic code graph pipeline during `cognify()` instead of LLM-based extraction. See the "Code Loader" accordion below.
* **TextLoader**: Reads text files with UTF-8 encoding.
* **CsvLoader**: Reads CSV files and converts each row into a structured text format (`Row N: key: value`).
* **ImageLoader**: Uses an LLM vision model to transcribe the image content. By default the transcription uses an extraction-oriented prompt that asks for entities, relationships, verbatim text, and structured content (tables, charts, diagrams) rather than a short caption. An optional local OCR pass can be enabled to append recognized text to the transcription; it requires `pip install cognee[rapidocr]`. See the "Image transcription and OCR" accordion below.
* **AudioLoader**: Uses an audio transcription API to transcribe audio files. This depends on your configured LLM provider supporting transcription endpoints; see [LLM Providers](/setup-configuration/llm-providers) for provider-specific caveats.
* **VideoLoader**: Transcribes a video's audio track into text with inline `[HH:MM:SS]` segment timestamps, then feeds it through the normal text pipeline. `.mp4` and `.webm` do not require `ffmpeg`; other containers do (see the VideoLoader usage accordion below).
### External Loaders
These require additional dependencies to be installed:
* **PyPdfLoader**: Extracts text from PDFs page by page using the `pypdf` library, preserving page boundaries with `Page N:` markers in the extracted text. Requires `pip install cognee[docs]` (or `pip install pypdf`).
* **AdvancedPdfLoader**: Layout-aware PDF extraction using the `unstructured` library. Extracts text, tables (as HTML), and image placeholders per page. Falls back to `PyPdfLoader` automatically if extraction fails. Requires `pip install cognee[docs]`, plus `poppler` and `tesseract` installed on the system.
* **UnstructuredLoader**: Handles many office and document formats (`.docx`, `.xlsx`, `.pptx`, `.odt`, `.rtf`, `.eml`, `.epub`, `.html`, and more) via `unstructured`'s auto-partition. Requires `pip install cognee[docs]`.
* **BeautifulSoupLoader**: Extracts text from HTML files using `BeautifulSoup`. Applies CSS selector rules to pull structured content from specific tags. Requires `pip install cognee[scraping]`.
* **DoclingLoader**: Catch-all fallback that converts a wide range of document formats (PDF, DOCX, XLSX, PPTX, HTML, Markdown, and more) to plain text via [Docling](https://github.com/docling-project/docling). Supported extensions are discovered dynamically from Docling's `FormatToExtensions` map. Requires `pip install cognee[docling]`.
* **DltCsvLoader**: Ingests `.csv` files through the [dlt structured path](/integrations/dlt-integration) instead of flattening them to text — the rows are loaded into a dlt staging database and the loader emits one manifest per CSV file, which cognify turns into row nodes deterministically from the schema, with no LLM extraction. It registers **above** `CsvLoader` in the priority order, so once the extra is installed every CSV takes this route by default. Requires `pip install 'cognee[dlt]'`.
## Supported File Extensions
Cognee selects loaders based on file type. The table below shows the supported extensions and their default loaders.
Extensions are matched case-insensitively for the text-only formats that carry no content signature — `.txt`, `.csv`, `.md`, `.json`, `.xml`, `.yaml`, and `.yml` — so `REPORT.CSV` and `report.csv` are detected as the same type and get the same loader. `.log` files are case-insensitive too: an unrecognized extension like `.LOG` falls back to plain text, which lands on the same TextLoader. Formats identified from their content, such as PDFs, images, audio, and video, were never case-sensitive.
<Accordion title="Supported extensions reference">
| Extension(s) | Default loader | Notes |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `.txt`, `.md`, `.json`, `.xml`, `.yaml`, `.yml`, `.log` | TextLoader | Plain text and structured text-like files |
| `.c`, `.cc`, `.cpp`, `.cs`, `.cxx`, `.dart`, `.fs`, `.go`, `.h`, `.hcl`, `.hh`, `.hpp`, `.java`, `.js`, `.jsx`, `.kt`, `.kts`, `.php`, `.proto`, `.py`, `.rake`, `.rb`, `.rs`, `.scala`, `.svelte`, `.swift`, `.tf`, `.ts`, `.tsx`, `.vb`, `.vue` | CodeLoader | Source-code files; matched by extension only and processed by the code graph pipeline rather than LLM extraction |
| `.csv` | DltCsvLoader (with `cognee[dlt]`), otherwise CsvLoader | Tabular data. With the `dlt` extra installed, `DltCsvLoader` takes precedence and routes rows through dlt structured ingestion; without it, `CsvLoader` converts rows to text. Force text flattening per call with `preferred_loaders=[{"csv_loader": {}}]` |
| `.pdf` | PyPdfLoader | Default PDF extraction; `AdvancedPdfLoader` is optional for layout-aware parsing |
| `.docx`, `.doc`, `.odt` | UnstructuredLoader | Word-processor formats; requires `unstructured` |
| `.xlsx`, `.xls`, `.ods` | UnstructuredLoader | Spreadsheet formats; requires `unstructured` |
| `.pptx`, `.ppt`, `.odp` | UnstructuredLoader | Presentation formats; requires `unstructured` |
| `.rtf`, `.html`, `.htm`, `.eml`, `.msg`, `.epub` | UnstructuredLoader | Additional document and markup formats; requires `unstructured` |
| `.png`, `.jpg`, `.jpe`, `.jpeg`, `.gif`, `.webp`, `.bmp`, `.tif`, `.tiff`, `.heic`, `.avif`, `.ico`, `.psd`, `.apng`, `.cr2`, `.dwg`, `.xcf`, `.jxr`, `.jpx` | ImageLoader | Raster, design, raw, and CAD-adjacent image formats transcribed by a vision LLM; optional local OCR text is appended when `IMAGE_OCR_ENABLED="true"` |
| `.mp3`, `.wav`, `.aac`, `.flac`, `.ogg`, `.m4a`, `.mid`, `.amr`, `.aiff` | AudioLoader | Audio transcription via a Whisper-compatible model |
| `.mp4`, `.webm` | VideoLoader | Audio track transcribed with inline `[HH:MM:SS]` timestamps; no `ffmpeg` required (`ffmpeg` is used automatically when available) |
| `.mov`, `.mkv`, `.avi`, `.m4v` | VideoLoader | Audio track transcribed with inline `[HH:MM:SS]` timestamps; requires system `ffmpeg` on `PATH` to extract audio |
> **Note:** Files with extensions not in this table cannot be remembered by default. Use a [custom loader](#registering-custom-loaders) to handle additional formats.
>
> When no registered loader can handle a file, `remember()` raises a `ValueError` that names the file's extension and lists the currently supported extensions. For office and document formats that ship only via an optional loader (for example `.pptx`, `.docx`, `.xlsx`, `.html`, `.epub`), install `cognee[docling]` for Docling or `cognee[docs]` for Unstructured-backed loaders, then retry.
</Accordion>
## Usage
<AccordionGroup>
<Accordion title="Using Preferred Loaders">
You can override the default loader selection by specifying `preferred_loaders`. This is useful when you want to pass specific configuration options to a loader.
```python theme={null}
import cognee
await cognee.remember(
data=["example_website.html"],
preferred_loaders=[
{
"beautiful_soup_loader": {
"extraction_rules": {
"title": "h1",
"body": "article.main-content"
}
}
}
]
)
```
**Opting out of the dlt CSV route.** When `cognee[dlt]` is installed, `DltCsvLoader` outranks `CsvLoader` for every `.csv` file. Name `csv_loader` explicitly to restore text flattening for a single call:
```python theme={null}
await cognee.remember(
data=["employees.csv"],
preferred_loaders=[{"csv_loader": {}}]
)
```
The same channel carries per-call dlt options — `primary_key`, `write_disposition`, `max_rows_per_table`, and `column_value_columns` — to `DltCsvLoader`:
```python theme={null}
await cognee.remember(
data=["employees.csv"],
preferred_loaders=[
{"dlt_csv_loader": {"primary_key": "id", "write_disposition": "merge"}}
]
)
```
</Accordion>
<Accordion title="Code Loader">
`CodeLoader` is a core loader that claims source-code files so they take Cognee's deterministic code graph pipeline instead of the LLM extraction path.
```python theme={null}
import cognee
await cognee.remember(data=["src/service.py"])
```
**Matched by extension only.** Content sniffing cannot identify a programming language — it reports code as plain text — so `CodeLoader` looks at the file-name extension and ignores the detected MIME type entirely. It claims these 31 extensions:
`.c`, `.cc`, `.cpp`, `.cs`, `.cxx`, `.dart`, `.fs`, `.go`, `.h`, `.hcl`, `.hh`, `.hpp`, `.java`, `.js`, `.jsx`, `.kt`, `.kts`, `.php`, `.proto`, `.py`, `.rake`, `.rb`, `.rs`, `.scala`, `.svelte`, `.swift`, `.tf`, `.ts`, `.tsx`, `.vb`, `.vue`
Extensions that double as generic config or markup are deliberately **not** claimed — `.yaml` and `.yml` (Ansible), `.json` (OpenAPI), and template formats such as `.xaml`, `.razor`, `.cshtml`, and `.hbs`. Claiming those would hijack ordinary documents, so files with those extensions keep going to `TextLoader` as before. `.graphql` is also left out, for a different reason: the extractor only reads GraphQL schemas inside a larger project, not as lone files.
**Stored under the real extension.** Like `TextLoader`, `CodeLoader` stores content verbatim under a content-hash file name, but it keeps the original suffix — `code_<content_hash>.<ext>` (for example `code_a1b2c3....py`). The storage file name is the only place the extension survives, and language detection downstream depends on it.
**No LLM or embedding calls.** Claimed files are classified as code files and routed by `cognify()` down a dedicated code route whose task list runs the code graph extraction directly. Chunking, entity extraction with an LLM, and summarization do not run for them, so they also cost nothing in a `cognify(dry_run=True)` estimate — see [Dry-run cost estimation](/python-api/cognify#dry-run-cost-estimation).
**Requires the enola binary.** The code graph extraction is performed by [enola](https://github.com/enola-labs/enola), an external Go CLI, not by Python code inside Cognee. The first `cognify()` run over a code file downloads and installs a pinned enola release automatically; set `ENOLA_PATH` to use a binary you installed yourself, or `ENOLA_AUTO_INSTALL=false` to disable the auto-install, in which case a missing binary raises `EnolaNotInstalledError`. See the "Where the enola binary comes from" accordion below for the download source, cache location, and offline setup.
**Opting out.** `preferred_loaders` is tried before the default priority order, so you can send a code file back down the normal text and LLM path:
```python theme={null}
import cognee
await cognee.remember(
data=["src/service.py"],
preferred_loaders=[{"text_loader": {}}]
)
```
</Accordion>
<Accordion title="Where the enola binary comes from">
Cognee resolves the enola binary in a fixed order: `ENOLA_PATH` first, then an `enola` on your `PATH`, and only then the automatic download.
**Download source.** There is no PyPI package or private registry — enola is a compiled Go CLI, so Cognee fetches the release archive straight from the upstream GitHub releases of [enola-labs/enola](https://github.com/enola-labs/enola):
```
https://github.com/enola-labs/enola/releases/download/v0.4.12/enola-0.4.12-<platform>.tar.gz
```
The pinned version is `0.4.12`. Every archive is verified against a SHA-256 checksum pinned inside Cognee before anything is installed; a mismatch aborts the install rather than running an unexpected binary.
**Cache location.** The extracted binary is written to `~/.cognee/bin/enola-0.4.12-<platform>` (`.exe` suffix on Windows). The file name is version-scoped, so bumping the pinned version downloads a new file instead of reusing the old one. Installation is idempotent: once the file exists, later runs use it with no network access at all.
**Platform builds.** Only these pinned builds exist. On any other platform the install raises `EnolaInstallError` naming your system/machine and asking you to install enola yourself and set `ENOLA_PATH`.
| Platform key | Machine |
| --------------- | ---------------------- |
| `darwin-arm64` | macOS, Apple Silicon |
| `darwin-amd64` | macOS, Intel |
| `linux-amd64` | Linux, x86-64 |
| `linux-arm64` | Linux, arm64 / aarch64 |
| `windows-amd64` | Windows, x86-64 |
**Air-gapped and offline setups.** `ENOLA_PATH` always wins over the auto-install, so pointing it at a binary you shipped yourself means the download never runs:
```dotenv theme={null}
# .env
ENOLA_PATH="/opt/enola/enola"
ENOLA_AUTO_INSTALL="false"
```
`ENOLA_AUTO_INSTALL="false"` (default `"true"`) disables the download outright — a missing binary then raises `EnolaNotInstalledError` instead of reaching for GitHub. Cognee's global `ALLOW_HTTP_REQUESTS="false"` switch is honored too and refuses the download with an error telling you to install enola manually. If `ENOLA_PATH` is set but no file exists there, Cognee raises `EnolaNotInstalledError` rather than silently falling back to a download.
For the repository-level code graph pipeline that uses this binary, see the [Code Graph guide](/guides/code-graph).
</Accordion>
<Accordion title="Advanced PDF Loader">
`AdvancedPdfLoader` uses the `unstructured` library to perform layout-aware extraction, preserving page structure, tables, image metadata, and page numbers. It groups content by page and prepends `Page N:` markers to each page's extracted text, making source pages traceable in downstream chunks. Because `PyPdfLoader` has higher default priority, you need to request it explicitly with `preferred_loaders`. It accepts a `strategy` parameter that controls the trade-off between speed and accuracy:
If you want to inspect those page markers after ingestion, retrieve raw chunks with `SearchType.CHUNKS`. The page number is kept in the chunk `text`, not in a separate metadata field.
<Note>
Make sure `poppler` and `tesseract` are installed on your system before using `AdvancedPdfLoader`, in addition to installing the Python dependencies with `pip install cognee[docs]`.
</Note>
| Strategy | Description |
| ------------------ | ------------------------------------------------------------- |
| `"auto"` (default) | Automatically selects the best strategy based on the document |
| `"fast"` | Fast text extraction without layout analysis |
| `"hi_res"` | High-resolution extraction with full layout analysis (slower) |
| `"ocr_only"` | Uses OCR for text extraction, useful for scanned PDFs |
<AccordionGroup>
<Accordion title="Scanned vs. text PDFs (enabling OCR)">
Cognee does **not** automatically distinguish scanned (image-only) PDFs from text PDFs. The default `PyPdfLoader` reads the embedded text layer page by page and skips pages with no extractable text. For a scanned PDF — where each page is an image with no text layer — this silently produces empty or near-empty output, and no OCR is performed.
A text layer is the selectable, machine-readable text embedded in the PDF. If a page has no text layer, `PyPdfLoader` omits that page and no `Page N:` marker is added for it. If `pypdf` errors on a malformed page, Cognee logs a warning, skips that page, and continues loading the rest of the document.
To process scanned PDFs, explicitly select `AdvancedPdfLoader` with an OCR strategy. Because `PyPdfLoader` has higher default priority, you must request the OCR-capable loader through `preferred_loaders`:
```python theme={null}
import cognee
await cognee.remember(
data=["scanned_document.pdf"],
preferred_loaders=[
{"advanced_pdf_loader": {"strategy": "ocr_only"}}
]
)
```
Use `strategy="ocr_only"` for fully image-based or scanned PDFs, and `strategy="hi_res"` for documents that mix a text layer with scanned images. OCR requires `pip install cognee[docs]` plus `poppler` and `tesseract` installed on the system. For non-English or CJK scanned documents, also install the matching Tesseract language packs (see the next accordion).
</Accordion>
<Accordion title="OCR for non-English and CJK PDFs">
Standard PDF text extraction via `PyPdfLoader` or `AdvancedPdfLoader` with `strategy="fast"` often fails silently on CJK and other non-Latin documents. These PDFs commonly embed glyphs as images or use non-standard font encodings, which can lead to empty or garbled output.
Use `AdvancedPdfLoader` with an OCR-based strategy and install Tesseract language packs for your target language.
**Step 1 — install Tesseract language packs** (Ubuntu/Debian):
```bash theme={null}
# Japanese
sudo apt-get install tesseract-ocr-jpn tesseract-ocr-jpn-vert
# Chinese (Simplified / Traditional)
sudo apt-get install tesseract-ocr-chi-sim tesseract-ocr-chi-tra
# Korean
sudo apt-get install tesseract-ocr-kor
```
**Step 2 — use `ocr_only` strategy with the `languages` parameter**:
```python theme={null}
import cognee
# Japanese PDF
await cognee.remember(
data=["japanese_document.pdf"],
preferred_loaders=[
{
"advanced_pdf_loader": {
"strategy": "ocr_only",
"languages": ["jpn"]
}
}
]
)
```
The `languages` list accepts [ISO 639-2 Tesseract language codes](https://tesseract-ocr.github.io/tessdoc/Data-Files-in-different-versions.html). Common values:
| Language | Code |
| ------------------- | ----------- |
| Japanese | `"jpn"` |
| Chinese Simplified | `"chi_sim"` |
| Chinese Traditional | `"chi_tra"` |
| Korean | `"kor"` |
Use `strategy="hi_res"` for better layout accuracy when the document has mixed text and images, and `strategy="ocr_only"` for fully image-based or scanned PDFs.
<Note>
If you also need translation after OCR extraction, use the [Multilingual Ingestion](/guides/multilingual-ingestion) pipeline before building the knowledge graph.
</Note>
</Accordion>
</AccordionGroup>
</Accordion>
<Accordion title="Unstructured Loader">
`UnstructuredLoader` handles a wide range of office and document formats using `unstructured`'s auto-partition feature. It supports the same `strategy` options as `AdvancedPdfLoader`.
Supported file types:
| Category | Extensions |
| -------------- | ------------------------------------------------ |
| Word documents | `.docx`, `.doc`, `.odt` |
| Spreadsheets | `.xlsx`, `.xls`, `.ods` |
| Presentations | `.pptx`, `.ppt`, `.odp` |
| Other | `.rtf`, `.html`, `.htm`, `.eml`, `.msg`, `.epub` |
```python theme={null}
import cognee
await cognee.remember(
data=["presentation.pptx"],
preferred_loaders=[
{
"unstructured_loader": {
"strategy": "fast"
}
}
]
)
```
</Accordion>
<Accordion title="Docling Loader">
`DoclingLoader` is the lowest-priority loader and acts as a catch-all fallback for any file type that no other loader can ingest. It converts the document with [Docling](https://github.com/docling-project/docling) and exports plain text via Docling's `export_to_text()`. Supported extensions are pulled at runtime from Docling's `FormatToExtensions` (PDF, DOCX, XLSX, PPTX, HTML, Markdown, and more), so the available formats track your installed Docling version.
Install with `pip install 'cognee[docling]'`. Because it sits last in the loader priority, Cognee only reaches for it when no higher-priority loader matches — to force it on a file another loader would normally handle (for example, to use Docling's layout-aware parsing on a PDF instead of `PyPdfLoader`), pass it through `preferred_loaders`:
```python theme={null}
import cognee
await cognee.remember(
data=["report.pdf"],
preferred_loaders=[{"docling_loader": {}}]
)
```
**When to use Docling vs. `AdvancedPdfLoader`**: prefer `AdvancedPdfLoader` for PDFs when you need per-page markers, HTML-formatted tables, or OCR strategy control (see the Advanced PDF Loader accordion above). Reach for `DoclingLoader` when you want a single unified converter across many formats, or for non-PDF files Cognee does not otherwise handle.
</Accordion>
<Accordion title="BeautifulSoup Loader">
`BeautifulSoupLoader` parses HTML files using CSS selectors. By default it applies a comprehensive set of extraction rules covering common HTML content areas (headings, paragraphs, articles, tables, code blocks, etc.). You can pass your own `extraction_rules` dict to target specific elements.
Each rule is a dict with the following optional keys:
| Key | Type | Description |
| ----------- | ------ | -------------------------------------------------------------- |
| `selector` | `str` | CSS selector to match elements |
| `xpath` | `str` | XPath expression (requires `lxml`) |
| `attr` | `str` | HTML attribute to extract instead of text content |
| `all` | `bool` | If `True`, extract all matches; otherwise only the first |
| `join_with` | `str` | String used to join multiple extracted values (default: `" "`) |
```python theme={null}
import cognee
await cognee.remember(
data=["page.html"],
preferred_loaders=[
{
"beautiful_soup_loader": {
"extraction_rules": {
"title": {"selector": "h1", "all": False},
"body": {"selector": "article.main-content", "all": True, "join_with": "\n\n"},
"og_image": {"selector": "meta[property='og:image']", "attr": "content"}
}
}
}
]
)
```
**Overlapping CSS rules are deduplicated.** Rules are applied in the order they appear in `extraction_rules`, against a single parsed document. Once a rule extracts an element, elements nested inside it are skipped by later rules, so content covered by a broad selector is not emitted a second time by a narrower one. With the default rules, for example, `article` runs before `paragraphs`, so a `<p>` inside an `<article>` is extracted once as part of the article rather than repeated. Deduplication works in one direction only: a broad rule listed *after* a narrower one is not suppressed by it and re-emits the shared content anyway, so list broader selectors before narrower ones — as the default rules do — to avoid duplicates.
Deduplication only skips *descendants* of already-extracted elements. Two rules whose selectors match the exact same element still extract that element twice. Matches within a single rule are likewise not deduplicated against each other, so one selector matching both an element and its descendant emits the shared text twice.
For XPath-based extraction (requires `pip install lxml`):
```python theme={null}
await cognee.remember(
data=["page.html"],
preferred_loaders=[
{
"beautiful_soup_loader": {
"extraction_rules": {
"content": {"xpath": "//div[@class='content']//p"}
}
}
}
]
)
```
<Note>
XPath rules are evaluated against a separate `lxml` tree and do not take part in the deduplication described above. An XPath rule never suppresses a later CSS rule, and it is never suppressed by an earlier one — so mixing XPath and CSS rules that cover the same elements can still repeat content.
</Note>
</Accordion>
<Accordion title="Configuring Vision Models for ImageLoader">
`ImageLoader` uses your configured LLM to describe image content — there is no separate VLM configuration. To process images, set `LLM_MODEL` to a vision-capable model.
**Vision-capable models by provider:**
| Provider | Example model |
| -------------- | ---------------------------- |
| OpenAI | `gpt-4o`, `gpt-4o-mini` |
| Google Gemini | `gemini/gemini-2.0-flash` |
| Anthropic | `claude-3-5-sonnet-20241022` |
| Azure OpenAI | `azure/gpt-4o` |
| Ollama (local) | `llava`, `llava-llama3` |
```dotenv theme={null}
# .env — enable vision by choosing a vision-capable model
LLM_PROVIDER="openai"
LLM_MODEL="gpt-4o-mini"
LLM_API_KEY="sk-..."
```
If your `LLM_MODEL` does not support vision, remembering an image file will fail at the description step. Switch to a vision-capable model and retry.
<Info>
**Ollama users**: Pull a vision-capable model (e.g. `ollama pull llava`) and set `LLM_MODEL="llava"`. Text-only models such as `llama3.1` cannot process images. See [LLM Providers](/setup-configuration/llm-providers) for full Ollama setup.
</Info>
For the prompt, token cap, and optional OCR pass used during transcription, see the "Image transcription and OCR" accordion below.
</Accordion>
<Accordion title="Image transcription and OCR">
`ImageLoader` turns an image into text in two steps: a vision-LLM transcription, and — when enabled — a local OCR pass whose recognized text is appended to that transcription. Both are controlled by environment variables.
| Variable | Default | Description |
| ------------------------------------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `IMAGE_EXTRACTION_ENABLED` | `"true"` | Transcribe with an extraction-oriented prompt instead of a short caption. Set to `"false"` to restore the legacy `"What's in this image?"` prompt and its 300-token cap; the three `IMAGE_TRANSCRIPTION_*` settings are then ignored. |
| `IMAGE_TRANSCRIPTION_PROMPT_PATH` | `"transcribe_image_prompt.txt"` | Prompt template used for the transcription. |
| `IMAGE_TRANSCRIPTION_MAX_COMPLETION_TOKENS` | `1024` | Completion-token cap for the transcription request. |
| `IMAGE_TRANSCRIPTION_REASONING_EFFORT` | `"low"` | Reasoning effort hint: `minimal`, `low`, `medium`, or `high`. Ignored by models without reasoning support. |
| `IMAGE_OCR_ENABLED` | `"false"` | Run a local OCR pass and append its text to the transcription. Requires `pip install cognee[rapidocr]`. |
**Extraction-oriented transcription (default).** The default prompt asks the vision model for concise, factual text aimed at graph extraction: the entities shown and their attributes, the relationships between them, all visible text, numbers, dates, and labels transcribed verbatim, and structured content (tables as rows, charts as series and data points, diagrams as how the elements connect). This produces longer text than a caption, which is why the token cap defaults to `1024`.
```dotenv theme={null}
# .env — restore the previous short-caption behavior
IMAGE_EXTRACTION_ENABLED="false"
```
**Optional local OCR.** OCR is off by default. When switched on, Cognee runs [RapidOCR](https://github.com/RapidAI/RapidOCR) locally — a pip-only dependency, no system binary — and appends the recognized text to the vision transcription under an `[OCR extracted text]` heading:
```
A bar chart titled "Quarterly revenue" with four bars...
[OCR extracted text]
Quarterly revenue
Q1 1.2M
Q2 1.8M
```
Because the OCR text is part of the loaded text, it flows through chunking, extraction, and storage like any other content. This is most useful for screenshots, scanned pages, and dense charts where the vision model paraphrases labels instead of reproducing them.
```bash theme={null}
pip install cognee[rapidocr]
```
```dotenv theme={null}
# .env
IMAGE_OCR_ENABLED="true"
```
<Note>
OCR text is truncated at 8000 characters (the tail is replaced with `...`). If the OCR pass itself fails, Cognee logs an error and continues with the vision transcription alone rather than failing the ingestion.
</Note>
**Custom transcription prompt.** `IMAGE_TRANSCRIPTION_PROMPT_PATH` accepts either a file name inside Cognee's built-in prompt directory (`cognee/infrastructure/llm/prompts`) or an absolute path, so you can keep your own prompt file anywhere on disk:
```dotenv theme={null}
# .env
IMAGE_TRANSCRIPTION_PROMPT_PATH="/etc/cognee/prompts/my_image_prompt.txt"
```
<Warning>
**Empty transcriptions on reasoning models.** `IMAGE_TRANSCRIPTION_MAX_COMPLETION_TOKENS` caps reasoning tokens as well as output tokens, so on a reasoning model (including the default `openai/gpt-5-mini`) a small cap can consume the whole budget on reasoning and return empty content. Cognee logs a warning naming the file and suggesting a higher `IMAGE_TRANSCRIPTION_MAX_COMPLETION_TOKENS`, then continues with empty text for that image. If images ingest but contribute nothing to the graph, raise the cap.
</Warning>
</Accordion>
<Accordion title="Video Loader">
`VideoLoader` ingests a video by transcribing its **audio track** and inlining per-segment `[HH:MM:SS]` timestamps into the resulting text, for example:
```
[00:00:00] Welcome to the walkthrough.
[00:00:12] First we configure the environment.
```
Because the timestamps are part of the text, they survive chunking and stay searchable. From there the transcript flows through the normal `TextDocument` pipeline (chunking, entity/relationship extraction, graph + vector storage) — there is no separate video document type, so a video becomes queryable memory with no special handling downstream.
`VideoLoader` is a core loader, available in every installation. Supported extensions: `.mp4`, `.m4v`, `.mov`, `.webm`, `.mkv`, `.avi`.
```python theme={null}
import cognee
await cognee.remember(data=["walkthrough.mp4"])
```
<Note>
**`ffmpeg` requirement.** `.mp4` and `.webm` do not require any extra tooling: if `ffmpeg` is unavailable, Cognee can send those containers straight to the transcription endpoint. Other containers (`.mov`, `.mkv`, `.avi`, `.m4v`) require system `ffmpeg` on your `PATH` to extract the audio track first. When `ffmpeg` is present it is also used for `.mp4`/`.webm` to keep the upload small.
If a container needs `ffmpeg` and none is found, `remember()` raises a `RuntimeError` explaining that `ffmpeg` must be installed and on your `PATH`, or that you can supply the video as `.mp4` or `.webm` instead. There is no `cognee[video]` extra — install `ffmpeg` through your system package manager.
</Note>
<Info>
Inline `[HH:MM:SS]` markers require a transcription model that supports segmented (`verbose_json`) output, such as `whisper-1`. Providers or models without segmented output fall back to a plain transcript with no timestamp markers; the transcript is still ingested normally.
</Info>
</Accordion>
<Accordion title="Registering Custom Loaders">
If you need to handle a custom file format, you can create your own loader class and register it with Cognee.
<Note>
`supported_extensions` values must **not** include a leading dot — use bare extensions like `"custom"` and `"jpeg"`, not `".custom"` or `".jpeg"`. Loader matching compares against the file's dot-free extension, so a dotted value would never match.
</Note>
```python theme={null}
from cognee.infrastructure.loaders import use_loader
from cognee.infrastructure.loaders.LoaderInterface import LoaderInterface
class MyCustomLoader(LoaderInterface):
loader_name = "my_custom_loader"
supported_extensions = ["custom"]
supported_mime_types = ["application/x-custom"]
async def load(self, file_path, **kwargs):
# Your custom logic to read the file and return text
return "Extracted text content"
# Register the loader so Cognee can use it
use_loader("my_custom_loader", MyCustomLoader)
await cognee.remember("data/file.custom")
```
**`load()` receives ingestion context.** Beyond the loader-specific options you pass via `preferred_loaders`, Cognee forwards the current ingestion context to `load()` as keyword arguments: `dataset_name`, `dataset_id`, `user`, and `original_file_name` (the user's real file name, which can differ from `file_path` when the bytes are a localized copy of an `s3://` source or an upload). Accept `**kwargs` — as the example above does — and ignore what you don't need.
**Returning a `LoaderResult`.** `load()` normally returns the stored derived-text path as a plain `str`. A loader that also owns the record's identity and routing can instead return a `LoaderResult`:
```python theme={null}
from cognee.infrastructure.loaders.LoaderInterface import LoaderInterface, LoaderResult
class MyCustomLoader(LoaderInterface):
...
async def load(self, file_path, **kwargs):
return LoaderResult(
file_path=stored_text_path, # required: the derived-text path
data_id=my_stable_uuid, # optional: pins the record's id
system_metadata={...}, # optional: routing stamp for cognify
)
```
| Field | Type | Description |
| ----------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `file_path` | `str` | Path to the stored derived text — the same value a plain-`str` return would provide |
| `data_id` | `UUID` or `None` | Pins the ingested record to this id instead of the one ingestion mints, so repeated runs update the same record |
| `system_metadata` | `dict` or `None` | Stamped onto the record and used by cognify's per-item routing |
| `file_metadata` | `FileMetadata` or `None` | Describes the derived text the loader just wrote — content hash, size, mime type, and name — computed while the content was still in memory |
Omitting `file_metadata` (or returning a plain `str`) makes ingestion re-open the stored file to derive the hash, size, and mime type it needs for the `Data` row; over the S3 backend that read-back costs an extra HEAD plus a full GET of content the loader already had. Loaders that write their text through the `store_derived_text` helper get the field filled in for free — it stores and describes in one step.
Returning a plain string remains fully supported; only loaders that need stable identity or custom routing need `LoaderResult`. The built-in `DltCsvLoader` uses it to carry a CSV manifest's stable `data_id` and its no-LLM cognify route stamp.
</Accordion>
</AccordionGroup>
# NodeSets
Source: https://docs.cognee.ai/core-concepts/further-concepts/node-sets
Tag and group data in Cognee with NodeSets.
## What are NodeSets?
A **NodeSet** lets you group parts of your AI memory at the dataset level. You create them as a simple list of tags when adding data to Cognee:
```python theme={null}
await cognee.remember(..., node_set=["projectA", "finance"])
```
These tags travel with your data into the knowledge graph, where they become first-class nodes connected with belongs\_to\_set edges — and you can later filter retrieval to only those subsets.
## How they flow through Cognee
* **[Remember](../main-operations/remember)**:
* NodeSets are attached as simple tags to datasets or documents
* This happens when you first ingest data
* The underlying graph-building step carries them into Documents, Chunks, and the entities extracted from those chunks
* They are materialized as real `NodeSet` nodes in the graph and connected with `belongs_to_set` edges
* **[Recall](../main-operations/recall)**:
* NodeSets help define meaningful retrieval subsets
* Use `recall()` with `node_name` and `node_name_filter_operator` to scope retrieval to specific node-set subsets
* **[Improve](../main-operations/improve)**:
* The default improvement path runs memify-style enrichment
* Related enrichment flows can create node sets such as `coding_agent_rules` or `user_sessions_from_cache`
## Why they matter
* Provide a lightweight way to organize and tag your data
* Enable graph-based filtering, traversal, and reporting
* Ideal for creating project-, domain-, or user-defined subsets of your knowledge graph
## See it in action
[NodeSet Grouping](/guides/nodeset-grouping) is a short runnable guide that walks through all of this end to end: it remembers three passages under overlapping labels — `["AI", "FinTech"]`, `["AI"]`, and `["MedTech"]` — renders the resulting graph so you can trace the `belongs_to_set` edges, and shows how to scope a later `recall()` to one group with `node_name`.
<Columns>
<Card title="Remember" icon="plus" href="/core-concepts/main-operations/remember">
Where NodeSets are first attached
</Card>
<Card title="Improve" icon="brain-cog" href="/core-concepts/main-operations/improve">
How enrichment flows add more NodeSet-based structure
</Card>
<Card title="Search" icon="search" href="/core-concepts/main-operations/legacy-operations/search">
Use NodeSets as anchors in queries
</Card>
</Columns>
# Ontologies
Source: https://docs.cognee.ai/core-concepts/further-concepts/ontologies
Enrich your knowledge graph with external vocabularies.
## What is an ontology in Cognee?
An **ontology** is an optional RDF/OWL file you can provide to Cognee.
It acts as a **reference vocabulary**, making sure that entity types ("classes") and entity mentions ("individuals") extracted from your data are linked to canonical, well-defined concepts.
## How it works
* You supply an ontology when running [Cognify](../main-operations/legacy-operations/cognify) in one of two ways (see the [practical example](#additional-details-and-examples) below):
* Set the `ONTOLOGY_FILE_PATH` environment variable and call `cognee.cognify()` normally.
* Pass an ontology resolver through the `config` argument.
* Cognee parses the file with [RDFLib](https://rdflib.dev/) and loads its classes and relationships.
* Once the LLM has extracted a graph from a chunk, its entities and types are checked against the ontology before any graph nodes are built from it:
* If a match is found, the node is marked `ontology_valid=True`.
* Parent classes and object-property links from the ontology are attached as extra edges.
* By default grounding only *annotates*: entities the ontology does not recognize are kept verbatim. Set `ONTOLOGY_MODE=strict` (or `ontology_mode` inside `ontology_config`) to have grounding drop them instead — see [Grounding modes](#grounding-modes).
* If no ontology is provided, extraction still works, just without validation or enrichment.
## Grounding modes
`ONTOLOGY_MODE` controls what grounding is allowed to do with entities the ontology does **not** recognize:
| Value | Behavior |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `annotate` *(default)* | Annotation only. Matched entities are canonicalized and enriched; unmatched entities are kept verbatim with `ontology_valid = False`. Nothing is dropped. |
| `strict` | Grounding also filters. An extracted node is **dropped** unless the ontology matched *either* its `type` against a class *or* its `name` against an individual, and every edge touching a dropped node is dropped with it. |
The default is `annotate`, so a pipeline that never sets the variable behaves exactly as it did before the option existed. Set it per deployment through the environment:
```bash theme={null}
ONTOLOGY_FILE_PATH=/path/to/subset.owl
ONTOLOGY_MODE=strict
```
…or per call inside `ontology_config`, where it overrides the environment value:
```python theme={null}
config: Config = {
"ontology_config": {
"ontology_resolver": RDFLibOntologyResolver(ontology_file="subset.owl"),
"ontology_mode": "strict",
}
}
await cognee.cognify(datasets=["my_dataset"], config=config)
```
The value is trimmed and lowercased before it is read, so `Strict` and `STRICT` work. An unrecognized value never fails a run: Cognee logs a warning naming the valid values and falls back to `annotate`. Since `config` is forwarded to Cognify, the same `ontology_config` works in [`remember()`](/python-api/remember).
<Note>
Grounding — and therefore the mode — only runs when an ontology resolver exists. Passing `config` replaces environment-based resolver lookup for that call, so a `config` carrying `ontology_mode` but no `ontology_resolver` runs with no ontology at all and the mode has no effect. Pass both keys together, or configure both through the environment.
</Note>
### What strict mode checks
Strict mode is **entity grounding only** — it is not an OWL validator:
* **Either match is enough.** An entity name the ontology has never heard of still survives if its *type* matches a declared class. This is deliberate: a recognized type is evidence the entity belongs to your domain.
* **Relationship names are never checked** against the ontology's object properties, so no edge is dropped for its own sake — only for having a dropped endpoint.
* **No axioms are evaluated.** There is still no domain/range, cardinality, or disjointness reasoning, and no reasoner runs.
Collapsing is unchanged: nodes matched to the same ontology individual are still merged onto a single survivor and their edges rewired, in both modes.
### What strict mode does not prune
Strict mode prunes **only the extracted graph**. Chunks have already been stored and embedded by the time grounding runs, so `CHUNKS`, `CHUNKS_LEXICAL`, and `RAG_COMPLETION` still retrieve chunk text that mentions dropped entities. Only graph-based search types see the pruned view.
### Strict mode over an empty ontology is a hard error
A mistyped `ONTOLOGY_FILE_PATH` produces a resolver whose lookup is empty — the resolver itself only warns — and strict mode over an empty ontology would drop every extracted entity while the run still reported success. Cognee refuses instead, raising `EmptyOntologyInStrictModeError`. When both the mode and the resolver come from the environment, the check runs while the resolver is being configured, before any pipeline work; when either comes from a per-call `ontology_config`, it runs at canonicalization. Custom resolvers that expose no `lookup` dictionary are not checked.
### Reading the drop summary
A strict run that dropped anything logs an aggregate warning per chunk batch — dropped and total nodes, the retained percentage, dropped and total edges, and the number of graphs in that batch — rather than one warning per chunk. A high drop ratio is the signal that the ontology does not cover your corpus's vocabulary; strict mode expects an ontology broad enough for the text you are ingesting, so check that summary on a small run before enabling it over a large corpus.
## Why use an ontology
* **Consistency**: standardize how entities and types are represented
* **Enrichment**: bring in inherited relationships from a domain schema
* **Control**: align Cognee's graph with existing enterprise or scientific vocabularies
## Where to get ontologies
Cognee works best with **manually curated, focused ontologies** that fit your dataset. Ontology design itself is outside the scope of Cognee, so if you need to create or model an ontology from scratch, use dedicated ontology tools and references first, then bring the resulting RDF/OWL file into Cognee.
Public resources like **Wikidata** or **DBpedia** define millions of classes and entities, which makes them too big to use directly in Cognee. If you start from a public ontology, always work with a subset, not the full ontology:
* **Select only the pieces you need** (specific classes, properties, or individuals)
* **Save the subset** in a format Cognee can parse with [`rdflib`](https://rdflib.readthedocs.io/)
* **If needed, enrich the subset manually** by adding extra classes or relationships relevant to your domain
* **Keep it small and relevant** so matching stays precise and performance remains fast
<AccordionGroup>
<Accordion title="Common sources">
- **General vocabularies**: schema.org, Dublin Core Terms (DC/Terms), SKOS, PROV-O, FOAF
- **Knowledge graph backbones**: DBpedia Ontology, Wikidata (Wikibase RDF ontology)
- **Domain examples**:
* Healthcare: SNOMED CT (licensed), ICD, UMLS, MeSH, HL7/FHIR RDF
* Finance: FIBO (Financial Industry Business Ontology)
* Geo/IoT: GeoSPARQL, SOSA/SSN, GeoNames
* Units: QUDT
</Accordion>
<Accordion title="Why subsetting is essential">
Every public ontology is **too broad to ingest wholesale**. Creating a subset is what makes them usable in Cognee:
* Improves matching precision (fewer false matches when mapping LLM output)
* Keeps performance acceptable (smaller graphs → faster resolution)
* Lets you curate only the relevant parts of a domain
</Accordion>
<Accordion title="How subsetting works">
Different communities provide different ways to extract subsets (e.g., "slims" in OBO ontologies, WDumper for Wikidata, module extraction in Protégé). The details vary, but the general principle is the same:
1. Pick the terms (classes or properties) you care about
2. Extract those terms plus their immediate context (e.g. parent classes, related properties)
3. Save the result in an `rdflib`-readable RDF format
</Accordion>
</AccordionGroup>
## Supported formats
Any format [RDFLib](https://rdflib.readthedocs.io/) can parse:
* RDF/XML (`.owl`, `.rdf`)
* Turtle (`.ttl`)
* N-Triples, JSON-LD, and others
## RDF read/write surface
Beyond consuming an ontology as extraction scaffolding, Cognee can preserve external IRIs end-to-end and treat the memory graph as RDF. This is aimed at teams that maintain knowledge natively as RDF and want Cognee as a complementary agentic-memory layer over their RDF knowledge base. The export direction — serializing the memory graph to RDF or querying it with SPARQL — is covered under [Additional details and examples](#additional-details-and-examples).
<Note>
The RDF surface relies on [`rdflib`](https://rdflib.dev/), which ships with Cognee's ontology support. Everything here is **backward compatible**: `ontology_uri` defaults to `None` and existing text-extraction ingestion is unchanged — nothing is required for existing workflows.
</Note>
### URI preservation
When an extracted entity or type matches your ontology, Cognee now keeps the matched IRI on the persisted node in [`DataPoint.ontology_uri`](/core-concepts/building-blocks/datapoints#core-structure) instead of flattening it to a local label. Grounded `Entity`/`EntityType` nodes carry their stable external IRI; ungrounded nodes keep `ontology_uri = None`. The field never affects node identity.
### Ingesting RDF into datapoints
The `cognee.modules.ontology.rdf_xml.rdf_ingest` module ingests an RDF T-Box + A-Box directly into Cognee datapoints, keeping the external IRIs verbatim rather than canonicalizing entities into a local vocabulary.
```python theme={null}
from cognee.modules.ontology.rdf_xml.rdf_ingest import ingest_rdf
# Accepts a file path, list of paths, file-like object, or a parsed rdflib.Graph.
data_points = await ingest_rdf("knowledge_base.ttl")
```
* **OWL classes** become `EntityType` nodes; **individuals** typed by a known class become `Entity` nodes. `rdf:type` and `rdfs:subClassOf` are kept as `is_a` relationships.
* **Node identity is derived from the IRI** (not the label), so distinct IRIs stay distinct and **re-ingesting the same RDF is idempotent** — this is an open-world model with no fuzzy canonicalization.
* **Object-property assertions** between two ingested individuals become explicit graph edges that preserve the original RDF predicate IRI as `predicate_uri`, so they round-trip back out on export.
* Parsing reuses the ontology resolver, so any RDF syntax RDFLib understands (RDF/XML, Turtle, N-Triples, JSON-LD, …) is supported.
Lower-level helpers are available when you need them: `load_rdf_graph(source)` parses a source into an `rdflib.Graph`, and `build_datapoints_from_rdf(graph)` / `build_graph_from_rdf(graph)` turn a parsed graph into datapoints (and custom edges) without persisting.
<Note>
**Scope / limits.** RDF ingestion preserves object-property assertions only when both endpoints are ingested individuals. Blank nodes, arbitrary literal/data-property round-trip, and RDF reasoning/entailment (e.g. OWL RL) are out of scope — no inference is applied to the ingested graph.
</Note>
## Additional details and examples
<AccordionGroup>
<Accordion title="Practical example">
`cognee.cognify()` has **no `ontology_file_path` parameter** — passing one raises `Unrecognized request argument supplied: ontology_file_path`. Supply the ontology through the `config` argument or the `ONTOLOGY_FILE_PATH` environment variable instead.
<Tabs>
<Tab title="Python API">
```python theme={null}
import cognee
from cognee.modules.ontology.rdf_xml.RDFLibOntologyResolver import RDFLibOntologyResolver
from cognee.modules.ontology.ontology_config import Config
config: Config = {
"ontology_config": {
"ontology_resolver": RDFLibOntologyResolver(ontology_file="subset.owl"), # your curated subset here
# Optional: "strict" drops entities the ontology does not ground.
# Omit to inherit ONTOLOGY_MODE (default "annotate").
"ontology_mode": "annotate",
}
}
await cognee.cognify(datasets=["my_dataset"], config=config)
```
</Tab>
<Tab title="Environment variable">
```bash theme={null}
ONTOLOGY_FILE_PATH=/path/to/subset.owl
# Optional: annotate (default) or strict — see "Grounding modes" above.
ONTOLOGY_MODE=annotate
```
```python theme={null}
import cognee
# With ONTOLOGY_FILE_PATH set, cognify picks up the ontology automatically.
await cognee.cognify(datasets=["my_dataset"])
```
</Tab>
</Tabs>
</Accordion>
<Accordion title="Default behavior without an ontology">
Cognee does **not** ship with or apply a built-in default ontology. When you don't set `ONTOLOGY_FILE_PATH` (or pass an ontology resolver via `config`), no ontology resolver is constructed at all: the grounding step is skipped entirely and extracted graphs go straight through the ontology-free construction path. There is no reference vocabulary to validate or enrich against, and no per-entity ontology lookups are performed.
In that mode, node labels and relationship names are produced **directly by the LLM** during graph extraction. There is no fixed, predefined list of relationship types — the model infers them from your text. Consistency is *guided* by the extraction prompt rather than *enforced*, so the same concept may occasionally surface under slightly different labels across chunks. The prompt asks the model to:
* Use **basic node types** (e.g. label a person as `Person`, not `Mathematician` or `Scientist` — those become properties).
* Use **snake\_case relationship names** (e.g. `acted_in`).
* Apply **coreference resolution** so an entity referred to by different names or pronouns maps to a single, consistent node.
Typical relationship names the model produces are plain, real-world verbs and roles derived from the text, for example:
* People: `married_to`, `parent_of`, `friend_of`, `works_at`, `colleague_of`
* Ownership and roles: `owns`, `owned_by`, `member_of`, `founder_of`, `employed_by`
* Things and places: `produces`, `located_in`, `part_of`, `created_by`, `acted_in`
Provide an ontology when you need these labels to be **standardized and validated** against a fixed vocabulary instead of inferred per document.
</Accordion>
<Accordion title="Using multiple ontology files">
Cognee can load several OWL files and merge them into one in-memory graph, which is useful when you split a large ontology into focused modules.
**Environment variable — comma-separated paths:**
```bash theme={null}
ONTOLOGY_FILE_PATH=/path/to/domain.owl,/path/to/entities.owl
```
**Python API — list of paths:**
```python theme={null}
from cognee.modules.ontology.rdf_xml.RDFLibOntologyResolver import RDFLibOntologyResolver
from cognee.modules.ontology.ontology_config import Config
resolver = RDFLibOntologyResolver(
ontology_file=["/path/to/domain.owl", "/path/to/entities.owl"]
)
config: Config = {"ontology_config": {"ontology_resolver": resolver}}
await cognee.cognify(config=config)
```
Files that cannot be found or parsed are skipped with a warning; at least one valid file is required for ontology grounding to take effect.
**How the merge works.**
`RDFLibOntologyResolver` parses each file with RDFLib and adds all triples to the same in-memory `rdflib.Graph`. The result is a single unified graph — Cognee does not create disjoint subgraphs, even when the ontologies share no classes or properties. There is no explicit conflict-resolution step: RDFLib performs an additive merge of triples, and any classes or properties that happen to share IRIs across files naturally coexist in the same graph.
**One ontology per `cognify()` run.**
The ontology resolver is configured at the `cognify()` call level (via `ONTOLOGY_FILE_PATH` or the `ontology_config` in the `Config` payload), not per dataset or per document. If you need different vocabularies for different data, run `cognify()` separately for each dataset with its own resolver instance.
</Accordion>
<Accordion title="Creating or editing ontologies">
Cognee does not provide ontology-authoring features. If you need to create, edit, or validate an ontology, use dedicated RDF/OWL tooling such as Protégé or your team's existing ontology workflow, then load the resulting file into Cognee.
When preparing a file for Cognee:
* Keep the ontology focused on the classes, properties, and individuals relevant to your dataset
* Prefer a curated subset over a large general-purpose ontology
* Save it in a format RDFLib can parse, such as RDF/XML (`.owl`, `.rdf`) or Turtle (`.ttl`)
</Accordion>
<Accordion title="How does an ontology relate to a custom graph?">
An ontology **extends** the graph Cognee builds — it never replaces it. The LLM extracts a graph from each chunk first, and grounding then runs as a **canonicalize-first pre-pass**: it rewrites the extracted graph *before* any graph nodes are constructed from it, rather than validating nodes one by one as they are built. For every extracted entity and entity type, Cognee looks the name up in the ontology. Then:
* **On a match**, the node is *canonicalized* in place: its `name` and `type` are rewritten to the ontology term, so the id derived from that name makes different surface forms collapse into one node. When several nodes in the same extracted graph resolve to the same ontology individual, only one of them survives and the edges of the collapsed nodes are rewired onto the survivor. The surviving node gets `ontology_valid = True` and the matched IRI in `ontology_uri`.
* Cognee then walks the matched term's neighbourhood in the OWL file and **adds** those classes and individuals as extra nodes, along with their `is_a` (`rdf:type` / `rdfs:subClassOf`) and object-property edges. An ontology edge is attached only when **both** of its endpoints are part of the matched subgraph; an edge pointing at a term outside it is skipped rather than inventing a node for that endpoint. This is how parent classes and related individuals show up in your graph even when the text never mentioned them.
* **On no match**, the node is kept exactly as the LLM produced it, with `ontology_valid = False`. In the default `annotate` mode nothing is rejected or discarded; under [`ONTOLOGY_MODE=strict`](#grounding-modes) a node with neither a class match on its type nor an individual match on its name is dropped instead, together with its edges.
So an OWL file layers a curated skeleton on top of the extracted graph. In the default `annotate` mode it cannot remove nodes — [strict mode](#grounding-modes) is the opt-in that lets it — and attaching one later does not retro-fit data that has already been processed — re-run [Cognify](../main-operations/legacy-operations/cognify) for that.
The other way to shape the graph is a [custom graph model](/guides/custom-graph-model). Neither of them is where entities come from: the entities are always read out of **your data** by the LLM. A `graph_model` constrains which shapes that extraction may return, and an ontology renames and extends what it did return — so an ontology is *not* "the graph model", and a graph model is *not* an ontology. Both shape the graph, but they act at different stages and are not alternatives to one another:
| | [Custom graph model](/guides/custom-graph-model) | Ontology |
| --------------------- | ---------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| **What it is** | A Pydantic `DataPoint` schema passed as `graph_model` | An RDF/OWL file passed via `ontology_config` or `ONTOLOGY_FILE_PATH` |
| **When it acts** | *Before* extraction — it **is** the LLM's structured-output schema | *After* extraction — extracted names are matched against the vocabulary |
| **What it controls** | Which node types, fields, and relationships the LLM may produce at all | Which of the produced entities and types get canonical names, IRIs, and inherited edges |
| **How strict** | Hard: the LLM cannot return anything outside the schema | Soft by default (`annotate`): unmatched entities are kept as-is. [`ONTOLOGY_MODE=strict`](#grounding-modes) opts into dropping them |
| **Reach for it when** | You know the exact *shape* you want (invoices, tickets, people → activities) | You already own a domain vocabulary and want *naming and typing* aligned to it |
**Combining the two.**
Grounding runs only inside the default `KnowledgeGraph` extraction path. If you pass a `graph_model` that is not a `KnowledgeGraph` subclass, Cognee stores your model's output directly and skips grounding, so **the two do not stack within a single `cognify()` / `remember()` call**. A [`custom_prompt`](/guides/custom-prompts) is independent of this: it replaces the extraction system prompt and works with either mode. Practical ways to get both worlds:
* **Ontology + [custom prompt](/guides/custom-prompts).** Keep the default schema and ontology grounding, and use `custom_prompt` to steer the LLM toward your ontology's class names so more entities match. This is the closest thing to "both" in one run.
* **Two passes over the same data.** Run one `cognify()` with your custom `graph_model` for the strictly-shaped part of the graph, and another with the default schema plus an ontology for the grounded part. Both write into the same graph.
* **Ingest the vocabulary directly.** If your ontology already contains the individuals you care about, [`ingest_rdf`](#ingesting-rdf-into-datapoints) loads its classes and individuals as datapoints with IRIs preserved, alongside anything extracted from text.
</Accordion>
<Accordion title="Which OWL constructs grounding actually reads">
Grounding uses a deliberately small slice of OWL. RDFLib parses the whole file, but `RDFLibOntologyResolver` only ever queries these triples:
| Construct | How it is used |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `rdf:type owl:Class` | Builds the **classes** lookup — matched against an extracted node's `type`, materialized as an `EntityType` |
| `rdf:type <SomeClass>` | Builds the **individuals** lookup (only for classes already in the lookup) — matched against an extracted node's `name`, materialized as an `Entity`; also re-emitted as an `is_a` edge |
| `rdfs:subClassOf` | Followed upward from the matched term and emitted as `is_a` edges, so parent classes come along |
| `rdf:type owl:ObjectProperty` | Any assertion whose predicate is a declared object property becomes an edge named after the predicate's local name — but only between two terms inside the matched subgraph |
Everything else is ignored:
* **`owl:DatatypeProperty` is never queried.** Literal-valued attributes in the OWL file are not copied onto graph nodes; an attached ontology node is created with only its `name`, a `description` mirroring it, `ontology_valid`, and `ontology_uri` — no attribute from the OWL file is copied over. Node properties come from extraction or from your [custom graph model](/guides/custom-graph-model)'s fields, never from the ontology.
* **No axioms are enforced.** `rdfs:domain` / `rdfs:range`, `owl:equivalentClass`, `owl:sameAs`, `owl:Restriction`, and cardinality all land in the in-memory graph but are never consulted — no reasoner runs.
* **`rdfs:label` is not used for matching.** Grounding keys on the **IRI local name** (the fragment after `#`, or the last path segment), lowercased with spaces replaced by underscores. A term with a readable label but an opaque IRI (e.g. `…#C0004096`) will never match text, so give your terms readable IRIs. ([`ingest_rdf`](#ingesting-rdf-into-datapoints) does read `rdfs:label`; grounding does not.)
A practical consequence of the lookup rules: a class must be declared `rdf:type owl:Class` to be findable at all. A class that only ever appears as the object of an `rdfs:subClassOf`, or an individual declared only as `owl:NamedIndividual` without a type from a declared class, never enters the lookup. The resolver still traverses such terms when walking a matched term's neighbourhood, but they are dropped when the subgraph is materialized: only declared classes and their typed individuals become nodes, and any edge touching an undeclared term is discarded with it. So declare every term you want in the graph as an `owl:Class`, or type it with one.
</Accordion>
<Accordion title="What ontology_valid actually means">
`ontology_valid` is a boolean marker on every [`DataPoint`](/core-concepts/building-blocks/datapoints#core-structure), defaulting to `False`. It records **whether grounding found a match** — nothing more:
* **It is not a gate in the default `annotate` mode.** Nodes with `ontology_valid = False` are stored, embedded, and returned by search exactly like grounded ones. [`ONTOLOGY_MODE=strict`](#grounding-modes) is the opt-in that does reject them: entities with neither a class match on their type nor an individual match on their name are dropped before any node is built. Note that strict mode does not make the flag redundant — an entity retained only because its *type* matched still carries `ontology_valid = False`. In either mode no node is auto-created in your `.owl` file — Cognee never writes back to the ontology.
* **It is not schema validation.** Matching is purely name-based: names are lowercased with spaces replaced by underscores, then compared using `difflib` fuzzy matching at a `0.8` similarity cutoff (`FuzzyMatchingStrategy` — the only strategy Cognee ships, `MATCHING_STRATEGY=fuzzy`). Only `owl:Class` terms (as *classes*) and individuals typed by one of those classes (as *individuals*) are candidates. OWL `rdfs:domain` / `rdfs:range` axioms are **not** enforced, and no reasoning or entailment is applied.
* **It is a node-level flag.** Only `Entity` and `EntityType` nodes carry `ontology_valid`. The edges Cognify writes do not have the property at all, whether they were copied in from the ontology subgraph or extracted by the LLM: relationship names are never matched against the ontology's object properties, so there is no edge-level grounding verdict to record. Filter on the endpoints, not the edge.
* **What it is good for.** It is provenance you can filter on: [`visualize_graph`](/guides/graph-visualization) colors grounded nodes differently, and you can select on the property in your own graph queries.
</Accordion>
<Accordion title="Exporting the memory graph to RDF">
The `cognee.modules.graph.rdf` module builds an RDF view over the live memory graph so you can serialize it or query it with SPARQL, decoupled from the underlying graph engine's query language.
```python theme={null}
from cognee.modules.graph.rdf import (
serialize_memory_graph,
query_memory_graph_sparql,
export_memory_graph_to_rdf, # returns an rdflib.Graph
graph_data_to_rdf, # pure builder over (nodes, edges) tuples
)
# Serialize the whole memory graph to RDF (Turtle by default).
turtle = await serialize_memory_graph(rdf_format="turtle")
# Or run SPARQL directly over an RDF view of the graph.
rows = await query_memory_graph_sparql(
"SELECT ?s WHERE { ?s a <http://example.org/mm#CNCMachine> }"
)
```
How nodes and edges map to RDF:
* **Grounded nodes** are emitted under their preserved `ontology_uri`. **Ungrounded nodes** get a minted IRI under `DEFAULT_BASE_IRI` (`https://cognee.ai/graph/…`) so the RDF stays well-formed and nothing is dropped.
* A node's `name` becomes an `rdfs:label`.
* The `is_a` relationship resolves to `rdf:type` for an individual→class link and to `rdfs:subClassOf` for a class→class link. Other relationships become predicate IRIs — a minted `…/prop/<name>` IRI, or, for RDF-ingested edges that carry a `predicate_uri`, that original RDF predicate IRI.
<Note>
`export_memory_graph_to_rdf` / `serialize_memory_graph` / `query_memory_graph_sparql` materialize the whole graph into an in-memory `rdflib` store on each call. This is convenient for querying and export but costs memory proportional to graph size — keep that in mind for very large graphs.
</Note>
</Accordion>
</AccordionGroup>
For more detailed examples of working with ontologies in Cognee, check out the demo scripts in the repository:
* [Basic ontology demo](https://github.com/topoteretes/cognee/tree/main/examples/guides) - Shows fundamental ontology integration
* [Advanced ontology demo](https://github.com/topoteretes/cognee/tree/dev/examples/advanced_guides/ontology_reference_vocabulary) - Demonstrates more complex ontology workflows
# User Preferences
Source: https://docs.cognee.ai/core-concepts/further-concepts/user-preferences
How per-user preference personalization stores ratings and nudges retrieval ranking.
**Per-user preference personalization** lets one user's ratings shape that user's own retrieval
ranking, without changing what anyone else sees. Cognee distills the ratings a user gives in
sessions into a compact per-user preference record, and applies it as a ranking nudge on that
user's later recalls.
Personalization is off by default. Turn it on with:
```dotenv theme={null}
PERSONALIZATION_ENABLED="true"
```
The tunable knobs — `PERSONALIZATION_INFLUENCE`, `PREFERENCE_ALPHA`, `PREFERENCE_BETA` — are
documented in [Setup Configuration](/setup-configuration/overview). For a runnable walkthrough of
the rate → improve → recall loop, see the
[Feedback System guide](/guides/feedback-system#personalize-ranking-per-user). This page explains
the mechanism.
## Where the ratings come from
A rating can reach personalization two ways:
* **Explicitly**, via `cognee.session.add_feedback(..., feedback_score=1..5)`.
* **Inferred**, when `AUTO_FEEDBACK` is on. The per-turn analysis can read a 1–5 rating of the
*previous* answer out of what you say next ("that was exactly right", "no, that's wrong
again"). The rating question is only added to the analysis when `PERSONALIZATION_ENABLED` is
on, and a value outside 1–5 degrades to "no signal" instead of raising. With the flag off, a
turn that carries only a rating writes nothing.
An explicit `feedback_score` wins over an inferred rating for the same Q\&A entry. A rating of
`3` is neutral and is treated as a no-op.
The same ratings also feed the **global** [feedback weights](/guides/feedback-system), which move
ranking for every user. That mechanism is independent of personalization — you can run either,
both, or neither.
## How preferences are stored
Personalization keeps one internal preference node per **(user, dataset)** pair, grouped under
the `user_preferences` node set. From it, weighted `prefers` edges point at the graph nodes that
were actually used to build the answers that user rated (only nodes — edges are never `prefers`
targets).
* A weight of `0.5` is neutral, meaning "no signal".
* Each rating moves a weight toward its target by `PREFERENCE_ALPHA` (default `0.3`).
* Untouched weights **decay** back toward neutral by `PREFERENCE_BETA` (default `0.02`) per
conversation turn. Decay is computed on read from a turn counter, so nothing is rewritten in
the background and there are no timestamps involved. A weight that has decayed to within
`0.01` of neutral is pruned.
* Stated preferences ("always answer in bullet points") are also folded into the preference
node's text, newest first, capped at 2000 characters — the oldest lines fall off first, never
mid-line. That text is injected as guidance into graph, hybrid, and RAG completion prompts.
## How preferences are updated
The preference update runs as a stage of [`improve()`](/core-concepts/main-operations/improve)
when you pass `session_ids` — the same call that applies global feedback weights. It runs once
for all the sessions you pass, because preferences aggregate across a user's sessions. It is
safe to re-run: each turn is counted once and each rating spent once. Like its neighbouring
stages it is best-effort — a failure is logged and never blocks the rest of `improve()` — and it
is a no-op that writes nothing at all when `PERSONALIZATION_ENABLED` is off.
## What changes at retrieval time
When weights exist for the current user and dataset, they act as a multiplicative nudge on
ranking, capped by `PERSONALIZATION_INFLUENCE` (default `0.3`, i.e. at most 30%):
* **Graph completion** — the personal weight multiplies into triplet scoring.
* **Hybrid** — it multiplies alongside the existing importance and truth factors.
* **RAG completion (`RAG_COMPLETION`)** — when the loaded weights actually match rows in the
chunk collection, the candidate fetch widens to the retriever's existing `wide_search_top_k`
(default `100`), the results are re-sorted by personalized distance, and the list is trimmed
back to `top_k`. So personalization can change *which* chunks make the cut, not only their
order. Weights that match nothing leave the fetch at `top_k`.
At a neutral weight, or with `PERSONALIZATION_INFLUENCE=0`, the ranking factor is exactly `1.0`,
so an empty or non-matching weight map leaves every path arithmetically identical to an
un-personalized run.
Two conditions are needed for personalization to apply at all: a user must be in context, and
exactly one dataset must resolve. A search that spans several datasets never personalizes.
Reads are fail-open everywhere — a missing, empty, or broken preference node costs you the
personalization, never the search.
<Note>
Preference nodes are internal and are never surfaced in retrieval output. They are filtered at
the graph read chokepoints: graph projection for search, triplet embedding, contradiction
detection, natural-language search, and the provenance and schema-inventory views (which also
drop every edge touching an internal node). The one exception is `SearchType.CYPHER`, which runs
your Cypher verbatim and applies no filter.
</Note>
<Columns>
<Card title="Feedback System" icon="brain-circuit" href="/guides/feedback-system">
Rate answers and fold ratings into ranking
</Card>
<Card title="Improve" icon="sparkles" href="/core-concepts/main-operations/improve">
Enrich the graph and bridge session memory
</Card>
<Card title="Setup Configuration" icon="settings" href="/setup-configuration/overview">
Personalization environment variables
</Card>
</Columns>
# Forget
Source: https://docs.cognee.ai/core-concepts/main-operations/forget
Delete data, datasets, or memory-only state with forget.
## What is the forget operation
The `.forget` operation is the unified deletion command in Cognee v1.0.
* **Single data item deletion**: remove one data item from a dataset.
* **Dataset deletion**: remove an entire dataset and its graph/vector data.
* **Full cleanup**: remove everything the current user can delete.
* **Memory-only reset**: delete only graph/vector memory for a dataset or a single file, preserving raw files so the dataset can be re-cognified with different settings.
* **User-scoped deletion**: `forget()` covers the main v1.0 deletion cases, but it does **not** replace low-level destructive `prune` operations.
## Where forget fits
* Use `forget()` when you want to remove memory.
* Use it to clean up test data, reset a dataset, or fully wipe a local user’s memory.
* Use dataset-level forgetting for most operational cleanup.
* Use `prune` only for destructive developer resets that must wipe underlying storage or system metadata directly.
## What happens under the hood
### Forget a specific data item
* Requires both `data_id` and `dataset`.
* Resolves the dataset by name or UUID with delete-permission checks.
* Deletes that item from the dataset with `delete_dataset_if_empty=False`.
* Removes the session-cache turns whose answers were built on the graph elements this delete removed, so cached answers stop quoting the deleted document.
* Leaves the dataset itself intact.
### Forget a dataset
* Resolves the dataset by name or UUID with delete-permission checks.
* Deletes the dataset's relational records and contained data items.
* Deletes graph nodes and edges for that dataset.
* Deletes vector embeddings for that dataset.
* Deletes every session in the cache attributed to that dataset, since all of their content derives from data that is now gone, and removes the contaminated turns from sessions that carry no dataset attribution.
### Forget everything
* Deletes all datasets the current user can delete.
* Removes graph, vector, and relational data for those datasets.
* Also prunes the session cache when caching or usage logging is enabled.
* Does **not** wipe raw uploaded files or bypass permission checks the way `prune` does.
### Forget memory only (dataset)
* Requires `dataset`. `memory_only=True` cannot be used without a dataset.
* Resolves the dataset by name or UUID with delete-permission checks.
* Deletes all graph nodes and edges for the dataset.
* Deletes all vector embeddings for the dataset.
* Deletes every session attributed to the dataset — their cached answers assert graph content that no longer exists, and a re-cognify with different settings should not replay them — and removes the contaminated turns from sessions that carry no dataset attribution.
* Resets `pipeline_status` on all data records in the dataset, allowing `cognify` to re-process them.
* **Does not** remove raw files or the dataset/data relational records.
### Forget memory only (single file)
* Requires both `dataset` and `data_id`, plus `memory_only=True`.
* Resolves the dataset by name or UUID with delete-permission checks.
* Deletes the graph nodes and edges associated with that single data item.
* Deletes the vector embeddings for that data item.
* Removes only the contaminated session-cache entries: turns whose answers used the deleted graph elements, plus anything that inherited from them (see the scope matrix below). Untouched turns in the same session survive.
* Resets the `cognify_pipeline` status entry for that data record, allowing re-processing.
* **Does not** remove the raw file or the data record itself.
## After forget finishes
* **Single-item forget**: the specified item is removed from the dataset, while the dataset remains. Session turns that used the item's graph elements are removed; the rest of each session survives.
* **Dataset forget**: the dataset's relational, graph, and vector data are removed, along with the sessions attributed to it and the contaminated turns in unattributed sessions.
* **Everything forget**: all datasets the current user can delete are removed, and the session cache is pruned wholesale when session caching or usage logging is enabled.
* **Memory-only (dataset)**: graph, vector, and pipeline status are cleared, the dataset's sessions are deleted, and the contaminated turns in unattributed sessions are removed; the dataset, data records, and raw files remain intact and can be re-cognified.
* **Memory-only (single file)**: graph nodes/edges and vector embeddings for that file are removed, together with the session turns built on them; the data record and raw file are preserved.
## Examples and details
<Accordion title="Forget a single data item">
```python theme={null}
await cognee.forget(
data_id=item_id,
dataset=dataset_id,
)
```
</Accordion>
<Accordion title="Forget an entire dataset">
```python theme={null}
await cognee.forget(dataset="scientists")
```
</Accordion>
<Accordion title="Forget everything for the current user">
```python theme={null}
await cognee.forget(everything=True)
```
</Accordion>
<Accordion title="Forget only memory for a dataset (keep raw files)">
Use this when you want to re-cognify a dataset with different settings (e.g. a new graph model or custom prompt) without removing the original files.
```python theme={null}
await cognee.forget(dataset="scientists", memory_only=True)
```
Graph nodes/edges and vector embeddings are deleted and the pipeline status is reset.
Raw files and the dataset/data records are preserved.
</Accordion>
<Accordion title="Forget only memory for a single file">
Clears just the graph and vector memory for one file, without touching the rest of the dataset.
```python theme={null}
await cognee.forget(
dataset="scientists",
data_id=item_id,
memory_only=True,
)
```
</Accordion>
<Accordion title="Return values">
`forget()` returns a summary dictionary.
* Item deletion returns fields like `data_id`, `dataset_id`, and `status`.
* Dataset deletion returns the resolved `dataset_id` and `status`.
* Full deletion returns the number of datasets removed plus `status`.
* Memory-only dataset reset returns `dataset_id`, `data_records_reset` (count of data records in the dataset), and `status`.
* Memory-only single-file reset returns `data_id`, `dataset_id`, and `status`.
</Accordion>
<Accordion title="Deletion scope by mode">
| Mode | Relational data | Graph data | Vector data | Pipeline status | Raw files | Session cache |
| ------------------------------------------ | ------------------------------ | ------------------------------ | ------------------------------ | ----------------------- | --------- | ---------------------------------------------------------------------------------------- |
| `data_id` + `dataset` | removes the targeted item | removed for that item | removed for that item | unchanged | preserved | contaminated turns removed |
| `dataset` | removed | removed | removed | removed | preserved | sessions for that dataset deleted; contaminated turns removed from unattributed sessions |
| `everything=True` | removed for all owned datasets | removed for all owned datasets | removed for all owned datasets | removed | preserved | pruned when session caching or usage logging is enabled |
| `dataset` + `memory_only=True` | preserved | removed | removed | reset (cognify re-runs) | preserved | sessions for that dataset deleted; contaminated turns removed from unattributed sessions |
| `dataset` + `data_id` + `memory_only=True` | preserved | removed for that item | removed for that item | reset for that item | preserved | contaminated turns removed |
Every mode cleans up session memory, so cached answers stop quoting content the delete removed. The two granularities differ:
* **Dataset-level deletes** (`dataset`, `dataset` + `memory_only=True`, and each dataset visited by `everything=True`) drop whole sessions — every session attributed to the dataset, found through the `dataset_id` recorded on the session or the per-dataset default session id (`default_session_<dataset_id>`) — and then apply the surgical pass below to sessions with no dataset attribution, so an answer that drew on the deleted content from the global `default_session` is removed too.
* **Single-item deletes** (`data_id` + `dataset`, with or without `memory_only=True`) are surgical. A turn is *contaminated* when the graph elements it recorded using overlap the nodes and edges this delete removed. Contamination then propagates inside that session — a contaminated turn taints feedback that references it, feedback taints the distilled session-context lesson it fed, and that lesson taints later turns that consumed it — and the whole chain is removed. Everything else in the session is kept.
`forget(everything=True)` remains the only mode that prunes the cache **wholesale** — every mode reaches sessions with no dataset attribution, but the others only surgically, removing the contaminated turns and leaving the session standing. None of these modes is a raw-storage/system-metadata reset like `prune`. The `memory_only` modes are the only modes that preserve relational records while clearing derived knowledge.
</Accordion>
<Accordion title="Does forget re-run the LLM or cost tokens?">
No. `forget()` (and the legacy [`delete()`](/core-concepts/main-operations/legacy-operations/delete)) only **remove** data — they never call the LLM or embedding model, so they consume **no tokens** and incur no model cost.
Under the hood, forgetting performs deletes against the relational, graph, and vector stores. It does **not** re-extract entities, re-embed text, or rebuild graph relationships. Existing nodes and edges are simply dropped (or preserved when still shared, as described below); the remaining graph is left as-is and is not recomputed.
When you rebuild memory after a forget, model calls happen in the operations that build memory — [`remember`](/core-concepts/main-operations/remember)/`cognify` and [`memify`](/core-concepts/main-operations/legacy-operations/memify). Tokens are spent only when you re-cognify after a `memory_only` forget, not by the forget itself.
</Accordion>
<Accordion title="What is tracked per document: nodes and relationships">
A common misconception is that item-level deletion only considers graph *nodes*. In practice Cognee also records relationship ownership metadata per source document: when `cognify` builds the graph, it records which graph nodes and which edges (relationships) were derived from each source.
When you forget a single data item (with or without `memory_only`), Cognee uses those ownership records to identify graph memory tied to that item:
| Owned by the removed document | Also referenced by another document in scope | Result |
| ------------------------------------ | -------------------------------------------- | --------------------------------------------------- |
| node | no | **deleted** from graph + vector stores (orphaned) |
| node | yes | **preserved** so the rest of the graph stays intact |
| relationship metadata / edge indexes | no longer referenced | **cleaned up** along with the removed graph memory |
| relationship metadata | still referenced | **preserved** for the remaining graph memory |
The per-document ownership records are always cleared, even when every node the document referenced is shared and therefore kept in the graph. Graph relationships attached to deleted nodes are removed with those nodes; relationships between surviving shared nodes are not independently deleted by `forget()`.
To replace a document's extracted graph memory with memory from new content, use [`update()`](/python-api/update), which runs this same item-level deletion flow and then re-cognifies the new version.
</Accordion>
<Accordion title="User context and ownership">
`forget()` always runs in a user context.
* If you do not pass `user`, Cognee resolves the default user.
* Deletion scope is based on what that user owns or has delete access to.
* `everything=True` means "everything the current user can delete," not "everything in the whole system."
* This is why the `user` parameter matters in multi-user or permissioned setups.
</Accordion>
<Accordion title="Parameters">
<Tabs>
<Tab title="Basic Parameters">
| Option | What it does |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `dataset` | Deletes an entire dataset by name or UUID. |
| `data_id` | Deletes one specific data item, but only when `dataset` is also provided. |
| `everything` | Deletes all datasets and data the current user can delete. |
| `memory_only` | When `True`, deletes only memory (graph + vector) and resets pipeline status for the given `dataset`, preserving raw files. Requires `dataset`. Combine with `data_id` to target a single file. |
</Tab>
<Tab title="Advanced Parameters">
| Option | What it does |
| ------ | ----------------------------------------------------------------------------------------- |
| `user` | Runs forget under a specific user context, affecting ownership checks and deletion scope. |
</Tab>
</Tabs>
</Accordion>
<Accordion title="Permissions and safety">
* Deleting a specific item or dataset requires delete access to that dataset.
* Dataset resolution filters by the `delete` permission, so a dataset you can only read is indistinguishable from one that does not exist. A **name** that resolves to nothing fails with `DatasetNotFoundError: Dataset '<name>' not found or not accessible.` (HTTP **404**); a **UUID** you cannot delete fails with `PermissionDeniedError: Request owner does not have necessary permission: [delete] for all datasets requested.` (HTTP **403**). See [troubleshooting dataset resolution](/python-api/forget#troubleshooting).
* `data_id` cannot be used alone; it must be paired with `dataset`.
* `memory_only=True` requires `dataset` to be specified; omitting it raises a `ValueError`.
* `memory_only` operations check delete permission on the dataset before removing any graph or vector data.
* `everything=True` ignores `data_id` and `dataset` and wipes all data the current user can delete.
* Session-cache cleanup is best-effort by contract: it never fails the delete that triggered it. If the cache is unavailable or a session cannot be cleaned, Cognee logs a warning and the relational, graph, and vector deletions still complete. A `"status": "success"` result therefore does not guarantee that every session entry was removed — check the logs if stale cached answers persist.
* `forget()` is not a substitute for `prune_data()` or `prune_system(...)`, which are lower-level destructive maintenance tools.
</Accordion>
<Accordion title="Under the hood — legacy operations">
`forget()` wraps the [Delete](/core-concepts/main-operations/legacy-operations/delete) and dataset-deletion APIs under the hood, extending them with a unified interface and session cache cleanup.
Use legacy Delete directly only when maintaining older integrations or referencing older documentation. For destructive storage resets that bypass normal deletion logic, use `prune` rather than `forget()`.
</Accordion>
<Accordion title="Inspect what you've stored before forgetting">
To find the dataset name or `data_id` needed for `forget()`, list your datasets and their contents first:
```python theme={null}
import cognee
# See all datasets
datasets = await cognee.datasets.list_datasets()
for ds in datasets:
print(ds.name, ds.id)
# See items inside a specific dataset
items = await cognee.datasets.list_data(dataset_id=ds.id)
for item in items:
print(item.id, item.name)
```
See [datasets API reference](/python-api/datasets) for the full set of listing and management methods.
</Accordion>
<Columns>
<Card title="Remember" icon="brain" href="/core-concepts/main-operations/remember">
Add new permanent or session memory
</Card>
<Card title="Recall" icon="search" href="/core-concepts/main-operations/recall">
Verify what memory is currently retrievable
</Card>
</Columns>
# Improve
Source: https://docs.cognee.ai/core-concepts/main-operations/improve
Enrich an existing Cognee graph and bridge session memory into it.
## What is the improve operation
The `.improve` operation enriches an existing Cognee graph after data has already been ingested.
* **Graph enrichment**: by default, `improve()` runs Cognee's built-in enrichment pass on an existing dataset, adding derived retrieval structures that make later recall work better.
* **Session bridging**: with `session_ids`, it moves useful session memory into the permanent graph.
* **Session distillation**: with `session_ids`, it can turn gated session guidance into curated, entity-anchored lesson documents tagged under `session_learnings`.
* **Feedback-aware**: it can raise or lower the importance of graph elements based on feedback attached to session answers that used those elements during retrieval.
* **Global context indexing**: with `build_global_context_index=True`, it builds dataset-level bucket and root summaries that can later be prepended during graph completion retrieval.
* **Truth-subspace build**: with `build_truth_subspace=True`, it can build truth-subspace anchors from distilled `session_learnings`.
* **Sync back to sessions**: after enrichment, it can write new graph relationships back into session cache for faster future session recall.
## Where improve fits
* Use `improve()` after [Remember](/core-concepts/main-operations/remember) when you want to enrich an existing graph further.
* Use it at the end of a chat or agent session to bridge short-term session memory into permanent memory.
* Use it when you want custom extraction or enrichment tasks.
* Use it instead of re-ingesting everything when the graph already exists and you want additive enrichment.
## What happens under the hood
<img alt="Cognee self-improvement flow" />
### Without session IDs
1. **Run graph enrichment**
* `improve()` runs an enrichment pass on the target dataset.
* By default, this extracts and indexes triplet datapoints when triplet embeddings are enabled.
* The default pass streams triplets directly from the graph database instead of projecting the whole graph into memory, so it avoids a full-graph scan and `O(graph size)` memory use on every run — including session-end auto-improve. The graph is still projected when you supply custom extraction or enrichment tasks without explicit `data`, since those tasks may consume the projected fragment. Enrichment output is the same either way.
* For coding-rule retrieval, pass explicit coding-rule extraction and enrichment tasks.
2. **Optionally build the [global context index](/core-concepts/further-concepts/global-context-index)**
* When `build_global_context_index=True`, Cognee builds semantic summary buckets over existing `TextSummary` nodes.
* It also creates a root summary for the dataset — a compact, dataset-wide "world summary".
* The first build is the most expensive; later runs only place newly added summaries. See the linked page for cost details.
* This index can later be included in `GRAPH_COMPLETION` and `HYBRID_COMPLETION` search with `include_global_context_index=True`.
### With session IDs
When `session_ids` is provided, Cognee can run these stages:
1. **Apply feedback weights**
* Session feedback updates `feedback_weight` on graph nodes and edges that were used during retrieval.
* In practice, this means highly rated answers can make their source graph elements more influential later, while poorly rated answers can reduce their influence.
2. **Persist session Q\&A**
* Question-and-answer content from the session is cognified into the permanent graph.
* Persisted session content is tagged under the `user_sessions_from_cache` node set.
3. **Persist agent traces**
* Structured trace steps from agent/tool activity are cognified into the graph, so tool outcomes can become long-term memory instead of staying only in cache.
4. **Extract session context**
* Pending trace windows can be summarized into session-context lessons before distillation.
* These lessons join the same gated guidance pool used by conversational session context.
5. **Distill sessions**
* Cognee loads session Q\&A plus active session-context guidance.
* Guidance is eligible only when it has not been rated harmful and its confidence passes the distillation gate.
* A curator proposes durable lessons, a writer/rejecter checks them against existing lessons and graph entities, and accepted lessons are added and cognified into the target dataset.
* Distilled documents are tagged under `session_learnings` and a session-specific node set.
6. **Update user preferences**
* Rated turns and stated preferences from those sessions are folded into the calling user's per-dataset preference subgraph: weighted `prefers` edges plus the preference node's text.
* This runs once for all the sessions you pass, because preferences aggregate across a user's sessions.
* It only does anything when `PERSONALIZATION_ENABLED` is on. With the flag off (the default) the stage writes nothing at all.
* It is best-effort: a failure here is logged and never blocks the rest of `improve()`.
* See [User Preferences](/core-concepts/further-concepts/user-preferences).
7. **Optionally build the truth subspace**
* When `build_truth_subspace=True`, Cognee builds truth-subspace anchors from the distilled `session_learnings`.
* This stage is opt-in and only applies when `session_ids` is provided.
8. **Run enrichment**
* The dataset goes through the normal enrichment pass.
9. **Optionally build the global context index**
* When `build_global_context_index=True`, Cognee builds the same retrieval-ready summary layer after enrichment.
10. **Sync graph back to sessions**
* Newly enriched graph relationships are copied back into the session cache as human-readable context.
## After improve finishes
* **Without session IDs**: the target dataset has gone through the enrichment pass and is ready for better downstream retrieval.
* **With session IDs**: session feedback, Q\&A, trace activity, and accepted distilled lessons can be persisted into the permanent graph, enrichment runs, and new graph context may be synced back into those sessions.
* **With session IDs and `PERSONALIZATION_ENABLED=true`**: the calling user's preference node and `prefers` weights for that dataset are also updated from rated turns and stated preferences, and start nudging that user's ranking on later recalls.
* **With `build_global_context_index=True`**: the dataset also has bucket and root summaries available for graph completion retrieval.
* **With `build_truth_subspace=True`**: distilled `session_learnings` can also become truth-subspace anchors for opt-in hybrid reranking.
## Examples and details
<Accordion title="What graph enrichment means">
In the context of `improve()`, **graph enrichment** means adding new derived retrieval structures or knowledge on top of an already-built graph instead of re-ingesting the original source data from scratch.
By default, that usually means:
* extracting and indexing triplet datapoints when triplet embeddings are enabled
* improving how the graph can be searched later
* optionally adding extra derived structures through custom tasks
So `improve()` is not the stage that first creates the graph. Instead, it makes an existing graph **more useful for future retrieval**.
</Accordion>
<Accordion title="What feedback weights mean">
**Feedback weights** are stored importance signals attached to graph elements that were used during retrieval.
* When a session answer has feedback and Cognee knows which graph nodes or edges helped produce that answer, `improve()` can update those elements' `feedback_weight`.
* Positive feedback can make those elements more influential in future ranking.
* Negative feedback can make them less influential.
* If no retrieval trace exists for the session, or if no feedback was captured, this stage may have little or nothing to update.
This is one of the ways Cognee lets memory quality improve over time without retraining the model itself.
</Accordion>
<Accordion title="What session distillation means">
Session distillation is the part of `improve(session_ids=[...])` that turns short-term session guidance into long-term graph memory.
During a session, Cognee can accumulate guidance such as goals, rules, preferences, and lessons learned. Distillation does not blindly persist every guidance entry. It first filters out entries that have harmful feedback or low confidence, then uses a curator/writer pass to keep only durable lessons that are supported by the session and not already known.
Accepted lessons are rendered as standalone markdown documents and written back through `add()` + `cognify()` into the target dataset. They are tagged with the `session_learnings` node set so later systems, including truth-subspace reranking, can find them.
If you want to run only this stage for one finished session, use:
```python theme={null}
result = await cognee.session.distill_session(
"support_chat_7",
dataset="product_docs",
)
print(result.status)
print(result.documents)
```
`result.status` is one of `completed`, `no_gated_entries`, `no_proposed_lessons`, or `no_accepted_lessons`.
</Accordion>
<Accordion title="How custom improvement tasks work">
`improve()` supports power-user overrides for custom extraction and enrichment tasks.
* `extraction_tasks` lets you define what intermediate subgraph or source material should be prepared.
* `enrichment_tasks` lets you define what new derived structures should be added to the graph.
* This is how you move beyond the default enrichment pass and create domain-specific memory behavior.
</Accordion>
<Accordion title="Build the global context index">
Use the global context index when later answers need document-wide or dataset-wide orientation in addition to local graph facts.
```python theme={null}
await cognee.improve(
dataset="project_memory",
build_global_context_index=True,
)
```
This builds `GlobalContextSummary` buckets over `TextSummary` nodes and one root summary for the dataset. During graph completion search, enable it with `retriever_specific_config={"include_global_context_index": True}`.
See [Global Context Index](/core-concepts/further-concepts/global-context-index) for the full model and retrieval example.
</Accordion>
<Accordion title="What improve produces">
* Enriched graph structures on the target dataset
* Triplet-embedding style retrieval artifacts (when triplet embeddings are enabled)
* Optional global context bucket and root summaries
* Optional persistence of session Q\&A into the permanent graph
* Optional persistence of agent trace steps into the permanent graph
* Optional distilled session-learning documents under `session_learnings`
* Optional feedback-based weighting updates on graph elements used during retrieval
* Optional truth-subspace anchors when `build_truth_subspace=True`
* Optional sync of newly enriched graph context back into session cache
</Accordion>
<Accordion title="What improve needs before it can help">
* A target dataset must already exist.
* You need `write` permission on that dataset. The dataset is resolved and authorized once, before any stage runs, so a UUID you cannot write to — or one that does not exist — fails the whole call with `PermissionDeniedError` instead of quietly enriching your own default dataset.
* A dataset owned by another user can only be targeted by UUID. Names are owner-scoped, so passing someone else's dataset *name* creates your own dataset with that name and leaves theirs untouched.
* That dataset should already contain graph memory, usually created by [Remember](/core-concepts/main-operations/remember).
* Session bridging only applies when you pass `session_ids`.
* Feedback-based weighting only helps when those sessions contain feedback and retrieval traces.
* Session distillation only produces documents when the session contains gated guidance that survives curation.
So if you run `improve()` on a dataset with no existing graph content, or pass sessions that have no useful cached interactions, the operation may run successfully but add little new value.
</Accordion>
<Accordion title="Bridge session memory into the graph">
```python theme={null}
await cognee.improve(
dataset="rules_demo",
session_ids=["chat_1", "chat_2"],
)
```
* This applies feedback weights from those sessions.
* It persists the session Q\&A into the permanent graph.
* It persists agent trace steps when present.
* It distills accepted session guidance into `session_learnings`.
* It enriches the graph and syncs new context back into the sessions.
</Accordion>
<Accordion title="Parameters">
<Tabs>
<Tab title="Basic Parameters">
| Option | What it does |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `dataset` | The dataset name or UUID to improve. Defaults to `main_dataset`. Requires `write` permission; a shared dataset must be given as a UUID. |
| `session_ids` | Bridges those sessions into the permanent graph and syncs graph context back. |
| `run_in_background` | Runs the improvement pipeline asynchronously. |
| `build_global_context_index` | Builds semantic bucket summaries and a root dataset summary after enrichment. Skipped in background mode. |
| `build_truth_subspace` | Builds truth-subspace anchors from distilled `session_learnings`. Only runs when `session_ids` is provided. |
| `node_name` | Restricts the projected graph fragment to specific named entities or node sets. Only applies when the graph is projected (custom extraction/enrichment tasks and no explicit `data`); the default pass skips projection. |
| `feedback_alpha` | Controls how strongly session feedback changes graph weights. |
</Tab>
<Tab title="Advanced Parameters">
| Option | What it does |
| --------------------------------------- | ------------------------------------------------------------------------------------------- |
| `extraction_tasks` / `enrichment_tasks` | Overrides the default enrichment task set. |
| `data` | Supplies explicit data to advanced improvement pipelines when supported. |
| `node_type` | Changes which node type the enrichment pass targets. |
| `user` | Runs improve under a specific user context, affecting dataset access and session ownership. |
| `vector_db_config` / `graph_db_config` | Overrides database backend configuration for the vector or graph stores. |
</Tab>
</Tabs>
</Accordion>
<Accordion title="Under the hood — legacy operations">
`improve()` runs [Memify](/core-concepts/main-operations/legacy-operations/memify) under the hood for its enrichment pass.
Use legacy Memify directly when you need fine-grained control over extraction and enrichment tasks — for example, to supply custom task lists or target specific pipeline stages. Note that `remember(..., self_improvement=True)` already calls `improve()` for you after permanent ingestion.
</Accordion>
<Columns>
<Card title="Remember" icon="brain" href="/core-concepts/main-operations/remember">
Ingest and build memory in one call
</Card>
<Card title="Recall" icon="search" href="/core-concepts/main-operations/recall">
Query the improved graph and session memory
</Card>
</Columns>
# Add
Source: https://docs.cognee.ai/core-concepts/main-operations/legacy-operations/add
Ingest and prepare data for processing in Cognee.
<Note>
`add()` is a legacy operation. In Cognee v1.0, most users should use [remember()](/core-concepts/main-operations/remember) instead, which replaces the `add()` + `cognify()` + `memify()` workflow with a single call.
</Note>
## What is the add operation
The `.add` operation is how you bring content into Cognee. It takes your files, directories, or raw text, normalizes them into plain text, and records them into a dataset that Cognee can later expand into vectors and graphs with [Cognify](/core-concepts/main-operations/legacy-operations/cognify).
* **Ingestion-only**: no embeddings, no graph yet
* **Flexible input**: raw text, local files, directories, any [Docling](https://github.com/docling-project/docling) supported format, S3 URIs, or HTTP/HTTPS URLs
* **Normalized storage**: everything is turned into text and stored consistently
* **Deduplicated**: Cognee uses content hashes to avoid duplicates
* **Dataset-first**: everything you add goes into a dataset
* Datasets are how Cognee keeps different collections organized (e.g. "research-papers", "customer-reports")
* Each dataset has its own ID, owner, and permissions for access control
* You can read more about them below
## Where add fits
* First step before you run [Cognify](/core-concepts/main-operations/legacy-operations/cognify)
* Use it to **create a dataset** from scratch, or **append new data** over time
* Ideal for both local experiments and programmatic ingestion from storage (e.g. S3)
## What happens under the hood
1. **Expand your input**
* Directories are walked, S3 paths are expanded, raw text is passed through
* Result: a flat list of items (files, text, handles)
2. **Ingest and register**
* Files are saved into Cognee's storage and converted to text
* Cognee computes a stable content hash to prevent duplicates
* Each item becomes a record in the database and is attached to your dataset
* **Text extraction**: Converts various file formats into plain text
* **Metadata preservation**: Keeps file-system metadata like name, extension, MIME type, file size, and content hash — not arbitrary user-defined fields
* **Content normalization**: Ensures consistent text encoding and formatting
3. **Return a summary**
* You get a pipeline run info object that tells you where everything went and which dataset is ready for the next step
## After add finishes
After `.add` completes, your data is ready for the next stage:
* **Files are safely stored** in Cognee's storage system with metadata preserved
* **Database records** track each ingested item and link it to your dataset
* **Dataset is prepared** for transformation with [Cognify](/core-concepts/main-operations/legacy-operations/cognify) — which will chunk, embed, and connect everything
## Further details
<Accordion title="Input sources">
* Mix and match: `["some text", "/path/to/file.pdf", "s3://bucket/data.csv", "https://example.com/page"]`
* Works with directories (recursively), S3 prefixes, file handles, and HTTP/HTTPS URLs
* Local and cloud sources are normalized into the same format
* HTTP/HTTPS URLs are scraped as web pages — see [URL ingestion](#url-ingestion-httphttps) below for the distinction between web pages and direct file downloads
</Accordion>
<Accordion title="URL ingestion (HTTP/HTTPS)">
Passing an `http://` or `https://` URL to `cognee.add()` triggers **web page scraping** — the URL is fetched and its response is saved as HTML for processing. Cognee does **not** inspect the server's `Content-Type` or the URL's file extension to detect direct binary file downloads.
| URL type | What Cognee does |
| -------------------------------- | ------------------------------------------------------------------------------------- |
| `https://example.com/article` | Fetches the page HTML and extracts text |
| `https://example.com/report.pdf` | Fetches the HTTP response as HTML — the PDF binary is **not** downloaded or extracted |
The fetching backend is picked from the environment: Tavily if `TAVILY_API_KEY` is set, otherwise Keenable if `KEENABLE_API_KEY` is set, otherwise the built-in `DefaultUrlCrawler` (BeautifulSoup). See the [Web URL Ingestion guide](/guides/web-url-ingestion) for a complete walkthrough, [Python API: add()](/python-api/add) for more URL ingestion examples, the [web URL content ingestion demo](https://github.com/topoteretes/cognee/blob/dev/examples/guides/web_url_content_ingestion_example.py) for a complete example, and [Loaders](/core-concepts/further-concepts/loaders) for `preferred_loaders` examples.
</Accordion>
<Accordion title="Structured data (dlt)">
Cognee integrates with [dlt](https://dlthub.com/) to ingest structured relational data directly into the knowledge graph:
* **dlt resources**: Pass `@dlt.resource()` decorated generators directly to `cognee.add()`
* **CSV files**: `.csv` files take the dlt route through the loader engine's `dlt_csv_loader` when `cognee[dlt]` is installed; without the extra they are flattened to text by `CsvLoader`
* **Database connections**: Pass a connection string (`postgresql://...`, `sqlite:///...`) to ingest tables directly
* Foreign key relationships become graph edges automatically
* Structured data bypasses LLM extraction — the graph is built deterministically from the schema
* See the full [dlt integration guide](/integrations/dlt-integration) for details
</Accordion>
<Accordion title="LlamaIndex documents">
`cognee.add()` can also accept LlamaIndex `Document` and `ImageDocument` objects when the `llama-index` extra is installed. See [Installation](/getting-started/installation) for the package extra list.
* Works with LlamaIndex loaders and connectors without a manual conversion step
* If a `Document` includes `metadata["file_path"]`, Cognee ingests that original file directly
* If an `ImageDocument` includes `image_path`, Cognee ingests the image file directly
* Otherwise, Cognee saves the document text to a temporary file and continues through the normal add pipeline
</Accordion>
<Accordion title="Supported formats">
Cognee automatically selects the best loader based on file extension. The table below lists all supported extensions and whether optional extras are needed:
| Loader | Extensions | Install extra |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ |
| **CodeLoader** | `.py` `.js` `.ts` `.go` `.rs` `.java` and 25 more source-code extensions — see [Loaders](/core-concepts/further-concepts/loaders); these take the code graph pipeline instead of LLM extraction | — (built-in) |
| **TextLoader** | `.txt` `.md` `.json` `.xml` `.yaml` `.yml` `.log` | — (built-in) |
| **DltCsvLoader** | `.csv` — takes precedence over `CsvLoader` when installed; routes rows through [dlt structured ingestion](/integrations/dlt-integration) | `pip install cognee[dlt]` |
| **CsvLoader** | `.csv` | — (built-in) |
| **PyPdfLoader** | `.pdf` | — (built-in) |
| **ImageLoader** | `.png` `.jpg` `.jpe` `.jpeg` `.gif` `.webp` `.bmp` `.tif` `.tiff` `.heic` `.avif` `.ico` `.psd` `.apng` `.cr2` `.dwg` `.xcf` `.jxr` `.jpx` | — (built-in) |
| **AudioLoader** | `.mp3` `.wav` `.aac` `.flac` `.ogg` `.m4a` `.mid` `.amr` `.aiff` | — (built-in) |
| **UnstructuredLoader** | `.docx` `.doc` `.odt` `.xlsx` `.xls` `.ods` `.pptx` `.ppt` `.odp` `.rtf` `.html` `.htm` `.eml` `.msg` `.epub` | `pip install cognee[docs]` |
| **AdvancedPdfLoader** | `.pdf` (layout-aware, preserves tables) | `pip install cognee[docs]` |
| **BeautifulSoupLoader** | `.html` | `pip install cognee[scraping]` |
| **DoclingDocument** | Pre-converted `DoclingDocument` objects | `pip install cognee[docling]` |
* **ImageLoader** uses a vision-capable LLM to transcribe image content (extraction-oriented prompt by default, with an optional local OCR pass).
* **AudioLoader** transcribes audio using a Whisper-compatible model.
* **AdvancedPdfLoader** preserves page layout and table structure; falls back to PyPdfLoader automatically on error.
You can learn more about how loaders work, override defaults, or register custom loaders in the [Loaders](/core-concepts/further-concepts/loaders) section.
</Accordion>
<Accordion title="Datasets">
* A dataset is your "knowledge base" — a grouping of related data that makes sense together
* Datasets are **first-class objects in Cognee's database** with their own ID, name, owner, and permissions
* They provide **scope**: `.add` writes into a dataset, [Cognify](/core-concepts/main-operations/legacy-operations/cognify) processes per-dataset
* Think of them as separate shelves in your library — e.g., a "research-papers" dataset and a "customer-reports" dataset
* If you name a dataset that doesn't exist, Cognee creates it for you; if you don't specify, a default one is used
* More detail: [Datasets](/core-concepts/further-concepts/datasets)
</Accordion>
<Accordion title="Users and ownership">
* Every dataset and data item belongs to a user
* If you don't pass a user, Cognee creates/uses a default one
* Ownership controls who can later read, write, or share that dataset
</Accordion>
<Accordion title="Node sets">
* Optional labels to group or tag data on ingestion
* Example: `node_set=["AI", "FinTech"]`
* Useful later when you want to focus on subgraphs
* More detail: [NodeSets](/core-concepts/further-concepts/node-sets)
</Accordion>
<Accordion title="Hash-based file storage, deduplication, and filename collisions">
When Cognee stores an ingested file, it renames it using the pattern `text_<md5_hash>.txt`, where the hash is computed from the original file's byte content. For example, adding `report.pdf` produces a stored file like `text_a3f1c8b2....txt` rather than `report.txt`.
This naming scheme is intentional and powers deduplication:
* The MD5 hash is derived from the **file content**, not the filename
* Re-adding the same file, even with a different name, produces the same hash, so Cognee detects the existing record and skips re-ingestion (`incremental_loading=True` by default)
* All loaders, including text, PDF, image, and audio, follow the same convention, so your storage directory will contain hash-named `.txt` files regardless of the original format
This is why stored files look unfamiliar when you inspect your `DATA_ROOT_DIRECTORY`. The original filename is preserved in the relational database as metadata on the `Data` record, but the on-disk representation uses the content hash.
Cognee's copy of an **uploaded source file** (a file sent through the API or the UI) is content-addressed too, but in a different shape: it lands under `<content_md5>/<original_filename>` — for example `a3f1c8b2…/report.pdf` — so you will see one hash-named directory per distinct payload rather than a flat directory of files. The hash goes in the directory prefix rather than the filename because the real basename still matters downstream: loaders select by file suffix, and the code-graph route derives stable node identity from it. Derived text is unaffected and keeps its flat `text_<md5>.txt` name, as does raw text you pass to `add()` directly.
Because the hash is part of the key, two uploads that share a filename but differ in content no longer overwrite each other's stored copy, and re-uploading identical bytes writes to the same key instead of adding another object.
Cognee deduplicates by **file content**, not by filename. Deduplication is a **lookup, not an identity**: on every `add()`, Cognee looks for an existing `Data` row matching `(dataset_id, owner_id, tenant_id, content_hash)`. A hit reuses that row; a miss creates a new row with a fresh random `uuid4` id.
The record id therefore carries no content information and is **stable for the lifetime of the document** — it does not change when the document's content changes (see [`update()`](/python-api/update)).
| Scenario | Result |
| ------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Same file added twice to the same dataset | Second call is a no-op because the lookup finds the existing record in that dataset |
| Same file added to a different dataset | No new storage copy, but a **second, independent `Data` record** with its own id — the two documents can be updated and deleted without affecting each other |
| Two files with the **same name** but **different contents** | Two separate records because different content produces different hashes — and, for uploads, two separate stored copies, because the content hash prefixes the storage key |
| Two files with **different names** but **identical contents** | One record per dataset because the same content produces the same hash |
If two files arrive simultaneously with the same filename but different contents, Cognee computes separate hashes and stores them as distinct records. Filename is metadata only and plays no role in deduplication.
Deduplication never crosses dataset, user, or tenant boundaries: the lookup filters on the dataset, the owner's user ID, and the tenant ID, so the same file added to another dataset, uploaded by a different user, or added in a different tenant becomes a separate record rather than being collapsed into a shared one.
<Note>
Records added before Cognee 1.5.0 had ids derived from `content_hash + user_id + tenant_id` and a single record could be shared by several datasets. The upgrade migration splits those shared records per dataset — see [`run_migrations`](/python-api/run-migrations). Ids issued before the split keep resolving, so data ids you stored externally remain valid.
</Note>
</Accordion>
<Accordion title="Custom metadata, labels, and grouping">
`cognee.add()` automatically preserves only **file-system metadata** like name, MIME type, extension, content hash.
If you need to associate extra information with ingested data, three mechanisms are available:
<AccordionGroup>
<Accordion title="node_set tags">
Pass a list of string tags to mark every item in that `add()` call:
```python theme={null}
await cognee.add(
"Quarterly earnings report Q4 2024.",
node_set=["finance", "Q4-2024"]
)
```
Tags flow into the knowledge graph as `NodeSet` nodes connected with `belongs_to_set` edges, and can be used to scope searches later — see [NodeSets](/core-concepts/further-concepts/node-sets).
</Accordion>
<Accordion title="DataItem metadata and labels">
Wrap individual data items in `DataItem` when you need to attach metadata or control per-item identifiers:
```python theme={null}
import cognee
from cognee.tasks.ingestion.data_item import DataItem
item = DataItem(
data="/path/to/report.pdf",
label="q4-earnings-report",
external_metadata={
"title": "Q4 Financial Report",
"author": "Jane Smith",
"date": "2024-12-31",
"department": "Finance",
},
)
await cognee.add(item, dataset_name="reports")
```
You can also pass a list of `DataItem` objects to ingest multiple files with different metadata in one call:
```python theme={null}
items = [
DataItem(
data="/path/to/paper1.pdf",
label="paper-one",
external_metadata={"title": "Paper One", "author": "Alice"},
),
DataItem(
data="/path/to/paper2.pdf",
label="paper-two",
external_metadata={"title": "Paper Two", "author": "Bob"},
),
]
await cognee.add(items, dataset_name="research")
```
**`DataItem` fields**
| Field | Type | Description |
| ------------------- | ----------------- | ---------------------------------------------------------------------------------------------------- |
| `data` | any | The content to ingest — same types accepted as plain `cognee.add()` (text, file path, binary stream) |
| `external_metadata` | `dict` (optional) | Arbitrary key-value metadata stored alongside the ingested record |
| `label` | `str` (optional) | A short label attached to the data record |
| `data_id` | `UUID` (optional) | A stable ID to use instead of the auto-generated content-hash ID |
The `external_metadata` dictionary and `label` are stored on the `Data` record in Cognee's relational database. `external_metadata` does not automatically become graph structure; use `node_set` when you need tags that flow into the knowledge graph.
Arbitrary key-value metadata must be passed via `DataItem(external_metadata=...)`; it is not inferred automatically from plain strings or file paths passed directly to `add()`.
**Over HTTP**
`label` and `external_metadata` are not SDK-only. The `POST /api/v1/add` and `POST /api/v1/remember` endpoints accept `labels` and `external_metadata` multipart form fields — each a single JSON array whose entries pair positionally with the uploaded files — and map them onto the same `Data` record fields. See [how per-file labels and metadata work](/cognee-cloud/functionality/data-ingestion#how-per-file-labels-and-metadata-work) for the wire format and validation rules.
**Re-ingesting without a label**
Omitting the label means "leave unchanged", not "clear it". If you ingest a file with a label and later re-ingest the same file without one, the previously stored label is preserved. To change a label, re-ingest with the new value.
</Accordion>
<Accordion title="dataset_name grouping">
Separate collections of data into [named datasets](/core-concepts/further-concepts/datasets) to keep different knowledge domains apart:
```python theme={null}
await cognee.add("Legal contract text.", dataset_name="legal-docs")
await cognee.add("Product spec text.", dataset_name="product-specs")
```
</Accordion>
</AccordionGroup>
</Accordion>
<Accordion title="Troubleshooting 409 Conflict errors">
`POST /api/v1/add` returns **409 Conflict** whenever an unhandled exception occurs during the add operation. It is a catch-all — the actual problem is always in the `error` field of the response body:
```json theme={null}
{
"error": "<description of what went wrong>"
}
```
Read that message first. The table below maps the most common error patterns to their fixes.
| Symptom (error field contains…) | Cause | Fix |
| -------------------------------------------------------------------------------- | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `"API key"`, `"authentication"`, `"invalid_api_key"`, `"401"` | Missing or invalid LLM API key | Set `LLM_API_KEY` in your environment (`.env` file or shell). Even though `add` itself does not call the LLM, the database setup that runs on every request uses the configured provider. |
| `"connection refused"`, `"could not connect"`, `"timeout"`, `"OperationalError"` | Database unreachable | Verify your database service is running. For Docker setups, use `DB_HOST=host.docker.internal` instead of `localhost`. Check `DB_HOST`, `DB_PORT`, `DB_USERNAME`, `DB_PASSWORD`, and `DB_NAME`. |
| `"permission denied"`, `"not authorized"`, `"forbidden"`, `"403"` | The authenticated user lacks write access to the target dataset | Either use `datasetId` of a dataset you own, or disable access control with `ENABLE_BACKEND_ACCESS_CONTROL=False` in development. |
| `"decode"`, `"encoding"`, `"UnicodeDecodeError"`, `"failed to process"` | Corrupted or non-text file content | Confirm the file is readable and in a [supported format](/core-concepts/main-operations/legacy-operations/add#supported-formats). As a workaround, read the file yourself and pass the text string directly instead of a file path. |
| `"No such file or directory"`, `"FileNotFoundError"` | The file path does not exist on the server | Use an absolute path. If calling the HTTP API, upload the file as a multipart form attachment instead of passing a path string. |
| `"SSL"`, `"certificate"` | TLS/certificate issue connecting to an external database or S3 | Check SSL settings for your database or storage backend. Set `DB_SSL=false` in development if certificates are self-signed. |
### Enabling debug logs
For errors not covered above, enable verbose logging to see the full stack trace:
```bash theme={null}
LITELLM_LOG="DEBUG"
ENV="development"
```
Then re-run the request. The server logs will show exactly where the failure occurred.
### Still stuck?
* Check [Setup Configuration](/setup-configuration/overview) to verify your environment variables.
* Ask in the [Discord community](https://discord.gg/m63hxKsp4p) with the full `error` field from the 409 response.
* Open an issue on [GitHub](https://github.com/topoteretes/cognee/issues).
</Accordion>
<Columns>
<Card title="Cognify" icon="brain-cog" href="/core-concepts/main-operations/legacy-operations/cognify">
Expand data into chunks, embeddings, and graphs
</Card>
<Card title="DataPoints" icon="circle" href="/core-concepts/building-blocks/datapoints">
The units you'll see after Cognify
</Card>
<Card title="Building Blocks" icon="puzzle" href="/core-concepts/building-blocks/tasks">
Learn about Tasks and Pipelines behind Add
</Card>
</Columns>
# Cognify
Source: https://docs.cognee.ai/core-concepts/main-operations/legacy-operations/cognify
Transform ingested data into a knowledge graph.
<Note>
`cognify()` is a legacy operation. In Cognee v1.0, most users should use [remember()](/core-concepts/main-operations/remember) instead, which replaces the `add()` + `cognify()` + `memify()` workflow with a single call.
</Note>
## What is the cognify operation
The `.cognify` operation takes the ingested data with [Add](/core-concepts/main-operations/legacy-operations/add) and turns plain text into structured knowledge: chunks, embeddings, summaries, nodes, and edges that live in Cognee's vector and graph stores. It prepares your data for downstream operations like [Search](/core-concepts/main-operations/legacy-operations/search).
* **Transforms ingested data**: builds chunks, embeddings, and summaries
* **Graph creation**: extracts entities and relationships to form a knowledge graph
* **Vector indexing**: makes everything searchable via embeddings
* **Dataset-scoped**: runs per dataset, respecting ownership and permissions
<Note>
`.cognify` can be run multiple times as the dataset grows, and Cognee will skip what's already processed. Read more about **Incremental loading** in **[Examples and details](#examples-and-details)**
</Note>
## What happens under the hood
The `.cognify` pipeline is made of six ordered [Tasks](/core-concepts/building-blocks/tasks), plus two optional tasks you can switch on. Each task takes the output of the previous one and moves your data closer to becoming a searchable knowledge graph.
1. **Classify documents** — wrap each ingested file as a `Document` object with metadata and optional node sets
2. **Check permissions** — enforce that you have write access to the target dataset
3. **Extract chunks** — split documents into smaller pieces (paragraphs, sections)
4. **Extract graph** — use LLMs to identify entities and relationships, inserting them into the graph DB
5. **Summarize text** — generate summaries for each chunk, stored as `TextSummary` [DataPoints](/core-concepts/building-blocks/datapoints)
6. **Add data points** — embed nodes and summaries, write them into the vector store, and update graph edges
Steps 4 and 5 are the part you can swap. `GRAPH_EXTRACTOR=gliner` — or `cognify(extractor="gliner")` — fills both with a local GLiNER2 model, making no LLM call for either; that run also plans a closed label schema up front, and step 6 still embeds what it writes. See [LLM-free extraction with GLiNER](/python-api/cognify#llm-free-extraction-with-gliner).
Two further tasks are appended only when you opt in:
7. **Record provenance** *(opt-in, off by default)* — append an audit-ledger entry for every document, chunk, entity, and relationship this run produced. Enable it with `PROVENANCE_TRACKING=true`; when it is off, the pipeline is exactly the tasks above. See [Provenance ledger](/python-api/cognify#provenance-ledger).
8. **Detect contradictions** *(opt-in, off by default)* — compare the facts this run touched against the facts already stored around them and record each conflict as a `contradicts` edge. Enable it with `CONTRADICTION_DETECTION=true`; when it is off, the pipeline is exactly the tasks above. See [Contradiction detection](/python-api/cognify#contradiction-detection).
The result is a fully searchable, structured knowledge graph connected to your data.
## After cognify finishes
When `.cognify` completes for a dataset:
* **DocumentChunks** exist in memory as the granular breakdown of your files
* **Summaries** are stored and indexed in the vector database for semantic search
* **Knowledge graph nodes and edges** are committed to the graph database
* **Dataset metadata** is updated with token counts and pipeline status
* **`contradicts` edges** may also be present if you enabled contradiction detection — each one records the two conflicting facts, the reason, and a confidence score
* Your dataset is now **query-ready**: you can run [Search](/core-concepts/main-operations/legacy-operations/search) or graph queries immediately
<Note>
Because `cognify()` calls the LLM for entity extraction and summarization, it can fail when the configured LLM provider (or LiteLLM proxy) reports that its token budget is exhausted. In that case it raises `LLMPaymentRequiredError`, which the API surfaces as **HTTP 402 (Payment Required)** with body `{"error": "Token budget exhausted", "detail": "..."}`. This error is **terminal** — Cognee does not retry budget-exhaustion failures — so treat a `402` as final for the request and prompt the user to top up their token budget rather than reissuing the call.
</Note>
## Examples and details
<Accordion title="Pipeline tasks (detailed)">
1. **Classify documents**
* Turns raw `Data` rows into `Document` objects
* Chooses the right document type (PDF, text, image, audio, etc.)
* Attaches metadata and optional node sets
2. **Check permissions**
* Verifies that the user has write access to the dataset
3. **Extract chunks**
* Splits documents into `DocumentChunk`s using a chunker
* You can customize the chunk size and strategy — see [Chunkers](/core-concepts/further-concepts/chunkers) for details
* Updates token counts in the relational DB
4. **Extract graph**
* Calls the LLM to extract entities and relationships
* Deduplicates nodes and edges, commits to the graph DB
5. **Summarize text**
* Generates concise summaries per chunk
* Stores them as `TextSummary` [DataPoints](/core-concepts/building-blocks/datapoints) for vector search
6. **Add data points**
* Converts summaries and other [DataPoints](/core-concepts/building-blocks/datapoints) into graph + vector nodes
* Embeds them in the vector store, persists in the graph DB
7. **Record provenance** *(opt-in — set `PROVENANCE_TRACKING=true`)*
* Runs right after **Add data points**, where node ids are persisted and stable, and before the contradiction check, so the ledger never depends on contradiction edges
* Appends document → chunk → entity → relationship lineage entries to the append-only `provenance_entries` table in the relational DB, one chained transaction per task invocation
* Every entry carries a SHA-256 checksum linking it to the previous one, so `verify_chain()` detects deletion, reordering, and single-field edits
* Fail-safe: it returns its input unchanged and swallows its own errors, so it can never break ingestion — which also means a failed batch is silently absent from the ledger
* Read entries back through `ProvenanceManager`, and note the migration requirement — see [Provenance ledger](/python-api/cognify#provenance-ledger)
8. **Detect contradictions** *(opt-in — set `CONTRADICTION_DETECTION=true`)*
* Runs last, so both the new facts and the already-stored ones are persisted and comparable
* Gathers the facts one hop from the entities this run touched (structural edges such as `contains` and `made_from` are skipped) and asks the LLM which pairs conflict
* Logs a warning per conflict and writes a `contradicts` edge carrying `first_fact`, `second_fact`, `reason`, and `confidence`
* Non-destructive and fail-safe: it only adds edges and swallows its own errors, so it can never break ingestion
* Tunable with `CONTRADICTION_CONFIDENCE_THRESHOLD` (default `0.5`) and `CONTRADICTION_MAX_FACTS` (default `500`) — see [Contradiction detection](/python-api/cognify#contradiction-detection)
</Accordion>
<Accordion title="Default extraction prompts">
Cognee ships with several built-in system prompts for entity and relationship extraction, stored in `cognee/infrastructure/llm/prompts/`. The active prompt is controlled by the `GRAPH_PROMPT_PATH` environment variable (default: `generate_graph_prompt.txt`).
| Prompt file | Use case | What it does |
| ---------------------------------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `generate_graph_prompt.txt` | Default balanced extraction | Extracts entities and relationships using the standard Cognee rules: basic node types, human-readable IDs, normalized dates, `snake_case` relationships, and coreference consistency. |
| `generate_graph_prompt_simple.txt` | Lightweight extraction | Uses a shorter, more compact rule set for straightforward graph extraction while keeping the same core conventions around node types, IDs, dates, and relationship naming. |
| `generate_graph_prompt_strict.txt` | Tighter schema control | Applies a more explicit prompt with named node categories, stronger relationship constraints, examples, and a strict instruction not to infer facts that are not present in the text. |
| `generate_graph_prompt_guided.txt` | More directed graph shaping | Adds guidance for edge direction, allows multi-word entity labels, and encourages logically implied facts when they improve graph clarity without repeating the same fact. |
To switch to a different built-in prompt, set the environment variable:
```bash theme={null}
GRAPH_PROMPT_PATH=generate_graph_prompt_strict.txt
```
Or configure it at runtime via `cognee.config`:
```python theme={null}
import cognee
cognee.config.llm_config.graph_prompt_path = "generate_graph_prompt_strict.txt"
```
<Note>
If you need to use a custom prompt, refer to our [Custom Prompts guide](/guides/custom-prompts)
</Note>
</Accordion>
<Accordion title="Datasets and permissions">
* Cognify always runs on a dataset
* You must have **write access** to the target dataset
* Permissions are enforced at pipeline start
* Each dataset maintains its own cognify status and token counts
</Accordion>
<Accordion title="Incremental loading and deduplication">
`incremental_loading=True` is the default on `cognee.add()`, `cognee.cognify()`, and [`update()`](/python-api/update), as is its companion `data_cache=True`. The per-item skip runs whenever **either** flag is on, so disabling deduplication means passing both as `False`. Failure behavior is controlled by an environment variable, not a function parameter — see **When a data item fails** below.
The two flags give you two layers of deduplication:
**Layer 1 — content-hash deduplication in `add()`**
Before `cognify()` runs, `add()` already deduplicates by content hash within the dataset. Re-adding unchanged content is skipped at ingestion time, while changed content hashes differently and becomes a **new record with its own id** — the previous record stays. To replace a document's content while keeping its id, use [`update()`](/python-api/update).
For the full behavior and scenario table, see [Hash-based deduplication](/core-concepts/main-operations/legacy-operations/add#hash-based-file-storage-deduplication-and-filename-collisions) on the Add page.
**Layer 2 — pipeline-status tracking in `cognify()`**
Before processing each data item, `cognify()` checks a `pipeline_status` field on the record. If the status for the `cognify_pipeline` in the current dataset is already `COMPLETED`, the item is skipped entirely — no LLM calls, no re-embedding, no graph writes.
| Scenario | What happens |
| ------------------------------------------------------------------- | -------------------------------------------------------------- |
| Same file added and cognified again (unchanged) | Skipped — `pipeline_status` is already `COMPLETED` |
| File content changes, then `add()` + `cognify()` | `pipeline_status` is reset by `add()`; `cognify()` reprocesses |
| New file added to an existing dataset, then `cognify()` | Only the new file is processed; existing ones are skipped |
| `incremental_loading=False, data_cache=False` passed to `cognify()` | All items are reprocessed regardless of previous status |
Common usage patterns:
<AccordionGroup>
<Accordion title="Appending new data to an existing dataset">
You can grow a dataset over time without reprocessing what's already there:
```python theme={null}
import cognee
# Initial load
await cognee.add("First document content", dataset_name="my_dataset")
await cognee.cognify(datasets=["my_dataset"])
# Later: add more data — existing items are skipped automatically
await cognee.add("Second document content", dataset_name="my_dataset")
await cognee.cognify(datasets=["my_dataset"]) # only processes the new document
```
</Accordion>
<Accordion title="Forcing a full reprocess">
To reprocess everything regardless of status, pass both skip flags as `False` — the skip runs whenever either one is on:
```python theme={null}
await cognee.cognify(datasets=["my_dataset"], incremental_loading=False, data_cache=False)
```
This bypasses the pipeline-status check but does not re-ingest files — use `cognee.datasets.empty_dataset()` first if you also need to clear the stored data.
</Accordion>
<Accordion title="Re-ingesting a source that keeps growing">
For a table, database, or CSV that gains rows between runs, two independent settings decide what happens. The [`write_disposition`](/integrations/dlt-integration#write-dispositions) you pass to dlt ingestion decides what the staged snapshot contains on a re-run; `incremental_loading` / `data_cache` decide whether Cognee looks at that snapshot at all. `write_disposition` alone is not enough: a plain re-run is skipped before the grown snapshot is ever compared (see [Re-Ingesting a Source](/integrations/dlt-integration#re-ingesting-a-source)), so turn off both skip flags on `add()` to force the ingestion layer to look:
```python theme={null}
await cognee.add(
orders_resource,
dataset_name="orders",
primary_key="id",
write_disposition="replace",
incremental_loading=False,
data_cache=False,
)
await cognee.cognify(datasets=["orders"]) # keep the default here
```
`cognify()` can keep its defaults: `add()` clears the record's `pipeline_status` whenever the content hash changed, so the grown source is reprocessed while every unchanged document in the dataset is still skipped. Note that `append` also disables orphan cleanup, so rows deleted upstream stay in the graph.
</Accordion>
<Accordion title="When a data item fails">
Deduplication never suppresses errors, and the behavior is not a function parameter: it is the `RAISE_INCREMENTAL_LOADING_ERRORS` environment variable, default `true`.
| Value | Behavior |
| ---------------- | ------------------------------------------------------------------------------------------- |
| `true` (default) | The first failing item aborts the run — the remaining items are not processed |
| `false` | The error is logged and the remaining items keep processing before the run is marked failed |
In both modes `add()` returns a `PipelineRunErrored` run info rather than raising. A foreground `cognify()` instead raises `CognifyFailedError`, carrying the root cause of the failing item; pass `raise_on_error=False` to get the errored run info back as before — see [Failed runs](/python-api/cognify#failed-runs). A failed `cognify()` run also rolls back its partial artifacts so the next run resumes cleanly — see [Crash recovery and stuck pipelines](/core-concepts/building-blocks/pipelines) for how the rollback is scoped; `add()` has no rollback step. Cases where deduplication cannot identify rows are logged as warnings rather than raised: duplicate primary keys within a dlt table are reported and the last row loaded wins for foreign-key targeting.
</Accordion>
</AccordionGroup>
</Accordion>
<Accordion title="Batching for faster processing">
Two batching parameters control how much work Cognee runs at once during ingestion and graph building:
For new workflows, prefer [remember()](/core-concepts/main-operations/remember). It is the current API and accepts these batching controls for permanent-memory ingestion. Use `add()` and `cognify()` directly only when you need lower-level control over ingestion and graph building as separate legacy steps.
| Parameter | Applies to | What it controls | Default |
| ------------------ | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------- |
| `data_per_batch` | `remember()` permanent memory; legacy `add()` / `cognify()` | Maximum number of data items processed concurrently within one dataset pipeline run | `20` |
| `chunks_per_batch` | `remember()` permanent memory; legacy `cognify()` | Number of chunks emitted to each chunk-level Cognify task batch, including graph extraction/summarization and data-point persistence | Default Cognify: `2000`; temporal Cognify: `10`; `remember` HTTP endpoint form default: `36` |
`data_per_batch` is the outer concurrency limit. Cognee schedules the data items in a dataset and uses a semaphore so at most this many items are processed at the same time. The default of `20` is a deliberate concurrency cap — pushing many more items through the pipeline at once overwhelms it regardless of backend. On the default SQLite backend there is an additional, sharper constraint to know about before raising it: every in-flight item writes its own `Data` row, and at high concurrency the parallel read-then-write transactions hit WAL snapshot-upgrade conflicts that fail immediately with `database is locked`, bypassing `busy_timeout`. Deployments on Postgres are not subject to that particular failure. Raise `data_per_batch` explicitly when you want more ingestion concurrency, and lower it if your deployment is memory-constrained or your model provider's rate limits are tight.
`chunks_per_batch` is the inner chunk-task batch size. In the default Cognify pipeline, Cognee passes it as `batch_size` to the graph extraction/summarization task and to `add_data_points`. If you do not pass it directly, Cognee checks the `chunks_per_batch` value from `CognifyConfig`; when that is unset, the default Cognify pipeline uses `2000`.
**Where to configure them**
* Pass `data_per_batch` and `chunks_per_batch` to permanent-memory `remember()` for the current API path.
* Use legacy `add()` / `cognify()` only when you intentionally split ingestion and graph building; `data_per_batch` applies to both, while `chunks_per_batch` applies to `cognify()`.
* For the legacy `/api/v1/cognify` endpoint, both values are accepted in the JSON request body.
* For `/api/v1/remember`, `chunks_per_batch` is exposed as a multipart form field. The current remember endpoint does not expose `data_per_batch` as a form field, so tune `data_per_batch` through the Python SDK or the lower-level Cognify API when you need that control.
**Tuning guidance**
| Scenario | Suggested starting point |
| -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Small local datasets | Keep the defaults. Lower values rarely help unless you are debugging provider limits. |
| Large datasets on a larger machine | Raise `data_per_batch` above the default to get more ingestion concurrency — safest on Postgres, since on SQLite high ingestion concurrency risks immediate `database is locked` failures. Increase `chunks_per_batch` gradually, for example to `100` or higher, while watching memory, database load, and provider rate limits. |
| Memory-constrained environments | Lower both values, for example `data_per_batch=2` to `5` and `chunks_per_batch=10` to `25`, to reduce concurrent model calls and in-memory intermediate results. |
| Faster ingestion with many independent files | Increase `data_per_batch` first, because it controls how many data items can move through the pipeline concurrently. |
| Faster graph extraction for long documents | Tune `chunks_per_batch`, because it controls chunk-level batches after documents are split. |
Larger batches can improve throughput by keeping the pipeline, model provider, and databases busier, but they also increase memory pressure and can hit LLM, embedding, or database rate limits sooner. The best values depend on document size, chunk count, model/provider limits, embedding batch behavior, graph/vector database capacity, and the CPU/RAM available to the Cognee process.
</Accordion>
<Accordion title="How entity and relationship names are determined">
During the **Extract graph** step, Cognee asks the LLM to turn each chunk into graph nodes and edges. The names and types in that graph are inferred from your content rather than fixed in advance.
| Element | Fields |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Node (vertex)** | `id` - graph-local handle the model assigns, used only to wire up that chunk's edges; `name` - human-readable label; `type` - semantic category such as `Person` or `Organization`; `description` - short summary |
| **Edge (relationship)** | `source_node_id`, `target_node_id`, `relationship_name` - a free-text verb phrase such as `works_at` or `produces` |
The extraction prompt instructs the model to:
* Capture entities, names, nouns, and implied mentions exhaustively
* Form relationships as `(start_node, relationship_name, end_node)` triplets using explicit and inferred connections
* Avoid duplicates and overly generic terms
That means the resulting graph schema emerges from the data you ingest. Different datasets, prompts, or LLMs can produce slightly different node types and relationship names for similar content.
**How extracted nodes become graph nodes.** The `id` the model returns is only a local handle, used to wire up that chunk's edges. The identity of the stored node is derived from its `name` (normalized to lowercase, with spaces turned into underscores and apostrophes stripped), so every mention of the same name — across chunks, across documents, and across later `.cognify` runs — resolves to the same `Entity` node instead of creating a duplicate.
Three consequences worth knowing:
* If a single extracted chunk graph contains several distinct nodes that share one name, they are **not** merged: each gets a deterministic chunk-scoped id so they stay separate.
* An extracted edge whose endpoint is not among that chunk's extracted nodes is dropped, rather than creating a placeholder node for the missing endpoint.
* Entity nodes written before Cognee moved to name-derived ids keep their old ids, so re-running `.cognify` over data that is already in the graph can create a second node alongside the existing one. Re-process those datasets from scratch if you need the ids to line up.
<Note>
If you need tighter control over naming, use an OWL ontology or a custom graph model. See [Ontologies](/core-concepts/further-concepts/ontologies) and [Custom Graph Model](/guides/custom-graph-model).
</Note>
</Accordion>
<Accordion title="Inspect extracted graph schema">
Once `.cognify` finishes, the graph schema is inspectable because the extracted node types and relationship names now exist in the graph store.
### Python SDK
Use the graph engine directly to inspect the stored nodes and edges:
```python theme={null}
from cognee.infrastructure.databases.graph import get_graph_engine
graph_engine = await get_graph_engine()
# Returns all nodes and edges
nodes, edges = await graph_engine.get_graph_data()
# Inspect unique node types
node_types = {props.get("type") for _, props in nodes if props.get("type")}
print("Node types:", node_types)
# Inspect unique relationship names
relationship_names = {rel_name for _, _, rel_name, _ in edges}
print("Relationship names:", relationship_names)
```
`get_graph_data()` returns:
* **Nodes** as `(node_id: str, properties: dict)`
* **Edges** as `(source_id: str, target_id: str, relationship_name: str, properties: dict)`
If you only need aggregate information, inspect graph metrics instead:
```python theme={null}
metrics = await graph_engine.get_graph_metrics()
# Returns: num_nodes, num_edges, mean_degree, edge_density,
# num_connected_components (int), and
# sizes_of_connected_components (list[int])
metrics = await graph_engine.get_graph_metrics(include_optional=True)
print(metrics)
```
### HTTP server mode
When you run the Cognee HTTP server, you can inspect graph data through the dataset graph endpoint:
```text theme={null}
GET /api/v1/datasets/{dataset_id}/graph
```
To explore the same graph visually, use the [Graph Visualization guide](/guides/graph-visualization).
</Accordion>
<Accordion title="Re-cognify after schema changes">
If you update your data model (e.g., add new entity fields or relationships) and want to reprocess existing data:
1. **Delete the dataset** first, then re-add and re-cognify:
```python theme={null}
# Clear existing processed data
await cognee.datasets.empty_dataset(dataset_id=my_dataset.id)
# Re-add source files
await cognee.add(source_files, dataset_name="my_dataset")
# Re-cognify with the updated schema
await cognee.cognify()
```
2. **Alternatively, use [Memify](/core-concepts/main-operations/legacy-operations/memify)** for additive enrichment — it runs extraction and enrichment tasks over the existing graph without re-ingesting data. This is useful when you want to add new derived facts without reprocessing from scratch.
<Warning>
`.cognify` skips already-processed data by default. Simply re-running `.cognify` on unchanged files will not pick up schema changes. You must delete and re-add the data, or use memify for enrichment.
</Warning>
</Accordion>
<Accordion title="Final outcome">
* Vector database contains embeddings for summaries and nodes
* Graph database contains entities and relationships
* Relational database tracks token counts and pipeline run status
* Your dataset is now ready for [Search](/core-concepts/main-operations/legacy-operations/search) (semantic or graph-based)
</Accordion>
<Accordion title="Checking indexing status">
If you are using the current v1.0 API, see [Remember](/core-concepts/main-operations/remember#checking-indexing-status) for the recommended indexing-status workflow built around `remember()` and `recall()`.
If you are working directly with legacy `.cognify()` or the MCP `cognify_status` tool, the same dataset status primitives still apply:
* `cognee.datasets.get_status([dataset_id])`
* `GET /api/v1/datasets/status?dataset=<dataset-uuid>`
* `GET /api/v1/activity/pipeline-runs?dataset_id=<dataset-uuid>`
In MCP mode, `cognify_status(dataset_name="main_dataset")` provides a text summary of recent cognify runs.
</Accordion>
<Accordion title="LLM call count and cost estimation">
The default cognify pipeline makes **2 LLM calls per chunk**:
1. **Graph extraction** — identifies entities and relationships from the chunk text
2. **Summarization** — generates a concise summary of the chunk
**Estimating total calls**
The number of chunks depends on your document size and the configured `chunk_size`:
```
chunks = ceil(document_tokens / chunk_size)
total_llm_calls = chunks × 2
```
When `chunk_size` is not set explicitly, Cognee auto-calculates it as:
```
chunk_size = min(embedding_model_max_tokens, llm_max_tokens ÷ 2)
```
With typical defaults (e.g., `gpt-4o-mini` + `text-embedding-3-small`) this usually falls in the **1 024 – 8 192 token** range. See [Chunkers](/core-concepts/further-concepts/chunkers) for details.
**Example estimates** at `chunk_size = 1024`:
| Document size | Chunks | LLM calls |
| ------------- | ------ | --------- |
| 100 tokens | 1 | 2 |
| 1 000 tokens | 1 | 2 |
| 10 000 tokens | 10 | 20 |
**Tips for reducing API usage**
* **Increase `chunk_size`** — fewer, larger chunks mean fewer calls:
```python theme={null}
await cognee.cognify(chunk_size=4096)
```
* **Skip summarization** — use a [custom pipeline](/guides/custom-tasks-pipelines) that omits the `summarize_text` task, reducing calls to 1 per chunk.
* **Pace requests from the start** — Cognee already turns the RPM limiter on by itself once a provider reports overload (see [Rate Limiting](/setup-configuration/llm-providers)), but you can set `LLM_RATE_LIMIT_ENABLED=true` to pace every call from the first request and avoid bursting your provider quota when processing many chunks in parallel.
</Accordion>
<Accordion title="Concurrent search while cognify is running">
You can run [`search`](/core-concepts/main-operations/legacy-operations/search) while a `cognify` pipeline is active — there is no global lock that blocks one from the other.
Cognee's locks are **session-level**: they serialize short read-modify-write operations (such as `update_qa` or `add_feedback`) within the same `(session_id, operation)` pair. They do not apply across `cognify` and `search`.
**Within a single process**, the default LanceDB vector store uses an asyncio lock per adapter instance to serialize concurrent write coroutines, so interleaved searches and writes within the same worker are safe.
**Across multiple processes**, sharing the same LanceDB data directory is not supported — Cognee's embedded stores are single-process. If two workers do open the same directory, LanceDB's commit-conflict detection surfaces errors like:
```
CommitConflict: Too many concurrent writers
```
Seeing this error means two processes are writing to the same embedded vector store. Route all writes through one Cognee process, or switch to an external vector store such as PGVector — see the single-process callout in [remember](/core-concepts/main-operations/remember#examples-and-details).
<Note>
For single-process deployments (the default), concurrent search during cognify works without any special configuration.
</Note>
</Accordion>
<Columns>
<Card title="Add" icon="plus" href="/core-concepts/main-operations/legacy-operations/add">
First bring data into Cognee
</Card>
<Card title="Search" icon="search" href="/core-concepts/main-operations/legacy-operations/search">
Query embeddings or graph structures built by Cognify
</Card>
<Card title="Memify" icon="sparkles" href="/core-concepts/main-operations/legacy-operations/memify">
Enrich your graph with derived facts after cognify
</Card>
</Columns>
# Delete
Source: https://docs.cognee.ai/core-concepts/main-operations/legacy-operations/delete
Remove data from your knowledge graph.
<Note>
`delete`-style APIs are legacy operations. In Cognee v1.0, most users should use [forget()](/core-concepts/main-operations/forget) instead for item, dataset, and full-memory deletion. Low-level `prune` resets remain separate destructive maintenance tools.
</Note>
Cognee allows you to remove data at various levels of granularity: individual data items, entire datasets, or all datasets you have permission to delete.
## Overview
When you delete data in Cognee, the system performs a cleanup operation that removes:
1. **Graph Data:** Nodes and edges associated with the data item are removed from the graph database (e.g., Neo4j, FalkorDB).
2. **Vector Embeddings:** Embeddings generated for the data are removed from the vector store (e.g., Qdrant, LanceDB).
3. **Metadata:** Records in the relational database tracking the data item are deleted.
4. **Storage:** Raw files are removed only when Cognee owns the file and no other data item references it.
<Note>
**Shared Nodes and Relationship Metadata:** Cognee tracks graph nodes and relationship ownership metadata per source document. If a node, like the entity "New York", is shared by multiple data items, it is only removed when the *last* data item referencing it is deleted. Relationships attached to deleted nodes are removed with those nodes, while relationships between surviving shared nodes are not independently deleted by item-level deletion. This keeps the remaining graph intact.
</Note>
## Permissions
Deletion is a privileged operation. To delete data, you must have the `delete` permission on the dataset containing the data.
* If you created the dataset, you typically have this permission by default.
* In multi-user environments, administrators can grant or revoke this permission.
## Python SDK
The Python SDK is the primary way to manage data programmatically.
### Delete a Single Data Item
To delete a specific document or text entry, you need its `data_id` and the `dataset_id` it belongs to.
```python theme={null}
import cognee
# 1. Find the dataset and data ID
datasets = await cognee.datasets.list_datasets()
my_dataset = datasets[0]
data_items = await cognee.datasets.list_data(my_dataset.id)
item_to_delete = data_items[0]
# 2. Delete the item
await cognee.datasets.delete_data(
dataset_id=my_dataset.id,
data_id=item_to_delete.id
)
```
### Delete an Entire Dataset
Removing a dataset deletes all data items contained within it and the dataset container itself.
```python theme={null}
await cognee.datasets.empty_dataset(
dataset_id=my_dataset.id
)
```
### Delete All Data
You can wipe all datasets and data that your user has permission to delete.
```python theme={null}
await cognee.datasets.delete_all()
```
### Development Reset (`prune`)
For local development and testing, you might want to completely reset the system, including caches and system metadata.
<Warning>
**Destructive Operation:** `prune` commands bypass permission checks and wipe the underlying storage directly. Do not use in production.
</Warning>
Cognee stores data in two separate places that must be cleared independently:
| Layer | What it holds | How to clear |
| ---------------- | ---------------------------------------------------------------- | ------------------- |
| **File storage** | Raw uploaded files on disk or S3 | `prune_data()` |
| **Databases** | Graph nodes/edges, vector embeddings, relational metadata, cache | `prune_system(...)` |
```python theme={null}
import cognee
# Step 1 — wipe raw files from disk / S3 (DATA_ROOT_DIRECTORY)
await cognee.prune.prune_data()
# Step 2 — wipe all databases (graph, vector, relational metadata, cache)
await cognee.prune.prune_system(metadata=True)
```
You can also do a partial reset — for example, clear just the graph and vector
stores while keeping relational metadata and raw files intact:
```python theme={null}
# Clear only graph and vector stores (keep metadata and files)
await cognee.prune.prune_system(graph=True, vector=True, metadata=False, cache=False)
```
See the [`prune` API reference](/python-api/prune) for the full parameter list.
## CLI and HTTP API
For command-line usage and HTTP API endpoints, refer to their dedicated documentation pages:
<CardGroup>
<Card title="CLI Reference" icon="terminal" href="/cognee-cli/overview#delete-data">
Learn how to delete data using `cognee-cli delete`.
</Card>
<Card title="HTTP API Reference" icon="code" href="/api-reference/introduction#data-deletion">
See `DELETE` endpoints and `curl` examples.
</Card>
</CardGroup>
<Note>
`DELETE /api/v1/delete` is deprecated. Use the `datasets` endpoints instead.
</Note>
# Memify
Source: https://docs.cognee.ai/core-concepts/main-operations/legacy-operations/memify
Enrich a knowledge graph with derived knowledge.
<Note>
`memify()` is a legacy operation. In Cognee v1.0, most users should use [improve()](/core-concepts/main-operations/improve) instead for graph enrichment and session-to-graph bridging.
</Note>
## What is the memify operation
The `.memify` operation runs enrichment pipelines on an existing knowledge graph. It requires a graph built by [Add](/core-concepts/main-operations/legacy-operations/add) and [Cognify](/core-concepts/main-operations/legacy-operations/cognify) — it does not ingest raw data or build the graph from scratch.
Every memify pipeline is composed of two stages:
* **Extraction** — selects or prepares data from the existing graph. For example, pulling document chunks, loading graph triplets, or reading cached sessions.
* **Enrichment** — processes the extracted data and writes new or updated nodes and edges back to the graph. Depending on the pipeline, that can mean indexing triplet datapoints, deriving coding rules, or consolidating entity descriptions.
Memify chains extraction tasks and enrichment tasks into a single pipeline and runs them in sequence. When you call `await cognee.memify()` with no arguments, it runs the default pipeline. You can also call one of the other built-in pipelines directly, or supply your own custom tasks.
<Accordion title="Parameters (cognee.memify)">
* **`extraction_tasks`** (`List[Task | str]`, default: config-dependent) — tasks that select or prepare the data to process. By default, memify uses triplet-datapoint extraction when triplet embeddings are enabled; otherwise the default extraction stage can be empty.
* **`enrichment_tasks`** (`List[Task | str]`, default: `[Task(index_data_points, task_config={"batch_size": 100})]`) — tasks that create or update nodes and edges from the extracted data. When omitted, memify indexes the default extracted datapoints.
* **`data`** (`Any`, default: `None`) — input data forwarded to the first extraction task. When `None`, memify loads the graph (or a filtered subgraph) as input.
* **`dataset`** (`str` or `UUID`, default: `"main_dataset"`) — the dataset to process. The user must have write access.
* **`node_type`** (`Type`, default: `NodeSet`) — filter the graph to nodes of this type. Only used when `data` is `None`.
* **`node_name`** (`List[str]`, default: `None`) — filter the graph to nodes with these names. Only used when `data` is `None`.
* **`run_in_background`** (`bool`, default: `False`) — if `True`, memify starts processing and returns immediately. Use the returned `pipeline_run_id` to monitor progress.
Both task parameters accept `Task` instances, names of built-in tasks, or a mix of the two. See [Supported task names](/python-api/memify#supported-task-names) for the names memify resolves and the validation error raised for unknown ones.
</Accordion>
## Built-in pipelines
Cognee ships a default memify pipeline plus several convenience pipelines. The default pipeline runs when you call `cognee.memify()` with no arguments. Other helpers wrap `cognee.memify()` with their own task sets.
<Accordion title="Default enrichment (triplet datapoints)">
Runs when you call `await cognee.memify()` with no task arguments.
* **Extraction** (`get_triplet_datapoints`) — when triplet embeddings are enabled, reads graph triplets (source -> relationship -> target) and converts each to an indexable datapoint
* **Enrichment** (`index_data_points`) — indexes those datapoints in the vector DB
**Produces:** indexed triplet datapoints for triplet-style retrieval. This default behavior is driven by the current memify defaults in Cognee's codebase and is what [Improve](/core-concepts/main-operations/improve) uses under the hood for its enrichment stage.
<Note>
The default extraction stage is config-dependent. When triplet embeddings are disabled, memify does not run the triplet extraction task automatically.
</Note>
</Accordion>
<Accordion title="Triplet embeddings">
Calls `cognee.memify()` with triplet-specific tasks via `await create_triplet_embeddings(user, dataset)`.
* **Extraction** (`get_triplet_datapoints`) — reads graph triplets (source → relationship → target) and converts each to an embeddable text
* **Enrichment** (`index_data_points`) — indexes those texts in the vector DB under the `Triplet_text` collection
**Produces:** a searchable `Triplet_text` vector collection. Enables [`SearchType.TRIPLET_COMPLETION`](/core-concepts/main-operations/legacy-operations/search) queries.
Guide: [Triplet Embeddings Guide](/guides/memify-triplet-embeddings)
</Accordion>
<Accordion title="Coding rules (custom workflow)">
Coding-rule extraction still exists, but it is no longer the default memify pipeline.
* **Extraction** (`extract_subgraph_chunks`) — pulls document chunk texts from the existing graph
* **Enrichment** (`add_rule_associations`) — sends chunks to the LLM, which derives coding-rule associations
**Produces:** `Rule` nodes connected to source chunks via `rule_associated_from` edges, grouped under the `coding_agent_rules` [node set](/core-concepts/further-concepts/node-sets). Enables [`SearchType.CODING_RULES`](/core-concepts/main-operations/legacy-operations/search) queries.
Demo: [Mine Coding Rules From Team Chat](/examples/coding-rule-mining)
</Accordion>
<Accordion title="Session persistence">
Calls `cognee.memify()` with session-specific tasks via `await persist_sessions_in_knowledge_graph_pipeline(user, session_ids)`. Requires [caching to be enabled](/core-concepts/sessions-and-caching).
* **Extraction** (`extract_user_sessions`) — reads Q\&A data from the session cache for the specified session IDs
* **Enrichment** (`cognify_session`) — processes session data through `cognee.add` + `cognee.cognify`
**Produces:** new graph nodes from the session content, grouped under the `user_sessions_from_cache` [node set](/core-concepts/further-concepts/node-sets).
Guide: [Session Persistence Guide](/guides/memify-session-persistence)
</Accordion>
<Accordion title="Entity consolidation">
Calls `cognee.memify()` with entity-consolidation tasks via `await consolidate_entity_descriptions_pipeline()`. Useful when entity descriptions are fragmented or repetitive across chunks after [cognify](/core-concepts/main-operations/legacy-operations/cognify).
* **Extraction** (`get_entities_with_neighborhood`) — loads `Entity` nodes along with their edges and neighbors
* **Enrichment** (`generate_consolidated_entities` → `add_data_points`) — sends each entity and its neighborhood to the LLM, which returns a refined description
**Produces:** updated `Entity` descriptions written back in place — no new nodes are created.
Guide: [Entity Consolidation Guide](/guides/memify-entity-consolidation)
</Accordion>
<Accordion title="Cross-connect entities">
Calls `cognee.memify()` with link-prediction tasks via `await cross_connect_entities_pipeline()`, imported from `cognee.memify_pipelines.cross_connect_entities`. Proposes *new* edges between `Entity` nodes that look related but are not yet connected — it links, it never merges.
* **Extraction** (`get_entity_nodes`) — loads every `Entity` node and its properties from the graph
* **Enrichment** (`cross_connect_entities`) — pairs up candidate entities, asks the LLM to name the relationship between each pair, and writes the confident ones back as edges
**Produces:** new edges between existing `Entity` nodes. Each written edge carries `inferred: True`, the LLM's `confidence`, and `feedback_weight: 0.2`, so inferred links are distinguishable from extraction-time relationships and start with a low weight, letting future cleanup prune them first if they turn out to be noise. No nodes are created, rewritten, or deleted.
<Warning>
`dry_run` defaults to **`False`**, so calling `await cross_connect_entities_pipeline()` with no arguments writes inferred edges to your graph. Pass `dry_run=True` for a preview run.
A dry run still calls the LLM once per surviving candidate pair — inference happens *before* the write branch. `dry_run=True` saves the graph writes, not the token cost.
</Warning>
**Parameters**
| Argument | Type | Default | Description |
| ------------------------ | ------- | ------- | --------------------------------------------------------------------------------------- |
| `similarity_threshold` | `float` | `0.5` | Minimum vector similarity for a nearest-neighbor pair to become a candidate. |
| `overlap_threshold` | `float` | `0.2` | Minimum IDF-weighted neighbor overlap for a shared-neighbor pair to become a candidate. |
| `max_new_edges_per_node` | `int` | `5` | Cap on how many new edges a single entity may gain in one run. |
| `confidence_threshold` | `float` | `0.7` | Minimum LLM confidence for a proposed relationship to be kept. |
| `dry_run` | `bool` | `False` | When `True`, propose edges and log the count, but perform zero graph or index writes. |
Unlike `consolidate_entities_pipeline()`, this pipeline takes no `user`, `dataset`, or `run_in_background` arguments — it runs against the default dataset.
<Accordion title="Under the hood">
Candidates come from the union of two independent signals, so a pair only needs to clear one of them:
* **Shared-neighbor overlap** — entities are scored with an IDF-weighted Jaccard over their neighbor sets. Neighbors that every entity has (a shared `EntityType`, for example) carry zero IDF and are skipped as pivots, so they cannot pair up half the graph.
* **Vector nearest neighbors** — each entity name is searched against the `Entity_name` collection, with the returned cosine distance flipped to a similarity.
Pairs that are already linked in the graph are dropped **before** any LLM call, so re-runs do not re-pay for existing edges. Surviving pairs are labeled by the LLM, filtered by `confidence_threshold` and the per-node cap, and then — on a real run — passed through a `has_edge` guard that removes any edge that already exists before writing. Survivors are written with `add_edges` and indexed with `index_graph_edges`.
The enrichment task returns `{"proposed": [...], "written": N, "dry_run": bool}`, where `written` is always `0` on a dry run.
Both tasks are SDK-only: they are not in the memify task-name registry, so they cannot be passed as strings. Import them from `cognee.tasks.memify.cross_connect_entities` if you want to build the tasks yourself.
</Accordion>
<Note>
The vector signal is optional. If the `Entity_name` collection does not exist — for example, no embedding provider has indexed entity names yet — vector candidates are skipped silently and only shared-neighbor overlap contributes candidates.
</Note>
</Accordion>
<Accordion title="cognify vs memify: which to use">
| | `cognify()` | `memify()` |
| ----------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------ |
| **Purpose** | Build a knowledge graph from raw data | Enrich an existing knowledge graph |
| **Requires** | Data added via [`add()`](/core-concepts/main-operations/legacy-operations/add) | A graph already built by [`cognify()`](/core-concepts/main-operations/legacy-operations/cognify) |
| **Pipeline** | Fixed: classify → chunk → extract → summarize → store | Customizable extraction + enrichment tasks |
| **Pipeline cache** | `True` — skips already-processed data by default | `False` — always re-processes |
| **Incremental loading** | `True` by default | Not applicable |
| **When to use** | Initial graph construction or adding new documents | Derive new facts, index triplets, consolidate entities |
**Use `cognify`** when you have new raw documents to turn into a knowledge graph.
**Use `memify`** when you already have a graph and want to enrich it — for example, to index [triplet embeddings](/guides/memify-triplet-embeddings), extract coding rules, or [consolidate entity descriptions](/guides/memify-entity-consolidation) — without re-ingesting source data.
</Accordion>
<Columns>
<Card title="Cognify" icon="brain-cog" href="/core-concepts/main-operations/legacy-operations/cognify">
Build the knowledge graph that memify enriches
</Card>
<Card title="Memify Quickstart" icon="brain" href="/guides/memify-quickstart">
Run the default memify pipeline step by step
</Card>
<Card title="Search" icon="search" href="/core-concepts/main-operations/legacy-operations/search">
Query the enriched graph with specialized search types
</Card>
</Columns>
# Search
Source: https://docs.cognee.ai/core-concepts/main-operations/legacy-operations/search
Query your AI memory with vectors, graphs, and LLMs.
<Note>
`search()` is a legacy operation. In Cognee v1.0, most users should use [recall()](/core-concepts/main-operations/recall) instead. Legacy `search()` remains useful when you need explicit control over retrievers and lower-level retrieval parameters.
</Note>
## What is search
`search` lets you ask questions over everything you've ingested and cognified.\
Under the hood, Cognee blends **vector similarity**, **graph structure**, and **LLM reasoning** to return answers with context and provenance.
## The big picture
* **Dataset-aware**: searches run against one or more datasets you can read *(requires `ENABLE_BACKEND_ACCESS_CONTROL=true`)*
* **Multiple modes**: from simple chunk lookup to graph-aware Q\&A
* **Hybrid retrieval**: vectors find relevant pieces; graphs provide structure; LLMs compose answers
* **Conversational memory**: for completion search types — GRAPH\_COMPLETION and its variants, RAG\_COMPLETION, HYBRID\_COMPLETION, TRIPLET\_COMPLETION, TEMPORAL, and AGENTIC\_COMPLETION — use `session_id` to maintain conversation history across searches *(requires caching enabled)*. Omitting `session_id` still stores history, under the dataset-scoped [default session](/core-concepts/sessions-and-caching#what-is-a-session). Retrieval-only search types do not use session history.
* **Safe by default**: permissions are checked before any retrieval
* **Observability**: telemetry is emitted for query start/completion
<Warning>
**Dataset scoping** requires specific configuration. See [permissions system](/core-concepts/multi-user-mode/permissions-system/datasets#dataset-isolation-how-access-is-enforced) for details on access control requirements and supported database setups.
</Warning>
## Where search fits
Use `search` after you've run `.add` and `.cognify`.
At that point, your dataset has chunks, summaries, embeddings, and a knowledge graph—so queries can leverage both **similarity** and **structure**.
<Warning>
If you see `DatabaseNotCreatedError: please call await setup() first`, Cognee has not been initialized yet. In practice, that usually means you are trying to search or use an operation that resolves the default user before initialization has happened. Run `cognee.add(...)` or `cognee.remember(...)` first, then retry your operation.
</Warning>
<Note>
Completion-type search modes (`GRAPH_COMPLETION`, `RAG_COMPLETION`, `TRIPLET_COMPLETION`, and the other `*_COMPLETION` types) call the LLM to compose an answer, so they can fail when the configured LLM provider (or LiteLLM proxy) reports that its token budget is exhausted. In that case `search()` raises `LLMPaymentRequiredError`, which the API surfaces as **HTTP 402 (Payment Required)** with body `{"detail": "LLM provider requires payment or token budget is exhausted. [LLMPaymentRequiredError]"}` — or, when the provider's own budget sentence can be identified, `{"detail": "LLM budget exhausted: <provider sentence> [LLMPaymentRequiredError]"}` (see [402 Payment Required](/api-reference/introduction)). This error is **terminal** — Cognee does not retry budget-exhaustion failures — so treat a `402` as final for the request and prompt the user to top up their token budget rather than reissuing the call.
</Note>
## How it works (conceptually)
1. **Scope & permissions**\
Resolve target datasets (by name or id) and enforce read access.
For most search types, this scope can contain multiple datasets. `AGENTIC_COMPLETION` is stricter and requires the resolved scope to contain exactly one dataset.
2. **Mode dispatch**\
Pick a search mode (default: **graph-aware completion**) and route to its retriever.
3. **Retrieve → (optional) generate**\
Collect context via vectors and/or graph traversal; some modes then ask an LLM to compose a final answer.
4. **Return results**\
Depending on mode: a natural-language answer string, chunk/summary dicts with metadata, graph records, or Cypher results.
For a practical guide to using search with examples and detailed parameter explanations, see [Search Basics](/guides/search-basics).
## Retrievers
Each search type is handled by a **retriever**. The pipeline is: `get_retrieved_objects` → `get_context_from_objects` → `get_completion_from_context` (skipped when `only_context=True`).
| Search type | Retriever |
| ------------------------------------- | ---------------------------------------- |
| GRAPH\_COMPLETION | GraphCompletionRetriever |
| RAG\_COMPLETION | CompletionRetriever |
| HYBRID\_COMPLETION | HybridRetriever |
| CHUNKS | ChunksRetriever |
| SUMMARIES | SummariesRetriever |
| GRAPH\_SUMMARY\_COMPLETION | GraphSummaryCompletionRetriever |
| GRAPH\_COMPLETION\_COT | GraphCompletionCotRetriever |
| GRAPH\_COMPLETION\_CONTEXT\_EXTENSION | GraphCompletionContextExtensionRetriever |
| TRIPLET\_COMPLETION | TripletRetriever |
| CHUNKS\_LEXICAL | BM25ChunksRetriever |
| CODING\_RULES | CodingRulesRetriever |
| CODE | CodeRetriever |
| TEMPORAL | TemporalRetriever |
| CYPHER | CypherSearchRetriever |
| NATURAL\_LANGUAGE | NaturalLanguageRetriever |
| SKILLS | SkillsRetriever |
You can register a custom retriever for a search type via `use_retriever(SearchType, RetrieverClass)`; the class must implement the same three-step interface (`BaseRetriever`). See the API reference for `BaseRetriever` and `register_retriever`.
### Multi-query (batch)
**GraphCompletionRetriever**, **GraphCompletionCotRetriever**, and **GraphCompletionContextExtensionRetriever** support **batch mode**: pass `query_batch` (a non-empty list of strings) instead of `query`. You get one result per query; session cache is not used in batch mode. The public `cognee.search()` API accepts only a single `query_text`; batch is available when you use the retrievers directly (e.g. in custom pipelines).
<Accordion title="GRAPH_COMPLETION">
Graph-aware question answering.
* **What it does**: Finds relevant graph triplets using vector hints across indexed fields, resolves them into readable context, and asks an LLM to answer your question grounded in that context.
* **Why it’s useful**: Combines fuzzy matching (vectors) with precise structure (graph) so answers reflect relationships, not just nearby text.
* **Output**: `str` by default. With `only_context=True`, returns the formatted context string instead of an answer. With `verbose=True`, returns a payload containing `text_result`, `context_result`, and `objects_result`. With `ENABLE_BACKEND_ACCESS_CONTROL=true`, results are wrapped with `dataset_name` and `dataset_id`.
**The concrete lookup process**
One `GRAPH_COMPLETION` call runs five stages (`GraphCompletionRetriever` → `brute_force_triplet_search`):
1. **Vector seeds** — the query is embedded once, then searched concurrently across every vector index collection registered by the loaded DataPoint models — typically `Entity_name`, `EntityType_name`, `TextSummary_text`, `DocumentChunk_text` — plus the edge collection `EdgeType_relationship_name`. Each collection returns at most `wide_search_top_k` (default `100`) scored hits; the node hits' ids become the traversal seeds, while edge hits are kept for scoring in the next steps.
2. **Graph traversal** — the seeds are resolved against the graph store into an in-memory subgraph ("memory fragment") containing every edge with at least one endpoint among the seed ids, i.e. a 1-hop expansion around the seeds, with the properties `id`, `name`, `type`, `description`, `text`, and `importance_weight` projected per node. Graph adapters without ID-filtered projection fall back to projecting the whole graph. The cosine distances from step 1 are then mapped onto the projected nodes and edges; anything with no vector hit keeps the fallback distance `triplet_distance_penalty` (default `6.5`), which ranks it below every real match.
3. **Triplet ranking** — each projected edge is scored as a triplet (`node1`, `edge`, `node2`) by summing the three elements' distances, each scaled by `(2 - importance_weight)` and, when `feedback_influence > 0`, blended with the element's `feedback_weight`. The `top_k` **lowest-scoring** (closest) triplets are kept.
4. **Context assembly** — the kept triplets are rendered into one string: a `Nodes:` section listing each distinct endpoint once with its content (a chunk/summary node contributes its `text`, an entity its `description`), followed by a `Connections:` section of `source --[relationship]--> target` lines. `include_global_context_index=True` prepends a [global-context](/core-concepts/further-concepts/global-context-index) prelude. With `only_context=True` this string is returned and the search stops here.
5. **LLM completion** — the question and context are rendered into the `graph_context_for_question.txt` user prompt and sent with the `answer_simple_question.txt` system prompt. With [sessions](/core-concepts/sessions-and-caching) enabled, conversation history is prepended to the system prompt; with `include_references=True` an `Evidence:` block is appended to the answer.
**How `top_k` affects it**
`top_k` (default `15` through `recall()` and `search()`; `5` when you construct `GraphCompletionRetriever` yourself) is the number of **triplets** kept in step 3 — not nodes, chunks, or characters. So it bounds the context at `top_k` connection lines and at most `2 × top_k` distinct nodes; raising it deepens recall and grows prompt tokens roughly linearly, lowering it tightens the answer. It must be a positive integer, or `None` to use the retriever's own default; non-positive values raise `QueryValidationError`.
`top_k` only selects from what steps 1–2 surfaced, so on large graphs also raise `wide_search_top_k` to widen the candidate pool. Two variants change step 2: setting `neighborhood_depth` replaces the 1-hop projection with a *k*-hop neighborhood grown from up to `neighborhood_seed_top_k` of the vector-hit seeds (all seeds through `search()` and `recall()`, which default it to `None`; `10` when you construct the retriever yourself — the capped subset is not distance-ranked), and passing `node_name` drops the per-collection cap (vector hits are node-set filtered instead) without narrowing the projection by seed ids. The nodes a *k*-hop expansion adds beyond the seeds had no vector hit in step 1; on LanceDB, PGVector, and Turso, Cognee scores them with a second id-filtered vector query so they rank on their real distance, and on adapters without that capability they keep the `triplet_distance_penalty` fallback.
</Accordion>
<Accordion title="RAG_COMPLETION">
Retrieve-then-generate over text chunks.
* **What it does**: Pulls top-k chunks via vector search, stitches a context window, then asks an LLM to answer.
* **When to use**: You want fast, text-only RAG without graph structure.
* **Scope**: Honors `node_name` and `node_name_filter_operator`, so chunk retrieval can be limited to specific node sets.
* **Output**: `str` by default. With `only_context=True`, returns the formatted context string instead of an answer. With `verbose=True`, returns a payload containing `text_result`, `context_result`, and `objects_result`. With `ENABLE_BACKEND_ACCESS_CONTROL=true`, results are wrapped with `dataset_name` and `dataset_id`.
</Accordion>
<Accordion title="HYBRID_COMPLETION (default)">
Blended chunk + entity retrieval with LLM completion.
* **What it does**: Builds a single context from three channels — BM25 lexical chunks, semantic (vector) chunks, and entity/graph context (matched entities plus their connected edges) — then asks an LLM to answer grounded in that combined context. The lexical and vector chunks are merged and de-duplicated up to `chunks_top_k`. Each entity's edge bullets are ordered by how relevant the edge text is to the query (entity-type edges are pinned first, then query-ranked edges, then remaining edges in graph order). The combined context also ends with a `## Related facts` section: up to `facts_top_k` edge-derived facts ranked by similarity to the query, excluding facts already shown as entity edge bullets. Facts whose text comes from chunk→entity "contains" edges are rendered as `Name: description` glossary-style lines.
* **When to use**: You want both keyword precision (BM25) and semantic recall (vectors) alongside entity relationships in one answer, without choosing between lexical and vector chunk search.
* **Output**: `str` by default. With `only_context=True`, returns the formatted context string instead of an answer. With `verbose=True`, returns a payload containing `text_result`, `context_result`, and `objects_result`. With `ENABLE_BACKEND_ACCESS_CONTROL=true`, results are wrapped with `dataset_name` and `dataset_id`.
* **Cost**: Issues both a BM25 lexical chunk lookup and a semantic vector-store query per search, so expect slightly higher query load than a single-channel mode.
* **Limitation**: `query_batch` cannot be combined with the session cache.
**Configuration (`retriever_specific_config`)**
| Key | Default | What it does |
| ------------------------------ | ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `chunks_top_k` | `min(top_k, 10)` (`10`) | Max merged chunks (BM25 + vector) included in context. |
| `entities_top_k` | `min(top_k, 10)` (`10`) | Number of entities retrieved for entity/graph context. |
| `max_edges_per_entity` | `10` | Max connected edges listed per entity. |
| `facts_top_k` | `min(top_k, 10)` (`10`) | Max edge-derived facts included in the `## Related facts` section. Set to `0` to disable the section. The fact budget is tied to the entity lane, so with `entities_top_k` at `0` no facts are returned. On `node_name`-scoped searches, only facts expressed by an edge on an in-scope entity are kept. |
| `include_global_context_index` | `false` | When `true`, prepends a global-context section (root text + top global-context summaries) to the answer context. |
| `global_context_index_top_k` | `3` | Number of global-context summaries to include when `include_global_context_index=true`. |
Pass these through `retriever_specific_config`.
</Accordion>
<Accordion title="CHUNKS">
Direct chunk retrieval.
* **What it does**: Returns the most similar text chunks to your query via vector search.
* **When to use**: You want raw passages/snippets to display or post-process.
* **Output**: `list[dict]`. Each chunk item includes `id`, `text`, `chunk_index`, `chunk_size`, and `cut_type`. With `ENABLE_BACKEND_ACCESS_CONTROL=true`, results are wrapped with `dataset_name` and `dataset_id`. See [Citation and Source Tracking](/guides/search-basics#citation-and-source-tracking) for details and examples.
</Accordion>
<Accordion title="SUMMARIES">
Search over precomputed summaries.
* **What it does**: Vector search on `TextSummary` content for concise, high-signal hits.
* **When to use**: You prefer short summaries instead of full chunks.
* **Output**: `list[dict]`. Each summary item includes `id` and `text`. With `ENABLE_BACKEND_ACCESS_CONTROL=true`, results are wrapped with `dataset_name` and `dataset_id`. See [Citation and Source Tracking](/guides/search-basics#citation-and-source-tracking) for details and examples.
</Accordion>
<Accordion title="GRAPH_SUMMARY_COMPLETION">
Graph-aware summary answering.
* **What it does**: Builds graph context like GRAPH\_COMPLETION, then condenses it before answering.
* **When to use**: You want a tighter, summary-first response.
* **Output**: `str` by default. With `only_context=True`, returns the formatted context string instead of an answer. With `verbose=True`, returns a payload containing `text_result`, `context_result`, and `objects_result`. With `ENABLE_BACKEND_ACCESS_CONTROL=true`, results are wrapped with `dataset_name` and `dataset_id`.
</Accordion>
<Accordion title="GRAPH_COMPLETION_COT">
Chain-of-thought over the graph.
* **What it does**: Starts from `GRAPH_COMPLETION` (retrieve triplets, draft an answer), then runs `max_iter` reasoning rounds. Each round validates the current answer, generates a follow-up question, fetches new triplets for it, merges them into the context, and regenerates the answer.
* **When to use**: Complex multi-hop questions where the answer depends on traversing several relationships (for example `A → B → C → D`). Each round can pull in one additional hop's worth of triplets, so deeper chains need more iterations.
* **Output**: `str` by default. With `only_context=True`, returns the formatted context string instead of an answer. With `verbose=True`, returns a payload containing `text_result`, `context_result`, and `objects_result`. With `ENABLE_BACKEND_ACCESS_CONTROL=true`, results are wrapped with `dataset_name` and `dataset_id`.
**Controlling depth with `max_iter`**
`max_iter` (default `4`) is the number of follow-up rounds after the initial retrieval. Higher values let the retriever traverse further from the seed entities but multiply LLM calls — each round adds a validation call, a follow-up-generation call, a new triplet fetch, and a completion call. Latency and token cost scale roughly linearly with `max_iter`.
Pass it through `retriever_specific_config`.
</Accordion>
<Accordion title="GRAPH_COMPLETION_CONTEXT_EXTENSION">
Iterative context expansion.
* **What it does**: Starts like `GRAPH_COMPLETION`: vector similarity seeds relevant triplets, then graph retrieval resolves them into context. It then runs up to `context_extension_rounds` extension rounds (default `4`), using each generated answer as a follow-up query to fetch and merge additional triplets until no new triplets are found.
* **When to use**: Open-ended or exploratory queries that need broader subgraph coverage than a single `GRAPH_COMPLETION` pass.
* **Output**: `str` by default. With `only_context=True`, returns the formatted context string instead of an answer. With `verbose=True`, returns a payload containing `text_result`, `context_result`, and `objects_result`. With `ENABLE_BACKEND_ACCESS_CONTROL=true`, results are wrapped with `dataset_name` and `dataset_id`.
</Accordion>
<Accordion title="NATURAL_LANGUAGE">
Natural language to Cypher to execution.
* **What it does**: Infers a Cypher query from your question using the graph schema, runs it, returns the results.
* **When to use**: You want structured graph answers without writing Cypher.
* **Output**: `list[dict[str, Any]]` containing executed graph-query result rows. With `ENABLE_BACKEND_ACCESS_CONTROL=true`, results are wrapped with `dataset_name` and `dataset_id`.
* **Availability**: Disabled when `ALLOW_CYPHER_QUERY=false`.
</Accordion>
<Accordion title="CYPHER">
Run Cypher directly.
* **What it does**: Executes your Cypher query against the graph database.
* **When to use**: You know the schema and want full control.
* **Output**: `list[dict[str, Any]]` containing raw Cypher result rows. With `ENABLE_BACKEND_ACCESS_CONTROL=true`, results are wrapped with `dataset_name` and `dataset_id`.
* **Availability**: Disabled when `ALLOW_CYPHER_QUERY=false`.
</Accordion>
<Accordion title="CODING_RULES">
Code-focused retrieval (coding rules / codebase search).
* **What it does**: Retrieves rules or code context from the `coding_agent_rules` nodeset and returns structured code information.
* **When to use**: Codebases or coding guidelines indexed by Cognee (e.g. via memify).
* **Output**: `list[dict[str, Any]]` containing structured code/rule retrieval objects. Exact fields depend on the retrieved rule/context objects. With `ENABLE_BACKEND_ACCESS_CONTROL=true`, results are wrapped with `dataset_name` and `dataset_id`.
* **Prereq**: The `coding_agent_rules` nodeset must be populated (e.g. via the [coding-rule mining demo](/examples/coding-rule-mining), which runs memify with the rule-extraction tasks).
</Accordion>
<Accordion title="CODE">
Deterministic queries over a code graph.
* **What it does**: Loads the dataset's code-graph facts (modules, symbols, routes, storage, services, dependencies, insights, …) into an in-memory snapshot and runs exactly one graph operation on it — list facts, explore a neighborhood, traverse or find a path between facts, analyse impact, read explainer insights, draw the module-level architecture, or report what the last ingestion changed. It uses only the graph adapter: no LLM, embedding, or vector call, and no session-turn step.
* **When to use**: Repositories indexed by the [code-graph pipeline](/guides/code-graph) or the [GitHub integration](/integrations/github-integration), when you want exact structural answers rather than a generated one.
* **Input**: The operation and its arguments come from the `code_query` dict, not from `query_text` or `retriever_specific_config`. Every operation, its arguments, and the fact kinds it can filter on are listed under [SearchType — CODE](/python-api/search-type#per-search-type-parameters).
* **Output**: One `dict` per searched dataset, keyed by the operation — for example `facts` with `total`/`has_more` for `query_facts`, `nodes`/`edges`/`stats` for `explore` and `traverse`, `by_depth` and a `summary` for `impact_analysis`, and a `diagram` block when requested. With `ENABLE_BACKEND_ACCESS_CONTROL=true`, results are wrapped with `dataset_name` and `dataset_id`.
* **Prereq**: A code graph in the dataset. Graph paths exist only inside one dataset, so index the repositories you want to connect into the same dataset.
</Accordion>
<Accordion title="TRIPLET_COMPLETION">
Triple-based retrieval with LLM completion (no full graph traversal).
* **What it does**: Retrieves graph triplets by vector similarity, resolves them to text, and asks an LLM to answer.
* **When to use**: You want triplet-level context without full graph expansion.
* **Scope**: Honors `node_name` and `node_name_filter_operator`, so triplet retrieval can be limited to specific node sets.
* **Output**: `str` by default. With `only_context=True`, returns the formatted context string instead of an answer. With `verbose=True`, returns a payload containing `text_result`, `context_result`, and `objects_result`. With `ENABLE_BACKEND_ACCESS_CONTROL=true`, results are wrapped with `dataset_name` and `dataset_id`.
* **Prereq**: Triplet embeddings must exist—set `TRIPLET_EMBEDDING=true` before running [cognify](/core-concepts/main-operations/legacy-operations/cognify) or run the [`create_triplet_embeddings`](/guides/memify-triplet-embeddings) memify pipeline (retriever uses the `Triplet_text` collection).
</Accordion>
<Accordion title="CHUNKS_LEXICAL">
Lexical (keyword-style) chunk search.
* **What it does**: Returns chunks that match your query using token-based BM25 lexical ranking, not semantic embeddings.
* **When to use**: Exact-term or keyword-style lookups; stopword-aware search.
* **Output**: `list[dict]` containing ranked chunk payloads (chunk text and metadata). BM25 orders the results but does not attach a score to them. With `ENABLE_BACKEND_ACCESS_CONTROL=true`, results are wrapped with `dataset_name` and `dataset_id`.
</Accordion>
<Accordion title="TEMPORAL">
Time-aware retrieval.
* **What it does**: Retrieves and ranks content by temporal relevance (dates, events) and answers with time context.
* **When to use**: Queries about "before/after X", "in 2020", or event timelines.
* **Output**: `str` by default. With `only_context=True`, returns the formatted temporal context instead of an answer. With `ENABLE_BACKEND_ACCESS_CONTROL=true`, results are wrapped with `dataset_name` and `dataset_id`. See [Time-awareness](/guides/time-awareness) for setup.
</Accordion>
<Accordion title="FEELING_LUCKY">
Automatic mode selection.
* **What it does**: Uses an LLM to choose the most suitable search type for your query, then runs it. Falls back to `RAG_COMPLETION` if mode selection fails.
* **When to use**: Exploratory or one-off queries when you are unsure which mode fits best. For production or latency-sensitive workloads, prefer a specific search type.
* **Output**: Varies. Returns the same shape as whichever search type the router selects.
</Accordion>
<Accordion title="INSIGHTS (MCP only)">
Graph-edge result format specific to the Cognee MCP server.
* **What it does**: Runs search in the Cognee MCP server and formats the raw triplet results as readable relationship lines instead of asking an LLM to compose an answer.
* **When to use**: You want to inspect raw graph edges in MCP rather than receive a natural-language response.
* **Output**: A newline-separated list of relationship statements. `INSIGHTS` is MCP-only and is not available through the Python `SearchType` enum.
</Accordion>
<Info>
**Feedback** is handled via [Sessions](/core-concepts/sessions-and-caching) and the [Feedback System](/guides/feedback-system)—use `cognee.session.add_feedback` and `cognee.session.delete_feedback`. See the [Sessions Guide](/guides/sessions) and [Feedback System](/guides/feedback-system) for full details.
</Info>
## Further details
<AccordionGroup>
<Accordion title="Searching across both graph and vector stores">
You do **not** need to run two separate searches and merge the results — the graph-completion family already queries both stores in a single call.
For `GRAPH_COMPLETION`, `GRAPH_COMPLETION_DECOMPOSITION`, `GRAPH_SUMMARY_COMPLETION`, `GRAPH_COMPLETION_COT`, and `GRAPH_COMPLETION_CONTEXT_EXTENSION`, retrieval works in two combined steps:
1. **Vector store** — seed nodes and edges are found by embedding similarity across indexed fields (`brute_force_triplet_search` over the vector index collections).
2. **Graph store** — those seeds are resolved against the knowledge graph so the surrounding triplets (nodes + relationships) come back as structured context.
The merged graph + vector context is then formatted and (unless `only_context=True`) passed to the LLM. So a single `GRAPH_COMPLETION` call is the built-in way to "look at both the graph and the vector store" at once.
`TRIPLET_COMPLETION` is related but different: it searches precomputed triplet text in the vector store (`Triplet_text`) and uses those triplet payloads as context, without expanding the seeds through the graph store.
```python theme={null}
import cognee
from cognee import SearchType
# One call uses vector similarity to seed and graph structure to expand
results = await cognee.search(
query_text="How are X and Y connected?",
query_type=SearchType.GRAPH_COMPLETION,
)
```
If you instead want raw vector-only matches (`CHUNKS`, `SUMMARIES`) and raw graph rows (`CYPHER`, `NATURAL_LANGUAGE`) and prefer to merge them yourself, run each search type separately and combine the returned lists. For multi-part questions, `GRAPH_COMPLETION_DECOMPOSITION` with `retriever_specific_config={"decomposition_mode": "combined_triplets_context"}` splits the query into subqueries, fetches triplets for each, and de-duplicates them into one merged context before answering.
</Accordion>
<Accordion title="Retrieval pipeline and raw results">
Every retriever follows the same three-step internal pipeline:
* **`get_retrieved_objects`** — fetches raw graph triplets (edges + nodes) or vector chunks from the store
* **`get_context_from_objects`** — formats those objects into a context string (e.g. `"Nodes: … Connections: …"` for graph modes)
* **`get_completion_from_context`** — sends context to the LLM to produce a natural-language answer *(skipped when `only_context=True`)*
**Completion modes** (`GRAPH_COMPLETION`, `RAG_COMPLETION`, `TRIPLET_COMPLETION`, etc.) run all three steps and return a **string answer**. That is why `GRAPH_COMPLETION` returns a plain string by default: the raw graph objects are consumed internally to assemble the LLM prompt.
**Retrieval-only modes** (`CHUNKS`, `SUMMARIES`) skip the LLM step and return **structured dicts** directly.
**Inspecting the raw retrieved objects**: add `verbose=True` to receive `text_result` (LLM answer), `context_result` (formatted context string), and `objects_result` (raw graph edges/nodes or chunk objects) together. Use `only_context=True` to skip LLM generation and return the formatted context string directly. For structured chunk dicts with no LLM call at all, use `SearchType.CHUNKS`. See [Search Basics — raw source objects](/guides/search-basics#citation-and-source-tracking) for code examples.
</Accordion>
</AccordionGroup>
<Columns>
<Card title="Add" icon="plus" href="/core-concepts/main-operations/legacy-operations/add">
First bring data into Cognee
</Card>
<Card title="Cognify" icon="brain-cog" href="/core-concepts/main-operations/legacy-operations/cognify">
Build the knowledge graph that search queries
</Card>
<Card title="Architecture" icon="building" href="/core-concepts/architecture">
Understand how vector and graph stores work together
</Card>
<Card title="Sessions and Caching" icon="message-square" href="/core-concepts/sessions-and-caching">
Enable conversational memory with sessions
</Card>
</Columns>
# Push
Source: https://docs.cognee.ai/core-concepts/main-operations/push
Upload a local Cognee knowledge graph to a remote Cognee instance.
## What is the push operation
The `.push` operation uploads a local dataset's already-built knowledge graph to Cognee Cloud or another remote Cognee instance.
Unlike a raw-data sync, `push()` ships the graph you already extracted locally. Cognee exports the dataset to a [COGX archive](/core-concepts/further-concepts/cogx), uploads it, and imports it on the remote instance so entities and relationships can be preserved.
It is available from both the Python SDK and the CLI:
```python theme={null}
await cognee.push("onboarding")
```
```bash theme={null}
cognee push onboarding
```
## Where push fits
* Use `push()` when you have built a graph locally and want the same graph available remotely.
* Use it when local graph extraction has already done the expensive LLM work and you want the remote side to preserve that result.
* Use it after [Remember](/core-concepts/main-operations/remember) or [Cognify](/core-concepts/main-operations/legacy-operations/cognify), not before them.
* Use [syncing a local instance](/cognee-cloud/connections/syncing-local-instance) instead when you want to send raw data and let the remote instance rebuild memory itself.
## What happens under the hood
1. **Resolve the local dataset** - Cognee finds the dataset by name or UUID and checks that it can be read.
2. **Export the graph** - the dataset's graph is exported as a COGX archive.
3. **Upload the archive** - the archive is sent to the configured remote Cognee instance.
4. **Import remotely** - the remote instance imports the archive into the target dataset.
5. **Report the result** - Cognee returns the remote status, target dataset, graph size, and any pipeline run id.
<Note>
The dataset must already have a knowledge graph. If the export finds no graph nodes, run `cognee.remember()` or `cognee.cognify()` on the dataset first.
</Note>
## Import modes
`push()` supports three remote import modes:
| Mode | Remote behavior | LLM calls |
| ----------- | ------------------------------------------------------------------- | --------- |
| `preserve` | Map the exported entities and facts directly into the remote graph | None |
| `hybrid` | Preserve the exported graph and also cognify the raw content | Yes |
| `re-derive` | Ignore the exported graph and rebuild from the raw content remotely | Yes |
The default mode is `preserve`, which is the graph-preserving path.
## Authentication
`push()` uses the same remote credential stack as `cognee.serve()`:
1. Explicit `url` and `api_key` arguments
2. An active `cognee.serve()` connection
3. `COGNEE_SERVICE_URL` and `COGNEE_API_KEY` environment variables
4. Saved credentials from a previous `serve` login
If no remote credentials are available, `push()` raises an authentication error instead of uploading anywhere implicitly.
## Examples and details
<Accordion title="Push with the Python SDK">
```python theme={null}
import cognee
# Build a graph locally.
await cognee.remember("docs/handbook.pdf", dataset="onboarding")
# Connect once so push can reuse the remote credentials.
await cognee.serve(url="https://your-tenant.aws.cognee.ai", api_key="your-api-key")
# Push the local graph to the remote dataset with the same name.
result = await cognee.push("onboarding")
print(result.target_dataset)
print(result.num_nodes, result.num_edges)
```
</Accordion>
<Accordion title="Push with the CLI">
```bash theme={null}
# Build local memory first.
cognee remember docs/handbook.pdf --dataset-name onboarding
# Log in once.
cognee serve
# Push the graph to the remote instance.
cognee push onboarding
```
</Accordion>
<Accordion title="Push to a different remote dataset">
Use `target_dataset` when the remote dataset should have a different name from the local dataset.
```python theme={null}
await cognee.push(
"onboarding",
target_dataset="prod_onboarding",
)
```
```bash theme={null}
cognee push onboarding --target-dataset prod_onboarding
```
</Accordion>
<Accordion title="Run large imports in the background">
For larger graphs, schedule the remote import in the background and track the returned pipeline run id.
```python theme={null}
result = await cognee.push(
"onboarding",
run_in_background=True,
)
print(result.pipeline_run_id)
```
```bash theme={null}
cognee push onboarding --background
```
</Accordion>
<Accordion title="Push to an explicit remote instance">
If you do not want to rely on a saved `serve` login, pass the remote URL and API key directly.
```python theme={null}
await cognee.push(
"onboarding",
url="https://your-tenant.aws.cognee.ai",
api_key="your-api-key",
)
```
```bash theme={null}
cognee push onboarding \
--url https://your-tenant.aws.cognee.ai \
--api-key your-api-key
```
</Accordion>
## See also
* [COGX archives](/core-concepts/further-concepts/cogx)
* [Syncing a local instance](/cognee-cloud/connections/syncing-local-instance)
* [Cognee CLI](/cognee-cli/overview)
# Recall
Source: https://docs.cognee.ai/core-concepts/main-operations/recall
Query Cognee memory with auto-routing and session-aware retrieval.
## What is the recall operation
The `.recall` operation is the main retrieval entry point in Cognee v1.0. It searches memory using the best available source for the request.
* **Auto-routing by default**: when you do not specify a search type, `recall()` classifies the query and picks the best retrieval strategy automatically.
* **Session-aware**: with `session_id`, it can search session cache entries first and fall through to the permanent graph if needed.
* **Graph-backed by default**: for permanent memory, `recall()` runs graph retrieval — not plain embedding similarity. `HYBRID_COMPLETION` is the fallback when auto-routing does not choose a more specific strategy.
* **Source tagging**: each recall result includes a `source` field (`"graph"`, `"session"`, `"trace"`, `"session_context"`, ...) so you can tell where it came from — see the [full source table](/python-api/recall#return-value).
## Where recall fits
* Use `recall()` as the default way to ask questions over memory in v1.0.
* Use it after [Remember](/core-concepts/main-operations/remember) has created either permanent or session memory.
* Use explicit `query_type` only when you want to force a specific retrieval mode.
* Use datasets to scope results to a specific knowledge base.
## What happens under the hood
1. **Check session scope**
* If you pass `session_id` without `datasets` and without `query_type`, Cognee first searches the session cache directly.
* Session search is keyword-based over stored question, context, and answer fields.
2. **Choose the retrieval strategy**
* If `query_type` is provided, Cognee uses it directly.
* Otherwise, if no usable LLM is configured, Cognee resolves to `CHUNKS` — plain vector search over chunks — because nothing could write a completion answer. This applies under either `auto_route` setting, and is decided by LLM availability alone, never by which extractor built the graph.
* Otherwise, if `auto_route=True`, a rule-based router picks the best strategy based on your query. See [Auto-routing behavior](#auto-routing-behavior) below for the kinds of patterns it recognizes.
* If `auto_route=False`, Cognee falls back to `HYBRID_COMPLETION`.
3. **Run graph retrieval when needed**
* If session search finds nothing, or if graph retrieval is requested, `recall()` queries the permanent knowledge graph.
* The retrieval strategy selected in step 2 determines how that graph query is executed.
## After recall finishes
* **Session-only recall**: you get matching session entries when cache hits exist, each tagged with `source="session"`.
* **Graph-backed recall**: you get normalized graph result objects tagged with `source="graph"`.
* **Hybrid behavior**: with `session_id` plus graph-scoping inputs like `datasets` or `query_type`, recall uses the permanent graph path rather than session-only lookup.
## Examples and details
<Accordion title="Prerequisites before calling recall">
`recall()` only reads from memory — it never initializes anything itself. Populate memory first with [`remember()`](/core-concepts/main-operations/remember) (or the legacy [`add()`](/core-concepts/main-operations/legacy-operations/add) + [`cognify()`](/core-concepts/main-operations/legacy-operations/cognify) sequence). The first ingestion run creates the relational, vector, and graph databases and the default user.
```python theme={null}
import cognee
await cognee.remember("Einstein was born in Ulm.") # creates databases + ingests
results = await cognee.recall("Where was Einstein born?")
```
If you call `recall()` before any data exists, it raises `RecallPreconditionError` (HTTP 422), triggered by the underlying `DatabaseNotCreatedError` (*"The database has not been created yet. Please call `await setup()` first."*) or `UserNotFoundError`. The fix is to run `remember()` (or `add()` + `cognify()`) first.
`recall()` can also fail when the configured LLM provider (or LiteLLM proxy) reports that its token budget is exhausted. In that case it raises `LLMPaymentRequiredError`, which the API surfaces as **HTTP 402 (Payment Required)** with body `{"detail": "LLM provider requires payment or token budget is exhausted. [LLMPaymentRequiredError]"}` — or, when the provider's own budget sentence can be identified, `{"detail": "LLM budget exhausted: <provider sentence> [LLMPaymentRequiredError]"}` (see [402 Payment Required](/api-reference/introduction)). This error is **terminal** — Cognee does not retry budget-exhaustion failures — so treat a `402` as final for the request and prompt the user to top up their token budget rather than reissuing the call.
`recall()` also accepts only its documented parameters — there is no catch-all `**kwargs`. Passing an unsupported keyword such as `node_type` raises a `TypeError`. Use `node_name` to scope retrieval to specific nodes or node sets; `node_type` belongs to the legacy [`search()`](/core-concepts/main-operations/legacy-operations/search) API. See the [`recall()` API reference](/python-api/recall) for the full parameter list.
</Accordion>
<Accordion title="Recall while indexing is still running">
`recall()` can run while another dataset is still being ingested or indexed in the background.
* There is no global lock that blocks retrieval while `remember()` is processing.
* `recall()` only sees data that has already made it through indexing.
* If a dataset is mid-run, results may be incomplete until that run reaches completion.
If you need to confirm a dataset is fully ready before querying it, check its status with [`cognee.datasets.get_status()`](/python-api/datasets#datasetsget_status) or the indexing-status guidance on [Remember](/core-concepts/main-operations/remember#checking-indexing-status).
</Accordion>
<Accordion title="Auto-routing behavior">
Auto-routing is on by default, and it is **rule-based, not LLM-based**. `recall()` decides the search type in one of these ways:
| How you call it | What decides the search type | LLM call to decide |
| ------------------------------------------------ | ---------------------------------------------------------------------------------------------------- | ------------------ |
| `recall(q)` — default, `auto_route=True` | Rule-based query router: regex cues with weighted scores | No |
| `recall(q, auto_route=False)` | Always `HYBRID_COMPLETION` | No |
| `recall(q, query_type=...)` | Your value — it always wins | No |
| `recall(q, query_type=SearchType.FEELING_LUCKY)` | Separate LLM-based selector (falls back to `RAG_COMPLETION` if the model returns an unusable answer) | Yes |
**What the router recognizes**
| Cue in the query | Routed search type | What you get back |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ | ----------------------------------------- |
| `summarize`, `summary`, `overview`, `outline`, `tl;dr`, `gist`, `main points`, `key takeaways`, `high-level` | `GRAPH_SUMMARY_COMPLETION` | Generated answer |
| `why`, `explain`, `reasoning`, `step by step`, `chain of thought` (weaker: `because`, `therefore`, `consequently`) | `GRAPH_COMPLETION_COT` | Generated answer |
| `how is/are X related/connected/linked`, `what connects/links/ties`, `path between`, `degree of separation` (weaker: `connection`, `relationship`, `related to`, `linked to`) | `GRAPH_COMPLETION_CONTEXT_EXTENSION` | Generated answer |
| `when`, `before`, `after`, `during`, `since`, `until`, `timeline`, `chronolog…`, `era`, `decade`, `century`, a 4-digit year, `between 1990 and 2000` | `TEMPORAL` | Generated answer |
| The whole query is one quoted phrase (`"exact wording"`), or it contains `exact`, `verbatim`, `literal`, `word for word` | `CHUNKS_LEXICAL` | Raw chunk payloads — no generated answer |
| `coding rules`, `code review`, `best practice`, `lint`/`linting`/`linter`, `refactor`/`refactoring` | `CODING_RULES` | Structured payloads — no generated answer |
| Query starts with `MATCH `, `RETURN `, `CREATE `, or `MERGE ` (uppercase), or contains `--(` / `)--` | `CYPHER` | Raw graph rows — no generated answer |
| **No cue matched** | `HYBRID_COMPLETION` | Generated answer |
The router **never** selects `SUMMARIES`, `CHUNKS`, `RAG_COMPLETION`, `TRIPLET_COMPLETION`, `GRAPH_COMPLETION_DECOMPOSITION`, `NATURAL_LANGUAGE`, or `AGENTIC_COMPLETION`. `HYBRID_COMPLETION` is the fallback it lands on when no pattern matches; no rule selects `GRAPH_COMPLETION`, so that type is only reachable by passing `query_type` yourself. Only `FEELING_LUCKY`'s LLM selector can land on retrieval-only types such as `SUMMARIES` or `CHUNKS`, which return payloads instead of an answer — if you want an answer every time, do not use `FEELING_LUCKY`. For the full answer-vs-payload breakdown per type, see [SearchType](/python-api/search-type#values) and [What recall returns](#what-recall-returns).
Cues are weighted and accumulate; the highest total wins, and anything unmatched falls back to `HYBRID_COMPLETION`. A cue is ignored when a negation (`not`, `n't`, `no`, `never`, `without`, `lack`) appears within 20 characters before it, so "topics **not related to** billing" does not route to context extension.
Incidental code tokens — a stray `def `, `return `, `async `, `await `, `import `, `class X(`, `.py`, or `function x(` inside an otherwise natural-language question — are a deliberately weak signal. On their own they are not enough to select coding-rules retrieval, so a query like "Where is `def recover()` mentioned in the incident notes?" falls back to `HYBRID_COMPLETION` rather than routing to coding rules. They are also outranked whenever the same query carries a stronger cue, such as a summary, relationship, year-range, or fully quoted phrase. Explicit coding vocabulary still routes to coding-rules retrieval even when it appears alongside a code token, as in "What coding rules apply to `gate.py`?".
**Confirming what the router chose**
The router logs its decision at `INFO` level under the `query_router` logger (see [Logging](/setup-configuration/logging)):
```text theme={null}
query_router: routed=TEMPORAL score=3.0 query='What happened before the outage?' scores={'TEMPORAL': 3.0}
query_router: no patterns matched, default=HYBRID_COMPLETION query='Who won Nobel Prizes?'
```
`query_router: no patterns matched` is **not** an error — it is the normal line for a plain factual question, and it means recall ran `HYBRID_COMPLETION`. You can also read the type that actually ran off each result's `search_type` field.
If you pass `query_type` while `auto_route` is still `True`, the router still runs and logs its own decision, and when that decision differs from your choice it also logs `Router override recorded: routed=…, user_chose=…`. It costs no LLM call and your explicit `query_type` is what executes.
**When to override `query_type`**
* You want raw payloads rather than a generated answer — pass `CHUNKS` or `SUMMARIES`.
* A cue word routes somewhere you did not intend: "Why did the migration fail?" routes to the slower `GRAPH_COMPLETION_COT`, and "What changed in 2024?" routes to `TEMPORAL`. Pass `query_type=SearchType.HYBRID_COMPLETION` to keep the fast default.
* Your query needs a mode it carries no cue for — a time-scoped question phrased without a time word, or a multi-part question that suits `GRAPH_COMPLETION_DECOMPOSITION`.
* You need a type outside the router's set (`GRAPH_COMPLETION`, `RAG_COMPLETION`, `TRIPLET_COMPLETION`, `NATURAL_LANGUAGE`, `AGENTIC_COMPLETION`, …).
* You want predictable latency and cost across calls — pinning `query_type` makes every call use the same retriever. `auto_route=False` pins it to `HYBRID_COMPLETION`.
<Warning>
A query that begins with `MATCH `, `RETURN `, `CREATE `, or `MERGE ` routes to `CYPHER`. If `ALLOW_CYPHER_QUERY=false`, that raises `UnsupportedSearchTypeError` ("Cypher query search types are disabled."). Rephrase the query or pass an explicit `query_type`.
</Warning>
</Accordion>
<Accordion title="How recall relates to retrievers">
Most users should think in terms of **asking memory a question**, not selecting a retriever manually.
Under the hood, retrieval strategies — graph completion, summary, temporal, lexical, coding-rules — do the actual work. `recall()` is intentionally one layer above them:
* you call `recall()`
* Cognee chooses or accepts a `query_type`
* the matching retriever runs underneath
* the result comes back through the recall API
If you want to understand or control the lower-level retrieval behavior directly, see the [lower-level search reference](/core-concepts/main-operations/legacy-operations/search).
</Accordion>
<Accordion title="What recall returns">
* Session-only recall returns matching session cache entries.
* Graph-backed recall returns the output of the underlying retrieval mode selected by the router or `query_type`.
* Depending on the retrieval path, each graph result object's `text` can contain a plain answer, retrieved context, chunk text, or rendered structured output.
* Each recall result is tagged with `source` so callers can distinguish session, graph, trace, and graph-context results.
* `only_context=True` skips the final LLM answer-generation step and returns retrieved context instead.
* `verbose=True` exposes extra retrieval details from the lower-level graph search path.
<Tabs>
<Tab title="Session Results">
When `recall()` returns session cache hits, the result is a list of `ResponseQAEntry` objects tagged with `source="session"`.
These entries follow the session QA shape documented in [Sessions and Caching](/core-concepts/sessions-and-caching#session-data-structure), with fields such as:
* `time`
* `qa_id`
* `question`
* `context`
* `answer`
* `feedback_text`
* `feedback_score`
* `source`
</Tab>
<Tab title="Graph Results">
When `recall()` falls through to graph retrieval, it returns normalized `ResponseGraphEntry` objects tagged with `source="graph"`. Each entry includes `text`, `kind`, `search_type`, optional `dataset_id` and `dataset_name`, `metadata`, `raw`, and `structured`.
* `kind` is the normalized result shape and can be more precise than `search_type`. Graph-completion variants — including `AGENTIC_COMPLETION` — collapse to `kind="graph_completion"`, while retrieval-only types map to structural kinds (`chunk`, `summary`, `cypher`, `temporal`, `coding_rule`, `natural_language`). `RAG_COMPLETION` and `TRIPLET_COMPLETION` keep their own `rag_completion`/`triplet_completion` kinds.
* `structured` is populated only when you pass a Pydantic model — either through the first-class [`response_model`](/python-api/recall#structured-output-with-response_model) parameter or the equivalent `retriever_specific_config={"response_model": ...}` dict key — and parsing succeeds; it holds the parsed model as a dict (mirrored in `raw`), and `kind` is set to `structured`. For plain-text answers it is `null`, so text-only renderers can keep reading `text` unchanged. Over HTTP (and from the SDK in remote mode) the model travels as a JSON Schema, which reconstructs structure but not custom validators or value constraints — re-validate with `MyModel.model_validate(result.structured)` when you need those to run.
* With `FEELING_LUCKY`, the router resolves an effective search type before retrieval runs, so `kind` and `search_type` reflect the retriever that actually ran (for example `CHUNKS` → `kind="chunk"`), not the `FEELING_LUCKY` placeholder.
For chunk and summary results (`CHUNKS`, `CHUNKS_LEXICAL`, `SUMMARIES`), `metadata` carries stable source identifiers — `data_id` (the ingested `Data` item's id), `chunk_id` (the chunk's own node id), `chunk_index`, and `document_name` — so you can map a result back to what you ingested and inspect the exact cited chunk. Only keys present in the underlying payload are included; completion-style results carry an empty `metadata` dict unless `include_references=True`. With references on, `GRAPH_COMPLETION` (and other completion) answers carry `metadata.evidence`, a structured list of the chunks, graph nodes, and graph edges placed in the LLM context, and the same `data_id`/`chunk_id` are surfaced inline in the `Evidence:` bullets (`- chunk N of document NAME (data_id: …, chunk_id: …)`, with a `"snippet"` suffix on `RAG_COMPLETION` only). Graph-completion citations are resolved through the [edge-evidence sidecar](/setup-configuration/overview#edge-evidence), which needs its Alembic revision applied on an existing deployment.
The `text` field is derived from the underlying retrieval mode selected by the router or `query_type`:
| Search type(s) | What `text` represents | Notes |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `GRAPH_COMPLETION`, `RAG_COMPLETION`, `HYBRID_COMPLETION`, `TRIPLET_COMPLETION`, `GRAPH_COMPLETION_DECOMPOSITION`, `GRAPH_SUMMARY_COMPLETION`, `GRAPH_COMPLETION_COT`, `GRAPH_COMPLETION_CONTEXT_EXTENSION` | Natural-language answer | With `only_context=True`, `text` contains the formatted context string instead. |
| `TEMPORAL` | Time-aware answer text | `only_context=True` returns retrieved temporal context instead of an answer. |
| `AGENTIC_COMPLETION` | Agentic answer text | Normalized to `kind="graph_completion"`. Requires the resolved graph scope to contain exactly one dataset. |
| `CHUNKS` | Chunk text | `raw` preserves the normalized chunk payload. `score` is the vector store's raw distance — lower is better; see [Relevance in `score`](/python-api/recall#relevance-in-score). |
| `CHUNKS_LEXICAL` | Ranked chunk text | `raw` preserves the chunk payload. `score` is `None`: BM25 ranks the results but does not populate the field. |
| `SUMMARIES` | Summary text | `raw` preserves the normalized summary payload. `score` carries the same raw distance as `CHUNKS`. |
| `CYPHER`, `NATURAL_LANGUAGE`, `CODING_RULES` | Rendered structured row/object output | `raw` preserves the normalized structured payload. `CYPHER` and `NATURAL_LANGUAGE` are disabled when `ALLOW_CYPHER_QUERY=false`. |
| `FEELING_LUCKY` | Varies | The router resolves an effective search type before retrieval, so `kind` and `search_type` report the resolved retriever rather than `FEELING_LUCKY`. |
<Note>
Read recall results with attribute access (`result.text`, `result.raw`, `result.source`), not `result.get(...)`. The `text_result`, `context_result`, and `objects_result` keys belong to the legacy [`search(verbose=True)`](/python-api/search) API, which returns plain dicts. See [recall() — Return value](/python-api/recall#return-value) for the full per-`source` field list.
</Note>
</Tab>
</Tabs>
</Accordion>
<Accordion title="Using only_context">
Set `only_context=True` when you want the retrieved context without asking Cognee to produce a final answer.
When `only_context=True`, Cognee stops after retrieval formatting:
* the LLM answer is **not** generated
* the final completion step is skipped
* the returned value is the retrieved context rather than a synthesized answer
```python theme={null}
results = await cognee.recall(
query_text="Tell me about NLP",
only_context=True,
)
```
This is useful when you want to:
* inspect what was retrieved before answer generation
* feed the retrieved context into your own prompt or downstream logic
* debug retrieval quality separately from answer quality
In other words, `only_context=True` keeps recall focused on **retrieval output** instead of answer generation.
The context on its own is less than a real completion receives, which also carries the session guidance, the conversation history, and the rendered prompt templates. If you are feeding your own model rather than just inspecting retrieval, add `context_format="prompt"` to get that whole envelope — `question`, `context`, `session_context`, `user_prompt`, and `system_prompt` — as a single item, rebuilt read-only with no LLM call and no session write. See [`context_format`](/python-api/recall#additional-keyword-options) for the exact shape and its limits.
</Accordion>
<Accordion title="Session cache reads and writes during recall">
`session_id` makes recall session-aware, but the exact behavior depends on the recall mode.
**Session reads:**
* With `session_id` and no explicit `query_type`, `recall()` searches session-cache QA entries by keyword and returns matches tagged with `source="session"`.
* If `session_id` is used with `datasets` or `dataset_ids`, recall can combine session lookup with graph retrieval.
* If you pass an explicit `query_type`, the default path is graph-backed retrieval. Session history can still be used during the LLM completion step, but session-cache QA lookup is not part of the default source set unless you pass `scope`.
* `scope` also reaches the two session sources that `auto` never picks on its own: `scope=["trace"]` searches recorded agent traces, and `scope=["session_context"]` renders the session's accumulated guidance. The guidance read takes a `context_profile` — `"qa"` (the default) for what conversation turns taught the session, or `"agent"` for lessons drawn from agent tool traces — and the two profiles are separate, so a query under one never returns the other's entries. Both reads are read-only: serving guidance stamps nothing and ages no entry. See the [`recall()` API reference](/python-api/recall#additional-keyword-options) for every scope value and `context_profile`.
**Graph-backed completion with session history:**
When graph-backed recall runs with `session_id` and the LLM completion step is enabled, Cognee loads prior session conversation history and prepends it to the completion prompt. If [`improve()`](/core-concepts/main-operations/improve) has built the [global context index](/core-concepts/further-concepts/global-context-index) and the retriever is configured with `include_global_context_index`, a background-knowledge prelude — the dataset's root summary plus the top matching bucket summaries — is also prepended. That prelude comes from the index, not from a per-session snapshot.
**Session writes:**
When the LLM completion step runs through session-enabled graph retrieval, Cognee appends a new QA entry to the session cache with the `question`, stored `context`, and generated `answer`. The next compatible recall on the same `session_id` can see that entry.
**Using `only_context=True`:**
Pass `only_context=True` to skip the final LLM completion. Because the QA write happens during completion, `only_context=True` also avoids adding a new QA entry to the session cache.
```python theme={null}
# Session-aware recall with answer generation can write a new QA entry
answer = await cognee.recall(
query_text="What did we decide about pricing?",
session_id="chat-42",
)
# Returns retrieved context only; skips final LLM completion and QA write
context = await cognee.recall(
query_text="What did we decide about pricing?",
session_id="chat-42",
only_context=True,
)
```
Use `only_context=True` when you want retrieval output without LLM summarization and without growing the session cache.
</Accordion>
<Accordion title="Dataset scoping">
```python theme={null}
answers = await cognee.recall(
query_text="Give me an overview of this dataset.",
datasets=["product_docs"],
)
```
* **Scoping is exclusive.** When you pass `datasets` (or `dataset_ids`), retrieval runs against **only** those datasets — Cognee does not pull from any other dataset, even ones the current user can read.
* **When you omit both**, recall searches across **all** datasets the current user has read access to. Supply a dataset to narrow that down to a single knowledge base.
* `datasets` limits graph retrieval to named datasets. With backend access control enabled, names are resolved only against datasets owned by the current user.
* `dataset_ids` scopes retrieval by dataset UUID instead of name. Use this for shared datasets that the current user can access but did not create. When supplied, it takes precedence over `datasets` and the name-to-UUID resolution step is skipped.
* With `session_id` plus either `datasets` or `dataset_ids`, recall becomes hybrid: session context is available, but graph retrieval is scoped to the selected dataset or dataset UUIDs.
<Warning>
If you set `query_type=SearchType.AGENTIC_COMPLETION`, the resolved graph scope must contain exactly one dataset. Passing multiple dataset names or UUIDs, or leaving scope broad enough to match multiple readable datasets, can raise `422 InvalidAgenticDatasetScope`.
</Warning>
<Warning>
If Alice created `shared_dataset` and Bob only has permission to use it, Bob should query it with `dataset_ids=[shared_id]`, not `datasets=["shared_dataset"]`. Name-based lookup can fail for non-owners even when they have read access.
</Warning>
</Accordion>
<Accordion title="Parameters">
<Tabs>
<Tab title="Basic Parameters">
| Option | What it does |
| -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `query_text` | The natural-language query to answer. |
| `query_type` | Forces a specific underlying search type instead of using auto-routing. |
| `datasets` | Restricts graph retrieval to specific dataset names. Names are resolved only against datasets owned by the current user. |
| `dataset_ids` | Restricts graph retrieval by dataset UUIDs. Use this for shared datasets the current user did not create. Takes precedence over `datasets` when both are set. |
| `top_k` | Limits the number of returned results. Defaults to `15`. |
| `auto_route` | Enables the rule-based query router. Defaults to `True`. |
| `session_id` | Enables session-aware retrieval and session-cache lookup. |
| `only_context` | Returns retrieved context only. The final LLM answer is not generated. |
| `context_format` | Shape of the `only_context=True` result. Defaults to `"context"`; `"prompt"` returns the full prompt envelope instead. |
| `system_prompt` / `system_prompt_path` | Customizes the generation prompt. |
</Tab>
<Tab title="Advanced Parameters">
| Option | What it does |
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `node_name` / `node_name_filter_operator` | Restricts graph retrieval to matching node names or node sets. |
| `wide_search_top_k` | Expands the initial candidate set used by graph-completion retrieval before ranking. |
| `triplet_distance_penalty` | Adjusts scoring for triplet-based retrieval paths. |
| `feedback_influence` | Applies stored feedback weights during ranking where supported. |
| `verbose` | Returns extra retrieval details from the lower-level graph search path. |
| `retriever_specific_config` | Passes advanced configuration directly to the selected retriever. |
| `response_model` | A Pydantic model class for structured completion output. The validated payload lands in each result's `structured` field. Shorthand for `retriever_specific_config={"response_model": ...}` — passing two different models across the two raises `CogneeValidationError`. |
| `user` | Runs recall under a specific user context, affecting dataset access and session-cache lookup. |
</Tab>
</Tabs>
</Accordion>
<Accordion title="Under the hood — legacy operations">
`recall()` runs [Search](/core-concepts/main-operations/legacy-operations/search) under the hood for graph-backed retrieval.
Use legacy Search directly when you need to select a specific retriever, inspect retrieval internals, or use advanced parameters not exposed by `recall()`. See also [Search Basics](/guides/search-basics) for the full set of retrieval options.
</Accordion>
<Accordion title="recall() vs search(): when to use each">
| | `recall()` | `search()` (legacy) |
| ------------------------- | ------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------- |
| **Recommended for** | Most v1.0 use cases | Advanced retriever control |
| **Search type selection** | Auto-routes by default; pass `query_type` to override | Always explicit; defaults to `HYBRID_COMPLETION` |
| **Session behavior** | Searches session-cache QA entries by keyword; falls through to graph on miss | `session_id` writes/reads conversation history only — no cache lookup |
| **Source tagging** | Results carry `source` (`"graph"`, `"session"`, `"trace"`, `"session_context"`, ...) | No source tagging |
| **Parameter surface** | Compact; explicit keyword options documented in the API reference | Full surface — `neighborhood_depth`, `node_type`, `triplet_distance_penalty` with explicit defaults |
**Use `recall()`** when you want to ask a question and get an answer. It handles routing, session lookup, and source attribution automatically.
**Use `search()` directly** when you need a specific retriever, lower-level parameters such as `neighborhood_depth` or `node_type`, or when building custom pipelines.
</Accordion>
<Columns>
<Card title="Remember" icon="brain" href="/core-concepts/main-operations/remember">
Create permanent or session memory
</Card>
<Card title="Improve" icon="sparkles" href="/core-concepts/main-operations/improve">
Enrich the graph for better future recall
</Card>
<Card title="Understand Recall with RAG Completion" icon="search" href="/guides/rag-recall">
Hands-on guide to recall() and RAG\_COMPLETION
</Card>
</Columns>
# Remember
Source: https://docs.cognee.ai/core-concepts/main-operations/remember
Store data in Cognee as permanent graph memory or session memory.
## What is the remember operation
The `.remember` operation is the main ingestion entry point in Cognee v1.0. It stores information in memory with a single API call.
It has two modes:
* **Permanent memory**: without `session_id` provided, `remember()` runs the full ingestion pipeline for you — normalizing your data, building the knowledge graph, and enriching it for retrieval, all in one call. Cognee does not create a session ID for this path.
* **Session memory**: with `session_id`, `remember()` writes to the session cache for fast short-term memory. If `self_improvement=True` (the default), it then starts a background [Improve](/core-concepts/main-operations/improve) pass to bridge that session content into the permanent graph.
* **Dataset-aware**: permanent memory is written into a named dataset, which defaults to `main_dataset`.
## Where remember fits
* Use `remember()` when you want the simplest way to get data into Cognee.
* Use it for raw text, files, file lists, and `DataItem` objects.
* Use `session_id` when you want fast conversational memory first and long-term graph sync second.
* Use `self_improvement=False` if you want to skip the follow-up improvement pass.
## What happens under the hood
### Permanent memory
1. **Ingest** — data is normalized and attached to a named dataset.
2. **Build the graph** — documents are chunked, entities and relationships are extracted, and embeddings are created.
3. **Enrich** — by default, Cognee runs a follow-up [Improve](/core-concepts/main-operations/improve) pass that adds derived retrieval structures to the graph.
### Session memory
1. **Store in session cache** — the content is written to the cache keyed by user and session.
2. **Return immediately** — the call completes quickly with a session-stored result.
3. **Optional background bridge** — when `self_improvement=True` (default), Cognee immediately starts [Improve](/core-concepts/main-operations/improve) in the background to sync that session into the permanent graph.
4. **Manual bridge when disabled** — when `self_improvement=False`, nothing is bridged automatically; the content stays in session cache until you explicitly run `cognee.improve(dataset=..., session_ids=[...])`.
## After remember finishes
* **Permanent memory**: your data is stored in a dataset, chunked, turned into graph structure, embedded for retrieval, and usually enriched with an immediate follow-up improve pass.
* **Session memory**: your data is available quickly through the session cache; it only becomes permanent graph memory if the improvement bridge runs.
## Examples and details
<Accordion title="Accepted inputs">
`remember()` accepts these input types (any of these, or a list mixing them):
* **Raw text strings** — e.g. `"Einstein was born in Ulm."`
* **Local file paths as strings** — `file://` URLs, or absolute (`"/path/to/doc.pdf"`) and relative paths that resolve to an existing file. A bare path string is only treated as a file reference when that file exists; otherwise it is ingested as raw text (see below)
* **HTTP/HTTPS URLs as strings** — fetched and ingested in permanent mode. A GitHub/GitLab repository URL is the exception: it is shallow-cloned and indexed as a code graph rather than fetched as a page (see [Code repository URLs](/python-api/add#code-repository-urls))
* **S3 paths as strings** — `"s3://bucket/key"` (requires `s3fs`)
* **`DataItem` objects** — a lightweight wrapper for attaching metadata to any of the above (see below)
* **HTTP upload objects with `.file` and `.filename` attributes** — e.g. FastAPI `UploadFile`. This is the supported path for "file-like" inputs in permanent-memory mode; it is what the HTTP API uses internally
<Note>
In the default blocking mode, plain Python file handles such as `open("doc.pdf", "rb")` (i.e. `BufferedReader`) are **not** accepted and will raise `IngestionError: Data type not supported`. Pass the file path as a string instead, wrap the bytes in an object that exposes `.file` and `.filename`, or use `run_in_background=True` so Cognee can materialize stream-like inputs before scheduling the pipeline.
</Note>
In permanent-memory mode:
* Plain text is stored directly as memory content.
* File paths are ingested and normalized into text before graph building.
* HTTP/HTTPS URLs are fetched and passed through the ingestion pipeline; a repository URL is cloned into `COGNEE_REPOS_DIR` and handed to `cognify()` as a code repository instead.
* Dataset scoping still applies through `dataset_name`.
<Note>
An absolute-looking string that does not point to an existing file is stored as text, not resolved as a path. So `await cognee.remember("/remember to call the dentist")` records that sentence as memory content on every platform, including Windows. Only strings that actually resolve to an existing local file — or explicit `file://` URLs — are read from disk.
</Note>
<Note>
In session-memory mode, a URL string is stored in the session cache as text. It is not fetched or scraped. Permanent mode is required for URL ingestion to work end-to-end.
</Note>
**What is `DataItem`?**
`DataItem` is a small dataclass exported from `cognee.tasks.ingestion.data_item` that lets you attach metadata to a single input alongside its content. Use it when you want a stable ID or extra labels travelling with the data through the ingestion pipeline.
```python theme={null}
from cognee.tasks.ingestion.data_item import DataItem
import cognee
await cognee.remember(
DataItem(
data="Einstein was born in Ulm.", # any accepted input above
label="biography-note", # optional free-form label
external_metadata={"source": "wiki"}, # optional dict
data_id=None, # optional UUID to pin an ID
)
)
```
The `data` field of a `DataItem` accepts the same types as `remember()` itself (string text, file path, URL, etc.). You can also pass a list of `DataItem` objects to ingest several items in one call.
</Accordion>
<Accordion title="Supported formats">
In permanent-memory mode, `remember()` supports the following file formats out of the box:
| Loader | Extensions | Install extra |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ |
| **CodeLoader** | `.py` `.js` `.ts` `.go` `.rs` `.java` and 25 more source-code extensions — see [Loaders](/core-concepts/further-concepts/loaders); these take the code graph pipeline instead of LLM extraction | built-in |
| **TextLoader** | `.txt` `.md` `.json` `.xml` `.yaml` `.yml` `.log` | built-in |
| **DltCsvLoader** | `.csv` — takes precedence over `CsvLoader` when installed; routes rows through [dlt structured ingestion](/integrations/dlt-integration) | `pip install cognee[dlt]` |
| **CsvLoader** | `.csv` | built-in |
| **PyPdfLoader** | `.pdf` | built-in |
| **ImageLoader** | `.png` `.jpg` `.jpe` `.jpeg` `.gif` `.webp` `.bmp` `.tif` `.tiff` `.heic` `.avif` `.ico` `.psd` `.apng` `.cr2` `.dwg` `.xcf` `.jxr` `.jpx` | built-in |
| **AudioLoader** | `.mp3` `.wav` `.aac` `.flac` `.ogg` `.m4a` `.mid` `.amr` `.aiff` | built-in |
| **UnstructuredLoader** | `.docx` `.doc` `.odt` `.xlsx` `.xls` `.ods` `.pptx` `.ppt` `.odp` `.rtf` `.html` `.htm` `.eml` `.msg` `.epub` | `pip install cognee[docs]` |
| **AdvancedPdfLoader** | `.pdf` with layout-aware extraction | `pip install cognee[docs]` |
| **BeautifulSoupLoader** | `.html` | `pip install cognee[scraping]` |
| **DoclingDocument** | pre-converted `DoclingDocument` objects | `pip install cognee[docling]` |
<Note>
This formats table applies only to permanent-memory `remember()`. In session-memory mode, `remember()` does not parse files or fetch URLs; it stores the provided content as session text, so plain text inputs are the recommended form. Non-text inputs become text representations rather than parsed source content.
</Note>
</Accordion>
<Accordion title="Users and ownership">
* Permanent-memory `remember()` writes data into a dataset owned by a user.
* If you do not pass a user explicitly, Cognee resolves or creates the default user context.
* Ownership affects who can later read, write, share, improve, or forget that dataset.
* In session-memory mode, entries are still scoped to the current user in the session cache, so the same `session_id` under different users does not mean shared memory.
If you need the deeper dataset-permissions model, see [Datasets](/core-concepts/further-concepts/datasets) and the multi-user docs.
</Accordion>
<Accordion title="What permanent remember produces">
When `remember()` runs without `session_id`, the result is not just stored text. It produces the full retrieval-ready memory stack:
* **Dataset records** in the relational store
* **Normalized source content** attached to that dataset
* **Chunks** created from the ingested content
* **Graph nodes and edges** extracted from those chunks
* **Embeddings and summaries** used for retrieval
* **Improvement artifacts** from the follow-up improve pass when `self_improvement=True`, depending on the active enrichment configuration
In practice, this means the dataset becomes immediately usable by [Recall](/core-concepts/main-operations/recall).
</Accordion>
<Accordion title="What session remember produces">
When `remember()` runs with `session_id`, it behaves differently:
* The content is written to the session cache for that user and session.
* The call returns quickly with a session-stored result.
* The content is available for session-aware [Recall](/core-concepts/main-operations/recall) immediately.
* No permanent graph write happens unless the improvement bridge runs.
So session memory is best thought of as **fast short-term memory first**, with optional long-term graph persistence second.
</Accordion>
<Accordion title="Session availability and configuration">
Session memory is **enabled by default** in the current Cognee 1.0 configuration.
* `session_id` works through the cache layer used for session storage.
* The default cache backend is filesystem-based cache.
* If you want to disable session caching entirely, set `CACHING=false`.
* If you want a shared or production-oriented backend, set `CACHE_BACKEND=redis`.
```dotenv theme={null}
# Disable session caching
CACHING=false
# Or keep sessions enabled and use Redis
CACHING=true
CACHE_BACKEND=redis
```
If session caching is disabled or unavailable, `remember(..., session_id=...)` cannot persist session memory in the normal way. For broader cache behavior and adapters, see [Sessions and Caching](/core-concepts/sessions-and-caching).
<Note>
If the session cache is unavailable, `remember(..., session_id=...)` can still return a session-style result object, but the session content may not actually have been stored. If session memory matters for your workflow, make sure caching is enabled and available.
</Note>
</Accordion>
<Accordion title="Latency and cost">
The two remember modes have very different cost and latency profiles:
* **Session memory** is the fast path. It writes to session cache and returns quickly.
* **Permanent memory** is the heavier path. It runs ingestion, chunking, graph extraction, embedding, and usually improvement.
That means permanent `remember()` is the right choice when you want durable graph memory, but it costs more time and more model work than session-only storage.
The biggest cost drivers are:
* How much data you ingest
* How many chunks the content becomes
* The graph extraction and summarization work during graph building
* Whether `self_improvement=True` adds a follow-up improvement pass
If you want more control over chunking or prompt behavior, use `chunk_size`, `chunker`, and `custom_prompt`. If you want the lightest permanent ingestion path, set `self_improvement=False`.
</Accordion>
<Accordion title="Background execution and result object">
* `remember()` returns a promise-like `RememberResult`.
* In blocking mode, it finishes only after the permanent pipeline completes.
* With `run_in_background=True`, it returns immediately and you can `await` the result later.
* Useful fields include status, dataset name, elapsed time, and the raw pipeline result.
<Note>
With `run_in_background=True`, file-like inputs (HTTP upload objects exposing `.file`/`.filename`, or stream-like objects with `.read()`) are read up-front and copied into owned in-memory buffers before the background task is scheduled. This lets background runs outlive the originating request or stream — you can pass an upload object or stream and let the call return immediately without the underlying stream being closed out from under the pipeline. Plain text strings, file paths, and URLs are unaffected.
</Note>
```python theme={null}
result = await cognee.remember(
"Einstein was born in Ulm.",
run_in_background=True,
)
print(result) # status='running'
await result
print(result) # status='completed'
```
</Accordion>
<Accordion title="Concurrent recall while indexing is running">
You can call [recall](/core-concepts/main-operations/recall) while a permanent-memory `remember()` run is still indexing in the background. Cognee does not place a global lock over ingestion and retrieval.
* Existing indexed data remains searchable while new data is being processed.
* Newly ingested content only becomes available to graph-backed `recall()` once its indexing work has completed.
* In practice, that means a background `remember()` run may produce **partial retrieval coverage** until the dataset finishes processing.
<Note>
Cognee runs as a single process. The embedded file-based defaults (Kuzu graph, SQLite, LanceDB) must not be shared across processes — a second process opening the same files can see stale or empty data. If more than one process or agent needs access to the same memory, the supported alternative is external stores: [Neo4j](/setup-configuration/graph-stores) for the graph and [Postgres](/setup-configuration/relational-databases) for the relational store (with [PGVector](/setup-configuration/vector-stores) for vectors), with all access going through a single Cognee service — see the [deployment guides](/how-to-guides/cognee-sdk/deployment).
</Note>
</Accordion>
<Accordion title="Checking indexing status">
When `remember()` writes to permanent memory, it runs a multi-step indexing pipeline in the background of that operation. For larger datasets, you may want to verify that indexing completed before calling [recall](/core-concepts/main-operations/recall).
<Tabs>
<Tab title="Python SDK">
```python theme={null}
import cognee
result = await cognee.remember(
"Cognee turns documents into AI memory.",
dataset_name="docs",
run_in_background=True,
)
dataset_id = result.dataset_id
status = await cognee.datasets.get_status([dataset_id])
if status.get(str(dataset_id)) == "DATASET_PROCESSING_COMPLETED":
answers = await cognee.recall(
query_text="What does Cognee do?",
datasets=["docs"],
)
```
</Tab>
<Tab title="HTTP API">
```http theme={null}
GET /api/v1/datasets/status?dataset=<dataset-uuid>
GET /api/v1/activity/pipeline-runs?dataset_id=<dataset-uuid>
```
* `GET /api/v1/datasets/status` returns the current indexing status for one or more datasets
* `GET /api/v1/activity/pipeline-runs` returns recent activity — pipeline runs plus single-row operation records, told apart by the `kind` field — with timestamps and run IDs
</Tab>
</Tabs>
Possible status values:
| Status | Meaning |
| ------------------------------ | -------------------------------- |
| `DATASET_PROCESSING_INITIATED` | Pipeline queued, not yet started |
| `DATASET_PROCESSING_STARTED` | Pipeline actively processing |
| `DATASET_PROCESSING_COMPLETED` | Indexing finished successfully |
| `DATASET_PROCESSING_ERRORED` | Pipeline encountered an error |
For the low-level API reference, see [cognee.datasets.get\_status()](/python-api/datasets#datasetsget_status).
</Accordion>
<Accordion title="RememberResult format">
`remember()` returns a `RememberResult` object. In docs terms, it behaves like a small status object you can print, inspect, or await.
**Common fields**
| Field | What it means |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `status` | Current state of the operation. Common values include `running`, `completed`, `errored`, and `session_stored`. A blocking permanent-memory run no longer reports `errored` by default — it raises `CognifyFailedError` instead, so you see `errored` on background runs or when you pass `raise_on_error=False`. See [Failed runs](/python-api/cognify#failed-runs). |
| `dataset_name` | The target dataset name used for the operation. |
| `dataset_id` | Dataset UUID when available. |
| `session_ids` | Session IDs associated with the result when session bridging is involved. |
| `pipeline_run_id` | Pipeline run UUID when the permanent pipeline has produced one. |
| `elapsed_seconds` | Wall-clock time from start to completion. |
| `items_processed` | Number of processed items when available. |
| `content_hash` | Content hash of the first processed item when available. |
| `items` | Per-item metadata such as IDs, names, token counts, MIME type, or content hashes when available. |
| `raw_result` | The raw pipeline result payload for advanced inspection. |
| `error` | Error text when the operation fails. |
**Typical shapes**
<AccordionGroup>
<Accordion title="Permanent memory completed">
```python theme={null}
{
"status": "completed",
"dataset_name": "main_dataset",
"dataset_id": "...",
"pipeline_run_id": "...",
"items_processed": 1,
"elapsed_seconds": 4.2
}
```
</Accordion>
<Accordion title="Session memory stored in cache">
```python theme={null}
{
"status": "session_stored",
"dataset_name": "main_dataset",
"session_ids": ["chat_1"],
"elapsed_seconds": 0.02
}
```
</Accordion>
<Accordion title="Background execution running">
```python theme={null}
{
"status": "running",
"dataset_name": "main_dataset"
}
```
</Accordion>
</AccordionGroup>
</Accordion>
<Accordion title="Parameters">
<Tabs>
<Tab title="Basic Parameters">
| Option | What it does |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `data` | The content to store. Can be a string (text, local path, `file://`, `http(s)://`, or `s3://` URL — a GitHub/GitLab repository URL is cloned and indexed as a code graph), a `DataItem`, an HTTP upload object exposing `.file`/`.filename`, or a list mixing any of these. Plain `open(..., "rb")` handles are not accepted — see "Accepted inputs". |
| `dataset_name` | Chooses the target permanent dataset. Defaults to `main_dataset`. |
| `session_id` | Switches `remember()` into session-memory mode. |
| `self_improvement` | Controls whether [Improve](/core-concepts/main-operations/improve) starts automatically after storage. In session mode, this is what enables or disables automatic bridging into the permanent graph. Defaults to `True`. |
| `run_in_background` | Starts the permanent pipeline asynchronously. |
| `chunk_size` / `chunker` | Customizes chunking for the permanent pipeline. |
| `custom_prompt` | Overrides the graph-extraction prompt used during graph building. |
| `session_ids` | Syncs new permanent graph knowledge back into specific sessions during the self-improvement pass. |
</Tab>
<Tab title="Advanced Parameters">
| Option | What it does |
| -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `dataset_id` | Targets a specific existing dataset by UUID. Takes precedence over `dataset_name` when both are supplied. Not supported for `MemorySource` imports or typed `MemoryEntry` payloads, which resolve by name only. |
| `graph_model` | Overrides the graph schema/model used during graph building. Defaults to `KnowledgeGraph`; pass a `DataPoint` subclass to constrain extraction to your own fields and relationships. Only the `llm` extractor fills a custom schema. See [Custom Graph Model](/guides/custom-graph-model). |
| `extractor` | Which implementation fills the extract-and-summarize step of the internal `cognify()` call — `"llm"` (default) or `"gliner"`, which builds the graph and the chunk summaries with a local GLiNER2 model instead of the LLM. Omitting it falls back to the `GRAPH_EXTRACTOR` setting. See [LLM-free extraction with GLiNER](/python-api/cognify#llm-free-extraction-with-gliner). |
| `node_set` | Attaches one or more node-set tags during permanent ingestion. |
| `preferred_loaders` | Controls which loaders are preferred when ingesting source files. |
| `incremental_loading` | Controls whether only new or changed data should be processed when reusing a dataset. |
| `data_per_batch` | Tunes batching for ingestion and pipeline processing in the current `remember()` API. See [Cognify](/core-concepts/main-operations/legacy-operations/cognify) for lower-level batching details. |
| `chunks_per_batch` | Tunes batching during chunk processing in the graph-building stage in the current `remember()` API. See [Cognify](/core-concepts/main-operations/legacy-operations/cognify) for lower-level batching details. |
| `raise_on_error` | Controls how a failed graph build is reported. Defaults to `True`, which raises `CognifyFailedError` out of a blocking permanent-memory run. Pass `False` to get a `RememberResult` with `status="errored"` back instead. Background runs always report on the result object. See [Failed runs](/python-api/cognify#failed-runs). |
| `user` | Runs `remember()` under a specific user context instead of the default user. This affects dataset ownership, permissions, and session-cache scoping. |
| `vector_db_config` / `graph_db_config` | Overrides database backend configuration for the vector or graph stores. |
| `primary_key` | dlt structured-ingestion option — names the column used for upsert/dedup behavior when `write_disposition="merge"` is used with database connection strings or dlt resources. Forwarded to the underlying ingestion step. For CSV files, pass it through `preferred_loaders=[{"dlt_csv_loader": {"primary_key": "..."}}]` instead. See [dlt integration](/integrations/dlt-integration). |
| `write_disposition` | dlt structured-ingestion option — `replace` (default), `merge`, or `append`. Forwarded to the underlying ingestion step. See [dlt integration](/integrations/dlt-integration). |
| `query` | dlt structured-ingestion option — a `SELECT` whose `WHERE` clause filters what is ingested from a database connection string. The `FROM` target may be schema-qualified and may carry a table alias; a `WHERE` that references the alias, and a `FROM` clause containing `JOIN`, raise a `ValueError`. Forwarded to the underlying ingestion step. See [dlt integration](/integrations/dlt-integration#database-connection-string). |
| `max_rows_per_table` | dlt structured-ingestion option — caps the number of rows pulled per table for a single call, overriding `DLT_MAX_ROWS_PER_TABLE` for that call. |
</Tab>
</Tabs>
</Accordion>
<Accordion title="Under the hood — legacy operations">
`remember()` runs [Add](/core-concepts/main-operations/legacy-operations/add) → [Cognify](/core-concepts/main-operations/legacy-operations/cognify) → [Improve](/core-concepts/main-operations/improve) under the hood.
Use the legacy operations directly when you need explicit control over each step — for example, to inspect intermediate results, tune pipeline parameters independently, or integrate ingestion and graph-building into a more complex workflow.
</Accordion>
<Accordion title="Inspect what you've stored">
After calling `remember()`, you can list datasets and browse their contents using `cognee.datasets`:
```python theme={null}
import cognee
# List all datasets
datasets = await cognee.datasets.list_datasets()
for ds in datasets:
print(ds.name, ds.id)
# List data items inside a dataset
items = await cognee.datasets.list_data(dataset_id=ds.id)
for item in items:
print(item.id, item.name)
```
See [datasets API reference](/python-api/datasets) for the full set of listing and management methods.
</Accordion>
<Columns>
<Card title="Recall" icon="search" href="/core-concepts/main-operations/recall">
Query memory with auto-routing and session awareness
</Card>
<Card title="Improve" icon="sparkles" href="/core-concepts/main-operations/improve">
Enrich the graph and bridge session memory
</Card>
</Columns>
# Serve
Source: https://docs.cognee.ai/core-concepts/main-operations/serve
Connect the Cognee SDK to Cognee Cloud or a remote Cognee instance.
## What is the serve operation
The `.serve` operation connects your local Cognee Python SDK to Cognee Cloud or another remote Cognee instance.
After `cognee.serve()` connects, SDK calls such as `remember()`, `recall()`, `improve()`, and `forget()` run against the remote instance instead of local storage. This lets the same Python code work with local Cognee during development and with a hosted or self-hosted Cognee backend in production.
```python theme={null}
import cognee
await cognee.serve()
```
The CLI command `cognee serve` starts a local Cognee backend process. The Python function `cognee.serve()` connects the SDK client to a remote or local backend.
## Where serve fits
* Use `serve()` when you want SDK operations to target Cognee Cloud.
* Use it when you want to connect to a self-hosted Cognee API server.
* Use it before [Push](/core-concepts/main-operations/push) when you want `push()` to reuse saved or active remote credentials.
* Use `disconnect()` when you want the SDK to return to local execution.
* Use [syncing a local instance](/cognee-cloud/connections/syncing-local-instance) for a cloud-focused walkthrough of the same connection flow.
## What happens under the hood
1. **Resolve connection settings** - Cognee looks for an explicit URL/API key, environment variables, saved credentials, or the device login flow (which needs `COGNEE_AUTH0_DEVICE_CLIENT_ID`). It then probes an authenticated endpoint on the instance, so a rejected API key raises `CogneeConfigurationError` here instead of failing on the first operation (see [Connection resolution](/python-api/serve#connection-resolution)).
2. **Create a remote client** - the SDK stores a client that knows how to call the remote Cognee API.
3. **Route SDK operations remotely** - supported high-level operations execute against the connected instance.
4. **Persist credentials when applicable** - Cloud login credentials can be saved and reused on later runs.
5. **Disconnect on request** - `cognee.disconnect()` clears the active remote client and returns the SDK to local mode.
### Reconnecting with saved credentials
On the Cloud path — `serve()` called without a `url` — Cognee tries the saved instance before it tries Auth0. When `~/.cognee/cloud_credentials.json` holds both a service URL and an API key, the SDK health-checks that URL and then probes an authenticated endpoint with the cached API key, without first consulting the stored Auth0 access token's expiry. If the instance responds and accepts the key, `serve()` connects with the cached credentials and skips Auth0 entirely, so an expired token on its own does not force a new login.
Auth0 is contacted only when the instance does not respond or rejects the cached key:
* **Stored token still valid** — Cognee goes straight to the device-code login.
* **Stored token expired and a refresh token is saved** — Cognee refreshes the token and repeats both checks against the saved URL; the device login follows only if that retry also fails.
The device login needs a device client ID (`COGNEE_AUTH0_DEVICE_CLIENT_ID` or the `auth0_client_id` argument). Without one, `serve()` raises `CogneeConfigurationError` naming the credentials file and the account on it instead of attempting to log in.
The practical effect is that a reachable instance with a working key keeps starting up during an Auth0 outage, and a failed connection points at the instance or the key rather than at token freshness.
### Request timeouts
Remote calls are bounded so a stalled connection cannot hang your process indefinitely:
* **Ordinary operations** (`remember()`, `recall()`, `improve()`, `add()`, `cognify()`, `search()`, `forget()`, `update()`, `datasets.list_data()`) allow up to **600 seconds** total per request, with connection failures surfacing after **30 seconds**. The 600 second total gives long blocking server-side work — for example `cognify()` over a large dataset — room to finish.
* **Archive uploads** used by [`push()`](/core-concepts/main-operations/push) have **no total cap**, and are instead bounded by **600 seconds of read inactivity**. An upload plus its synchronous server-side import can legitimately outlast any fixed total, so the upload is cut off only when the server stops sending data.
<Note>
These timeouts are fixed values in the remote client. There is no environment variable or `serve()` parameter to change them. If a blocking `cognify()` run is likely to exceed 600 seconds, submit it as a background run and poll for status instead of holding the request open.
</Note>
### Filenames for raw-text uploads
When you pass a raw string — or a list containing strings — to `remember()`, `add()`, or `update()` while connected, the remote client uploads it as a file named `text_<md5_hash>.txt`, where the hash is computed from that string's UTF-8 bytes. Each string in a list gets its own hash-derived name. This matches the naming local ingestion already uses for nameless text (see [Hash-based file storage, deduplication, and filename collisions](/core-concepts/main-operations/legacy-operations/add#hash-based-file-storage-deduplication-and-filename-collisions)), so the same text produces the same object name whether it is ingested locally or over a `serve()` connection.
File-like objects are unaffected: they keep uploading under their own `name` attribute, falling back to `upload` when they have none.
Because every string is uploaded as content, a **file path passed as a string is uploaded as that path's text**, not as the file it points at. Local ingestion resolves `"/path/to/document.pdf"`, `"file://…"`, and `"s3://…"` to the file; the remote client does not. Pass an open file object when you want the file's contents to cross the connection. The exceptions are `remember()`'s `content_type="code"` and `content_type="skills"`, which are path-based by design — code specs are sent as strings for the server to clone or read itself, and skill directories are walked locally so each `SKILL.md` is uploaded by content.
<Note>
Previously every raw-text upload was sent as `data.txt`. Because the name was fixed, all text uploads for a tenant landed on one remote object, so concurrent adds raced each other against the server's content-hash read-back and could fail with a `FileContentHashingError` 409. Content-derived names remove that collision.
Like the timeouts, this filename is a fixed value in the remote client — there is no parameter to supply your own. If you have tooling or tests that assert the uploaded name is `data.txt`, update them to expect `text_<md5_hash>.txt`. The uploaded text content itself is unchanged.
</Note>
## Connection modes
<Tabs>
<Tab title="Cognee Cloud">
Call `serve()` without arguments to use the Cognee Cloud device login flow. It needs a device client ID; without `COGNEE_AUTH0_DEVICE_CLIENT_ID` set, a bare `serve()` raises `CogneeConfigurationError` instead of opening the login.
```bash theme={null}
export COGNEE_AUTH0_DEVICE_CLIENT_ID="your-device-client-id"
```
```python theme={null}
import cognee
await cognee.serve()
```
After login, Cognee stores reusable credentials at `~/.cognee/cloud_credentials.json`.
</Tab>
<Tab title="Explicit credentials">
For self-hosted, staging, or non-interactive environments, pass the remote URL and API key directly.
```python theme={null}
await cognee.serve(
url="https://your-tenant.aws.cognee.ai",
api_key="your-api-key",
)
```
</Tab>
<Tab title="Environment variables">
Use environment variables when you do not want credentials in code.
```bash theme={null}
export COGNEE_SERVICE_URL="https://your-tenant.aws.cognee.ai"
export COGNEE_API_KEY="your-api-key"
```
```python theme={null}
await cognee.serve()
```
</Tab>
<Tab title="Local backend">
Start a local backend with the CLI, then connect the SDK to it. A server with its default posture requires authentication, so pass an API key; omit it only when the server runs with `ENABLE_BACKEND_ACCESS_CONTROL=false`.
```bash theme={null}
cognee serve
```
```python theme={null}
await cognee.serve(url="http://localhost:8000", api_key="your-api-key")
```
</Tab>
</Tabs>
## After serve connects
Supported SDK operations run on the connected remote instance.
```python theme={null}
await cognee.serve(
url="https://your-tenant.aws.cognee.ai",
api_key="your-api-key",
)
await cognee.remember("Einstein developed general relativity in 1915.")
results = await cognee.recall("What did Einstein develop?")
await cognee.disconnect()
```
<Note>
`serve()` changes where SDK operations execute. It does not copy local datasets to the remote instance by itself. Use [Push](/core-concepts/main-operations/push) to upload an already-built local graph, or run `remember()` while connected to ingest data directly into the remote instance.
</Note>
## Examples and details
<Accordion title="Cloud login">
```python theme={null}
import cognee
# Opens the device login flow and discovers your Cloud tenant.
# Requires COGNEE_AUTH0_DEVICE_CLIENT_ID; otherwise pass url= and api_key=.
await cognee.serve()
await cognee.remember("Cloud memory note")
results = await cognee.recall("What notes are stored?")
```
</Accordion>
<Accordion title="Self-hosted server">
```python theme={null}
import cognee
await cognee.serve(
url="https://memory.example.com",
api_key="ck_live_...",
)
await cognee.remember("Runbook: deploys happen on Tuesdays.")
```
</Accordion>
<Accordion title="Return to local mode">
```python theme={null}
await cognee.disconnect()
# This now runs against local Cognee storage again.
await cognee.remember("Local-only note")
```
</Accordion>
## See also
* [Push](/core-concepts/main-operations/push)
* [Syncing a local instance](/cognee-cloud/connections/syncing-local-instance)
* [Cloud SDK](/cognee-cloud/connections/cloud-sdk)
* [Deploy REST API server](/guides/deploy-rest-api-server)
# Dataset Database Handlers: How to use them?
Source: https://docs.cognee.ai/core-concepts/multi-user-mode/dataset-database-handlers/dataset-database-handlers-how-to-use-them
Learn how to use Dataset Database Handlers in Cognee multi-user mode.
## Core Responsibilities and Lifecycle
All handlers implement the `DatasetDatabaseHandlerInterface`, which defines three lifecycle entry points.
<Accordion title="Handler Interface Methods">
Handlers must implement the following methods:
* **`create_dataset(dataset_id, user) -> dict`**
Creates or resolves backing storage for the dataset and returns a dictionary of connection and identification fields.
This may:
* Provision new infrastructure (e.g., create a Neo4j Aura instance), or
* Return connection details to an existing shared or pooled backend.
* **`resolve_dataset_connection_info(dataset_database) -> DatasetDatabase`** *(optional override)*
Converts stored references into runtime-ready connection details.
Typical use cases include:
* Decrypting stored secrets
* Fetching short-lived access tokens
The default implementation returns the input unchanged.
* **`delete_dataset(dataset_database) -> None`**
Deprovisions, deletes, or prunes the dataset’s backing storage.
</Accordion>
## Persisted Dataset Database Records
The dictionary returned from `create_dataset()` is persisted as a `DatasetDatabase` row in the relational database and later merged into runtime connection flows.
Typical stored fields include:
<Accordion title="Vector database fields">
* `vector_database_provider`
* `vector_database_url`
* `vector_database_key`
* `vector_database_name`
* `vector_dataset_database_handler`
* `vector_database_connection_info`
*(JSON dictionary for extended or sensitive parameters such as usernames, passwords, or custom options)*
</Accordion>
<Accordion title="Graph database fields">
* `graph_database_provider`
* `graph_database_url`
* `graph_database_key`
* `graph_database_name`
* `graph_dataset_database_handler`
* `graph_database_connection_info`
*(JSON dictionary for extended or sensitive parameters such as usernames, passwords, or custom options)*
</Accordion>
## Where Dataset Database Handlers Are Used
Handlers are invoked automatically by the system based on configuration.
* **Vector storage**
* Selected via `VECTOR_DATASET_DATABASE_HANDLER` environment variable
* **Graph storage**
* Selected via `GRAPH_DATASET_DATABASE_HANDLER` environment variable
When a dataset is accessed:
* A new `DatasetDatabase` row is created if one does not already exist.
* The handler name, provider, and connection metadata are stored for reuse.
* At runtime, the handler may resolve secrets or transform stored references before connections are opened.
* On dataset deletion, the handler is responsible for cleaning up the underlying storage. The dataset deletion call happens when pruning data in Cognee and when datasets are explicitly deleted.
## List of Supported Dataset Database Handlers:
### Core handlers included with Cognee:
**Graph handlers:**
* neo4j\_aura\_dev → Neo4j Aura development cloud handler
* neo4j → Neo4j graph handler, one database per dataset (requires Neo4j Enterprise or AuraDB; on Community edition it raises `Neo4jMultiDatabaseSupportError` — use `neo4j_community` instead)
* neo4j\_community → [Neo4j Community handler](/core-concepts/multi-user-mode/dataset-database-handlers/existing-dataset-database-handlers/neo4j-community) — one Neo4j Community Docker container per dataset (requires Docker)
* ladybug / kuzu → embedded Ladybug (formerly Kuzu) graph handler — both names register the same handler
* postgres\_graph → Postgres graph handler, one database per dataset (demo — the [Postgres graph store](/setup-configuration/graph-stores) is not production-ready)
* postgres\_graph\_shared → Postgres graph handler that isolates each dataset in its own schema (`ds_<dataset_id>`) of Cognee's relational database instead of provisioning a database per dataset; requires `GRAPH_DATABASE_PROVIDER=postgres_demo` (the alias `postgres` is also accepted)
* turso\_graph → Turso graph handler, one libSQL file per dataset at `<SYSTEM_ROOT_DIRECTORY>/databases/graph_<dataset_id>.db`, so `SYSTEM_ROOT_DIRECTORY` must be an **absolute local filesystem path** — a non-local root such as an `s3://…` one, or a relative path assigned through `cognee.config.system_root_directory()`, raises `EnvironmentError` at dataset creation, before any directory is made. Deleting a dataset evicts the cached graph engine by database name and waits for it to close before removing that file together with its `-wal`/`-shm` WAL companions
**Vector handlers:**
* lancedb → LanceDB vector handler
* pgvector → PGVector vector handler, one database per dataset
* pgvector\_shared → [PGVector vector handler](/core-concepts/multi-user-mode/dataset-database-handlers/existing-dataset-database-handlers/pgvector) that isolates each dataset in its own schema (`ds_<dataset_id>`) of Cognee's relational database instead of provisioning a database per dataset; requires `VECTOR_DB_PROVIDER=pgvector`
* turso → Turso vector handler
<Note>
Each handler declares which database provider it works with, and Cognee raises an `EnvironmentError` at startup when the selected handler does not match the configured provider. The Ladybug handler accepts **both** provider names: `GRAPH_DATASET_DATABASE_HANDLER` set to either `ladybug` or `kuzu` works with `GRAPH_DATABASE_PROVIDER` set to either `ladybug` or `kuzu`, so the current and legacy names can be mixed without tripping that check.
</Note>
### Community-contributed handlers from the [community repository](https://github.com/topoteretes/cognee-community):
* qdrant → Qdrant vector handler using local docker
* falkor\_vector\_local -> FalkorDB vector handler using local docker
* falkor\_graph\_local -> FalkorDB graph handler using local docker
## Using Custom Dataset Database Handlers
You can add your own at runtime with a register function, then point configuration to your handler name.
For example:
```python theme={null}
# --- STEP 1: Tell Cognee to look for custom handlers ---
# Note: This would typically be done in a .env file or environment configuration, it has to be done before Cognee initialization
os.environ["VECTOR_DATASET_DATABASE_HANDLER"] = "custom_vector_handler"
os.environ["GRAPH_DATASET_DATABASE_HANDLER"] = "custom_graph_handler"
# --- STEP 2: Import your actual custom logic ---
from my_custom_handlers import CustomVectorDatasetDatabaseHandler, CustomGraphDatasetDatabaseHandler
# --- STEP 3: Register the handlers ---
from cognee.infrastructure.databases.dataset_database_handler.use_dataset_database_handler import use_dataset_database_handler
# Register custom vector database handler
use_dataset_database_handler(
"custom_vector_handler", # -> Name to register for the handler, should match the env var above
CustomVectorDatasetDatabaseHandler, # -> Your custom class implementing DatasetDatabaseHandlerInterface
"vector_db_name" # -> What is the vector database provider for this handler
)
# Register custom graph database handler
use_dataset_database_handler(
"custom_graph_handler", # -> Name to register for the handler, should match the env var above
CustomGraphDatasetDatabaseHandler, # -> Your custom class implementing DatasetDatabaseHandlerInterface
"graph_db_name" # -> What is the graph database provider for this handler
)
```
By writing your own Dataset Database Handlers, you can integrate Cognee with any graph or vector storage backend while maintaining clean separation of concerns and secure handling of connection details. This extensibility allows Cognee to adapt to a wide range of deployment scenarios and infrastructure setups (AWS, GCP, Azure, Local and etc.).
* For a more detailed example of writing a custom handler, see the [Custom Dataset Database Handlers Example](https://github.com/topoteretes/cognee/blob/main/cognee/tests/test_dataset_database_handler.py).
* For an example of an existing handler implementation, see the [Neo4j Aura Dev Dataset Database Handler source code](https://github.com/topoteretes/cognee/blob/main/cognee/infrastructure/databases/graph/neo4j_driver/Neo4jAuraDevDatasetDatabaseHandler.py).
## Dataset Database Handler Interface
Below is the full interface definition that all Dataset Database Handlers must implement along with docstrings explaining each method and its purpose.
```python theme={null}
class DatasetDatabaseHandlerInterface(ABC):
@classmethod
@abstractmethod
async def create_dataset(cls, dataset_id: Optional[UUID], user: Optional[User]) -> dict:
"""
Return a dictionary with database connection/resolution info for a graph or vector database for the given dataset.
Function can auto handle deploying of the actual database if needed, but is not necessary.
Only providing connection info is sufficient, this info will be mapped when trying to connect to the provided dataset in the future.
Needed for Cognee multi-tenant/multi-user and backend access control support.
Dictionary returned from this function will be used to create a DatasetDatabase row in the relational database.
From which internal mapping of dataset -> database connection info will be done.
The returned dictionary is stored verbatim in the relational database and is later passed to
resolve_dataset_connection_info() at connection time. For safe credential handling, prefer
returning only references to secrets or role identifiers, not plaintext credentials.
Each dataset needs to map to a unique graph or vector database when backend access control is enabled to facilitate a separation of concern for data.
Args:
dataset_id: UUID of the dataset if needed by the database creation logic
user: User object if needed by the database creation logic
Returns:
dict: Connection info for the created graph or vector database instance.
"""
pass
@classmethod
async def resolve_dataset_connection_info(
cls, dataset_database: DatasetDatabase
) -> DatasetDatabase:
"""
Resolve runtime connection details for a dataset’s backing graph/vector database.
Function is intended to be overwritten to implement custom logic for resolving connection info.
This method is invoked right before the application opens a connection for a given dataset.
It receives the DatasetDatabase row that was persisted when create_dataset() ran and must
return a modified instance of DatasetDatabase with concrete connection parameters that the client/driver can use.
Do not update these new DatasetDatabase values in the relational database to avoid storing secure credentials.
In case of separate graph and vector database handlers, each handler should implement its own logic for resolving
connection info and only change parameters related to its appropriate database, the resolution function will then
be called one after another with the updated DatasetDatabase value from the previous function as the input.
Typical behavior:
- If the DatasetDatabase row already contains raw connection fields (e.g., host/port/db/user/password
or api_url/api_key), return them as-is.
- If the row stores only references (e.g., secret IDs, vault paths, cloud resource ARNs/IDs, IAM
roles, SSO tokens), resolve those references by calling the appropriate secret manager or provider
API to obtain short-lived credentials and assemble the final connection DatasetDatabase object.
- Do not persist any resolved or decrypted secrets back to the relational database. Return them only
to the caller.
Args:
dataset_database: DatasetDatabase row from the relational database
Returns:
DatasetDatabase: Updated instance with resolved connection info
"""
return dataset_database
@classmethod
@abstractmethod
async def delete_dataset(cls, dataset_database: DatasetDatabase) -> None:
"""
Delete the graph or vector database for the given dataset.
Function should auto handle deleting of the actual database or send a request to the proper service to delete/mark the database as not needed for the given dataset.
Needed for maintaining a database for Cognee multi-tenant/multi-user and backend access control.
Args:
dataset_database: DatasetDatabase row containing connection/resolution info for the graph or vector database to delete.
"""
pass
```
# Dataset Database Handlers: What are they?
Source: https://docs.cognee.ai/core-concepts/multi-user-mode/dataset-database-handlers/dataset-database-handlers-what-are-they
Learn how Cognee maps datasets to graph and vector storage backends.
# What is a Dataset Database Handler?
Dataset Database Handlers are small, pluggable classes that define how a Cognee dataset maps to concrete storage backends for:
* **Graph databases**
* **Vector databases**
They act as the abstraction layer between a logical dataset and its physical storage.
Extensible by design: You can write custom handlers to integrate Cognee with any database and database backend (AWS, Azure, GCP, local setups and etc.) without modifying core code.
<img alt="Dataset Database Handler Diagram" />
In this diagram, we can see that each dataset is mapped to its own unique Neo4j graph database instance using a Neo4j Aura Dataset Database Handler.
This connection is automatically resolved at runtime based on the authenticated user and dataset being accessed and is done when recalling, remembering, forgetting, creating, or otherwise modifying a dataset.
<Info> If you're using Cognee with `ENABLE_BACKEND_ACCESS_CONTROL` set to `False`, you don't need to configure handlers. </Info>
## How graph isolation works
Dataset Database Handlers matter specifically in multi-user mode, because that is when Cognee stops treating datasets as logical partitions inside one shared graph and starts routing each dataset to its own storage backend.
| | Single-user mode | Multi-user mode |
| ------------------------ | ---------------------------- | -------------------------------------- |
| **Graph databases** | One shared instance | One per dataset |
| **Separation mechanism** | `dataset_id` metadata filter | Physical database boundary |
| **Handler required** | No | Yes (`GRAPH_DATASET_DATABASE_HANDLER`) |
<Tabs>
<Tab title="Single-User Mode">
All datasets share one physical graph database. There are no hard graph-level boundaries between them. Dataset separation is tracked logically in the relational database through `dataset_id`, and queries are scoped using that metadata at runtime.
</Tab>
<Tab title="Multi-User Mode">
Each dataset is routed to its own dedicated physical graph database. When a dataset is first accessed, Cognee invokes the configured graph dataset database handler to provision or resolve the correct backend for that dataset. Connection metadata is stored in the relational database and resolved at query time, so each dataset's graph operations run against an isolated database instance.
</Tab>
</Tabs>
## What Dataset Database Handlers Do
Handlers encapsulate all backend-specific logic required to manage dataset storage, including:
* **Provisioning or resolution** of per-dataset storage
* **Runtime connection resolution**, such as:
* Secret decryption
* Fetching short-lived credentials
* **Teardown and deletion** of per-dataset storage
Handlers are selected through configuration and registered in a central handler registry. This allows Cognee to support multiple providers and custom storage setups without modifying core pipeline code.
## Why Dataset Database Handlers Exist
Dataset Database Handlers solve several core system needs:
* **Multi-tenant isolation** — Each dataset can map to its own graph and/or vector database, enabling clean separation when backend access control is enabled.
* **Pluggability** — Providers like LanceDB, Kùzu, or Neo4j Aura can be added or swapped without changing application logic.
* **Secure secret handling** — Credentials can be resolved at connection time instead of being stored in plaintext in the relational database.
* **Lifecycle control** — All create, resolve, and delete semantics for a backend live in one well-defined place.
<Warning> Cognee requires database instances provisioned via the Dataset Database Handler to be active and running. While Cognee facilitates the creation and deletion of these databases, it does not manage the operational lifecycle—such as starting or stopping containers—during its own startup or shutdown processes. </Warning>
<Columns>
<Card title="Datasets" icon="user" href="/core-concepts/multi-user-mode/permissions-system/datasets">
Learn about datasets as the core unit of storage in Cognee
</Card>
<Card title="Dataset Database Handlers: How to use them" icon="building" href="/core-concepts/multi-user-mode/dataset-database-handlers/dataset-database-handlers-how-to-use-them">
Deep dive into configuring and using dataset database handlers
</Card>
</Columns>
# FalkorDB Dataset Database Handler
Source: https://docs.cognee.ai/core-concepts/multi-user-mode/dataset-database-handlers/existing-dataset-database-handlers/falkor
Handler for using FalkorDB in Cognee multi-user mode.
The FalkorDB adapter is one of Cognee's community adapters, which can be found on our [community repo](https://github.com/topoteretes/cognee-community/tree/main/packages/hybrid/falkordb).
<Warning>
Make sure that `ENABLE_BACKEND_ACCESS_CONTROL` in your `.env` file is **NOT** set to `False`.
Multi-user mode is enabled by default, therefore `ENABLE_BACKEND_ACCESS_CONTROL=True` by default.
</Warning>
## Installation
Firstly, you will need to install the adapter package:
```bash theme={null}
pip install cognee-community-hybrid-adapter-falkor
```
## Setup
You will need a running FalkorDB database instance, and the existing handler works with local setups.
All you need to do is provide the necessary connection information, like the following example:
```dotenv theme={null}
VECTOR_DB_PROVIDER=falkordb
VECTOR_DB_URL=localhost
VECTOR_DB_PORT=6379
GRAPH_DATABASE_PROVIDER=falkor
GRAPH_DATABASE_URL=localhost
GRAPH_DATABASE_PORT=6379
```
Falkor is a hybrid vector/graph store, therefore we need to provide Cognee with information about both
databases, even though it is the same Falkor database instance. You can run a local instance with the
following command:
```bash theme={null}
docker run -p 6379:6379 -p 3000:3000 -it --rm falkordb/falkordb:edge
```
## Usage
After setting up your instance and connection information, you will need to register the adapter and handler to Cognee,
so it knows which database to use. Registering the adapter also registers the dataset database handler.
In your code, add the following import statement:
```bash theme={null}
from cognee_community_hybrid_adapter_falkor import register
```
The final important piece of information is to let Cognee know which handler you are using.
This can be done by setting the following `.env` variables:
```dotenv theme={null}
GRAPH_DATASET_DATABASE_HANDLER="falkor_graph_local"
VECTOR_DATASET_DATABASE_HANDLER="falkor_vector_local"
```
Since Falkor is a hybrid adapter, we have to set variables for both vector and graph databases.
<CardGroup>
<Card title="Falkor Adapter" icon="book" href="/setup-configuration/community-maintained/falkordb">
Details About Cognee's FalkorDB Adapter
</Card>
<Card title="Multi-User Overview" icon="users" href="/core-concepts/multi-user-mode/multi-user-mode-overview">
More Details About Multi-User Mode
</Card>
</CardGroup>
# Kuzu Dataset Database Handler
Source: https://docs.cognee.ai/core-concepts/multi-user-mode/dataset-database-handlers/existing-dataset-database-handlers/kuzu
Handler for using Kuzu in Cognee multi-user mode.
<Warning>
Make sure that `ENABLE_BACKEND_ACCESS_CONTROL` in your `.env` file is **NOT** set to `False`.
Multi-user mode is enabled by default, therefore `ENABLE_BACKEND_ACCESS_CONTROL=True` by default.
</Warning>
The Kuzu adapter is one of Cognee's core vector adapters, along with Neo4j.
## Installation
Kuzu is part of Cognee's core dependencies, so no extra installation steps are required.
## Setup and Usage
Since Kuzu is a file-based database, and its handler is registered in Cognee by default,
all that needs to be set is the information about the provider, and to let Cognee know which
handler you are using:
```dotenv theme={null}
GRAPH_DATABASE_PROVIDER="kuzu"
GRAPH_DATASET_DATABASE_HANDLER="kuzu"
```
Graph files are stored by default in the .cognee\_system folder (within the Cognee package), with each dataset assigned its own persistent graph file.
<CardGroup>
<Card title="Graph Stores" icon="book" href="/setup-configuration/graph-stores">
Details About Cognee's Graph Stores
</Card>
<Card title="Multi-User Overview" icon="users" href="/core-concepts/multi-user-mode/multi-user-mode-overview">
More Details About Multi-User Mode
</Card>
</CardGroup>
# LanceDB Dataset Database Handler
Source: https://docs.cognee.ai/core-concepts/multi-user-mode/dataset-database-handlers/existing-dataset-database-handlers/lancedb
Handler for using LanceDB in Cognee multi-user mode.
<Warning>
Make sure that `ENABLE_BACKEND_ACCESS_CONTROL` in your `.env` file is **NOT** set to `False`.
Multi-user mode is enabled by default, therefore `ENABLE_BACKEND_ACCESS_CONTROL=True` by default.
</Warning>
The LanceDB adapter is one of Cognee's core vector adapters, along with PGVector.
## Installation
LanceDB is part of Cognee's core dependencies, so no extra installation steps are required.
## Setup and Usage
Since LanceDB is a file-based database, and its handler is registered in Cognee by default,
all that needs to be set is the information about the provider, and to let Cognee know which
handler you are using:
```dotenv theme={null}
VECTOR_DB_PROVIDER="lancedb"
VECTOR_DATASET_DATABASE_HANDLER="lancedb"
```
LanceDB vector files are stored by default in the .cognee\_system folder (within the Cognee package), with each dataset assigned its own persistent vector database file.
<Note>
When the handler provisions a dataset, it ensures the per-user databases directory (`<system_root_directory>/databases/<user_id>`) exists through Cognee's storage abstraction rather than a direct local `mkdir`. Provisioning therefore routes to the correct backend based on your configured path: it creates a local filesystem directory for local `system_root_directory` values (ensure the process has write permission; defaults to `.cognee_system`), and uses S3 object storage when `system_root_directory` is an `s3://` URI. In the standard S3 setup, `STORAGE_BACKEND=s3` is used together with S3 data and system roots, so LanceDB dataset provisioning no longer fails for S3-backed targets.
</Note>
## Schema Migrations
When you upgrade Cognee and the LanceDB payload schema changes, `run_migrations()` migrates existing collections to the new schema. In multi-user mode this runs for every dataset database automatically.
**Migration behavior for existing rows:**
* If all existing rows can be backfilled to the new schema (including required fields that have defaults or types that admit a safe fallback), the migration completes and the collection is updated in place.
* If any existing row cannot be backfilled — for example because a newly added field is required and has no default — the migration **aborts with a `RuntimeError`** and leaves the collection unchanged.
The error message names the collection and how many rows failed, along with sample validation errors. To resolve it:
1. Add an explicit default value to the new required field in your DataPoint model, **or**
2. Make the new field `Optional[YourType]` (which defaults to `None`).
Then re-run `run_migrations()`.
<Note>
If a dataset database migration fails, Cognee logs the error and continues migrating the remaining dataset databases. The failed dataset is left in its original state and can be retried after you fix the schema.
</Note>
## Local Cleanup with prune
When you wipe the vector store — for example via `cognee.prune.prune_system(vector=True)` — the LanceDB handler also removes its on-disk database directory for local filesystem paths. This local directory cleanup now works for any local path regardless of OS path format, including Windows paths (e.g. `C:\...`) and relative paths; previously it only triggered for POSIX absolute paths beginning with `/`, leaving the directory behind on Windows or when a relative path was configured.
Removal is skipped for remote-backed LanceDB instances, identified by a URL scheme of `db://`, `http://`, `https://`, `s3://`, `gs://`, or `az://`.
<Note>
If you ran cleanup on Windows or with a relative LanceDB path on an earlier version, leftover database directories may remain on disk. Re-running the prune workflow on this version removes them.
</Note>
<CardGroup>
<Card title="Vector Stores" icon="book" href="/setup-configuration/vector-stores">
Details About Cognee's Vector Stores
</Card>
<Card title="Multi-User Overview" icon="users" href="/core-concepts/multi-user-mode/multi-user-mode-overview">
More Details About Multi-User Mode
</Card>
</CardGroup>
# Neo4j Aura Dataset Database Handler
Source: https://docs.cognee.ai/core-concepts/multi-user-mode/dataset-database-handlers/existing-dataset-database-handlers/neo4j-aura-dev
Handler for using Neo4j Aura in Cognee multi-user mode.
<Warning>
Make sure that `ENABLE_BACKEND_ACCESS_CONTROL` in your `.env` file is **NOT** set to `False`.
Multi-user mode is enabled by default, therefore `ENABLE_BACKEND_ACCESS_CONTROL=True` by default.
</Warning>
The Neo4j adapter is one of Cognee's core graph adapters, along with Kuzu. Multi-user mode with Neo4j
requires a dataset database handler: this one, which provisions a Neo4j Aura Cloud instance per dataset,
or the [`neo4j_community` handler](/core-concepts/multi-user-mode/dataset-database-handlers/existing-dataset-database-handlers/neo4j-community),
which runs a local Neo4j Community Docker container per dataset instead. You can read more about Aura
in the official [Neo4j Aura docs](https://neo4j.com/docs/aura/).
## Local vs. Cloud Storage
By default, Cognee stores graph data locally using Kuzu (a file-based database). When you switch to Neo4j,
your data moves to a Neo4j server — which can be local (self-hosted) or cloud-based (Neo4j Aura).
* **Self-hosted Neo4j** — You run Neo4j yourself (locally or on a server). Data stays on that server.
* **Neo4j Aura** — Neo4j's fully managed cloud service. Data is stored in Neo4j's cloud infrastructure; nothing is saved locally on your machine. This is useful for teams, production deployments, or when you need managed backups.
## Installation
Both options use Neo4j as the graph database provider, so install the Neo4j dependencies first:
```bash theme={null}
pip install "cognee[neo4j]"
```
## Two Ways to Use Neo4j Aura
<Tabs>
<Tab title="Connect to an Existing Aura Instance">
If you already have a Neo4j Aura account and a database instance, you can point Cognee at it
directly using the connection URI from your Aura console:
```dotenv theme={null}
GRAPH_DATABASE_PROVIDER="neo4j"
GRAPH_DATABASE_URL="neo4j+s://<your-instance-id>.databases.neo4j.io"
GRAPH_DATABASE_NAME="neo4j"
GRAPH_DATABASE_USERNAME="neo4j"
GRAPH_DATABASE_PASSWORD="<your-aura-password>"
```
This is the simplest approach. All datasets share the same Aura instance.
For the general Neo4j provider setup and more context on using a single shared Aura database,
see [Graph Stores](/setup-configuration/graph-stores#neo4j-aura-cloud).
</Tab>
<Tab title="Auto-Provisioned Aura Instances per Dataset (this handler)">
The `Neo4jAuraDevDatasetDatabaseHandler` goes further: it automatically creates a dedicated
Neo4j Aura instance for each Cognee dataset via the Neo4j Aura API, and deletes it when the
dataset is removed. This enables strong per-dataset isolation for multi-user deployments.
This handler requires **Neo4j Aura API OAuth credentials**, not a regular database password.
### Getting Aura API Credentials
To use the auto-provisioning handler you need OAuth credentials from the Neo4j Aura console:
1. Log in to [console.neo4j.io](https://console.neo4j.io)
2. Navigate to **Account → API Keys**
3. Create a new API key to obtain a `client_id` and `client_secret`
4. Find your **Tenant ID** on the Aura console home page or under account settings
### Environment Variables
Set the following variables in your `.env` file:
```dotenv theme={null}
GRAPH_DATABASE_PROVIDER="neo4j"
NEO4J_CLIENT_ID=<your_oauth_client_id>
NEO4J_CLIENT_SECRET=<your_oauth_client_secret>
NEO4J_TENANT_ID=<your_aura_tenant_id>
NEO4J_ENCRYPTION_KEY=<a_secret_key_for_encrypting_stored_credentials>
GRAPH_DATASET_DATABASE_HANDLER="neo4j_aura_dev"
```
| Variable | Required | Description |
| -------------------------------- | ----------- | ----------------------------------------------------- |
| `NEO4J_CLIENT_ID` | Yes | OAuth client ID from the Neo4j Aura API console |
| `NEO4J_CLIENT_SECRET` | Yes | OAuth client secret from the Neo4j Aura API console |
| `NEO4J_TENANT_ID` | Yes | Your Neo4j Aura tenant (organization) ID |
| `NEO4J_ENCRYPTION_KEY` | Recommended | Key used to encrypt database passwords at rest |
| `GRAPH_DATASET_DATABASE_HANDLER` | Yes | Selects the `neo4j_aura_dev` dataset database handler |
<Warning>
**`NEO4J_ENCRYPTION_KEY`** defaults to `"test_key"` if not set. Always set a strong random key in production to protect the Aura instance credentials stored in Cognee's relational database.
</Warning>
### How It Works
When a dataset is created, Cognee:
1. Calls the Neo4j Aura API to provision a new database instance (1 GB, Neo4j 5, GCP)
2. Polls until the instance is ready (up to \~5 minutes)
3. Encrypts the instance password and stores the connection details in Cognee's relational database
4. Returns connection info so subsequent operations use that dataset's dedicated graph
When a dataset is deleted, Cognee calls the Aura API to tear down the corresponding instance.
<Info>
Each dataset gets its own Aura instance. For applications with many datasets, this results in
multiple Aura instances running in parallel. Check your Aura plan limits accordingly.
</Info>
</Tab>
</Tabs>
<CardGroup>
<Card title="Graph Stores" icon="book" href="/setup-configuration/graph-stores">
Details About Cognee's Graph Stores (including Neo4j Aura Option 1)
</Card>
<Card title="Multi-User Overview" icon="users" href="/core-concepts/multi-user-mode/multi-user-mode-overview">
More Details About Multi-User Mode
</Card>
</CardGroup>
# Neo4j Community Dataset Database Handler
Source: https://docs.cognee.ai/core-concepts/multi-user-mode/dataset-database-handlers/existing-dataset-database-handlers/neo4j-community
Handler for per-dataset isolation on Neo4j Community edition using one Docker container per dataset.
<Warning>
Make sure that `ENABLE_BACKEND_ACCESS_CONTROL` in your `.env` file is **NOT** set to `False`.
Multi-user mode is enabled by default, therefore `ENABLE_BACKEND_ACCESS_CONTROL=True` by default.
</Warning>
Neo4j Community edition serves exactly **one** database per running server — `CREATE DATABASE`
(used by the plain `neo4j` handler) is an Enterprise-only feature. The `neo4j_community` handler
gets around this by running **one Neo4j Community Docker container per dataset**, mirroring the
[`neo4j_aura_dev` handler](/core-concepts/multi-user-mode/dataset-database-handlers/existing-dataset-database-handlers/neo4j-aura-dev)'s
one-instance-per-dataset model with Docker in place of the Aura REST API.
Use it when you want per-dataset graph isolation on the free Neo4j edition, on a host you control.
<Note>
The plain `neo4j` handler now detects this limitation up front: pointed at a Community server, it fails
fast — typically before even running `CREATE DATABASE` — with a `Neo4jMultiDatabaseSupportError` whose
message names this handler as one of the ways forward. If you arrived here from that error, this page is option (2) — keep your
existing server and set `GRAPH_DATASET_DATABASE_HANDLER="neo4j_community"`.
</Note>
## Requirements
This handler requires a **reachable Docker daemon** on the host running Cognee. Before provisioning
a dataset, Cognee checks that the `docker` binary is on `PATH` and that `docker info` responds; if
either check fails, dataset creation raises a `RuntimeError` explaining what is missing.
Install the Neo4j dependencies, since this handler uses the Neo4j graph database provider:
```bash theme={null}
pip install "cognee[neo4j]"
```
## Environment Variables
```dotenv theme={null}
GRAPH_DATABASE_PROVIDER="neo4j"
GRAPH_DATASET_DATABASE_HANDLER="neo4j_community"
NEO4J_ENCRYPTION_KEY=<a_secret_key_for_encrypting_stored_credentials>
# Optional tuning
NEO4J_COMMUNITY_MAX_CONTAINERS=6
NEO4J_COMMUNITY_IMAGE="neo4j:5-community"
NEO4J_COMMUNITY_STARTUP_TIMEOUT=120
```
| Variable | Required | Default | Description |
| --------------------------------- | ----------- | ----------------------------------- | ------------------------------------------------------------------------------------------------- |
| `GRAPH_DATABASE_PROVIDER` | Yes | — | Must be `neo4j`; the handler rejects any other provider |
| `GRAPH_DATASET_DATABASE_HANDLER` | Yes | — | Selects the `neo4j_community` dataset database handler |
| `NEO4J_ENCRYPTION_KEY` | Recommended | `test_key` | Key used to encrypt the generated per-dataset passwords at rest (same scheme as `neo4j_aura_dev`) |
| `NEO4J_COMMUNITY_MAX_CONTAINERS` | No | `DATABASE_MAX_LRU_CACHE_SIZE` (`6`) | Ceiling on **concurrently running** containers |
| `NEO4J_COMMUNITY_IMAGE` | No | `neo4j:5-community` | Docker image used for the per-dataset containers |
| `NEO4J_COMMUNITY_STARTUP_TIMEOUT` | No | `120` | Seconds to wait for a container to accept bolt connections |
<Warning>
**`NEO4J_ENCRYPTION_KEY`** defaults to `"test_key"` if not set. Always set a strong random key in
production to protect the per-dataset container passwords stored in Cognee's relational database.
</Warning>
Every running container is a full Neo4j server and consumes CPU and RAM. Tune
`NEO4J_COMMUNITY_MAX_CONTAINERS` to the concurrency your host can actually sustain — it defaults to
`DATABASE_MAX_LRU_CACHE_SIZE` so the container pool tracks the graph-engine cache capacity.
## How It Works
**When a dataset is created**, Cognee:
1. Verifies Docker is available.
2. Picks a free localhost port and runs a `neo4j:5-community` container named
`cognee-neo4j-<dataset_id>` with a named data volume `cognee-neo4j-data-<dataset_id>`, where
`<dataset_id>` is the dataset's UUID **without dashes**. The bolt port is published on
`127.0.0.1` only, and stays fixed for the container's lifetime.
3. Generates a random per-dataset password, encrypts it with Fernet, and stores it together with the
container name, volume name, and host port in the `DatasetDatabase` row.
4. Waits until a Neo4j server answers a bolt protocol handshake on the published port (a plain TCP
connect is not a readiness signal — `docker-proxy` accepts connections before Neo4j listens).
Containers are labelled `ai.cognee.neo4j_community=true` and `ai.cognee.dataset_id=<dataset_id>`, so
Cognee only ever inspects, stops, or removes containers it created itself.
**Auto-start.** Before every operation on a dataset, Cognee decrypts the credentials and starts the
container if it is stopped. Graph data lives on the named volume, so it survives container stops and
Cognee restarts.
**Auto-stop.** Container lifetime follows the graph-engine LRU cache: once a dataset's engine is
evicted and no caller holds it, the dataset is idle and its container is stopped. The next operation
on that dataset starts it again.
**Container ceiling.** Before a container starts, Cognee counts the running containers it manages
(including ones that survived a restart, found via a `docker ps` label filter) and stops
least-recently-used ones until there is room under `NEO4J_COMMUNITY_MAX_CONTAINERS`.
<Warning>
**Deleting a dataset is irreversible.** `delete_dataset` removes both the dataset's container **and**
its named Docker volume, so all graph data for that dataset is destroyed. Back up the volume first if
you need to keep it.
</Warning>
If a container is removed outside of Cognee, the next operation on that dataset fails with an error
telling you to delete and re-create the dataset so a fresh container can be provisioned. Container
startup problems are visible through `docker logs cognee-neo4j-<dataset_id>`.
<CardGroup>
<Card title="Graph Stores" icon="book" href="/setup-configuration/graph-stores">
Details About Cognee's Graph Stores
</Card>
<Card title="Multi-User Overview" icon="users" href="/core-concepts/multi-user-mode/multi-user-mode-overview">
More Details About Multi-User Mode
</Card>
</CardGroup>
# PGVector Dataset Database Handler
Source: https://docs.cognee.ai/core-concepts/multi-user-mode/dataset-database-handlers/existing-dataset-database-handlers/pgvector
Handler for using PGVector in Cognee multi-user mode.
<Warning>
Make sure that `ENABLE_BACKEND_ACCESS_CONTROL` in your `.env` file is **NOT** set to `False`.
Multi-user mode is enabled by default, therefore `ENABLE_BACKEND_ACCESS_CONTROL=True` by default.
</Warning>
The PGVector adapter is one of Cognee's core vector adapters, along with LanceDB. Even though PGVector is
an extension of Postgres, you can use PGVector as a vector store with other relational databases, such as SQLite.
## Installation
Firstly, you will need to install specific dependencies necessary for working with Postgres and PGVector:
```bash theme={null}
pip install "cognee[postgres]"
# or for binary version
pip install "cognee[postgres-binary]"
```
## Setup
You will need a running Postgres database instance, and the existing handler works with both **local** and **cloud** setups.
All you need to do is provide the necessary connection information, like the following example does for a local setup:
```dotenv theme={null}
VECTOR_DB_PROVIDER=pgvector
VECTOR_DB_NAME=<your_db_name>
VECTOR_DB_URL=127.0.0.1
VECTOR_DB_PORT=5432
VECTOR_DB_USERNAME=<your_db_username>
VECTOR_DB_PASSWORD=<your_db_password>
```
For the server setup, you can use the one from the Cognee `docker-compose.yml` file, or your own:
```bash theme={null}
docker compose --profile postgres up
```
## Usage
The PGVector handler is registered in Cognee by default, so all that is left to do is to let
Cognee know which handler you are using. This can be done by setting the following `.env` variable:
```dotenv theme={null}
VECTOR_DATASET_DATABASE_HANDLER="pgvector"
```
### Schema-per-dataset isolation (`pgvector_shared`)
The `pgvector` handler above gives every dataset its own Postgres database (`CREATE DATABASE
"<dataset_id>"`). If you would rather keep everything in a single database, select the
`pgvector_shared` handler instead:
```dotenv theme={null}
VECTOR_DB_PROVIDER=pgvector
VECTOR_DATASET_DATABASE_HANDLER="pgvector_shared"
```
In this mode each dataset's vector collections live in a dedicated **schema** of Cognee's existing
relational database rather than in a separate database:
* **The schema name is derived automatically.** Cognee names it `ds_<dataset_id_hex>` (the dataset
UUID with hyphens stripped) and stores it in the dataset's `vector_database_connection_info`. You
do not supply a schema yourself.
* **Connection details come from the relational configuration.** The host, port, database name,
username, and password are taken from your `DB_HOST` / `DB_PORT` / `DB_NAME` / `DB_USERNAME` /
`DB_PASSWORD` settings — not from the `VECTOR_DB_*` variables — because the shared database *is*
Cognee's relational database. Credentials are read from the live relational config at connection
time and are never persisted in the dataset record.
* **Only `CREATE SCHEMA` privilege is required**, not `CREATE DATABASE`.
* **Provisioning is automatic and idempotent.** On first use Cognee runs `CREATE EXTENSION IF NOT
EXISTS vector` and `CREATE SCHEMA IF NOT EXISTS`, so re-running `add`/`cognify` is safe and you do
not need to create the schema or install the pgvector extension by hand.
* **Deleting a dataset drops its schema** with a single `DROP SCHEMA ... CASCADE`, removing every
table and index the dataset created.
Isolation is enforced by pinning the per-dataset PGVector engine's `search_path` to
`"<dataset schema>, public"`. The dataset schema comes first, so reads and writes never fall through
to `public` (which holds Cognee's shared relational tables), while `public` stays on the path so the
`vector` type remains resolvable. Table listing and deletion for these engines are likewise scoped
to the dataset's own schema.
There is a matching graph handler, `postgres_graph_shared`, which isolates each dataset's
`graph_node`/`graph_edge` tables in the same way. See the
[list of supported handlers](/core-concepts/multi-user-mode/dataset-database-handlers/dataset-database-handlers-how-to-use-them).
<Warning>
`pgvector_shared` requires `VECTOR_DB_PROVIDER=pgvector`. Selecting it with any other vector
provider raises a `ValueError` when the dataset database is created.
</Warning>
## Connection pool tuning
In multi-user mode each dataset gets its own PGVector engine, so the total number of open
connections grows with the number of active datasets (number of datasets × pool size). To keep
this fan-out in check, per-dataset PGVector engines fall back to a small connection pool
(`pool_size=2`, `max_overflow=20`) when you have not sized the pool yourself.
With the `pgvector_shared` handler every dataset engine points at the same database, so those pools
all land on one connection target instead of fanning out across a separate database per dataset —
but each dataset still has its own engine and its own pool, so the same sizing guidance applies.
Pool arguments for these engines are resolved from the first source that is set:
1. **`VECTOR_POOL_ARGS`** — explicit PGVector sizing, which always wins.
2. **The relational `POOL_ARGS`** — inherited when `VECTOR_POOL_ARGS` is unset, so an operator who
sized the pool explicitly outranks the built-in default.
3. **The built-in access-control default** (`pool_size=2`, `max_overflow=20`) — used only when
neither variable is set.
`VECTOR_POOL_ARGS` takes a JSON object whose keys are passed through to SQLAlchemy's
`create_async_engine`:
```dotenv theme={null}
VECTOR_POOL_ARGS='{"pool_size": 2, "max_overflow": 5, "pool_recycle": 1800}'
```
It applies only to PGVector per-dataset engines; leave it unset if you want these engines to follow
`POOL_ARGS` instead. The value must be a JSON object; invalid JSON raises a configuration error
(`VECTOR_POOL_ARGS must be valid JSON`) at startup. Restart your process or container after
changing it so the new pool settings take effect.
## SSL / connect args for managed Postgres
When a per-dataset PGVector engine connects to managed Postgres that enforces SSL (for example
Neon), it now inherits the relational `DATABASE_CONNECT_ARGS` connection arguments, so asyncpg SSL options
supplied there are honored for these engines too. Leaving `DATABASE_CONNECT_ARGS` unset is a no-op.
See [Relational Databases → Managed Postgres with SSL](/setup-configuration/relational-databases)
for the `DATABASE_CONNECT_ARGS` format.
<CardGroup>
<Card title="Vector Stores" icon="book" href="/setup-configuration/vector-stores">
Details About Cognee's Vector Stores
</Card>
<Card title="Multi-User Overview" icon="users" href="/core-concepts/multi-user-mode/multi-user-mode-overview">
More Details About Multi-User Mode
</Card>
</CardGroup>
# Qdrant Dataset Database Handler
Source: https://docs.cognee.ai/core-concepts/multi-user-mode/dataset-database-handlers/existing-dataset-database-handlers/qdrant
Handler for using Qdrant in Cognee multi-user mode.
The Qdrant adapter is one of Cognee's community adapters, which can be found on our [community repo](https://github.com/topoteretes/cognee-community/tree/main/packages/vector/qdrant).
<Warning>
Make sure that `ENABLE_BACKEND_ACCESS_CONTROL` in your `.env` file is **NOT** set to `False`.
Multi-user mode is enabled by default, therefore `ENABLE_BACKEND_ACCESS_CONTROL=True` by default.
</Warning>
## Installation
Firstly, you will need to install the adapter package:
```bash theme={null}
pip install cognee-community-vector-adapter-qdrant
```
## Setup
You will need a running Qdrant database instance, and the existing handler works with both **local** and **cloud** setups.
All you need to do is provide the necessary connection information, like the following example does for a local setup:
```dotenv theme={null}
VECTOR_DB_PROVIDER="qdrant"
VECTOR_DB_URL="http://localhost:6333"
```
For a local database instance, you can use docker, as in the code below. For a cloud setup,
you can read [Qdrant's docs](https://qdrant.tech/documentation/cloud-intro/) on how to set it up.
```bash theme={null}
docker run -p 6333:6333 -p 6334:6334 \
-v "$(pwd)/qdrant_storage:/qdrant/storage:z" \
qdrant/qdrant
```
## Usage
After setting up your instance and connection information, you will need to register the adapter and handler to Cognee,
so it knows which database to use. Registering the adapter also registers the dataset database handler.
In your code, add the following import statement:
```bash theme={null}
from cognee_community_vector_adapter_qdrant import register
```
The final important piece of information is to let Cognee know which handler you are using.
This can be done by setting the following `.env` variable:
```dotenv theme={null}
VECTOR_DATASET_DATABASE_HANDLER="qdrant"
```
<CardGroup>
<Card title="Qdrant Adapter" icon="book" href="/setup-configuration/community-maintained/qdrant">
Details About Cognee's Qdrant Adapter
</Card>
<Card title="Multi-User Overview" icon="users" href="/core-concepts/multi-user-mode/multi-user-mode-overview">
More Details About Multi-User Mode
</Card>
</CardGroup>
# Multi-User Mode Overview
Source: https://docs.cognee.ai/core-concepts/multi-user-mode/multi-user-mode-overview
Learn how Cognee handles multiple users and data isolation.
Multi-user mode is the architectural directive in Cognee that enforces strict data isolation between different users and [datasets](/core-concepts/further-concepts/datasets). It is primarily controlled by the environment variable `ENABLE_BACKEND_ACCESS_CONTROL`.
Starting with version 0.5.0, this mode is enabled by default when your configured storage setup supports it. This keeps Cognee secure by default without forcing unsupported database combinations into multi-user mode.
<Info>Data Isolation Enforcement — When active, Cognee partitions the knowledge graph and vector stores, ensuring data created by one user is neither visible nor accessible to another, unless read permission has been given to the other user.</Info>
## Upgrading to v0.5.0 or Later
<Warning>
If you are upgrading from a version earlier than `0.5.0`, data ingested before the upgrade may be inaccessible in multi-user mode because it was not associated with a specific user. To restore access to pre-upgrade data, start Cognee with `ENABLE_BACKEND_ACCESS_CONTROL=false`, then migrate or re-ingest that data before re-enabling multi-user mode.
</Warning>
For configuration requirements and supported handler/provider combinations, see [Permissions Setup](/setup-configuration/permissions) and [Dataset Database Handlers](/core-concepts/multi-user-mode/dataset-database-handlers/dataset-database-handlers-what-are-they).
## When multi-user mode is active
When multi-user mode is active, the system unlocks several multi-tenant features:
* **Isolated Recall**: Retrieval operations are strictly scoped to datasets the authenticated user has explicit read access to. To learn more about the permissions system and access types, read about our [Permission System](/core-concepts/multi-user-mode/permissions-system/overview).
* **Granular Management**: Remembering or forgetting documents is scoped at the dataset level, preventing global knowledge pool pollution.
* **Automatic Routing**: The system automatically determines which local/cloud database or logical schema to connect to based on the dataset. This is done with the help of [Dataset Database Handlers](/core-concepts/multi-user-mode/dataset-database-handlers/dataset-database-handlers-what-are-they).
## Where isolation happens: datasets vs. databases
Dataset isolation is handled differently depending on whether backend access control is enabled. This applies to the **graph and vector stores**, not the relational store:
| | Single-user mode (`ENABLE_BACKEND_ACCESS_CONTROL=false`) | Multi-user mode (`ENABLE_BACKEND_ACCESS_CONTROL=true`) |
| ------------------------------- | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Graph / vector store** | One shared instance for all datasets | One backend per dataset, resolved by a [Dataset Database Handler](/core-concepts/multi-user-mode/dataset-database-handlers/dataset-database-handlers-what-are-they) |
| **Separation between datasets** | None at query time — all datasets share one store and `datasets`/`dataset_ids` filters are ignored | Physical — each dataset's data lives in its own database |
| **Access control** | Off (no auth) | Enforced per [dataset](/core-concepts/multi-user-mode/permissions-system/datasets) via read/write/share/delete permissions |
| **Tenant scope** | N/A | A [tenant](/core-concepts/multi-user-mode/permissions-system/tenants) is a group of users that shares dataset permissions — it is not itself a separate database |
So a "workspace" or "tenant" is not a database boundary. The database boundary in multi-user mode is the **dataset**: each dataset is routed to its own physical graph/vector store, while tenants and users simply control *who can reach which datasets*.
<Columns>
<Card title="Permission System" icon="user" href="/core-concepts/multi-user-mode/permissions-system/overview">
Learn about the permission system that powers multi-user mode
</Card>
<Card title="Dataset Database Handlers" icon="building" href="/core-concepts/multi-user-mode/dataset-database-handlers/dataset-database-handlers-what-are-they">
Understand database connection resolution per dataset
</Card>
</Columns>
# ACL
Source: https://docs.cognee.ai/core-concepts/multi-user-mode/permissions-system/acl
Access control lists for permission storage and inheritance in Cognee.
# ACL: Permission Storage and Inheritance
The ACL (Access Control List) system stores all permissions and handles permission checking at runtime. ACL entries are stored in the `acls` table, with each row linking a [principal](./principals) to a [dataset](./datasets) with a specific permission.
<Info>**Runtime permission calculation** — The system doesn't store "effective permissions" anywhere—it calculates them on demand by querying ACL entries.</Info>
## How ACL Works
When a [user](./users) tries to access data, the system queries all relevant ACL entries and aggregates the permissions. The permission checking function `get_all_user_permission_datasets()` unions the [user](./users)'s direct permissions with those inherited from their [tenant](./tenants) and [roles](./roles), combining all three sources: direct [user](./users) permissions, [tenant](./tenants)-level permissions, and [role](./roles)-level permissions.
This approach ensures permissions are always current and allows for complex permission inheritance without data duplication.
## ACL Storage Schema
The ACL system uses a simple but powerful schema to store permissions:
<Accordion title="ACL Model Fields">
The ACL model defines what gets stored in the SQL database. The `acls` table contains:
* `id`: Unique identifier (UUID primary key)
* `principal_id`: References the [principal](./principals) ([user](./users), [tenant](./tenants), or [role](./roles))
* `dataset_id`: References the [dataset](./datasets)
* `permission_id`: References the permission type
* `created_at`: Timestamp when created
* `updated_at`: Timestamp when last modified
</Accordion>
<Accordion title="Permission Checking Functions">
* `get_all_user_permission_datasets(user, permission)`: Queries ACL entries and returns [datasets](./datasets) the [user](./users) can access
* `give_permission_on_dataset(principal, dataset_id, permission)`: Inserts a new ACL entry. If a matching entry already exists for the same principal, dataset, and permission, the call is a no-op (the existing entry is left untouched). Grants are additive — calling it once per permission name (e.g. `"read"`, then `"write"`) adds each as a separate ACL row.
* `revoke_permission_on_dataset(principal, dataset_id, permission)`: Removes the matching ACL entry. Each permission must be revoked individually.
</Accordion>
## Permission Resolution Order
The system evaluates permissions in a specific order:
1. **Direct [user](./users) permissions** — Explicitly granted to the [user](./users)
2. **[Role](./roles) permissions** — Inherited through the [user](./users)'s role memberships
3. **[Tenant](./tenants) permissions** — Inherited through the [user](./users)'s tenant membership
This order allows for flexible permission management where more specific permissions can override broader ones.
## ACL Operations
The ACL system supports several key operations:
* **Grant permissions** — Add new ACL entries to grant access
* **Revoke permissions** — Remove ACL entries to revoke access
* **Check permissions** — Query ACL entries to determine access
* **List permissions** — Get all permissions for a principal or dataset
## Permission Inheritance
The ACL system implements a three-tier inheritance model:
* **User level** — Direct permissions granted to individual users
* **Role level** — Permissions granted to roles, inherited by role members
* **Tenant level** — Permissions granted to tenants, inherited by all tenant members
Users receive the union of all permissions from these three sources, giving them the most permissive access available.
## Performance Considerations
The ACL system is designed for performance:
* **Indexed queries** — Database indexes on principal\_id, dataset\_id, and permission\_id
* **Efficient lookups** — Single query to get all permissions for a user
* **Caching opportunities** — Permission results can be cached for frequently accessed datasets
* **Batch operations** — Support for granting/revoking multiple permissions at once
## Security Features
The ACL system includes several security features:
* **Immutable ownership** — Dataset ownership cannot be changed
* **Permission validation** — All permission checks go through the ACL system
* **Audit trail** — All permission changes are logged with timestamps
* **Isolation** — Users can only access datasets they have permissions for
## Troubleshooting
Common ACL-related issues and solutions:
<AccordionGroup>
<Accordion title="PermissionDeniedError on memory writes (Error code #403)">
**Error:** `Request owner does not have necessary permission: [write] for all datasets requested`
This error occurs when `ENABLE_BACKEND_ACCESS_CONTROL=true` and the calling user does not have `write` permission on the target dataset. This commonly happens when running `remember()`, `improve()`, or the lower-level `add()` / `cognify()` flow on a dataset owned by a different user.
Check the following:
* **Required permission** — memory-writing operations such as `remember()`, `improve()`, `add()`, and `cognify()` require `write` permission on every target dataset. Having `read` access is not enough.
* **ACL entries** — Verify that ACL rows actually exist for the expected principal and dataset. If no matching ACL entries exist, the user will not inherit access.
* **Role or tenant inheritance** — If access should come from a role or tenant, confirm that the user is a member of that role or tenant. Effective permissions are the union of direct user grants plus inherited tenant and role grants.
If the user is missing `write`, the dataset owner (or any user with `share` permission on the dataset) must grant it.
**Python SDK:**
```python theme={null}
from cognee.modules.users.permissions.methods import authorized_give_permission_on_datasets
# Called by the dataset owner (or a user with 'share' permission)
await authorized_give_permission_on_datasets(
user.id, # user who needs write access
[dataset.id], # target dataset UUIDs
"write", # permission type
owner.id, # must have 'share' permission on the dataset
)
```
**HTTP API** (requires authentication as the dataset owner):
```bash theme={null}
curl -X POST "http://localhost:8000/v1/permissions/datasets/{user_id}" \
-H "Authorization: Bearer <owner_token>" \
-H "Content-Type: application/json" \
-d '{"permission_name": "write", "dataset_ids": ["<dataset_uuid>"]}'
```
To retrieve the user's `id`, call `GET /v1/users/me`. To list available datasets and their IDs, call `GET /v1/datasets`.
See [Permission Snippets](/guides/permission-snippets) for complete setup examples.
<Note>
`PermissionDeniedError` has a second, quieter variant: `Request owner does not have permission: [<type>] for any dataset.`, raised when the caller has **no** accessible datasets at all rather than being denied a dataset they asked for. That case is logged at `DEBUG` — the denial above is logged at `ERROR` — because read paths such as `get_readable_datasets()` turn it into an empty result, which is the normal outcome on a fresh install that has not ingested anything yet. Its absence from your logs therefore does not mean the permission check was skipped; set [`LOG_LEVEL=DEBUG`](/setup-configuration/overview#environment-variable-quick-reference) if you need to observe it.
</Note>
</Accordion>
<Accordion title="Dataset not accessible when deleting data (Error code #401)">
If you get `UnauthorizedDataAccessError: Dataset ... not accessible (401)` from `cognee.datasets.delete_data()` or the legacy `delete()`, the most common causes are:
**`delete` permission not granted** — `read` and `write` access are not sufficient for deletion. The `delete` permission must be granted separately.
Verify which datasets a user can delete:
```python theme={null}
from cognee.modules.users.permissions.methods import get_all_user_permission_datasets
deletable_datasets = await get_all_user_permission_datasets(user, "delete")
```
Grant `delete` permission (requires the granting user to have `share` permission on the dataset):
```python theme={null}
from cognee.modules.users.permissions.methods import authorized_give_permission_on_datasets
await authorized_give_permission_on_datasets(
user.id, # user to receive permission
[dataset.id], # target datasets
"delete", # permission type
owner.id, # must have 'share' permission on the dataset
)
```
**Wrong user context** — When `ENABLE_BACKEND_ACCESS_CONTROL` is enabled, every operation runs as a specific user. Ensure the `user` object passed to `forget()` or lower-level delete functions is the user that owns or has `delete` access to the dataset.
**Dataset owned by another user** — Only the dataset owner and users with an explicit `delete` grant can remove data. If the dataset was created by `user1`, `user2` cannot delete from it without a permission grant from `user1`.
<Note>
`forget()` resolves the dataset itself before it reaches these helpers, so the same missing grant surfaces earlier and under a different name: `DatasetNotFoundError` (404) for a dataset name, `PermissionDeniedError` (403) for a `dataset_id`. See [troubleshooting dataset resolution](/python-api/forget#troubleshooting).
</Note>
</Accordion>
<Accordion title="Slow permission checks">
Review database indexes and query patterns, especially on `principal_id`, `dataset_id`, and `permission_id`, since those fields drive ACL lookups.
</Accordion>
</AccordionGroup>
<Columns>
<Card title="Snippets" icon="code" href="/guides/permission-snippets">
See practical snippets of ACL operations
</Card>
<Card title="Setup Configuration" icon="settings" href="/setup-configuration/permissions">
Learn how to configure ACL and multi-tenant mode
</Card>
</Columns>
# Datasets in the Permissions System
Source: https://docs.cognee.ai/core-concepts/multi-user-mode/permissions-system/datasets
The core unit of data in Cognee's permission system.
A dataset is a logical container for related documents and their processed knowledge graphs. All data in Cognee belongs to a dataset. When you store data with `cognee.remember()`, it is processed and stored within a specific dataset.
<Info>**Dataset-scoped permissions** — All permissions in Cognee are defined at the dataset level, never for individual documents.</Info>
## Ownership and Permissions
When a principal creates a dataset, they become its **owner**. A principal is any entity that can have permissions, like a [user](./users), [tenant](./tenants), or [role](./roles). Ownership cannot be changed. The owner has full control and can grant permissions to others.
There are four types of permissions you can grant on a dataset:
* **Read** — View documents and query stored memory.
* **Write** — Remember, improve, or otherwise modify documents and data.
* **Delete** — Remove the entire dataset.
* **Share** — Grant permissions to other principals.
## Dataset Isolation: How Access Is Enforced
Cognee can enforce strict data isolation between datasets, but it's important to understand when this happens.
* **Isolation is a default setting**: Dataset boundaries are enforced by default, meaning the `ENABLE_BACKEND_ACCESS_CONTROL` setting is `true` by default.
* **Without isolation**: If this setting is `false`, dataset parameters are ignored during searches, and queries will run across all data in the system, regardless of permissions.
* **Database support**: True isolation is currently supported when using the following database backends (others do not support dataset isolation.):
* **Relational Databases**: SQLite, Postgres
* **Vector Databases**: LanceDB, PGVector, Qdrant
* **Graph Databases**: Kùzu, Neo4j Aura
* **Hybrid Databases**: FalkorDB
See [ACL](./acl) for details on how permissions are stored and checked. For setup instructions, see [Permissions Setup](/setup-configuration/permissions).
## Using Datasets in Operations
Datasets integrate with Cognee's main operations:
* **`remember`**: Direct new content into a specific dataset by name or ID. If no dataset is specified, a default `main_dataset` is used.
* **`improve`**: Apply optional semantic enrichment on a per-dataset basis.
* **`recall`**: Scope queries to run only against datasets you have read access to.
* **`forget`**: Remove data or full datasets within the current user's permission scope.
## Technical Details
<Accordion title="Operation Permission Requirements">
Different operations require different permissions:
* `remember`/`improve` operations → require `write` permission
* `recall` operations → require `read` permission
* `forget` operations → require `delete` permission
* Permission management → requires `share` permission
</Accordion>
<Accordion title="Dataset Creation Methods">
Cognee provides two helper methods for creating datasets:
* `create_dataset()`: This is a lower-level function that only inserts the dataset record. It expects the caller to manage the Access Control List (ACL) entries separately.
* `create_authorized_dataset()`: This is the recommended method for most user-facing flows. It wraps `create_dataset()` and then immediately grants the creator full `read/write/delete/share` permissions. This ensures the dataset is usable as soon as it's created, especially when `ENABLE_BACKEND_ACCESS_CONTROL` is active.
</Accordion>
<Accordion title="Dataset Model Fields">
The core dataset metadata is stored in a relational (SQL) database. The `datasets` table includes:
* `id`: Unique identifier (UUID primary key)
* `name`: Human-readable name
* `owner_id`: ID of the principal who created the dataset
* `created_at`: Timestamp when created
* `updated_at`: Timestamp when last modified
</Accordion>
<Columns>
<Card title="Main Operations" icon="play" href="/core-concepts/main-operations/remember">
See how datasets work with Remember, Improve, and Recall
</Card>
<Card title="Building Blocks" icon="puzzle" href="/core-concepts/building-blocks/datapoints">
Learn about the DataPoints that populate datasets
</Card>
</Columns>
# Permissions System Overview
Source: https://docs.cognee.ai/core-concepts/multi-user-mode/permissions-system/overview
Introduction to Cognee's permission system and access control architecture.
# Cognee Permissions System
The Cognee permission system manages access to data through an access control architecture. This system provides data isolation and access control through dataset-scoped permissions and per-dataset storage, enabling multiple users or organizations to use the same Cognee instance while keeping their data completely separate.
<Info>**Enable Backend Access Control (EBAC)** is the configuration flag that activates this multi-tenant mode, enforcing user authentication and complete data isolation.</Info>
## Core Components
The permission system is built around several key concepts:
* **Dataset** — The basic unit of data in Cognee. All documents and their processed knowledge graphs belong to a dataset. Permissions are always defined at the dataset level. See [Datasets](./datasets) for details.
* **Principal** — Any entity that can hold permissions. Principals come in three forms: [Users](./users), [Tenants](./tenants), and [Roles](./roles). This unified design supports flexible access control across individuals and organizations.
* **User** — An individual who creates and interacts with datasets. Users can own datasets and be granted permissions on others. Each user belongs to at most one tenant.
* **Tenant** — An organization or group. Tenants contain users and can be granted permissions on datasets, which apply to all members.
* **Role** — A group of users within a tenant. Roles can also be granted dataset permissions, which apply to their members.
* **ACL** — The Access Control List records all permission assignments. Each entry links a principal to a dataset with a specific permission type. See [ACL](./acl) for details.
## Permission Types
There are four types of permissions that can be granted on datasets:
* **Read** — View documents and query stored memory
* **Write** — Remember data, improve datasets, or otherwise modify stored memory
* **Delete** — Remove datasets or dataset-scoped memory
* **Share** — Grant permissions to other principals
Deletion operations (removing a data item or a full dataset) require `delete` permission on the target dataset. If a user lacks `delete`, the API returns a not-found or unauthorized response depending on the endpoint and auth mode.
## How It Works
When `ENABLE_BACKEND_ACCESS_CONTROL` is set to true, Cognee runs in access control mode:
* **Authentication becomes mandatory** (even if `REQUIRE_AUTHENTICATION=false`)
* **Data isolation is enforced** at the user + dataset level for graph and vector stores
* **Database routing is automatic** — Databases are configured per request via context variables through the [Dataset Database Handler Mechanism](/core-concepts/multi-user-mode/dataset-database-handlers/dataset-database-handlers-what-are-they)
* **Vector handler routing stays automatic** — When access control is enabled, Cognee routes vector storage through a dataset database handler. Built-in providers such as `pgvector` and `turso` auto-select their matching handler when you leave `VECTOR_DATASET_DATABASE_HANDLER` at its default value; explicitly setting an incompatible handler still causes 401/403 errors. See [Permissions Setup](/setup-configuration/permissions) for supported combinations.
See [Setup Configuration](/setup-configuration/permissions) for configuration details.
## Permission Resolution
When a user tries to access data, the system evaluates their effective permissions by combining:
1. **Direct user permissions** — explicitly granted to the user
2. **Role permissions** — inherited through the user's role memberships
3. **Tenant permissions** — inherited through the user's tenant membership
The system doesn't store "effective permissions" anywhere—it calculates them on demand by querying ACL entries. This approach ensures permissions are always current and allows for complex permission inheritance without data duplication.
## Data Storage Layout
When EBAC is enabled, Cognee automatically organizes data by user and dataset for file based databases like LanceDB and Kuzu:
**Filesystem layout**:
```
.cognee_system/databases/<user_uuid>/
├── <dataset_uuid>.pkl # Kùzu graph database
└── <dataset_uuid>.lance.db/ # LanceDB vector database
.data_storage/<tenant_uuid_or_user_uuid>/
└── ... # Raw and processed files
```
**Key points:**
* Each user gets their own database directory
* Each dataset gets its own database files within the user's directory
<Columns>
<Card title="Datasets" icon="database" href="/core-concepts/multi-user-mode/permissions-system/datasets">
Learn about datasets as the core unit of data
</Card>
<Card title="Setup Configuration" icon="settings" href="/setup-configuration/permissions">
Configure multi-tenant mode and access control
</Card>
</Columns>
# Principals
Source: https://docs.cognee.ai/core-concepts/multi-user-mode/permissions-system/principals
The unified abstraction for entities that can hold permissions in Cognee.
# Principals: The Abstraction
A principal is any entity that can hold permissions in Cognee. This abstraction allows the permission system to work with different types of entities in a unified way, eliminating the need for separate permission systems for users, tenants, and roles.
<Info>**Polymorphic design** — All principal types use the same permission mechanism, making the system flexible and consistent.</Info>
## Principal Types
There are three types of principals:
* **[Users](./users)** — Individual people who interact with the system
* **[Tenants](./tenants)** — Organizations or groups that contain users
* **[Roles](./roles)** — Groups of users within a tenant
All three types inherit from the same base Principal class, which means they can all be granted permissions on datasets using the same functions and mechanisms.
## How Principals Work with Permissions
The system stores permissions by linking principals to datasets. You can grant permissions to any of the principals using built-in functions like `give_permission_on_dataset()` and `get_principal_datasets()`. When you grant a permission, you specify:
* Which principal gets the permission
* Which dataset the permission applies to
* What type of permission (read, write, delete, share)
This unified approach means you can grant permissions to:
* Individual [users](./users) for personal access
* [Tenants](./tenants) for organization-wide access
* [Roles](./roles) for team-based access within a tenant
<Accordion title="Principal Model Fields">
The base Principal model defines what gets stored in the SQL database. The `principals` table contains:
* `id`: Unique identifier (UUID primary key)
* `created_at`: Timestamp when created
* `updated_at`: Timestamp when last modified
* `type`: Discriminator field for polymorphic inheritance
Each principal type (User, Tenant, Role) has its own table that references the principals table via foreign key, storing additional fields specific to that type.
</Accordion>
<Accordion title="Permission Storage Schema">
The permission system links principals to datasets with permissions:
* `principal_id`: References the principal ([user](./users), [tenant](./tenants), or [role](./roles))
* `dataset_id`: References the [dataset](./datasets)
* `permission_id`: References the permission type
This many-to-many relationship allows flexible permission management across different entity types.
</Accordion>
<Accordion title="Key Functions">
* `give_permission_on_dataset(principal, dataset_id, permission)`: Writes a single ACL row (or reuses an existing one) so a [user](./users), [tenant](./tenants), or [role](./roles) gains read, write, delete, or share on that dataset. It's the building block used after dataset creation or whenever access is delegated.
* `get_principal_datasets(principal, permission)`: Queries those ACL entries (and the related dataset records) so you can list every dataset where that same principal holds the requested permission—handy for permission checks or UI listings. It takes a principal object and performs no authorization of its own, so call it only when you have already established that the caller may ask about that principal.
* `authorized_get_principal_datasets(principal_id, permission_name, requester_id)`: The authorization-checked entry point for the same lookup, importable from `cognee.modules.users.permissions.methods`. It takes the principal's id rather than the object, resolves the requester, checks that they may ask about that principal, and returns the resulting `list[Dataset]`. This is what backs `GET /api/v1/permissions/principals/{principal_id}/datasets`.
The requester's [tenant](./tenants) is read off the requester rather than passed in, so a caller cannot name a different tenant, and the returned datasets are always filtered down to that tenant. Which principals a requester may ask about depends on the principal's type:
* **[User](./users)** — themselves, or any user in the tenant if they are the tenant owner or have user-management permission.
* **[Role](./roles)** — a role of their tenant that they are a member of, or any role of that tenant if they are the tenant owner or have user-management permission. A `principal_id` for a role in another tenant raises `RoleNotFoundError` (`404`), which is indistinguishable from a role that does not exist.
* **[Tenant](./tenants)** — only the tenant they are currently in; any other tenant raises `PermissionDeniedError` (`403`).
Anything else — including an unsupported principal type, or a requester who fails the tenant-owner-or-user-management check the rules above fall back to — raises `PermissionDeniedError` (`403`).
</Accordion>
## Permission Inheritance
The principal system supports hierarchical permission inheritance:
1. **Direct permissions** — Explicitly granted to a specific principal
2. **[Role permissions](./roles)** — Inherited through role memberships
3. **[Tenant permissions](./tenants)** — Inherited through tenant membership
When a [user](./users) tries to access data, the system evaluates their effective permissions by combining all three sources. This allows for flexible access control patterns:
* Grant broad permissions at the [tenant](./tenants) level
* Refine access with [role](./roles)-specific permissions
* Override with direct [user](./users) permissions when needed
## Benefits of the Principal System
* **Unified interface** — Same functions work for all principal types
* **Flexible access control** — Support for individual, team, and organization-level permissions
* **Scalable management** — Easy to add new principal types or modify existing ones
* **Consistent behavior** — All principals follow the same permission rules and patterns
<Columns>
<Card title="Users" icon="user" href="/core-concepts/multi-user-mode/permissions-system/users">
Learn about individual users and authentication
</Card>
<Card title="Tenants" icon="building" href="/core-concepts/multi-user-mode/permissions-system/tenants">
Understand organization-level access control
</Card>
</Columns>
# Roles
Source: https://docs.cognee.ai/core-concepts/multi-user-mode/permissions-system/roles
Role-based permissions within tenants for granular access control.
# Roles
A role is a group of [users](./users) within a [tenant](./tenants). [Roles](./roles) can be granted permissions on [datasets](./datasets), which apply to their members. This enables fine-grained access control within organizations and makes it easier to manage permissions for different teams.
<Info>**Role-based permissions** — When a role is granted a permission on a dataset, all users assigned to that role inherit that permission.</Info>
## Role Concept
[Roles](./roles) are created by [tenant](./tenants) owners and are scoped to that specific [tenant](./tenants). The role belongs to exactly one [tenant](./tenants) as soon as it's created. Because of that foreign-key link, a role can't be moved or shared with another [tenant](./tenants); you would need to create a new role under the other [tenant](./tenants) instead.
[Users](./users) can be assigned to multiple [roles](./roles) within their [tenant](./tenants), and [roles](./roles) can contain multiple [users](./users). This many-to-many relationship allows flexible permission management across teams.
## Role-Based Permissions
When a [role](./roles) is granted a permission on a [dataset](./datasets), all [users](./users) assigned to that role inherit that permission. [Users](./users) receive the union of their direct permissions, [tenant](./tenants)-level permissions, and [role](./roles)-level permissions.
[Roles](./roles) allow you to create permission groups like "editors" or "viewers" within a [tenant](./tenants), making it easier to manage access for different teams without granting permissions to individual [users](./users).
<Accordion title="Role Model Fields">
The Role model defines what gets stored in the SQL database. The `roles` table contains:
* `id`: Unique identifier (UUID primary key, references principals.id)
* `name`: Human-readable name (unique within tenant)
* `tenant_id`: ID of the tenant this role belongs to (required)
</Accordion>
<Accordion title="Role Creation">
* `create_role(role_name, owner_id)`: Creates a new role (tenant owner only)
* `add_user_to_role(user_id, role_id, owner_id)`: Assigns a user to a role (tenant owner only)
</Accordion>
<Accordion title="Role and Member Deletion">
* `delete_role(role_id, owner_id)`: Deletes a role, removes all user-role memberships for that role, and revokes dataset permissions granted to the role principal. Exposed via `DELETE /api/v1/permissions/roles/{role_id}`.
* `remove_user_from_role(user_id, role_id, owner_id)`: Removes a [user](./users) from a specific role without removing them from the [tenant](./tenants). Exposed via `DELETE /api/v1/permissions/users/{user_id}/roles`.
Use `remove_user_from_tenant` (see [Tenants](./tenants)) only when you want to remove a [user](./users) from the entire [tenant](./tenants). That operation also strips the user from all roles in the tenant and revokes their direct dataset permissions for tenant-owned datasets.
</Accordion>
<Accordion title="Limitations">
* Roles are tenant-scoped and cannot cross tenants
* API endpoints for role management
</Accordion>
## Role Management
Tenant owners can:
* Create roles within their tenant
* Assign users to roles
* Remove users from roles
* Grant permissions to roles
* Delete roles
### Role Visibility
Administering roles is owner-only, but reading them is scoped to membership:
* **Roles you belong to** — `GET /api/v1/permissions/tenants/{tenant_id}/roles` returns `200` for any authenticated caller when the tenant exists (a nonexistent `tenant_id` still returns `404`). Callers who are the [tenant](./tenants) owner or have user-management permission (for example, an Admin role) see every role in the tenant; everyone else sees only the roles they are a member of. A caller who belongs to no role in that tenant — including a caller who passes a `tenant_id` for a tenant they are not part of — receives an empty list rather than an error.
* **Members of those roles** — `GET /api/v1/permissions/tenants/{tenant_id}/roles/{role_id}/users` is visible to members of the role itself, so [users](./users) can see who shares their roles. A caller who is not a member and lacks user-management permission receives `403`. The role is resolved within the [tenant](./tenants) in the path, so a `role_id` belonging to another tenant returns `404` instead of that tenant's members.
* **The full tenant user directory** — `GET /api/v1/permissions/tenants/{tenant_id}/users` still requires tenant ownership or user-management permission. Role membership alone does not grant it.
* **The datasets granted to a role** — `GET /api/v1/permissions/principals/{principal_id}/datasets` returns the [datasets](./datasets) a principal holds a permission on, so you can answer "which datasets does this team have?". The optional `permission_name` query parameter selects which permission to list and defaults to `read`; the other accepted values are `write`, `delete`, and `share`. For a role `principal_id`, the endpoint is visible to members of that role, so [users](./users) can list the datasets of the [roles](./roles) they belong to; a non-member who is not the tenant owner and lacks user-management permission receives `403`. The role is resolved within the caller's current [tenant](./tenants), so a `role_id` belonging to another tenant returns `404` rather than revealing that tenant's datasets. The response is a JSON list of dataset objects, and it is always narrowed to the caller's current tenant.
## Common Role Patterns
Roles are typically organized around job functions or responsibilities:
* **Editors** — Can modify content and run cognify operations
* **Viewers** — Can only read and search data
* **Administrators** — Can manage permissions and users
* **Project Managers** — Can access specific project datasets
* **Reviewers** — Can read and provide feedback on content
## Permission Inheritance Hierarchy
Users receive permissions through a three-level hierarchy:
1. **Direct permissions** — Explicitly granted to the user
2. **Role permissions** — Inherited through role memberships
3. **Tenant permissions** — Inherited through tenant membership
The system calculates effective permissions by combining all three sources, giving users the most permissive access available to them.
## Best Practices
* **Create meaningful role names** — Use descriptive names that reflect the role's purpose
* **Keep roles focused** — Each role should have a clear, specific purpose
* **Regular role reviews** — Periodically review and update role assignments
* **Document role purposes** — Keep clear documentation of what each role is for
* **Principle of least privilege** — Grant only the minimum permissions necessary
## Role vs Tenant Permissions
* **Tenant permissions** — Broad, organization-wide access
* **Role permissions** — Specific, team-based access within the tenant
* **Direct permissions** — Individual, user-specific access
This three-tier system allows for flexible and scalable permission management.
<Columns>
<Card title="ACL" icon="shield" href="/core-concepts/multi-user-mode/permissions-system/acl">
Learn how permissions are stored and checked
</Card>
<Card title="Snippets" icon="code" href="/guides/permission-snippets">
See practical snippets of role-based permissions
</Card>
</Columns>
# Tenants
Source: https://docs.cognee.ai/core-concepts/multi-user-mode/permissions-system/tenants
Organization-level access control and permission inheritance in Cognee.
# Tenants
A tenant represents an organization or group. [Tenants](./tenants) contain [users](./users) and can be granted permissions on [datasets](./datasets), which apply to all members. This enables organization-wide access control and simplifies permission management for teams.
<Info>**Tenant-level permissions** — When a tenant is granted a permission on a dataset, all users in that tenant automatically inherit that permission.</Info>
## Tenant Concept
[Tenants](./tenants) are created by [users](./users) who become the tenant owner. The owner can add other [users](./users) to the tenant. [Users](./users) can belong to at most one tenant, but [tenants](./tenants) can contain multiple [users](./users).
## Tenant-Level Permissions
When a [tenant](./tenants) is granted a permission on a [dataset](./datasets), all [users](./users) in that tenant automatically inherit that permission. This happens through the permission checking mechanism: `get_all_user_permission_datasets()` unions the [user](./users)'s direct permissions with their [tenant](./tenants)'s permissions.
[Tenants](./tenants) start with zero permissions. You can leave the tenant principal empty and manage access purely through individual [user](./users) permissions, or grant tenant-wide permissions for organization-wide resources.
## Permission Inheritance
Tenant-level grants are blanket: once a [dataset](./datasets) permission is assigned to the tenant principal, every [user](./users) whose `tenant_id` matches inherits it. [Users](./users) can also receive direct permissions that differ from the tenant defaults, giving you flexibility to customize access for specific [users](./users) within the same tenant.
<Accordion title="Tenant Model Fields">
The Tenant model defines what gets stored in the SQL database. The `tenants` table contains:
* `id`: Unique identifier (UUID primary key, references principals.id)
* `name`: Human-readable name (unique)
* `owner_id`: ID of the [user](./users) who created the tenant
</Accordion>
<Accordion title="Tenant Creation">
* `create_tenant(tenant_name, user_id)`: Creates a new tenant with the specified [user](./users) as owner
* `add_user_to_tenant(user_id, tenant_id, owner_id)`: Adds an existing [user](./users) to a tenant (owner only)
</Accordion>
<Accordion title="Removing Members and Tenant Deletion">
* `remove_user_from_tenant(user_id, tenant_id, owner_id)`: Removes a [user](./users) from the tenant. Also strips them from all [roles](./roles) in the tenant and revokes their permissions on tenant datasets. The tenant owner cannot be removed. Exposed via `DELETE /v1/permissions/tenants/{tenant_id}/users/{user_id}`.
**Tenant deletion is not yet supported.** There is no `delete_tenant` API or function. As a workaround, depopulate the tenant by calling `remove_user_from_tenant` for every non-owner member, then stop using the tenant.
</Accordion>
<Accordion title="Limitations">
* [Users](./users) without a tenant exist but are isolated
* API endpoints for tenant management
</Accordion>
## Tenant Management
Tenant owners can:
* Add [users](./users) to the tenant
* Remove [users](./users) from the tenant
* Grant permissions to the tenant principal
* Manage tenant-level access to [datasets](./datasets)
* Delete the tenant
<Info>**Tenant deletion** — Any authenticated user can delete a tenant.</Info>
## Use Cases
Tenants are ideal for:
* **Organization-wide access** — Grant broad permissions to all team members
* **Department-level isolation** — Keep different departments' data separate
* **Project-based grouping** — Organize users around specific projects or initiatives
* **Scalable permission management** — Avoid granting individual permissions to many users
## Data Isolation
Each tenant's data is completely isolated:
* **Database separation** — Each user's data is stored in their own directory
* **Permission boundaries** — [Users](./users) can only access [datasets](./datasets) they have permissions for
* **No cross-tenant access** — Data from one tenant cannot be accessed by [users](./users) from another tenant
## Best Practices
* **Start with tenant-level permissions** — Grant broad access at the tenant level
* **Refine with [user](./users) permissions** — Override tenant defaults for specific [users](./users) when needed
* **Use [roles](./roles) for granular control** — Create [roles](./roles) within tenants for more specific access patterns
* **Regular permission audits** — Review and update permissions as team structure changes
<Columns>
<Card title="Roles" icon="users" href="/core-concepts/multi-user-mode/permissions-system/roles">
Learn about role-based permissions within tenants
</Card>
<Card title="ACL" icon="shield" href="/core-concepts/multi-user-mode/permissions-system/acl">
Understand how permissions are stored and checked
</Card>
</Columns>
# Users
Source: https://docs.cognee.ai/core-concepts/multi-user-mode/permissions-system/users
Individual users and authentication in Cognee's permission system.
# Users
Users are the most common type of principal and the primary way people access the system. They authenticate through email and password and can own datasets, be granted permissions on others.
<Info>**Default user behavior** — When no user is specified, Cognee uses a default user with email "[default\_user@example.com](mailto:default_user@example.com)" for development and testing.</Info>
## User Authentication
Users authenticate through email and password.
## User Management
Users can:
* Own [datasets](./datasets) and be granted permissions on others
* Belong to a [tenant](./tenants)
* Have direct permissions on [datasets](./datasets)
* Inherit permissions from their [tenant](./tenants) and [roles](./roles)
<Accordion title="User Model Fields">
The User model defines what gets stored in the SQL database. The `users` table contains:
* `id`: Unique identifier (UUID primary key, references principals.id)
* `email`: User's email address (unique)
* `hashed_password`: Encrypted password
* `tenant_id`: ID of the [tenant](./tenants) the user belongs to (nullable)
* `parent_user_id`: ID of the parent user (UUID, nullable, self-referencing FK to `users.id`). When an agent or service user creates datasets, the parent user automatically inherits full permissions on those datasets. Set to `null` for regular human users.
* `is_active`: Whether the user account is active
* `is_verified`: Whether the user's email is verified
* `is_superuser`: Whether the user has superuser privileges
</Accordion>
<Accordion title="User Creation">
* `create_user(email, password, is_superuser=False, is_active=True, is_verified=False, auto_login=False, parent_user_id=None)`: Creates a new user with specified credentials.
* `parent_user_id` (optional): UUID of a parent user. Pass this when creating agent or service users so the parent user automatically receives permissions on any datasets those users create.
* `is_active`, `is_verified`, and `is_superuser`: Account-state and privilege flags stored on the user.
* `auto_login`: Refreshes the created user record so it is ready for an immediate login flow.
* `tenant_id` is stored on the user model, but it is not passed to `create_user(...)`; assign users to tenants through the tenant membership methods instead.
* Default user behavior: System creates "[default\_user@example.com](mailto:default_user@example.com)" if no user exists.
* See [users](/python-api/users#create_user) for the full Python API reference and examples.
</Accordion>
<Accordion title="Environment Variables">
* `DEFAULT_USER_EMAIL`: Override default user email (default: "[default\_user@example.com](mailto:default_user@example.com)")
* `DEFAULT_USER_PASSWORD`: Override default user password (default: "default\_password")
* `ENABLE_BACKEND_ACCESS_CONTROL`: Canonical posture switch (default: "true"). `true` enables multi-tenant mode with per-user/dataset isolated databases **and** auth required on HTTP endpoints. `false` switches to single-user mode (shared DB, auth requirement off).
* `REQUIRE_AUTHENTICATION`: Optional override for the HTTP auth requirement. When unset, it inherits from `ENABLE_BACKEND_ACCESS_CONTROL`; when `ENABLE_BACKEND_ACCESS_CONTROL=true`, auth is always required. See [Security & Privacy](/setup-configuration/security) for the full posture table.
* `FASTAPI_USERS_RESET_PASSWORD_TOKEN_SECRET`: Secret for password reset tokens
* `FASTAPI_USERS_VERIFICATION_TOKEN_SECRET`: Secret for email verification tokens
</Accordion>
## User Permissions
Users can receive permissions in three ways:
1. **Direct permissions** — Explicitly granted to the user
2. **[Tenant permissions](./tenants)** — Inherited through [tenant](./tenants) membership
3. **[Role permissions](./roles)** — Inherited through [role](./roles) memberships
The system calculates effective permissions by combining all three sources, giving users the union of their direct permissions, [tenant](./tenants)-level permissions, and [role](./roles)-level permissions.
## User Isolation
When `ENABLE_BACKEND_ACCESS_CONTROL=true`, each user's data is completely isolated:
* **Database routing is automatic** — Databases are configured per request via context variables and with the help of [Dataset Database Handlers](/core-concepts/multi-user-mode/dataset-database-handlers/dataset-database-handlers-what-are-they)
* **Filesystem isolation** — Each user gets their own database directory
* **No unauthorized access** — Users can only access [datasets](./datasets) they have explicit permissions for
## Superuser Privileges
Users with `is_superuser=True` have additional privileges:
* Can manage other [users](./users), [tenants](./tenants), and [roles](./roles)
* Can write system LLM and vector-database settings (`POST /api/v1/settings`) — any other authenticated caller receives `403 Forbidden` with `{"error": "Superuser privileges required to modify settings"}`. Reading the settings with `GET /api/v1/settings` only requires authentication.
* Can perform administrative operations
Superuser status does **not** bypass [dataset](./datasets) permissions. Dataset access — including the dataset list the web UI shows — always resolves through explicit grants (direct, [tenant](./tenants), or [role](./roles)), so a superuser sees only datasets they hold `read` on, like any other user. To give an admin visibility into datasets created by agent or service users, create those users with `parent_user_id` set to the admin (permissions on new datasets are then inherited automatically), or grant access per dataset — see [Permission Snippets](/guides/permission-snippets).
The auto-created default user (`default_user@example.com`) is a superuser, so single-user and local setups can save settings out of the box. Users created with `create_user(...)` are not superusers unless you pass `is_superuser=True`, so integrations and automation that change LLM or vector-DB configuration over HTTP must authenticate as a superuser account.
<Warning>**Production Security** — Superuser privileges should be carefully managed in production environments.</Warning>
<Columns>
<Card title="Tenants" icon="building" href="/core-concepts/multi-user-mode/permissions-system/tenants">
Learn about organization-level access control
</Card>
<Card title="Roles" icon="users" href="/core-concepts/multi-user-mode/permissions-system/roles">
Understand role-based permissions within tenants
</Card>
</Columns>
# Core Concepts Overview
Source: https://docs.cognee.ai/core-concepts/overview
Learn the core concepts behind Cognee memory, retrieval, and pipeline design.
## Introduction
Cognee is an open source tool and platform that transforms your raw data into intelligent, searchable memory. It combines vector search with graph databases to make your data both searchable by meaning and connected by relationships.
<Info>**Dual storage architecture** gives you both semantic search and structural reasoning</Info>
<Tip>**Modular design** composes [Tasks](./building-blocks/tasks), [Pipelines](./building-blocks/pipelines), and [DataPoints](./building-blocks/datapoints)</Tip>
<Note>**Main operations** in Cognee v1.0 are `remember`, `recall`, `improve`, and `forget`, with legacy `add`, `cognify`, `memify`, and `search` still available as lower-level building blocks.</Note>
## Table of Contents
<Accordion title="Architecture">
Cognee uses three complementary storage systems, each playing a different role:
* **Relational store** — Tracks documents, chunks, and provenance (where data came from and how it's linked)
* **Vector store** — Holds embeddings for semantic similarity (numerical representations that find conceptually related content)
* **Graph store** — Captures entities and relationships in a knowledge graph (nodes and edges that show connections between concepts)
This architecture makes your data both **searchable** (via vectors) and **connected** (via graphs). Cognee ships with lightweight defaults that run locally, and you can swap in production-ready backends when needed.
For detailed information about the storage architecture, see [Architecture](./architecture).
</Accordion>
<Accordion title="Building Blocks">
Cognee's processing system is built from three fundamental components:
* **[DataPoints](./building-blocks/datapoints)** — Structured data units that become graph nodes, carrying both content and metadata for indexing
* **[Tasks](./building-blocks/tasks)** — Individual processing units that transform data, from text analysis to relationship extraction
* **[Pipelines](./building-blocks/pipelines)** — Orchestration of Tasks into coordinated workflows, like assembly lines for data transformation
These building blocks work together to create a flexible system where you can:
* Use built-in Tasks for common operations
* Create custom Tasks for domain-specific logic by extending DataPoints
* Compose Tasks into Pipelines that match your workflow
</Accordion>
<Accordion title="Main Operations">
Cognee provides four main operations that users interact with:
* **[Remember](./main-operations/remember)** — Store new memory in one call, either as permanent graph-backed memory or as session memory
* **[Recall](./main-operations/recall)** — Query stored memory with session-aware and graph-backed retrieval
* **[Improve](./main-operations/improve)** — Enrich existing memory and optionally bridge session memory into the permanent graph
* **[Forget](./main-operations/forget)** — Remove memory at the item, dataset, or user scope
**Note:** These v1.0 operations are the primary user-facing workflow.
</Accordion>
<Accordion title="Further Concepts">
Beyond the core workflow, Cognee offers advanced features for sophisticated knowledge management:
* **[Node Sets](./further-concepts/node-sets)** — Tagging and organization system that helps categorize and filter your knowledge base content
* **[Agent Memory Decorator](./further-concepts/agent-memory-decorator)** — A clean way to attach Cognee memory retrieval to an async agent function
* **[Ontologies](./further-concepts/ontologies)** — External knowledge grounding through RDF/XML ontologies that connect your data to established knowledge structures
* **[Loaders](./further-concepts/loaders)** — Components that handle reading and normalizing various file formats into text
* **[Chunkers](./further-concepts/chunkers)** — Tools for splitting documents into manageable pieces for processing and embedding
These concepts extend Cognee's capabilities for:
* **Organization** — Managing growing knowledge bases with systematic tagging
* **Knowledge grounding** — Connecting your data to external, validated knowledge sources
* **Domain expertise** — Leveraging existing ontologies for specialized fields like medicine, finance, or research
</Accordion>
## Next steps
A good way to learn Cognee is to start with its [architecture](./architecture), move on to [building blocks](./building-blocks/datapoints), practice the [main operations](./main-operations/remember), and finally explore [advanced features](./further-concepts/node-sets).
<Columns>
<Card title="Architecture" icon="building" href="/core-concepts/architecture">
Understand Cognee's three storage systems and how they work together
</Card>
<Card title="Building Blocks" icon="puzzle" href="/core-concepts/building-blocks/datapoints">
Learn about DataPoints, Tasks, and Pipelines that power the system
</Card>
<Card title="Main Operations" icon="play" href="/core-concepts/main-operations/remember">
Learn the v1.0 workflow with Remember, Recall, Improve, and Forget
</Card>
</Columns>
# Sessions and Caching
Source: https://docs.cognee.ai/core-concepts/sessions-and-caching
Learn how Cognee handles short-term memory with sessions and caching.
In Cognee, a session defines the scope for a single conversation or agent run. It maintains a cache of short-term information, including recent queries, responses, and the context used to answer them.
## What Is a Session?
A session is Cognee's short-term memory for a specific user. It is identified by `(user_id, session_id)` and stores an ordered list of recent interactions.
In the v1.0 API, you interact with sessions through [`remember()`](/core-concepts/main-operations/remember) and [`recall()`](/core-concepts/main-operations/recall):
* `cognee.remember(data, session_id="my_session")` — writes content directly into the session cache for fast retrieval.
* `cognee.recall(query_text, session_id="my_session")` — searches session cache entries first, then falls through to the permanent graph if nothing matches.
The lower-level [`cognee.search()`](/core-concepts/main-operations/legacy-operations/search) also accepts `session_id`. Session-aware retrieval is used across the main completion-oriented search paths, including graph-completion variants, RAG, hybrid, triplet, temporal, and agentic retrieval.
For session-aware retrieval, omitting `session_id` still stores the turn when caching is enabled — it does not disable sessions. The session it lands in is **scoped to the dataset**: when Cognee knows which dataset the call runs against, the default resolves to `default_session_<dataset_id>` (the dataset's UUID appended to `default_session`). Because every dataset gets its own default session, two datasets can no longer mix their turns into one shared conversation. Only when no dataset is known at all does the default fall back to the plain global `default_session`. This is different from `remember()`, where omitting `session_id` writes directly to permanent memory instead of creating a session.
Reads follow the same rule as writes, so a turn written without a `session_id` is readable back without one. A bare `cognee.session.get_session()` outside any dataset context resolves to the default session of your existing `main_dataset` — the dataset Cognee uses when no dataset is stated. If no `main_dataset` exists yet, the call raises a `SessionPreconditionError` (a `CogneeValidationError`) rather than silently returning the unscoped global session; add data first, run inside a dataset context, or pass an explicit `session_id`. To scope conversations explicitly, pass your own `session_id` — an explicit value is always stored and read unchanged, with no dataset suffix applied.
Previously, an omitted `session_id` always resolved to the single global `default_session`, so turns from different datasets shared one history. Entries already stored there are not migrated — pass `session_id="default_session"` explicitly to keep reading them.
Sessions only affect completion-oriented search. The completion search types — `GRAPH_COMPLETION` and its variants (`GRAPH_COMPLETION_COT`, `GRAPH_COMPLETION_CONTEXT_EXTENSION`, `GRAPH_SUMMARY_COMPLETION`, `TEMPORAL`), `RAG_COMPLETION`, `HYBRID_COMPLETION`, `TRIPLET_COMPLETION`, and `AGENTIC_COMPLETION` — read and write session history. Retrieval-only types (`CHUNKS`, `SUMMARIES`, etc.) accept `session_id` but do not use or write session history. For multi-tenant or background jobs, pass an explicit `user` so the default user is not used.
Cognee reads from session memory at the start of a retrieval to recover earlier turns. When the retrieval finishes, it writes a new interaction to the session so the history grows over time.
Using the same `session_id` across calls allows Cognee to include previous interactions as conversational history in the LLM prompt, enabling follow-up questions and contextual awareness.
To inspect stored history, use `cognee.session.get_session(session_id=..., last_n=...)`. To annotate a stored entry, use `cognee.session.add_feedback(...)` and `cognee.session.delete_feedback(...)`.
<Note>
Sessions require caching to be enabled. See the next sections and Configuration Details below. If caching is disabled or unavailable, searches still work but without access to previous interactions.
</Note>
## Session Cache vs Permanent Memory
Cognee keeps two distinct kinds of memory. `remember()` writes to one or the other depending on whether you pass `session_id`:
| | Session cache (short-term) | Permanent memory (knowledge graph) |
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **How to write** | `remember(data, session_id="...")` | `remember(data)` (no `session_id`) |
| **What happens** | Raw text is written straight to the cache as a Q\&A entry — no chunking, no entity extraction, no embeddings | Runs the full [Add](/core-concepts/main-operations/legacy-operations/add) + [Cognify](/core-concepts/main-operations/legacy-operations/cognify) pipeline: chunking, entity/relationship extraction, and embeddings, plus an [Improve](/core-concepts/main-operations/improve) pass |
| **Latency / cost** | Near-instant, no LLM calls | Heavier — LLM and embedding calls scale with input size |
| **Scope** | One conversation, keyed by `(user_id, session_id)` | A named dataset, shared across all sessions |
| **Lifetime** | Expires roughly `SESSION_TTL_SECONDS` (default 7 days) after the session's last write, and is also cut short when the data it was built on is deleted (see *Invalidation when the underlying data is deleted* under [Additional Information](#additional-information)) | Durable until you [Forget](/core-concepts/main-operations/forget) it |
| **Best for** | Conversation turns, scratch context, recent interactions | Documents, facts, anything you want to query later as a graph |
Passing `session_id` does **not** run graph extraction on that content — the write is raw and fast by design. This is why `remember(data, session_id=...)` does not, on its own, place data in the permanent graph. When `self_improvement=True` (the default), it additionally kicks off a background [Improve](/core-concepts/main-operations/improve) pass that bridges cached turns, agent traces, and accepted distilled session guidance into the permanent graph; with `self_improvement=False`, the content stays in the cache only until you explicitly call `cognee.improve(dataset=..., session_ids=[...])`. To write straight to permanent memory, call `remember()` without a `session_id`.
## How Sessions Work
Sessions integrate with both the v1.0 operations and the lower-level search pipeline.
**v1.0 session flow (via `recall`):**
When you call `cognee.recall(query_text, session_id="my_session")`:
1. **Check session cache** – Cognee searches the session cache for matching entries using keyword matching
2. **Fall through to graph** – If no session entries match, retrieval continues against the permanent knowledge graph
3. **Return tagged results** – Results include a `_source` field indicating whether they came from `"session"` or `"graph"`
**Lower-level session flow (via `search`):**
When you call `cognee.search()` with a `session_id`:
1. **Retrieve context** – Cognee finds relevant graph elements for your query
2. **Load conversation history** – If caching is enabled, previous interactions for `(user_id, session_id)` are loaded
3. **Generate answer** – The LLM receives the query, graph context, and retrieved history
4. **Save interaction** – A new Q\&A entry is stored in the session cache
## Cache Adapters
Cognee supports three cache adapters for storing sessions: SQL (the default — SQLite or Postgres), Redis, and Filesystem. Redis or Postgres keep the cache in an external service so it outlives the local machine, while SQLite (the default) and Filesystem give you a simple local cache without network dependencies. All provide the same functionality; only the storage backend differs. Below are the configuration options for each adapter with additional details.
<Tabs>
<Tab title="SQL (default)">
The default backend stores sessions in a SQL database via SQLAlchemy — SQLite for a zero-setup local cache, or Postgres for a cache hosted in an external database:
```dotenv theme={null}
CACHING=true
CACHE_BACKEND=sqlite # default; or postgres
```
With `CACHE_BACKEND=sqlite` and no further configuration, sessions live in a `cache.db` file next to the relational SQLite database. With `CACHE_BACKEND=postgres`, the connection falls back to the relational `DB_*` settings. Either backend can point at a specific database with `CACHE_DB_URL`:
```dotenv theme={null}
CACHE_DB_URL=postgresql+asyncpg://cognee:cognee@localhost:5432/cognee_db
```
* `sqlite`: zero setup, no network dependency; local to one machine
* `postgres`: hosted in an external database, so the cache survives the local machine
* Both run the same SQL adapter; only the connection-URL resolution differs
</Tab>
<Tab title="Redis">
Add to your `.env` file:
```dotenv theme={null}
CACHING=true
CACHE_BACKEND=redis
CACHE_HOST=localhost
CACHE_PORT=6379
```
**Start Redis:**
```bash theme={null}
# Using Docker
docker run -d -p 6379:6379 redis:latest
# Or using local installation
redis-server
```
For managed Redis services that require in-transit encryption, add TLS settings to the same Redis configuration:
```dotenv theme={null}
CACHE_HOST=my-cache.example.cache.amazonaws.com
CACHE_USERNAME=default
CACHE_PASSWORD=your_password
CACHE_SSL=true
CACHE_SSL_CERT_REQS=required
```
* Fast in-memory storage
* Requires a running Redis instance and network connectivity
* Optional TLS for managed Redis via `CACHE_SSL` / `CACHE_SSL_CERT_REQS`
</Tab>
<Tab title="Filesystem">
**Configuration:**
Add to your `.env` file:
```dotenv theme={null}
CACHING=true
CACHE_BACKEND=fs
```
* Sessions are stored in `{DATA_ROOT_DIRECTORY}/.cognee_fs_cache/sessions_db`.
* Stores session data on the local filesystem using `diskcache`
* No network dependency
</Tab>
</Tabs>
## Additional Information
<AccordionGroup>
<Accordion title="Invalidation when the underlying data is deleted">
A session's lifetime is not only bounded by `SESSION_TTL_SECONDS`. Deleting the data a session was built on also removes the cached turns that quoted it, so completions and session context stop asserting content that no longer exists.
Sessions are attributed to a dataset — through the `dataset_id` recorded on the session, or through the per-dataset default session id (`default_session_<dataset_id>`) — and that attribution decides which sessions a delete drops **whole**. It does not bound what a delete reaches: on top of dropping attributed sessions, every delete also runs a targeted pass over the sessions carrying **no** dataset attribution — the plain global `default_session` an unscoped or cross-dataset search runs in. Matching turns by the graph element ids they recorded is what makes that wider scan safe: only turns that actually used the deleted elements are removed, so an answer built from the deleted content is scrubbed even when its session was never tied to the dataset.
* **Dataset-level deletes** — [`forget(dataset=...)`](/core-concepts/main-operations/forget), `forget(dataset=..., memory_only=True)`, and [`datasets.empty_dataset()`](/python-api/datasets#datasets-empty_dataset) — delete every session attributed to that dataset, then run the targeted pass below across dataset-unattributed sessions.
* **Single-document deletes** — [`forget(data_id=..., dataset=...)`](/core-concepts/main-operations/forget) and [`datasets.delete_data()`](/python-api/datasets#datasets-delete_data) — remove only the contaminated entries. A turn is contaminated when the graph elements it recorded using overlap the nodes and edges the delete removed; contamination then follows the provenance chain inside that session (turn → feedback referencing it → session-context lesson distilled from that feedback → later turns that consumed the lesson) and everything on the chain is removed. Unrelated turns in the same session survive.
* **`forget(everything=True)`** still prunes the whole cache.
* **dlt ingestion paths** — re-ingesting a changed [dlt source](/integrations/dlt-integration#re-ingesting-a-source) (which purges the source's previously derived artifacts before re-emitting the current rows) and dlt orphan cleanup (rows removed upstream) both run the same targeted, contamination-propagating pass as a single-document delete. Invalidation is therefore not limited to explicit deletes: an ingestion that removes graph elements scrubs the session turns built on them too.
This cleanup is **best-effort by contract**: it must never fail the delete that triggered it. A cache error is logged as a warning and the graph, vector, and relational deletions complete regardless.
Known limits:
* Agent-trace entries store context as text without graph element ids, so they are not matched by the targeted pass.
* The `tapes` cache backend is append-only and never sees deletes.
* Sessions created before dataset attribution existed are only discoverable through the `default_session_<dataset_id>` naming. This limits which sessions can be dropped whole, not what the targeted pass reaches — a session with no attribution at all is scanned regardless of when it was created.
</Accordion>
<Accordion title="Session Lifecycle Persistence (Relational DB)">
In addition to the cache layer, Cognee persists session lifecycle metadata to the relational database (SQLite or Postgres). Running database migrations — either via `await cognee.run_migrations()` or `alembic upgrade head` — creates the required relational tables for this metadata.
Two tables are created:
<Tabs>
<Tab title="session_records">
One row per `(user_id, session_id)`:
| Column | Type | Description |
| ------------------ | -------------------- | --------------------------------------------------------------------------- |
| `session_id` | String (PK) | The caller-supplied session identifier. |
| `user_id` | UUID (PK) | The owning user. Same `session_id` from two users is two separate sessions. |
| `dataset_id` | UUID (nullable) | Associated dataset, if any. |
| `status` | String | Stored status: `running`, `completed`, or `failed`. |
| `started_at` | Timestamp | When the session started. |
| `last_activity_at` | Timestamp | When the session last received an LLM call. |
| `ended_at` | Timestamp (nullable) | When the session was marked completed or failed. |
| `tokens_in` | Integer | Cumulative input tokens across all LLM calls in this session. |
| `tokens_out` | Integer | Cumulative output tokens. |
| `cost_usd` | Float | Estimated cumulative cost in USD. |
| `error_count` | Integer | Number of errors recorded in this session. |
| `last_model` | Text (nullable) | Most recently used LLM model name. |
</Tab>
<Tab title="session_model_usage">
One row per `(session_id, user_id, model)`:
| Column | Type | Description |
| ------------ | ----------- | ------------------------------------------- |
| `session_id` | String (PK) | The session. |
| `user_id` | UUID (PK) | The owning user. |
| `model` | Text (PK) | The model name (e.g. `openai/gpt-4o-mini`). |
| `tokens_in` | Integer | Input tokens attributed to this model. |
| `tokens_out` | Integer | Output tokens attributed to this model. |
| `cost_usd` | Float | Cost attributed to this model. |
| `updated_at` | Timestamp | When this row was last updated. |
</Tab>
</Tabs>
Splitting per-model usage out of `session_records` allows mixed-model sessions (e.g. a completion model plus an embedding model) to attribute cost correctly.
Because the token/cost estimate is computed locally from prompt and completion text, this tracking works the same way across every configured [LLM provider](/setup-configuration/llm-providers) — OpenAI, Anthropic, Gemini (Google AI Studio and Vertex AI), Bedrock, Mistral, Ollama, and any `custom` LiteLLM-routed provider. It does not depend on the provider returning usage metadata, and it works in Docker and self-hosted deployments as long as caching is enabled.
Cost figures come from Cognee's built-in pricing table keyed on `LLM_MODEL`. If your model is not in that table (for example, a Vertex AI custom endpoint or a self-hosted `custom` model), `tokens_in` and `tokens_out` are still recorded, but `cost_usd` may be `0` or based on a fallback rate. Use the token counts for those models and compute cost from your provider's pricing.
Tracking depends on caching being available, and only reflects **session-scoped completion calls**, not every LLM-capable step in the broader ingestion pipeline — see [Recall](/core-concepts/main-operations/recall) for retrieval and `only_context=True`, and [Cognify](/core-concepts/main-operations/legacy-operations/cognify) for ingestion-time call counts. If you omit `session_id`, usage is still tracked, attributed per dataset to `default_session_<dataset_id>` (see [What Is a Session?](#what-is-a-session) above) rather than accumulating on one shared `default_session`.
**Session visibility rules**
Each `session_records` row remains keyed by the `user_id` that actually created the session, but the HTTP read paths can surface any rows visible to the requesting user at read time. That includes the requesting user's own sessions, sessions created by child-agent users whose `parent_user_id` matches the requesting user's `id`, and sessions visible through dataset read permissions. See [Users](/core-concepts/multi-user-mode/permissions-system/users) for how to create agent users with `parent_user_id`.
```http theme={null}
GET /api/v1/sessions
GET /api/v1/sessions/{session_id}
GET /api/v1/sessions/stats?range=30d
GET /api/v1/sessions/cost-by-model?range=30d
```
* `GET /api/v1/sessions` lists sessions visible to the caller
* `GET /api/v1/sessions/{session_id}` returns per-session fields such as `tokens_in`, `tokens_out`, and `cost_usd`
* `GET /api/v1/sessions/stats?range=30d` returns aggregate totals for `24h`, `7d`, `30d`, or `all`
* `GET /api/v1/sessions/cost-by-model?range=30d` breaks usage down by model
**Session status lifecycle:**
Sessions move through: `running` → `completed` or `failed`. The `abandoned` status is never written to the database — it is computed at read time: a session whose `last_activity_at` is older than the abandonment threshold and is still in `running` state is reported as `abandoned`. The threshold defaults to 30 minutes and is configurable:
```dotenv theme={null}
SESSION_ABANDON_AFTER_SECONDS=1800 # default: 30 minutes
```
This means no background sweeper is needed to mark stale sessions. Reads include the effective status automatically.
<Note>
Token counts use a character-based estimate (`len(text) // 4`) when the LLM client does not return exact usage counts. These are approximate and suitable for dashboard aggregates rather than precise billing.
</Note>
</Accordion>
<Accordion title="Session Data Structure">
Sessions store interactions as JSON entries in a list. Each item returned by `cognee.session.get_session()` is a `SessionQAEntry` model with the following fields:
| Field | Type | Description |
| ------------------------ | -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `time` | `str` | ISO 8601 timestamp when the entry was created. |
| `qa_id` | `Optional[str]` | Unique identifier for the Q\&A turn. Cognee assigns a UUID when the turn is stored; use this ID with feedback and per-entry update APIs. |
| `question` | `str` | The user's original query text. |
| `context` | `str` | Retrieved context used to answer the question. May be empty if summarization is not enabled. |
| `answer` | `str` | The generated answer. |
| `feedback_text` | `Optional[str]` | Free-form feedback text, or `None` if not set. |
| `feedback_score` | `Optional[int]` | Integer rating from `1` to `5`, or `None` if not set. |
| `used_graph_element_ids` | `Optional[Dict[str, List[str]]]` | Graph node and edge IDs used during retrieval. Keys are `node_ids` and `edge_ids`. |
| `memify_metadata` | `Optional[Dict[str, bool]]` | Session persistence and memify status flags, such as `feedback_weights_applied`. |
Sessions are keyed by `agent_sessions:{user_id}:{session_id}`.
Each user can have multiple sessions, each maintaining its own cache of short-term information.
</Accordion>
<Accordion title="Reading Session History (get_session())">
Use `cognee.session.get_session()` to retrieve stored Q\&A entries for a session. Entries are returned in chronological order (oldest first) — use `entries[-1]` for the most recent entry. If the session does not exist or the cache backend is unavailable, this call returns an empty list instead of raising an error.
<ParamField type="Optional[str]">
Identifier of the session to retrieve. When set, it must match the `session_id` previously passed to `cognee.recall()`. When `None`, it resolves to the same dataset-scoped default session the write side uses (see [What Is a Session?](#what-is-a-session) above). Pass the literal `"default_session"` to read the global (legacy) session instead.
</ParamField>
<ParamField type="Optional[int]">
Maximum number of most-recent entries to return. When `None`, all stored entries are returned.
</ParamField>
<ParamField type="Optional[User]">
User that owns the session. When `None`, Cognee resolves it from the current session context or falls back to the default user.
</ParamField>
Returns `List[SessionQAEntry]` (see [Session Data Structure](#session-data-structure) above), which may be empty.
```python theme={null}
import cognee
entries = await cognee.session.get_session(session_id="conversation_1", last_n=5)
for entry in entries:
print(f"[{entry.time}] Q: {entry.question}")
print(f" A: {entry.answer}")
print(f" feedback score: {entry.feedback_score}")
```
</Accordion>
<Accordion title="Upstream context and the include_context flag">
Every Q\&A entry stored in the session cache contains a `context` field. Depending on how the completion was generated, this field may be empty or may contain a stored summary of the retrieved context for that turn. You can inspect it programmatically when reading session history.
**The `include_context` flag:**
<Note>
`get_session_manager()` is an internal, lower-level API rather than the usual SDK entry point. Prefer `cognee.session.get_session()` unless you specifically need formatted history control such as `include_context`.
</Note>
`SessionManager.get_session()` and `SessionManager.format_entries()` both accept `include_context: bool` (default `True`). When `True`, a `CONTEXT:` line is included for each entry in the formatted history string; when `False`, it is omitted.
This flag is not exposed on `cognee.recall()` or `cognee.session.get_session()`. If you need it, use the lower-level `SessionManager` directly.
**Additional information:**
<AccordionGroup>
<Accordion title="Example: reading context from past entries">
```python theme={null}
import cognee
entries = await cognee.session.get_session(session_id="my_session", last_n=5)
for entry in entries:
print("Q:", entry.question)
print("Stored context:", entry.context) # may be empty or a stored summary
print("A:", entry.answer)
```
</Accordion>
<Accordion title="Does the LLM automatically see context from previous turns?">
No. When Cognee builds conversation history for the LLM during `cognee.recall()`, it uses `include_context=False` internally — previous questions and answers are included in the prompt, but the stored context from those earlier turns is omitted. Fresh graph context is retrieved for the current query only. This keeps prompts compact and avoids re-sending large context blobs.
</Accordion>
<Accordion title="Using SessionManager directly">
If your use-case requires the LLM to see stored context from a prior turn — for example to trace provenance or build a richer prompt — use the lower-level `SessionManager` to retrieve formatted history with `include_context=True`, then pass that history to your own LLM call.
```python theme={null}
from cognee.infrastructure.session.get_session_manager import get_session_manager
from cognee.modules.users.methods import get_default_user
user = await get_default_user()
sm = get_session_manager()
# Formatted string with CONTEXT included per entry (default)
history_with_context = await sm.get_session(
user_id=str(user.id),
session_id="my_session",
formatted=True,
include_context=True, # includes CONTEXT: line for each Q&A turn
)
```
</Accordion>
</AccordionGroup>
</Accordion>
<Accordion title="Configuration Details">
**Environment Variables:**
* `CACHING` (bool): Enable/disable caching (default: `true`). Set to `false` to disable session storage and conversational memory.
* `AUTO_FEEDBACK` (bool): Enable automatic session-context guidance and feedback detection on each answered turn (default: `true`). Requires `CACHING` to be on and uses the resolved session (the dataset-scoped default session, `default_session_<dataset_id>`, when `session_id` is omitted). When enabled, every search turn — retrieval-only types included — runs one extra structured-output LLM call to analyze the turn against the previous one (skipped when `only_context=True`). `SESSION_SEARCH_MODE` decides whether that call runs alongside the answer or before retrieval (see [Session-context guidance](#session-context-guidance-auto-feedback) below). Set to `false` to disable the extra call and restore plain history-only sessions.
* `SESSION_SEARCH_MODE` (str): How one session turn executes — `"concurrent"` (default) or `"sequential"`. Both modes make the same two LLM calls per answered turn; they differ in how those calls are sequenced and therefore in which turn the analysis can influence. `"concurrent"` runs the turn analysis alongside retrieval and answer generation, so a turn costs roughly one answer call of wall-clock time. `"sequential"` runs the analysis first, so its rewritten query drives retrieval and its context updates reach the same turn's answer. The setting is deployment-wide — there is no per-request override. Setting `AUTO_FEEDBACK=false` skips the analysis call in both modes, but does not disable the mode itself: eligible calls in `"concurrent"` mode still run the dual-query retrieval and merge described below. See [Session-context guidance](#session-context-guidance-auto-feedback) below for the full behavioral difference and the cases that always fall back to sequential.
* `CACHE_BACKEND` (str): `"sqlite"` (default), `"postgres"`, `"redis"`, `"fs"`, or `"tapes"`. `"sqlite"` and `"postgres"` both use the SQL cache adapter and differ only in how the connection URL is resolved; when set to `"fs"`, sessions are stored on local disk; when set to `"redis"`, sessions are stored in Redis and shared across processes; when set to `"tapes"`, sessions are stored locally and new Q\&A turns are mirrored to a running Tapes ingest service.
* `CACHE_DB_URL` (str, optional): SQLAlchemy async URL for the SQL cache backends (e.g. `postgresql+asyncpg://cognee:cognee@localhost:5432/cognee_db`). When unset, `"sqlite"` uses a `cache.db` file next to the relational SQLite database and `"postgres"` falls back to the relational `DB_*` settings.
* `CACHE_HOST` (str): Redis hostname (default: `"localhost"`)
* `CACHE_PORT` (int): Redis port (default: `6379`)
* `CACHE_USERNAME` (str, optional): Redis username
* `CACHE_PASSWORD` (str, optional): Redis password
* `CACHE_SSL` (bool): Connect to Redis over TLS (default: `false`). Enable for managed Redis with in-transit encryption (AWS ElastiCache, GCP Memorystore, Azure Cache for Redis). Applies only to the `redis` backend.
* `CACHE_SSL_CERT_REQS` (str): TLS certificate verification when `CACHE_SSL` is enabled — `"required"` (default), `"optional"`, or `"none"`. Use `"none"` to skip verification (e.g. self-signed certificates).
* `SESSION_TTL_SECONDS` (int, optional): Time-to-live for cached session entries in seconds (default: `604800` — 7 days). The TTL is measured from the session's **last write**, not from when an entry was created: every write slides the session's expiry forward. Set to `0` to disable expiry — rows are then stored without an expiry, nothing is ever purged, and the sliding-TTL writes are skipped too, which is the lightest-I/O setting for long-lived agent sessions on the SQLite backend.
`CACHE_SSL` and `CACHE_SSL_CERT_REQS` are forwarded to both the sync and async Redis clients as `ssl` / `ssl_cert_reqs`. Existing plaintext Redis deployments are unaffected because `CACHE_SSL` defaults to `false`.
<Note>
**Upgrading an existing SQL-cache deployment:** the `sqlite` and `postgres` backends now write session-context entries as an upsert keyed on `(user_id, session_id, entry_id)`, which requires a unique index on the `cache_session_context` table. Fresh databases get that index when the cache tables are created, but an existing one must be migrated: run [`cognee.run_migrations()`](/python-api/run-migrations) (or `alembic upgrade head`) to remove duplicate rows accumulated before the fix and create the index.
</Note>
**Conversation history window:**
* Cognee includes up to the last 10 session entries when building LLM conversation history.
* Each entry is a full question/answer turn — a single `SessionQAEntry` holding both the user's `question` and the generated `answer` (see [Session Data Structure](#session-data-structure) above). So the window covers up to 10 prior exchanges, not 10 individual messages.
* This 10-entry window is **fixed and not configurable** — there is no environment variable for it. The configurable session settings are the cache toggle and backend (`CACHING`, `CACHE_BACKEND`, `CACHE_DB_URL`, `CACHE_HOST`, `CACHE_PORT`, `CACHE_USERNAME`, `CACHE_PASSWORD`), the Redis TLS options (`CACHE_SSL`, `CACHE_SSL_CERT_REQS`), the entry expiry (`SESSION_TTL_SECONDS`), and the abandonment threshold (`SESSION_ABANDON_AFTER_SECONDS`).
Sessions expire automatically after `SESSION_TTL_SECONDS` when that value is greater than `0`. The clock runs from the session's last write — each write slides the expiry of the session's entries forward, so an actively used session does not expire underneath you. If you set `SESSION_TTL_SECONDS=0`, sessions persist until the cache is cleared — use `cognee.prune.prune_system(..., cache=True)`, or wipe your cache backend directly (e.g. Redis keys or the filesystem cache directory).
On the SQL backends (`sqlite` and `postgres`), that slide is applied **lazily**: an entry is only re-stamped once its recorded expiry has fallen more than 5% of the TTL behind the current target. Entries therefore expire somewhere between 0.95 × `SESSION_TTL_SECONDS` and 1.0 × `SESSION_TTL_SECONDS` after the session's last write — with the 7-day default, up to about 8.4 hours before the full TTL. Treat the TTL as a lower bound with a small slack window rather than an exact deadline; if a session must survive a precise interval, size `SESSION_TTL_SECONDS` accordingly or set it to `0`. Per-user usage logs are re-stamped under the same rule. The Redis backend is unaffected and keeps exact `EXPIRE` semantics.
<Note>
**Operator note (SQL backends):** because entries are re-stamped at most once per slack window instead of on every write, a write costs roughly its own bytes rather than a rewrite of every row in the session. This removes the write amplification that could grow a SQLite `cache.db-wal` file to many times the size of `cache.db` under sustained agent traffic.
</Note>
**Graceful fallback behavior:**
* If no cache backend is configured or the cache is unavailable, `cognee.session.get_session()` returns `[]`.
* In the same situation, `cognee.session.add_feedback()` and `cognee.session.delete_feedback()` return `False`.
</Accordion>
<Accordion title="Session-context guidance (AUTO_FEEDBACK)">
When `AUTO_FEEDBACK` is enabled (the default) and `CACHING` is on, session-capable completion searches run a lightweight analysis step under the resolved session (the dataset-scoped default session, `default_session_<dataset_id>`, when `session_id` is omitted). This step is on by default — existing session usage performs one additional structured-output LLM call per answered turn.
On every turn, the analysis compares the current query against the previous turn's question, answer, and the context that was served for it, and accumulates durable, per-session **guidance** grouped into `goals`, `rules`, `preferences`, and `lessons_learned`, which can be injected into later answers in the same session.
**When** that analysis runs is set by `SESSION_SEARCH_MODE`, and the mode decides what else the analysis is allowed to do:
**`concurrent` (default)** — the analysis runs alongside retrieval and answer generation, so an answered turn costs roughly one LLM call of wall-clock time rather than two calls in a row. Its guidance is applied after the answer is produced, so it shapes the *next* turn in the session rather than the current one. In this mode the analysis' routing outputs are ignored: it can neither substitute an effective query nor gate the turn, so every turn is retrieved and answered. The analysis is bounded by a 30-second timeout and falls back to no context updates if it exceeds it.
Because retrieval cannot wait for a rewritten query in this mode, a concurrent turn retrieves **twice** — once with the raw question, and once with a deterministic, LLM-free rewrite that prefixes the question with up to the last two question/answer turns of the session (capped at 2000 characters). The retriever merges the two result sets into one under its usual `top_k` budget: items found by both lanes rank first, then the raw question's remaining items, and roughly a third of the budget is reserved for items only the rewrite found (nothing is reserved when the limit is 2 or less). The merged total never exceeds `top_k`. If one lane fails, the surviving lane's results are used as-is.
**`sequential`** — the analysis runs **before retrieval**, so its outputs can take effect on the same turn. It may derive an **effective query** used for retrieval and answer generation instead of the raw query (for example, resolving a terse follow-up into a self-contained question), and it may **gate** the turn: when the analysis determines the turn does not require retrieval, the search returns a short acknowledgement (the analysis-provided reply, or `"Got it."`) instead of running retrieval and completion. Retrieval runs once, with the effective query.
**Automatic fallback to sequential.** Concurrent mode applies only to `search()` / `recall()` calls whose resolved retriever is exactly `GraphCompletionRetriever` (`GRAPH_COMPLETION`), `HybridRetriever` (`HYBRID_COMPLETION`), `CompletionRetriever` (`RAG_COMPLETION`), or `TripletRetriever` (`TRIPLET_COMPLETION`). The match is by exact class, so subclass-based search types — `GRAPH_COMPLETION_COT`, `GRAPH_COMPLETION_CONTEXT_EXTENSION`, `GRAPH_SUMMARY_COMPLETION`, `TEMPORAL`, `AGENTIC_COMPLETION` — do not qualify. These cases run sequentially with no configuration change on your part:
* Any retriever outside the four exact classes above.
* `only_context=True` (which skips the analysis entirely, in either mode).
* Batch queries.
* `FEELING_LUCKY`, whose retriever is only resolved after routing.
* Calls with no session available for the user.
The step **fails open** in both modes — if analysis errors or no session is available, the original query is answered normally. Because guidance (and, in sequential mode, the effective query) can change retrieval inputs, answers may differ from history-only sessions. To disable this behavior and keep only conversation-history replay, set `AUTO_FEEDBACK=false` — that removes the analysis call in both modes. It does not switch off concurrent mode's dual-query retrieval: eligible calls still retrieve with both the raw question and the deterministic rewrite and merge the results. Set `SESSION_SEARCH_MODE=sequential` as well to restore single-query retrieval.
<Note>
This adds one structured-output LLM call per answered turn and its token usage in both modes. In `concurrent` mode that call overlaps the answer, so it adds little to a turn's latency; in `sequential` mode it is serialized ahead of retrieval and its latency adds to the turn. Sessions still work without it; set `AUTO_FEEDBACK=false` to opt out.
</Note>
</Accordion>
<Accordion title="Session distillation into long-term memory">
Session-context guidance is short-term until it is bridged into the graph. When you run `cognee.improve(dataset=..., session_ids=[...])`, Cognee can distill gated guidance from those sessions into permanent lesson documents.
Distillation:
* Loads session Q\&A and active session-context entries.
* Keeps only guidance that was never rated harmful and has enough confidence.
* Curates proposed durable lessons, checks them against previously distilled lessons and graph entities, and rejects lessons that are already known, unsupported, or not durable.
* Writes accepted lessons back into the dataset through `add()` + `cognify()`.
* Tags distilled lessons with `session_learnings` and a session-specific node set.
You can run this directly for one finished session:
```python theme={null}
result = await cognee.session.distill_session(
"my_session",
dataset="my_dataset",
)
```
`result.documents` contains the rendered lesson documents when the status is `completed`. Empty output can be normal when the session has no gated entries or no accepted lessons.
See [Session Distillation](/guides/session-distillation) for a full end-to-end example.
</Accordion>
<Accordion title="Adapter Comparison">
| Feature | SQLite (default) | Postgres | Redis | Filesystem | Tapes |
| ---------------- | -------------------------------- | -------------------------------- | ----------------- | ---------------------- | ---------------------------------------- |
| Storage | Local `cache.db` file (SQL) | External SQL database | In-memory (Redis) | Local disk (diskcache) | Local disk + mirrored ingest |
| Performance | Fast (local I/O) | Fast | Very fast | Fast (local I/O) | Fast local writes + network mirror |
| Network required | ❌ No | ✅ Yes | ✅ Yes | ❌ No | ⚠️ Only for mirroring to Tapes |
| Setup complexity | Low | Medium | Medium | Low | Medium |
| Best for | Default local setup, development | Production on existing SQL infra | Production | Development, local | Local session cache with Tapes ingestion |
</Accordion>
</AccordionGroup>
<Note>
Cached sessions can be persisted into the knowledge graph for long-term retrieval using [`improve()`](/core-concepts/main-operations/improve). The older [session persistence memify pipeline](/guides/memify-session-persistence) documents the legacy Q\&A persistence path.
</Note>
<Columns>
<Card title="Search" icon="search" href="/core-concepts/main-operations/legacy-operations/search">
Learn how sessions integrate with search
</Card>
<Card title="Sessions Guide" icon="code" href="/guides/sessions">
Practical examples with Redis and filesystem
</Card>
<Card title="Setup Configuration" icon="settings" href="/setup-configuration/overview">
Configure cache adapters
</Card>
</Columns>
# Teach an Agent From Its Own Tool Traces
Source: https://docs.cognee.ai/examples/agent-trace-lessons
Turn an agent's tool-call successes and failures into agent-profile session guidance, distill it into the knowledge graph, and read it back before the next run
Your coding agent has already hit the errors that matter — a test run that failed on a missing dependency, a lint pass that flagged an unused import — and the next run will hit them again unless something remembers what happened. This demo turns that raw tool-call history into guidance the agent can be handed before it starts.
## What You'll Build
Five tool-call traces from one agent working a task — a failing `pytest` run, a `uv sync` that fixed it, a passing test run, a file read, and a lint failure — are stored one by one into a session as `TraceEntry` records. Cognee extracts agent-profile lessons from them as they accumulate, then distillation rewrites the accepted lessons into markdown documents and cognifies them into the dataset, so they outlive the session. The payoff is the last act: `recall()` returns those lessons as a read-only context block for the agent-profile question "what should I know before running tests in this repo?", while the same query under the QA profile returns nothing and the raw traces are still retrievable as evidence.
The complete runnable script is
[`examples/demos/sessions/agentic_session_context_demo.py`](https://github.com/topoteretes/cognee/blob/main/examples/demos/sessions/agentic_session_context_demo.py) —
this page walks through its key moments rather than reproducing it.
## Features in Play
* [Agent Session Traces](/guides/agent-session-traces) — each tool call becomes a `TraceEntry` in the session, carrying its parameters, return value, or error message
* [Sessions and Caching](/core-concepts/sessions-and-caching) — the session cache holds both the raw traces and the agent-profile guidance extracted from them
* [Session Distillation](/guides/session-distillation) — rewrites the accepted guidance into markdown documents and cognifies them into the dataset
* [Recall](/core-concepts/main-operations/recall) — reads the guidance back with `scope` and `context_profile`, and the raw traces alongside it as evidence
## What to Expect
The excerpts below are from a real run, trimmed. The narration streams to stderr in three acts, every session-memory entry is labeled with its section and its source, and the run finishes by writing `agentic_session_context_demo_output.json` and printing its path; exact lesson wording varies by model.
**Act 1 — a failing trace becomes a lesson instantly.** Storing the failed `pytest` run writes a deterministic `failure_lessons` entry, no LLM involved: `source=live trace`.
```text theme={null}
Session memory content after trace 1
Trace
tool: run_tests
status: error
input:
command: pytest -q
error:
ModuleNotFoundError: No module named 'dotenv'
Session memory
entries (1):
- [failure_lessons] run_tests failed: ModuleNotFoundError: No module named 'dotenv' (confidence=0.85, source=live trace)
```
**Act 1 — the batch pass builds typed guidance around it.** After trace 2, the first extraction pass reads the failure-plus-fix pair and adds `success_patterns`, `workflow_state`, and a generalized failure lesson, all marked `source=batch LLM`.
```text theme={null}
Session memory content after trace 2 (first batch context extraction)
Trace
tool: run_command
status: success
input:
command: uv sync
output:
Installed 42 packages in 1.2s
Session memory
entries (4):
- [failure_lessons] run_tests failed: ModuleNotFoundError: No module named 'dotenv' (confidence=0.85, source=live trace)
- [success_patterns] When tests fail with a ModuleNotFoundError, install the missing Python packages via run_command (e.g., pip) and then re-run the tests. (confidence=0.9, source=batch LLM)
- [workflow_state] After a successful package installation step, the immediate next action should be to re-run the test suite to verify the dependency issue is resolved. (confidence=0.9, source=batch LLM)
- [failure_lessons] Avoid running tests before installing project dependencies; proactively run dependency installation (for example pip install -r requirements.txt) to prevent ModuleNotFoundError failures. (confidence=0.85, source=batch LLM)
```
**Act 1 — later traces widen the picture.** The second batch pass, after the file read, adds `environment_facts` and `tool_rules` on top; memory is now eight entries drawn from five kinds of section.
```text theme={null}
Session memory content after trace 4 (second batch context extraction)
...
Session memory
entries (8):
...
- [environment_facts] The project's config file contains a comment indicating it is managed with the 'uv' tool. (confidence=0.9, source=batch LLM)
- [environment_facts] The project's declared name in the config is 'demo'. (confidence=0.9, source=batch LLM)
- [tool_rules] When a project configuration names a specific package manager, prefer invoking that manager (via run_command) to install dependencies instead of an ad-hoc tool. (confidence=0.8, source=batch LLM)
- [success_patterns] Treat a run_command output that reports 'Installed N packages' as a successful dependency installation signal and proceed to re-run the test suite. (confidence=0.85, source=batch LLM)
```
**Act 2 — distillation makes the lessons permanent.** The pending tail is flushed, then the accepted guidance is rewritten into markdown documents and cognified into the dataset, where it outlives the session.
```text theme={null}
final batch context extraction: pending 1 -> 0
context entries touched: 3
...
Graph documents written
status: completed
document 1:
# Session learning — 2026-09-02 (session agentic_demo_session)
Install a project's Python dependencies before running its test suite (for example, pip install -r requirements.txt); if tests fail with a ModuleNotFoundError, install the missing package(s) (for example, pip install python-dotenv) and immediately re-run the test suite to verify the issue is resolved. (Learned after run_tests failed with ModuleNotFoundError: No module named 'dotenv' during a test run.)
...
```
**Act 3 — recall hands the guidance back, read-only.** The agent-profile query renders the distilled lessons as an Active session guidance block; the QA profile returns nothing, the raw traces stay retrievable as evidence, and recall performs no writes.
```text theme={null}
Agent-profile session memory recall
## Active session guidance
Items are listed oldest to newest within each section. When guidance conflicts, prefer the later item.
### Tool rules
- [09:33:44] When a project configuration names a specific package manager, prefer invoking that manager (via run_command) to install dependencies instead of an ad-hoc tool.
...
### Failure lessons
- [09:33:30] Avoid running tests before installing project dependencies; proactively run dependency installation (for example pip install -r requirements.txt) to prevent ModuleNotFoundError failures.
...
QA-profile session memory results: 0
raw trace recall results: 1
writes during recall: False
```
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — lesson extraction and distillation are LLM-backed, so the full run needs one; `--offline` skips both
* The script sets its own environment before importing cognee: `CACHING=true`, `CACHE_BACKEND=fs`, `AUTO_FEEDBACK=true`, `ENABLE_BACKEND_ACCESS_CONTROL=false`, and `LOG_LEVEL=ERROR` unless you already set it. The [filesystem cache adapter](/core-concepts/sessions-and-caching#cache-adapters) is what stores the session
* Run it from a checkout of the cognee repo; it writes `agentic_session_context_demo_output.json` into the working directory with every snapshot the run took
* The run starts by pruning data and system metadata and deleting any previous `agentic_demo_session`, so point it at a scratch instance rather than memory you want to keep
## How It Works
### Stage 1: Script the Agent's Tool Traces
```python theme={null}
TRACES = [
{
"origin_function": "run_tests",
"status": "error",
"method_params": {"command": "pytest -q"},
"error_message": "ModuleNotFoundError: No module named 'dotenv'",
},
{
"origin_function": "run_command",
"status": "success",
"method_params": {"command": "uv sync"},
"method_return_value": "Installed 42 packages in 1.2s",
},
{
"origin_function": "run_tests",
"status": "success",
"method_params": {"command": "uv run pytest -q"},
"method_return_value": "188 passed in 4.1s",
},
{
"origin_function": "read_file",
"status": "success",
"method_params": {"path": "pyproject.toml"},
"method_return_value": "[project] name = 'demo' # managed with uv",
},
{
"origin_function": "run_lint",
"status": "error",
"method_params": {"command": "uv run ruff check ."},
"error_message": "F401 'os' imported but unused",
},
]
```
These five dictionaries stand in for what a real agent's tool layer would emit: the tool name, whether the call succeeded, its input, and either the return value or the error. The story they tell — bare `pytest` fails, `uv sync` fixes it, `uv run pytest` passes — is the raw material a lesson can be drawn from, and nothing in the run tells the extractor what that lesson is.
### Stage 2: Store Each Trace and Extract as You Go
```python theme={null}
for index, trace in enumerate(TRACES, start=1):
before = await snapshot(user)
await cognee.remember(
TraceEntry(**trace),
dataset_name=DATASET_NAME,
session_id=SESSION_ID,
self_improvement=False,
user=user,
)
if drive_periodic_extraction:
await extract_pending_agent_context(
session_manager=get_session_manager(),
user_id=str(user.id),
session_id=SESSION_ID,
min_new_traces=DEMO_TRACE_EXTRACTION_INTERVAL,
overlap=DEMO_TRACE_EXTRACTION_OVERLAP,
)
```
`remember()` with a `session_id` writes each `TraceEntry` to the session cache rather than the graph. The trace-write path already runs the periodic extraction pass for you on its own interval; the demo calls it directly with `DEMO_TRACE_EXTRACTION_INTERVAL = 2` and `DEMO_TRACE_EXTRACTION_OVERLAP = 1` so a five-trace run actually shows the pass firing — after traces 2 and 4 — instead of only at the end. The snapshot taken before and after each trace is what lets the run print whether that trace advanced the processed-trace watermark.
### Stage 3: Flush the Tail and Distill Into the Graph
```python theme={null}
before_flush = await snapshot(user)
touched_ids = await extract_pending_agent_context(
session_manager=get_session_manager(),
user_id=str(user.id),
session_id=SESSION_ID,
min_new_traces=1,
)
distillation_input = await snapshot(user)
result = await distill_session(SESSION_ID, dataset=DATASET_NAME, user=user)
```
Traces 2 and 4 triggered extraction, which leaves the fifth trace — the lint failure — pending. Dropping `min_new_traces` to 1 forces that tail through, so distillation sees the complete set of lessons. `distill_session()` then rewrites the accepted guidance into markdown documents and cognifies them into the demo dataset, which is what makes the lessons outlast this session. The script reaches it through its internal module path; in your own code call it as [`cognee.session.distill_session()`](/guides/session-distillation), or let [`improve(session_ids=[...])`](/core-concepts/main-operations/improve) run the same distillation as part of a wider pass.
### Stage 4: Recall Guidance by Profile
```python theme={null}
agent_ctx = await cognee.recall(
"what should I know before running tests in this repo?",
scope=["session_context"],
context_profile="agent",
session_id=SESSION_ID,
only_context=True,
user=user,
)
qa_ctx = await cognee.recall(
"what should I know before running tests in this repo?",
scope=["session_context"],
context_profile="qa",
session_id=SESSION_ID,
only_context=True,
user=user,
)
raw_traces = await cognee.recall(
"dotenv",
scope=["trace"],
session_id=SESSION_ID,
only_context=True,
user=user,
)
```
Three recalls against the same session show what the profile does. `scope=["session_context"]` with `context_profile="agent"` returns the distilled guidance block, ready to prepend to the next agent run; the identical query under `context_profile="qa"` comes back empty, because this demo wrote no QA-profile entries. `scope=["trace"]` reaches past the lessons to the raw evidence they were drawn from, so a lesson about `dotenv` can be traced back to the call that failed. `only_context=True` keeps all three as context reads rather than completions.
### Stage 5: Prove the Reads Changed Nothing
```python theme={null}
async def served_state(user) -> dict:
"""Map of agent-lesson id -> last_served_at, to prove recall does not stamp anything."""
rows = await get_session_manager().get_session_context_entries(
user_id=str(user.id), session_id=SESSION_ID
)
return {
row.get("id"): row.get("last_served_at")
for row in rows
if row.get("context_profile", "qa") == "agent"
}
```
The recall act captures this map of agent-lesson id to `last_served_at` before and after the three queries and compares them. Serving guidance to an agent is a read: nothing is stamped, no entry is aged, and the run prints the comparison so you do not have to take that on faith.
## Run It
```bash theme={null}
uv run python examples/demos/sessions/agentic_session_context_demo.py
```
## Offline Mode
```bash theme={null}
uv run python examples/demos/sessions/agentic_session_context_demo.py --offline
```
`--offline` runs the same trace capture and recall without any LLM calls: the periodic extraction passes are skipped, Act 2 is skipped entirely, and the only session-memory entries that appear are the deterministic ones written when a failing trace is stored. Use it to see the capture-and-recall shape of the demo without a configured provider — and, in the full run, as the baseline that shows which entries the LLM added.
<Columns>
<Card title="Agent Session Traces" icon="footprints" href="/guides/agent-session-traces">
Recording tool calls as traces and recalling them later.
</Card>
<Card title="Session Distillation" icon="graduation-cap" href="/guides/session-distillation">
How gated session guidance becomes permanent lessons in the graph.
</Card>
<Card title="Sessions and Caching" icon="message-square" href="/core-concepts/sessions-and-caching">
The session cache behind traces, guidance, and the `fs` backend this demo uses.
</Card>
<Card title="Watch a Session Become Permanent Memory" icon="repeat" href="/examples/memory-loop-walkthrough">
The same distillation loop, driven by a conversation instead of tool traces.
</Card>
</Columns>
# Agentic Procurement Decisions
Source: https://docs.cognee.ai/examples/agentic-procurement
Build an agent that researches vendor conversations, purchase history, and policy in separate memory categories, then recommends a vendor with evidence
Your team is about to sign off on 50 laptops, and the evidence for that call is scattered across two vendor sales conversations, a file of past purchase records, and a procurement policy document. A procurement agent has to read all three, keep them straight, and justify whichever vendor it picks.
## What You'll Build
Four procurement documents — two vendor conversations, a purchase-history record, and the company's procurement policies — go into cognee memory under three separate category labels. The agent then runs a research phase: nine questions, each answered only from the category that can answer it, so pricing questions never get answered from the policy file and rating questions never get answered from a sales pitch. The nine question-and-answer pairs are compiled into a single evidence block, and one final LLM call turns that block into a vendor recommendation justified by the research it just did.
The complete runnable script is
[`examples/demos/agentic/agentic_reasoning_procurement_example.py`](https://github.com/topoteretes/cognee/blob/main/examples/demos/agentic/agentic_reasoning_procurement_example.py) —
this page walks through its key moments rather than reproducing it.
## Features in Play
* [NodeSets](/core-concepts/further-concepts/node-sets) — labels each document with its memory category at write time, and scopes each recall to one category at read time
* [Remember](/core-concepts/main-operations/remember) — ingests the four documents into three labeled slices of one graph
* [Recall](/core-concepts/main-operations/recall) — answers each research question against a single category, via `node_name`
* [Inspecting Graph Completion Context](/guides/graph-completion) — `SearchType.GRAPH_COMPLETION` is the search type behind every research answer, grounding it in graph triplets
* [Low-Level LLM](/guides/low-level-llm) — `LLMGateway.acreate_structured_output` makes the final vendor call from the compiled evidence, with no retrieval of its own
## What to Expect
The excerpts below are from a real run, trimmed. Ingestion takes a few minutes before the first question prints, and because every answer and the final recommendation are live LLM calls, the wording varies from run to run. One formatting note: the script prints each recall result as a one-line `ResponseGraphEntry(...)` object; the answers below are shown with their line breaks restored so you can read them.
**Memory goes in first.** The four procurement documents — two vendor conversations, the purchase history, and the policy sheet — are ingested and cognified into their node set categories before any question is asked.
```text theme={null}
Building AI Procurement System with Memory: Cognee Integration...
Setting up procurement memory data...
...
Memory successfully populated and processed.
```
**Nine questions, each scoped to its category.** Every recall names the category it should draw from, so vendor claims, past performance, and policy limits stay separable — the same pattern repeats across `vendor_conversations`, `purchase_history`, and `procurement_policies`.
```text theme={null}
Running contextual research questions...
Category: vendor_conversations
Question:
What are the laptops that are discussed, together with their vendors?
Answer:
- Dell Precision 5570 — offered by TechSupply Solutions (Dell)
- Lenovo ThinkPad P1 — offered by TechSupply Solutions (Lenovo)
- HP ZBook Power G9 — offered by Office Solutions (HP)
...
Category: purchase_history
...
Question:
What were the satisfaction ratings for each vendor?
Answer:
1) TechSupply Solutions — 5/5 (from the 2024-01-15 purchase)
2) Office Solutions — 2/5 (from the 2024-02-20 purchase)
...
Category: procurement_policies
...
Question:
What is the minimum vendor rating for new contracts?
Answer:
Minimum vendor rating for new contracts: 4 out of 5.
...
```
**The evidence is compiled.** All nine Q/A pairs are folded into one research summary — this text, not the raw graph, is what the decision prompt sees.
```text theme={null}
Compiling structured research information for decision-making...
Compiled Research Summary:
Q: What are the laptops that are discussed, together with their vendors?
A: - Dell Precision 5570 — offered by TechSupply Solutions (Dell)
- Lenovo ThinkPad P1 — offered by TechSupply Solutions (Lenovo)
- HP ZBook Power G9 — offered by Office Solutions (HP)
...
```
**One decision, justified from the evidence.** The recommendation cites the price after discount, the delivery windows, policy compliance, and both vendors' track records — including Office Solutions failing the minimum-rating requirement.
```text theme={null}
Passing research to LLM for final procurement recommendation...
Final Decision:
Recommendation: Purchase Dell Precision 5570 from TechSupply Solutions (50 units).
Justification (concise evidence from the QA data)
- Price: $1,334 ea after the vendor’s 8% bulk discount → total $66,700 for 50 units. This is the lowest cost option (Lenovo = $71,300; HP = $75,200).
- Delivery: 2–3 weeks (well within the company max of 30 days). Lenovo is 3–4 weeks; HP standard 4–5 weeks (may exceed 30 days unless expedited).
- Policy fit: TechSupply’s 8% discount exceeds the procurement minimum (≥5% for orders > $50k).
- Vendor performance/qualification: TechSupply has a 5/5 satisfaction rating, delivered early on prior order, and had no red flags. Office Solutions scores 2/5 with delivery and communication complaints and therefore fails the minimum vendor rating requirement (≥4/5).
- Risk: Office Solutions’ HP option would need an expedite fee ($75/unit) to meet delivery limits and still costs substantially more.
...
Conclusion: TechSupply Solutions (Dell Precision 5570) is the best fit on price, delivery, policy compliance, and proven past performance.
```
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — the research phase and the final decision are both live LLM calls
* Use Ladybug or Neo4j as your [graph store](/setup-configuration/graph-stores): node sets are only supported on those two backends. The script sets `GRAPH_DATABASE_PROVIDER` to `ladybug` itself, before importing cognee, so no configuration is needed — but a `GRAPH_DATABASE_PROVIDER` in your environment will not win
* Run it from a checkout of the cognee repo: it reads its four `.txt` inputs from the sibling `agentic_reasoning_procurement_example_data/` folder, and loads your `.env` with `load_dotenv()`
* The script starts with `cognee.forget(everything=True)`, so point it at a scratch instance rather than memory you want to keep — see [Forget](/core-concepts/main-operations/forget)
## How It Works
### Stage 1: Categorize Memory by Node Set
```python theme={null}
# Initializing and pruning databases
await cognee.forget(everything=True)
# Store data in different memory categories
await cognee.remember(
data=[vendor_conversation_text_techsupply, vendor_conversation_text_office_solutions],
node_set=["vendor_conversations"],
self_improvement=False,
)
await cognee.remember(
data=previous_purchases_text,
node_set=["purchase_history"],
self_improvement=False,
)
await cognee.remember(
data=procurement_preferences_text,
node_set=["procurement_policies"],
self_improvement=False,
)
```
Three `remember()` calls write into one graph but tag their data with three different node sets: `vendor_conversations`, `purchase_history`, and `procurement_policies`. Those labels are what make the research phase possible — without them, a question about vendor ratings would retrieve sales-pitch text just as readily as the actual rating records.
### Stage 2: Scope Every Recall to One Category
```python theme={null}
async def search_memory(self, query, search_categories=None):
"""Search across different memory layers"""
results = {}
for category in search_categories:
category_results = await cognee.recall(
query_type=SearchType.GRAPH_COMPLETION,
query_text=query,
node_name=[category],
top_k=30,
)
results[category] = category_results
return results
```
`node_name` restricts retrieval to the node set named by the category, so each answer is grounded in one memory layer only. `SearchType.GRAPH_COMPLETION` means the answer is generated from graph triplets rather than raw chunks, and `top_k=30` gives each question a wide slice of that layer to reason over.
### Stage 3: Write the Research Plan
```python theme={null}
research_questions = {
"vendor_conversations": [
"What are the laptops that are discussed, together with their vendors?",
"What pricing was offered by each vendor before and after discounts?",
"What were the delivery time estimates for each product?",
],
"purchase_history": [
"Which vendors have we worked with in the past?",
"What were the satisfaction ratings for each vendor?",
"Were there any complaints or red flags associated with specific vendors?",
],
"procurement_policies": [
"What are our company’s bulk discount requirements?",
"What is the maximum acceptable delivery time for non-critical items?",
"What is the minimum vendor rating for new contracts?",
],
}
```
The research plan is a dictionary keyed by category: three questions per memory layer, each one asked only where its answer lives. Offers and delivery estimates come from the vendor conversations, past performance from the purchase history, and the thresholds a vendor must clear from the policy document.
### Stage 4: Run the Research Loop
```python theme={null}
for category, questions in research_questions.items():
print(f"Category: {category}")
research_notes[category] = []
for q in questions:
print(f"Question: \n{q}")
results = await procurement_system.search_memory(q, search_categories=[category])
top_answer = results[category][0]
print(f"Answer: \n{top_answer}\n")
research_notes[category].append({"question": q, "answer": top_answer})
```
Nine scoped recalls run in sequence, and the top answer of each is kept alongside the question that produced it. This is the agent's research phase: it gathers its own evidence before anything decides anything, and every note carries the question that justifies its presence.
### Stage 5: Compile the Evidence
```python theme={null}
research_information = "\n\n".join(
f"Q: {note['question']}\nA: {note['answer']}"
for section in research_notes.values()
for note in section
)
```
The per-category notes are flattened into one plain-text block of Q/A pairs. Category boundaries mattered during retrieval — they are what kept each answer honest — but the decision step needs to weigh price against rating against policy, so the evidence is deliberately merged back together here.
### Stage 6: Decide from the Compiled Evidence
```python theme={null}
final_decision = await LLMGateway.acreate_structured_output(
text_input=research_information,
system_prompt="""You are a procurement decision assistant. Use the provided QA pairs that were collected through a research phase. Recommend the best vendor,
based on pricing, delivery, warranty, policy fit, and past performance. Be concise and justify your choice with evidence.
""",
response_model=str,
)
```
One direct LLM call turns the compiled research into a recommendation. It does no retrieval of its own — the only facts it can cite are the ones the nine scoped recalls put in front of it, which is what makes the resulting justification traceable back to memory.
## Run It
```bash theme={null}
uv run python examples/demos/agentic/agentic_reasoning_procurement_example.py
```
## Adapting It to Your Data
The shape here generalizes to any research-then-decide agent: pick the categories your decision actually depends on, tag each source with a node set at `remember()` time, and write one small set of questions per category. Two rules keep it working — a question is only asked in the category that can answer it, and the deciding call sees the compiled notes rather than the raw documents. Swapping in your own vendors, policies, or history files means editing the data folder and the `research_questions` dictionary, not the loop around them.
<Columns>
<Card title="NodeSets" icon="tags" href="/core-concepts/further-concepts/node-sets">
How node-set labels are written and how `node_name` filters retrieval by them.
</Card>
<Card title="Recall" icon="search" href="/core-concepts/main-operations/recall">
The retrieval operation behind every research question, and its other parameters.
</Card>
<Card title="Low-Level LLM" icon="cpu" href="/guides/low-level-llm">
Calling `acreate_structured_output` directly, including Pydantic response models.
</Card>
<Card title="Inspecting Graph Completion Context" icon="route" href="/guides/graph-completion">
What `GRAPH_COMPLETION` retrieves before an answer is generated.
</Card>
</Columns>
# Mine Coding Rules From Team Chat
Source: https://docs.cognee.ai/examples/coding-rule-mining
Turn overlapping team conversations about code standards into deduplicated Rule nodes a coding agent can query before it writes code
Your team's coding standards live in chat: a principal engineer listing formatting expectations, a manager repeating that Susan reviews every merge before it lands. A coding agent needs those standards as a clean, queryable list of rules — not as two overlapping conversations it has to read again every time.
## What You'll Build
Two short chat transcripts about the team's coding standards go into memory as an ordinary graph. A second pass — a `memify()` pipeline you assemble yourself from two tasks — walks that graph's document chunks, asks an LLM which coding rules each chunk states, and writes them back as `Rule` nodes grouped under the `coding_agent_rules` node set. The enrichment task reads the rules already in that node set before extracting more, so the standards both chats mention — Susan's review, no Friday releases — land as single rules rather than duplicates. What you get out is a flat list of team rules returned by one `SearchType.CODING_RULES` recall, plus two HTML graph visualizations that show what the enrichment pass added.
The complete runnable script is
[`examples/demos/custom_pipelines/memify_coding_agent_rule_extraction_example.py`](https://github.com/topoteretes/cognee/blob/main/examples/demos/custom_pipelines/memify_coding_agent_rule_extraction_example.py) —
this page walks through its key moments rather than reproducing it.
## Features in Play
* [Remember](/core-concepts/main-operations/remember) — stores the two chat transcripts as the base graph the enrichment pass runs over
* [Memify](/core-concepts/main-operations/legacy-operations/memify) — runs the custom extraction → enrichment pipeline against that existing graph instead of ingesting anything new
* [Custom Tasks and Pipelines](/guides/custom-tasks-pipelines) — the two `Task` objects, and the `batch_size` config that decides how many chunks reach the enrichment task at once
* [NodeSets](/core-concepts/further-concepts/node-sets) — `coding_agent_rules` is both where the new `Rule` nodes are filed and where the deduplication check looks
* [Recall](/core-concepts/main-operations/recall) — pinned to [`SearchType.CODING_RULES`](/core-concepts/main-operations/legacy-operations/search) and scoped to that node set, so it returns the rules themselves rather than a generated answer
* [Graph Visualization](/guides/graph-visualization) — the before and after HTML renders that make the added rule layer visible
## What to Expect
The excerpts below come from one real run, trimmed of most log lines. Both the graph build and the rule extraction are live LLM calls, so the rule wording and the node and edge counts vary from run to run; the shape of the output does not.
**The reset and the remember call produce the "before" graph.** `Data reset complete.` follows the `forget(everything=True)` wipe, and `Text remembered successfully.` follows two add-and-cognify pipeline runs, one per chat. The `Retrieved 31 nodes` line is the visualization reading that base graph: documents, chunks, and entities, and no `Rule` node yet.
```text theme={null}
Resetting cognee data...
...
Data reset complete.
...
Text remembered successfully.
...
2026-09-10T11:19:07.910648 [info ] Retrieved 31 nodes and 49 edges in 0.00 seconds [cognee.shared.logging_utils]
...
Open file to see graph visualization after remember: ...
```
**The memify pass runs the enrichment task once per chunk.** `Retrieving full graph.` is what a `memify()` call with no `data` argument does first, and the projection is the same 31-node graph. `add_rule_associations` then starts and completes twice, because `batch_size: 1` hands it one chunk at a time; the second call only starts after the first has written its rules, which is why it can see them. The warnings between those lines (elided here) name two new `Rule` nodes after the first call and five after the second: seven rules in total, so the two standards both chats state were written once.
```text theme={null}
2026-09-10T11:19:09.775850 [info ] Retrieving full graph. [CogneeGraph]
2026-09-10T11:19:09.778634 [info ] Graph projection completed: 31 nodes, 49 edges in 0.00s [CogneeGraph]
2026-09-10T11:19:09.795031 [info ] Pipeline run started: `7371376c-3c8d-565f-86c7-ad38d09a525b` [run_tasks_with_telemetry()]
2026-09-10T11:19:09.801307 [info ] Async Generator task started: `extract_subgraph_chunks` [run_tasks_base]
2026-09-10T11:19:09.807345 [info ] Coroutine task started: `add_rule_associations` [run_tasks_base]
...
2026-09-10T11:19:22.335390 [info ] Coroutine task completed: `add_rule_associations` [run_tasks_base]
2026-09-10T11:19:22.348521 [info ] Coroutine task started: `add_rule_associations` [run_tasks_base]
...
2026-09-10T11:19:44.787578 [info ] Coroutine task completed: `add_rule_associations` [run_tasks_base]
2026-09-10T11:19:44.800784 [info ] Async Generator task completed: `extract_subgraph_chunks` [run_tasks_base]
2026-09-10T11:19:44.810365 [info ] Pipeline run completed: `7371376c-3c8d-565f-86c7-ad38d09a525b` [run_tasks_with_telemetry()]
```
**The recall returns the rules themselves, one bullet each.** The router line shows that the words "coding rules" in the query would have selected `CODING_RULES` anyway; the script pins it so the result never depends on that. Seven rules come back for the six standards the principal engineer listed: the LLM split "Typing and Docstrings" into two rules, and Susan's review and the Friday freeze each appear once even though both chats state them. Note how far the wording drifts from the chat: the extraction prompt turns each standard into a fuller policy, naming tools the team never mentioned and generalizing Susan into "a qualified reviewer". Treat the rule text as a draft to edit, not a transcript.
```text theme={null}
2026-09-10T11:19:44.839687 [info ] query_router: routed=CODING_RULES score=5.0 query='List me the coding rules' scores={'CODING_RULES': 5.0} [query_router]
...
2026-09-10T11:19:53.940534 [info ] recall: 7 results across sources=['graph'] (session=-) [recall]
Coding rules created by memify:
- Avoid scheduling production releases or deployments on Fridays or immediately before weekends and major holidays. ...
- Annotate complex or non-obvious code segments with an explicit NOTE: comment that explains why the code is complex, the intended behavior, and references to design documents or tests. ...
- Enforce PEP8-style formatting automatically: include a canonical formatter (e.g., black) and linters (e.g., flake8) in the repository, enable them via pre-commit hooks, and fail CI if code is not formatted/linted. ...
- Require type annotations for functions, methods, and public APIs, and run a static type checker (e.g., mypy or pyright) in CI. ...
- Avoid duplicate code: refactor duplicated logic into a single, well-tested helper function or module. ...
- Require docstrings for all public modules, classes, and functions following a documented style (Google, NumPy, or project standard). ...
- Require at least one approved code review by a qualified reviewer (someone other than the author) before merging changes into the main branch. ...
```
**The "after" graph is exactly the rule layer bigger.** The second visualization reads 39 nodes and 63 edges against the 31 and 49 before: the eight new nodes are the seven `Rule` nodes plus the `coding_agent_rules` `NodeSet` node, and the fourteen new edges are one `rule_associated_from` edge back to the source chunk and one `belongs_to_set` edge into the node set per rule. Open both HTML files to see that layer.
```text theme={null}
2026-09-10T11:19:53.969445 [info ] Retrieved 39 nodes and 63 edges in 0.00 seconds [cognee.shared.logging_utils]
...
Open file to see graph visualization after memify enhancment: ...
```
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — both the initial graph build and the rule extraction call the LLM
* Run it from a checkout of the cognee repo: the script writes its two visualizations into an `.artifacts/` folder next to the script file
* The run opens with `cognee.forget(everything=True)`, which wipes all data and system state — point it at a scratch instance rather than memory you want to keep
## How It Works
### Stage 1: Write Down the Team's Rule Chatter
```python theme={null}
coding_rules_chat_from_principal_engineer = """
We want code to be formatted by PEP8 standards.
Typing and Docstrings must be added.
Please also make sure to write NOTE: on all more complex code segments.
If there is any duplicate code, try to handle it in one function to avoid code duplication.
Susan should also always review new code changes before merging to main.
New releases should not happen on Friday so we don't have to fix them during the weekend.
"""
```
The demo's input is two plain strings standing in for messages a team actually sends. The second one, `coding_rules_chat_from_manager`, restates the last two rules above almost word for word — that overlap is deliberate, and collapsing it is the job the enrichment pass has to do.
### Stage 2: Remember the Chats and Snapshot the Graph
```python theme={null}
await cognee.remember(
[coding_rules_chat_from_principal_engineer, coding_rules_chat_from_manager],
self_improvement=False,
)
print("Text remembered successfully.\n")
# Visualize graph after remembering
file_path = os.path.join(
pathlib.Path(__file__).parent, ".artifacts", "graph_visualization_after_remember.html"
)
await visualize_graph(file_path)
print(f"Open file to see graph visualization after remember: {file_path}\n")
```
`remember()` builds the ordinary graph — documents, chunks, entities — with `self_improvement=False`, because this demo drives its own enrichment rather than the default one. The visualization written here is the "before" half of the comparison: chunks and entities, and not a single `Rule` node yet.
### Stage 3: Assemble the memify Task Pair
```python theme={null}
# extract_subgraph_chunks is a function that returns all document chunks from specified subgraphs (if no subgraph is specifed the whole graph will be sent through memify)
subgraph_extraction_tasks = [Task(extract_subgraph_chunks)]
# add_rule_associations is a function that handles processing coding rules from chunks and keeps track of
# existing rules so duplicate rules won't be created. As the result of this processing new Rule nodes will be created
# in the graph that specify coding rules found in conversations.
coding_rules_association_tasks = [
Task(
add_rule_associations,
rules_nodeset_name="coding_agent_rules",
task_config={"batch_size": 1},
),
]
```
Every memify pipeline is an extraction stage feeding an enrichment stage. Extraction here yields the text of every `DocumentChunk` in the graph; enrichment sends each chunk to the LLM together with the rules already filed under `coding_agent_rules`, and writes back only what is new. `batch_size: 1` is what makes that deduplication work chunk by chunk: one chunk per call, so the rules the first chat produced are already in the node set when the manager's chat is processed.
### Stage 4: Run memify Over the Existing Graph
```python theme={null}
# Memify accepts these tasks and orchestrates forwarding of graph data through these tasks (if data is not specified).
# If data is explicitely specified in the arguments this specified data will be forwarded through the tasks instead
await memify(
extraction_tasks=subgraph_extraction_tasks,
enrichment_tasks=coding_rules_association_tasks,
)
```
With no `data` argument, `memify()` loads the graph itself and pushes it through the pair. Each new `Rule` node is filed in the `coding_agent_rules` node set and linked back to the chunk it came from by a `rule_associated_from` edge, so a rule is always traceable to the conversation that stated it.
### Stage 5: Read the Rules Back
```python theme={null}
# Find the new specific coding rules added to graph through memify (created based on chat conversation between team members)
coding_rules = await cognee.recall(
query_text="List me the coding rules",
query_type=cognee.SearchType.CODING_RULES,
node_name=["coding_agent_rules"],
)
print("Coding rules created by memify:")
for result in coding_rules:
print("- " + result.text)
```
Pinning `query_type` to `SearchType.CODING_RULES` takes the choice away from the router and skips the completion step entirely: the retriever reads the `coding_agent_rules` node set directly and hands back the rule texts, which is what an agent wants before it edits a file. The script then writes a second visualization, `graph_visualization_after_memify.html`, so the rule layer can be compared against the "before" render from Stage 2.
## Run It
```bash theme={null}
uv run python examples/demos/custom_pipelines/memify_coding_agent_rule_extraction_example.py
```
<Columns>
<Card title="Memify" icon="sparkles" href="/core-concepts/main-operations/legacy-operations/memify">
The extraction and enrichment stages behind this pipeline, and the other built-in pairs.
</Card>
<Card title="Custom Tasks and Pipelines" icon="workflow" href="/guides/custom-tasks-pipelines">
Writing your own `Task` functions and wiring them into a pipeline.
</Card>
<Card title="NodeSets" icon="tags" href="/core-concepts/further-concepts/node-sets">
How node sets group the rules and scope the query that reads them back.
</Card>
<Card title="Search" icon="search" href="/core-concepts/main-operations/legacy-operations/search">
What `SearchType.CODING_RULES` returns and the other search types alongside it.
</Card>
</Columns>
# Company Brain for Docs, Code, and Conversations
Source: https://docs.cognee.ai/examples/company-brain
Teach one memory a written fact, a code repository, and a rule stated in conversation, then answer one question that needs all three at once
What a new engineer needs to know about your payments API lives in three different places: one sentence in a doc, the duplicate-charge guard in the code itself, and a release rule somebody stated in chat months ago. Answering "how do I ship this safely?" means holding all three at once.
## What You'll Build
Three kinds of knowledge go into a single cognee dataset: a plain-text fact about who owns the payments API and which database it uses, a tiny Python module extracted into a code graph, and a two-turn conversation in which a release rule is stated and then distilled into permanent memory. One question — who maintains the API, which database it uses, what release rule the team learned, and which function guards against duplicate charges — is then asked from a brand-new session. Each part is answerable from exactly one of the three sources, and the answer draws on all of them even though the session that learned the rule is gone.
The complete runnable script is
[`examples/demos/company_brain/company_brain_demo.py`](https://github.com/topoteretes/cognee/blob/dev/examples/demos/company_brain/company_brain_demo.py) —
this page walks through its key moments rather than reproducing it.
## Features in Play
* [Remember](/core-concepts/main-operations/remember) — writes the text fact and the code fixture into one dataset, with `content_type="code"` selecting the code pipeline for the second
* [Code Graph](/guides/code-graph) — extracts symbols from the sample module with Enola, with no LLM call; `index_vectors=True` also embeds them so the final answer can reach them
* [Sessions](/guides/sessions) — carries the scripted two-turn conversation in which the release rule is stated
* [Session Distillation](/guides/session-distillation) — promotes that conversation's lesson out of the session cache and into the permanent graph
* [Recall](/core-concepts/main-operations/recall) — answers the final question with `GRAPH_COMPLETION` from a fresh session id, proving the knowledge outlived the conversation
## What to Expect
A successful run prints five numbered steps; the excerpts below cover the three worth watching — the code graph (step 2), distillation (step 4), and the final answer (step 5). They come from a real run, trimmed: startup banners and per-task pipeline lines are cut, home directories are shortened, and session ids are truncated. The lesson wording, how many lessons the curator accepts, and the final answer all come from live model calls, so yours will read differently. Expect a couple of minutes end to end, most of it in the distillation step.
**Enola installs itself, then the code graph is built.** The first code run downloads the pinned binary into `~/.cognee/bin`, extracts the two functions, and writes them as graph nodes and edges. Extraction is deterministic — no LLM call — and `index_vectors=True` adds the symbol embeddings the final answer needs.
```text theme={null}
Downloading enola v0.4.12 (darwin-arm64) from https://github.com/enola-labs/enola/releases/download/v0.4.12/enola-0.4.12-darwin-arm64.tar.gz
Installed enola v0.4.12 at ~/.cognee/bin/enola-0.4.12-darwin-arm64
Parsed 3 fact(s) (0 from insights.json) from .cognee-readme-demo/payments-example/.enola/facts.jsonl
Mapped 3 enola fact(s) to 4 data point(s).
Code graph node delta: 3 added, 0 updated, 0 unchanged.
Code graph edge delta: 3 new, 0 already present.
```
**`query_facts` reads the symbols straight back out.** Both functions come back as `CodeSymbol` facts with their file and line, and `payments.replay_test` carries the `calls` edge to `payments.charge_once` that Enola derived from the source. This is a deterministic listing from the graph index, not a similarity ranking.
```text theme={null}
[
{
"operation": "query_facts",
"facts": [
{
"kind": "symbol",
"type": "CodeSymbol",
"name": "payments.charge_once",
"file": "payments.py",
"line": 4,
"repo": "payments-example",
"description": "symbol: cyclomatic=2, exported=True, language=python, return_type=bool, symbol_kind=function",
...
},
{
"kind": "symbol",
"type": "CodeSymbol",
"name": "payments.replay_test",
"file": "payments.py",
"line": 12,
"repo": "payments-example",
...
"relations": [
{
"type": "calls",
"target": "payments.charge_once"
},
...
]
}
],
"total": 2,
"offset": 0,
"limit": 10,
"has_more": false
}
]
```
**The session's rule becomes a permanent lesson document.** Distillation reports `completed` and publishes the lessons the curator accepted, each written as a markdown document and cognified into the dataset. The lesson is rewritten rather than quoted, and one conversation can yield more than one.
```text theme={null}
Distillation: completed; 1 lesson documents
# Session learning — 2026-09-14 (session readme-learning-821c92e6…)
For every Payments API release, run the replay test before deploying to prevent duplicate webhook delivery that can cause double charges. (Learned after a prior incident in which duplicate webhook delivery caused a double charge during a Payments API release.)
```
**A brand-new session answers from all three sources.** The question is asked under a fresh random `session_id`, so none of the earlier conversation is in scope. Each line traces to a different place the knowledge came from: the maintainer and database from the remembered document, the release rule from the distilled lesson, and the function from the code graph.
```text theme={null}
Answer in a fresh session:
- Maintainer: Alice
- Database: PostgreSQL
- Release rule: Run the replay test before deploying every Payments API release (to prevent duplicate webhook delivery)
- Function: payments.charge_once
```
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) and [Embedding Providers](/setup-configuration/embedding-providers) configured — text ingestion, the session turns, distillation, and the final answer are all live model calls
* Code extraction downloads the Enola binary automatically on first use; see [Code Graph](/guides/code-graph) for how that pipeline works and how to pin the binary yourself
* The code step needs an embedding provider but makes no LLM call: extraction itself is deterministic, and `index_vectors=True` embeds the extracted symbols so `GRAPH_COMPLETION` can retrieve them later
* Run it from a checkout of the cognee repo: the script loads your `.env` with `load_dotenv()`, writes its sample module beside itself, and defaults its data, system, cache, and log directories into `.cognee-readme-demo` next to the script
* Storage variables are set with `setdefault`, so a `DATA_ROOT_DIRECTORY` (or any sibling) already in your environment wins and the demo writes into your existing instance instead
* The script sets `CACHING=true` and `AUTO_FEEDBACK=true` itself — distillation depends on the per-turn analysis those enable, so they are not optional here
* Nothing is deleted: the demo only adds to memory, and re-running it adds again
## How It Works
### Stage 1: Pin Storage Beside the Script
```python theme={null}
root = Path(__file__).resolve().parent / ".cognee-readme-demo"
for variable, directory in (
("DATA_ROOT_DIRECTORY", "data"),
("SYSTEM_ROOT_DIRECTORY", "system"),
("CACHE_ROOT_DIRECTORY", "cache"),
("COGNEE_LOGS_DIR", "logs"),
):
os.environ.setdefault(variable, str(root / directory))
# Feedback analysis captures durable guidance during the scripted session.
os.environ["CACHING"] = "true"
os.environ["AUTO_FEEDBACK"] = "true"
```
This runs before `import cognee`, which is what makes it effective — cognee reads these on first import. The `setdefault` calls keep the demo self-contained without overriding a real instance you have already configured, while `CACHING` and `AUTO_FEEDBACK` are forced on because the session lesson in Stage 4 has nothing to distill without them.
### Stage 2: Remember the Written Fact
```python theme={null}
print("1. Remember the document.", flush=True)
await cognee.remember(DOCUMENT, dataset_name=DATASET, self_improvement=False)
```
One sentence — "Alice maintains the payments API. The payments API uses PostgreSQL." — becomes graph memory. `self_improvement=False` skips the enrichment pass, which is wasted work on a single short document and keeps the demo's first step fast.
### Stage 3: Build and Query the Code Graph
```python theme={null}
async def index_code(cognee, root):
repo = root / "payments-example"
repo.mkdir(parents=True, exist_ok=True)
(repo / "payments.py").write_text(CODE)
print("\nIndexing the sample code:", flush=True)
# index_vectors writes CodeSymbol embeddings too, so the final
# GRAPH_COMPLETION answer can reach the code alongside text and lessons.
await cognee.remember(str(repo), dataset_name=DATASET, content_type="code", index_vectors=True)
facts = await cognee.search(
query_type=cognee.SearchType.CODE,
query_text="",
datasets=[DATASET],
code_query={"operation": "query_facts", "kinds": ["symbol"], "limit": 10},
)
print(json.dumps(facts, indent=2, default=str))
```
The script writes a two-function module — a `charge_once` duplicate guard and the `replay_test` that exercises it — and hands the folder to `remember(content_type="code")`, which routes it through the deterministic Enola pipeline rather than LLM extraction. The `query_facts` call reads the symbols straight back out, which is how you confirm the graph was built. `index_vectors=True` is what makes Stage 6 possible: by default the code facts live only in the graph index, reachable through `SearchType.CODE` but invisible to every semantic retriever.
### Stage 4: State a Rule Inside a Session
```python theme={null}
print("3. Learn a release rule during a session.", flush=True)
session_id = f"readme-learning-{uuid4().hex}"
await cognee.recall(
"Who maintains the payments API?",
query_type=cognee.SearchType.RAG_COMPLETION,
datasets=[DATASET],
session_id=session_id,
)
await cognee.recall(
LESSON,
query_type=cognee.SearchType.RAG_COMPLETION,
datasets=[DATASET],
session_id=session_id,
)
```
Two `recall()` calls share one `session_id`, which is what makes them a conversation rather than two unrelated queries. The first is an ordinary question; the second states the lesson — always run the replay test before a payments API release, because duplicate webhook delivery once caused a double charge — as a turn in that conversation. With `AUTO_FEEDBACK` on, the turn analysis recognizes it as durable guidance.
### Stage 5: Distill the Session into Permanent Memory
```python theme={null}
print("4. Distill the session into permanent memory.", flush=True)
distilled = await cognee.session.distill_session(session_id, dataset=DATASET)
print(f"Distillation: {distilled.status}; {len(distilled.documents)} lesson documents")
for document in distilled.documents:
print(document)
if distilled.status != "completed" or not distilled.documents:
raise RuntimeError(
"No new lesson was published. Inspect the distillation status and provider logs. "
"An existing equivalent lesson can also cause the curator to reject a duplicate."
)
```
`distill_session` turns the conversation's guidance into lesson documents in the graph, so the rule survives the session that produced it. Distillation is model-dependent and a curator can reject a lesson it considers a duplicate of one already stored, so the script fails loudly rather than letting the final answer quietly miss the rule.
### Stage 6: Answer from a Fresh Session
```python theme={null}
async def recall_saved_memory(cognee):
print("\nAnswer in a fresh session:", flush=True)
print_answers(
await cognee.recall(
QUESTION,
query_type=cognee.SearchType.GRAPH_COMPLETION,
datasets=[DATASET],
session_id=f"readme-verification-{uuid4().hex}",
)
)
```
The one question that needs all three sources is asked under a brand-new random `session_id`, so none of the earlier conversation is in scope. Anything the answer gets right came from permanent memory — which is the whole point of the previous five stages. The retriever matters here: `GRAPH_COMPLETION` builds its search list from every indexed node type, so it reaches the embedded `CodeSymbol` names. `RAG_COMPLETION` reads document chunks only and would miss the code entirely.
## Run It
```bash theme={null}
uv run python examples/demos/company_brain/company_brain_demo.py
```
## Running One Part at a Time
Two mutually exclusive flags shorten the loop once memory exists:
```bash theme={null}
uv run python examples/demos/company_brain/company_brain_demo.py --recall-only
uv run python examples/demos/company_brain/company_brain_demo.py --code-only
```
`--recall-only` skips straight to Stage 6 and re-asks the question against whatever is already stored, which is the cheapest way to confirm the memory persisted across processes. `--code-only` runs just Stage 3, so you can check that Enola installed and extracted symbols without spending any LLM calls — it does embed them, so an embedding provider is still required.
<Columns>
<Card title="Session Distillation" icon="graduation-cap" href="/guides/session-distillation">
How a stated rule becomes a permanent lesson, and what the curator accepts.
</Card>
<Card title="Code Graph" icon="code" href="/guides/code-graph">
The Enola pipeline behind `content_type="code"` and the `code_query` operations.
</Card>
<Card title="Sessions" icon="message-square" href="/guides/sessions">
Session ids, the session cache, and how turns become conversational memory.
</Card>
<Card title="Remember" icon="brain" href="/core-concepts/main-operations/remember">
The write operation behind both ingestion steps, including `self_improvement`.
</Card>
</Columns>
# Resolve Conflicting Facts in Memory
Source: https://docs.cognee.ai/examples/contradiction-handling
Feed cognee two documents that disagree and watch it flag the conflict, resist a five-star rating, and change its answer only when a correction is remembered
Two people file reports about the same project and the numbers do not match — one says the budget is 2 million euros, the other says 5 million. Your memory layer has to hold both, tell you they conflict, and be honest about what it takes to settle the disagreement.
## What You'll Build
Two one-line documents about Project Falcon go into an isolated dataset, and they contradict each other on both the lead and the budget. With contradiction detection on, the second ingestion records a `contradicts` edge carrying both fact texts, the reason, and a confidence score — nothing is overwritten or deleted — and the session layer tracks which graph elements each answer used, so a rating can be folded into retrieval weights. What comes out is an answer that reports the conflict, keeps reporting it even after a 5/5 rating, and flips to the corrected budget only when an explicit correction is remembered.
The complete runnable script is
[`examples/demos/feedback/contradiction_feedback_demo.py`](https://github.com/topoteretes/cognee/blob/main/examples/demos/feedback/contradiction_feedback_demo.py) —
this page walks through its key moments rather than reproducing it.
## Features in Play
* [Remember](/core-concepts/main-operations/remember) — ingests each document into the demo dataset, and it is the `cognify()` stage inside it that the contradiction check hangs off
* [Contradiction Detection](/python-api/cognify#contradiction-detection) — the opt-in check that compares the newly touched facts against the ones already stored and writes the `contradicts` edge
* [Sessions](/guides/sessions) — records each answered question as a QA entry, including which graph nodes and edges the answer used, so feedback has something to attach to
* [Feedback System](/guides/feedback-system) — `add_feedback()` puts a 5/5 rating and a comment on that QA entry, and a feedback-weights pipeline pushes it into the graph
* [Feedback-Weighted Ranking](/guides/truth-subspace-reranking#relationship-to-the-feedback-loop) — `DEFAULT_FEEDBACK_INFLUENCE` is what makes those weights count during retrieval scoring
## What to Expect
Every box is read back from the graph or the session store rather than echoed from the inputs — you see the actual edges and weights, not a narration of them. The boxes below are from a real run, trimmed. Because ingestion, the contradiction judgement, and every answer are live LLM calls, the exact wording, the confidence scores, and even the number of contradictions flagged vary from run to run.
**STEP 1** — the first report goes in, and the box lists the facts extracted from it: Anna as the lead, the 2 million euro budget, and their types. With only one document stored there is nothing to disagree with yet.
```text theme={null}
+--------- STEP 1 REMEMBER: 'Anna leads Falcon. Budget is 2M EUR.' ---------+
| facts now in the knowledge graph: |
| |
| (anna) --leads--> (project falcon) |
| (anna) --is_a--> (person) |
| (project falcon) --has_budget--> (2 million euros) |
| (project falcon) --is_a--> (project) |
| (2 million euros) --is_a--> (monetaryamount) |
| |
| contradictions flagged: 0 |
+----------------------------------------------------------------------------+
```
**STEP 2** — the conflicting report lands. Both budget facts now sit in the graph side by side, and the detection pass flags the disagreements — in this run both the budget and the lead — each as a `FACT A`/`FACT B` pair with the model's reason and confidence. The flagged line says it explicitly: nothing was deleted.
```text theme={null}
+-------- STEP 2 REMEMBER: 'Marko leads Falcon. Budget is 5M EUR.' ---------+
| facts now in the knowledge graph: |
| |
| (anna) --leads--> (project falcon) |
| (anna) --is_a--> (person) |
| (project falcon) --has_budget--> (2 million euros) |
| (project falcon) --is_a--> (project) |
| (project falcon) --has_budget--> (5 million euros) |
| (2 million euros) --is_a--> (monetaryamount) |
| (marko) --leads--> (project falcon) |
| (marko) --is_a--> (person) |
| (5 million euros) --is_a--> (monetaryamount) |
| |
| contradictions flagged: 2 (nothing was deleted) |
...
| FACT A : project falcon has budget 2 million euros |
| FACT B : project falcon has budget 5 million euros |
| reason : Both facts state different budgets for the same subject (proje |
| ct falcon); the project cannot simultaneously have two different monetary |
| amounts as its budget. |
| confidence: 1.0 |
+----------------------------------------------------------------------------+
```
**STEP 3** — the question is asked for the first time. Retrieval sees both budget facts, so the answer reports the conflict instead of picking a side. The box also shows the session bookkeeping the next step depends on: which graph elements the answer used, all still at the neutral weight `0.5`, and the `qa_id` the exchange was recorded under.
```text theme={null}
+----------- STEP 3 ASK: 'What is the budget of Project Falcon?' -----------+
| answer: |
| The sources conflict: Project Falcon's budget is listed as either 2 mil |
| lion euros or 5 million euros. |
| |
| graph elements used by this answer: 27 |
| node b7c10547-0775: weight 0.5 |
...
| recorded in session 'board_demo_session' as qa_id 389844ab... |
+----------------------------------------------------------------------------+
```
**STEP 4** — the 5/5 rating is attached to that `qa_id` and the weights pipeline pushes it into the graph: every element the rated answer used moves from `0.5` to `0.55`, uniformly. That uniformity is the setup for the next box.
```text theme={null}
+--------------- STEP 4 FEEDBACK: rated 5/5 -> weights shift ---------------+
| feedback weight per element (before -> after): |
| |
| node b7c10547-0775: 0.5 -> 0.55 |
| node 1ee4a665-25e7: 0.5 -> 0.55 |
...
| high-rated elements now rank higher in future searches |
| (DEFAULT_FEEDBACK_INFLUENCE=0.2 blends weight into retrieval scoring); |
| both original facts and the contradicts edge remain stored. |
+----------------------------------------------------------------------------+
```
**STEP 5** — the same question in a fresh session, so only the graph and its new weights are in play. Both budget facts were raised by the same amount, so their relative ranking is unchanged and the answer still reports the conflict: a rating steers attention, it does not decide what is true.
```text theme={null}
+---------- STEP 5 ASK AGAIN: a rating alone cannot pick a winner ----------+
| answer (fresh session, graph + weights only): |
| The sources conflict: one chunk states Project Falcon's budget is 2 mil |
| lion euros, another states 5 million euros. |
...
+----------------------------------------------------------------------------+
```
**STEP 6** — the correction is remembered as a document, and retrieval finally has a statement that explicitly supersedes the 2 million figure. The answer in a third fresh session flips to 5 million euros, while the old fact stays stored and auditable through its `contradicts` edge.
```text theme={null}
+----------- STEP 6 REMEMBER THE CORRECTION -> the answer flips ------------+
| new document: 'The approved budget is 5M EUR; the 2M figure is |
| outdated.' (this is what textual feedback becomes when persisted) |
| |
| answer (fresh session): |
| The approved budget for Project Falcon is 5 million euros. (The earlier |
| 2 million euro figure is outdated.) |
| |
| contradictions now flagged in the graph: 2 |
| the 2M fact is still stored (auditable), but retrieval now has a |
| correction that explicitly supersedes it -- new knowledge, not the |
| rating, is what changed the answer. |
+----------------------------------------------------------------------------+
```
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — ingestion, the contradiction judgement, and every answer are live calls, so the wording and the confidence score vary by model
* Run it from a checkout of the cognee repo with dependencies installed and a configured `.env`
* Expect the script to set its own environment before importing cognee: it points cognee's data and system roots at `/tmp/conflict_demo` (deleted at startup), turns on `CONTRADICTION_DETECTION`, keeps `CACHING` on, and raises `DEFAULT_FEEDBACK_INFLUENCE` to `0.2` from its default of `0.0` — see [Contradiction detection](/python-api/cognify#contradiction-detection) for the contradiction tuning knobs
* The run starts with `prune_data()` and `prune_system(metadata=True)`, which is safe here only because those roots are the isolated demo directory rather than your real storage
## How It Works
### Stage 1: Isolate Storage and Enable Detection
```python theme={null}
os.environ.update(
{
"DATA_ROOT_DIRECTORY": str(DEMO_ROOT / "data"),
"SYSTEM_ROOT_DIRECTORY": str(DEMO_ROOT / "system"),
"CONTRADICTION_DETECTION": "true",
"CACHING": "true",
"DEFAULT_FEEDBACK_INFLUENCE": "0.2",
}
)
```
These five settings run before `import cognee`, which is what makes them take effect. Two of them are the demo's subject: `CONTRADICTION_DETECTION` appends the conflict check to the end of the ingestion pipeline (it is off by default), and `DEFAULT_FEEDBACK_INFLUENCE` lifts feedback weights from ignored to a fifth of the retrieval score, so a rating can actually move ranking. The storage roots keep the whole run inside `/tmp/conflict_demo`.
### Stage 2: Remember the Second, Conflicting Document
```python theme={null}
await cognee.remember(
"Marko leads Project Falcon. The budget of Project Falcon is 5 million euros.",
dataset_name=DATASET,
self_improvement=False,
)
facts, conflicts = await read_facts()
```
The first `remember()` call stores Anna and the 2 million euro budget; this second one disagrees on both counts. Because entity ids are derived from entity names, "Project Falcon" lands on the same node, which puts the new budget fact one hop from the stored one — the neighbourhood the contradiction check compares. `self_improvement=False` skips the enrichment pass so nothing but ingestion is in play.
### Stage 3: Read the Conflict Back From the Graph
```python theme={null}
async def read_facts():
"""Read every semantic fact and every contradicts edge back from the graph."""
graph = await get_graph_engine()
nodes, edges = await graph.get_graph_data()
names = {str(node_id): props.get("name", str(node_id)[:8]) for node_id, props in nodes}
facts, conflicts = [], []
for source, target, relationship, props in edges:
if relationship == "contradicts":
conflicts.append(props)
elif relationship not in STRUCTURAL:
facts.append(
f"({names.get(str(source))}) --{relationship}--> ({names.get(str(target))})"
)
return facts, conflicts
```
Every step re-reads the graph through this helper rather than trusting what the previous call returned. It splits the edges into two piles: `contradicts` edges, whose properties carry `first_fact`, `second_fact`, `reason`, and `confidence` for the printout, and ordinary semantic facts. Structural edges such as `contains` and `is_part_of` are filtered out so the printed list is only human-meaningful statements — and both budget facts stay in that list, because flagging a conflict never removes either side.
### Stage 4: Ask, and Capture What the Answer Used
```python theme={null}
results = await cognee.recall(
QUESTION,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET],
session_id=SESSION,
)
answer = first_answer(results)
user = await get_default_user()
qa_entries = await get_session_manager().get_session(user_id=str(user.id), session_id=SESSION)
assert isinstance(qa_entries, list) and qa_entries, "session recorded no QA entry"
qa = qa_entries[-1]
weights_before = await element_weights(qa.used_graph_element_ids)
```
Asking for the budget with a `session_id` writes a QA entry into the session store, and that entry's `used_graph_element_ids` is the link between an answer and the graph elements behind it. The demo reads the current feedback weight of each of those elements now, before any rating exists, so the next step has a baseline to compare against. Retrieval sees both budget facts, so the answer reports the conflict rather than choosing.
### Stage 5: Rate the Answer and Push the Rating Into the Graph
```python theme={null}
await cognee.session.add_feedback(
session_id=SESSION,
qa_id=qa.qa_id,
feedback_score=5,
feedback_text="Correct — 5 million is the approved budget; Marko took over in June.",
)
await apply_feedback_weights_pipeline(
user=user, session_ids=[SESSION], dataset=DATASET, alpha=0.1
)
weights_after = await element_weights(qa.used_graph_element_ids)
```
`add_feedback()` attaches the score and comment to that one QA entry — at this point the feedback lives in the session and nothing in the graph has moved. The weights pipeline is the second half of the loop: it walks the elements the rated answer used and updates their `feedback_weight`, with `alpha` controlling how hard one rating pulls. Printing the before and after side by side shows the shift. In an ordinary application this is what [`improve()`](/core-concepts/main-operations/improve#with-session-ids) does for you.
### Stage 6: Ask Again — a Rating Cannot Break a Tie
```python theme={null}
# The 5/5 rating up-weighted every element the answer used, INCLUDING both
# budget facts (the answer needed both to report the conflict). A symmetric
# signal cannot break the tie, so the answer still reports the conflict.
results = await cognee.recall(
QUESTION,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET],
session_id="fresh_session_1",
)
```
The same question runs in a fresh session, so nothing but the graph and its new weights can influence the answer — no conversation history carries the rating's text forward. The rating raised both budget facts equally, because the answer it praised had used both, and equal raises leave their relative order unchanged. The answer still reports the conflict, which is the point of the step: feedback steers which memories get attention, not which ones are true.
### Stage 7: Remember the Correction and Watch the Answer Flip
```python theme={null}
await cognee.remember(
"The approved budget of Project Falcon is 5 million euros. "
"The earlier 2 million euro figure is outdated.",
dataset_name=DATASET,
self_improvement=False,
)
_, conflicts = await read_facts()
results = await cognee.recall(
QUESTION,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET],
session_id="fresh_session_2",
)
```
The same claim the feedback comment made in prose is now remembered as a document — this is what textual feedback becomes when you persist it. Retrieval finally has a statement that explicitly supersedes the 2 million figure, and the answer in a third fresh session flips to 5 million. The 2 million fact is still stored and still auditable through its `contradicts` edge; what changed the answer was new knowledge, not the rating.
## Run It
```bash theme={null}
uv run python examples/demos/feedback/contradiction_feedback_demo.py
```
<Columns>
<Card title="Cognify" icon="brain" href="/python-api/cognify#contradiction-detection">
What the contradiction check compares, what it skips, and how to tune it.
</Card>
<Card title="Feedback System" icon="brain-circuit" href="/guides/feedback-system">
Rating a session answer and pushing that rating into graph weights.
</Card>
<Card title="Truth Subspace Reranking" icon="compass" href="/guides/truth-subspace-reranking">
How feedback weights and truth weighting reach retrieval scoring.
</Card>
<Card title="Improve" icon="sparkles" href="/core-concepts/main-operations/improve">
The operation that applies feedback weights for you outside a demo.
</Card>
</Columns>
# Data Silos
Source: https://docs.cognee.ai/examples/data-silos
Unify siloed enterprise data in a single Cognee memory layer.
## Enterprise Data Unification
Every enterprise has the same problem: valuable data locked in silos. Your CRM doesn't talk to your ERP. Your knowledge base doesn't connect to your support tickets. Your strategic documents live in SharePoint while operational data lives in Snowflake.
Cognee creates a unified memory layer that connects these silos without replacing them.
## The Siloed Data Problem
When someone asks "What's the full context on the Acme Corp relationship?", the answer requires piecing together:
* CRM opportunity and contact data
* Support ticket history and resolution patterns
* Contract terms and renewal dates
* Invoice and payment history
* Relevant Slack conversations and email threads
No single system has the complete picture. Neither does traditional RAG.
## Why Standard RAG Fails Here
Vector search treats each chunk independently. It might find a support ticket mentioning Acme Corp and a contract with their name, but it doesn't understand that:
* The support ticket was about a feature that the contract specifically excludes
* The escalation pattern matches a trend you're seeing with other enterprise customers
* The contract renewal is approaching and the recent ticket volume is a risk signal
Relationships matter as much as content. Cognee captures both.
## Implementation: Remember, Improve, Recall
The Cognee Memory Layer sits on top of your existing data infrastructure:
### Step 1: Remember Your Sources
Cognee supports 30+ data sources out of the box:
```python theme={null}
import cognee
from cognee import SearchType
# Connect structured data
await migrate_relational_database(graph, schema=schema)
# Connect unstructured documents
await cognee.remember("s3://bucket/product-docs/", dataset_name="enterprise_memory")
await cognee.remember("https://www.your-website.com", dataset_name="enterprise_memory")
# Connect semi-structured data
await cognee.remember("path-to-your-folders", dataset_name="enterprise_memory")
...
```
### Step 2: Improve the Shared Memory
Optionally run an explicit enrichment pass when you want to deepen the shared graph after ingestion:
```python theme={null}
# Enrich the unified memory layer
await cognee.improve(dataset="enterprise_memory")
```
Cognee automatically:
* Enriches the existing knowledge graph instead of re-ingesting the source data
* Builds additional retrieval structures on top of what `remember()` already stored
* Improves later `recall()` quality for the unified dataset
* Can add session-bridging and feedback-weight updates when you provide `session_ids`
### Step 3: Recall with Context
Now queries return connected knowledge, not isolated chunks:
```python theme={null}
results = await cognee.recall(
query_text="Full context on Acme Corp",
query_type=SearchType.GRAPH_COMPLETION,
datasets=["enterprise_memory"],
)
```
Ready to unify your data silos? [Start with the open-source SDK](https://github.com/topoteretes/cognee) or [talk to our team](https://calendly.com/vasilije-topoteretes/) about enterprise deployment.
# Developer Knowledge Base
Source: https://docs.cognee.ai/examples/developer-knowledge-base
Turn an engineer's profile, past copilot conversations, and a coding-principles document into one memory you can question across all three
Everything you know about how an engineer works is scattered: a short profile, months of assistant conversations full of code, and a team document of coding principles nobody cross-references. Answering "does the code I wrote actually follow the principles we agreed on?" means holding all three in your head at once.
## What You'll Build
Three heterogeneous sources — a plain-text developer profile, a JSON export of human/assistant coding conversations, and a Markdown guide to the Zen of Python — go into memory under two node sets, with an OWL ontology grounding the entities that get extracted. A `memify()` pass then consolidates the graph, and you get two interactive HTML snapshots (before and after) plus answers to two questions: one that has to reach across the conversations and the principles document at once, and one deliberately scoped to the principles alone.
The complete runnable script is
[`examples/demos/comprehensive_example/cognee_comprehensive_example.py`](https://github.com/topoteretes/cognee/blob/main/examples/demos/comprehensive_example/cognee_comprehensive_example.py) —
this page walks through its key moments rather than reproducing it.
## Features in Play
* [NodeSet Grouping](/guides/nodeset-grouping) — labels the profile and conversations as `developer_data` and the Zen guide as `principles_data`, so the second question can be answered from the principles alone
* [Ontology Quickstart](/guides/ontology-support) — the bundled OWL file grounds extraction in a shared vocabulary instead of letting each source invent its own entity names
* [Memify](/core-concepts/main-operations/legacy-operations/memify) — the consolidation pass that runs between the two snapshots and enriches the connections across sources
* [Graph Visualization](/guides/graph-visualization) — renders the graph twice, so the effect of consolidation is something you can look at rather than infer
* [Recall](/core-concepts/main-operations/recall) — answers both questions with `GRAPH_COMPLETION`, once across the whole graph and once filtered to one node set
## What to Expect
The output below is from a real run, trimmed. Both answers come from live LLM calls, so their wording varies from run to run, and the script prints each one as a one-line `ResponseGraphEntry(...)` object — the excerpts restore the line breaks so you can read them. Alongside the console output, the run leaves two HTML files in an `.artifacts` folder next to the script — `graph_visualization_nodesets_and_ontology.html` and `graph_visualization_after_memify.html` — which you open in a browser to compare the graph before and after `memify()`.
**The unscoped question is answered from both sources.** `AsyncWebScraper` lives only in the conversation export and the design principles only in the Zen guide, so the mapping the answer draws — scraper techniques to named principles — is the cross-source assembly that Stage 5 of [How It Works](#how-it-works) sets up.
```text theme={null}
Python Pattern Analysis: I don't have your scraper code here, but given your background (heavy asyncio/aiohttp pipelines, Pydantic, pytest-asyncio, low‑latency APIs), an AsyncWebScraper that follows those technologies will align well with core Python design principles. Brief mapping:
- Concurrency via asyncio + aiohttp → explicit, idiomatic async I/O (”Explicit is better than implicit”, “There should be one—obvious—way to do it” for async network code).
- Async context managers (async with ClientSession) and small coroutine functions → clear resource management and simpler, flatter control flow (”Readability counts”, “Flat is better than nested”).
- Pydantic or type hints for parsed responses → explicit contracts and validation (”Explicit is better than implicit”, “Readability counts”).
- pytest-asyncio tests → surface errors early and avoid silent failures (”Errors should never pass silently”).
- Modular design, single-responsibility coroutines, and backoff/retry/logging → practical, robust, and maintainable (”Simple is better than complex”, “Practicality beats purity”).
...
```
**The scoped question stays inside one node set.** Restricted to `principles_data` via `node_name`, the naming answer is drawn from the principles document alone — general conventions, with nothing from the engineer's own past code leaking in.
```text theme={null}
Filtered search result: Short answer: follow PEP 8 + the Zen of Python: use clear, consistent, descriptive names that make code readable and explicit.
Practical rules (brief):
- Use snake_case for variables and functions: my_value, parse_response.
- Use CapWords (PascalCase) for classes: AsyncWebScraper, UserModel.
- Use UPPER_SNAKE_CASE for module-level constants: DEFAULT_TIMEOUT = 10.
- Use a single leading underscore for “private” names: _cache, _connect().
...
```
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) and [Embedding Providers](/setup-configuration/embedding-providers) configured
* The script sets `os.environ["LLM_API_KEY"] = "your_api_key"` at the top as a placeholder — replace it with your own key, or delete the line if your `.env` already carries one
* Run it from a cognee repo checkout rather than a copy-paste: it reads three bundled files from the sibling `data/` folder — `copilot_conversations.json`, `zen_principles.md`, and `basic_ontology.owl`
* The run starts with `cognee.forget(everything=True)`, which wipes all existing Cognee data — use a setup you can afford to reset
## How It Works
### Stage 1: Locate the Bundled Sources
```python theme={null}
data_dir = Path(__file__).resolve().parent / "data"
asset_paths = {
"human_agent_conversations": str(data_dir / "copilot_conversations.json"),
"python_zen_principles": str(data_dir / "zen_principles.md"),
"ontology": str(data_dir / "basic_ontology.owl"),
}
```
The three inputs are resolved relative to the script file, so the demo works from any working directory as long as it runs inside the repo checkout. Two of them are data to ingest; the third is the ontology that shapes how that data is read.
### Stage 2: Ground Extraction in the OWL Ontology
```python theme={null}
os.environ["ONTOLOGY_FILE_PATH"] = ontology_path
```
The ontology is configured through the environment rather than passed as an argument, and this assignment happens **before** `import cognee` — Cognee reads env-backed settings at import time, so setting it afterwards would not take effect. Every source ingested below is extracted against this shared vocabulary.
### Stage 3: Ingest Three Sources into Two Node Sets
```python theme={null}
await cognee.forget(everything=True)
await cognee.remember(developer_intro, node_set=["developer_data"], self_improvement=False)
await cognee.remember(
human_agent_conversations,
node_set=["developer_data"],
self_improvement=False,
)
await cognee.remember(
python_zen_principles,
node_set=["principles_data"],
self_improvement=False,
)
```
A clean slate, then three `remember()` calls that differ in what they take — an inline string, a JSON path, and a Markdown path — but land in the same graph. The node sets do the sorting: the engineer's profile and their conversations become `developer_data`, the Zen guide becomes `principles_data`. `self_improvement=False` skips the automatic enrichment pass so that the `memify()` call in the next stage is the only consolidation step, and its effect is visible in isolation.
### Stage 4: Snapshot the Graph Before and After Consolidation
```python theme={null}
# generate the initial graph visualization showing nodesets and ontology structure
initial_graph_visualization_path = os.path.join(
os.path.dirname(__file__), artifacts_path, "graph_visualization_nodesets_and_ontology.html"
)
await cognee.visualize_graph(initial_graph_visualization_path)
# enhance the knowledge graph with memory consolidation for improved connections
await cognee.memify()
# generate the second graph visualization after memory enhancement
enhanced_graph_visualization_path = os.path.join(
os.path.dirname(__file__), artifacts_path, "graph_visualization_after_memify.html"
)
await cognee.visualize_graph(enhanced_graph_visualization_path)
```
The same render runs on either side of `memify()`. The first file shows what ingestion alone produced — the two node sets and the ontology-grounded entities under them; the second shows the graph after consolidation has enriched the connections between them. Opening both is the point of the stage.
### Stage 5: Ask a Question That Spans Sources
```python theme={null}
results = await cognee.recall(
query_text="How does my AsyncWebScraper implementation align with Python's design principles?",
query_type=cognee.SearchType.GRAPH_COMPLETION,
)
```
`AsyncWebScraper` appears only in the conversation export; "Python's design principles" only in the Zen guide. Neither source answers this question alone, and with no node-set filter the recall traverses the whole graph — so the answer has to be assembled from both.
### Stage 6: Scope Recall to One Node Set
```python theme={null}
results = await cognee.recall(
query_text="How should variables be named?",
query_type=cognee.SearchType.GRAPH_COMPLETION,
node_name=["principles_data"],
)
```
The same search type, now restricted to `principles_data` via `node_name`. This is what the tagging in Stage 3 bought: a question about conventions answered from the principles document, without the engineer's own past code influencing the answer.
## Run It
```bash theme={null}
uv run python examples/demos/comprehensive_example/cognee_comprehensive_example.py
```
<Columns>
<Card title="NodeSet Grouping" icon="layers" href="/guides/nodeset-grouping">
Learn the tagging this demo uses to keep two topics apart in one graph.
</Card>
<Card title="Ontology Quickstart" icon="git-branch" href="/guides/ontology-support">
Step through grounding extraction in your own OWL vocabulary.
</Card>
<Card title="Graph Visualization" icon="network" href="/guides/graph-visualization">
Render and bound the graph snapshots this demo compares.
</Card>
<Card title="Recall" icon="search" href="/core-concepts/main-operations/recall">
See the full set of recall parameters behind both questions.
</Card>
</Columns>
# Edge AI
Source: https://docs.cognee.ai/examples/edge-ai
Run Cognee memory pipelines on edge devices with cognee-RS.
## Edge AI & On-Device Memory
Cognee is bringing AI memory to the edge with **cognee-RS**, our Rust-based SDK designed for resource-constrained devices. Run the full memory pipeline (ingestion, semantic organization, retrieval) directly on-device, sub-100ms recall and data stay local.
## The Edge AI Opportunity
Picture this: Your smart glasses capture a conversation during a run, instantly recall your to-do list, and feed you directions - all offline, with zero data uploaded. Or your smart-home hub analyzes your evening routine, suggests energy optimizations for better sleep, and monitors wellness patterns without sending a single byte to the cloud.
This is the future and the promise of edge AI memory.
## cognee-RS: Rust-Powered Memory for Devices
cognee-RS is our experimental Rust SDK. It is a port of cognee's proven memory architecture to edge devices like phones, smartwatches, glasses, and smart-home hubs.
It combines:
* A lean retrieval engine optimized for constrained resources
* Support for on-device LLMs
* Seamless hybrid switching to cloud when needed
* Full multimodal support (text, images, audio)
### Core Capabilities
**Fully Offline Operation**
Run with Phi-4-class LLMs and local embeddings—no internet required for queries or retrieval. Toggle to hosted models with a single config flag when you have connectivity and need more power.
**High Accuracy**
We're targeting 90%+ answer accuracy, matching our Python SDK. The local semantic layer ensures retrieval fidelity even with smaller models. Graph-aware retrieval boosts accuracy 15-25% through structural cues.
**Hybrid Execution**
Route tasks intelligently: local for embeddings, cloud for heavy entity extraction, or split dynamically based on connectivity, battery, and latency requirements.
**Multimodal Fusion**
Handles text, images, audio, and sensor data. Real-time fusion from device inputs (mic + camera) creates holistic context that a cloud-only approach can't match.
**Resource Orchestration**
Dynamic scheduling caps memory and CPU usage. Heavy processing doesn't interrupt core device functions—retrieval stays prioritized while batch ingestion happens during idle time.
## Use Cases: Where Edge Memory Excels
### Personal Voice Assistants
Smart earbuds and wearables that remember your conversations, preferences, and context—without uploading your private discussions to the cloud.
> "What did Sarah say about the project deadline during our walk yesterday?"
Local conversation memory enables instant recall. Sync only opt-in summaries, never raw audio.
### Smart Home & Wellness
Baby monitors, vital-sign wearables, and home hubs that analyze patterns locally—complying with GDPR and HIPAA by design.
* Sleep pattern analysis without cloud dependency
* Anomaly detection that works during internet outages
* Behavioral insights that stay on your network
Your health data stays yours.
### Robotics & Autonomous Systems
Drones, robots, and autonomous vehicles need real-time memory access for navigation and decision-making—especially in dead zones.
```
Robot enters new environment
│
▼
cognee-RS builds local context map
│
▼
Real-time retrieval: "Have I seen this obstacle type before?"
│
▼
Decision without connectivity delay
```
No connectivity? No problem. Local context drives decisions.
### Industrial IoT
Factory-floor sensors, offline kiosks, and field equipment often operate in network-constrained environments.
Edge AI enables:
* 24/7 local reasoning without persistent connection
* Anomaly detection at the source
* Bandwidth savings—only critical events sync to cloud
* Continued operation during network outages
## Trade-Offs and Mitigations
Edge isn't effortless. Smaller models have tighter context windows. Devices have limited compute and battery budgets. Complex reasoning may exceed local capabilities.
cognee-RS addresses these constraints:
| Challenge | Mitigation |
| ---------------------- | ---------------------------------------- |
| Limited context window | Graph-aware retrieval for precision |
| Complex reasoning | Hybrid execution—offload when needed |
| Battery constraints | Dynamic scheduling, idle-time processing |
| Storage limits | Semantic compression, smart eviction |
| Model size | Support for Phi-4 class, upgradeable |
cognee-RS is currently experimental. Early conversations with partners are giving promising results.
## The Vision: Memory Everywhere
The future isn't cloud-only AI. It's AI that runs where you are: on your phone, your glasses, your watch, your car. AI that remembers your context without uploading your life to someone else's servers.
cognee-RS is how we get there: the same semantic memory layer that powers enterprise deployments, compiled to run on the devices in your pocket.
Privacy-first. Real-time. Offline-capable. Memory-enabled.
***
# Watch Feedback Reshape the Graph
Source: https://docs.cognee.ai/examples/feedback-loop-app
Run a local chat app where every answer can be rated and a memify pass turns those ratings into feedback weights you can see move on the graph
Your users rate the answers your assistant gives, and you want to know what that rating actually did to the memory behind it — not in aggregate, weeks later, but on the next screen refresh.
## What You'll Build
A small FastAPI service plus a browser UI, running side by side on your machine. Three short candidate profiles are remembered into an isolated dataset, and the knowledge graph built from them is drawn in the left panel with node size and edge thickness scaled by each element's `feedback_weight`. In the right panel you ask questions in one session, rate the answers 1–5, and then run the feedback memify pipeline — which reads the session's rated Q\&A entries, updates the `feedback_weight` of the graph nodes and edges those answers used, and hands back a before/after snapshot with a per-element delta. The graph redraws from the "after" side, so a rating you gave a moment ago is visible as a node that grew or shrank.
The complete runnable demo is
[`examples/demos/sessions/session_feedback_lifecycle_demo/`](https://github.com/topoteretes/cognee/tree/main/examples/demos/sessions/session_feedback_lifecycle_demo) —
this page walks through the key moments of its backend rather than reproducing it.
## Features in Play
* [Remember](/core-concepts/main-operations/remember) — loads the bundled candidate profiles into the demo dataset in one call
* [Recall](/core-concepts/main-operations/recall) — answers every question with `GRAPH_COMPLETION` against that dataset, inside the demo session
* [Sessions](/guides/sessions) — the shared `session_id` that makes every question, answer, and rating part of one conversation
* [Feedback System](/guides/feedback-system) — `cognee.session.add_feedback` attaches a 1–5 score and a comment to a specific `qa_id`
* [Session-Context Guidance](/core-concepts/sessions-and-caching#session-context-guidance-auto-feedback) — with `AUTO_FEEDBACK` on, a rating typed as an ordinary chat message can land on the previous answer without an explicit API call
* [Improve](/core-concepts/main-operations/improve#with-session-ids) — the feedback-weight pass, which this demo triggers directly as a memify pipeline so it can snapshot the graph on either side of it
## How It Looks
<Note>
The interface in these screenshots is not the [Cognee UI](/cognee-cloud/local-ui) — it is a small frontend bundled with this demo (`frontend/` in the demo folder), built only to make the feedback loop visible. Beyond the logo, it shares nothing with the Cognee UI.
</Note>
The frontend is plain HTML, CSS, and JavaScript with D3 for the graph; there is no build step, and the backend serves it directly. On first load the app initializes itself behind a loading overlay — isolated storage, ingestion, the first graph build — and then opens a walkthrough panel with the intended order: inspect the ingested documents, ask a question, praise the answer, ask another, criticize it, run the scripted replay, and look at what changed. Clicking a step highlights the part of the UI it refers to, and the `How This Demo Works` button reopens the panel any time.
<Frame>
<img alt="The demo on first launch: a knowledge graph of uniformly sized nodes on the left, the session chat panel on the right, and the seven-step How This Demo Works walkthrough open in the middle." />
</Frame>
The left panel is the graph. The header counts nodes and edges and `Reset View` re-centers the layout; the legend at the bottom states the scaling formulas — node size is `10 + weight * 18`, edge width `1.6 + weight * 4.4`, and color follows the same signal — which is why an untouched graph draws uniform and a memify run is a visible change rather than a number: praised regions grow and turn green, criticized ones shrink toward red. Clicking any node or edge fills the `Selected Element` panel below with its properties, `feedback_weight` included.
The right panel is the session. Across the top sit the four controls: `Run_demo` (the scripted replay), `Run_memify_pipeline` (fold the session's ratings into the graph and redraw), `View Ingested Docs`, and `How This Demo Works`. Below them, the current `session_id` badge, the chat itself, the `Search depth (top_k)` slider that sets `top_k` for the next question, and a single input that takes both questions and feedback — type "Wrong answer, 1/5" as an ordinary message and `AUTO_FEEDBACK` attaches it to the previous answer. `Session Content` at the bottom lists the session's Q\&A entries with the scores they have accumulated.
<Frame>
<img alt="The Ingested Documents modal listing the three candidate profiles for Emily Carter, Michael Rodriguez, and Sarah Nguyen." />
</Frame>
A few rated turns later, the same graph is legible at a glance:
<Frame>
<img alt="The demo after rated turns: the knowledge graph drawn with visibly uneven node sizes and colors, the Selected Element panel showing the machine learning node at feedback weight 0.809, and the chat holding the rated questions." />
</Frame>
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — ingestion and every answer are live calls, so wording varies by model
* Run it from a checkout of the cognee repo: the backend serves `frontend/` as static assets and reads `data/demo_documents.json` and `data/scripted_flow.json` from the demo folder, so a copy-pasted single file will not work
* Leave the session settings the app expects in place: it pins `CACHING=true`, `AUTO_FEEDBACK=true`, and `CACHE_BACKEND=fs` with `os.environ.setdefault`, so a conflicting value already exported in your shell or `.env` wins and the app refuses to initialize with a `412` naming the mismatch. See [Sessions and Caching](/core-concepts/sessions-and-caching#cache-adapters) for what those control
* Expect it to write its storage inside the demo folder — `.data_storage/` and `.cognee_system/` — and to start by calling `cognee.forget(everything=True)` against those roots, so it clears its own demo storage rather than memory you want to keep
* Open the UI in a browser with network access: the page loads D3 from a CDN to draw the graph
## Run It
```bash theme={null}
uv run python examples/demos/sessions/session_feedback_lifecycle_demo/backend/app.py
```
The entry point is the backend, and it serves the frontend itself: the script starts uvicorn on `http://127.0.0.1:8765` and then waits, so run it in its own terminal and stop it with Ctrl-C. Success is a browser, not a console — open that address and the page calls `POST /demo/init`, which streams back its activity log as it configures the isolated storage, forgets it, ingests the bundled documents, and reports the node count of the graph it built. From there the terminal only shows request logs; the demo's own narration is the activity log in the UI, which records each question with its `top_k`, each answer, any auto-detected feedback with the score and text it found, each manual rating with its `qa_id`, and — after a memify run — how many nodes and edges changed weight.
## How It Works
### Stage 1: Pin the Session Feedback Settings
Source: `examples/demos/sessions/session_feedback_lifecycle_demo/backend/app.py`
```python theme={null}
os.environ.setdefault("CACHING", "true")
os.environ.setdefault("CACHE_BACKEND", "fs")
os.environ.setdefault("AUTO_FEEDBACK", "true")
os.environ.setdefault("ENV", "dev")
```
These four lines run before `import cognee`, which is what makes them take effect. `CACHING` and `AUTO_FEEDBACK` are already the defaults, so this pins them rather than enabling them; `CACHE_BACKEND=fs` is the real departure, putting the session cache in files. The same three names are re-read from the environment when the app initializes — the UI checks `GET /demo/config_gate` at page load, and `POST /demo/init` refuses with a `412` naming the mismatch — so if one of them was set to something else before the process started, the demo tells you which one instead of half-working.
### Stage 2: Reset Storage and Ingest the Candidate Profiles
Source: `examples/demos/sessions/session_feedback_lifecycle_demo/backend/app.py`
```python theme={null}
record_step("Reset", "Configuring isolated demo directories")
DEMO_DATA_DIR.mkdir(parents=True, exist_ok=True)
DEMO_SYSTEM_DIR.mkdir(parents=True, exist_ok=True)
cognee.config.data_root_directory(str(DEMO_DATA_DIR))
cognee.config.system_root_directory(str(DEMO_SYSTEM_DIR))
record_step("Reset", "Forgetting demo-local data and metadata")
await cognee.forget(everything=True)
record_step("Ingest", "Loading deterministic demo documents")
documents = _get_demo_documents()
await cognee.remember(
documents,
dataset_name=DATASET_NAME,
self_improvement=False,
)
```
`POST /demo/init` is what the UI calls on load. It points cognee's data and system roots at folders inside the demo, wipes them, and ingests the bundled profiles of Emily Carter, Michael Rodriguez, and Sarah Nguyen. `self_improvement=False` keeps the enrichment pass out of the way, so the only thing that will ever change a `feedback_weight` in this demo is the feedback you give. Each step is appended to an activity log the frontend renders as it goes.
### Stage 3: Answer Every Question in One Session
Source: `examples/demos/sessions/session_feedback_lifecycle_demo/backend/app.py`
```python theme={null}
await _ensure_dataset_context()
search_order = [SearchType.GRAPH_COMPLETION]
results = None
for search_type in search_order:
try:
results = await cognee.recall(
query_text=question,
query_type=search_type,
datasets=[DATASET_NAME],
session_id=session_id,
top_k=max(1, min(10, int(top_k))),
)
```
Every question — typed or scripted — goes through this one helper. Pinning `query_type` to `GRAPH_COMPLETION` keeps answers on the graph path rather than letting the router pick, which is what makes the graph the thing under test; passing `session_id` is what records the question and its answer as a rateable Q\&A entry. `top_k` comes from the slider in the UI and is clamped to 1–10.
### Stage 4: Catch Feedback Typed Into the Chat
Source: `examples/demos/sessions/session_feedback_lifecycle_demo/backend/app.py`
```python theme={null}
latest_after = await _latest_qa_for_session(session_id)
qa_id_after = getattr(latest_after, "qa_id", None) if latest_after else None
created_new_entry = bool(qa_id_after and qa_id_after != qa_id_before)
updated_feedback_on_same_entry = bool(
latest_after
and qa_id_after
and qa_id_after == qa_id_before
and (
getattr(latest_after, "feedback_score", None) != feedback_score_before
or getattr(latest_after, "feedback_text", None) != feedback_text_before
)
)
```
`POST /demo/send` reads the session's latest entry before and after the search and compares the two. A new `qa_id` means the message was a question; the same `qa_id` with changed feedback fields means `AUTO_FEEDBACK` read the message as a comment on the previous answer and attached it there. That is the branch behind typing "Wrong answer, 1/5" into the chat box: no feedback API is called, and the rating still lands on the right Q\&A entry.
### Stage 5: Attach an Explicit Score to a Q\&A
Source: `examples/demos/sessions/session_feedback_lifecycle_demo/backend/app.py`
```python theme={null}
user = await get_default_user()
state.session_id = payload.session_id
await _ensure_dataset_context()
ok = await cognee.session.add_feedback(
session_id=payload.session_id,
qa_id=payload.qa_id,
feedback_text=payload.feedback_text,
feedback_score=payload.feedback_score,
user=user,
)
```
`POST /demo/feedback` is the explicit path the UI uses when you rate a specific message rather than talking to it. `add_feedback` returns `False` only when the `qa_id` does not exist or caching is disabled, which the endpoint turns into a `404` — a rating that silently failed would be indistinguishable from a rating that moved no weights. An unreachable cache is not a `False`: it raises out of the endpoint instead, per the [feedback return contract](/guides/feedback-system#return-contract-and-errors).
### Stage 6: Read Feedback Weights Off the Graph
Source: `examples/demos/sessions/session_feedback_lifecycle_demo/backend/app.py`
```python theme={null}
graph_engine = await get_graph_engine()
nodes_data, edges_data = await graph_engine.get_graph_data()
node_ids = [str(node_id) for node_id, _ in nodes_data]
node_weights = await graph_engine.get_node_feedback_weights(node_ids) if node_ids else {}
```
`_snapshot_graph` is how the demo sees anything at all: it pulls the whole graph from the engine, then asks the engine for the current `feedback_weight` of every node and every identifiable edge. Weights are clamped to `0.0`–`1.0` with `0.5` as the neutral default, so an untouched graph draws as uniform and any variation you see is feedback.
### Stage 7: Fold the Session's Ratings Into the Graph
Source: `examples/demos/sessions/session_feedback_lifecycle_demo/backend/app.py`
```python theme={null}
before = await _snapshot_graph()
result = await apply_feedback_weights_pipeline(
user=user,
session_ids=[payload.session_id],
dataset=DATASET_NAME,
alpha=MEMIFY_ALPHA,
batch_size=100,
run_in_background=False,
)
after = await _snapshot_graph()
deltas = _compute_deltas(before, after)
```
This is the "Run\_memify\_pipeline" button. The pipeline reads the rated Q\&A entries out of the session, maps them back to the graph elements those answers were retrieved from, and streams the ratings into their weights — the same feedback-weight update [`improve()`](/core-concepts/main-operations/improve#with-session-ids) performs with `session_ids`, called directly here so the demo can bracket it with snapshots. `alpha` is the smoothing factor for that update, fixed at `0.619` in this demo, and `run_in_background=False` makes the request wait so the response can carry a real "after". `_compute_deltas` diffs the two snapshots and reports every element whose weight moved, which is what the UI highlights.
### Stage 8: Replay the Whole Loop From a Script
Source: `examples/demos/sessions/session_feedback_lifecycle_demo/backend/app.py`
```python theme={null}
qa_id = getattr(latest_qa, "qa_id", None)
feedback_score = int(turn.get("feedback_score", 3))
feedback_text = str(turn.get("feedback_text", "Scripted demo feedback"))
if qa_id is not None:
await cognee.session.add_feedback(
session_id=session_id,
qa_id=qa_id,
feedback_text=feedback_text,
feedback_score=feedback_score,
user=user,
)
```
`POST /demo/run_demo` walks the six question-and-rating turns in `data/scripted_flow.json` — high marks for the answers about Emily Carter and Sarah Nguyen, `1` for the two Michael Rodriguez turns — asking each question, rating the entry it produced, and then running the same memify pipeline once at the end. The UI's replay button drives the same six turns a different way: it loops client-side through `/demo/send`, `/demo/feedback`, and `/demo/run_memify_pipeline`, so in the browser the weights shift after every rating rather than once at the end. Either way, it is the fast way to get a graph with visibly uneven weights before you start asking your own questions.
<Columns>
<Card title="Feedback System" icon="brain-circuit" href="/guides/feedback-system">
Recording and clearing feedback on session Q\&A entries.
</Card>
<Card title="Improve" icon="sparkles" href="/core-concepts/main-operations/improve">
The operation that turns session feedback into graph weights.
</Card>
<Card title="Sessions" icon="message-square" href="/guides/sessions">
Working with `session_id` for conversational memory.
</Card>
<Card title="Learn a User's Preferences From Conversation Alone" icon="message-square-quote" href="/examples/live-session-feedback">
The sibling demo, where the conversation itself is the feedback.
</Card>
</Columns>
# Tune How Strongly Ratings Steer an Answer
Source: https://docs.cognee.ai/examples/feedback-score-shifting
Rate one answer up and another down, bake both into graph feedback weights, then sweep feedback_influence from 0.0 to 1.0 over one ambiguous question and watch the ranking move
Your users rate the answers your assistant gives, and now you have to decide how much those ratings should count when memory is searched again. Too little and the feedback is decorative; too much and a single thumbs-up decides everything — and there is no way to pick the number without watching it move.
## What You'll Build
Two bundled text files — five German car manufacturers and five US tech companies — are remembered into one memory, so the graph holds two distinct bodies of context. One session then asks a cars question and rates the answer `5`, asks a companies question and rates that answer `1`, and a feedback-weights pipeline bakes both ratings into the `feedback_weight` of the graph elements each answer used. What comes out is a sweep: the deliberately ambiguous question "List the companies in the context" is asked six times at `feedback_influence` `0.0`, `0.2`, `0.4`, `0.6`, `0.8`, and `1.0`, with every answer printed under its beta, so you can read the dial's effect off one screen instead of guessing at a default.
The complete runnable script is
[`examples/demos/feedback/feedback_score_shifting_example.py`](https://github.com/topoteretes/cognee/blob/main/examples/demos/feedback/feedback_score_shifting_example.py) —
this page walks through its key moments rather than reproducing it.
## Features in Play
* [Remember](/core-concepts/main-operations/remember) — loads both text files into one memory in a single call, so the two topics compete for the same retrieval slots
* [Sessions](/guides/sessions) — one shared `session_id` records each answered question as a QA entry, including which graph elements the answer used, giving the ratings something to attach to
* [Feedback System](/guides/feedback-system) — `add_feedback()` puts the `5` and the `1` on those two QA entries
* [Improve](/core-concepts/main-operations/improve#with-session-ids) — the feedback-weights pipeline the demo calls directly is the stage `improve()` runs for you outside a demo
* [Feedback-Weighted Ranking](/guides/truth-subspace-reranking#relationship-to-the-feedback-loop) — `feedback_influence` is the per-call dial that decides how much the stored weights count during triplet scoring
## What to Expect
The excerpts below come from a real run, trimmed: cognee's progress logs are elided. The step lines are what the script prints; the answers are shown as the text they returned, because `print(str(answer))` wraps each one in the `ResponseGraphEntry(...)` object your terminal will actually show — same content, more scaffolding around it. Ingestion and all eight answers are live LLM calls, so the wording of every answer — and the exact beta where the ranking shifts — varies from run to run.
**Steps 1 to 3 set the experiment up.** Two ratings are stored against the session, then the pipeline writes them into the graph. Nothing has been swept yet — `Feedback weights applied.` is the line that says the graph now carries the `5` and the `1`.
```text theme={null}
Step 1: Ask cars-specific question and give positive feedback (5).
...
Added feedback score=5 for cars context.
Step 2: Ask companies-specific question and give negative feedback (1).
...
Added feedback score=1 for companies context.
Step 3: Apply feedback into graph feedback_weight values (memify).
...
Feedback weights applied.
```
**At low beta the ratings barely register.** The ambiguous question pulls from both documents: all ten names, technology companies first, exactly as semantic similarity ranks them.
```text theme={null}
Step 4: Ask one neutral query while sweeping beta.
As beta increases, ranking should shift toward positively-rated context (companies focused on car manufacturers). 1 means only feedback score is taken into account nothing else.
--- beta = 0.0 (0% feedback influence) ---
- Apple
- Google
- Microsoft
- Amazon
- Meta (formerly Facebook)
- Audi
- BMW
- Mercedes‑Benz
- Porsche
- Volkswagen
--- beta = 0.2 (20% feedback influence) ---
Apple
Google
Microsoft
Amazon
Meta (formerly Facebook)
Audi
BMW
Mercedes-Benz
Porsche
Volkswagen
```
**From `0.4` on the up-rated context owns the answer.** The five technology companies drop out entirely and only the German car manufacturers — the context the `5` was attached to — come back. The answer at `0.6` and `0.8` is identical to both blocks below, elided here: past the tipping point, turning the dial further changes nothing, which is exactly what the sweep is for.
```text theme={null}
--- beta = 0.4 (40% feedback influence) ---
Audi
BMW
Mercedes-Benz
Porsche
Volkswagen
...
--- beta = 1.0 (100% feedback influence) ---
Audi
BMW
Mercedes-Benz
Porsche
Volkswagen
```
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — ingestion and all eight answers are live calls, so the wording of every answer in the sweep varies from run to run
* Set `CACHING=true` and `CACHE_BACKEND=fs`; the script checks both at import time and raises `CogneeConfigurationError` if either is wrong. Sessions and feedback work on any [cache adapter](/core-concepts/sessions-and-caching#cache-adapters) — the script pins the filesystem one so the run needs no external service and leaves its session files where you can inspect them
* Run it from a checkout of the cognee repo: it reads its two documents from the `feedback_score_shifting_example_data/` folder next to the script, so a copy-pasted copy has nothing to ingest
* The run opens with `cognee.forget(everything=True)` and the script does **not** redirect cognee's storage roots, so it clears whatever memory the current configuration points at — run it against a scratch instance rather than storage you want to keep
## How It Works
### Stage 1: Require a Filesystem Session Cache
```python theme={null}
cache_config = get_cache_config()
if not cache_config.caching or cache_config.cache_backend != "fs":
raise CogneeConfigurationError(
"feedback_score_shifting_example requires caching=True and CACHE_BACKEND=fs."
)
```
The check runs at import time, before anything is ingested, because the whole demo hangs off session storage: with `CACHING=false` there are no QA entries to rate, and therefore no ratings for the weights pipeline to read. Any cache backend would store those entries; requiring `fs` is the demo keeping its run self-contained. Either way the check fails loudly on a misconfigured instance instead of quietly sweeping a graph whose weights never moved.
### Stage 2: Put Two Rival Topics in One Memory
```python theme={null}
async def main():
await cognee.forget(everything=True)
await cognee.remember([TEXT_1, TEXT_2], self_improvement=False)
```
`TEXT_1` and `TEXT_2` are the two bundled documents — Audi, BMW, Mercedes-Benz and their peers in one, Apple, Google and theirs in the other — and they go in as a single list, into a single memory. That shared memory is what makes the closing question ambiguous: both files describe companies, so nothing but the ratings can tell retrieval which set to favor. `self_improvement=False` skips the `improve()` pass that `remember()` would otherwise run after cognify, so the graph the sweep queries is exactly what was ingested — and the demo's own pipeline call in Stage 5 is the only thing that ever writes a `feedback_weight`.
### Stage 3: Rate the Cars Context Up
```python theme={null}
await cognee.recall(
query_text="Which German car manufacturers are described and what are they known for?",
query_type=SearchType.GRAPH_COMPLETION,
user=user,
session_id=session_id,
)
qa_cars = (await cognee.session.get_session(session_id=session_id, user=user, last_n=1))[0]
await cognee.session.add_feedback(
session_id=session_id,
qa_id=qa_cars.qa_id,
feedback_score=5,
feedback_text="Cars-focused context is exactly what I want.",
user=user,
)
```
Everything from here on runs as the default user (`get_default_user()`) under one named session the script opens just above. Asking with that `session_id` is what records the exchange, and `get_session(..., last_n=1)` reads that one entry straight back so its `qa_id` can be rated. The question is narrow on purpose: it pulls the car manufacturers into the answer, so the 5/5 lands on exactly the graph elements that describe them.
### Stage 4: Rate the Companies Context Down
```python theme={null}
qa_companies = (await cognee.session.get_session(session_id=session_id, user=user, last_n=1))[0]
await cognee.session.add_feedback(
session_id=session_id,
qa_id=qa_companies.qa_id,
feedback_score=1,
feedback_text="Companies-focused context is less useful for this goal.",
user=user,
)
```
The mirror image, in the same session: a question about technology companies and their products, then a `1` on the answer it produced. Two ratings pulling in opposite directions is what makes the sweep readable — a symmetric signal would raise both topics equally and leave their relative order untouched.
### Stage 5: Bake the Ratings Into the Graph
```python theme={null}
await apply_feedback_weights_pipeline(user=user, session_ids=[session_id], alpha=0.9)
```
Until this call the two ratings live only in the session; the graph has not moved. The pipeline walks the elements each rated answer used and updates their `feedback_weight` toward the normalized rating, with `alpha` setting how far one rating pulls — the default is `0.1`, and the demo turns it up to `0.9` so a single rating per topic is enough to separate them. In an ordinary application this is what [`improve()`](/core-concepts/main-operations/improve#with-session-ids) does for you.
### Stage 6: Sweep the Dial Over One Ambiguous Query
```python theme={null}
final_query = "List the companies in the context"
for beta in [0.0, 0.2, 0.4, 0.6, 0.8, 1.0]:
answer = await cognee.recall(
query_text=final_query,
query_type=SearchType.GRAPH_COMPLETION,
user=user,
feedback_influence=beta,
)
print(f"\n--- beta = {beta:.1f} ({beta * 100:.0f}% feedback influence) ---")
print(str(answer))
```
The same question, six times, with only `feedback_influence` changing. It is asked without a `session_id`, so no conversation history carries the earlier ratings' text forward and the graph plus its new weights are the only thing in play. At `0.0` the weights are ignored and ranking is pure semantic similarity; as beta rises, the script's own note says ranking should shift toward the positively-rated context, and at `1.0` — as it puts it, "only feedback score is taken into account nothing else" — similarity drops out of the triplet score entirely.
## Run It
```bash theme={null}
uv run python examples/demos/feedback/feedback_score_shifting_example.py
```
<Columns>
<Card title="Feedback System" icon="brain-circuit" href="/guides/feedback-system">
Rating a session answer and pushing that rating into graph weights.
</Card>
<Card title="Truth-Subspace Reranking" icon="compass" href="/guides/truth-subspace-reranking">
Where `feedback_weight` meets retrieval scoring, alongside truth weighting.
</Card>
<Card title="Improve" icon="sparkles" href="/core-concepts/main-operations/improve">
The operation that applies feedback weights for you outside a demo.
</Card>
<Card title="Resolve Conflicting Facts in Memory" icon="scale" href="/examples/contradiction-handling">
The sibling demo, where a rating steers attention but cannot decide what is true.
</Card>
</Columns>
# HR Resume Screening
Source: https://docs.cognee.ai/examples/hr-resume-screening
Turn a pile of CVs into a queryable candidate graph, then ask who has a given skill — rebuilding the graph only when the corpus changes
A recruiter has five CVs on disk and one question per role: who here actually has the skill we are hiring for? The CVs never change while the questions keep coming, so the expensive part — reading them into memory — should happen once and the asking should be cheap to repeat.
## What You'll Build
Five plain-text CVs from a sibling data folder — three data scientists, a graphic designer, and a sales manager — are ingested into one candidate knowledge graph, and a single graph-grounded query asks which of them has experience with design tools. The run is split into four phases (prune data, prune system, remember, retrieve), each one gated behind its own boolean, so the first run builds the graph and every later run can skip straight to the query against the graph that is already there.
The complete runnable script is
[`examples/demos/custom_pipelines/dynamic_steps_resume_analysis_hr_example.py`](https://github.com/topoteretes/cognee/blob/main/examples/demos/custom_pipelines/dynamic_steps_resume_analysis_hr_example.py) —
this page walks through its key moments rather than reproducing it.
## Features in Play
* [Remember](/core-concepts/main-operations/remember) — turns the five CV texts into a candidate graph in one call, with `self_improvement=False` to keep ingestion to the plain add-and-cognify path
* [Recall](/core-concepts/main-operations/recall) — asks the screening question against the finished graph with a pinned `query_type`
* [Inspecting Graph Completion Context](/guides/graph-completion) — `SearchType.GRAPH_COMPLETION` is what grounds the answer in graph triplets extracted from the CVs rather than raw resume text
* [Delete](/core-concepts/main-operations/legacy-operations/delete) — `prune_data()` and `prune_system(metadata=True)` wipe the previous run's storage and metadata when the rebuild phases are switched on
## What to Expect
The excerpts below come from one real run with both phases switched on, trimmed of most log lines. Ingestion and the closing query are live LLM calls, so the answer's wording varies from run to run; the shape of the output does not.
**The two resets confirm the run starts from nothing.** `Data pruned.` follows the file-storage wipe, and `System pruned.` follows the graph, vector, relational, and cache deletions — the relational database is dropped because the script passes `metadata=True`, which is also why the next phase begins by rebuilding the schema from scratch.
```text theme={null}
Data pruned.
...
System pruned.
```
**One `Remembering text:` line per CV, then a single confirmation.** Each line prints the first 35 characters of the resume, which is why the candidate's name wraps onto a second line, and the `Relevant` / `Not Relevant` labels are the first line of the sample files themselves, not something cognee added. Between the fifth line and `Knowledge graph created.` the remember call runs the add and cognify pipelines over all five resumes — about a minute and a half in this run, most of it in the `extract_graph_and_summarize` tasks.
```text theme={null}
Remembering text: CV 1: Relevant
Name: Dr. Emily Cart...
Remembering text: CV 2: Relevant
Name: Michael Rodrig...
Remembering text: CV 3: Relevant
Name: Sarah Nguyen
C...
Remembering text: CV 4: Not Relevant
Name: David Thom...
Remembering text: CV 5: Not Relevant
Name: Jessica Mi...
...
Knowledge graph created.
```
**The router log shows why `query_type` is pinned.** Left to itself, recall would have routed this question to `HYBRID_COMPLETION`; the script's explicit `SearchType.GRAPH_COMPLETION` overrides that, and the retrieval lines that follow show the graph path at work — a subgraph of 142 nodes and 358 edges is projected and narrowed to the 13 nodes and 15 connections that become the answer's context. The final line is the script's own print: a one-element list naming the graphic designer and the tools he is connected to, even though the question never named any of them.
```text theme={null}
2026-09-10T10:49:20.947808 [info ] query_router: no patterns matched, default=HYBRID_COMPLETION query='Who has experience in design tools?' [query_router]
2026-09-10T10:49:20.947891 [info ] Router override recorded: routed=HYBRID_COMPLETION, user_chose=GRAPH_COMPLETION (total=1) [query_router]
...
2026-09-10T10:49:21.612145 [info ] Graph projection completed: 142 nodes, 358 edges in 0.00s [CogneeGraph]
2026-09-10T10:49:21.613129 [info ] Completed resolving edges to text [cognee.shared.logging_utils] extra={'node_count': 13, 'connection_count': 15}
2026-09-10T10:49:31.367201 [info ] recall: 1 results across sources=['graph'] (session=-) [recall]
['David Thompson — experienced with design tools including Adobe Photoshop, Illustrator, and InDesign.']
```
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — both the ingestion phase and the closing query make live LLM calls
* Run it from a checkout of the cognee repo: the script reads `cv_1.txt` through `cv_5.txt` from the sibling `dynamic_steps_resume_analysis_hr_example_data/` folder
* Point it at a scratch instance on the first run: with the rebuild phases on, it calls `prune_data()` and `prune_system(metadata=True)`, which wipe stored data and drop the relational database rather than deleting one dataset — see [Delete](/core-concepts/main-operations/legacy-operations/delete)
## How It Works
### Stage 1: Reset the Previous Run's State
```python theme={null}
if enable_steps.get("prune_data"):
await cognee.prune.prune_data()
print("Data pruned.")
if enable_steps.get("prune_system"):
await cognee.prune.prune_system(metadata=True)
print("System pruned.")
```
Every phase in `main()` is wrapped in an `enable_steps.get(...)` check, and the two resets are the first of them. They exist so a rebuild starts from an empty store instead of layering a second copy of the same five candidates onto the graph — which is also why they are the phases you turn off once the graph is built.
### Stage 2: Remember the CV Corpus
```python theme={null}
if enable_steps.get("remember"):
text_list = [job_1, job_2, job_3, job_4, job_5]
for text in text_list:
print(f"Remembering text: {text[:35]}...")
await cognee.remember(text_list, self_improvement=False)
print("Knowledge graph created.")
```
One `remember()` call takes the list of five CV strings and does the whole ingestion: the resumes are chunked, entities like people, skills, tools, and employers are extracted, and the result is stored as a graph plus embeddings. `self_improvement=False` stops the follow-up [improve](/core-concepts/main-operations/improve) pass, so this is the plain permanent-ingestion path and nothing enriches the graph behind the scenes.
### Stage 3: Ask the Screening Question
```python theme={null}
if enable_steps.get("retriever"):
results = await cognee.recall(
query_type=SearchType.GRAPH_COMPLETION, query_text="Who has experience in design tools?"
)
print([result.text for result in results])
```
The screening question goes to `recall()` with `query_type` pinned to `SearchType.GRAPH_COMPLETION`, so the answer is generated from graph triplets rather than from whichever resume chunk happens to look similar to the question. That matters for a question like this one: "design tools" appears in none of the CVs verbatim, and the answer has to come from the tools a candidate is connected to.
### Stage 4: Choose Which Phases Run
```python theme={null}
rebuild_kg = True
retrieve = True
steps_to_enable = {
"prune_data": rebuild_kg,
"prune_system": rebuild_kg,
"remember": rebuild_kg,
"retriever": retrieve,
}
```
Two booleans at the entry point drive the four phases: `rebuild_kg` groups both prunes and the ingestion, `retrieve` covers the query. Both start out `True`, which is what a first run needs — and editing them is how the same script becomes either a rebuild or a query-only run.
## Run It
```bash theme={null}
uv run python examples/demos/custom_pipelines/dynamic_steps_resume_analysis_hr_example.py
```
A full run takes about two minutes, nearly all of it in the remember phase. Its output is walked through in [What to Expect](#what-to-expect) above.
## Re-Running Only the Query
Once the graph exists, set `rebuild_kg = False` and leave `retrieve = True`. Both prunes and the `remember()` call are skipped, the script goes straight to the recall against the stored graph, and the question is answered in about ten seconds instead of waiting on the rebuild. Edit `query_text` and re-run to screen the same corpus for a different skill; flip `rebuild_kg` back to `True` only when the CVs in the data folder change.
<Columns>
<Card title="Remember" icon="brain" href="/core-concepts/main-operations/remember">
The ingestion call behind the candidate graph, and what `self_improvement` changes.
</Card>
<Card title="Recall" icon="search" href="/core-concepts/main-operations/recall">
Query routing, pinning a `query_type`, and the other retrieval parameters.
</Card>
<Card title="Inspecting Graph Completion Context" icon="route" href="/guides/graph-completion">
What `GRAPH_COMPLETION` retrieves from the graph before an answer is generated.
</Card>
<Card title="Human Resources" icon="link" href="/examples/human-resources">
The wider HR use case: aligning CVs with job posts through entity resolution.
</Card>
</Columns>
# Learn a User's Preferences From Conversation Alone
Source: https://docs.cognee.ai/examples/live-session-feedback
Run a ten-turn consulting conversation in which stated preferences, corrections, and style rules become session guidance without anyone calling a feedback API
You are building an assistant that plans work with a user over a long conversation, and the user keeps changing the brief mid-stream — a constraint you missed, an ordering rule, a switch from two bullet points to four. You would like the assistant to absorb each of those as it happens, without the user ever having to say "remember this."
## What You'll Build
Fifteen sentences about a fictional logistics company — four offices, four projects, audit windows, and the people who lead them — are remembered into an isolated dataset. A single session then runs ten turns of a consulting conversation on top of it, in which the user asks for an audit itinerary and then repeatedly amends the brief: Singapore must come before Toronto, Priya before Mateo, Lisbon can be a video call, answers should now be four bullets instead of two, customer-facing notes should be operational rather than technical. Because `AUTO_FEEDBACK` is on, every answered turn is analyzed for exactly that kind of statement, and whatever the analysis judges durable is written into the session's guidance layer as a gated entry. The script prints the growing list after every turn along with the QA history and which guidance IDs the latest answer used. The run also opens with an `only_context` probe that shows a context read adds no QA entry, and ends by dumping the whole trace as JSON.
The complete runnable script is
[`examples/demos/sessions/live_session_context_feedback_demo.py`](https://github.com/topoteretes/cognee/blob/main/examples/demos/sessions/live_session_context_feedback_demo.py) —
this page walks through its key moments rather than reproducing it.
## Features in Play
* [Remember](/core-concepts/main-operations/remember) — loads the fifteen Northstar Labs facts into the demo dataset in one call
* [Recall](/core-concepts/main-operations/recall) — answers every turn with `GRAPH_COMPLETION` against that dataset
* [Sessions](/guides/sessions) — the shared `session_id` that makes ten separate `recall()` calls one conversation
* [Session-Context Guidance](/core-concepts/sessions-and-caching#session-context-guidance-auto-feedback) — `AUTO_FEEDBACK` is what turns a stated preference or correction into a gated guidance entry on the turn that states it
* [Search Basics](/guides/search-basics#parameters-reference) — `only_context=True` returns the retrieval context instead of an answer, which the probe uses to show that reading context is side-effect-free
## What to Expect
The excerpts below are from a real run, trimmed. Per-turn snapshots stream to stderr and one JSON document lands on stdout at the end; answer wording and the exact guidance entries the analysis absorbs vary from run to run. Re-run with `--no-ingest` to keep the ingested dataset and only reset the session.
**The probe proves context reads are free.** A real question runs with `only_context=True` before the first turn, and the QA count is identical on both sides — reading retrieval context stores no Q\&A turn.
```text theme={null}
[live-session-demo] Running context-only probe; this should not register a QA entry.
[live-session-demo] Context-only probe complete: QA count before=0, after=0.
```
**A stated preference becomes guidance on the turn that states it.** By turn 2 the session context already holds six gated entries — the three absorbed from turn 1 plus the goal, ordering rule, and visit-order preference this very message stated — and `used_session_context_ids` shows the answer drew on the three that existed when it ran.
```text theme={null}
--- Turn 2: goal_and_order_preference ---
user: That helps. My goal is to create a practical audit itinerary, and I prefer visiting Berlin and Lisbon before Singapore and Toronto. For now, answer with 2 informative bullet points.
assistant: - Proposed itinerary (in your preferred order): Mon — Berlin (RoutePulse): review traffic feeds, weather alerts, customs delay reports; stakeholders: RoutePulse/local ops; window: Monday morning. Tue — Lisbon (HarborLens): review vessel schedules, berth availability, labor notices; stakeholders: HarborLens/port liaison; window: Tuesday afternoon. ...
...
qa_count: 2
latest_qa.used_session_context_ids: ['c84daa5c-e8a3-4844-aca8-3157ffc557ce', 'b679d9b1-b055-48a8-a773-0d334e83a641', '67d26d3f-0b60-4564-9129-25e871b095c3']
session_context:
- [goals] User's objective is to create a practical, prioritized audit itinerary covering Northstar Labs' offices, key projects, and relevant audit topics. (helpful=0, harmful=0)
- [preferences] Future recommendations should list offices, associated high-priority projects, audit topics, suggested stakeholders, and estimated time per site. (helpful=0, harmful=0)
- [rules] Prioritize offices and projects that handle sensitive data, critical infrastructure, or have high regulatory exposure when recommending audit coverage. (helpful=0, harmful=0)
- [goals] Create a practical, efficient audit itinerary for Northstar Labs offices covering projects, audit topics, and logistics. (helpful=0, harmful=0)
- [rules] Respect each office's stated availability windows and minimize unnecessary backtracking when scheduling visits. (helpful=0, harmful=0)
- [preferences] Visit Berlin and Lisbon before Singapore and Toronto (preferred visit order). (helpful=0, harmful=0)
...
```
**Helpful and harmful counts move as guidance gets used — or overridden.** By the final snapshot the availability rule has proven helpful twice, and the superseded two-bullet format preference carries a harmful mark, with its four-bullet replacement stored alongside it.
```text theme={null}
...
- [rules] Respect each office's stated availability windows and minimize unnecessary backtracking when scheduling visits. (helpful=2, harmful=0)
...
- [preferences] When requested, provide answers in the concise format specified (e.g., exactly two informative bullet points). (helpful=0, harmful=1)
...
- [preferences] Use exactly four concise bullet points for future responses. (helpful=0, harmful=0)
...
```
**The closing JSON repeats everything in structured form** — the dataset and session IDs, the probe result, and one entry per turn with the serialized response and the evidence snapshot taken after it.
```text theme={null}
{
"dataset": "northstar_labs_live_session_demo",
"session_id": "northstar_live_session",
...
"qa_was_registered": false
...
```
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — ingestion, every answer, and the per-turn feedback detection are all live calls, so answer wording and the learned guidance text vary by model
* Expect the script to set its own environment before importing cognee: it pins `CACHING=true` and `AUTO_FEEDBACK=true` — both already the defaults — switches `CACHE_BACKEND` to `fs` from the default `sqlite`, and defaults `LOG_LEVEL` to `ERROR`. See [Sessions and Caching](/core-concepts/sessions-and-caching#cache-adapters) for what those control
* Run it from a checkout of the cognee repo: it points cognee's data, system, and cache roots at `examples/temp/live_session_context_feedback_demo/` and works only inside that folder
* A full run starts with `cognee.forget(everything=True)` against that isolated root, so it clears its own demo storage rather than memory you want to keep
## How It Works
### Stage 1: Pin the Session Feedback Settings
```python theme={null}
os.environ["CACHING"] = "true"
os.environ["CACHE_BACKEND"] = "fs"
os.environ["AUTO_FEEDBACK"] = "true"
os.environ.setdefault("LOG_LEVEL", "ERROR")
```
These four lines run before `import cognee`, which is what makes them take effect. `CACHING` and `AUTO_FEEDBACK` are both on by default, so setting them here is not what enables the behavior — it pins it, so the demo runs the same way on an instance where either was switched off. `AUTO_FEEDBACK` is the setting behind the per-turn analysis call; with it off, the session would still replay conversation history but would learn nothing from what the user says. `CACHE_BACKEND=fs` is the one real departure from the defaults, putting the session cache in files under the demo's own root instead of the default `sqlite` backend.
### Stage 2: Seed the Northstar Labs Facts
```python theme={null}
async def setup_demo_data():
await configure_demo_storage(reset_storage=True)
progress("Clearing previous demo state.")
await cognee.forget(everything=True)
progress(f"Ingesting {len(DOCUMENTS)} Northstar Labs facts.")
await cognee.remember(DOCUMENTS, dataset_name=DATASET_NAME, self_improvement=False)
progress("Ingestion complete.")
```
One `remember()` builds the permanent graph the conversation will be grounded in: offices, the project each one owns, the data each project consumes, and the audit windows and leads. `self_improvement=False` skips the enrichment pass, because this demo is about what the session learns, not what the graph does. Nothing in these documents states a visit order or a bullet-point preference — those can only come from the conversation. Setup then deletes any previous copy of the demo session, so guidance growth starts from zero and every printed entry is attributable to this run; the closing JSON reports whether it found one as `session_was_deleted`.
### Stage 3: Route Every Question Through One Session
```python theme={null}
async def ask(message: str, *, user, only_context: bool) -> Any:
return await cognee.recall(
query_text=message,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET_NAME],
session_id=SESSION_ID,
user=user,
only_context=only_context,
)
```
Every turn and the probe go through this one helper, so the only thing that differs between them is `only_context`. The shared `SESSION_ID` is what makes ten independent `recall()` calls a single conversation, and pinning `query_type` keeps each turn on the graph-completion path rather than letting the router pick.
### Stage 4: Probe the Context Without Writing to It
```python theme={null}
async def run_context_only_probe(user) -> dict:
progress("Running context-only probe; this should not register a QA entry.")
before = await session_evidence(user)
context = await ask(
"Which Northstar offices are mentioned?",
user=user,
only_context=True,
)
after = await session_evidence(user)
```
The probe asks a real question with `only_context=True` and takes an evidence snapshot on either side of it. The QA count should be identical before and after — the probe records that comparison as `qa_was_registered` in its result, which is how the demo shows that reading retrieval context stores no Q\&A turn. The per-turn analysis is skipped for `only_context` calls as well, but the probe runs before the first turn, when the guidance layer is empty either way, so that half is documented behavior rather than something this output demonstrates — see [Session-Context Guidance](/core-concepts/sessions-and-caching#session-context-guidance-auto-feedback).
### Stage 5: Amend the Brief Mid-Conversation
```python theme={null}
{
"label": "communication_preference_update",
"message": (
"Actually, change my communication preference: I now prefer 4 concise bullet "
"points instead of 2 informative bullet points."
),
},
{
"label": "customer_facing_style_rule",
"message": (
"Good. Also remember that customer-facing audit notes should be operational, "
"not technical."
),
},
```
`TURNS` is a list of ten such messages, and these two are the eighth and ninth: the first supersedes a formatting preference stated back in turn two, the second adds a style rule. Neither is phrased as an instruction to the memory — the labels are narration for the printout, and the session decides on its own what is worth keeping.
### Stage 6: Read Back the Session Evidence
```python theme={null}
async def session_evidence(user) -> dict:
qa_entries = await cognee.session.get_session(session_id=SESSION_ID, user=user)
context_entries = await get_session_manager().get_session_context_entries(
user_id=str(user.id),
session_id=SESSION_ID,
)
return {
"qa_count": len(qa_entries),
"latest_qa": serialize_latest_qa(qa_entries),
"session_context_entries": serialize_context_entries(context_entries),
}
```
This helper is the instrument the demo reads the session with, and every snapshot in the run is one call to it, drawing on two sources: the stored Q\&A history, and the session's context entries. The script keeps only entries whose `kind` is `context` and prints each one's section, content, and helpful and harmful counts — the running tally that feeds an entry's ranking score alongside its section, confidence, and overlap with the query. Alongside them it prints the latest QA's `used_session_context_ids`, which links an answer back to the guidance entries that shaped it.
### Stage 7: Watch the Guidance Grow, Turn by Turn
```python theme={null}
for index, turn in enumerate(TURNS, start=1):
progress(f"Turn {index}: {turn['label']}")
response = await ask(turn["message"], user=user, only_context=False)
evidence = await session_evidence(user)
print_turn_snapshot(
turn_number=index,
label=turn["label"],
user_message=turn["message"],
response=response,
evidence=evidence,
)
output["turns"].append(
{
"turn": index,
"label": turn["label"],
"user_message": turn["message"],
"assistant_response": serialize_response(response),
"evidence": evidence,
}
)
```
Everything above comes together in these twenty lines: answer the turn, immediately re-read the session, print the two side by side, and keep the same pairing in the JSON. Because the snapshot is taken after every turn rather than once at the end, the printed guidance list is a running record of what the conversation taught the session — and the only input to any of it is a user message.
One thing to expect while you watch: the effect arrives one turn later than the statement. The script leaves `SESSION_SEARCH_MODE` at its `concurrent` default, where the analysis runs alongside answer generation, so a new entry lands after the current turn's reply and shapes the next one — turn eight still answers in two bullets, and the switch to four shows up from turn nine. Set `SESSION_SEARCH_MODE=sequential` if you want guidance to reach the same turn that stated it; see [Session-Context Guidance](/core-concepts/sessions-and-caching#session-context-guidance-auto-feedback) for the trade.
## Run It
```bash theme={null}
uv run python examples/demos/sessions/live_session_context_feedback_demo.py
```
<Columns>
<Card title="Sessions" icon="message-square" href="/guides/sessions">
Working with `session_id` for conversational memory.
</Card>
<Card title="Sessions and Caching" icon="message-square" href="/core-concepts/sessions-and-caching">
What `AUTO_FEEDBACK` analyzes on each turn, and when its guidance applies.
</Card>
<Card title="Recall" icon="search" href="/core-concepts/main-operations/recall">
The query path every turn takes, including `only_context`.
</Card>
<Card title="Watch a Session Become Permanent Memory" icon="repeat" href="/examples/memory-loop-walkthrough">
The sibling demo, where session guidance is distilled into the graph.
</Card>
</Columns>
# Watch a Session Become Permanent Memory
Source: https://docs.cognee.ai/examples/memory-loop-walkthrough
Follow a narrated run in which conversation rules and lessons are absorbed turn by turn, distilled into the knowledge graph, and answered back from a brand-new session
You have been told that session memory becomes permanent memory, and you would like to see it happen rather than take it on faith — which turn absorbed which rule, when the distillation fired, and whether a session that never heard the conversation can still answer from it.
## What You'll Build
Five sentences about a fictional robotics company are remembered into a permanent graph. A six-turn session then runs on top of it, mixing rules, lessons, and preferences that appear nowhere in those seed documents with ordinary questions — and every turn prints the guidance the session absorbed from it. Every third turn, an `improve()` checkpoint distills that accumulated guidance into the graph. The payoff is the last two steps: the same lesson is asked for from a brand-new session that has no conversation history to lean on, and the graph is rendered to HTML with the distilled nodes ringed in gold.
The complete runnable script is
[`examples/demos/sessions/session_flow_stepwise_demo.py`](https://github.com/topoteretes/cognee/blob/main/examples/demos/sessions/session_flow_stepwise_demo.py) —
this page walks through its key moments rather than reproducing it.
## Features in Play
* [Remember](/core-concepts/main-operations/remember) — seeds the permanent graph with the five robotics documents in one call
* [Sessions](/guides/sessions) — the shared `session_id` that makes six separate `recall()` calls one conversation
* [Sessions and Caching](/core-concepts/sessions-and-caching) — `AUTO_FEEDBACK` is what turns a stated rule into a gated guidance entry on the turn that states it
* [Session Distillation](/guides/session-distillation) — curates those gated entries into permanent `session_learnings` lessons
* [Improve](/core-concepts/main-operations/improve) — the call the demo fires every third turn to trigger that distillation
* [Graph Visualization](/guides/graph-visualization) — renders the finished graph so the distilled nodes can be found by eye
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — every turn is a live recall, and the distillation checkpoints add more calls on top
* Keep [caching enabled](/core-concepts/sessions-and-caching#cache-adapters) so the session cache is available; `CACHE_BACKEND=sqlite` is the default, and the script warns on stderr and degrades if the cache is unavailable
* Run it from a checkout of the cognee repo: it loads the repo-root `.env` with `override=True` before importing cognee, and writes its narrated log and the graph HTML into a sibling `logs/` folder
* Expect it to set some environment for you — it forces `AUTO_FEEDBACK=true`, defaults `LLM_MODEL` to `openai/gpt-4o-mini` on the OpenAI path when unset, mirrors `LLM_API_KEY` into `OPENAI_API_KEY`, and skips cognee's LLM preflight. Set `DEMO_USE_OLLAMA=1` instead to run it fully locally against `ollama serve` (see [Local Ollama](/guides/local-ollama))
* The run starts by pruning data and system metadata, so point it at a scratch instance rather than memory you want to keep
## How It Works
### Stage 1: Seed Permanent Memory
```python theme={null}
banner("(A) remember(documents) -> PERMANENT memory (runs add -> cognify -> improve)")
await cognee.prune.prune_data()
await cognee.prune.prune_system(metadata=True)
step(f"remember() {len(DOCUMENTS)} documents into dataset '{DATASET_NAME}'")
result = await cognee.remember(DOCUMENTS, dataset_name=DATASET_NAME)
```
A single `remember()` with a `dataset_name` and no `session_id` takes the permanent path: add, cognify, then improve. The five documents describe products, firmware, and people at Aurora Robotics — the baseline the session will later be measured against, because nothing in them says what a technician must do after flashing firmware.
### Stage 2: Script a Session of Rules and Questions
```python theme={null}
SESSION_TURNS = [
("rule", "Always run the HALT test suite before a VoltaArm firmware release."),
(
"lesson",
"Flashing VoltaArm firmware wipes calibration data, so calibration must be re-run afterwards.",
),
("question", "How does the TerraScout rover navigate warehouses?"),
("preference", "Keep firmware answers to a few short bullet points."),
(
"lesson",
"After re-running VoltaArm calibration, verify it against the battery-backed memory bank.",
),
("question", "Who leads the VoltaArm firmware team?"),
]
# Demo cadence: automatically extract learnings every N turns (option #1). Kept small and
# in-demo on purpose — real cadence/triggering would live in the session lifecycle, not here.
AUTO_DISTILL_EVERY = 3
```
The turns interleave two kinds of message: durable statements (a release rule, two firmware lessons, a formatting preference) and plain questions the seed documents can already answer. The labels are narration only — the session has no idea which is which, and `AUTO_FEEDBACK` decides on its own what is worth keeping.
### Stage 3: Absorb Guidance Turn by Turn
```python theme={null}
async def run_multi_turn_session(user) -> None:
banner("(B+C) multi-turn session -> recall() absorbs guidance; auto-distill every N turns")
for turn_no, (label, message) in enumerate(SESSION_TURNS, start=1):
step(f"turn {turn_no} [{label}]", message)
await cognee.recall(
query_text=message,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET_NAME],
session_id=SESSION_ID,
user=user,
)
if turn_no % AUTO_DISTILL_EVERY == 0:
await auto_distill_checkpoint(user, turn_no)
```
Every turn is an ordinary `recall()`; the only thing making them a conversation is the shared `SESSION_ID`. Because `AUTO_FEEDBACK` is on, each answered turn is also analyzed, and any rule or lesson it states is written into the session's active-guidance layer as a gated entry — the raw material distillation will later curate.
### Stage 4: Distill the Session into the Graph
```python theme={null}
step(
f"auto-extraction checkpoint (after turn {turn_no}, every {AUTO_DISTILL_EVERY} turns)",
"calling improve(session_ids=[...]) -> distills accumulated learnings into the graph",
)
await cognee.improve(dataset=DATASET_NAME, session_ids=[SESSION_ID], user=user)
await show_gated_guidance(user)
```
`improve(session_ids=[...])` is what promotes the session's gated guidance into permanent lessons, so no explicit distillation call is needed. The demo fires it on a simple every-third-turn cadence and then prints the guidance entries with their sections and confidences, so you can see exactly what went in.
### Stage 5: Verify from a Fresh Session
```python theme={null}
result = await cognee.recall(
query_text=LESSON_QUESTION,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET_NAME],
session_id="verification_session",
user=user,
)
```
The question — what a technician must do after flashing VoltaArm firmware, and why — is asked under a different `session_id`, so there is no conversation history to answer from. If the answer still carries the firmware-and-calibration lesson, it can only have come from the graph, which is the whole point of the run.
### Stage 6: Find the Distilled Nodes in the Graph
```python theme={null}
html = await cognee.visualize_graph(
destination_file_path=str(VIZ_PATH),
user=user,
)
has_ring = "#FFC53D" in html
```
The final step renders the graph to an HTML file next to the run's log. Distilled session-learning nodes are drawn with a dashed gold ring, and the script checks for that color in the markup so the console tells you whether any distilled nodes exist before you open the file.
## Run It
```bash theme={null}
uv run python examples/demos/sessions/session_flow_stepwise_demo.py
```
The run narrates itself in five stages, `(A)` through `(E)`. `(A)` prints the `RememberResult` — status, dataset name, and item count — for the five seeded documents. `(B+C)` prints each of the six turns with its label and message, and after turns 3 and 6 an auto-extraction checkpoint followed by the absorbed guidance entries, each with its section, confidence, and truncated text. `(D)` prints the fresh-session question and the answer the long-term graph alone produced. `(E)` prints the path of the rendered HTML and whether the gold memory ring was found in it. A closing `DONE` banner repeats the log and visualization paths; the wording of answers and distilled lessons varies by model.
<Columns>
<Card title="Session Distillation" icon="graduation-cap" href="/guides/session-distillation">
How gated session guidance becomes permanent `session_learnings` lessons.
</Card>
<Card title="Improve" icon="sparkles" href="/core-concepts/main-operations/improve">
The operation behind the distillation checkpoints, and what else it bridges.
</Card>
<Card title="Sessions" icon="message-square" href="/guides/sessions">
Working with `session_id` for conversational memory.
</Card>
<Card title="Graph Visualization" icon="network" href="/guides/graph-visualization">
Rendering a graph to HTML and controlling what it shows.
</Card>
</Columns>
# Migrate Memory Systems with COGX
Source: https://docs.cognee.ai/examples/migrate-memory-systems
Import memories from Mem0, LangMem, Letta, Zep, Graphiti, or another Cognee instance using the COGX exchange format.
Already storing memories in another system? Cognee can import them directly into
a knowledge graph — no manual reformatting. This guide explains the **COGX**
format that makes that possible, walks through a runnable Mem0 example, and
lists every other source you can import from.
<Note>
Migrating a large amount of data? [Chat with us](https://calendly.com/vasilije-topoteretes/) and we'll help you plan it.
</Note>
## What is COGX?
**COGX (the Cognee eXchange format) is a common shape that all memory imports
are translated into before they enter Cognee.** Instead of writing one importer
for every memory tool, Cognee defines a single intermediate format and a single
loader:
```
Mem0 / LangMem / Letta / Zep / Graphiti ──► COGX records ──► cognee.remember() ──► knowledge graph
(a "source") (common shape) (one loader) (queryable memory)
```
A **source** is a small adapter that reads one provider's export and emits COGX
records. Because every source produces the same COGX shape, the rest of the
pipeline — loading, graph extraction, storage — is identical no matter where
your data came from. COGX is also what `cognee.export()` writes, so the same
format powers backup, restore, and Cognee-to-Cognee migration. For a full
breakdown of the format and its record kinds, see
[COGX Exchange Format](/core-concepts/further-concepts/cogx).
You never construct COGX records by hand. You hand a source object to
`cognee.remember()` and it does the rest:
```python theme={null}
import cognee
from cognee.migration import Mem0Source
await cognee.remember(Mem0Source("mem0_export.json"), dataset_name="my_memories")
```
## Quickstart: import from Mem0
The import itself is a single call — construct a `Mem0Source` from your export
and hand it to `cognee.remember()`:
```python theme={null}
result = await cognee.remember(
Mem0Source(MEM0_MEMORIES, mode="re-derive"),
dataset_name=DATASET,
)
```
That's the whole migration: read the export with `Mem0Source`, pass it to
`cognee.remember()`, then query with `cognee.recall()`. Your Mem0 memories are
now a queryable Cognee knowledge graph.
<Accordion title="Simple runnable example" icon="play">
This script uses a small **inline sample** in the exact shape Mem0 returns, so
you can run it without a Mem0 account. Swap in real data using the patterns at
the [bottom of this page](#using-real-data).
<Note>
Requires `LLM_API_KEY` to be set (in `.env` or your environment) — both the
import and `recall()` use the LLM.
</Note>
```python migrate_from_mem0.py theme={null}
import asyncio
import cognee
from cognee.migration import Mem0Source
DATASET = "mem0_import"
# A sample Mem0 export — exactly the shape Mem0 produces: a list of memory
# objects (the OSS `client.get_all()` result, or the items inside a platform
# export's {"results": [...]} wrapper).
MEM0_MEMORIES = [
{
"id": "0a1b2c3d",
"memory": "Alex is a senior backend engineer who owns the payments service.",
"user_id": "alex",
"categories": ["work", "role"],
"created_at": "2026-05-01T10:00:00Z",
},
{
"id": "1b2c3d4e",
"memory": "Alex prefers Python and is wary of premature microservices.",
"user_id": "alex",
"categories": ["preferences"],
"created_at": "2026-05-02T09:30:00Z",
},
{
"id": "2c3d4e5f",
"memory": "The payments service had a timeout incident caused by a missing DB index.",
"user_id": "alex",
"categories": ["incident"],
"created_at": "2026-05-10T14:15:00Z",
},
]
async def main() -> None:
# Start clean so the example is reproducible.
await cognee.forget(everything=True)
# Import the Mem0 memories. mode="re-derive" (the default) runs Cognee's own
# extraction over each memory, building a real entity/relationship graph.
print(f"Importing {len(MEM0_MEMORIES)} Mem0 memories...")
result = await cognee.remember(
Mem0Source(MEM0_MEMORIES, mode="re-derive"),
dataset_name=DATASET,
)
print(f"Done: {result}\n")
# Query the migrated memory.
for question in (
"What does Alex work on?",
"What caused the payments service incident?",
"What are Alex's technical preferences?",
):
answer = await cognee.recall(question, datasets=[DATASET])
print(f"Q: {question}")
print(f"A: {answer}\n")
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
A fully runnable version of this walkthrough ships with the repo:
[`examples/demos/ingestion_and_migration/migrate_from_mem0/migrate_from_mem0.py`](https://github.com/topoteretes/cognee/blob/dev/examples/demos/ingestion_and_migration/migrate_from_mem0/migrate_from_mem0.py).
It imports a bundled sample export of four mem0 memories
([`data/mem0_export.json`](https://github.com/topoteretes/cognee/blob/dev/examples/demos/ingestion_and_migration/migrate_from_mem0/data/mem0_export.json))
in `preserve` mode — calling `cognee.cognify()` right after, since a preserve
import alone isn't recall-queryable (see [Import modes](#import-modes) below) —
and verifies the result with two `recall()` queries, then clears everything,
re-imports the same export in `re-derive` mode, and queries again so you can
compare the two modes side by side. It finishes with `forget(everything=True)`
so re-runs start clean. Run it with:
```bash theme={null}
uv run python examples/demos/ingestion_and_migration/migrate_from_mem0/migrate_from_mem0.py
```
## Import modes
Every source accepts a `mode` argument that controls how much work Cognee does
on import. Pick it based on whether your source already has an extracted graph
and how much you want to spend on LLM calls.
| Mode | What it does | When to use it |
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `re-derive` | Ingests the raw content and runs Cognee's own extraction (`cognify`). The source's own graph (if any) is ignored. | You want the richest graph and don't mind the LLM cost. **Default for Mem0, LangMem, and Letta.** |
| `preserve` | Maps the source's already-extracted entities and facts straight into the graph with **zero LLM calls**. Raw content is stored but not re-processed. | Your source already has a good graph, or you want a fast, free, deterministic import. **Default for COGX archives.** |
| `hybrid` | Keeps the source's graph **and** re-cognifies the raw content. | Your source has both verbatim content and a derived graph (e.g. Zep/Graphiti) and you want the best of both. **Default for Zep/Graphiti.** |
```python theme={null}
# Override the default for any source:
COGXArchiveSource("./archive", mode="hybrid") # preserve the graph and re-cognify raw content
ZepSource(data, mode="re-derive") # ignore Zep's graph, re-extract from scratch
```
Mem0 exports contain short memory records, not a graph, so use the default
`re-derive` mode for Mem0 imports when you want Cognee to build a knowledge
graph from those memories. `preserve` is also valid for Mem0: each memory is
stored as a raw data item with zero LLM calls — the cheapest way to get the
records in, ready for a later `cognify()` run.
That makes `preserve` a **two-call pattern** whenever you want to query the
imported memories: `remember()` lands the raw records, and a separate
`cognify()` builds the graph `recall()` reads from.
```python theme={null}
# 1. Land the raw memories — zero LLM calls, nothing queryable yet.
await cognee.remember(
Mem0Source(MEM0_MEMORIES, mode="preserve"),
dataset_name=DATASET,
)
# 2. Run extraction to make them recallable (this one does call the LLM).
await cognee.cognify(datasets=[DATASET])
answer = await cognee.recall("What does Alex work on?", datasets=[DATASET])
```
Skip step 2 and `recall()` returns nothing for the imported memories. With
`re-derive` or `hybrid` the extraction is part of the import, so no extra
`cognify()` call is needed. The
[runnable tutorial](https://github.com/topoteretes/cognee/blob/dev/examples/demos/ingestion_and_migration/migrate_from_mem0/migrate_from_mem0.py)
imports the same export both ways so you can compare.
## Other sources you can import from
Every source has the same interface — construct it from a file path (or
in-memory data) and pass it to `cognee.remember()`.
<Note>
When you construct a source from a **file path**, the export file is read
through the same allowlist as ordinary ingestion. That allowlist is unset by
default, so an export anywhere on disk works out of the box. Only once you set
`COGNEE_ALLOWED_LOCAL_FILE_ROOTS` — as you should on a deployment reachable by
untrusted callers — must the export live under one of the listed roots; an
export outside them raises
`ValueError: Local file path is outside allowed roots.`
See [allowed local roots](/setup-configuration/security#local-file-system-access).
Passing already-parsed data instead of a path skips the check entirely.
</Note>
<AccordionGroup>
<Accordion title="LangMem (JSON memory exports)" icon="note-sticky">
`LangMemSource` reads a LangMem memory export and imports each item as an
atomic **memory** record. It accepts a file path, an already-parsed list, or a
dict wrapping the list under `memories`, `results`, `items`, or `data` — any
other shape raises `ValueError`.
Those aliases are scanned in that order, and only dict items count as records.
An alias that is present but empty does not shadow a populated one later in the
order, so `{"memories": [], "results": [...]}` imports the `results` records.
A wrapper whose recognized aliases are all empty imports zero records without
raising.
Per memory:
* **content** is the first string found among `content`, `text`, `memory`,
`data`, and `message`; items with none of those are skipped
* **scope** takes `user_id` (falling back to `namespace`), plus `agent_id`,
`session_id`, and `run_id` when present
* **categories** accepts either a single string or a list
* **timestamps** are read from `created_at` / `createdAt` / `timestamp` and
`updated_at` / `updatedAt`
* any **`metadata`** is carried over nested under `langmem_metadata`
Because LangMem memories are free-form text with no derived graph, it defaults
to `re-derive` mode.
```python theme={null}
from cognee.modules.migration.sources.langmem import LangMemSource
await cognee.remember(LangMemSource("langmem_export.json"), dataset_name="langmem_import")
```
</Accordion>
<Accordion title="Letta / MemGPT (.af agent files)" icon="robot">
`LettaSource` reads a Letta **Agent File** (`.af`, a JSON serialization of one
or more agents) and imports:
* **core memory blocks** → memory blocks in the graph (`COGXMemoryBlock`)
* **message history** → one conversation episode per agent (`COGXEpisode`)
* **archival memory** → one document per passage (`COGXDocument`)
The parser tolerates key-name differences across Letta versions. Per message:
* **text** is read from `content`, falling back to the `text` alias when
`content` is missing *or* explicitly `null` — Letta serializers that write
unset fields as `null` rather than omitting them import correctly. An
explicitly empty `content` (`""`) counts as a message with no text and does
not fall through to `text`
* either key may hold a plain string or a list of typed parts, of which only
the text parts are imported
* messages that end up with no text, and messages whose role is `system` or
`tool`, are skipped; if that leaves an agent with no messages, no
conversation episode is emitted for it
```python theme={null}
from cognee.migration import LettaSource
await cognee.remember(LettaSource("my_agent.af"), dataset_name="letta_import")
```
</Accordion>
<Accordion title="Zep / Graphiti (graph exports)" icon="share-nodes">
`ZepSource` reads a JSON export of a Zep or Graphiti knowledge graph and
imports:
* **episodes** (verbatim ingested content) → `COGXEpisode`
* **entity nodes** → `COGXEntity`
* **relation edges** ("facts") → `COGXFact`, including their bi-temporal
`valid_at` / `invalid_at` validity windows
All three record types resolve their scope `session_id` the same way: from
`group_id`, falling back to a `session_id` key when `group_id` is absent. When
a record carries both, `group_id` wins.
It defaults to `hybrid` mode because Zep/Graphiti keep both verbatim episodes
and a derived graph.
```python theme={null}
from cognee.migration import ZepSource, GraphitiSource
# Zep export
await cognee.remember(ZepSource("zep_export.json"), dataset_name="zep_import")
# OSS Graphiti export (same shape; produce the JSON from a Cypher dump of
# EntityNode / EpisodicNode / RELATES_TO records)
await cognee.remember(GraphitiSource("graphiti_export.json"), dataset_name="graphiti_import")
```
</Accordion>
<Accordion title="Another Cognee instance (COGX archive)" icon="box-archive">
`COGXArchiveSource` re-imports an archive produced by `cognee.export(...,
format="cogx")`. This is the restore half of backup/restore and the receiving
end of Cognee-to-Cognee migration. It defaults to `preserve` mode (zero-LLM),
because a Cognee archive already carries a fully extracted graph.
```python theme={null}
from cognee.migration import COGXArchiveSource
await cognee.remember(COGXArchiveSource("./my_cogx_archive"), dataset_name="restored")
```
**Graph-only restore.** Pass `index_vectors=False` to persist the archive's graph
without initializing or writing a vector engine. The import then skips the
first-run LLM and embedding connection checks too, so it needs no API key at all:
```python theme={null}
await cognee.remember(
COGXArchiveSource("./my_cogx_archive"),
dataset_name="restored",
index_vectors=False,
)
```
The trade-off is retrieval coverage: nothing is embedded, so only
vector-independent search reaches the restored data — `CHUNKS_LEXICAL` (keyword
/ BM25) and graph traversal work, while embedding-backed retrieval such as
`CHUNKS`, `RAG_COMPLETION`, and the graph completion types do not. To get them
later, re-run the same import without `index_vectors=False` — imports are
idempotent, so repeating one is safe. This is the mode
[`cognee-cli demo`](/cognee-cli/overview#try-the-demo-graph) uses to load its
bundled archive on a machine with no keys configured.
</Accordion>
</AccordionGroup>
### Runnable tutorial: Letta + Zep
A runnable walkthrough of the two accordions above ships with the repo:
[`examples/demos/ingestion_and_migration/migrate_from_letta_and_zep/migrate_from_letta_and_zep.py`](https://github.com/topoteretes/cognee/blob/dev/examples/demos/ingestion_and_migration/migrate_from_letta_and_zep/migrate_from_letta_and_zep.py).
It runs in three parts, each starting from `forget(everything=True)`:
1. **Letta** — imports a bundled sample agent file
([`sample_letta_dump.json`](https://github.com/topoteretes/cognee/blob/dev/examples/demos/ingestion_and_migration/migrate_from_letta_and_zep/data/sample_letta_dump.json):
one agent with two core memory blocks, a three-message history, and two
archival passages) in `re-derive` mode, then verifies it with two `recall()`
queries.
2. **Zep** — imports a bundled sample graph export
([`sample_zep_dump.json`](https://github.com/topoteretes/cognee/blob/dev/examples/demos/ingestion_and_migration/migrate_from_letta_and_zep/data/sample_zep_dump.json):
two episodes, four entities, and four facts, each carrying a `valid_at`
timestamp and none an `invalid_at` end — every sample fact is still valid)
in `hybrid` mode, then runs three `recall()` queries against it.
3. **Both together** — imports the same two dumps into one graph and queries
across them, so you can see a combined Letta + Zep memory answer questions
that draw on both sources.
It finishes with one more `forget(everything=True)`, so it leaves nothing behind
and re-runs start clean.
Because the dumps are bundled, you can run all three parts without a Letta or
Zep account — only `LLM_API_KEY` is needed (`re-derive`, `hybrid`, and `recall()`
all call the LLM).
```bash theme={null}
uv run python examples/demos/ingestion_and_migration/migrate_from_letta_and_zep/migrate_from_letta_and_zep.py
```
## Write your own source
If your memory tool isn't listed above, you can add it yourself. A source is a
small adapter class: subclass `MemorySource`, name the system it reads from,
and implement one async generator — `records()` — that yields COGX records.
Everything else (loading, modes, deduplication, graph storage) is handled by
the shared machinery, exactly as for the built-in sources.
Everything you need is importable from `cognee.migration`: the `MemorySource`
base class and the record models (`COGXDocument`, `COGXEpisode`, `COGXTurn`,
`COGXEntity`, `COGXFact`, `COGXMemory`, `COGXMemoryBlock`, `COGXScope`).
Here is a complete source for a hypothetical notes app that exports a JSON
list of `{"id", "text", "user", "tags"}` objects:
```python theme={null}
import json
from pathlib import Path
from typing import AsyncIterator
from cognee.migration import COGXMemory, COGXRecord, COGXScope, MemorySource
class AcmeNotesSource(MemorySource):
source_system = "acme_notes"
def __init__(self, export_path, mode: str = "re-derive"):
super().__init__(mode=mode) # validates mode: re-derive | preserve | hybrid
self._export_path = Path(export_path)
async def records(self) -> AsyncIterator[COGXRecord]:
notes = json.loads(self._export_path.read_text(encoding="utf-8"))
for note in notes:
yield COGXMemory(
external_system=self.source_system,
external_id=str(note["id"]),
content=note["text"],
categories=note.get("tags", []),
scope=COGXScope(user_id=note.get("user")),
)
```
That's the whole integration — it plugs into `cognee.remember()` like any
built-in source:
```python theme={null}
await cognee.remember(AcmeNotesSource("acme_export.json"), dataset_name="acme_notes")
```
### Choosing record kinds
Pick the record type that matches what the source holds; the
[COGX concept page](/core-concepts/further-concepts/cogx) describes each in
detail. What happens to a record depends on its kind and the import mode:
| Your source holds | Emit | Processed in |
| --------------------------------- | -------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Files, passages, standalone text | `COGXDocument` | `re-derive` / `hybrid` (cognified); stored raw in `preserve` |
| Conversations with turns | `COGXEpisode` (with `COGXTurn`s) | same as documents — rendered as a timestamped transcript |
| Short derived facts, notes | `COGXMemory` | same as documents |
| Named core-memory blocks | `COGXMemoryBlock` | same as documents |
| An already-extracted entity graph | `COGXEntity` + `COGXFact` | `preserve` / `hybrid` (mapped directly into the graph, zero LLM calls); in `re-derive` they're re-ingested as digest documents and re-extracted by cognify |
Nothing you emit is silently dropped except raw nodes and entities with no
`description`, both of which carry no standalone text for `re-derive` to
re-extract.
A `COGXFact` references its endpoints by `subject_ref` / `object_ref` — use the
`external_id` of an entity record you also emit, or a plain entity name. A
plain-name reference that doesn't match an emitted entity becomes a new entity
of that name, so a source can emit facts on their own. A reference that looks
like a UUID but matches no emitted record is skipped and logged, never turned
into an entity named by a UUID. A fact's `valid_at` / `invalid_at` validity
window is preserved as edge properties in the graph.
### Rules the loader relies on
* **Stable `external_id`s make re-import idempotent.** Each record's identity
in Cognee is derived deterministically from
`(external_system, external_id)`, so importing the same export twice
doesn't duplicate data. Use the source system's own ids; only fall back to
synthetic ids (e.g. an index) for records that genuinely have none.
* **`records()` should be re-callable.** The streaming `preserve`-mode import
passes over the records three times — once to store the raw content, then
twice for the graph (nodes first, then facts) — calling `records()` once per
pass. File-backed sources get this for free by re-reading the file. If your
source is a one-shot cursor (e.g. a live API stream), set the class attribute
`replayable = False` to make the loader buffer records instead.
* **Preserve scope and timestamps.** Fill `COGXScope`
(`user_id`/`agent_id`/`session_id`/`run_id`) and `created_at`/`updated_at`
where the source has them — ownership and time information survive the
migration only if the source carries them across. Anything that doesn't fit
a typed field can go in the record's free-form `metadata` dict.
* **Pick a sensible default mode.** Match the [import mode](#import-modes) to
whether your source carries a pre-built graph; callers can always override.
For a fuller reference implementation, read the built-in sources in
[`cognee/modules/migration/sources/`](https://github.com/topoteretes/cognee/tree/dev/cognee/modules/migration/sources)
— `mem0.py` is the smallest, and `zep.py` shows entity/fact emission. If your
source would be useful to others, PRs adding it there (exported in the
package's `__init__.py`, with a sample-export test) are welcome.
## Export: Cognee → COGX
Migration runs both ways. `cognee.export()` writes a dataset's graph to a
portable COGX archive that you can back up, move to another Cognee instance, or
re-import later with `COGXArchiveSource`:
```python theme={null}
import cognee
# Write a COGX archive directory
await cognee.export(dataset="mem0_import", format="cogx", destination="./my_cogx_archive")
# ...later, on any Cognee instance:
from cognee.migration import COGXArchiveSource
await cognee.remember(COGXArchiveSource("./my_cogx_archive"), dataset_name="restored")
```
## Using real data
The Mem0 example above uses an inline sample. Here's how to point any source at
real data.
**From a provider's client (live API).** Fetch with the provider's own SDK and
pass the response straight in — sources accept already-parsed Python lists and
dicts, not just file paths:
```python theme={null}
from mem0 import MemoryClient
from cognee.migration import Mem0Source
client = MemoryClient(api_key="...")
memories = client.get_all(user_id="alex") # a list (or {"results": [...]})
await cognee.remember(Mem0Source(memories), dataset_name="mem0_import")
```
`Mem0Source` scans a wrapper dict for `results`, `memories`, then `items`, in
that order, counting only dict items as records; an alias that is present but
empty does not shadow a populated one later in the order, so
`{"results": [], "memories": [...]}` imports the `memories` records. A wrapper
whose recognized aliases are all empty imports zero records without raising,
and any other shape raises `ValueError`.
**From an exported file.** Point the source at the export on disk:
```python theme={null}
await cognee.remember(Mem0Source("mem0_export.json"), dataset_name="mem0_import")
```
## Next steps
<CardGroup>
<Card title="remember()" icon="brain" href="/core-concepts/main-operations/remember">
How import and ingestion work under the hood
</Card>
<Card title="recall()" icon="magnifying-glass" href="/core-concepts/main-operations/recall">
Query your migrated memory
</Card>
<Card title="COGX Exchange Format" icon="arrows-rotate" href="/core-concepts/further-concepts/cogx">
The portable format behind every import and export
</Card>
<Card title="Configuration" icon="gear" href="/setup-configuration/overview">
Configure LLM, embedding, and storage backends
</Card>
</CardGroup>
# Sharing Memory Across Users and Teams
Source: https://docs.cognee.ai/examples/multi-tenant-access-control
Give every user their own dataset, then open one up — first to a single colleague with an ACL grant, then to an entire role inside a tenant
Two people ingest documents into the same cognee deployment, and neither should see the other's data by default — until one of them asks for access, and later a whole research team needs it. That is the sequence every multi-user deployment runs into the first time someone says "can you share that dataset with me?"
## What You'll Build
Three users, two documents, and one instance with access control on. `user_1` remembers a PDF into an `AI` dataset while `user_2` remembers a text passage into a `QUANTUM` dataset, and the script then works through what `user_1` can and cannot do with someone else's dataset: reads and writes are refused, a read grant from the owner turns the refusal into an answer, and a `CogneeLab` tenant with a `Researcher` role finally lets `user_3` read a tenant-owned dataset on the strength of role membership alone. Every refusal along the way is a real `PermissionDeniedError` caught and printed, so a run reads as a transcript of the permission system making decisions.
The complete runnable script is
[`examples/demos/permissions/user_permissions_and_access_control_example.py`](https://github.com/topoteretes/cognee/blob/main/examples/demos/permissions/user_permissions_and_access_control_example.py) —
this page walks through its key moments rather than reproducing it.
## Features in Play
* [Permission Snippets](/guides/permission-snippets) — the tenant and role calls in short copy-paste form, with the four permission names spelled out; it grants with the unchecked `give_permission_on_dataset` rather than the authorizing wrapper used here
* [ACL](/core-concepts/multi-user-mode/permissions-system/acl) — `authorized_give_permission_on_datasets` writes the rows that decide each read in this demo, and raises `PermissionDeniedError` when there is no row to match
* [Tenants](/core-concepts/multi-user-mode/permissions-system/tenants) — the `CogneeLab` organization, and the active-tenant context that decides which datasets a grant may target
* [Roles](/core-concepts/multi-user-mode/permissions-system/roles) — the `Researcher` role, so access is granted once and every member inherits it
* [Remember](/core-concepts/main-operations/remember) — each ingestion call runs as a `user` and makes that user the sole owner of the dataset it creates
* [Recall](/core-concepts/main-operations/recall) — every read names both a `user` and explicit `dataset_ids`, and returns results only when an ACL allows it
## What to Expect
The excerpts below come from one real run, trimmed of most log lines. Every ingestion and every recall is a live LLM call, so the wording of the answers varies from run to run; the permission decisions do not.
**The instance is emptied and each owner is registered in turn.** `Data reset complete.` follows the two prunes, `Relational migrations applied (target head).` is `setup()` rebuilding the user-management tables, and each `create_user` call prints the address it registers before the new principal's id is logged.
```text theme={null}
Resetting cognee data...
...
Data reset complete.
...
Relational migrations applied (target head).
Creating user_1: user_1@example.com
User 000f9b41-bf09-4f1d-89f8-18442c626c29 has registered.
...
Creating user_2: user_2@example.com
User 494df8a4-d085-4545-8644-87e36b5329a9 has registered.
```
**Reading your own dataset needs no grant.** `user_1` recalls from `ai_dataset_id` and a `graph_completion` result comes straight back, summarizing the PDF it ingested a moment earlier — the owner holds every permission on the dataset its `remember()` created.
```text theme={null}
Recall results as user_1 on dataset owned by user_1:
kind='graph_completion' search_type='GRAPH_COMPLETION' text='A multi-page overview of artificial intelligence: definitions and types (narrow/weak, AGI/ASI), machine learning and deep learning ...
```
**The two refusals are the default posture.** Pointing the same recall at `user_2`'s dataset raises `PermissionDeniedError`, and so does a `remember()` into it. The `403` lines are cognee logging the exception before the script catches it and prints its own line, so expect them on a healthy run — this is what ownership alone gets everyone else: nothing.
```text theme={null}
Recall result as user_1 on the dataset owned by user_2:
...
2026-09-11T17:27:27.876388 [error ] PermissionDeniedError raised (Status code: 403) [cognee.shared.logging_utils]
2026-09-11T17:27:27.892307 [error ] PermissionDeniedError raised (Status code: 403) [cognee.shared.logging_utils]
User: <cognee.modules.users.models.User.User object at 0x124759010> does not have permission to read from dataset: QUANTUM
Attempting to remember new data as user_1 to dataset owned by user_2:
User: <cognee.modules.users.models.User.User object at 0x124759010> does not have permission to write to dataset: QUANTUM
```
**One `"read"` grant turns the refusal into a result.** The recall that failed two steps earlier is replayed unchanged, and this time `recall: 1 results` is logged and a completion is returned instead of a 403. The ACL row is the only thing that changed between the two attempts.
```text theme={null}
Operation started as user_2 to give read permission to user_1 for the dataset owned by user_2
Recall result as user_1 on the dataset owned by user_2:
...
2026-09-11T17:27:44.407459 [info ] recall: 1 results across sources=['graph'] (session=-) [recall]
kind='graph_completion' search_type='GRAPH_COMPLETION' ...
```
**The tenant is built in a forced order, and the first role grant is refused.** Each line is one constraint being satisfied: tenant, active tenant, role, new user, tenant membership, role membership, and finally `user_3` selecting `CogneeLab` as its own active tenant. Read the last two lines closely — `user_2` owns both `CogneeLab` and the `QUANTUM` dataset, but that dataset belongs to `user_2` personally, so it is out of scope for a grant made from inside the tenant.
```text theme={null}
User 2 is creating CogneeLab tenant/organization
User 2 is selecting CogneeLab tenant/organization as active tenant
User 2 is creating Researcher role
Creating user_3: user_3@example.com
Operation started as user_2 to add user_3 to CogneeLab tenant/organization
Operation started by user_2, as tenant owner, to add user_3 to Researcher role inside the tenant/organization
Operation as user_3 to select CogneeLab tenant/organization as active tenant
...
Operation started as user_2, with CogneeLab as its active tenant, to give read permission to Researcher role for the dataset QUANTUM owned by user_2
User 2 could not give permission to the role as the QUANTUM dataset is not part of the CogneeLab tenant
```
**The grant lands once the dataset lives in the tenant, and `user_3` reads it.** No grant anywhere in the run names `user_3`: the final read succeeds purely through membership of `Researcher`, which is the reason to grant at the role level at all.
```text theme={null}
Operation started as user_2, with CogneeLab as its active tenant, to give read permission to Researcher role for the dataset QUANTUM owned by the CogneeLab tenant
Recall result as user_3 on the QUANTUM dataset owned by the CogneeLab organization:
...
2026-09-11T17:28:40.308421 [info ] recall: 1 results across sources=['graph'] (session=-) [recall]
kind='graph_completion' search_type='GRAPH_COMPLETION' ...
```
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — the ingestion calls and every `GRAPH_COMPLETION` recall in the script make live LLM calls
* Leave multi-user mode on: the whole demo depends on `ENABLE_BACKEND_ACCESS_CONTROL`, which is enabled by default, and on graph and vector backends that support per-dataset isolation — see [Permissions Setup](/setup-configuration/permissions) and [Multi-User Mode Overview](/core-concepts/multi-user-mode/multi-user-mode-overview)
* Run it from a checkout of the cognee repo: the script reads `artificial_intelligence.pdf` from the sibling `data/` folder next to it
* Point it at a scratch instance: it opens with `prune_data()` and `prune_system(metadata=True)`, and the second of those drops the relational database — users, tenants, and ACLs included — rather than deleting one dataset
## How It Works
### Stage 1: Create the User-Management Tables
```python theme={null}
# Set up the necessary databases and tables for user management.
await setup()
```
After the two prunes have emptied the instance, `setup()` recreates the relational tables the permission system lives in: principals, tenants, roles, and the ACL rows that join them to datasets. Nothing later in the script — not even creating the first user — works without it.
### Stage 2: Give Each Owner Their Own Dataset
```python theme={null}
print("Creating user_1: user_1@example.com")
user_1 = await create_user("user_1@example.com", "example")
ai_remember_result = await cognee.remember(
[explanation_file_path],
dataset_name="AI",
user=user_1,
self_improvement=False,
)
```
Passing `user=user_1` is what makes this ingestion belong to someone: the `AI` dataset is created under `user_1`, who is initially the only principal with any permission on it. `user_2` is created the same way a few lines down and remembers an inline passage about quantum computing into a `QUANTUM` dataset, which gives the rest of the script two datasets with two different owners to negotiate over.
### Stage 3: Address Datasets by ID, Not Name
```python theme={null}
def get_dataset_id(remember_result):
"""Extract dataset_id from remember output."""
return UUID(remember_result.dataset_id)
# Get dataset IDs from remember results
# Note: When we want to work with datasets from other users (recall, remember, and etc.) we must supply dataset
# information through dataset_ids; using dataset names only looks for datasets owned by current user
ai_dataset_id = get_dataset_id(ai_remember_result)
quantum_dataset_id = get_dataset_id(quantum_remember_result)
```
Each `remember()` result carries the id of the dataset it wrote to, and those ids are how the script refers to datasets from here on. The comment names the reason: a dataset *name* is only ever resolved among the datasets the calling user owns, so `dataset_name="QUANTUM"` as `user_1` would look for a second, different dataset instead of reaching `user_2`'s. Ids are used for the owner's own data too: the first recall passes `dataset_ids=[ai_dataset_id]` and returns an answer immediately.
### Stage 4: Watch the Default Denial
```python theme={null}
# But user_1 cant read the dataset owned by user_2 (QUANTUM dataset)
print("\nRecall result as user_1 on the dataset owned by user_2:")
try:
await cognee.recall(
query_type=SearchType.GRAPH_COMPLETION,
query_text="What is in the document?",
user=user_1,
dataset_ids=[quantum_dataset_id],
)
except PermissionDeniedError:
print(f"User: {user_1} does not have permission to read from dataset: QUANTUM")
```
The same recall that just worked against `user_1`'s own dataset raises `PermissionDeniedError` when it points at `user_2`'s, because ownership grants nothing to anyone else. Writes are refused on the same grounds: the block that follows sends a `remember()` at `dataset_id=quantum_dataset_id` as `user_1` and catches the identical error, since `user_1` holds neither read nor write on that dataset.
### Stage 5: Grant One User Read Access
```python theme={null}
# We've shown that user_1 can't interact with the dataset from user_2
# Now have user_2 give proper permission to user_1 to read QUANTUM dataset
# Note: supported permission types are "read", "write", "delete" and "share"
print(
"\nOperation started as user_2 to give read permission to user_1 for the dataset owned by user_2"
)
await authorized_give_permission_on_datasets(
user_1.id,
[quantum_dataset_id],
"read",
user_2.id,
)
# Now user_1 can read from quantum dataset after proper permissions have been assigned by the QUANTUM dataset owner.
print("\nRecall result as user_1 on the dataset owned by user_2:")
recall_results = await cognee.recall(
query_type=SearchType.GRAPH_COMPLETION,
query_text="What is in the document?",
user=user_1,
dataset_ids=[quantum_dataset_id],
)
for result in recall_results:
print(f"{result}\n")
```
The grant reads as who gets it, on what, which permission, and who is authorizing — the last argument is the acting user, and the call is refused unless that user is allowed to share the dataset. One `"read"` row later, the recall from Stage 4 is replayed unchanged and returns an answer. Only reads are open: remembering into `QUANTUM` as `user_1` would still fail, because `"write"` was never granted.
### Stage 6: Set Up a Tenant and a Role
```python theme={null}
# Users can also be added to Roles and Tenants and then permission can be assigned on a Role/Tenant level as well
# To create a Role a user first must be an owner of a Tenant
print("User 2 is creating CogneeLab tenant/organization")
tenant_id = await create_tenant("CogneeLab", user_2.id)
print("User 2 is selecting CogneeLab tenant/organization as active tenant")
await select_tenant(user_id=user_2.id, tenant_id=tenant_id)
print("\nUser 2 is creating Researcher role")
role_id = await create_role(role_name="Researcher", owner_id=user_2.id)
print("\nCreating user_3: user_3@example.com")
user_3 = await create_user("user_3@example.com", "example")
# To add a user to a role he must be part of the same tenant/organization
print("\nOperation started as user_2 to add user_3 to CogneeLab tenant/organization")
await add_user_to_tenant(user_id=user_3.id, tenant_id=tenant_id, owner_id=user_2.id)
print(
"\nOperation started by user_2, as tenant owner, to add user_3 to Researcher role inside the tenant/organization"
)
await add_user_to_role(user_id=user_3.id, role_id=role_id, owner_id=user_2.id)
```
Granting per user does not scale past a handful of colleagues, so `user_2` builds an organization instead: a `CogneeLab` tenant, selected as the active tenant, and a `Researcher` role inside it. The order is forced by the model — a role can only be created by a tenant owner, and a user can only join a role once they are in the same tenant, which is why `user_3` is added to `CogneeLab` before being added to `Researcher`. `user_3` then selects `CogneeLab` as its own active tenant.
### Stage 7: Hit the Tenant-Scoping Constraint
```python theme={null}
# Even though the dataset owner is user_2, the dataset doesn't belong to the tenant/organization CogneeLab.
# So we can't assign permissions to it when we're acting in the CogneeLab tenant.
try:
await authorized_give_permission_on_datasets(
role_id,
[quantum_dataset_id],
"read",
user_2.id,
)
except PermissionDeniedError:
print(
"User 2 could not give permission to the role as the QUANTUM dataset is not part of the CogneeLab tenant"
)
```
Handing the role the *original* `QUANTUM` dataset fails, and this refusal is the one worth understanding: `user_2` owns that dataset and owns the tenant, but the dataset was created before `CogneeLab` existed and therefore belongs to `user_2` personally. A grant is evaluated inside the acting user's active tenant, and a personal dataset is out of that scope — so the answer is the same `PermissionDeniedError` as Stage 4, for a completely different reason.
### Stage 8: Own the Dataset in the Tenant, Then Grant to the Role
```python theme={null}
# Note: We need to update user_2 from the database to refresh its tenant context changes
user_2 = await get_user(user_2.id)
quantum_cognee_lab_remember_result = await cognee.remember(
[text],
dataset_name="QUANTUM_COGNEE_LAB",
user=user_2,
self_improvement=False,
)
# The recreated Quantum dataset will now have a different dataset_id as it's a new dataset in a different organization
quantum_cognee_lab_dataset_id = get_dataset_id(quantum_cognee_lab_remember_result)
print(
"\nOperation started as user_2, with CogneeLab as its active tenant, to give read permission to Researcher role for the dataset QUANTUM owned by the CogneeLab tenant"
)
await authorized_give_permission_on_datasets(
role_id,
[quantum_cognee_lab_dataset_id],
"read",
user_2.id,
)
```
The fix is to put the data where the role can be given access to it: `user_2` is re-read from the database so the in-memory object carries the new active tenant, and the same passage is remembered again as `QUANTUM_COGNEE_LAB` — a distinct dataset with its own id, this time owned inside `CogneeLab`. The personal `QUANTUM` dataset is untouched and still reachable by selecting `user_2`'s personal tenant. Granting `"read"` to `role_id` now succeeds, and the closing recall proves the payoff: `user_3`, who was never named in any grant, reads the tenant dataset as a member of `Researcher`.
## Run It
```bash theme={null}
uv run python examples/demos/permissions/user_permissions_and_access_control_example.py
```
The run is a single pass with no arguments and no interactive steps; [What to Expect](#what-to-expect) above walks through its output.
<Columns>
<Card title="Permission Snippets" icon="code" href="/guides/permission-snippets">
Every call from this walkthrough as a standalone snippet, plus the permission names.
</Card>
<Card title="Permissions System Overview" icon="shield" href="/core-concepts/multi-user-mode/permissions-system/overview">
How principals, datasets, and ACLs fit together behind these calls.
</Card>
<Card title="Multi-User Mode Overview" icon="users" href="/core-concepts/multi-user-mode/multi-user-mode-overview">
What access control changes about storage and isolation across a deployment.
</Card>
<Card title="Permissions Setup" icon="shield" href="/setup-configuration/permissions">
The environment variables and backend support the demo assumes.
</Card>
</Columns>
# Org Chart from JSON Exports
Source: https://docs.cognee.ai/examples/org-chart-from-json
Turn two flat JSON exports of companies and people into a connected org-chart graph, with deterministic node IDs doing the deduplication
You have your org data as plain exports — one JSON file listing companies and their departments, another listing people and where they work — and you want it in memory as a connected graph of companies, departments, and employees, without an LLM re-deriving structure you already know.
## What You'll Build
Two JSON files — `companies.json` and `people.json` — are loaded and mapped onto typed `DataPoint` classes, then written straight into the graph and vector stores by a two-task custom pipeline: no chunking, no extraction, no cognify pass. What comes out is an org chart you can open in a browser: `Company` nodes linked to their `Department` nodes, each department linked to the `Person` nodes that work in it, with every name embedded for semantic search. Node IDs are derived from the data itself, so the same department name reaching the pipeline twice lands on one node rather than two.
The demo ships as two variants of the same scenario, so this page walks through both: a high-level script that lets `identity_fields` deduplicate for it, and a low-level script that assembles and deduplicates the graph by hand.
The complete runnable scripts are in
[`examples/demos/custom_pipelines/organizational_hierarchy`](https://github.com/topoteretes/cognee/tree/main/examples/demos/custom_pipelines/organizational_hierarchy) —
this page walks through their key moments rather than reproducing them.
## Features in Play
* [DataPoints](/core-concepts/building-blocks/datapoints) — `Person`, `Department`, `Company`, and `CompanyType` are the node types; `index_fields` decides what gets embedded and `identity_fields` derives the deterministic ID that collapses repeats
* [Custom Data Models](/guides/custom-data-models) — `add_data_points` persists the typed objects and their nesting as nodes and edges, with no LLM in the path
* [Custom Tasks and Pipelines](/guides/custom-tasks-pipelines) — two `Task` objects run by `run_tasks` against a dataset: one maps JSON to DataPoints, one stores them
* [Graph Visualization](/guides/graph-visualization) — both scripts close by rendering the org chart to an HTML file you can open
* [Search](/core-concepts/main-operations/legacy-operations/search) — the low-level variant finishes with a `GRAPH_COMPLETION` query, proving the hand-built nodes answer questions like any other memory
## How It Looks
This is the graph the high-level script writes, loaded from a real run of it. Drag a node to pull it out of the tangle, scroll to zoom, and hover one to fade out everything it does not touch — hovering a department is the quickest way to see exactly who works in it.
<Frame>
<CogneeGraph label="Org chart knowledge graph with 27 nodes and 26 edges" />
</Frame>
Every node and edge here comes from the run's `visualize_graph` export, so the shape is exactly what lands in your graph store: `Research & Development` sits on GreenFuture with no one attached to it, because the company declares the department but no one in `people.json` works there.
## What to Expect
The lines below come from one real run of the high-level script, trimmed. Nothing in this demo calls an LLM, so the node and edge counts are identical on every run — only the UUIDs, timestamps, and paths change. A first run also prints a long block of relational migrations as cognee builds its tables; that block is setup, not part of the demo.
**The pipeline starts with both JSON files already in hand.** `build_lightweight_data_object` has wrapped the two files into a single `LightweightData` DataPoint, so the payload printed at the start of the run is the whole input — five companies and ten people — before any task has touched it.
```text theme={null}
status='PipelineRunStarted' pipeline_run_id=UUID('40da9a6f-3948-43be-a90f-e551570c40dd') dataset_id=UUID('94add405-52f0-5e30-871b-b2e8eeda31c7') dataset_name='test_dataset' payload=[LightweightData(id=UUID('c22f2fa6-5eee-5a20-a8d5-f3a1d10dbbbd'), ...
```
**The two tasks finish in about three seconds.** `ingest_files` maps the payload and `add_data_points` writes the result. There is no extraction step between them, which is why the whole pipeline costs less than a single LLM call would.
```text theme={null}
2026-09-11T15:24:05.784975 [info ] Function task started: `ingest_files` [run_tasks_base]
2026-09-11T15:24:05.792215 [info ] Coroutine task started: `add_data_points` [run_tasks_base]
...
2026-09-11T15:24:08.755657 [info ] Coroutine task completed: `add_data_points` [run_tasks_base]
2026-09-11T15:24:08.762174 [info ] Function task completed: `ingest_files` [run_tasks_base]
2026-09-11T15:24:08.768537 [info ] Pipeline run completed: `07f15429-bcc9-56e7-8058-6ff9d3fd6372` [run_tasks_with_telemetry()]
```
**The graph that lands is 27 nodes and 26 edges, and the arithmetic is what tells you the mapping was right.** Five `Company` nodes, eleven `Department` nodes, ten `Person` nodes, and the one `CompanyType` all five companies point at make 27. Eleven company-to-department edges, ten department-to-employee edges, and five `is_type` edges make 26. Had a department name landed as two nodes, both numbers would be higher.
```text theme={null}
2026-09-11T15:24:08.791366 [info ] Retrieved 27 nodes and 26 edges in 0.00 seconds [cognee.shared.logging_utils]
2026-09-11T15:24:08.797151 [info ] Neighborhood retrieval (2-hop): 27 nodes and 26 edges in 0.01s [cognee.shared.logging_utils]
```
**One warning is expected and does not mean the run failed.** `add_data_points` did embed every `index_fields` value — the `Person_name`, `Department_name`, and `Company_name` collections are written and searchable. The warning comes from the renderer's semantic-map layer, which only recognizes cognee's built-in node types and so has nothing to plot for custom DataPoints. The graph view itself is complete.
```text theme={null}
2026-09-11T15:24:08.798832 [info ] fetch_node_embeddings: resolved 0/27 node embeddings across 0 collection(s) [embedding_join]
2026-09-11T15:24:08.798890 [warning ] fetch_node_embeddings: no embeddings resolved — the semantic map will be empty. Missing collections: none. Unmapped node types: ['Department', 'Company', 'Person', 'CompanyType']. [embedding_join]
```
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — `add_data_points` embeds every `index_fields` value, so an embedding provider is required even though neither script extracts anything with an LLM; the low-level variant's closing query also needs a completion model
* Run both scripts from a checkout of the cognee repo: they read `companies.json` and `people.json` from the `data/` folder next to them, and write their HTML renders into a sibling `.artifacts/` folder
* The high-level script starts with `prune.prune_data()` and `prune.prune_system(metadata=True)`, which wipes the configured instance — point it at a scratch instance rather than memory you want to keep. The low-level script pins its system root to a `.cognee_system` folder beside itself before pruning, so it stays self-contained
## How It Works
### Stage 1: Model the Org Chart as Deduplicating DataPoints
Source: `examples/demos/custom_pipelines/organizational_hierarchy/organizational_hierarchy_pipeline_example.py`
```python theme={null}
class Person(DataPoint):
name: str
# "index_fields": fields to embed for vector search
# "identity_fields": fields used to generate deterministic IDs (deduplication)
metadata: dict = {"index_fields": ["name"], "identity_fields": ["name"]}
class Department(DataPoint):
name: str
employees: list[Person]
metadata: dict = {"index_fields": ["name"], "identity_fields": ["name"]}
```
These classes are the entire schema of the resulting graph — `Company` and `CompanyType` follow the same pattern below them. Two things happen per class: `index_fields` names the field that gets embedded, so the graph is searchable by name; `identity_fields` makes the node's UUID a deterministic function of that name, so building `Person(name="John Doe")` twice lands on one node instead of two. The `employees` field is what makes this a graph rather than two tables — a list of `Person` objects on a `Department` becomes edges from that department to those people.
### Stage 2: Map the JSON Rows onto Nested DataPoints
Source: `examples/demos/custom_pipelines/organizational_hierarchy/organizational_hierarchy_pipeline_example.py`
```python theme={null}
def ingest_files(data: list[Any]) -> list[Company]:
# With identity_fields, DataPoints with the same name automatically get the same UUID.
# No manual dict-based deduplication needed — just create instances freely.
all_companies: list[Company] = []
# Single CompanyType node shared across all data items (deterministic ID via identity_fields)
company_type = CompanyType()
for data_item in data:
people = data_item.people
companies = data_item.companies
# Build departments with their employees
dept_employees: dict[str, list[Person]] = {}
for person in people:
dept_name = person["department"]
if dept_name not in dept_employees:
dept_employees[dept_name] = []
dept_employees[dept_name].append(Person(name=person["name"]))
departments = {
name: Department(name=name, employees=employees)
for name, employees in dept_employees.items()
}
for company in companies:
company_departments = [
departments.get(dept_name, Department(name=dept_name, employees=[]))
for dept_name in company["departments"]
]
all_companies.append(
Company(name=company["name"], departments=company_departments, is_type=company_type)
)
return all_companies
```
This is the first pipeline task, and it is ordinary Python: group people by their `department` field, build one `Department` per group, then hang the departments each company declares off a `Company`. Nothing here checks whether a node already exists. A company that declares a department nobody works in still gets a `Department` node — created on the spot with no employees — and if another company declares the same one, `identity_fields` gives both objects the same UUID, so the two collapse into a single shared node when they are stored. The function returns only the companies; the departments and people reach the graph because they hang off them.
### Stage 3: Run the Two-Task Pipeline
Source: `examples/demos/custom_pipelines/organizational_hierarchy/organizational_hierarchy_pipeline_example.py`
```python theme={null}
# Run tasks expects a list of data even if it is just one document
data = [{"companies": companies, "people": people}]
pipeline = run_tasks(
[Task(ingest_files), Task(add_data_points)],
dataset_id=datasets[0].id,
data=build_lightweight_data_object(data),
incremental_loading=False,
)
```
`main()` prunes, calls `setup()` to create the relational tables, resolves the default user, and creates a `test_dataset` with `load_or_create_datasets` before this point — the dataset is what the pipeline run is tracked against. `run_tasks` then chains the two tasks: `ingest_files` returns `Company` objects and `add_data_points` writes them, their nested departments and people, and the edges between them into the graph and vector stores. `build_lightweight_data_object` wraps the raw dict in a `LightweightData` DataPoint with a `uuid5` ID, because the pipeline expects DataPoints rather than plain dicts. The run ends with `visualize_graph`, which writes the org chart to `.artifacts/organizational_hierarchy_pipeline_example.html`.
### Stage 4: Model the Same Graph Without identity\_fields
Source: `examples/demos/custom_pipelines/organizational_hierarchy/organizational_hierarchy_pipeline_low_level_example.py`
```python theme={null}
class Person(DataPoint):
"""Represent a person."""
name: str
metadata: dict = {"index_fields": ["name"]}
class Department(DataPoint):
"""Represent a department."""
name: str
employees: list[Person]
metadata: dict = {"index_fields": ["name"]}
```
The low-level variant declares the same four node types, with one difference that shapes the rest of the script: no `identity_fields`. Names are still embedded, but each constructed instance gets its own random UUID, so two `Person(name="John Doe")` objects are two nodes. Everything the next stage does by hand exists to prevent that.
### Stage 5: Deduplicate People and Departments by Hand
Source: `examples/demos/custom_pipelines/organizational_hierarchy/organizational_hierarchy_pipeline_low_level_example.py`
```python theme={null}
def build_people_nodes(people: Iterable[Mapping[str, Any]]) -> dict:
"""Build person nodes keyed by name."""
nodes = {p["name"]: Person(name=p["name"]) for p in people if p.get("name")}
return nodes
def group_people_by_department(people: Iterable[Mapping[str, Any]]) -> dict:
"""Group person names by department."""
groups = defaultdict(list)
for person in people:
name = person.get("name")
if not name:
continue
dept = person.get("department", "Unknown")
groups[dept].append(name)
return groups
```
Every node here is created once and then referenced by name from a dict — that keying, not the model, is what makes the graph deduplicate. `build_companies` chains the helpers in order: collect people and companies from the payloads, build the person and department node dicts, then `attach_departments_to_companies` and `attach_employees_to_departments` wire the objects together by looking each name up. `remove_duplicates_preserve_order` is needed because the same person can be listed twice for a department, and `collect_declared_departments` unions the departments people work in with the ones companies declare, so a department with no employees still gets a node.
### Stage 6: Store the Graph, Index Its Edges, and Query It
Source: `examples/demos/custom_pipelines/organizational_hierarchy/organizational_hierarchy_pipeline_low_level_example.py`
```python theme={null}
# Build and run pipeline
tasks = [Task(ingest_payloads), Task(add_data_points)]
pipeline = run_tasks(tasks, dataset_id, None, user, "demo_pipeline")
async for status in pipeline:
logger.info("Pipeline status: %s", status)
# Post-process: index graph edges and visualize
await index_graph_edges()
await visualize_graph(str(GRAPH_HTML))
# Run query against graph
completion = await search(
query_text="Who works for GreenFuture Solutions?",
query_type=SearchType.GRAPH_COMPLETION,
)
```
The pipeline is the same two-task shape, run against a `demo_dataset` with `None` as the data argument — `ingest_payloads` falls back to loading the bundled JSON files itself. After the nodes land, `index_graph_edges` embeds the relationship names so edges are retrievable and not just traversable, and `visualize_graph` writes the HTML render. The closing `GRAPH_COMPLETION` search is the point of the whole exercise: nothing about the query knows the graph was assembled by hand, so asking who works for GreenFuture Solutions walks company → department → employee edges and answers from them.
## Run It
```bash theme={null}
uv run python examples/demos/custom_pipelines/organizational_hierarchy/organizational_hierarchy_pipeline_example.py
uv run python examples/demos/custom_pipelines/organizational_hierarchy/organizational_hierarchy_pipeline_low_level_example.py
```
The high-level run is the one walked through above; it closes by logging the path of the render it wrote to `.artifacts/organizational_hierarchy_pipeline_example.html`, next to the script — open that file to see companies linked to departments linked to people. The low-level run logs its pipeline statuses through `logging` as `Pipeline status: ...` lines, writes `.artifacts/graph_visualization.html`, and ends by logging `Graph completion result: ...` with the LLM's answer to "Who works for GreenFuture Solutions?" — the proof that the hand-built graph is queryable.
## Choosing Between the Two Variants
Start from the high-level script. Declaring `identity_fields` moves deduplication into the node's identity, so the mapping code is free to construct objects wherever it is convenient and the store collapses the repeats — the same property that makes re-running the import idempotent rather than doubling the graph. The low-level script is worth reading when you need to see what that buys you: it is the same org chart, with keyed dicts, ordered-unique helpers, and attach passes standing in for the deterministic IDs. Reach for its shape when identity is not a field you have — when nodes are distinguished by something you compute rather than something in the data — and for `index_graph_edges`, which either variant can call once the nodes are stored.
<Columns>
<Card title="DataPoints" icon="circle" href="/core-concepts/building-blocks/datapoints">
How `index_fields`, `identity_fields`, and nested models shape the graph a DataPoint becomes.
</Card>
<Card title="Custom Data Models" icon="circle-stop" href="/guides/custom-data-models">
More on modeling your own node types and storing them with `add_data_points`.
</Card>
<Card title="Custom Tasks and Pipelines" icon="workflow" href="/guides/custom-tasks-pipelines">
Writing your own tasks and running them as a pipeline, step by step.
</Card>
<Card title="Reading the Visualization" icon="eye" href="/guides/reading-the-visualization">
What each tab of the rendered org chart shows, and which one to reach for.
</Card>
</Columns>
# Organizing Your Data
Source: https://docs.cognee.ai/examples/organizing-your-data
Ingest the same mixed corpus three ways — one pile, node sets, separate datasets — and watch how each layout changes what recall answers
Your assistant's memory holds both your product documentation and your sales-call transcripts, and its answers have started blending the two — a question about API rate limits comes back quoting whatever a rep promised a customer last quarter. The memory is doing its job by linking related facts; the problem is that everything lives in one undifferentiated pile.
## What You'll Build
Four small files — two technical documents (an API reference and an architecture guide) and two sales-call transcripts (where a rep promises "unlimited requests" and "instant sync") — are ingested three ways: everything in one dataset, one dataset with node-set tags, and two walled-off datasets with node sets inside each. The same questions run under every layout, and each recall prints as a comparable `QUERY` / `SCOPE` / `ANSWER` block, so you can see exactly what each layout changes: the unfiltered pile blends promise with spec, node sets scope a query to one group while still answering cross-domain questions, and datasets make leakage impossible. The run ends with the rule of thumb the demo exists to teach: node sets are tags, datasets are walls.
The complete runnable demo — the script plus the four data files it reads — is
[`examples/demos/organizing_your_data/`](https://github.com/topoteretes/cognee/tree/dev/examples/demos/organizing_your_data) —
this page walks through its key moments rather than reproducing it.
The three sections below cover one layout each: the calls that build it, the output those calls produce in a real run, and what the layout is good and bad at. Output excerpts are verbatim from one run, trimmed with `...`. Every `ANSWER` is a live LLM call, so the wording varies from run to run; each `ANSWER` line is prefixed with the dataset it came from, so the same question can be compared across layouts by eye.
## Features in Play
* [NodeSets](/core-concepts/further-concepts/node-sets) — tags each source with soft, overlappable groups at write time; `node_name` scopes retrieval to a tag at read time
* [Datasets](/core-concepts/further-concepts/datasets) — the hard boundary with its own permissions and storage that keeps sales transcripts out of technical answers entirely
* [Remember](/core-concepts/main-operations/remember) — ingests the four bundled files under each layout's labels
* [Recall](/core-concepts/main-operations/recall) — runs the same questions under every layout, scoped by `datasets` and `node_name`
## Everything in One Dataset
The baseline, and the layout that produces the complaint the page opens with. All four files land in the default dataset with no labels, and one recall runs with no filter.
```python theme={null}
await cognee.remember(
TECH_DOCS + [SALES_CALL_INITECH, SALES_CALL_HOOLI], self_improvement=False
)
answer = await cognee.recall(RATE_LIMIT_QUERY, query_type=SearchType.HYBRID_COMPLETION)
```
Docs and sales calls share one retrieval pool, so the rate-limit question retrieves from the API reference and the transcripts alike. In this run the model resolved the conflict in favor of the documentation — but nothing in the query kept the rep's promise out of the pool. The `NOTE` line names the competition to watch for.
```text theme={null}
==============================================================================
(1) ALL DATA IN ONE DATASET — the setup that confuses the LLM
==============================================================================
QUERY : What rate limits does the Acme API enforce?
SCOPE : everything (no filter)
NOTE : docs and sales calls share one retrieval pool — the documented limit (100/min) competes with the rep's 'unlimited requests' promise
ANSWER: [main_dataset] - Default: 100 requests per minute per API key.
- Enterprise: can be raised to 500 requests per minute per key.
...
- No plan offers unlimited requests.
```
**Pros**
* The simplest layout: one `remember()` call, no labeling decisions, nothing to keep consistent later.
* Every fact can link to every other, so any question can draw on the whole corpus.
**Cons**
* There is no way to scope a query. Every recall sees everything, and sources that disagree compete in one retrieval pool.
* Whether the answer favors the spec or the promise is left to the LLM, so a correct answer in one run is not a guarantee for the next.
* One set of permissions for the whole pile: you cannot let one audience read the transcripts and another only the docs.
**Use it for**
* A corpus that covers one domain for one audience, where every question should see everything.
* Prototypes and small personal memories, before you know which groupings matter.
## One Dataset with Node Sets
The same dataset and the same graph, but every item now carries node-set tags. Tags are soft and overlappable: everything is also tagged `acme_api`, so a shared grouping cuts across the docs/calls split — something no folder-style hierarchy could express.
```python theme={null}
# An item can carry several tags at once: everything here is also tagged
# "acme_api", so a shared grouping cuts across the docs/calls split.
await cognee.remember(TECH_DOCS, node_set=["tech_docs", "acme_api"], self_improvement=False)
await cognee.remember(
[SALES_CALL_INITECH, SALES_CALL_HOOLI],
node_set=["sales_calls", "acme_api"],
self_improvement=False,
)
```
Tagging's effect shows the moment a recall is scoped to one tag. `node_name` restricts retrieval to the tagged subgraph, so the same question now answers from the documentation alone. The sales calls are still in the graph — they just don't participate in this query.
```python theme={null}
docs_answer = await cognee.recall(
RATE_LIMIT_QUERY,
query_type=SearchType.HYBRID_COMPLETION,
node_name=["tech_docs"],
)
```
Scoped to `tech_docs`, the answer is the documented limit. Scoped to `sales_calls` with the same call shape, it is what the rep said, with no mention of the spec.
```text theme={null}
QUERY : What rate limits does the Acme API enforce?
SCOPE : node_name=['tech_docs']
NOTE : the documented truth only
ANSWER: [main_dataset] - Default: 100 requests per minute per API key.
- Enterprise: can raise to 500 requests per minute per key.
...
- No plan offers unlimited requests; provision multiple keys to shard traffic.
QUERY : What did we promise customers about rate limits?
SCOPE : node_name=['sales_calls']
NOTE : what the reps actually said
ANSWER: [main_dataset] We promised to remove the 100 requests/minute cap for enterprise deals and to allow unlimited (no throttling) requests during the pilot — the AE said they'd include this in the proposal deck.
```
The payoff of tags over walls is the question you ask with no filter at all. Because docs and calls share one graph, entities from both link to each other, and an unfiltered recall can draw on both sides.
```python theme={null}
# The graph is still ONE graph — cross-domain questions work when you
# want them to, by simply not filtering.
cross_query = "Where do our sales promises contradict the technical documentation?"
cross_answer = await cognee.recall(cross_query, query_type=SearchType.GRAPH_COMPLETION)
```
The deliberately unscoped recall lines up the sales promises against the API reference and cites both sources. This is the question a dataset wall would make impossible.
```text theme={null}
QUERY : Where do our sales promises contradict the technical documentation?
SCOPE : everything (no filter, on purpose)
NOTE : a cross-domain question spanning both groups
ANSWER: [main_dataset] Short answer — two direct contradictions:
- Enterprise “remove rate limits entirely” (sales AE told Initech this) vs. the API docs stating Enterprise can raise the per-key limit only to 500 req/min and “No plan, including Enterprise, offers unlimited requests.” ...
- “Unlimited requests during the pilot / we won’t throttle you” (AE promised, and agreed to put it in the proposal deck) vs. the API docs stating rate limits are enforced per API key ...
```
**Pros**
* Scoped and cross-domain questions both work: `node_name` narrows a recall to one group, and leaving it out lets the graph connect the groups.
* Tags overlap. One item can belong to several groups at once, like `tech_docs` and `acme_api` here.
* Cheap to adopt: the same `remember()` call with a `node_set` argument, no new datasets or permissions to manage.
**Cons**
* The boundary is soft. An unscoped recall still sees everything, so keeping the promise out of the spec is the caller's job on every query.
* Tags are assigned at write time, so a new grouping means re-ingesting the affected items.
* No separate permissions, storage, or forget per tag — everything still lives in one dataset.
**Use it for**
* Mixed content that shares entities you still want linked, like documentation and call transcripts about the same product.
* Memory that has to answer both narrow questions ("what does the spec say?") and cross-cutting ones ("where do the two disagree?").
* Slicing one team's memory by topic, source, or account without splitting it up.
## Separate Datasets
The third layout gives each domain its own dataset. Graphs are built independently, permissions and storage are separate, and either side can be forgotten without touching the other. Node sets still work inside each dataset — here the sales dataset tags each call by account.
```python theme={null}
await cognee.remember(
TECH_DOCS,
dataset_name="tech_docs",
node_set=["api_reference"],
self_improvement=False,
)
await cognee.remember(
SALES_CALL_INITECH,
dataset_name="sales_calls",
node_set=["initech"],
self_improvement=False,
)
await cognee.remember(
SALES_CALL_HOOLI,
dataset_name="sales_calls",
node_set=["hooli"],
self_improvement=False,
)
```
The two mechanisms compose: `datasets` picks the wall, `node_name` picks the tag inside it, and the answer covers one account's calls only.
```python theme={null}
initech_query = "What was discussed with Initech?"
initech_answer = await cognee.recall(
initech_query,
query_type=SearchType.HYBRID_COMPLETION,
datasets=["sales_calls"],
node_name=["initech"],
)
```
The `[tech_docs]` prefix on the first answer shows it came from that dataset alone — the sales calls cannot leak in. The combined `datasets` plus `node_name` filter narrows the second answer to the Initech call.
```text theme={null}
==============================================================================
(3) SEPARATE DATASETS (tech vs sales) — hard walls, node sets inside
==============================================================================
QUERY : What rate limits does the Acme API enforce?
SCOPE : datasets=['tech_docs']
NOTE : sales calls cannot leak in
ANSWER: [tech_docs] - Default: 100 requests per minute per API key.
- Enterprise: up to 500 requests per minute per API key.
...
- No plan offers unlimited requests.
QUERY : What was discussed with Initech?
SCOPE : datasets=['sales_calls'] + node_name=['initech']
NOTE : one account's calls only
ANSWER: [sales_calls] They discussed Initech’s bursty batch jobs and that the 100 requests/min cap would be a blocker; Acme offered to remove rate limits for an enterprise deal and provide unlimited requests during the pilot, and Initech asked that the unlimited pilot offer be included in the proposal deck.
```
The layout has one footgun, and the demo closes on it. `recall()` without `datasets` spans every dataset you can read, so separating data at write time is not enough — queries must opt into the scope they want.
```python theme={null}
# Caution: recall() without `datasets` spans ALL datasets you can read —
# separating data at write time is not enough, queries must opt into the
# scope they want.
unscoped_answer = await cognee.recall(RATE_LIMIT_QUERY, query_type=SearchType.HYBRID_COMPLETION)
```
The same rate-limit question without `datasets` returns one `ANSWER` per readable dataset, `[tech_docs]` and `[sales_calls]`, so the rep's promise is back in the result. The run then closes with the `TAKEAWAY` banner.
```text theme={null}
QUERY : What rate limits does the Acme API enforce?
SCOPE : everything (no dataset filter)
NOTE : spans both datasets again — scope must be chosen per query
ANSWER: [tech_docs] - Default: 100 requests per minute per API key.
- Enterprise: up to 500 requests per minute per API key.
...
- No plan (including Enterprise) offers unlimited requests.
ANSWER: [sales_calls] By default a 100 requests per minute cap is mentioned; Acme said they can remove rate limits for an enterprise deal and offered unlimited (no throttling) requests during the pilot.
```
**Pros**
* Hard isolation: a recall scoped to one dataset can never surface content from another.
* Each dataset has its own permissions and storage, and can be forgotten independently.
* Node sets still slice inside a dataset, so you lose nothing from the previous layout within each wall.
**Cons**
* Graphs are built independently, so there is no cross-dataset reasoning. An unscoped recall returns one answer per dataset rather than one answer that connects them.
* The wall only holds when the query names it: `recall()` without `datasets` spans every dataset you can read.
* Dataset scoping walls datasets off only in [multi-user mode](/core-concepts/multi-user-mode/multi-user-mode-overview), which the default Ladybug and LanceDB stack turns on automatically — keep `ENABLE_BACKEND_ACCESS_CONTROL` at its default or set it to `true`.
**Use it for**
* Domains that must never contaminate each other's answers, like the spec and the sales promises here.
* Content with different access control or retention, where one audience or one lifecycle applies per dataset.
* Multi-tenant memory, with one dataset per customer or team.
## Run It
```bash theme={null}
uv run python examples/demos/organizing_your_data/organizing_your_data_demo.py
```
Run it from a checkout of the cognee repo with an [LLM provider](/setup-configuration/llm-providers) configured: the script reads its two documents and two transcripts from the sibling `data/` folder, and every answer is a live LLM call. Ladybug (the default) and Neo4j both support node-set filtering out of the box. The script calls `cognee.forget(everything=True)` before each of its three sections, so point it at a scratch instance rather than memory you want to keep — see [Forget](/core-concepts/main-operations/forget).
## Choosing a Layout
Start with the simplest layout that answers your queries cleanly. One dataset with no labels is fine while the corpus covers a single domain for a single audience. Reach for node sets the moment distinct content types share entities you still want linked — scoped questions stay clean, and cross-domain questions keep working. Move a domain into its own dataset when it must never contaminate the other's answers, or when it needs different access control or retention — and remember the two compose, so datasets can carry node sets inside them.
<Columns>
<Card title="NodeSets" icon="tags" href="/core-concepts/further-concepts/node-sets">
How node-set labels are written and how `node_name` filters retrieval by them.
</Card>
<Card title="Datasets" icon="database" href="/core-concepts/further-concepts/datasets">
What a dataset isolates — documents, permissions, and processing.
</Card>
<Card title="NodeSet Grouping" icon="layers" href="/guides/nodeset-grouping">
The minimal single-feature guide to tagging memories with node sets.
</Card>
<Card title="Agentic Procurement Decisions" icon="gavel" href="/examples/agentic-procurement">
A sibling demo where node-set scoping powers an agent's research phase.
</Card>
</Columns>
# Cognee examples overview
Source: https://docs.cognee.ai/examples/overview
Browse Cognee examples by use case and pattern.
AI systems still struggle with the messy realities of data.
**The core challenges:**
* **Complex Data at Scale**: Databases spanning hundreds of tables, documents in dozens of formats, knowledge scattered across systems
* **Lack of Business Context**: Without domain ontologies and relationships, even advanced LLMs produce hallucinations
* **Stale Knowledge**: Static RAG doesn't evolve as your organization and data change
Cognee solves these problems by creating a unified memory layer, combining knowledge graphs with vector search to give AI systems true understanding of your data.
***
## Example Use Cases
### [Vertical AI Agents](./vertical-ai-agents)
The memory layer that makes autonomous agents actually work. Agents without memory can't learn, can't understand organizational context, and can't improve over time. Cognee provides the missing piece.
**Key capabilities:**
* Persistent memory across agent sessions
* Domain-specific reasoning context
* Continuous learning and improvement
***
### [Enterprise Data Unification](./data-silos)
Connect data silos without replacing your existing systems. When the answer requires CRM + support tickets + contracts + operational data, Cognee provides the unified view.
**Key capabilities:**
* 30+ data source connectors
* Entity resolution across systems
* Granular access control by user, team, or organization
***
### [Edge AI & On-Device Memory](./edge-ai)
Bring AI memory to resource-constrained devices with cognee-RS, our Rust-based SDK. Run the full memory pipeline directly on phones, smartwatches, glasses, and smart-home hubs—sub-100ms recall, data stays local.
**Key capabilities:**
* Fully offline operation with on-device LLMs
* Hybrid execution—local or cloud based on connectivity
* Privacy-first architecture for sensitive data
***
## Common Patterns Across Use Cases
### Memory Enrichment
All use cases benefit from Cognee's ability to consolidate information over time, not just at ingestion, but continuously as new data arrives and patterns emerge.
### Ontology Management
Whether it's financial instrument definitions, research taxonomies, or codebase architecture, Cognee aligns your domain-specific terminology into a coherent knowledge structure.
### Hybrid Search
Every query leverages both graph traversal (understanding relationships) and vector similarity (semantic matching) for complete, accurate results.
### Modular Customization
Cognee provides building blocks such as chunkers, loaders, retrievers, ontology definitions that you can customize for your specific domain without building from scratch.
***
## Dive Deeper in Use Cases:
* [What You Can Build](./what-you-can-build) - Concrete ways teams put Cognee to work
* [Vertical AI Agents](./vertical-ai-agents) - The memory layer that makes autonomous agents actually work
* [Enterprise Data Unification](./data-silos) - Connect data silos without replacing your existing systems
* [Edge AI & On-Device Memory](./edge-ai) - Rust-powered AI memory for phones, wearables, and IoT devices
# Rebuild Add and Cognify by Hand
Source: https://docs.cognee.ai/examples/rebuild-cognify-pipeline
Reproduce cognee's built-in add and cognify stages as custom pipelines, so you can see and change every task they run
You need to change something in the middle of cognee's ingestion — swap a chunker, insert a task of your own, or just see which steps run in what order — and calling `add()` and `cognify()` hides all of it behind two function calls.
## What You'll Build
A short paragraph of text about natural language processing goes in, and a queryable knowledge graph comes out — except neither `add()` nor `cognify()` is ever called. The add stage is reassembled by hand from the two `Task` objects it wraps, and the cognify stage is run from the task list `cognify()` itself would have used, both handed to `run_custom_pipeline()`. A final `GRAPH_COMPLETION` search over the result proves the hand-driven graph is an ordinary cognee graph.
The complete runnable script is
[`examples/demos/custom_pipelines/custom_cognify_pipeline_example.py`](https://github.com/topoteretes/cognee/blob/main/examples/demos/custom_pipelines/custom_cognify_pipeline_example.py) —
this page walks through its key moments rather than reproducing it.
## Features in Play
* [Custom Tasks and Pipelines](/guides/custom-tasks-pipelines) — `run_custom_pipeline()` executes both stages from task lists the script controls
* [Tasks](/core-concepts/building-blocks/tasks) — `Task(...)` wraps `resolve_data_directories` and `ingest_data` together with the arguments they need
* [Add](/core-concepts/main-operations/legacy-operations/add) — the ingestion stage this demo reconstructs from its two underlying tasks
* [Cognify](/core-concepts/main-operations/legacy-operations/cognify) — `get_default_tasks()` hands over the real graph-building task list instead of running it
* [Inspecting Graph Completion Context](/guides/graph-completion) — `SearchType.GRAPH_COMPLETION` queries the graph the custom pipelines built
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — the cognify task list makes live LLM calls to extract entities and relationships. The script's own header asks you to copy `.env.template` to `.env` and set `LLM_API_KEY`
* Run it from a checkout of the cognee repo: it imports internals such as `cognee.modules.pipelines.Task` and `cognee.api.v1.cognify.cognify.get_default_tasks`, and the command below uses the repo-relative path
* The script begins with `prune_data()` and `prune_system(metadata=True)`, which wipe data and system state including users and pipeline runs — point it at a scratch instance rather than memory you want to keep
## How It Works
### Stage 1: Reset and Initialize the Databases
```python theme={null}
# Create a clean slate for cognee -- reset data and system state
print("Resetting cognee data...")
await cognee.prune.prune_data()
await cognee.prune.prune_system(metadata=True)
print("Data reset complete.\n")
# Create relational database and tables
await setup()
```
Pruning the system with `metadata=True` drops the relational database, so the tables the pipelines write to have to be recreated before anything runs. `setup()` is the call the high-level operations make for you; driving pipelines directly means making it yourself.
### Stage 2: Rebuild the Add Stage from Tasks
```python theme={null}
# Let's recreate the cognee add pipeline through the custom pipeline framework
from cognee.tasks.ingestion import ingest_data, resolve_data_directories
user = await get_default_user()
# Values for tasks need to be filled before calling the pipeline
add_tasks = [
Task(resolve_data_directories, include_subdirectories=True),
Task(
ingest_data,
"main_dataset",
user,
),
]
# Forward tasks to custom pipeline along with data and user information
await cognee.run_custom_pipeline(
tasks=add_tasks, data=text, user=user, dataset="main_dataset", pipeline_name="add_pipeline"
)
```
This is what `add()` does underneath: resolve whatever was passed into concrete data items, then ingest them into a dataset. A `Task` carries its own arguments — the dataset name and user are bound into `ingest_data` here — while the pipeline supplies the data flowing through. Naming the run `add_pipeline` keeps it identifiable in the pipeline-run history.
### Stage 3: Borrow the Default Cognify Task List
```python theme={null}
from cognee.api.v1.cognify.cognify import get_default_tasks
cognify_tasks = await get_default_tasks(user=user)
print("Recreating existing cognify pipeline in custom pipeline to create knowledge graph...\n")
await cognee.run_custom_pipeline(
tasks=cognify_tasks, user=user, dataset="main_dataset", pipeline_name="cognify_pipeline"
)
```
Rather than hand-writing the graph-building steps, the script asks cognify for its own task list — document classification, chunking, entity extraction, summarization, storage — and runs it through the same custom pipeline entry point. No `data` argument is needed: the tasks pick up the data items the add pipeline already wrote to `main_dataset`.
### Stage 4: Query the Hand-Built Graph
```python theme={null}
query_text = "Tell me about NLP"
print(f"Searching cognee for insights with query: '{query_text}'")
# Query cognee for insights on the added text
search_results = await cognee.search(
query_type=SearchType.GRAPH_COMPLETION, query_text=query_text
)
```
An ordinary `search()` closes the loop. Nothing about the query is aware that the graph was built task by task — which is the point of the demo: the custom pipeline path produces the same graph the built-in operations do.
## Run It
```bash theme={null}
uv run python examples/demos/custom_pipelines/custom_cognify_pipeline_example.py
```
A successful run prints the reset messages first, then echoes the NLP paragraph it is adding and confirms `Text added successfully.` once the add pipeline finishes. The cognify pipeline announces that it is recreating the existing cognify pipeline, takes the longest as it makes its LLM calls, and ends with `Cognify process complete.` The script then reports the query it is running and prints `Search results:` followed by the generated answer about natural language processing.
## Where to Change It
Both stages are plain Python lists of `Task` objects, so this is the shape to start from when the built-in flow is close to what you want but not exact. `get_default_tasks()` returns the cognify list, and you can reorder it, drop a task, or splice your own in before passing it to `run_custom_pipeline()` — see [Custom Tasks and Pipelines](/guides/custom-tasks-pipelines) for writing that task. It also takes the knobs cognify takes, including `graph_model` and `chunker`, if adjusting the arguments is enough.
<Columns>
<Card title="Custom Tasks and Pipelines" icon="workflow" href="/guides/custom-tasks-pipelines">
Write your own task and run it in a pipeline of your own.
</Card>
<Card title="Tasks" icon="square-check" href="/core-concepts/building-blocks/tasks">
What a `Task` wraps, and how its arguments and batching work.
</Card>
<Card title="run_custom_pipeline()" icon="route" href="/python-api/custom-pipeline">
Every parameter of the entry point both stages are run through.
</Card>
<Card title="Cognify" icon="brain-cog" href="/core-concepts/main-operations/legacy-operations/cognify">
The operation whose default task list this demo runs by hand.
</Card>
</Columns>
# Migrate a Relational Database into a Knowledge Graph
Source: https://docs.cognee.ai/examples/relational-db-migration
Convert a relational database schema and data into a searchable knowledge graph
A focused walkthrough for converting an existing relational database (SQLite or Postgres) into a knowledge graph that can be queried with natural language.
**Before you start:**
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured
* Have connection details for the database you want to migrate
## What This Does
Cognee reads your relational database schema (tables, columns, primary keys, foreign keys) and can map it into graph nodes and edges:
* In **full migration mode**, each **table** becomes a `TableType` node
* Each **row** becomes a `TableRow` node linked to its table
* Non-key column values can become `ColumnValue` nodes linked from each row
* **Foreign key relationships** become edges between row nodes
* Migrated nodes and edges are embedded and indexed for semantic search
Alternatively, pass `schema_only=True` to ingest only the structure (tables, columns, foreign keys, and a handful of sample rows) instead of every row — see [Ingestion options](#additional-information) below for all parameters.
After migration, you can query your relational data using `cognee.recall()` with natural language.
## 1. Configure the Migration Database
Set the source database connection in your `.env` file:
<Tabs>
<Tab title="SQLite">
```dotenv theme={null}
MIGRATION_DB_PROVIDER="sqlite"
MIGRATION_DB_PATH="/path/to/db/directory"
MIGRATION_DB_NAME="my_database.sqlite"
```
</Tab>
<Tab title="Postgres">
```dotenv theme={null}
MIGRATION_DB_PROVIDER="postgres"
MIGRATION_DB_HOST="127.0.0.1"
MIGRATION_DB_PORT="5432"
MIGRATION_DB_USERNAME="myuser"
MIGRATION_DB_PASSWORD="mypassword"
MIGRATION_DB_NAME="my_database"
```
</Tab>
</Tabs>
## 2. Run the Migration
```python theme={null}
import asyncio
import os
import tempfile
from pathlib import Path
import cognee
from cognee.api.v1.visualize.visualize import visualize_graph
from cognee.infrastructure.databases.graph import get_graph_engine
from cognee.infrastructure.databases.relational import get_migration_relational_engine
from cognee.infrastructure.databases.relational.config import get_migration_config
from cognee.tasks.ingestion import migrate_relational_database
# Isolate this example from any pre-existing local Cognee state so old SQLite
# metadata files do not interfere with the run.
example_root = Path(tempfile.gettempdir()) / "cognee-relational-migration-example"
os.environ.setdefault("SYSTEM_ROOT_DIRECTORY", str(example_root / "system"))
os.environ.setdefault("DATA_ROOT_DIRECTORY", str(example_root / "data"))
os.environ.setdefault("CACHE_ROOT_DIRECTORY", str(example_root / "cache"))
os.environ.setdefault("ENABLE_BACKEND_ACCESS_CONTROL", "false")
async def main():
migration_config = get_migration_config()
migration_config.migration_db_provider = os.environ.get("MIGRATION_DB_PROVIDER", "sqlite")
migration_config.migration_db_path = os.environ.get("MIGRATION_DB_PATH", "/path/to/db")
migration_config.migration_db_name = os.environ.get(
"MIGRATION_DB_NAME", "my_database.sqlite"
)
# Start from a clean local state for repeatable runs.
await cognee.forget(everything=True)
# Extract the schema from the source database
engine = get_migration_relational_engine()
schema = await engine.extract_schema()
print(f"Loaded schema with {len(schema)} tables: {sorted(schema)}")
# Migrate schema + all rows into the graph
graph = await get_graph_engine()
nodes, edges = await migrate_relational_database(graph, schema=schema)
print(f"Migrated graph: {len(nodes)} nodes / {len(edges)} edges")
# Query the migrated data
results = await cognee.recall(
query_text="What data does this database contain?",
top_k=100,
)
print("Recall results:")
print(results)
# Visualize the migrated graph
graph_path = Path(__file__).resolve().parent / "relational_migration_graph.html"
graph_html = await visualize_graph(str(graph_path))
graph_path.write_text(graph_html, encoding="utf-8")
print(f"Graph visualization saved at: {graph_path}")
asyncio.run(main())
```
<Note>
The migration pipeline itself uses lower-level APIs because it is importing an external relational schema directly into the graph. Once the graph is built, prefer `cognee.recall()` as the default query interface in v1.0. Set `top_k` higher (100-200) for broad exploratory queries over large databases, and lower (20-50) for specific lookups to keep LLM context manageable. The example above also isolates Cognee's local state in a temp directory so repeated runs do not pick up stale metadata from earlier experiments.
</Note>
## Additional information
<AccordionGroup>
<Accordion title="Ingestion options: schema only vs schema + row data">
`migrate_relational_database` accepts these parameters:
| Parameter | Default | What it does |
| --------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `graph_db` | — | Graph engine to write into, from `await get_graph_engine()` |
| `schema` | — | Schema dict from `await engine.extract_schema()` |
| `migrate_column_data` | `True` | In full mode, also create a `ColumnValue` node per non-key column value and link it to its row. Set to `False` to keep only `TableType`, `TableRow`, and foreign-key edges |
| `schema_only` | `False` | Ingest only the database structure instead of every row (see below) |
The `schema_only` flag selects between the two ingestion modes:
| Mode | Flag | What is migrated |
| ------------------ | ------------------- | ------------------------------------------------------------------------------------------------- |
| **Full** (default) | `schema_only=False` | `TableType`, `TableRow`, optional `ColumnValue`, and foreign-key edges |
| **Schema only** | `schema_only=True` | `DatabaseSchema`, `SchemaTable`, and `SchemaRelationship` datapoints for structural understanding |
Schema-only mode does **not** create `TableRow` nodes for every record. Instead, it ingests schema-level datapoints and attaches up to 5 sample rows, the column definitions, and a row-count estimate as metadata on each `SchemaTable` node. Use it when you want to explore table structure and relationships without migrating the full dataset:
```python theme={null}
# Structure only: tables, columns, foreign keys, plus a few sample rows
await migrate_relational_database(graph, schema=schema, schema_only=True)
# Schema + all row data, but without per-column value nodes
await migrate_relational_database(graph, schema=schema, migrate_column_data=False)
```
Schema-only mode is especially useful for large databases when your questions are structural, for example:
* "What tables exist in this database?"
* "Which tables are connected by foreign keys?"
* "What kind of data does the `invoices` table contain?"
Use full mode when you need to answer questions about individual records ("Which customers ordered in March?"), since only full mode creates and embeds a node for every row.
</Accordion>
<Accordion title="Mixing migration and application DB providers">
`MIGRATION_DB_PROVIDER` is independent of `DB_PROVIDER` — Cognee builds a separate engine for each from its own config (`MigrationConfig` vs `RelationalConfig`). You can freely combine providers: for example, keep Cognee's application metadata in Postgres while migrating from a local SQLite file.
```dotenv theme={null}
# Application DB — Cognee's internal metadata store
DB_PROVIDER="postgres"
DB_NAME="cognee_db"
DB_HOST="127.0.0.1"
DB_PORT="5432"
DB_USERNAME="cognee"
DB_PASSWORD="cognee"
# Migration DB — source data in a SQLite file
MIGRATION_DB_PROVIDER="sqlite"
MIGRATION_DB_PATH="/path/to/db/directory"
MIGRATION_DB_NAME="my_app_data.sqlite"
```
The reverse (`DB_PROVIDER="sqlite"` with `MIGRATION_DB_PROVIDER="postgres"`) is equally valid. See [Relational Databases](/setup-configuration/relational-databases) for additional same-server and cross-server combinations.
</Accordion>
<Accordion title="Visualize the Result">
```python theme={null}
from cognee.api.v1.visualize.visualize import visualize_graph
from pathlib import Path
graph_path = Path("./migration_graph.html")
graph_html = await visualize_graph(str(graph_path))
graph_path.write_text(graph_html, encoding="utf-8")
```
This generates an HTML file you can open in a browser to explore the graph structure. See [Graph Visualization](/guides/graph-visualization) for more options.
</Accordion>
</AccordionGroup>
<Columns>
<Card title="Relational Databases" icon="database" href="/setup-configuration/relational-databases">
Configure SQLite and Postgres connections
</Card>
<Card title="Recall and Search" icon="search" href="/guides/search-basics">
Learn the current recall flow and lower-level search options
</Card>
<Card title="Graph Visualization" icon="network" href="/guides/graph-visualization">
Explore your knowledge graph visually
</Card>
</Columns>
# Let an Agent Rewrite Its Own Weak Skill
Source: https://docs.cognee.ai/examples/self-improving-skills
Ingest three SKILL.md playbooks, run them against a real diff, score the one that does the wrong job, and apply the proposal that rewrites its instructions
One of the skills your agent loads on every pull request quietly does the wrong job — it grades the reviewer's tone and never looks at the code. Nothing errors, so nobody notices until someone reads a transcript, and then a person has to sit down and edit the playbook by hand.
## What You'll Build
Three `SKILL.md` playbooks in a bundled `skills/` folder — a diff explainer, a deliberately flawed PR-comment evaluator that judges politeness only, and a critic that grades the other two — are remembered into one dataset as skills. A single agentic run then loads all three in order against a two-line diff that drops a `None` check and a reviewer comment that says nothing more than "This is bad", and returns JSON naming which skill failed, the score it deserves, and the instruction it is missing. That verdict is recorded as a skill run, which drafts a skill-improvement proposal; applying the proposal rewrites the flawed skill's procedure in the graph, and the script prints the skill's text before and after so you can read the edit the loop made for you.
The complete runnable script is
[`examples/demos/feedback/skill_feedback_loop/skill_feedback_loop_demo.py`](https://github.com/topoteretes/cognee/blob/main/examples/demos/feedback/skill_feedback_loop/skill_feedback_loop_demo.py) —
this page walks through its key moments rather than reproducing it.
## Features in Play
* [Remember](/core-concepts/main-operations/remember) — `content_type="skills"` turns a folder of `SKILL.md` files into skill nodes in one dataset, which is what makes them loadable by name later
* [Recall](/core-concepts/main-operations/recall) — one `AGENTIC_COMPLETION` call is the agent run under test: it loads the three skills, does the work, and returns the JSON verdict the rest of the script acts on
* [Sessions](/guides/sessions) — a single `session_id` spans the agentic run and the record written about it, so the evaluation and the improvement belong to the same episode
* [Skill-improvement proposals](/api-reference/introduction#api-features) — a skill-run entry with a low `success_score` drafts a proposal; applying it by id is what actually rewrites the skill's procedure
## What to Expect
The excerpts below are from a real run, trimmed. The script prints five numbered lines to stdout; two LiteLLM notice blocks that appeared between the first and second line during the agentic loop are elided here. The score, the feedback wording, and the text of the rewritten skill are all live LLM output and vary from run to run.
**Three skills go in, and the flawed one comes out with a failing score.** The first line confirms the folder walk found all three `SKILL.md` files. The second is the agentic run's verdict: it singled out `pr-comment-evaluator`, as the task and the critic skill both steer it to, and scored it well under the `0.30` ceiling the critic sets for a tone-only evaluation.
```text theme={null}
1. remember -> stored 3 skills
...
2. evaluation -> pr-comment-evaluator scored 0.20
```
**A proposal is drafted and applied in one pass.** The third line names the proposal that the skill-run record drafted and that the script then applied by id — the same id you could inspect with `GET /api/v1/proposals/{proposal_id}` before applying it.
```text theme={null}
3. improve proposal -> applied proposal_id=5efe8e2e-ed96-4d2a-83d5-fbcfa03128cb
```
**The skill's procedure changes in place.** The last two lines are the same skill read back by name on either side of the rewrite, flattened onto one line each. Before, it is the one-sentence playbook that refuses to look at code. After, it is a procedure that takes the diff and the comment as inputs, parses the diff into hunks, and lists "removed None-checks" among the patterns to detect — then scores the comment on a separate `technical_score` next to `tone_score`. The full rewritten procedure runs to several hundred words; only its opening is shown.
```text theme={null}
4. skill before -> # pr-comment-evaluator Only judge whether the PR comment sounds polite. Do not discuss code risk or technical correctness.
5. skill after -> # pr-comment-evaluator Inputs provided to the skill: `diff` (unified diff or patch text) and `comment` (the reviewer comment text). Behavioral constraints: - Do not mutate state or modify any files. Do not run commands that change the repository. - Operate only on the provided `diff` and `comment` inputs. Procedure (follow exactly): 1. Parse the `diff` into changed hunks. For each hunk, extract file path, line ranges, removed lines, and added lines. 2. For each hunk, identify concrete code changes that could introduce a bug or behavior change. ...
```
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — the agentic pass is a live tool-calling loop and the proposal is LLM-drafted, so the score, the feedback text, and the rewritten skill differ from run to run
* Set `RECALL_WARMUP_SHORTCIRCUIT=false` in your environment before running: skills ingestion writes skill nodes without logging a graph build, so recall's [warm-up guard](/python-api/recall#warming-up-marker) reads the dataset as empty and returns a warming-up marker instead of an answer, which the script cannot parse. See [Recall warm-up](/setup-configuration/overview#recall-warm-up) for the variable
* Run it from a checkout of the cognee repo: the script reads its three playbooks from the `skills/` folder and its diff and reviewer comment from the `data/` folder next to it, so a copy-pasted script has nothing to ingest
* The run opens with `cognee.forget(everything=True)` and does not redirect cognee's storage roots, so it clears whatever memory the current configuration points at — run it against a scratch instance rather than storage you want to keep
## How It Works
### Stage 1: Ingest the Skills Folder
```python theme={null}
remembered = await cognee.remember(
str(SKILLS_ROOT),
dataset_name=DATASET_NAME,
content_type="skills",
)
print(f"1. remember -> stored {remembered.items_processed} skills")
user, datasets = await resolve_authorized_user_datasets(UUID(remembered.dataset_id))
dataset = datasets[0]
```
`content_type="skills"` walks the `skills/` directory for `SKILL.md` files and stores each one as a skill scoped to `toy-skill-feedback-loop` — skills are always dataset-scoped, and the agentic run later needs exactly one dataset to work in. The `user` and `dataset` resolved from the returned dataset id are what the script uses to read and rewrite a skill's body directly at the end.
### Stage 2: Write the Task That Exposes the Flaw
```python theme={null}
TASK_TEMPLATE = """Use the skills in this exact order:
1. Load diff-risk-explainer and explain the concrete bug risk in the diff.
2. Load pr-comment-evaluator and evaluate the reviewer comment.
3. Load skill-feedback-writer and decide which skill needs a better instruction.
The skills are plain instructions. After you load each skill, do the work yourself.
The pr-comment-evaluator skill is intentionally flawed because it judges tone only. If its output
does not compare the reviewer comment against the concrete bug risk, target pr-comment-evaluator
and give a score of 0.30 or lower.
Return only JSON with keys:
diff_risk_summary, comment_evaluation, skill_to_improve, score, feedback, missing_instruction.
Diff:
{diff_text}
Reviewer comment:
{comment_text}
"""
```
The task names the order the three skills run in and pins the output to a fixed set of JSON keys, so the script can read `skill_to_improve` and `score` out of a free-text answer. It also tells the agent what a failure looks like: `pr-comment-evaluator` is written to judge tone only, and an evaluation that never mentions the diff's dropped `None` check earns `0.30` or lower. The bundled `skill-feedback-writer` playbook carries the same rule, so the low score comes from the skills as much as from the prompt.
### Stage 3: Run the Three Skills in One Agentic Pass
```python theme={null}
answer = await cognee.recall(
task,
query_type=SearchType.AGENTIC_COMPLETION,
datasets=DATASET_NAME,
retriever_specific_config={
"skills": SKILL_NAMES,
"max_iter": 6,
},
session_id=SESSION_ID,
)
feedback = parse_json_answer(answer)
score = score_from_feedback(feedback)
skill_to_improve = str(feedback["skill_to_improve"])
```
`AGENTIC_COMPLETION` hands the agent the three skill names and lets it pull each procedure in with the `load_skill` tool, up to six tool rounds. The agent sees only names and descriptions up front — the bodies arrive when it asks for them — which is why the run is a fair test of the playbooks rather than of one long prompt. `datasets=DATASET_NAME` keeps the scope to a single dataset, which [`AGENTIC_COMPLETION` requires](/core-concepts/main-operations/recall#examples-and-details); the script's own helpers then pull the JSON out of the answer and clamp the score into `0.0`–`1.0`.
### Stage 4: Record the Weak Run and Draft a Proposal
```python theme={null}
proposal_result = await cognee.remember(
SkillRunEntry(
selected_skill_id=skill_to_improve,
task_text=task,
result_summary=feedback_summary(feedback),
success_score=score,
feedback=-1.0 if score < 0.7 else 1.0,
),
dataset_name=DATASET_NAME,
session_id=SESSION_ID,
skill_improvement={
"skill_name": skill_to_improve,
"apply": False,
"score_threshold": 0.9,
},
)
proposal_id = next(
item["proposal_id"]
for item in proposal_result.items
if item.get("kind") == "skill_improvement_proposal"
)
```
The verdict goes back into memory as a `SkillRunEntry`: which skill was used, the task it was used for, the critic's feedback and missing instruction as the result summary, and the score as both a raw `success_score` and a `-1.0`/`1.0` signal. `skill_improvement` is what turns that record into a rewrite — `score_threshold: 0.9` means any run scoring below `0.9` is bad enough to draft against, and `apply: False` stops at the draft so the proposal can be inspected before it changes anything. The proposal's id comes back among the entry's result items.
### Stage 5: Apply the Proposal and Read the Skill Back
```python theme={null}
before = await skill_body(skill_to_improve, dataset, user)
await improve_skill(
skill_to_improve,
dataset=dataset,
user=user,
proposal_id=proposal_id,
apply=True,
)
after = await skill_body(skill_to_improve, dataset, user)
```
Reading the skill's procedure on either side of `improve_skill(..., apply=True)` is what makes the loop visible: the same lookup by name, before and after the proposal is applied, over the same dataset. The `before` text is the flawed instruction that only judges politeness; the `after` text is the drafted replacement, which the critic's `missing_instruction` asked to compare the reviewer comment against the concrete bug risk. Nothing here re-ingests the folder — the skill node is edited in place, so the next agentic run loads the new text.
## Run It
```bash theme={null}
uv run python examples/demos/feedback/skill_feedback_loop/skill_feedback_loop_demo.py
```
<Columns>
<Card title="Recall" icon="search" href="/core-concepts/main-operations/recall">
How `AGENTIC_COMPLETION` and the other search types are chosen and scoped.
</Card>
<Card title="Remember" icon="brain" href="/core-concepts/main-operations/remember">
The operation that stores the skills folder, and everything else, in memory.
</Card>
<Card title="Feedback System" icon="brain-circuit" href="/guides/feedback-system">
Rating answers in a session, the other half of cognee's feedback loop.
</Card>
<Card title="Tune How Strongly Ratings Steer an Answer" icon="sliders-horizontal" href="/examples/feedback-score-shifting">
The sibling demo, where feedback moves retrieval ranking instead of a playbook.
</Card>
</Columns>
# Typed Claim Extraction
Source: https://docs.cognee.ai/examples/typed-extraction-pipeline
Build a three-step custom pipeline that turns a paragraph of research text into typed Person and ScientificClaim nodes, each claim attributed to the person who made it
You have text where the facts belong to people — who claimed what, and how sure they were — and you want the graph to record exactly that, in your own node types with your own fields, rather than whatever cognee's default extraction happens to produce.
## What You'll Build
A paragraph of science history is ingested into a dataset, then a custom pipeline of three chained tasks turns it into a graph: the first reads the ingested document and asks the LLM for the people and claims it can find, the second asks which person each claim belongs to and attaches them, and the third writes the resulting objects to the graph and vector stores and prints a summary of what it stored. What comes out is a small graph of `Person` nodes, each linked to the `ScientificClaim` nodes it owns, with every node stamped with the content hash of the document it came from and the claim text and person names indexed for semantic search — so the script can close by asking "Who worked on gravity?" and getting an answer back from the graph it just built.
The complete runnable script is
[`examples/demos/custom_pipelines/custom_pipeline_single_object_example.py`](https://github.com/topoteretes/cognee/blob/main/examples/demos/custom_pipelines/custom_pipeline_single_object_example.py) —
this page walks through its key moments rather than reproducing it.
## Features in Play
* [Custom Tasks and Pipelines](/guides/custom-tasks-pipelines) — `Task` wraps each step and `run_custom_pipeline` runs them in order against a dataset, feeding each task's return value to the next
* [Add](/core-concepts/main-operations/legacy-operations/add) — `add()` ingests the text into a dataset first, which is what gives the pipeline a document with a content hash to work from and a place for recall to look
* [DataPoints](/core-concepts/building-blocks/datapoints) — `Person` and `ScientificClaim` are the graph's node types; the `Embeddable` and `Dedup` annotations decide which fields get embedded and which give a node its identity
* [Custom Data Models](/guides/custom-data-models) — `add_data_points` persists the typed objects and their nesting as nodes and edges, without a cognify pass
* [Low-Level LLM](/guides/low-level-llm) — `LLMGateway.acreate_structured_output` runs both extraction passes against plain Pydantic response models
* [Recall](/core-concepts/main-operations/recall) — one `GRAPH_COMPLETION` query scoped to the dataset proves the hand-built nodes are searchable like any other memory
## What to Expect
The excerpts below come from one real run, trimmed of log lines. Two of the three tasks and the recall are live LLM calls, so the roles, claim wording, confidences, and answer vary from run to run; the shape of the output does not.
**The pipeline's summary is the first thing the script prints, one block per person.** Every line ends in the same twelve-character `source_hash`: all three people were extracted from the one document that was ingested, and the pipeline stamped its content hash onto each node it stored. A person with no attributed claims would show `(no claims linked)` instead of an indented list.
```text theme={null}
Albert Einstein (Physicist) [source_hash: f9bf9de04a56]
- Albert Einstein published the theory of general relativity in 1915. [confidence: 1.0]
- General relativity describes gravity as spacetime curvature. [confidence: 1.0]
Marie Curie (Physicist and chemist) [source_hash: f9bf9de04a56]
- Marie Curie discovered polonium and radium. [confidence: 1.0]
- Marie Curie won Nobel Prizes in both physics and chemistry. [confidence: 1.0]
Niels Bohr (Physicist) [source_hash: f9bf9de04a56]
- Niels Bohr proposed the atomic model with quantized electron orbits in 1913. [confidence: 1.0]
```
**The recall answers from the hand-built graph.** The entry is a `ResponseGraphEntry` carrying `dataset_name='science_claims'`, which is the proof that the custom pipeline's nodes landed in the dataset and are reachable through the ordinary retrieval path. If this section instead shows a `ResponseMarkerEntry` saying memory is warming up, the nodes were stored outside a dataset and recall cannot see them.
```text theme={null}
--- Recall: 'Who worked on gravity?' ---
[ResponseGraphEntry(kind='graph_completion', search_type='GRAPH_COMPLETION', text='Albert Einstein — he developed general relativity, which describes gravity as spacetime curvature.', score=None, dataset_id='3f474d11-b9ca-500b-a7ec-2f84dd88a49c', dataset_name='science_claims', ...
```
**The cleanup removes exactly one dataset.** `forget(everything=True)` reports the `science_claims` dataset that `add()` created, confirming the graph, its vectors, and the ingested document all lived there.
```text theme={null}
--- Forget everything ---
{'datasets_removed': 1, 'status': 'success'}
```
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — two of the three tasks are live LLM calls, and the script requires `LLM_API_KEY` in your `.env` or environment
* Run it from a checkout of the cognee repo, so the `uv run` command below resolves the repo's dependencies
* The script calls `cognee.forget(everything=True)` both before and after the pipeline, so point it at a scratch instance rather than memory you want to keep — see [Forget](/core-concepts/main-operations/forget)
## How It Works
### Stage 1: Model the Graph as Typed DataPoints
```python theme={null}
class ScientificClaim(DataPoint):
"""A factual claim extracted from text."""
text: Annotated[str, Embeddable("Claim text for semantic search"), Dedup()]
subject: str = ""
confidence: float = 1.0
class Person(DataPoint):
"""A person mentioned in the text."""
name: Annotated[str, Embeddable("Person name"), Dedup()]
role: str = ""
claims: list[ScientificClaim] | None = None
```
These two classes are the whole schema of the resulting graph. `Embeddable` marks the field that gets embedded and indexed for semantic search, `Dedup` marks the field that derives the node's identity — so re-running the pipeline over text that mentions the same person again lands on the same node instead of a second one. The `claims` field is what makes this a graph rather than a table: a list of `ScientificClaim` objects on a `Person` becomes edges from that person to those claims.
### Stage 2: Keep the LLM Schema Separate from the Graph Schema
```python theme={null}
class ExtractedPerson(BaseModel):
name: str
role: str = ""
class ExtractedClaim(BaseModel):
text: str
subject: str = ""
confidence: float = 1.0
class ExtractionResult(BaseModel):
people: list[ExtractedPerson] = Field(default_factory=list)
claims: list[ExtractedClaim] = Field(default_factory=list)
```
The LLM is asked for these plain Pydantic models, not for the `DataPoint` classes directly. A `DataPoint` carries an `id`, `metadata`, versioning and provenance fields, and a structured-output call would hand every one of them to the model to fill in — with invented ids and metadata as the result. Extracting into a plain schema and building the DataPoints from it afterwards keeps those fields under cognee's control.
### Stage 3: Extract People and Claims from the Ingested Document
```python theme={null}
text_parts = []
for data_item in data_items:
async with open_data_file(data_item.raw_data_location, mode="r", encoding="utf-8") as file:
text_parts.append(file.read())
extraction = await LLMGateway.acreate_structured_output(
text_input="\n".join(text_parts),
system_prompt=(
"Extract all people and scientific claims from the text. "
"For each person, provide their name and role. "
"For each claim, provide the claim text, subject, and confidence (0-1)."
),
response_model=ExtractionResult,
)
people = [Person(name=p.name, role=p.role) for p in extraction.people]
claims = [
ScientificClaim(text=c.text, subject=c.subject, confidence=c.confidence)
for c in extraction.claims
]
# Returned as one list of DataPoints so the pipeline stamps provenance —
# including the source document's content hash — on every node before the
# next task wires them together.
return [*people, *claims]
```
This is the body of the `extract_entities` task. Because the pipeline runs against a dataset, the task receives the dataset's `Data` records rather than a raw string, and reads the text back from where `add()` stored it. The LLM's `ExtractionResult` is then turned into real `Person` and `ScientificClaim` objects, whose ids now come from their `Dedup` fields. Returning them as one list matters: the pipeline stamps provenance onto every `DataPoint` a task returns, and the content hash of the `Data` record the task started from is part of that stamp.
### Stage 4: Attribute Each Claim to Its Author
```python theme={null}
people = [node for node in nodes if isinstance(node, Person)]
claims = [node for node in nodes if isinstance(node, ScientificClaim)]
assignments = await LLMGateway.acreate_structured_output(
text_input=(f"People: {[p.name for p in people]}\nClaims: {[c.text for c in claims]}"),
system_prompt=(
"Assign each claim to the person who made it or is most associated with it. "
"Return a list of assignments, each with a person_name and their claim_texts."
),
response_model=Assignments,
)
# Build lookup and attach claims to people
claim_lookup = {c.text: c for c in claims}
for assignment in assignments.assignments:
for person in people:
if person.name.lower() == assignment.person_name.lower():
person.claims = [
claim_lookup[t] for t in assignment.claim_texts if t in claim_lookup
]
return people
```
The body of the `link_claims_to_people` task sorts the incoming nodes back into people and claims, then splits attribution into its own LLM call over just the names and claim texts rather than the source document. `Assignments` is another plain response model. The returned assignments are resolved back to the actual objects through `claim_lookup` and hung off each matching `Person`, so the task returns people whose `claims` lists are populated — the edges the graph will get.
### Stage 5: Store the Objects and Read Their Provenance
```python theme={null}
# add_data_points persists nodes and edges to graph DB,
# and indexes embeddable fields in vector DB
await add_data_points(people)
lines = []
for person in people:
# source_content_hash is stamped by the pipeline provenance system;
# it carries the content hash of the source document this node came from
hash_display = person.source_content_hash or "N/A"
lines.append(f"{person.name} ({person.role}) [source_hash: {hash_display[:12]}]")
```
One call to `add_data_points` writes the people, the claims nested inside them, and the edges between them — plus the embeddings for every `Embeddable` field. Because the task runs inside a dataset pipeline, the nodes land in that dataset's graph and vector stores and are attributed to it, which is what lets `recall` find them later. The summary then reads `source_content_hash` back off each stored person: the same twelve characters on every line, because every node traces back to the one document that was ingested.
### Stage 6: Ingest, Then Run the Pipeline Against the Dataset
```python theme={null}
# Ingest the text into a dataset first. This creates the dataset, stores the
# text as a Data record with a content hash, and is what makes the graph the
# custom pipeline builds below both attributable and searchable.
await cognee.add(sample_text, dataset_name=DATASET_NAME)
# Run the custom pipeline over the dataset's ingested documents. With no
# `data` argument the first task receives the dataset's Data records.
await cognee.run_custom_pipeline(
tasks=[
Task(extract_entities),
Task(link_claims_to_people),
Task(store_and_summarize),
],
dataset=DATASET_NAME,
pipeline_name="entity_extraction",
)
```
`add()` does the ingestion work cognify would otherwise build on: it creates the `science_claims` dataset and stores the text as a `Data` record with a content hash. `run_custom_pipeline` then runs the three `Task` objects in order inside that dataset's context — each task's return value becomes the next task's first argument, and every document in the dataset goes through the chain. Running the pipeline against a dataset is also what gives the tasks a [pipeline context](/core-concepts/building-blocks/pipeline-context) with a dataset in it, so the stored nodes belong somewhere. The `pipeline_name` labels the run for logging and status tracking.
### Stage 7: Recall from the Hand-Built Graph
```python theme={null}
# Recall from the graph
print("\n--- Recall: 'Who worked on gravity?' ---")
answer = await cognee.recall(
"Who worked on gravity?",
query_type=cognee.SearchType.GRAPH_COMPLETION,
datasets=[DATASET_NAME],
)
print(f" {answer}")
```
Nothing about the query knows the graph was assembled by hand. `GRAPH_COMPLETION` traverses the `Person` and `ScientificClaim` nodes in the `science_claims` dataset exactly as it would traverse nodes produced by `cognify()`, which is the point of writing typed `DataPoint` models: custom extraction buys you a custom schema without giving up the ordinary retrieval path.
## Run It
```bash theme={null}
uv run python examples/demos/custom_pipelines/custom_pipeline_single_object_example.py
```
The run takes about a minute, most of it in the two extraction calls. Its output is walked through in [What to Expect](#what-to-expect) above.
## Adapting It to Your Data
The three-task shape generalizes to any typed extraction: define the `DataPoint` classes your domain actually has, annotate the field that should be searchable with `Embeddable` and the field that identifies a record with `Dedup`, then nest one model inside another wherever you want an edge. Give each `DataPoint` a matching plain Pydantic extraction model and build the DataPoints from what the LLM returns, as Stage 3 does. What changes per domain is the two system prompts and the response models; the pipeline around them stays as it is. To run it over a real corpus, `add()` the documents into the dataset instead of one string — every `Data` record in the dataset goes through the three tasks, and each stored node carries the hash of the document it came from.
<Columns>
<Card title="Custom Tasks and Pipelines" icon="workflow" href="/guides/custom-tasks-pipelines">
Writing your own tasks and running them as a pipeline, step by step.
</Card>
<Card title="DataPoints" icon="circle" href="/core-concepts/building-blocks/datapoints">
How `Embeddable`, `Dedup`, and nested models shape the graph a DataPoint becomes.
</Card>
<Card title="Custom Data Models" icon="circle-stop" href="/guides/custom-data-models">
More on `add_data_points` and modeling your own node types.
</Card>
<Card title="Low-Level LLM" icon="cpu" href="/guides/low-level-llm">
Calling `acreate_structured_output` directly, including Pydantic response models.
</Card>
</Columns>
# Vertical AI Agents
Source: https://docs.cognee.ai/examples/vertical-ai-agents
Use Cognee as the memory layer for vertical AI agents.
The future of AI is autonomous agents that execute complex, multi-step tasks in specialized domains. But agents without memory are agents without context. They can't learn from past interactions, can't understand organizational nuances, and can't improve over time.
Cognee provides the memory layer that makes agentic AI actually work.
## The Problem: Agents That Forget
Consider an AI agent designed to automate legal contract review. Without persistent memory, every document is a blank slate:
* The agent doesn't remember that your company uses specific non-standard clauses
* It can't recall that the counterparty had issues with similar terms last quarter
* It has no context about your organization's risk tolerance or negotiation patterns
## Why Memory Matters for Agents and What Cognee Brings
Agentic AI systems need three capabilities that standard RAG cannot provide:
### 1. Domain Understanding
The agent must understand how your enterprise works instead of only generic industry knowledge, in terms of your specific organizational structure, terminology, and processes.
### 2. Personalization
Each user, client, or session can have tailored context. The agent adapts its responses based on individual preferences, history, and past interactions stored in memory.
### 3. Dynamically Evolving Memory
As the agent operates, it should learn and improve. Patterns from successful task completions should inform future actions.
Our memory layer provides:
**Structured Context for Reasoning**
Rather than raw text chunks, agents receive graph-structured knowledge that captures relationships, hierarchies, and domain logic.
**Continuous Learning**
Through [`memify()`](/core-concepts/main-operations/legacy-operations/memify), agents run enrichment pipelines on the graph — for example, extracting coding rules or persisting session history — to consolidate experiences into persistent memory, improving task execution over time.
**Advanced Retrieval**
Multiple search types—graph completion, semantic chunks, summaries—let agents retrieve exactly the context they need for each decision.
### Example: Contract Review Agent with Memory
Define tools that give your agent persistent memory:
```python theme={null}
import cognee
from cognee import SearchType
# Tool 1: Remember information
async def remember(text: str):
"""Store information in long-term memory."""
await cognee.remember(text)
return "Saved to memory"
# Tool 2: Recall information
async def recall(query: str) -> list:
"""Search memory for relevant context."""
results = await cognee.recall(
query_text=query,
query_type=SearchType.GRAPH_COMPLETION,
)
return results
```
Wire them into your agent:
```python theme={null}
tools = [remember, recall]
agent = Agent(
model="gpt-4o",
system_prompt="You are a contract analyst. Use remember() to store important details and recall() to retrieve past context.",
tools=tools
)
```
Now the agent has memory:
```python theme={null}
# Session 1: Learn client preferences
agent.run("Remember: Acme Corp requires 30-day payment terms and California arbitration.")
# Session 2: Use memory for analysis
agent.run("Review this contract for Acme Corp: 60-day terms, New York jurisdiction.")
# Agent calls recall() → flags mismatches with stored preferences
```
## Integration with Agentic Frameworks
Cognee integrates with the frameworks you're already using:
* LangGraph, CrewAI, LlamaIndex, Agent Development Kit, etc.
* **Custom implementations**: Direct SDK integration with any agent framework
## Next Steps
Learn more about [Core Concepts](/core-concepts/overview) or review [Integrations](/integrations) for available options. If we don't have your favorite agent framework yet, let us know by [opening an issue on GitHub](https://github.com/topoteretes/cognee/issues).
# What you can build with Cognee
Source: https://docs.cognee.ai/examples/what-you-can-build
Concrete ways teams put Cognee to work.
Concrete ways teams put Cognee to work — find the pattern closest to yours.
<CardGroup>
<Card title="Company brain" icon="building-2">
Enterprise data shouldn't live in silos. Let every team member contribute documents to a shared memory, with permission controls where you need them. Cognee captures what was discussed, when, and by whom. Mark a colleague's insight as worth keeping, and cognee updates its weights so the right answer surfaces next time.
</Card>
<Card title="Sales & Deal Intelligence" icon="handshake">
Clients spread across calls, emails, and chat? Connect every channel to cognee and give each rep the full context of conversations they were never part of. Multiple CRMs collapse into a single source of truth, and most importantly your data is ready for your agents to pull the most up-to-date answer, every time.
</Card>
<Card title="Second brain" icon="brain">
Upload your documents, connect your email, and sync your meeting notes. Cognee will turn your scattered personal data into one connected memory. No more missed deadlines, no more fragmented files. Then build agents on top of it that actually act: surfacing next steps, flagging what's due, and remembering the things you forgot you knew.
</Card>
<Card title="Investment & Research" icon="trending-up">
You know some of the relationships between companies. Cognee finds the rest. Build a graph of the hidden and visible ties between firms and the people behind them, then trace every document and news article that mentions a given figure, and exactly how they are connected. Let your agents read the graph and make the call whether they should invest.
</Card>
<Card title="Docs & Manuals" icon="book-open">
Years of old manuals nobody can search? Hundreds of components, each with its own spec sheet? Drop them into cognee and just ask. It traverses the connections between parts, revisions, and diagrams to surface exactly what you need, in seconds.
</Card>
<Card title="Coding agents" icon="code">
Does your coding agent start from zero every time? Hitting the same bugs a teammate already solved last week? Plug your agents into cognee as shared memory across the whole engineering team. Now everyone's work is connected; so when a colleague spends two hours figuring out a Vercel deploy, your agent already knows the way through.
</Card>
</CardGroup>
# Docs for LLMs
Source: https://docs.cognee.ai/getting-started/docs-for-llms
Machine-readable exports of these docs for agents, LLMs, and RAG pipelines.
Every page on this site is also published as plain markdown, so you can hand the documentation to an LLM or agent instead of scraping HTML.
## Available endpoints
| Endpoint | What it contains |
| -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| [`/llms.txt`](https://docs.cognee.ai/llms.txt) | Curated index: recommended entry points plus links to the shard pages below. Small enough to drop into a system prompt. |
| [`/llms-full.txt`](https://docs.cognee.ai/llms-full.txt) | Full-site export — the complete text of every page in one file. Use this for RAG ingestion. |
| Any page URL + `.md` | A single page as markdown, for example [`/getting-started/quickstart.md`](https://docs.cognee.ai/getting-started/quickstart.md). |
If `llms-full.txt` is larger than the context you want to spend, use one of the per-surface shards instead. Each one lists the markdown URLs for a single area of the docs:
* [`/llms-core.md`](https://docs.cognee.ai/llms-core.md) — getting started, [core concepts](/core-concepts/overview), [setup](/setup-configuration/overview), guides, and examples
* [`/llms-cognee-cloud.md`](https://docs.cognee.ai/llms-cognee-cloud.md) — [Cognee Cloud](/cognee-cloud/overview)
* [`/llms-mcp.md`](https://docs.cognee.ai/llms-mcp.md) — [Cognee MCP](/cognee-mcp/mcp-overview) setup, tools, and client integrations
* [`/llms-integrations.md`](https://docs.cognee.ai/llms-integrations.md) — [third-party integrations](/integrations/index)
* [`/llms-api.md`](https://docs.cognee.ai/llms-api.md) — [REST](/api-reference/introduction) and [Python](/python-api/index) API references
## Copy a single page
Every page has a contextual menu next to its title with **Copy page**, **View as Markdown**, and shortcuts to open the page directly in ChatGPT, Claude, or Perplexity. Use it when you only need one page in an existing chat.
## Fetch from an agent
Any of these endpoints is a plain HTTP GET, so a tool-using agent can pull them on demand:
```bash theme={null}
curl https://docs.cognee.ai/llms.txt
curl https://docs.cognee.ai/getting-started/quickstart.md
```
## Ingest the docs into Cognee
You can also give the docs to Cognee itself and query them with [`recall`](/core-concepts/main-operations/recall):
```python theme={null}
import asyncio
import httpx
import cognee
async def main():
docs = httpx.get("https://docs.cognee.ai/llms-full.txt", timeout=60).text
await cognee.remember(docs, dataset_name="cognee_docs")
results = await cognee.recall(query_text="How do I configure a local LLM provider?")
for result in results:
print(result.text)
if __name__ == "__main__":
asyncio.run(main())
```
<Note>
For a setup-focused assistant you usually want the [LLM Quickstart Skill](/getting-started/llm-quickstart-skill) rather than the full docs export — it is a compact, curated set of instructions for installing and configuring Cognee.
</Note>
# Installation
Source: https://docs.cognee.ai/getting-started/installation
Install Cognee and configure the basics for your first memory workflow.
Set up your environment and install Cognee to start building AI memory.
<Info>
Python **3.10 – 3.14** is required to run Cognee.
</Info>
<Tip>
Using Claude Code or another LLM to set up Cognee? Copy the [LLM Quickstart Skill](/getting-started/llm-quickstart-skill) first so the assistant checks Python versions, provider settings, extras, and the first smoke test in the right order.
</Tip>
## Setup Notes
<AccordionGroup>
<Accordion title="Environment Configuration">
* We recommend creating a `.env` file in your project root
* Cognee supports many configuration options, and a `.env` file keeps them organized
</Accordion>
<Accordion title="API Keys & Models">
You have two main options for configuring LLM and embedding providers:
**Option 1: OpenAI (Simplest)**
* Single API key handles both LLM and embeddings
* Uses `openai/gpt-5-mini` for LLM and `openai/text-embedding-3-large` for embeddings by default
* Works out of the box with minimal configuration
**Option 2: Other Providers**
* Configure both LLM and embedding providers separately
* Supports Gemini, Anthropic, Ollama, and more
* Requires setting both `LLM_*` and `EMBEDDING_*` variables
<Info>
By default, Cognee uses OpenAI for both LLMs and embeddings. If you change the LLM provider but don't configure embeddings, it will still default to OpenAI.
</Info>
</Accordion>
<Accordion title="Virtual Environment">
We recommend creating a virtual environment before installing Cognee. Use whichever tool you prefer — [uv](https://github.com/astral-sh/uv) is fast, but the standard library `venv` works just as well if you don't use uv.
The activation command depends on your shell, not just your operating system. `source` is Unix-only; on Windows, PowerShell and Command Prompt each have their own activation script inside `.venv\Scripts\`.
<Tabs>
<Tab title="macOS / Linux">
```bash theme={null}
# uv
uv venv
source .venv/bin/activate
# or standard venv + pip
python -m venv .venv
source .venv/bin/activate
```
</Tab>
<Tab title="Windows (PowerShell)">
```powershell theme={null}
# uv
uv venv
.\.venv\Scripts\Activate.ps1
# or standard venv + pip
python -m venv .venv
.\.venv\Scripts\Activate.ps1
```
</Tab>
<Tab title="Windows (Command Prompt)">
```cmd theme={null}
:: uv
uv venv
.venv\Scripts\activate.bat
:: or standard venv + pip
python -m venv .venv
.venv\Scripts\activate.bat
```
`activate.bat` is the Command Prompt script — it does not work in PowerShell, and `Activate.ps1` does not work in Command Prompt.
</Tab>
</Tabs>
If PowerShell blocks activation with an execution-policy error, or `import cognee` later fails because the wrong interpreter is active, see the **Windows Setup** accordion below.
</Accordion>
<Accordion title="uv Projects (uv add)">
If your project is managed by uv — it has a `pyproject.toml`, usually created with `uv init` — use `uv add` instead of `uv pip install`. It records Cognee in your `pyproject.toml` dependencies, updates `uv.lock`, and installs it into the project's `.venv`:
```bash theme={null}
uv add cognee
```
Extras go in the package spec exactly as with pip. Quote it so your shell does not interpret the brackets:
```bash theme={null}
uv add "cognee[postgres]"
uv add "cognee[neo4j,aws]"
```
Then run your code with `uv run python your_script.py`, which uses the project environment without manual activation.
Two things to know:
* `uv add` requires a uv project. Run from a directory with no `pyproject.toml` (a plain `uv venv` environment, for example), it fails — use `uv pip install cognee` there instead.
* Cognee requires Python `>=3.10,<3.15`, so your project's own `requires-python` must stay inside that range or resolution fails.
</Accordion>
<Accordion title="Windows Setup">
On Windows the setup steps differ slightly from Linux/macOS.
<AccordionGroup>
<Accordion title="Install uv">
Install uv with the official standalone installer, which adds uv to your `PATH` automatically:
<Tabs>
<Tab title="PowerShell">
```powershell theme={null}
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
```
</Tab>
<Tab title="Command Prompt (CMD)">
Run the PowerShell command above, or download and run the installer from [the uv installation guide](https://docs.astral.sh/uv/getting-started/installation/).
</Tab>
</Tabs>
<Warning>
If you ran `pip install uv` and then hit `uv : The term 'uv' is not recognized...`, the package installed but Python's scripts folder is not available on your `PATH`. You can either reinstall uv with the standalone installer above, add Python's scripts folder to `PATH`, or call uv through Python instead:
```powershell theme={null}
python -m uv pip install cognee
```
</Warning>
</Accordion>
<Accordion title="Create and Activate the Virtual Environment">
Once `uv --version` works, create the environment and activate it with the script that matches your shell — PowerShell uses `.venv\Scripts\Activate.ps1`, Command Prompt uses `.venv\Scripts\activate.bat`:
<Tabs>
<Tab title="PowerShell">
```powershell theme={null}
uv venv
.\.venv\Scripts\Activate.ps1
```
If you see an execution-policy error, run this first (current user only):
```powershell theme={null}
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
```
</Tab>
<Tab title="Command Prompt (CMD)">
```cmd theme={null}
uv venv
.venv\Scripts\activate.bat
```
</Tab>
</Tabs>
</Accordion>
<Accordion title="Verify the Python Interpreter">
After installing Cognee in the [Setup](#setup) section, confirm the active interpreter can import it:
<Tabs>
<Tab title="PowerShell">
```powershell theme={null}
python -c "import cognee; print(cognee.__file__)"
```
</Tab>
<Tab title="Command Prompt (CMD)">
```cmd theme={null}
python -c "import cognee; print(cognee.__file__)"
```
</Tab>
</Tabs>
The printed path should point inside your `.venv` folder.
A common Windows error is `ModuleNotFoundError: No module named 'cognee'` even though the install succeeded. This happens when the script runs with system Python instead of the venv interpreter — for example after opening a new terminal without re-activating, double-clicking a `.py` file, or an IDE configured to use the global interpreter.
First confirm which Python the active terminal uses. The `.venv\Scripts\python.exe` path should be selected:
<Tabs>
<Tab title="PowerShell">
```powershell theme={null}
Get-Command python
```
</Tab>
<Tab title="Command Prompt (CMD)">
```cmd theme={null}
where python
```
</Tab>
</Tabs>
If it does not, re-activate the environment in that terminal:
<Tabs>
<Tab title="PowerShell">
```powershell theme={null}
.\.venv\Scripts\Activate.ps1
```
</Tab>
<Tab title="Command Prompt (CMD)">
```cmd theme={null}
.venv\Scripts\activate.bat
```
</Tab>
</Tabs>
To bypass activation entirely, call the venv interpreter explicitly when running your script:
<Tabs>
<Tab title="PowerShell">
```powershell theme={null}
.\.venv\Scripts\python.exe your_script.py
```
</Tab>
<Tab title="Command Prompt (CMD)">
```cmd theme={null}
.venv\Scripts\python.exe your_script.py
```
</Tab>
</Tabs>
In an IDE (VS Code, PyCharm), select the `.venv` interpreter as the project interpreter so the **Run** button uses it.
</Accordion>
<Accordion title="Configure Environment Files and Paths">
Copy the template from the project root, then open it in any text editor (Notepad, VS Code, etc.):
<Tabs>
<Tab title="PowerShell">
```powershell theme={null}
Copy-Item .env.template .env
```
</Tab>
<Tab title="Command Prompt (CMD)">
```cmd theme={null}
copy .env.template .env
```
</Tab>
</Tabs>
The `.env` file must be saved in the **project root** — the same directory from which you run Python. Cognee calls `load_dotenv()` at import time and searches upward from the working directory.
When setting `DATA_ROOT_DIRECTORY` or `SYSTEM_ROOT_DIRECTORY` in your `.env` file, use **forward slashes** or **double backslashes** — single backslashes are not valid in `.env` values:
```ini theme={null}
# Forward slashes (recommended)
DATA_ROOT_DIRECTORY="C:/Users/YourName/cognee/.cognee_data"
SYSTEM_ROOT_DIRECTORY="C:/Users/YourName/cognee/.cognee_system"
# Or double backslashes
DATA_ROOT_DIRECTORY="C:\\Users\\YourName\\cognee\\.cognee_data"
```
A `~` home-directory prefix also works and is cross-platform:
```ini theme={null}
DATA_ROOT_DIRECTORY="~/.cognee_data"
```
If you prefer to set variables directly in your shell session instead of using a file:
<Tabs>
<Tab title="PowerShell">
```powershell theme={null}
$env:LLM_API_KEY = "your_openai_api_key"
```
</Tab>
<Tab title="Command Prompt (CMD)">
```cmd theme={null}
set LLM_API_KEY=your_openai_api_key
```
</Tab>
</Tabs>
<Warning>
Variables set this way are session-scoped and lost when the terminal closes. A `.env` file is recommended for persistent configuration.
</Warning>
Python-dotenv handles both Windows (CRLF) and Unix (LF) line endings automatically, so line endings are not a concern.
</Accordion>
<Accordion title="Graph Database Fails to Open (`Could not find lbug C API shared library`)">
If the first call that opens the graph database fails with:
```
RuntimeError: Could not find lbug C API shared library.
```
the cause is missing OpenSSL DLLs: the Windows wheels of `ladybug` (Cognee's default graph database) stopped shipping the OpenSSL 3 libraries their native extension loads, so the extension fails to import and `ladybug` falls back to a backend whose shared library is not distributed at all.
**Cognee handles this for you.** It supplies those libraries from CPython's own OpenSSL 3 — copying them into a per-interpreter cache directory under the names the extension expects and adding that directory to the DLL search path — automatically, at import time, in the main process and in every database worker process. There is nothing to install, configure, or set in `.env`. The workaround turns itself off on non-Windows platforms and on installs whose `ladybug` wheel ships its own OpenSSL.
The one case it cannot cover is an interpreter that has no OpenSSL 3 to borrow: CPython 3.10 on Windows links OpenSSL 1.1 instead, and embedded distributions ship no `DLLs` folder at all. Cognee supports Python 3.10, so if you see this error, check which version the environment runs:
<Tabs>
<Tab title="PowerShell">
```powershell theme={null}
python --version
```
</Tab>
<Tab title="Command Prompt (CMD)">
```cmd theme={null}
python --version
```
</Tab>
</Tabs>
If it reports **3.10**, that is the cause. On Windows, use **Python 3.11 or newer** from a standard python.org or uv-managed installation — not an embedded distribution. Recreate the virtual environment with that interpreter and reinstall Cognee:
<Tabs>
<Tab title="PowerShell">
```powershell theme={null}
uv venv --python 3.12
.\.venv\Scripts\Activate.ps1
uv pip install cognee
```
</Tab>
<Tab title="Command Prompt (CMD)">
```cmd theme={null}
uv venv --python 3.12
.venv\Scripts\activate.bat
uv pip install cognee
```
</Tab>
</Tabs>
</Accordion>
</AccordionGroup>
</Accordion>
<Accordion title="Optional">
<AccordionGroup>
<Accordion title="Database">
* PostgreSQL database is required if you plan to use PostgreSQL as your relational database (requires `postgres` extra)
</Accordion>
</AccordionGroup>
</Accordion>
</AccordionGroup>
## Setup
<Tabs>
<Tab title="OpenAI (Recommended)">
<Card>
**Environment:** Add your OpenAI API key to your `.env` file:
```bash theme={null}
LLM_API_KEY="your_openai_api_key"
```
**Installation:** Install Cognee with the default package, using whichever tool manages your environment. With `pip` or `uv pip`, activate the virtual environment first; in a uv project, `uv add` installs into the project's `.venv` directly:
<CodeGroup>
```bash pip theme={null}
pip install cognee
```
```bash uv virtual environment theme={null}
uv pip install cognee
```
```bash uv project (pyproject.toml) theme={null}
uv add cognee
```
</CodeGroup>
**What this gives you**: Cognee installed with default local databases (SQLite, LanceDB, Kuzu) — no external servers required.
<Info>
This single API key handles both LLM and embeddings. The defaults are `openai/gpt-5-mini` for the LLM and `openai/text-embedding-3-large` (3072 dimensions) for embeddings.
</Info>
</Card>
</Tab>
<Tab title="Other Providers (Gemini, Anthropic, etc.)">
<Card>
**Environment:** Configure both LLM and embedding providers in your `.env` file. Here is an example for Gemini:
```bash theme={null}
# LLM
LLM_PROVIDER="gemini"
LLM_MODEL="gemini/gemini-flash-latest"
LLM_API_KEY="your_gemini_api_key"
# Embeddings
EMBEDDING_PROVIDER="gemini"
EMBEDDING_MODEL="gemini/gemini-embedding-001"
EMBEDDING_API_KEY="your_gemini_api_key"
```
<Info>
Make sure to configure both LLM and embedding settings. If you only set one, the other will default to OpenAI.
</Info>
**Installation:** Install Cognee, then add provider-specific extras only when needed:
| Provider path | Install command |
| ------------------------------- | ---------------------------------------- |
| Gemini through Google AI Studio | No extra package |
| Gemini through Vertex AI | `uv pip install google-cloud-aiplatform` |
| Anthropic | `uv pip install "cognee[anthropic]"` |
| Ollama | `uv pip install "cognee[ollama]"` |
| Groq | `uv pip install "cognee[groq]"` |
| Mistral | `uv pip install "cognee[mistral]"` |
**What this gives you**: Cognee installed with your chosen providers and default local databases.
For detailed configuration options, see our [LLM](/setup-configuration/llm-providers) and [Embeddings](/setup-configuration/embedding-providers) guides.
</Card>
</Tab>
<Tab title="Ollama (Local, No API Key)">
<Card>
**Environment:** Run both the LLM and embeddings locally with [Ollama](https://ollama.ai). Configure both providers in your `.env` file. Unlike cloud providers, Ollama needs a local base URL (`LLM_ENDPOINT` / `EMBEDDING_ENDPOINT`) pointing at your running Ollama server:
```bash theme={null}
# LLM — Ollama
LLM_PROVIDER="ollama"
LLM_MODEL="llama3.1:8b"
LLM_ENDPOINT="http://localhost:11434"
LLM_API_KEY="ollama"
# Embeddings — Ollama
EMBEDDING_PROVIDER="ollama"
EMBEDDING_MODEL="nomic-embed-text:latest"
EMBEDDING_ENDPOINT="http://localhost:11434/api/embed"
EMBEDDING_DIMENSIONS="768"
HUGGINGFACE_TOKENIZER="nomic-ai/nomic-embed-text-v1.5" # optional, recommended for accurate token counting
```
`LLM_API_KEY="ollama"` is a required placeholder — Ollama ignores the value, but Cognee needs it non-empty. `LLM_ENDPOINT` is the bare host with no path suffix — on the default structured output backend, LiteLLM appends Ollama's native `/api/generate` path itself, so a trailing `/v1` fails with a 404 (see [LLM Providers → Ollama](/setup-configuration/llm-providers#ollama-local)). `EMBEDDING_ENDPOINT` is handled separately and keeps its `/api/embed` path. `HUGGINGFACE_TOKENIZER` is the HuggingFace repo ID of the tokenizer matching your embedding model; it is optional — Cognee no longer requires it at startup — but recommended for accurate token counting.
**Installation:** Install Cognee with the Ollama extra, then pull the models:
```bash theme={null}
uv pip install "cognee[ollama]"
ollama pull llama3.1:8b
ollama pull nomic-embed-text:latest
```
**What this gives you**: A fully local setup — no cloud account or API key required.
<Info>
Configure **both** LLM and embeddings to a local backend. If you set only one, the other defaults to OpenAI. See the [Local Setup guide](/guides/local-setup) for an Ollama LLM + Fastembed alternative and troubleshooting.
</Info>
</Card>
</Tab>
</Tabs>
## Extras and Common Installation Combinations
Cognee's base installation (`pip install cognee`) includes everything needed to run with OpenAI and the default local databases (SQLite, LanceDB, Kuzu). Optional extras unlock additional providers, integrations, and features.
Install one or more extras with:
```bash theme={null}
pip install "cognee[extra1,extra2]"
# or with uv:
uv pip install "cognee[extra1,extra2]"
# or, in a uv project:
uv add "cognee[extra1,extra2]"
```
The tables below use `uv pip install`; the same package specs work with `pip install` and with `uv add` in a uv project (see the **uv Projects** note above).
<AccordionGroup>
<Accordion title="Common installation combinations">
If you already know the stack you want, these combinations cover the most common setups:
| Use case | Install |
| -------------------------------------------------------------------- | ------------------------------------------------- |
| PostgreSQL as the database backend | `uv pip install "cognee[postgres]"` |
| Neo4j graph store + AWS S3 storage | `uv pip install "cognee[neo4j,aws]"` |
| Code graph analysis | `uv pip install "cognee[codegraph]"` |
| OpenTelemetry tracing | `uv pip install "cognee[tracing]"` |
| Web scraping + extended document formats | `uv pip install "cognee[scraping,docs]"` |
| Gmail inbox ingestion | `uv pip install cognee-community-connector-gmail` |
| BAML structured output backend | `uv pip install "cognee[baml]"` |
| Anthropic Claude models | `uv pip install "cognee[anthropic]"` |
| LLM-free graph building (local GLiNER extraction + local embeddings) | `uv pip install "cognee[gliner,fastembed]"` |
</Accordion>
<Accordion title="LLM & Embedding Providers">
These extras install provider SDKs. You still need to set the corresponding environment variables. See [LLM Providers](/setup-configuration/llm-providers) and [Embedding Providers](/setup-configuration/embedding-providers).
| Extra | Packages installed | When to use |
| ------------- | -------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `anthropic` | `anthropic>=0.27` | Use Claude models (claude-3-5-sonnet, etc.) |
| `groq` | `groq>=0.8.0,<1.0.0` | Use Groq-hosted inference |
| `mistral` | `mistral-common`, `mistralai` | Use Mistral AI models |
| `huggingface` | `transformers>=4.46.3,<5` | Use HuggingFace models for LLM or embeddings |
| `ollama` | `transformers>=4.46.3,<5` | Use Ollama for local model serving |
| `llama-cpp` | `llama-cpp-python[server]>=0.3.0` | Run GGUF models locally via llama.cpp |
| `azure` | `azure-identity>=1.15.0,<2` | Azure OpenAI or other Azure-hosted models |
| `fastembed` | `fastembed<=0.8.0`, `onnxruntime<=1.23.2` (Python \< 3.14) or `onnxruntime>=1.24.1` (Python 3.14+) | Fast local embeddings without a GPU |
| `gliner` | `gliner2[local]>=2.0.0,<3`, `protobuf>=5.29.6`, `sentencepiece>=0.2.0` | Build the knowledge graph and chunk summaries with a local GLiNER2 model instead of the LLM (`GRAPH_EXTRACTOR=gliner`). Pulls in torch, and downloads \~800 MB of model weights on first use. See [LLM-free extraction with GLiNER](/python-api/cognify#llm-free-extraction-with-gliner) |
<Info>
There is no separate `gemini` extra. Gemini through Google AI Studio is supported through `litellm`, which is already part of the base installation. Vertex AI for Gemini additionally requires `google-cloud-aiplatform`.
</Info>
</Accordion>
<Accordion title="Vector & Graph Stores">
| Extra | Packages installed | When to use |
| ----------------- | ---------------------------------------- | ------------------------------------------------------------ |
| `postgres` | `psycopg2`, `pgvector`, `asyncpg` | Use PostgreSQL as relational DB and pgvector as vector store |
| `postgres-binary` | `psycopg2-binary`, `pgvector`, `asyncpg` | Same as `postgres` but uses pre-compiled binary wheels |
| `neo4j` | `neo4j>=5.28.0,<6` | Use Neo4j as the graph store |
| `neptune` | `langchain_aws>=0.2.22` | Use Amazon Neptune as the graph store |
| `chromadb` | `chromadb>=0.6,<0.7`, `pypika` | Use ChromaDB as the vector store |
</Accordion>
<Accordion title="Data Ingestion & Processing">
| Extra | Packages installed | When to use |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| `docs` | `unstructured` (with csv, doc, docx, epub, md, ppt, pptx, xlsx, pdf, and more), `lxml` | Parse Office documents, PDFs via unstructured, and other rich formats beyond the built-in PyPDF support |
| `docling` | `docling>=2.54`, `transformers>=4.55` | Use Docling for advanced document parsing |
| `scraping` | `tavily-python`, `beautifulsoup4`, `playwright`, `lxml`, `protego`, `APScheduler` | Web scraping, URL ingestion, and scheduled crawling |
| `codegraph` | `fastembed`, `transformers` | Build code graphs from Python repositories |
| `langchain` | `langsmith`, `langchain_text_splitters`, `langchain-core` | Use LangChain text splitters or LangSmith tracing |
| `llama-index` | `llama-index-core>=0.14.20,<0.15` | Use LlamaIndex data loaders and connectors |
| `dlt` | `dlt[sqlalchemy]>=1.9.0,<2` | Ingest data via DLT pipelines |
| `Gmail connector` | `cognee-community-connector-gmail`, `dlt[sqlalchemy]>=1.9.0,<2`, `google-api-python-client>=2.100.0,<3`, `google-auth>=2.23.0,<3`, `google-auth-oauthlib>=1.1.0,<2` | Ingest Gmail messages into memory (read-only OAuth, incremental sync) |
</Accordion>
<Accordion title="Infrastructure & Storage">
| Extra | Packages installed | When to use |
| ------- | ----------------------- | ----------------------------------------------------------------- |
| `redis` | `redis>=5.0.3,<6.0.0` | Use Redis for caching instead of the default in-memory/disk cache |
| `aws` | `s3fs[boto3]==2025.3.2` | Use Amazon S3 for file storage |
| `baml` | `baml-py==0.206.0` | Use BAML as a structured output backend |
</Accordion>
<Accordion title="Observability & Monitoring">
| Extra | Packages installed | When to use |
| --------- | ---------------------------------------------------------------------- | --------------------------------------------------------- |
| `tracing` | `opentelemetry-api`, `opentelemetry-sdk`, OTLP exporters (gRPC + HTTP) | Export traces via OpenTelemetry to any compatible backend |
| `posthog` | `posthog>=3.5.0,<4` | Send usage analytics to PostHog |
</Accordion>
<Accordion title="Evaluation">
| Extra | Packages installed | When to use |
| ---------- | --------------------------------------------------------- | ----------------------------------------------------- |
| `deepeval` | `deepeval>=3.0.1,<4` | Run LLM evaluation benchmarks with DeepEval |
| `evals` | `plotly`, `gdown`, `pandas`, `matplotlib`, `scikit-learn` | Internal evaluation tooling with plotting and metrics |
</Accordion>
<Accordion title="Development & Tooling">
| Extra | Packages installed | When to use |
| ---------- | ------------------------------------------------ | ------------------------------------------------------------------- |
| `notebook` | `notebook>=7.1.0,<8` | Run Jupyter notebooks |
| `dev` | pytest, mypy, ruff, pre-commit, mkdocs, and more | Full development environment for contributing to cognee |
| `debug` | `debugpy>=1.8.9,<2.0.0` | Attach a remote debugger (e.g. VS Code) to a running cognee process |
</Accordion>
<Accordion title="Missing dependency errors (ImportError)">
If you encounter an `ImportError` when using a cognee feature, it usually means a required extra has not been installed.
| ImportError mentions | Install |
| ------------------------------------------------------------------------------------ | ----------------------------------------------- |
| `neo4j` | `cognee[neo4j]` |
| `playwright`, `tavily`, `beautifulsoup4` | `cognee[scraping]` |
| `unstructured` | `cognee[docs]` |
| `docling` | `cognee[docling]` |
| `fastembed` | `cognee[fastembed]` or `cognee[codegraph]` |
| `gliner2` | `cognee[gliner]` |
| `psycopg2`, `asyncpg`, `pgvector` | `cognee[postgres]` or `cognee[postgres-binary]` |
| `redis` | `cognee[redis]` |
| `s3fs`, `boto3` | `cognee[aws]` |
| `baml` | `cognee[baml]` |
| `anthropic` | `cognee[anthropic]` |
| `groq` | `cognee[groq]` |
| `mistralai` | `cognee[mistral]` |
| `llama_cpp` | `cognee[llama-cpp]` |
| `opentelemetry` | `cognee[tracing]` |
| `chromadb` | `cognee[chromadb]` |
| `deepeval` | `cognee[deepeval]` |
| `dlt` | `cognee[dlt]` |
| `googleapiclient`, `google-api-python-client`, `google-auth`, `google-auth-oauthlib` | `cognee-community-connector-gmail` |
</Accordion>
</AccordionGroup>
## Verifying Release Authenticity
Cognee's release pipeline attaches supply-chain provenance to the packages it publishes: PEP 740 attestations shown in the "Provenance" section of the [PyPI project page](https://pypi.org/project/cognee/), plus a GitHub-hosted SLSA build provenance attestation. To confirm a downloaded wheel or sdist was built by CI from `topoteretes/cognee`:
```bash theme={null}
gh attestation verify ./cognee-<version>-py3-none-any.whl --repo topoteretes/cognee
```
### Verifying from the GitHub release page
The release pipeline also attaches the built distributions and their attestation to the [GitHub release](https://github.com/topoteretes/cognee/releases) for the tag, so you can verify without querying GitHub's attestation store:
| Asset | What it is |
| -------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| `cognee-<version>-py3-none-any.whl`, `cognee-<version>.tar.gz` | The distributions that were published to PyPI |
| `cognee-<version>.sigstore.json` | The Sigstore bundle — certificate, signature, and in-toto statement |
| `cognee-<version>.intoto.jsonl` | The DSSE envelope extracted from that bundle, the shape the SLSA ecosystem expects |
Point `gh attestation verify` at the bundle instead of the repo:
```bash theme={null}
gh attestation verify cognee-<version>-py3-none-any.whl \
--bundle cognee-<version>.sigstore.json \
--repo topoteretes/cognee
```
The upload runs only after the PyPI publish succeeds, so a release page never carries artifacts that did not ship. The nightly PyPI canaries (`X.Y.Z.devYYYYMMDD`) create no GitHub release and have no assets to verify; `.devN` pre-releases cut from `dev` do get them.
Releases published before the provenance pipeline landed (August 2026) carry no attestations, so verification has nothing to check for them; releases cut before this asset upload landed have the attestations but no release-page assets, so use the `--repo` form above for those. For the full mechanism, see [Supply-chain provenance & release attestations](https://github.com/topoteretes/cognee/blob/dev/docs/supply_chain_provenance.md) in the cognee repo.
## Next Steps
<CardGroup>
<Card title="Run Your First Example" href="/getting-started/quickstart" icon="play">
**Quickstart Tutorial**
Get started with Cognee by running your first knowledge graph example.
</Card>
<Card title="Explore Advanced Features" href="/core-concepts" icon="compass">
**Core Concepts**
Dive deeper into Cognee's powerful features and capabilities.
</Card>
</CardGroup>
# Introduction
Source: https://docs.cognee.ai/getting-started/introduction
Learn how Cognee turns your data into searchable AI memory.
<img alt="Diagram showing raw documents becoming chunks, extracted entities, derived concepts, induced ontologies, and searchable memory through remember, improve, and recall." />
<img alt="Diagram showing raw documents becoming chunks, extracted entities, derived concepts, induced ontologies, and searchable memory through remember, improve, and recall." />
Give Cognee your documents, and it creates a graph of raw information, extracted concepts, and meaningful relationships you can query.
## What is Cognee?
Cognee is a **memory layer** for AI agents and tools. It is **not an LLM**, and it is **more than plain RAG**. You keep using whatever model you already have (Claude, GPT, a local model); Cognee sits between your data and that model and gives it long-term, connected memory.
* **Not a model.** Cognee calls an LLM to build and query memory, but it does not replace your model. It prepares the right context for each call your agent makes.
* **More than RAG.** Classic RAG embeds text chunks and retrieves by similarity. Cognee adds a knowledge graph — a "brain" of entities and the relationships between them — on top, so recall can follow connections, not just match text.
* **A layer you plug in.** Use it from Python, the [CLI](/cognee-cli/overview), a [REST API](/guides/deploy-rest-api-server), direct integrations like [Claude Code](/integrations/claude-code-integration), or MCP-compatible clients like [Cursor](/cognee-mcp/integrations/cursor) and [Cline](/cognee-mcp/integrations/cline) through the [Cognee MCP server](/cognee-mcp/mcp-overview).
**When is the "brain" built?** When you store data as permanent memory, Cognee runs the ingestion and graph-building pipeline before that data becomes recallable. [Session memory](/core-concepts/sessions-and-caching) is the fast short-term path, and you can later bridge it into the permanent graph with [`.improve`](/core-concepts/main-operations/improve).
## Why AI memory matters
When you call an LLM, each request is stateless: it doesn't remember what happened in the last call, and it doesn't know about the rest of your documents.
That makes it hard to build applications that actually use your documents and carry context forward. You need a memory layer that can link your documents together and create the right context for every LLM call.
## How Cognee works
Cognee v1.0 exposes four operations that cover the full memory lifecycle:
* **[`.remember`](/core-concepts/main-operations/remember) — Store data in memory:**
Give Cognee text, files, or URLs. It ingests, chunks, extracts entities, and builds the knowledge graph for you in one call. Supports permanent graph memory or fast session memory.
* **[`.recall`](/core-concepts/main-operations/recall) — Query memory:**
Ask a question in natural language. Cognee picks the best retrieval strategy automatically, or you can specify one. Works across the permanent graph and session cache.
* **[`.improve`](/core-concepts/main-operations/improve) — Enrich existing memory:**
Runs enrichment passes on an already-built graph. With session IDs, it bridges short-term session memory into the permanent graph and applies feedback-based weighting.
* **[`.forget`](/core-concepts/main-operations/forget) — Remove memory:**
Delete a specific data item, an entire dataset, or everything owned by the current user.
If you want direct control over each pipeline step, the lower-level [Add](/core-concepts/main-operations/legacy-operations/add), [Cognify](/core-concepts/main-operations/legacy-operations/cognify), [Search](/core-concepts/main-operations/legacy-operations/search), and [Memify](/core-concepts/main-operations/legacy-operations/memify) operations are still available.
## License
Cognee is open source and released under the [Apache License 2.0](https://github.com/topoteretes/cognee/blob/main/LICENSE). You are free to use, modify, and distribute it in both personal and commercial projects, subject to the terms of the license.
## Ready to get started?
<CardGroup>
<Card title="Set up your environment" href="/getting-started/installation" icon="download">
**Installation Guide**
Set up your environment and install Cognee to start building AI memory.
</Card>
<Card title="Run your first example" href="/getting-started/quickstart" icon="play">
**Quickstart Tutorial**
Get started with Cognee by running your first knowledge graph example.
</Card>
<Card title="Keep exploring" href="/core-concepts/overview" icon="compass">
**Core Concepts**
Dive deeper into Cognee's powerful features and capabilities.
</Card>
</CardGroup>
# LLM Quickstart Skill
Source: https://docs.cognee.ai/getting-started/llm-quickstart-skill
Copy a Claude-ready skill to help an LLM set up Cognee.
Give this skill to Claude Code or another LLM assistant when you want help setting up Cognee. It focuses on the common first-run friction points: Python version, virtual environments, provider configuration, optional extras, and the first `remember` / `recall` smoke test.
## Install in Claude Code
Create a project skill at `.claude/skills/cognee-quickstart/SKILL.md` and copy the full block below into it.
```bash theme={null}
mkdir -p .claude/skills/cognee-quickstart
```
## Copy the skill
````markdown theme={null}
---
name: cognee-quickstart
description: Help users start or troubleshoot a Cognee project by choosing the right installation path, creating a clean Python environment, configuring LLM and embedding providers, installing optional extras, running the first remember/recall smoke test, and resolving dependency setup errors.
---
# Cognee Quickstart
Use this skill when the user is starting with Cognee, setting up dependencies, running Cognee from source, configuring providers, using local models, or reporting install/import/API key errors.
## First move
Classify the setup path before installing anything:
- **Fastest package setup**: User wants to try Cognee in a new app or notebook.
- **Provider-specific setup**: User wants OpenAI, Gemini, Anthropic, Ollama, Fastembed, or another backend.
- **Source setup**: User cloned `topoteretes/cognee` to inspect examples, change code, or contribute.
- **Existing broken setup**: User already has errors, stale environments, or dependency conflicts.
If the path is unclear, ask one short question. Otherwise make the conservative default: package install in a fresh virtual environment using OpenAI defaults.
## Environment rules
- Work from the project root, where the user's `.env` should live.
- Require Python 3.10 through 3.14. Check `python --version` or `python3 --version`.
- Prefer `uv` for environment and package work. Use `python -m venv` and `pip` only when `uv` is unavailable.
- Do not install packages globally unless the user explicitly asks.
- Do not mix package managers in the same environment.
- Treat `.env` values as secrets. Never print real API keys back to the user.
## Fastest package setup
Use this path for most first-time users with an OpenAI API key:
```bash
uv venv
source .venv/bin/activate
uv pip install cognee
```
Create `.env` in the project root:
```dotenv
LLM_API_KEY="your_openai_api_key"
```
Verify import before running a full example:
```bash
python -c "import cognee; print('cognee import ok')"
```
Then run the smoke test:
```python
import asyncio
import cognee
async def main():
await cognee.forget(everything=True)
await cognee.remember("Cognee turns documents into AI memory.")
results = await cognee.recall(query_text="What does Cognee do?")
for result in results:
print(result.text)
if __name__ == "__main__":
asyncio.run(main())
```
## Provider setup
The most common configuration mistake is setting only an LLM provider or only an embedding provider. If the user is not using OpenAI defaults, configure both sides explicitly.
### Gemini example
```dotenv
LLM_PROVIDER="gemini"
LLM_MODEL="gemini/gemini-flash-latest"
LLM_API_KEY="your_gemini_api_key"
EMBEDDING_PROVIDER="gemini"
EMBEDDING_MODEL="gemini/gemini-embedding-001"
EMBEDDING_API_KEY="your_gemini_api_key"
```
### Anthropic LLM example
Anthropic is an LLM provider, not an embedding provider. Pair it with an embedding provider such as OpenAI, Gemini, or Fastembed.
```bash
uv pip install "cognee[anthropic]"
```
```dotenv
LLM_PROVIDER="anthropic"
LLM_MODEL="anthropic/<your_claude_model>"
LLM_API_KEY="your_anthropic_api_key"
EMBEDDING_PROVIDER="openai"
EMBEDDING_MODEL="openai/text-embedding-3-small"
EMBEDDING_API_KEY="your_openai_api_key"
```
### Local no-API-key example
Use Ollama for the LLM and Fastembed for CPU-friendly embeddings:
```bash
ollama pull llama3.1:8b
```
```dotenv
LLM_PROVIDER="ollama"
LLM_MODEL="llama3.1:8b"
LLM_ENDPOINT="http://localhost:11434"
LLM_API_KEY="ollama"
EMBEDDING_PROVIDER="fastembed"
EMBEDDING_MODEL="sentence-transformers/all-MiniLM-L6-v2"
EMBEDDING_DIMENSIONS="384"
```
`LLM_ENDPOINT` is the bare host — no `/v1` suffix; on the default structured output backend LiteLLM appends Ollama's native `/api/generate` path itself, and a trailing `/v1` fails with a 404.
If the user changes embedding providers or dimensions after a previous run, advise a local reset before re-running:
```python
import asyncio
import cognee
async def main():
await cognee.prune.prune_system(metadata=True)
asyncio.run(main())
```
## Source setup
Use this path when the user cloned the Cognee repository:
```bash
git clone https://github.com/topoteretes/cognee.git
cd cognee
uv venv
source .venv/bin/activate
uv pip install -e ".[dev]"
```
If the development extra fails because the package metadata changed, fall back to:
```bash
uv pip install -e .
```
Create `.env` from `.env.template` if it exists, then set at least `LLM_API_KEY` for the default OpenAI path. Run the same import check and smoke test before changing examples or source code.
## Dependency triage
When setup fails, do not reinstall blindly. Read the first meaningful error and match it:
| Error clue | Likely fix |
|---|---|
| `No module named cognee` | Activate the virtual environment and install `cognee` in that environment. |
| `Python 3.9`, `SyntaxError`, or resolver rejects Python | Switch to Python 3.10 through 3.14 and recreate the virtual environment. |
| `anthropic` | Install `cognee[anthropic]`. |
| `psycopg2`, `asyncpg`, or `pgvector` | Install `cognee[postgres]` or `cognee[postgres-binary]`. |
| `neo4j` | Install `cognee[neo4j]`. |
| `playwright`, `tavily`, or `beautifulsoup4` | Install `cognee[scraping]`. |
| `unstructured` | Install `cognee[docs]`. |
| `docling` | Install `cognee[docling]`. |
| `fastembed` | Install `cognee[fastembed]` or `cognee[codegraph]`. |
| `redis` | Install `cognee[redis]`. |
| `baml` | Install `cognee[baml]`. |
| API key, auth, or provider fallback errors | Confirm `.env` is in the project root and both LLM and embedding settings are configured for non-default providers. |
| Embedding dimension mismatch or stale vector collections | Reset local metadata with `await cognee.prune.prune_system(metadata=True)` or use a new `SYSTEM_ROOT_DIRECTORY`. |
| LLM connection preflight times out on a local/small model | Add `COGNEE_SKIP_CONNECTION_TEST=true` to `.env`, then test with a small input. |
## Windows adjustments
There is no `source` on Windows. Pick the activation script that matches the shell the user is in — PowerShell uses `Activate.ps1`, Command Prompt uses `activate.bat`:
```powershell
uv venv
.\.venv\Scripts\Activate.ps1
```
```cmd
uv venv
.venv\Scripts\activate.bat
```
If PowerShell activation is blocked by an execution policy:
```powershell
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
```
If `uv` is not on `PATH` after `pip install uv`, call it through Python: `python -m uv pip install cognee`.
After installing, confirm the active interpreter resolves Cognee from the venv:
```powershell
python -c "import cognee; print(cognee.__file__)"
```
Use forward slashes or doubled backslashes in `.env` path values.
## Finish criteria
Before saying the setup is done, confirm:
- The intended Python interpreter is active.
- `import cognee` succeeds.
- `.env` is in the project root.
- LLM and embedding configuration are both explicit unless the user is using OpenAI defaults.
- The smoke test stores text and recalls an answer.
- Any installed extras match the feature the user is actually trying to use.
````
# Python Quickstart
Source: https://docs.cognee.ai/getting-started/quickstart
Run your first Cognee workflow with remember and recall.
After completing the [installation steps](https://docs.cognee.ai/getting-started/installation) successfully, run your first Cognee example to see AI memory in action.
## Run it without an API key
If you haven't configured `LLM_API_KEY` yet, one CLI command proves the install works:
```bash theme={null}
cognee-cli demo
```
It imports a small knowledge graph that ships inside the `cognee` package and answers two example questions from it. The import is graph-only (no embeddings are computed) and the queries use `CHUNKS_LEXICAL`, a keyword search — so the whole run makes **no LLM calls and no embedding calls**, and works on a machine with no network access. The only outbound request is Cognee's anonymous telemetry event, which is best-effort (it fails silently offline) and switched off entirely by [`TELEMETRY_DISABLED=true`](/setup-configuration/overview#observability--telemetry). Clean up afterwards with `cognee-cli forget --dataset demo`.
See the [CLI reference](/cognee-cli/overview#try-the-demo-graph) for its flags and output.
<Note>
The demo is a keyword search over a pre-built graph. The Python example below builds a graph from your own text and gets an LLM-written answer, so it does need `LLM_API_KEY` configured.
</Note>
## Basic Usage
This minimal example shows how to store content and retrieve it:
```python theme={null}
import cognee
import asyncio
async def main():
# Create a clean slate for cognee -- reset data and system state
await cognee.forget(everything=True)
# Store content in memory (ingests, builds knowledge graph, enriches)
text = "Cognee turns documents into AI memory."
await cognee.remember(text)
# Retrieve from memory
results = await cognee.recall(
query_text="What does Cognee do?"
)
# Print
for result in results:
print(result.text)
if __name__ == '__main__':
asyncio.run(main())
```
<Accordion title="Example output">
```text theme={null}
Cognee converts (transforms) documents into AI memory — a structured, queryable representation of document content for AI systems.
```
Output wording may vary by provider and model, but it should answer the question using the text stored with <code>remember</code>.
</Accordion>
<Accordion title="Visualisation">
<p>Interactive knowledge graph visualization -- drag nodes, zoom, and hover for details. Create your own visualization with 2 additional lines of code [here](/guides/graph-visualization).</p>
<CogneeGraph label="Knowledge graph built by the Quickstart: one remembered sentence, chunked, summarised, and mined for entities" />
</Accordion>
## What just happened
The code demonstrates Cognee's two primary v1.0 operations:
* **`.remember`** — Stores data in memory. Under the hood it runs ingestion, chunking, entity extraction, graph building, and a follow-up enrichment pass. The result is a fully queryable knowledge graph.
* **`.recall`** — Retrieves from memory. It auto-routes the query to the best retrieval strategy and returns contextual results from the knowledge graph.
## About `async` / `await` in Cognee
<Note>
**Cognee uses asynchronous code extensively.** That means many of its functions are defined with `async` and must be called with `await`. This lets Python handle waiting (e.g. for I/O or network calls) without blocking the rest of your program.
</Note>
<Accordion title="Async basics">
This example uses <code>async</code> / <code>await</code>, Python’s way of doing asynchronous programming.
Asynchronous programming is used when functions may block because they are waiting for something (for example, a reply from an API call). By writing <code>async def</code>, you define a function that can pause at certain points.
The <code>await</code> keyword marks those calls that may need to pause.
To run such functions, Python provides the <code>asyncio</code> library. It uses a loop, called the event loop, which executes your code in order but, whenever a function is waiting, can temporarily run another one. From inside your function, though, everything still runs top-to-bottom: each line after an <code>await</code> only executes once the awaited call has finished.
</Accordion>
<Accordion title="Async resources">
* A good starting point is this [guide](https://realpython.com/async-io-python/).
* Official documentation is available [here](https://docs.python.org/3/library/asyncio.html).
</Accordion>
## Next Steps
<CardGroup>
<Card title="Cognee core concepts" href="/core-concepts/overview" icon="compass">
Learn about Cognee's core concepts, architecture, building blocks, and main operations.
</Card>
<Card title="Improve and enrich memory" href="/core-concepts/main-operations/improve" icon="sparkles">
Enrich an existing graph and bridge session memory into permanent memory.
</Card>
</CardGroup>
# Agent Memory Quickstart
Source: https://docs.cognee.ai/guides/agent-memory-quickstart
Minimal end-to-end example showing session memory and graph memory with cognee.agent_memory
A minimal comparison of two agent-memory patterns: one agent that remembers the active session and one agent that only reads from the persistent knowledge graph.
**Before you start:**
* Complete [Quickstart](/getting-started/quickstart) or have Cognee installed and configured
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured
* Be familiar with the `remember()` workflow
* Keep caching enabled — it is on by default, and the session options used below require it:
```dotenv theme={null}
CACHING=true
```
<Warning>
`with_session_memory`, `save_session_traces`, and `persist_session_trace_after` all read and write the [session cache](/core-concepts/sessions-and-caching). With `CACHING=false`, the decorator raises a validation error as soon as it is applied — not when the function is called. The `with_memory` (graph retrieval) path works without caching.
</Warning>
## Shared Setup
Both agents use the same `LLMGateway` helper. The updated example starts clean with `forget(everything=True)`, stores one baseline fact with `remember()`, and then compares session memory against graph memory.
`LLMGateway.acreate_structured_output()` automatically picks up whatever memory the decorator retrieved and prepends it to the text input. In this flow, `save_session_traces=True` records each of the support agent's turns into the session cache, and `persist_session_trace_after=3` is what bridges those traces into the knowledge graph.
```python theme={null}
import asyncio
import os
import warnings
os.environ["LOG_LEVEL"] = "ERROR"
os.environ["COGNEE_LOG_FILE"] = "false"
warnings.filterwarnings("ignore")
import cognee # noqa: E402
from cognee.infrastructure.llm.LLMGateway import LLMGateway # noqa: E402
SESSION_ID = "ticket_001"
BUG = "Login fails with error XQ-99."
FIX = "Set XQ_TOKEN=1 in the .env file."
NO_INFO = "NO INFO AVAILABLE"
async def setup() -> None:
await cognee.forget(everything=True)
await cognee.remember(
["Our app is a web service. Users log in to access their account."], self_improvement=False
)
async def ask_llm(question: str, system_prompt: str) -> str:
return await LLMGateway.acreate_structured_output(
text_input=question,
system_prompt=system_prompt,
response_model=str,
)
```
## Compare the Agents
<Tabs>
<Tab title="Support Agent">
```python theme={null}
@cognee.agent_memory(
with_memory=False,
with_session_memory=True,
save_session_traces=True,
session_id=SESSION_ID,
session_memory_last_n=2,
persist_session_trace_after=3,
)
async def support_agent(question: str, system_prompt: str) -> str:
return await ask_llm(question, system_prompt)
```
This agent relies on session memory instead of graph retrieval. It can remember the recent conversation, and after enough turns its saved traces are persisted into the knowledge graph.
</Tab>
<Tab title="FAQ Bot">
```python theme={null}
@cognee.agent_memory(
with_memory=True,
with_session_memory=False,
save_session_traces=False,
memory_query_from_method="question",
)
async def faq_bot(question: str, system_prompt: str) -> str:
return await ask_llm(question, system_prompt)
```
This agent does the opposite: it skips session memory and reads only from the persistent knowledge graph. It learns only after the support agent's saved traces have been persisted.
</Tab>
</Tabs>
## From Session Traces to the Permanent Graph
The two agents above read from different places, so it helps to know which store each parameter touches:
| Parameter | Store | Effect |
| ------------------------------- | ---------------------------------------------------- | --------------------------------------------------------------------------------------- |
| `with_memory=True` | Permanent graph | Searches `dataset_name` (default `main_dataset`) before the call and injects the result |
| `with_session_memory=True` | [Session cache](/core-concepts/sessions-and-caching) | Injects the last `session_memory_last_n` trace summaries for `(user, session_id)` |
| `save_session_traces=True` | Session cache | Writes one trace step per call — inputs, output, and retrieved memory |
| `persist_session_trace_after=N` | Permanent graph | Every `N`th trace, memifies the recent traces into the graph |
`save_session_traces=True` alone keeps traces in the session cache, where only `with_session_memory` can see them. Adding `persist_session_trace_after=N` is what bridges them into the [permanent graph](/core-concepts/main-operations/remember): after every `N`th trace step in the session, Cognee runs a [memify](/core-concepts/main-operations/legacy-operations/memify) pass that writes the trace summaries into `dataset_name` under the `agent_trace_feedbacks` node set. That is why `faq_bot` — which reads only the graph — answers correctly only after `support_agent` has taken three turns.
<Note>
Trace persistence is best-effort: failures are logged and never propagate out of your agent function. See the [decorator parameters](/core-concepts/further-concepts/agent-memory-decorator) for the options that control what gets persisted and where.
</Note>
## Full Example
<Accordion title="Guide">
```python theme={null}
import asyncio
import os
import warnings
os.environ["LOG_LEVEL"] = "ERROR"
os.environ["COGNEE_LOG_FILE"] = "false"
warnings.filterwarnings("ignore")
import cognee # noqa: E402
from cognee.infrastructure.llm.LLMGateway import LLMGateway # noqa: E402
SESSION_ID = "ticket_001"
BUG = "Login fails with error XQ-99."
FIX = "Set XQ_TOKEN=1 in the .env file."
NO_INFO = "NO INFO AVAILABLE"
async def setup() -> None:
await cognee.forget(everything=True)
await cognee.remember(
["Our app is a web service. Users log in to access their account."], self_improvement=False
)
async def ask_llm(question: str, system_prompt: str) -> str:
return await LLMGateway.acreate_structured_output(
text_input=question,
system_prompt=system_prompt,
response_model=str,
)
@cognee.agent_memory(
with_memory=False,
with_session_memory=True,
save_session_traces=True,
session_id=SESSION_ID,
session_memory_last_n=2,
persist_session_trace_after=3,
)
async def support_agent(question: str, system_prompt: str) -> str:
return await ask_llm(question, system_prompt)
@cognee.agent_memory(
with_memory=True,
with_session_memory=False,
save_session_traces=False,
memory_query_from_method="question",
)
async def faq_bot(question: str, system_prompt: str) -> str:
return await ask_llm(question, system_prompt)
async def main() -> None:
print("=== Agent Memory Quickstart ===\n")
print("support_agent: session memory — knows what happened in this conversation.")
print("faq_bot: knowledge graph — knows only what has been formally filed.\n")
print("Setting up knowledge graph...")
await setup()
print("Ready.\n")
recall_prompt = f"Answer based on the available context. If it is not available in the context, say exactly: {NO_INFO}"
support_agent_q1 = f"A user just reported this: {BUG}"
print(f"support_agent_q: {support_agent_q1}")
support_agent_a1 = await support_agent(
support_agent_q1, f"Confirm you received it. Say exactly: {BUG}"
)
print(f"support_agent_a: {support_agent_a1}\n")
faq_bot_q = "How do I fix error XQ-99?"
support_agent_q2 = "What bug was just reported?"
print(f"support_agent_q: {support_agent_q2}")
support_agent_a2 = await support_agent(
support_agent_q2,
f"Use your session memory. If you know, say: {BUG} If not, say: {NO_INFO}",
)
print(f"support_agent_a: {support_agent_a2}")
print(f"faq_bot_q: {faq_bot_q}")
faq_bot_a_before = await faq_bot(faq_bot_q, recall_prompt)
print(f"faq_bot_a: {faq_bot_a_before}")
print("\n^ support_agent recalled the bug from session. faq_bot had no context yet.\n")
support_agent_q3 = f"Log this fix for the login crash: {FIX}"
print(f"support_agent_q: {support_agent_q3}")
support_agent_a3 = await support_agent(support_agent_q3, f"Confirm the fix. Say exactly: {FIX}")
print(f"support_agent_a: {support_agent_a3}")
print("(Session traces are now persisted to the knowledge graph.)\n")
print(f"faq_bot_q: {faq_bot_q}")
faq_bot_a_after = await faq_bot(faq_bot_q, recall_prompt)
print(f"faq_bot_a: {faq_bot_a_after}")
print("\n^ faq_bot now answered correctly — session traces reached the knowledge graph.")
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
<Note>
The decorator does not create memory by itself. The example works because baseline graph memory is stored first with `remember()`, and the session-aware support agent persists traces only after those conversation turns happen.
</Note>
<Columns>
<Card title="Agent Memory Decorator" icon="bot" href="/core-concepts/further-concepts/agent-memory-decorator">
Concept overview and parameter reference
</Card>
<Card title="Low-Level LLM" icon="cpu" href="/guides/low-level-llm">
`LLMGateway` for direct model calls
</Card>
<Card title="Remember" icon="brain-cog" href="/core-concepts/main-operations/remember">
Store memory before using the decorator
</Card>
<Card title="Sessions" icon="database" href="/guides/sessions">
Learn how trace-backed session context behaves
</Card>
</Columns>
# Agent Session Traces
Source: https://docs.cognee.ai/guides/agent-session-traces
Record what a decorated function did on each call, and recall those traces later
A minimal guide to agent session traces: decorate a function with `cognee.agent_memory(save_session_traces=True)`, call it a couple of times, then recall what happened — including calls that raised an error.
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — the initial `remember()` call needs one, even though tracing itself does not
* Read [Sessions](/guides/sessions) first — this guide assumes you already know what `session_id` is and how it works
## Code in Action
```python theme={null}
import asyncio
import cognee
SESSION_ID = "agent_demo"
@cognee.agent_memory(
session_id=SESSION_ID,
with_memory=False,
with_session_memory=False,
save_session_traces=True,
session_trace_summary=False,
)
async def lookup_teammate_status(name: str) -> str:
statuses = {"Alice": "available"}
if name not in statuses:
raise ValueError(f"No status on file for {name!r}")
return statuses[name]
async def main():
await cognee.remember(
"Alice and Bob are teammates.",
self_improvement=False,
)
status = await lookup_teammate_status("Alice")
print(f"Alice's status: {status}")
try:
await lookup_teammate_status("Bob")
except ValueError as error:
print(f"Expected error for Bob: {error}")
alice_traces = await cognee.recall(
query_text="Alice",
session_id=SESSION_ID,
scope="trace",
)
for trace in alice_traces:
print(
f"[{trace.status}] {trace.origin_function}({trace.method_params}) "
f"-> {trace.method_return_value or trace.error_message}"
)
bob_traces = await cognee.recall(
query_text="Bob",
session_id=SESSION_ID,
scope="trace",
)
for trace in bob_traces:
print(
f"[{trace.status}] {trace.origin_function}({trace.method_params}) "
f"-> {trace.method_return_value or trace.error_message}"
)
other_session_traces = await cognee.recall(
query_text="Alice",
session_id="a_different_session",
scope="trace",
)
print(f"Traces found in a fresh session: {len(other_session_traces)}")
if __name__ == "__main__":
asyncio.run(main())
```
## What Just Happened
### Step 1: Bootstrap Cognee and Decorate the Function
```python theme={null}
@cognee.agent_memory(
session_id=SESSION_ID,
with_memory=False,
with_session_memory=False,
save_session_traces=True,
session_trace_summary=False,
)
async def lookup_teammate_status(name: str) -> str:
statuses = {"Alice": "available"}
if name not in statuses:
raise ValueError(f"No status on file for {name!r}")
return statuses[name]
```
```python theme={null}
await cognee.remember(
"Alice and Bob are teammates.",
self_improvement=False,
)
```
The `remember()` call runs once, before anything else, purely to make sure Cognee's database and default user exist — the decorated function itself never reads this fact back, because `with_memory=False`. `save_session_traces=True` is what turns tracing on — without it, the decorator would still run the function but record nothing. `with_memory=False` and `with_session_memory=False` keep this example focused on traces alone: no graph memory lookup, no conversation-history retrieval, just "what happened when this function ran." `session_trace_summary=False` skips the LLM-generated summary Cognee would otherwise attempt for each trace.
### Step 2: Call It Once Successfully
```python theme={null}
status = await lookup_teammate_status("Alice")
print(f"Alice's status: {status}")
```
`"Alice"` is in `statuses`, so the function returns normally. Behind the scenes, the decorator records this call's trace with `status="success"` and the returned value — automatically, without you constructing any trace object yourself.
### Step 3: Call It Again With an Error
```python theme={null}
try:
await lookup_teammate_status("Bob")
except ValueError as error:
print(f"Expected error for Bob: {error}")
```
`"Bob"` is not in `statuses`, so the function raises `ValueError`. The decorator still records this call's trace — this time with `status="error"` and the error message — then re-raises the exception, which is why the call is wrapped in `try`/`except` here.
### Step 4: Recall the Traces
```python theme={null}
alice_traces = await cognee.recall(
query_text="Alice",
session_id=SESSION_ID,
scope="trace",
)
```
`scope="trace"` tells `recall()` to search recorded traces instead of the knowledge graph or conversation history. Trace search matches by keyword across each trace's function name, parameters, return value, and error message — `query_text="Alice"` matches the first call because `"Alice"` appears in its parameters. Each result exposes `origin_function`, `status`, `method_params`, `method_return_value`, and `error_message`, so both the successful call and the failed one are fully inspectable.
### Step 5: Confirm Isolation Between Sessions
```python theme={null}
other_session_traces = await cognee.recall(
query_text="Alice",
session_id="a_different_session",
scope="trace",
)
print(f"Traces found in a fresh session: {len(other_session_traces)}")
```
The same query against a `session_id` that never ran `lookup_teammate_status` returns an empty list — traces are scoped per session, just like the conversation history covered in [Sessions](/guides/sessions).
## Traces vs. Other Session Concepts
<AccordionGroup>
<Accordion title="Traces vs. conversation history">
A trace records one function call — its inputs, status, and outcome. Conversation history (covered in [Sessions](/guides/sessions)) records question/answer turns from `recall()`. Both live in the same session cache and are scoped by `session_id`, but they answer different questions: "what did this code do?" vs. "what did we discuss?"
</Accordion>
<Accordion title="Traces vs. memory_context and automatic guidance">
This guide keeps `with_memory=False` and `with_session_memory=False` so the decorated function never reads graph memory or session history back — it only writes traces. Enabling those flags (as `examples/guides/agent_memory_quickstart.py` does) lets a decorated function *use* memory during its own execution, which is a separate, more advanced concept from recording that the call happened.
</Accordion>
<Accordion title="Traces vs. graph persistence">
Traces recorded here stay in the session cache — this guide sets no option that writes them to the permanent knowledge graph. Two mechanisms bridge them: `improve(session_ids=...)`, the on-demand path covered in [Session Distillation](/guides/session-distillation), and `persist_session_trace_after=N` on the decorator itself, which memifies the recent traces into the graph every `N`th trace (see [Agent Memory Quickstart](/guides/agent-memory-quickstart#from-session-traces-to-the-permanent-graph)). Both are out of scope for this guide.
</Accordion>
</AccordionGroup>
<Columns>
<Card title="Sessions" icon="message-square" href="/guides/sessions">
Learn the session-cache concept traces are built on
</Card>
<Card title="Agent Memory Quickstart" icon="bot" href="/guides/agent-memory-quickstart">
See traces combined with session memory and graph memory
</Card>
</Columns>
# BaseRetriever Guide
Source: https://docs.cognee.ai/guides/base-retriever
Learn how BaseRetriever's interface works by building a tiny, fully offline retriever
A minimal guide to how `BaseRetriever`'s interface works, taught by building a tiny, fully offline retriever. No database, network connection, embeddings, or LLM required.
## Before You Start
* Have `cognee` installed (see [Installation](/getting-started/installation)) — needed only for the `BaseRetriever` import; no LLM provider or database configuration is required for this example.
* No prior OOP knowledge is assumed — the concepts you need (classes, inheritance, abstract methods) are explained at the end of the page.
## Code in Action
```python theme={null}
import asyncio
import json
from pathlib import Path
from typing import Any, Optional
from cognee.modules.retrieval.base_retriever import BaseRetriever
class JsonToyRetriever(BaseRetriever):
"""A minimal offline retriever."""
def __init__(self, output_path: str | Path = "toy_query.json"):
self.output_path = Path(output_path)
async def get_retrieved_objects(
self,
query: Optional[str] = None,
query_batch: Optional[list[str]] = None,
) -> dict[str, Any]:
query = query or ""
data = {
"query": query,
"character_count": len(query),
"word_count": len(query.split()),
}
with self.output_path.open("w", encoding="utf-8") as f:
json.dump(
data,
f,
indent=2,
ensure_ascii=False,
sort_keys=True,
)
return data
async def get_context_from_objects(
self,
query: Optional[str] = None,
query_batch: Optional[list[str]] = None,
retrieved_objects: Any = None,
) -> str:
return json.dumps(
retrieved_objects,
indent=2,
ensure_ascii=False,
sort_keys=True,
)
async def get_completion_from_context(
self,
query: Optional[str] = None,
query_batch: Optional[list[str]] = None,
retrieved_objects: Any = None,
context: Any = None,
) -> list[str]:
return [
(
"Toy completion: "
f"query={retrieved_objects['query']!r}; "
f"characters={retrieved_objects['character_count']}; "
f"words={retrieved_objects['word_count']}."
)
]
async def main():
retriever = JsonToyRetriever()
query = "How does BaseRetriever work?"
result = await retriever.get_completion(query)
print(result[0])
# Call the same three stages manually, to prove get_completion() just orchestrates them
objects = await retriever.get_retrieved_objects(query)
context = await retriever.get_context_from_objects(query, retrieved_objects=objects)
manual_result = await retriever.get_completion_from_context(
query, retrieved_objects=objects, context=context
)
print("Manual orchestration matches get_completion():", manual_result[0])
if __name__ == "__main__":
asyncio.run(main())
```
## What Just Happened
The three methods below — `get_retrieved_objects`, `get_context_from_objects`, and `get_completion_from_context` — are called, in order, by `get_completion()`, each explained one step at a time below. `JsonToyRetriever` inherits `get_completion()` from `BaseRetriever` rather than defining it.
### Step 1: Retrieve Objects
```python theme={null}
async def get_retrieved_objects(
self,
query: Optional[str] = None,
query_batch: Optional[list[str]] = None,
) -> dict[str, Any]:
query = query or ""
data = {
"query": query,
"character_count": len(query),
"word_count": len(query.split()),
}
with self.output_path.open("w", encoding="utf-8") as f:
json.dump(data, f, indent=2, ensure_ascii=False, sort_keys=True)
return data
```
`get_retrieved_objects` is the first of the three methods `BaseRetriever` requires every subclass to implement (`get_retrieved_objects`, `get_context_from_objects`, `get_completion_from_context`). It turns the raw query into a small dictionary and writes it to a JSON file. A production retriever would query a vector database, a graph database, or another storage backend here instead of building a toy dictionary.
### Step 2: Build Context
```python theme={null}
async def get_context_from_objects(
self,
query: Optional[str] = None,
query_batch: Optional[list[str]] = None,
retrieved_objects: Any = None,
) -> str:
return json.dumps(retrieved_objects, indent=2, ensure_ascii=False, sort_keys=True)
```
`get_context_from_objects`, the second stage, turns retrieved objects into the context that gets handed to the completion step. Here it's just the same dictionary formatted as JSON text; a real retriever might concatenate document chunks or format graph relationships instead.
### Step 3: Produce the Completion
```python theme={null}
async def get_completion_from_context(
self,
query: Optional[str] = None,
query_batch: Optional[list[str]] = None,
retrieved_objects: Any = None,
context: Any = None,
) -> list[str]:
return [
(
"Toy completion: "
f"query={retrieved_objects['query']!r}; "
f"characters={retrieved_objects['character_count']}; "
f"words={retrieved_objects['word_count']}."
)
]
```
`get_completion_from_context`, the third stage, would normally call an LLM with the context built in Step 2. This toy version returns a deterministic string instead, so the whole example stays offline and reproducible.
### Step 4: Run the Retriever
```python theme={null}
async def main():
retriever = JsonToyRetriever()
query = "How does BaseRetriever work?"
result = await retriever.get_completion(query)
print(result[0])
```
`get_completion()` is not implemented by `JsonToyRetriever` — it's inherited from `BaseRetriever`. `get_completion()` calls the three stages above in order, passing each stage's return value into the next, and returns the final result: the base class defines the workflow, while the subclass only defines the behavior of each step. Running this prints:
```text theme={null}
Toy completion: query='How does BaseRetriever work?'; characters=28; words=4.
```
and creates a `toy_query.json` file alongside it.
### Step 5: Prove the Orchestration Manually
```python theme={null}
objects = await retriever.get_retrieved_objects(query)
context = await retriever.get_context_from_objects(query, retrieved_objects=objects)
manual_result = await retriever.get_completion_from_context(
query, retrieved_objects=objects, context=context
)
print("Manual orchestration matches get_completion():", manual_result[0])
```
To prove that's really what's happening, calling the three stages manually — `get_retrieved_objects`, then `get_context_from_objects`, then `get_completion_from_context`, passing each result into the next — produces the exact same output as `get_completion()`. It's the same three calls either way, just written out instead of hidden inside the inherited method. Running this prints an extra line:
```text theme={null}
Manual orchestration matches get_completion(): Toy completion: query='How does BaseRetriever work?'; characters=28; words=4.
```
## OOP Concepts You Need
<AccordionGroup>
<Accordion title="Classes, Inheritance, and Abstract Methods">
A **class** is a blueprint describing what an object can do; an **instance** is one actual object created from it.
One class can **inherit** from another, gaining its methods for free, and a subclass can **override** an inherited method with its own implementation:
```python theme={null}
class Animal:
def speak(self):
return "..."
class Dog(Animal):
def speak(self):
return "woof"
```
Sometimes a base class needs to require a method without providing it — an **abstract method**: declared on the base class, but with no implementation there, forcing every subclass to supply its own. Python's `abc` module (Abstract Base Classes) provides this via the `@abstractmethod` decorator:
```python theme={null}
from abc import ABC, abstractmethod
class Animal(ABC):
@abstractmethod
def speak(self):
pass
```
If a subclass forgets to implement `speak`, Python raises an error before the object can even be created. See [Python's `abc` module docs](https://docs.python.org/3/library/abc.html) for the full mechanism.
`BaseRetriever` works the same way: it declares `get_retrieved_objects`, `get_context_from_objects`, and `get_completion_from_context` as abstract methods, so calling `BaseRetriever()` directly raises a `TypeError` unless all three methods are implemented.
</Accordion>
</AccordionGroup>
<Columns>
<Card title="Search" icon="search" href="/core-concepts/main-operations/legacy-operations/search">
See how built-in retrievers map to search types, and how to register a custom one
</Card>
<Card title="Search Basics" icon="search" href="/guides/search-basics">
Run your first real Cognee search once you're ready to move past this toy example
</Card>
</Columns>
# Code Graph
Source: https://docs.cognee.ai/guides/code-graph
Build a knowledge graph of a code repository with enola and query it with SearchType.CODE
A minimal guide to turning a code repository into a knowledge graph and querying it. The pipeline extracts facts such as modules, symbols, routes, storage, services, and dependencies with the external enola extractor, loads them as typed graph nodes and edges, and answers structured queries — no LLM or embedding provider involved.
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Read [Pipelines](/core-concepts/building-blocks/pipelines) and [Tasks](/core-concepts/building-blocks/tasks) for how a custom pipeline is assembled from tasks
* Have the enola binary available: it is installed automatically on the first run (pinned release, checksum-verified, placed in `~/.cognee/bin`), or [install it yourself](https://github.com/enola-labs/enola#installation) and point `ENOLA_PATH` at it
* Set `CODE_GRAPH_REPO_PATH` to the repository you want to index — it defaults to the current working directory
* No [LLM Providers](/setup-configuration/llm-providers) or embedding configuration is required: both the pipeline and `SearchType.CODE` are deterministic
## Code in Action
````python theme={null}
import asyncio
import json
import os
import cognee
from cognee import SearchType
from cognee.shared.logging_utils import ERROR, setup_logging
from cognee.tasks.code_graph import get_code_graph_tasks
async def main():
repo_path = os.getenv("CODE_GRAPH_REPO_PATH", os.getcwd())
# Start clean so the example is reproducible.
await cognee.prune.prune_data()
await cognee.prune.prune_system(metadata=True)
print(f"Extracting code graph from: {repo_path}")
await cognee.run_custom_pipeline(
# Pass index_vectors=True only if these facts should also be available
# to semantic/LLM retrievers; SearchType.CODE does not need it.
tasks=get_code_graph_tasks(repo_path),
data=repo_path,
dataset="code_graph_demo",
pipeline_name="code_graph_pipeline",
# This pipeline is deterministic (no LLM/embedding calls), so skip the
# first-run LLM/embedding connection checks and stay truly keyless.
skip_connection_test=True,
)
print("Listing the first indexed code facts")
search_results = await cognee.search(
query_type=SearchType.CODE,
query_text="",
datasets=["code_graph_demo"],
code_query={
"operation": "query_facts",
"kinds": ["module", "symbol", "route", "storage", "service"],
"limit": 20,
},
)
print(json.dumps(search_results, indent=2, default=str))
print("Module-level architecture overview, drawn as a Mermaid diagram")
architecture = await cognee.search(
query_type=SearchType.CODE,
query_text="",
datasets=["code_graph_demo"],
# Symbol-to-symbol edges are rolled up to the modules that declare
# them; routes/storage/services hang off their modules. The result
# includes deterministic Mermaid source (paste it into any Markdown
# renderer that supports ```mermaid fences); "diagram": "dot" gives
# Graphviz, and any other operation accepts the same option.
code_query={"operation": "architecture", "max_nodes": 40},
)
# search() returns one {dataset_id, dataset_name, search_result} entry per
# dataset; the CODE operation's result is the (single) search_result item.
for entry in architecture:
payload = entry.get("search_result") if isinstance(entry, dict) else None
if isinstance(payload, list) and payload:
payload = payload[0]
diagram = payload.get("diagram") if isinstance(payload, dict) else None
if diagram and diagram.get("source"):
print(diagram["source"])
print("Architecture findings enola's explainers produced (with their evidence facts)")
insights = await cognee.search(
query_type=SearchType.CODE,
query_text="",
datasets=["code_graph_demo"],
# Structural findings (cycles, declared-layer violations) score 1.0;
# heuristic ones (hotspots, god-class, complexity outliers) score below.
code_query={"operation": "insights", "min_confidence": 0.5, "limit": 10},
)
print(json.dumps(insights, indent=2, default=str))
# Other deterministic operations use the same API shape. Ids may be
# cognee node ids or enola's own 32-hex fact ids (from facts.jsonl):
# code_query={"operation": "explore", "id": "<fact id>", "max_depth": 2}
# code_query={"operation": "traverse", "node_ids": ["<fact id>"], "direction": "reverse"}
# code_query={"operation": "find_path", "source_id": "<id>", "target_id": "<id>"}
# code_query={"operation": "impact_analysis", "id": "<fact id>", "max_depth": 3}
# code_query={"operation": "query_facts", "kind": "dependency", "prop": "type",
# "prop_value": "package"} # declared packages from manifests (purl names)
# code_query={"operation": "delta"} # last ingestion's changes + the snapshot receipt
if __name__ == "__main__":
logger = setup_logging(log_level=ERROR)
asyncio.run(main())
````
## What Just Happened
### Step 1: Choose the Repository and Start Clean
```python theme={null}
repo_path = os.getenv("CODE_GRAPH_REPO_PATH", os.getcwd())
# Start clean so the example is reproducible.
await cognee.prune.prune_data()
await cognee.prune.prune_system(metadata=True)
```
The repository to index comes from `CODE_GRAPH_REPO_PATH`, falling back to the directory you run the script from. Pruning first means the graph you inspect afterwards contains only what this run extracted.
### Step 2: Run the Code Graph Pipeline
```python theme={null}
print(f"Extracting code graph from: {repo_path}")
await cognee.run_custom_pipeline(
# Pass index_vectors=True only if these facts should also be available
# to semantic/LLM retrievers; SearchType.CODE does not need it.
tasks=get_code_graph_tasks(repo_path),
data=repo_path,
dataset="code_graph_demo",
pipeline_name="code_graph_pipeline",
# This pipeline is deterministic (no LLM/embedding calls), so skip the
# first-run LLM/embedding connection checks and stay truly keyless.
skip_connection_test=True,
)
```
`get_code_graph_tasks()` returns the three ordered tasks the pipeline runs: extract (run enola over the repository and map its facts to DataPoints), load the graph nodes, then load the typed relations as edges. Because nothing here calls an LLM or an embedding model, `skip_connection_test=True` skips the first-run provider checks so the pipeline runs without any API key.
### Step 3: Query the Graph with SearchType.CODE
```python theme={null}
print("Listing the first indexed code facts")
search_results = await cognee.search(
query_type=SearchType.CODE,
query_text="",
datasets=["code_graph_demo"],
code_query={
"operation": "query_facts",
"kinds": ["module", "symbol", "route", "storage", "service"],
"limit": 20,
},
)
print(json.dumps(search_results, indent=2, default=str))
```
`SearchType.CODE` is driven by the structured `code_query` argument rather than by `query_text`, which stays empty here. The `query_facts` operation filters the extracted facts — by `kinds` in this case — and returns the first `limit` matches, so the result is a deterministic listing rather than a similarity ranking.
### Step 4: Draw the Architecture and Read the Insights
````python theme={null}
print("Module-level architecture overview, drawn as a Mermaid diagram")
architecture = await cognee.search(
query_type=SearchType.CODE,
query_text="",
datasets=["code_graph_demo"],
# Symbol-to-symbol edges are rolled up to the modules that declare
# them; routes/storage/services hang off their modules. The result
# includes deterministic Mermaid source (paste it into any Markdown
# renderer that supports ```mermaid fences); "diagram": "dot" gives
# Graphviz, and any other operation accepts the same option.
code_query={"operation": "architecture", "max_nodes": 40},
)
# search() returns one {dataset_id, dataset_name, search_result} entry per
# dataset; the CODE operation's result is the (single) search_result item.
for entry in architecture:
payload = entry.get("search_result") if isinstance(entry, dict) else None
if isinstance(payload, list) and payload:
payload = payload[0]
diagram = payload.get("diagram") if isinstance(payload, dict) else None
if diagram and diagram.get("source"):
print(diagram["source"])
print("Architecture findings enola's explainers produced (with their evidence facts)")
insights = await cognee.search(
query_type=SearchType.CODE,
query_text="",
datasets=["code_graph_demo"],
# Structural findings (cycles, declared-layer violations) score 1.0;
# heuristic ones (hotspots, god-class, complexity outliers) score below.
code_query={"operation": "insights", "min_confidence": 0.5, "limit": 10},
)
print(json.dumps(insights, indent=2, default=str))
````
The `architecture` operation rolls symbol-level edges up to the modules that declare them and returns a ready-to-render diagram alongside the nodes — Mermaid by default, Graphviz with `"diagram": "dot"`. The `insights` operation returns the findings enola's explainers produced over the same graph, each with the evidence facts behind it; `min_confidence` separates the structural findings that always score 1.0 from the heuristic ones below it.
### Step 5: Reuse the Same Shape for Other Operations
```python theme={null}
# Other deterministic operations use the same API shape. Ids may be
# cognee node ids or enola's own 32-hex fact ids (from facts.jsonl):
# code_query={"operation": "explore", "id": "<fact id>", "max_depth": 2}
# code_query={"operation": "traverse", "node_ids": ["<fact id>"], "direction": "reverse"}
# code_query={"operation": "find_path", "source_id": "<id>", "target_id": "<id>"}
# code_query={"operation": "impact_analysis", "id": "<fact id>", "max_depth": 3}
# code_query={"operation": "query_facts", "kind": "dependency", "prop": "type",
# "prop_value": "package"} # declared packages from manifests (purl names)
# code_query={"operation": "delta"} # last ingestion's changes + the snapshot receipt
```
Every other operation is the same `cognee.search()` call with a different `code_query`. Take a fact id from the `query_facts` output above and feed it to `explore` to see a fact's neighborhood, `traverse` to walk edges in one direction, `find_path` to connect two facts, or `impact_analysis` to see what depends on a fact. `query_facts` also filters on a fact property with `prop` and `prop_value`; the `dependency` example above lists the packages declared in the repository's manifests. There is also a `delta` operation, which needs no fact id: `code_query={"operation": "delta"}` reports what the last ingestion changed in each repository. Every operation's full argument list, defaults, and the fact kinds a query can filter on are in the [SearchType reference](/python-api/search-type#per-search-type-parameters).
## Advanced Usage
<AccordionGroup>
<Accordion title="Make the Facts Available to Semantic Retrievers">
`get_code_graph_tasks(repo_path, index_vectors=True)` also writes the extracted facts to the vector store, so semantic and LLM-backed retrievers can reach them. It is opt-in because `SearchType.CODE` reads the graph only; enabling it adds embedding calls and therefore needs an [embedding provider](/setup-configuration/embedding-providers) configured.
</Accordion>
<Accordion title="Query Across Repositories">
Graph paths only exist inside a single dataset. To follow paths across repositories, generate one Enola append/multi-repository snapshot covering all of them and ingest that into one dataset. Repositories indexed into separate datasets are searched independently, and no path can connect them.
</Accordion>
<Accordion title="Index a Public Repository by URL">
A GitHub/GitLab repository URL passed to plain `add()` or `remember()` — with **no** `content_type` — is recognised as a repository by its shape, shallow-cloned, and ingested as one code-repo item that takes the same code graph route, plus the repository's own documents:
```python theme={null}
await cognee.add("https://github.com/owner/repo", dataset_name="my_repo")
await cognee.cognify(datasets=["my_repo"])
# Or in one call
await cognee.remember("https://github.com/owner/repo", dataset_name="my_repo")
```
Use `remember(url, content_type="code")` instead when you want only the code graph (no repository documents), several repositories in one call, or `repo_credentials` for a private remote. Clones land under `COGNEE_REPOS_DIR` (default `~/.cognee/repos`) either way, and both paths need `git` on `PATH` and `ALLOW_HTTP_REQUESTS` enabled. Deeper forge URLs (`/blob/`, `/tree/`, `/issues`, GitLab's `/-/` pages) are still treated as web pages; see [`add()`](/python-api/add#code-repository-urls) for the exact detection rules.
</Accordion>
<Accordion title="Index a Private Repository">
`remember(url, content_type="code", repo_credentials="<token>")` clones a private https remote using an out-of-band token — a GitHub App installation token, for example. It is sent as HTTP basic auth under the `x-access-token` username, the scheme GitHub expects. The token reaches `git` through environment-level config rather than the URL, so nothing derived from the URL can carry it: the clone directory name, the persisted git remote, log lines, and git's own error output all use the credential-free URL. `repo_credentials` is code-only — passing it with any other `content_type` raises `ValueError`.
Credentials embedded in the URL userinfo (`https://x-access-token:<token>@github.com/org/repo.git`) still work, but are the legacy path. Either way, the `source` recorded on each result item is redacted, so a token never surfaces in a `remember()` result.
To connect a whole organization instead of one repository at a time, see the [GitHub integration](/integrations/github-integration).
</Accordion>
<Accordion title="Query the Code Graph Through recall()">
`recall()` reaches the same deterministic operations without assembling a `search()` call. The `"code"` scope is explicit opt-in — neither `"auto"` nor `"all"` includes it — and `code_query` takes the same operation dict this guide uses:
```python theme={null}
results = await cognee.recall(
"UserService",
scope=["code"],
code_query={"operation": "impact_analysis", "name": "UserService", "max_depth": 3},
)
```
Omitting `code_query` runs `explore` seeded with the query text, and entries come back with `source="code"`, so a `scope=["graph", "code"]` call keeps the two lanes apart in the results. For the full parameter behavior, see [`recall()`](/python-api/recall).
</Accordion>
<Accordion title="Provide Your Own enola Binary">
`ENOLA_PATH` always wins over the auto-installed binary, so point it at your own build to control the version. Setting `ENOLA_AUTO_INSTALL=false` disables the automatic download entirely — the run then fails with an install error instead of fetching the pinned release. For the exact release URL, the pinned version, the cache path, and the supported platform builds, see [Where the enola binary comes from](/core-concepts/further-concepts/loaders#usage).
</Accordion>
</AccordionGroup>
<Columns>
<Card title="Pipelines" icon="git-merge" href="/core-concepts/building-blocks/pipelines">
How tasks are orchestrated into a pipeline.
</Card>
<Card title="run_custom_pipeline()" icon="route" href="/python-api/custom-pipeline">
The full parameter surface of the call this guide uses.
</Card>
<Card title="Custom Tasks and Pipelines" icon="workflow" href="/guides/custom-tasks-pipelines">
Write your own tasks and assemble them into a pipeline.
</Card>
</Columns>
# Custom Data Models
Source: https://docs.cognee.ai/guides/custom-data-models
Step-by-step guide to creating custom data models and using add_data_points
A minimal guide to creating custom data models and inserting them directly into the knowledge graph using `add_data_points`.
**Before you start:**
* Complete [Quickstart](getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](setup-configuration/llm-providers) configured
* Have some structured data you want to model
## What Custom Data Models Do
* Define your own Pydantic models that inherit from `DataPoint`
* Insert structured data directly into the knowledge graph without `cognify`
* Create relationships between data points programmatically
* Control exactly what gets indexed and how
## Code in Action
### Step 1: Define Your Data Model
```python theme={null}
class Person(DataPoint):
name: str
knows: SkipValidation[Any] = None
# Recommended: specify which fields to index for search
metadata: dict = {"index_fields": ["name"]}
```
Create a Pydantic model that inherits from `DataPoint`. Use `SkipValidation[Any]` for fields that will hold other DataPoints to avoid forward reference issues. **Metadata is recommended** - it tells Cognee which fields to embed and store in the vector database for search.
### Step 2: Create Data Instances
```python theme={null}
alice = Person(name="Alice")
bob = Person(name="Bob")
charlie = Person(name="Charlie")
```
Instantiate your models with the data you want to store. Each instance becomes a node in the knowledge graph.
### Step 3: Create Relationships
```python theme={null}
alice.knows = bob
# Optional: add weights and custom relationship types
bob.knows = (Edge(weight=0.9, relationship_type="friend_of"), charlie)
```
Assign DataPoint instances to fields to create edges. The field name becomes the relationship label by default. **Weights are optional** - you can use `Edge` to add weights, custom relationship types, or other metadata to your relationships.
### Step 4: Insert into Graph
```python theme={null}
await add_data_points([alice, bob, charlie])
```
This converts your DataPoint instances into nodes and edges in the knowledge graph, automatically handling the graph structure and indexing. The `name` field gets embedded and stored in the vector database for search.
## Custom Data Model Fields
When `add_data_points` walks your model, it decides field-by-field whether a value is a relationship or a plain property:
* **Edges** — a field whose value is another `DataPoint`, a `list[DataPoint]`, or an `(Edge(...), DataPoint)` / `(Edge(...), list[DataPoint])` tuple. Each referenced DataPoint becomes its own node, and the field name (or the `Edge.relationship_type`) becomes the edge label.
* **Properties** — every other value type (`str`, `int`, `float`, `bool`, **`dict`**, or a list of scalar values) is stored on the node. It is not expanded into separate nodes or edges.
Only fields listed in `metadata.index_fields` are embedded for vector search — pick a text field (like `name`) for that, since a `dict` is stored but not meaningfully searchable. See [DataPoints](/core-concepts/building-blocks/datapoints) for more on indexing.
For complex nested values, such as a list of dictionaries, prefer serializing them yourself or modeling each nested object as its own `DataPoint` when you need portable graph behavior across database backends.
### Edge Metadata Fields
`Edge` accepts the following fields, all optional:
| Field | Type | Description |
| ------------------- | ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `weight` | `float` | A single numeric weight for the relationship. |
| `weights` | `dict[str, float]` | Multiple named weights (e.g. `{"strength": 0.8, "confidence": 0.9}`). Each one is also stored as a separate, queryable `weight_<name>` property on the edge. |
| `relationship_type` | `str` | Custom relationship label. When set, it overrides the field name as the edge's relationship name. |
| `properties` | `dict[str, Any]` | Arbitrary custom metadata to attach to the edge. Use this for any fields not covered above. |
| `edge_text` | `str` | A rich, natural-language description of the relationship that is embedded for semantic edge/triplet retrieval. When omitted, Cognee builds fallback retrieval text from the source node, relationship name, and target node. |
Use `properties` for custom edge metadata, such as `properties={"since": 2015, "context": "college"}`. Advanced users can also subclass `Edge`; subclass fields are included in the stored edge properties.
### Custom Fields and Read-Back
Use plain scalar fields when you need to keep external identifiers, labels, statuses, or other simple properties on a node. Do not add those fields to `metadata.index_fields` unless you actually want Cognee to embed them as searchable text.
```python theme={null}
from cognee.infrastructure.engine import DataPoint
from cognee.infrastructure.engine.models.DataPoint import MetaData
class Note(DataPoint):
text: str
external_id: str
category: str
metadata: MetaData = {"index_fields": ["text"]}
```
Here, `text` is embedded for semantic search, while `external_id` and `category` are stored as normal node properties.
When you have the `DataPoint` object itself, read custom fields directly:
```python theme={null}
note.external_id
note.model_dump()["category"]
```
When a search result returns the stored payload directly, custom fields on that payload are plain keys:
```python theme={null}
import cognee
from cognee import SearchType
results = await cognee.search(
query_text="my query",
query_type=SearchType.CHUNKS,
)
results[0]["external_id"]
```
When using `recall()`, normalized graph entries reserve `metadata` for stable provenance keys such as `data_id`, `chunk_id`, `chunk_index`, and `document_name`. Custom payload fields from the result are available on `raw`:
```python theme={null}
import cognee
from cognee import SearchType
entries = await cognee.recall(
query_text="my query",
query_type=SearchType.CHUNKS,
)
entries[0].raw["external_id"]
```
## Use in Custom Tasks and Pipelines
This approach is particularly useful when creating custom tasks and pipelines where you need to:
* Insert structured data programmatically
* Define specific relationships between known entities
* Control exactly what gets indexed and how
* Integrate with external data sources or APIs
You can combine this with `cognify` to extract knowledge from unstructured text, then add your own structured data on top.
## Linking DataPoints to a Dataset
When you call `add_data_points` standalone, nodes are inserted globally with no dataset association. Dataset-level [`forget()`](/core-concepts/main-operations/forget) calls will **not** remove them. To delete those unassociated DataPoints, call `prune_system()` instead of `forget(dataset=...)`.
To associate DataPoints with a dataset so that `forget(dataset=...)` can clean them up, pass a `PipelineContext` as the `ctx` argument:
```python theme={null}
from cognee.modules.pipelines.models import PipelineContext
await add_data_points(
[alice, bob, charlie],
ctx=PipelineContext(
user=user, # authenticated user object
dataset=dataset, # dataset object
data_item=data_item, # source data item for provenance
),
)
```
When `ctx` carries all three values, each node and edge is tagged with `dataset_id` and `data_id` in the relational database. `forget(dataset=...)` then finds and removes exactly those records — nodes shared across other datasets are preserved.
When using `Task(add_data_points)` inside `cognee.run_custom_pipeline()`, the pipeline machinery builds and injects `ctx` automatically. If you write a custom task that calls `add_data_points` internally, declare `ctx` in your task signature so the pipeline forwards it:
```python theme={null}
from cognee.modules.pipelines.models import PipelineContext
async def my_custom_task(data, ctx: PipelineContext = None) -> list:
points = build_data_points(data)
return await add_data_points(points, ctx=ctx) # forward ctx for dataset linking
```
## Additional examples
Additional examples about Custom data models are available on our [github](https://github.com/topoteretes/cognee/tree/main/examples/guides).
## Full Example
<Accordion title="Latest guide">
```python theme={null}
import asyncio
from typing import Any
from pydantic import SkipValidation
import cognee
from cognee.infrastructure.engine import DataPoint
from cognee.infrastructure.engine.models.Edge import Edge
from cognee.tasks.storage import add_data_points
class Person(DataPoint):
name: str
# Keep it simple for forward refs / mixed values
knows: SkipValidation[Any] = None # single Person or list[Person]
# Recommended: specify which fields to index for search
metadata: dict = {"index_fields": ["name"]}
async def main():
# Start clean (optional in your app)
await cognee.forget(everything=True)
alice = Person(name="Alice")
bob = Person(name="Bob")
charlie = Person(name="Charlie")
# Create relationships - field name becomes edge label
alice.knows = bob
# You can also do lists: alice.knows = [bob, charlie]
# Optional: add weights and custom relationship types
bob.knows = (Edge(weight=0.9, relationship_type="friend_of"), charlie)
await add_data_points([alice, bob, charlie])
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
<Accordion title="Legacy guide">
```python theme={null}
import asyncio
from typing import Any
from pydantic import SkipValidation
import cognee
from cognee.infrastructure.engine import DataPoint
from cognee.infrastructure.engine.models.Edge import Edge
from cognee.tasks.storage import add_data_points
class Person(DataPoint):
name: str
# Keep it simple for forward refs / mixed values
knows: SkipValidation[Any] = None # single Person or list[Person]
# Recommended: specify which fields to index for search
metadata: dict = {"index_fields": ["name"]}
async def main():
# Start clean (optional in your app)
await cognee.prune.prune_data()
await cognee.prune.prune_system(metadata=True)
alice = Person(name="Alice")
bob = Person(name="Bob")
charlie = Person(name="Charlie")
# Create relationships - field name becomes edge label
alice.knows = bob
# You can also do lists: alice.knows = [bob, charlie]
# Optional: add weights and custom relationship types
bob.knows = (Edge(weight=0.9, relationship_type="friend_of"), charlie)
await add_data_points([alice, bob, charlie])
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
<Note>
This example shows the complete workflow with metadata for indexing and optional edge weights. In practice, you can create complex nested models with multiple relationships and sophisticated data structures.
</Note>
<Columns>
<Card title="Low-Level LLM" icon="cpu" href="/guides/low-level-llm">
Learn about direct LLM interaction
</Card>
<Card title="Core Concepts" icon="brain" href="/core-concepts/overview">
Understand knowledge graph fundamentals
</Card>
<Card title="API Reference" icon="code" href="/api-reference/introduction">
Explore API endpoints
</Card>
</Columns>
# Custom Graph Model
Source: https://docs.cognee.ai/guides/custom-graph-model
Step-by-step guide to creating custom graph models and using remember with them
A minimal guide to creating custom graph models and loading them into Cognee with `remember()`. Reach for one when you want extraction to return a specific set of entity types and relationships instead of a free-form knowledge graph.
<Warning>
Typed `Edge[...]` fields and `FromIdentity()` are **Python SDK only**. A graph model sent over HTTP (`POST /cognify`, `POST /remember`) travels as a JSON schema and is rebuilt from it, and that schema carries neither the `Edge` generics nor the annotation markers — the typed edges are silently lost. Pass the model class directly through the SDK to use them.
</Warning>
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured
* Read [DataPoints](/core-concepts/building-blocks/datapoints) for the conceptual overview of nodes, edges, and metadata
* Have some structured data you want to model
## Code in Action
```python theme={null}
import asyncio
import os
from typing import Annotated, Literal
from cognee import forget, remember, visualize_graph
from cognee.low_level import DataPoint, Edge, FromIdentity
CUSTOM_PROMPT = (
"Extract every person, the role they hold, and every group with its members. "
"Extract friendships, family links (married_to or sibling_of), who reports to whom, "
"and other named relationships between people."
)
class Role(DataPoint):
name: str
metadata: dict = {"index_fields": ["name"], "identity_fields": ["name"]}
class Person(DataPoint):
name: str
is_a: Annotated[Role, FromIdentity()] | None = None
# An edge can also live on the node that owns it. Endpoints of the same type have to
# be named as strings here, because Person is not bound inside its own body yet.
# Put an edge here when one side clearly owns it, as each person has one manager.
reports_to: list[Edge["Person", "Person"]] = []
metadata: dict = {"index_fields": ["name"], "identity_fields": ["name"]}
class Group(DataPoint):
name: str
members: list[Person] | None = None
metadata: dict = {"index_fields": ["name"], "identity_fields": ["name"]}
class PeopleGraph(DataPoint):
# Edges on the root suit a relationship with no obvious owner. Each one shows a way
# of naming: fixed by the field, chosen from a Literal, or free-form from the LLM.
# When building Edge values by hand, set source= explicitly here: an omitted source
# falls back to the declaring node, and this root is not a Person.
people: list[Person]
groups: list[Group] = []
friends_with: list[Edge[Person, Person]] = []
family_links: list[Edge[Person, Person, Literal["married_to", "sibling_of"]]] = []
other_links: list[Edge[Person, Person, str]] = []
async def main():
await forget(everything=True)
text = (
"Maya and Owen are engineers on the Search team and are friends. "
"Priya is a manager and Maya's sibling. Owen mentors Maya. "
"Maya and Owen both report to Priya."
)
await remember(
text,
graph_model=PeopleGraph,
custom_prompt=CUSTOM_PROMPT,
self_improvement=False,
)
graph_path = os.path.join(os.path.dirname(__file__), ".artifacts", "custom_graph.html")
await visualize_graph(graph_path)
if __name__ == "__main__":
asyncio.run(main())
```
## What Just Happened
### Step 1: Define Your Entity Classes and Relationships
```python theme={null}
from typing import Annotated, Literal
from cognee import forget, remember, visualize_graph
from cognee.low_level import DataPoint, Edge, FromIdentity
class Role(DataPoint):
name: str
metadata: dict = {"index_fields": ["name"], "identity_fields": ["name"]}
class Person(DataPoint):
name: str
is_a: Annotated[Role, FromIdentity()] | None = None
# An edge can also live on the node that owns it. Endpoints of the same type have to
# be named as strings here, because Person is not bound inside its own body yet.
# Put an edge here when one side clearly owns it, as each person has one manager.
reports_to: list[Edge["Person", "Person"]] = []
metadata: dict = {"index_fields": ["name"], "identity_fields": ["name"]}
class Group(DataPoint):
name: str
members: list[Person] | None = None
metadata: dict = {"index_fields": ["name"], "identity_fields": ["name"]}
```
Create Pydantic models that inherit from `DataPoint` to represent your node types. A field holding another `DataPoint` becomes an edge named after the field — `members: list[Person]` extracts `Person` nodes and the `members` edges from each `Group` — while `Annotated[Role, FromIdentity()]` asks the LLM for a role's identity string instead of a nested object, and `list[Edge["Person", "Person"]]` asks for flat `reports_to` relationship rows. The `metadata` dict sets which fields are embedded for retrieval (`index_fields`) and which derive a deterministic node id (`identity_fields`), so repeated mentions of `Maya` collapse into one node instead of duplicating.
### Step 2: Define Your Top-Level Graph Container
```python theme={null}
class PeopleGraph(DataPoint):
# Edges on the root suit a relationship with no obvious owner. Each one shows a way
# of naming: fixed by the field, chosen from a Literal, or free-form from the LLM.
# When building Edge values by hand, set source= explicitly here: an omitted source
# falls back to the declaring node, and this root is not a Person.
people: list[Person]
groups: list[Group] = []
friends_with: list[Edge[Person, Person]] = []
family_links: list[Edge[Person, Person, Literal["married_to", "sibling_of"]]] = []
other_links: list[Edge[Person, Person, str]] = []
```
Wrap your top-level entities in a container model; this is what you pass as `graph_model`. Declaring an edge on the root fits a relationship neither side owns, such as a friendship, while `reports_to` sits on `Person` because the reporting line belongs to the person rather than to the graph as a whole. The third `Edge` type parameter decides how the relationship is named.
### Step 3: Remember Your Data with the Custom Model
```python theme={null}
CUSTOM_PROMPT = (
"Extract every person, the role they hold, and every group with its members. "
"Extract friendships, family links (married_to or sibling_of), who reports to whom, "
"and other named relationships between people."
)
text = (
"Maya and Owen are engineers on the Search team and are friends. "
"Priya is a manager and Maya's sibling. Owen mentors Maya. "
"Maya and Owen both report to Priya."
)
await remember(
text,
graph_model=PeopleGraph,
custom_prompt=CUSTOM_PROMPT,
self_improvement=False,
)
```
This ingests the text and builds the graph in one call. The custom `graph_model` acts as the extraction schema while `CUSTOM_PROMPT` tells the LLM what to look for, and the prompt names the `married_to` and `sibling_of` options explicitly so the LLM fills `family_links` with the values that `Literal` allows.
### Step 4: Visualize Your Data
```python theme={null}
graph_path = os.path.join(os.path.dirname(__file__), ".artifacts", "custom_graph.html")
await visualize_graph(graph_path)
```
This renders the generated graph to `custom_graph.html` so you can verify nodes, relationships, and overall schema behavior.
## Advanced Usage
<Accordion title="Custom graph models and ontologies">
A custom graph model constrains extraction *before* the LLM runs, while an [ontology](/core-concepts/further-concepts/ontologies) grounds names *after* it. See "How does an ontology relate to a custom graph?" under [Additional details and examples](/core-concepts/further-concepts/ontologies#additional-details-and-examples) for the trade-offs and for how to combine them.
</Accordion>
<Accordion title="How graph_model constrains LLM extraction">
When you pass `graph_model=...`, that model **is** the structured-output schema the LLM must fill in. Internally, Cognee hands your model to the LLM as the `response_model` for structured extraction, so the LLM can only return entities and relationships that fit the fields you declared — it is not free to invent an arbitrary shape.
* **Default:** without `graph_model`, Cognee uses its general-purpose `KnowledgeGraph` schema (free-form nodes and edges).
* **Custom model:** when you pass a `DataPoint` subclass, the extraction schema is built from every domain field the model carries — including fields it inherits from your own intermediate `DataPoint` subclasses, not just the ones annotated on the class you pass. If `Animal(DataPoint)` declares `species: str` and `Dog(Animal)` adds `breed: str`, the LLM is asked for both. Only the fields defined on `DataPoint` itself (`id`, `version`, `type`, `created_at`, `metadata`, and the rest of the infrastructure fields) are stripped out, so they do not expand the LLM's response schema; the stripping is by field name, so a subclass that overrides `metadata` — as `Person` does above — still keeps it out of the schema. Adding a field (e.g. `age: int` on `Person`) tells the LLM to extract that value; nested `DataPoint` fields (like `members: list[Person]` on `Group`) tell it to extract those related entities and the edges between them, and `Edge[...]` fields ask for relationship rows it resolves against those entities.
* **`custom_prompt` vs `graph_model`:** they play different roles. `graph_model` defines the *shape* (which fields and relationships are allowed), while `custom_prompt` replaces the system prompt that tells the LLM *what to look for*. Use them together for predictable, domain-specific extraction.
</Accordion>
<Accordion title="Naming a typed relationship">
The third type parameter of `Edge` controls how the relationship is named:
* Omitted (`friends_with`) — the edge takes the field's name.
* `Literal["married_to", "sibling_of"]` (`family_links`) — the LLM picks one of the listed names.
* `str` (`other_links`) — the LLM supplies a free-form name, which Cognee normalizes.
</Accordion>
<Accordion title="How edge rows are resolved">
Cognee resolves each row's `source` and `target` against the nodes extracted from the same text, matching on the identity value. When a row names an endpoint that no extracted node matches, that single row is dropped and logged (`Skipping unresolved edge on <field>: <row>`); the rest of the chunk still stores.
Because both features address nodes by their identity value, every class at either end of an edge — and every `FromIdentity()` target — needs exactly one `identity_fields` entry, and a `FromIdentity()` target must be constructible from that field alone. `FromIdentity()` accepts `Target`, `Target | None`, `list[Target]`, and `list[Target] | None`; any other shape would hand the LLM a whole nested object and mint a duplicate node, so it is rejected outright. Cognee raises `InvalidReferenceTypeError` when a model breaks either rule, while it builds the extraction schema and before the LLM is called.
Those rules cover edges extraction fills in. When you construct `Edge` values **by hand** instead, an omitted `source` falls back to the node the field is declared on — which is what you want on an owner-declared edge like `reports_to`, but not on a root container: `PeopleGraph` is not a `Person`, so a parametrized edge there raises `ValueError` unless you set `source=` explicitly. A missing `target` is always an error. The older unparametrized tuple form records no generics and keeps the permissive fallback.
</Accordion>
<Accordion title="Keeping the container out of the graph">
The container is stored as a node too, so the graph gains a `PeopleGraph` node with a `people` edge to each `Person`. To keep it as an extraction schema only, add `metadata: dict = {"index_fields": [], "transparent": True}` — Cognee then skips the wrapper and promotes each `Person` to a top-level node. See [Transparent containers](/core-concepts/building-blocks/datapoints#transparent-containers-nodes-that-group-rather-than-describe).
</Accordion>
## Declaring a model in JSON
The same model can be written as a plain JSON document — a **graph schema spec** — and compiled with `graph_model_from_spec()`, which returns the `DataPoint`-derived class you pass as `graph_model`. It is the shape the Cognee UI's graph-model editor produces, so a model built in the UI can be handed straight to the SDK. See [Graph Model from JSON](/guides/graph-model-from-json) for the spec reference, the validation rules, and a runnable walkthrough.
## Use in Custom Tasks and Pipelines
This pattern is useful when you need predictable, domain-specific extraction inside custom workflows.
* Reuse the same graph schema across tasks to keep outputs consistent
* Run `remember(graph_model=...)` in workflows where downstream logic expects a specific graph shape
* Combine with [custom prompts](/guides/custom-prompts) or [custom tasks](/guides/custom-tasks-pipelines) to refine extraction
* Validate pipeline results with `visualize_graph` before promoting changes to production
<Columns>
<Card title="DataPoints" icon="circle" href="/core-concepts/building-blocks/datapoints">
The node, edge, and metadata model these classes are built on.
</Card>
<Card title="remember()" icon="brain" href="/python-api/remember">
Every parameter `remember()` accepts, including `graph_model`.
</Card>
<Card title="Custom Data Models" icon="circle-stop" href="/guides/custom-data-models">
Build DataPoints yourself and insert them with `add_data_points`, skipping LLM extraction.
</Card>
<Card title="More examples" icon="github" href="https://github.com/topoteretes/cognee/tree/main/examples/guides">
Runnable guide scripts, including this one, in the cognee repo.
</Card>
</Columns>
# Custom Prompts
Source: https://docs.cognee.ai/guides/custom-prompts
Step-by-step guide to using custom prompts to control graph extraction
A minimal guide to shaping graph extraction with a custom LLM prompt. You'll pass your prompt via `custom_prompt` to `cognee.remember()` to control entity types, relationship labels, and extraction rules.
<Tip>
For the built-in graph extraction prompts selected through `GRAPH_PROMPT_PATH`, see [Cognify](/core-concepts/main-operations/legacy-operations/cognify).
</Tip>
**Before you start:**
* Complete [Quickstart](getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](setup-configuration/llm-providers) configured
* Have some text or files to process
## Code in Action
### Step 1: Write a Custom Prompt
```python theme={null}
custom_prompt = """
Extract only people and cities as entities.
Connect people to cities with the relationship "lives_in".
Ignore all other entities.
"""
```
The custom prompt overrides the default system prompt used during entity/relationship extraction. It constrains node types, enforces relationship naming, and reduces noise.
<Note>
`custom_prompt` is ignored when `temporal_cognify=True`.
</Note>
### Step 2: Remember with Your Custom Prompt
```python theme={null}
await cognee.forget(everything=True)
await cognee.remember(
[
"Alice moved to Paris in 2010, while Bob has always lived in New York.",
"Andreas was born in Venice, but later settled in Lisbon.",
"Diana and Tom were born and raised in Helsinki. Diana currently resides in Berlin, while Tom never moved.",
],
custom_prompt=custom_prompt,
self_improvement=False,
)
```
This resets the local state and then uses `remember()` to ingest the text and build the graph in one pass. The same approach works with multiple documents, files, or entire datasets.
### Step 3: Ask Questions
```python theme={null}
res = await cognee.recall(
query_type=SearchType.GRAPH_COMPLETION,
query_text="Where does Alice live?",
)
```
Use `cognee.recall(...)` with `SearchType.GRAPH_COMPLETION` to get answers that leverage your custom extraction rules.
## `custom_prompt` vs `system_prompt`: which prompt goes where
Cognee uses **two distinct prompts at two different stages**, and they are not interchangeable. Passing one where the other is expected will silently have no effect.
<Tabs>
<Tab title="Cognee v1.0">
| Operation | Prompt parameter | What it influences | Stage |
| ------------------------------------------- | ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
| [`cognee.remember()`](/python-api/remember) | `custom_prompt` | Entity & relationship extraction — `remember()` runs `add()` + `cognify()` under the hood, so this overrides the graph extraction prompt used during graph build. | Graph build |
| [`cognee.recall()`](/python-api/recall) | `system_prompt` (or `system_prompt_path`) | Answer generation — instructs the LLM how to compose the final answer from retrieved context. Only applies to completion-style retrieval (`GRAPH_COMPLETION`, `RAG_COMPLETION`, `TRIPLET_COMPLETION`, etc.); ignored by `only_context=True` and by non-completion retrieval paths. | Query time |
</Tab>
<Tab title="Legacy">
| Operation | Prompt parameter | What it influences | Stage |
| ----------------------------------------- | ----------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------- |
| [`cognee.add()`](/python-api/add) | *none* | — (pure ingestion; no LLM call) | Ingestion |
| [`cognee.cognify()`](/python-api/cognify) | `custom_prompt` | Entity & relationship extraction — overrides the default graph extraction prompt (see [`GRAPH_PROMPT_PATH`](/core-concepts/main-operations/legacy-operations/cognify#default-extraction-prompts)) | Graph build |
| [`cognee.search()`](/python-api/search) | `system_prompt` (or `system_prompt_path`) | Answer generation — instructs the LLM how to compose the final answer from retrieved context. Only applies to completion-style search types; ignored by `CHUNKS`, `SUMMARIES`, `CYPHER`, and when `only_context=True`. | Query time |
</Tab>
</Tabs>
## Additional examples
Additional examples about Custom prompts are available on our [github](https://github.com/topoteretes/cognee/tree/main/examples/guides).
## Full Example
<Accordion title="Latest guide">
```python theme={null}
import asyncio
import cognee
from cognee.api.v1.search import SearchType
custom_prompt = """
Extract only people and cities as entities.
Connect people to cities with the relationship "lives_in".
Ignore all other entities.
"""
async def main():
await cognee.forget(everything=True)
await cognee.remember(
[
"Alice moved to Paris in 2010, while Bob has always lived in New York.",
"Andreas was born in Venice, but later settled in Lisbon.",
"Diana and Tom were born and raised in Helsinki. Diana currently resides in Berlin, while Tom never moved.",
],
custom_prompt=custom_prompt,
self_improvement=False,
)
res = await cognee.recall(
query_type=SearchType.GRAPH_COMPLETION,
query_text="Where does Alice live?",
)
print(res)
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
<Accordion title="Legacy guide">
```python theme={null}
import asyncio
import cognee
from cognee.api.v1.search import SearchType
custom_prompt = """
Extract only people and cities as entities.
Connect people to cities with the relationship "lives_in".
Ignore all other entities.
"""
async def main():
await cognee.add([
"Alice moved to Paris in 2010, while Bob has always lived in New York.",
"Andreas was born in Venice, but later settled in Lisbon.",
"Diana and Tom were born and raised in Helsingy. Diana currently resides in Berlin, while Tom never moved."
])
await cognee.cognify(custom_prompt=custom_prompt)
res = await cognee.recall(
query_type=SearchType.GRAPH_COMPLETION,
query_text="Where does Alice live?",
)
print(res)
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
<Note>
This simple example uses a few strings for demonstration. In practice, you can add multiple documents, files, or entire datasets - the custom prompt processing works the same way across all your data.
</Note>
<Tip>
If you are running Cognee as a server and want to infer a schema or generate a prompt through HTTP instead of writing it by hand, see the **LLM Utility Endpoints** examples in [Deploy REST API Server](/guides/deploy-rest-api-server#llm-utility-endpoints).
</Tip>
<Columns>
<Card title="Core Concepts" icon="brain" href="/core-concepts/overview">
Understand knowledge graph fundamentals
</Card>
<Card title="Ontology Quickstart" icon="git-branch" href="/guides/ontology-support">
Learn about ontology integration
</Card>
<Card title="API Reference" icon="code" href="/api-reference/introduction">
Explore API endpoints
</Card>
</Columns>
# Custom Tasks and Pipelines
Source: https://docs.cognee.ai/guides/custom-tasks-pipelines
Step-by-step guide to creating custom tasks and pipelines
A minimal guide to creating custom tasks and pipelines. The updated example builds a lightweight ingestion object, extracts people with an LLM task, stores the resulting graph nodes, and then visualizes the result.
**Before you start:**
* Complete [Quickstart](getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](setup-configuration/llm-providers) configured
* Have some text data to process
## What Custom Tasks and Pipelines Do
* Define custom processing steps using `Task` objects
* Chain multiple operations together in a custom pipeline
* Use LLMs to extract structured data from text
* Insert structured data directly into the knowledge graph
* Control the entire data processing workflow
## Code in Action
### Step 1: Define Your Pipeline Models
```python theme={null}
class PersonLLM(BaseModel):
name: str
knows: list[str] = []
class Person(DataPoint):
name: str
knows: list["Person"] = []
metadata: dict[str, Any] = {"index_fields": ["name"]}
class LightweightData(DataPoint):
id: UUID
text: str
```
`PersonLLM` and `PeopleLLM` describe the structured output the LLM should return. `Person` is the graph-ready `DataPoint`, and `LightweightData` gives the pipeline a simple ingestion object with a stable ID and text field.
### Step 2: Create Your Custom Task
```python theme={null}
async def extract_people(data: LightweightData) -> list[Person]:
system_prompt = (
"Extract people mentioned in the text. "
"Return as `persons: Person[]` with each Person having `name` and optional `knows` relations. "
"Infer ‘knows’ only when there is a clear interpersonal interaction in the text."
)
person_map: dict[str, Person] = {}
...
```
This task uses the LLM to extract lightweight people records, then resolves those names into graph-ready `Person` objects with `knows` relationships.
<Tip>
`acreate_structured_output` is backend-agnostic (LiteLLM Native by default; Instructor and BAML are opt-in). Configure it through `STRUCTURED_OUTPUT_FRAMEWORK` in `.env`.
</Tip>
### Step 3: Build and Run Your Pipeline
```python theme={null}
tasks = [
Task(extract_people),
Task(add_data_points),
]
await cognee.run_custom_pipeline(
tasks=tasks, data=build_lightweight_data_object(text_data), dataset="people_demo"
)
await cognee.cognify()
```
The custom pipeline inserts the extracted `Person` nodes directly into the graph. The follow-up `cognify()` call builds the rest of Cognee's retrieval stack on top of that stored graph data.
<Note>
If a task raises an exception while processing a data item, the pipeline run yields a `PipelineRunErrored` status and then re-raises the original exception to the caller, instead of failing silently. Wrap your pipeline run in `try`/`except` so you can handle the propagated error.
</Note>
### Step 4: Visualize the Result
```python theme={null}
visualize_graph_path = os.path.join(
os.path.dirname(__file__), ".artifacts", "custom_tasks_and_pipelines.html"
)
await visualize_graph(visualize_graph_path)
```
The example writes an HTML graph visualization so you can inspect the entities and inferred `knows` edges produced by the custom pipeline.
## Use Cases
This approach is particularly useful when you need to:
* Extract structured data from unstructured text
* Process data through multiple custom steps
* Control the entire data processing workflow
* Combine LLM extraction with programmatic data insertion
* Build complex data processing pipelines
## Additional information
<Accordion title="Importing an existing graph">
If you already have nodes and edges — exported from another graph database, or stored as JSON/CSV — you don't need the LLM to re-extract them. Map your data to [`DataPoint`](/guides/custom-graph-model) models and store it directly with `add_data_points`. Because `run_custom_pipeline` can work with already-built graphs, this recreates your graph deterministically (no LLM extraction step):
* **Nodes** become `DataPoint` subclasses, one per entity type.
* **Edges** are expressed as nested `DataPoint` fields — the field name becomes the relationship label (e.g. `employees: list[Person]` creates `employees` edges).
* **Typed or weighted edges** use the `Edge` model: declare the field as `SkipValidation[Any]` and set values to `(Edge(relationship_type="manager", weight=0.9), node)` tuples.
```python theme={null}
import cognee
from cognee.infrastructure.engine import DataPoint
from cognee.modules.pipelines import Task
from cognee.tasks.storage import add_data_points
class Person(DataPoint):
name: str
metadata: dict = {"index_fields": ["name"], "identity_fields": ["name"]}
class Company(DataPoint):
name: str
employees: list[Person] # the field name becomes the edge label
metadata: dict = {"index_fields": ["name"], "identity_fields": ["name"]}
def build_nodes(rows) -> list[Company]:
# Map your existing nodes and edges (a graph export, JSON, CSV, ...) onto DataPoints
return [
Company(name=r["name"], employees=[Person(name=p) for p in r["people"]])
for r in rows
]
tasks = [Task(build_nodes), Task(add_data_points)]
await cognee.run_custom_pipeline(tasks=tasks, data=my_rows, dataset="imported_graph")
```
For deterministic re-imports, either map stable source IDs into each DataPoint's `id` field or configure `identity_fields` as shown above. Reusing the same node `id` updates the existing node instead of creating a duplicate. For a complete example that loads nodes and edges from JSON files, see the [organizational hierarchy pipeline](https://github.com/topoteretes/cognee/tree/dev/examples/demos/custom_pipelines/organizational_hierarchy) on GitHub.
</Accordion>
## Additional examples
Additional examples about custom tasks and pipelines are available on our [github](https://github.com/topoteretes/cognee/tree/main/examples/guides).
## Full Example
<Accordion title="Latest guide">
```python theme={null}
import asyncio
import os
from typing import Any
from uuid import NAMESPACE_OID, UUID, uuid5
from pydantic import BaseModel
import cognee
from cognee import visualize_graph
from cognee.infrastructure.engine import DataPoint
from cognee.infrastructure.llm.LLMGateway import LLMGateway
from cognee.modules.engine.operations.setup import setup
from cognee.modules.pipelines import Task
from cognee.tasks.storage import add_data_points
class PersonLLM(BaseModel):
"""Lightweight Pydantic model for LLM extraction only."""
name: str
knows: list[str] = [] # Just names for now, we'll resolve to Person instances later
class PeopleLLM(BaseModel):
"""Lightweight Pydantic model for LLM extraction only."""
persons: list[PersonLLM]
class Person(DataPoint):
name: str
# Optional relationships (we'll let the LLM populate this)
knows: list["Person"] = []
# Make names searchable in the vector store
metadata: dict[str, Any] = {"index_fields": ["name"]}
class LightweightData(DataPoint):
"""Lightweight DataPoint model for data ingestion only."""
id: UUID
text: str
def build_lightweight_data_object(text_data):
return LightweightData(id=uuid5(NAMESPACE_OID, text_data), text=text_data)
async def extract_people(data: LightweightData) -> list[Person]:
system_prompt = (
"Extract people mentioned in the text. "
"Return as `persons: Person[]` with each Person having `name` and optional `knows` relations. "
"Infer ‘knows’ only when there is a clear interpersonal interaction in the text."
)
# Create a mapping of name -> Person DataPoint
person_map: dict[str, Person] = {}
for data_item in data:
people_llm = await LLMGateway.acreate_structured_output(
data_item.text, system_prompt, PeopleLLM
)
for person_llm in people_llm.persons:
person_map[person_llm.name] = Person(name=person_llm.name)
# Resolve knows relationships
for person_llm in people_llm.persons:
person = person_map[person_llm.name]
person.knows = [person_map[name] for name in person_llm.knows if name in person_map]
return list(person_map.values())
async def main(text_data):
await cognee.forget(everything=True)
await setup()
tasks = [
Task(extract_people), # input: text -> output: list[Person]
Task(add_data_points), # input: list[Person] -> output: list[Person]
]
await cognee.run_custom_pipeline(
tasks=tasks, data=build_lightweight_data_object(text_data), dataset="people_demo"
)
await cognee.cognify()
visualize_graph_path = os.path.join(
os.path.dirname(__file__), ".artifacts", "custom_tasks_and_pipelines.html"
)
await visualize_graph(visualize_graph_path)
if __name__ == "__main__":
text = "Alice knows Mark. Mark had dinner with Bob and Alice. Bob knows Mary."
asyncio.run(main(text))
```
</Accordion>
<Accordion title="Legacy guide">
```python theme={null}
import asyncio
from typing import Any, Dict, List
from pydantic import BaseModel, SkipValidation
import cognee
from cognee.modules.engine.operations.setup import setup
from cognee.infrastructure.llm.LLMGateway import LLMGateway
from cognee.infrastructure.engine import DataPoint
from cognee.tasks.storage import add_data_points
from cognee.modules.pipelines import Task, run_pipeline
class Person(DataPoint):
name: str
# Optional relationships (we'll let the LLM populate this)
knows: List["Person"] = []
# Make names searchable in the vector store
metadata: Dict[str, Any] = {"index_fields": ["name"]}
class People(BaseModel):
persons: List[Person]
async def extract_people(text: str) -> List[Person]:
system_prompt = (
"Extract people mentioned in the text. "
"Return as `persons: Person[]` with each Person having `name` and optional `knows` relations. "
"If the text says someone knows someone set `knows` accordingly. "
"Only include facts explicitly stated."
)
people = await LLMGateway.acreate_structured_output(text, system_prompt, People)
return people.persons
async def main():
await cognee.prune.prune_data()
await cognee.prune.prune_system(metadata=True)
await setup()
text = "Alice knows Bob."
tasks = [
Task(extract_people), # input: text -> output: list[Person]
Task(add_data_points) # input: list[Person] -> output: list[Person]
]
async for _ in run_pipeline(tasks=tasks, data=text, datasets=["people_demo"]):
pass
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
<Note>
This updated example uses a lightweight ingestion object, a custom extraction task, and a visualization step. In practice, you can create larger pipelines with additional transforms and storage stages.
</Note>
<Columns>
<Card title="Custom Data Models" icon="circle-stop" href="/guides/custom-data-models">
Learn about custom data models
</Card>
<Card title="Low-Level LLM" icon="cpu" href="/guides/low-level-llm">
Learn about direct LLM interaction
</Card>
<Card title="Core Concepts" icon="brain" href="/core-concepts/overview">
Understand knowledge graph fundamentals
</Card>
</Columns>
# Deploy REST API Server
Source: https://docs.cognee.ai/guides/deploy-rest-api-server
Deploy Cognee as a REST API server using Docker or Python
Deploy Cognee as a REST API server to expose its functionality via HTTP endpoints.
## Setup
```bash theme={null}
# Clone repository
git clone https://github.com/topoteretes/cognee.git
cd cognee
# Configure environment
cp .env.template .env
```
<Info>
Edit `.env` with your preferred configuration. See [Setup Configuration](/setup-configuration/overview) guides for all available options.
</Info>
## Deployment Methods
<Tabs>
<Tab title="Docker">
### Start Server
```bash theme={null}
# Start API server
docker compose up --build cognee
# Check status
docker compose ps
```
</Tab>
<Tab title="Python (Local)">
### Setup
```bash theme={null}
# Create virtual environment
uv venv && source .venv/bin/activate
# Install with all extras
uv sync --all-extras
```
### Start Server
```bash theme={null}
# Run API server
uvicorn cognee.api.client:app --host 0.0.0.0 --port 8000
```
Alternatively, launch the server through the module entry point, which accepts a `--agent-mode` flag:
```bash theme={null}
# Standard mode (defaults to port 8000)
python cognee/api/client.py
# Agent mode (defaults to port 8011)
python cognee/api/client.py --agent-mode
```
Passing `--agent-mode` sets `COGNEE_AGENT_MODE=true` for the process and overrides the environment variable if it is also set. Agent mode is intended for ephemeral deployments where an orchestrator spins up Cognee for one or more agents; the server tracks agents that call `POST /api/v1/agents/register` and shuts itself down once the active count drops back to zero (driven by `POST /api/v1/agents/unregister`). The auto-shutdown watchdog only starts after the first connection registers, so a fresh agent-mode server stays up while it waits. Host and port can still be overridden with `HTTP_API_HOST` and `HTTP_API_PORT`.
</Tab>
</Tabs>
## Access API
* **API:** [http://localhost:8000](http://localhost:8000)
* **Documentation:** [http://localhost:8000/docs](http://localhost:8000/docs)
## Agent Mode
Cognee can run in **agent mode**, which tracks active agent connections and shuts the
server down once they all disconnect. This is intended for ephemeral deployments where
an external orchestrator launches a Cognee server for one or more agents and tears it
down when they finish.
Enable agent mode in either of two ways:
```bash theme={null}
# CLI flag (when running the API server directly)
python -m cognee.api.client --agent-mode
# Environment variable
COGNEE_AGENT_MODE=true uvicorn cognee.api.client:app --host 0.0.0.0
```
When agent mode is enabled:
* The **default port becomes `8011`** (instead of `8000`). The CLI flag overrides the
`COGNEE_AGENT_MODE` env var. `HTTP_API_PORT` still wins if you set it explicitly.
* A background watchdog starts **after the first** `POST /api/v1/agents/register` call
and checks the active connection count every **60 seconds**. When the count drops to
zero, the watchdog sends `SIGTERM` to the server process.
* The server stays alive indefinitely while waiting for the first registration —
the watchdog does not arm until then.
Agents call `POST /api/v1/agents/register` on connect and `POST /api/v1/agents/unregister`
on disconnect; see the **Agent Management** accordion below for the full surface.
## Authentication
If `REQUIRE_AUTHENTICATION=true` in your `.env` file:
1. **Register:** `POST /api/v1/auth/register`
2. **Login:** `POST /api/v1/auth/login`
3. **Use token:** Include `Authorization: Bearer <token>` header or use cookies
## Python SDK Client
After deploying the server, connect the Python SDK to your running instance using `cognee.serve()`:
```python theme={null}
import cognee
import asyncio
async def main():
client = await cognee.serve(url="http://localhost:8000", api_key="...")
# serve() probes an authenticated endpoint and raises on a rejected key.
# Omit api_key only when the server runs with ENABLE_BACKEND_ACCESS_CONTROL=false.
# Ingest data and build the knowledge graph in one step
await client.remember("Cognee turns documents into AI memory.", dataset_name="docs")
# Query the knowledge graph
results = await client.recall("What does Cognee do?")
for result in results:
print(result)
await cognee.disconnect()
asyncio.run(main())
```
You can also configure the connection via environment variables instead of passing arguments to `serve()`:
```bash theme={null}
export COGNEE_SERVICE_URL="http://localhost:8000"
export COGNEE_API_KEY="<your-key>" # omit only when ENABLE_BACKEND_ACCESS_CONTROL=false
```
```python theme={null}
client = await cognee.serve() # reads COGNEE_SERVICE_URL and COGNEE_API_KEY
```
The `CloudClient` returned by `serve()` exposes four methods that map to the server's V2 endpoints: `remember()` (ingest + cognify), `recall()` (search), `improve()` (enrich graph), and `forget()` (delete). Call `await cognee.disconnect()` to revert to local mode.
### Uploading skills
`client.remember(..., content_type="skills")` ingests local `SKILL.md` files as Skill nodes. Pass either a single `SKILL.md` file path or a directory; directories are searched recursively for `SKILL.md` files. The client reads the local file contents and uploads their bytes (preserving the relative folder layout), so the path is resolved on the **caller's** machine rather than on the server:
```python theme={null}
client = await cognee.serve(url="http://localhost:8000")
# A directory tree — every SKILL.md under ./skills is uploaded
await client.remember("./skills", dataset_name="agent_skills", content_type="skills")
# Or a single SKILL.md file
await client.remember("./skills/demo/SKILL.md", dataset_name="agent_skills", content_type="skills")
```
The client raises `FileNotFoundError` when the path does not exist and `ValueError` when a directory contains no `SKILL.md` files.
<Note>
When a skill push reaches the server without any file named `SKILL.md` — for example a direct `POST /api/v1/remember` upload with `content_type=skills` whose uploaded files use other names — the server now ingests each uploaded file as an individual skill instead of skipping the push. Pushes that already contain `SKILL.md` files are ingested as before, preserving their folder layout.
</Note>
## HTTP API Examples
<AccordionGroup>
<Accordion title="Authentication">
**Register a user:**
```bash theme={null}
curl -X POST "http://localhost:8000/api/v1/auth/register" \
-H "Content-Type: application/json" \
-d '{"email": "user1@example.com", "password": "strong_password"}'
```
**Login and get token:**
```bash theme={null}
TOKEN="$(curl -s -X POST http://localhost:8000/api/v1/auth/login \
-H 'Content-Type: application/x-www-form-urlencoded' \
-d 'username=user1@example.com&password=strong_password' | jq -r .access_token)"
```
</Accordion>
<Accordion title="Dataset Management">
**Create a dataset:**
```bash theme={null}
curl -X POST http://localhost:8000/api/v1/datasets \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{"name": "project_docs"}'
```
**List datasets:**
```bash theme={null}
curl -H "Authorization: Bearer $TOKEN" http://localhost:8000/api/v1/datasets
```
</Accordion>
<Accordion title="Data Operations">
<Tabs>
<Tab title="Cognee v1.0">
**Remember data and build memory in one call:**
```bash theme={null}
curl -X POST http://localhost:8000/api/v1/remember \
-H "Authorization: Bearer $TOKEN" \
-F "data=@/absolute/path/to/file.pdf" \
-F "datasetName=project_docs" \
-F "chunk_size=1024" \
-F "chunks_per_batch=20" \
-F "run_in_background=false"
```
**Recall from a dataset with explicit retrieval settings:**
```bash theme={null}
curl -X POST http://localhost:8000/api/v1/recall \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{"query": "What are the main topics?", "datasets": ["project_docs"], "search_type": "GRAPH_COMPLETION", "top_k": 10}'
```
**Stream the recall answer as server-sent events:**
```bash theme={null}
curl -N -X POST http://localhost:8000/api/v1/recall \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{"query": "What are the main topics?", "datasets": ["project_docs"], "stream": true}'
```
Setting `"stream": true` in the body switches the response to `text/event-stream` (SSE). Omitting the field leaves the decision to content negotiation: streaming happens only when the `Accept` header names `text/event-stream` explicitly — a `text/*` wildcard does not count — and ranks it strictly above `application/json`, so the `Accept: */*` default sent by `curl`, `fetch`, `httpx`, and `requests` — and the MCP-style `application/json, text/event-stream` — stay on the plain JSON response. `"stream": false` forces JSON regardless of the header.
Nothing is sent until the recall has either produced output or failed, so permission and validation failures still arrive as the same `403`/`422` responses the JSON path returns — only failures after the first byte are reported inside the stream. The body carries these event types:
* `stage` — `{"stage": "retrieving"}`, then `{"stage": "generating"}` when a streamable answer call starts
* `delta` — `{"text": "..."}`, a fragment of the answer as it is generated. Deltas are a preview and can legitimately be absent
* `answer_done` — the answer text is complete
* `reset` — discard any rendered deltas and wait for `final` (sent after an LLM retry, or when preview frames were dropped)
* `error` — `{"message": "...", "status": ...}`, carrying the status code the JSON response would have used
* `final` — `{"results": [...]}`, where `results` is exactly the array the JSON response returns: the authoritative payload
`delta` events carry preconditions the other event types do not, and a request that misses any one of them still returns a valid stream — it simply skips from `stage` straight to an unchanged `final`. Deltas flow only when all of the following hold:
* The server sets `LLM_ANSWER_STREAMING=true` and the configured provider path supports token streaming (see [LLM providers](/setup-configuration/llm-providers#configuration))
* The answer runs as a concurrent session turn: `CACHING` is on and `SESSION_SEARCH_MODE` is left at `concurrent` (see [Sessions and Caching](/core-concepts/sessions-and-caching))
* The request asks for a plain-text answer — neither `only_context: true` nor a `response_schema`
* `search_type` is one of `HYBRID_COMPLETION` (the default), `GRAPH_COMPLETION`, `RAG_COMPLETION`, or `TRIPLET_COMPLETION`. The graph-completion variants (`GRAPH_COMPLETION_COT`, `GRAPH_COMPLETION_DECOMPOSITION`, `GRAPH_COMPLETION_CONTEXT_EXTENSION`, `GRAPH_SUMMARY_COMPLETION`) and the retrieval-only types do not stream
Comment lines (`: keepalive`) are emitted every 15 seconds during quiet phases so proxies do not drop the connection — SSE clients ignore them automatically. If the client disconnects mid-stream, the recall still runs to completion and the session turn is persisted.
**Improve an existing dataset in the background:**
```bash theme={null}
curl -X POST http://localhost:8000/api/v1/improve \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{"dataset_name": "project_docs", "run_in_background": true}'
```
**Forget only derived memory and keep the uploaded files:**
```bash theme={null}
curl -X POST http://localhost:8000/api/v1/forget \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{"dataset": "project_docs", "memory_only": true}'
```
</Tab>
<Tab title="Legacy Operations">
Use these lower-level endpoints when you want to keep ingestion, graph building, and retrieval as separate steps.
**Add data (upload file):**
```bash theme={null}
curl -X POST http://localhost:8000/api/v1/add \
-H "Authorization: Bearer $TOKEN" \
-F "data=@/absolute/path/to/file.pdf" \
-F "datasetName=project_docs"
```
**Build the knowledge graph with a custom chunk size:**
```bash theme={null}
curl -X POST http://localhost:8000/api/v1/cognify \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{"datasets": ["project_docs"], "chunk_size": 1024}'
```
**Search data:**
```bash theme={null}
curl -X POST http://localhost:8000/api/v1/search \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $TOKEN" \
-d '{"query": "What are the main topics?", "datasets": ["project_docs"], "top_k": 10}'
```
</Tab>
</Tabs>
</Accordion>
<Accordion title="Uploading files, raw text, and remote servers">
Both `POST /api/v1/remember` and `POST /api/v1/add` expect **`multipart/form-data`**, where `data` is one or more **file uploads** — not a JSON body or a plain form string. Sending text directly (for example `-F "data=some text"` or a JSON `{"data": "..."}` body) fails validation with:
```
Value error, Expected UploadFile, received: <class 'str'>
```
Attach a file with curl's `@` prefix instead:
```bash theme={null}
curl -X POST http://localhost:8000/api/v1/remember \
-H "Authorization: Bearer $TOKEN" \
-F "data=@/absolute/path/to/file.pdf" \
-F "datasetName=project_docs"
```
To ingest raw text, write it to a file first and upload that file:
```bash theme={null}
echo "Cognee turns documents into AI memory." > note.txt
curl -X POST http://localhost:8000/api/v1/remember \
-H "Authorization: Bearer $TOKEN" \
-F "data=@note.txt" \
-F "datasetName=project_docs"
```
You can attach multiple files by repeating `-F "data=@..."`. If you prefer to send raw strings as JSON, use the [Python SDK](/python-api/remember) (`await client.remember("some text", ...)`) or the [`POST /api/v1/skills`](/api-reference/introduction) JSON endpoint for skill markdown — the multipart endpoints always require file uploads.
**Targeting a remote (non-localhost) server:** replace `http://localhost:8000` with your server's address, e.g. `http://<host-or-ip>:8000` on a private network or `https://cognee.example.com` behind a reverse proxy. Bind the server to a reachable interface with `--host 0.0.0.0` (see the [Python (Local)](#deployment-methods) tab), and keep [authentication](#authentication) enabled whenever the server is not on a trusted, private network.
</Accordion>
<Accordion title="Activity and Observability">
The `/api/v1/activity` router exposes endpoints for pipeline run history, trace data, tenant or agent monitoring, and dataset export. All endpoints require authentication.
| Endpoint | Description |
| ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GET /api/v1/activity/pipeline-runs` | Recent activity, newest first — both pipeline runs and single-row operation records (`search`, `recall`, `remember`, `forget`, `delete`, `prune`), told apart by the `kind` field. Accepts `?dataset_id=<uuid>`, `?pipeline_name=<name>`, `?limit=` (1–500, default 50) and `?offset=` (default 0). |
| `GET /api/v1/activity/spans` | In-memory OTEL span buffer (last 50 traces). Requires `COGNEE_TRACING_ENABLED=true`. Returns an empty list when tracing is disabled. |
| `GET /api/v1/activity/users` | All active users in the current tenant. |
| `GET /api/v1/activity/agents` | Registered agents with `status` (`LIVE` / `INACTIVE`), API key count, and recent activity flag. |
| `GET /api/v1/activity/export/{dataset_id}` | Downloads the dataset's knowledge graph as a Markdown export. |
**Reading the pipeline-runs feed**
Every row carries a `kind` discriminator (never null):
* `"pipeline"` — a pipeline run; `pipeline_name` is set.
* `"operation"` — a single-row operation record; `pipeline_name` and `status` are `null`, so status-based readers do not see these rows at all.
Alongside the original `id`, `pipeline_name`, `status`, `dataset_id`, `dataset_name`, `owner_id`, `owner_email`, `created_at` and `pipeline_run_id` keys, each row carries the operation columns below. **All of them are nullable** — rows written before this feature were not backfilled, and each writer sets only the subset it knows.
| Field | Notes |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------- |
| `operation_name` | For pipeline rows this mirrors `pipeline_name`, so it does *not* distinguish the two kinds — use `kind` for that. |
| `origin` | Initiating surface: `sdk`, `api`, `cli`, `mcp`, `background`. |
| `outcome` | `succeeded` / `failed`; `null` on non-terminal rows. |
| `background` | `true` when the call launched background work. |
| `error_class` | Exception class name when `outcome` is `failed`. |
| `tokens_in`, `tokens_out` | Provider-billed token counts. |
| `started_at`, `ended_at` | ISO-8601 timestamps. |
| `user_id` | Triggering user. |
| `session_id` | Session-cache id; joins `session_model_usage`. |
| `parent_operation_id` | Parent row's `pipeline_run_id`. |
Two values are easy to misread:
* `tokens_in` / `tokens_out` of `null` means **not measured**; `0` means **measured zero**. Do not conflate them, and do not use a truthiness check.
* When `background` is `true`, an `outcome` of `"succeeded"` means the work was *accepted and started*, not that it finished. Counting those rows as completions inflates any success-rate or cost figure derived from the feed.
The table behind the feed is append-only, so `tokens_*` cannot be summed row by row either — a pipeline run contributes several rows sharing one `pipeline_run_id`, and `parent_operation_id` chains child totals into their parent. See [what gets stored in a pipeline run record](/core-concepts/building-blocks/pipelines) for the deduplication rules.
**Pagination.** The response is a bare JSON array with no `total` — it has always been a top-level array, so no paging envelope was added. `len(results) == limit` means another page may exist.
**Visibility.** Without `dataset_id`, the feed returns rows authored by the caller or their child agents, plus rows on any dataset shared with them. Note that `recall`, `prune` and multi-dataset `search` records carry no `dataset_id`, so passing `dataset_id` omits them entirely; passing a `dataset_id` the caller cannot read is still a `403`.
<Warning>
A row can be visible because the caller authored it even when the caller has no read permission on its dataset (write-only access, or read revoked after the run). In that case `dataset_name`, `owner_id` and `owner_email` come back as `null` while `dataset_id` is still returned. Clients that render activity entries should tolerate a missing dataset name rather than assuming it is always present.
</Warning>
<Tabs>
<Tab title="Pipeline Runs">
```bash theme={null}
# Second page of 100, newest first
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:8000/api/v1/activity/pipeline-runs?limit=100&offset=100"
# Only one pipeline's history — also excludes operation records,
# which carry no pipeline_name
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:8000/api/v1/activity/pipeline-runs?pipeline_name=cognify_pipeline"
```
</Tab>
<Tab title="Trace Buffer">
```bash theme={null}
curl -H "Authorization: Bearer $TOKEN" \
"http://localhost:8000/api/v1/activity/spans"
```
</Tab>
<Tab title="Dataset Export">
```bash theme={null}
curl -L -H "Authorization: Bearer $TOKEN" \
"http://localhost:8000/api/v1/activity/export/<dataset_id>"
The `/api/v1/activity/spans` response mirrors the same in-memory trace buffer used by the Python OpenTelemetry helpers such as `get_all_traces()`.
```
</Tab>
</Tabs>
</Accordion>
<Accordion title="LLM Utility Endpoints">
When running Cognee as a server, two `/api/v1/llm` endpoints can help you bootstrap a custom extraction prompt from sample text:
* `POST /api/v1/llm/infer-schema` — analyze sample text and return a graph schema
* `POST /api/v1/llm/custom-prompt` — generate a custom extraction prompt from that schema
Typical flow: infer a schema from sample text, generate a prompt, then pass that prompt to `POST /api/v1/cognify`.
<Tabs>
<Tab title="Infer Schema">
```bash theme={null}
curl -X POST "http://localhost:8000/api/v1/llm/infer-schema" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"text": "Alice moved to Paris. Bob founded Acme Corp in New York."}'
```
</Tab>
<Tab title="Generate Prompt">
```bash theme={null}
curl -X POST "http://localhost:8000/api/v1/llm/custom-prompt" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"graphModel": {"title": "PersonCityNetwork", "$defs": {...}, ...}}'
```
</Tab>
<Tab title="Use with Cognify">
```bash theme={null}
curl -X POST "http://localhost:8000/api/v1/cognify" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"datasets": ["my_dataset"], "custom_prompt": "<prompt from step 2>"}'
```
</Tab>
</Tabs>
Optional `parameters` keys for the LLM endpoints include `temperature`, `max_tokens`, `top_p`, and `seed`.
</Accordion>
<Accordion title="Agent Management">
The `/api/v1/agents` router exposes two groups of endpoints: **agent management**
(create / list / get / delete an agent identity) and **agent connections**
(register, unregister, and inspect live sessions). All endpoints require
authentication. Agent identities are persisted as child users of the calling user,
keyed by UUID (`agentId` in API responses), and authenticate to Cognee using the API key returned
on creation — agents do not have passwords.
| Endpoint | Description |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `POST /api/v1/agents/create?name=<name>` | Create an agent identity. Returns `agentId` (UUID), a synthesized `agentEmail`, and the one-time `agentApiKey`. Store the key — it is not retrievable later. Returns `409` if an agent with that name already exists for the caller. |
| `GET /api/v1/agents/list` | List agents created by the authenticated user. Each item includes `agentId`, `agentEmail`, and `apiKeyLabel`. |
| `GET /api/v1/agents/{agent_id}` | Fetch a single agent. `404` if not found, `403` if the caller does not own it. |
| `DELETE /api/v1/agents/{agent_id}` | Delete an agent. `404` / `403` as above. |
| `POST /api/v1/agents/register` | Register an agent connection (session). Body uses `RegisterAgentRequest` (see below). Returns the created `AgentConnection`. `201 Created`. |
| `POST /api/v1/agents/unregister` | Deactivate a connection. Body: `{ "agent_session_name": "<name>" }`. Returns `{ "activeAgents": <count> }`. |
| `GET /api/v1/agents/connections` | List agent connections visible to the caller. Filters: `agent_id`, `range` (`24h`/`7d`/`30d`/`all`, default `30d`), `status` (`active`/`inactive`/`unknown`), `include_sources`, `active_only`, `limit` (1–500, default 50), `offset`. |
| `GET /api/v1/agents/connections/me` | Connection detail for the authenticated user's own agent connection. Optional `agent_session_name` query filter. `404` if no matching connection. |
| `GET /api/v1/agents/connections/{agent_id}` | Connection detail for a specific agent. Optional `agent_session_name` query filter. |
**`RegisterAgentRequest` body fields:** `agent_session_name` (required — combined
with the caller's user ID to form the connection ID), `type`
(`sdk`/`api`/`mcp`/`claude_code`/`opencode`/`workflow`/`unknown`, default `api`),
`memory_mode` (`session`/`cognee`/`hybrid`/`none`/`unknown`), `session_id`,
`dataset_ids`, `dataset_names`, `source`, `origin_function`, `metadata`.
<Tabs>
<Tab title="Create an Agent">
```bash theme={null}
curl -X POST "http://localhost:8000/api/v1/agents/create?name=my-agent" \
-H "Authorization: Bearer $TOKEN"
# {
# "agentId": "f3b0...-...",
# "agentEmail": "my-agent@cognee.agent",
# "agentApiKey": "ck_..."
# }
```
</Tab>
<Tab title="Register a Connection">
```bash theme={null}
curl -X POST "http://localhost:8000/api/v1/agents/register" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{
"agent_session_name": "my-agent-session",
"type": "sdk",
"memory_mode": "cognee",
"dataset_names": ["project_docs"]
}'
```
</Tab>
<Tab title="Unregister">
```bash theme={null}
curl -X POST "http://localhost:8000/api/v1/agents/unregister" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"agent_session_name": "my-agent-session"}'
# {"activeAgents": 0}
```
</Tab>
</Tabs>
<Note>
When the server runs in [agent mode](#agent-mode), `register` and `unregister`
drive the auto-shutdown watchdog. The same `agent_session_name` registered twice
by the same user counts as a single connection — registration is idempotent on
the connection ID.
</Note>
</Accordion>
<Accordion title="Multi-tenant Operations">
**Create tenant:**
```bash theme={null}
curl -X POST "http://localhost:8000/api/v1/permissions/tenants?tenant_name=acme" \
-H "Authorization: Bearer $TOKEN"
```
**Add user to tenant:**
```bash theme={null}
curl -X POST "http://localhost:8000/api/v1/permissions/users/<user_id>/tenants?tenant_id=<tenant_id>" \
-H "Authorization: Bearer $TOKEN"
```
**Create role:**
```bash theme={null}
curl -X POST "http://localhost:8000/api/v1/permissions/roles?role_name=editor" \
-H "Authorization: Bearer $TOKEN"
```
**Assign user to role:**
```bash theme={null}
curl -X POST "http://localhost:8000/api/v1/permissions/users/<user_id>/roles?role_id=<role_id>" \
-H "Authorization: Bearer $TOKEN"
```
**Grant dataset permissions:**
```bash theme={null}
curl -X POST "http://localhost:8000/api/v1/permissions/datasets/<principal_id>?permission_name=read&dataset_ids=<ds_uuid_1>&dataset_ids=<ds_uuid_2>" \
-H "Authorization: Bearer $TOKEN"
```
</Accordion>
</AccordionGroup>
<Columns>
<Card title="API Reference" icon="book" href="/api-reference/introduction">
Explore all API endpoints
</Card>
<Card title="Setup Configuration" icon="settings" href="/setup-configuration/overview">
Configure providers and databases
</Card>
<Card title="MCP Integration" icon="plug" href="/cognee-mcp/mcp-overview">
Set up AI assistant integration
</Card>
</Columns>
# Fact Validity
Source: https://docs.cognee.ai/guides/fact-validity
Close superseded facts with close_node() and filter stale ones with is_valid()
A minimal guide to closing superseded facts. When a fact changes ("Alice works at Acme" becomes "Alice works at Globex"), close the old node instead of deleting it — the graph keeps the history, and `is_valid()` tells you whether any node is still current.
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured
* Read [DataPoints](/core-concepts/building-blocks/datapoints) for the `valid_to` field and the rest of the node schema
* Use the default Ladybug [graph store](/setup-configuration/graph-stores) — currently the only backend where `close_node()` can persist the `valid_to` stamp (see [Backend Support](#backend-support))
## Code in Action
```python theme={null}
import asyncio
import cognee
from cognee.infrastructure.databases.graph import get_graph_engine
from cognee.modules.engine.models import Entity
# close_node and is_valid are not re-exported from cognee.tasks.storage,
# so import them from their module.
from cognee.tasks.storage.close_node import close_node, is_valid
async def main():
# Step 1: start clean and remember the original fact.
await cognee.forget(everything=True)
await cognee.remember("Alice works at Acme.", dataset_name="employment_facts")
# Step 2: entity node ids are deterministic — derive Acme's id from its name.
acme_id = Entity.id_for("Acme")
# Step 3: Alice changes jobs — close the old fact instead of deleting it.
closed = await close_node(acme_id)
print("closed:", closed) # True — the node existed and valid_to was stamped
# Step 4: remember the replacement fact.
await cognee.remember("Alice works at Globex.", dataset_name="employment_facts")
# Step 5: the closed node is still in the graph, just stale.
graph = await get_graph_engine()
node = await graph.get_node(str(acme_id))
print("is_valid:", is_valid(node)) # False — the fact was superseded
if __name__ == "__main__":
asyncio.run(main())
```
The complete runnable script is on GitHub: [`examples/guides/fact_validity.py`](https://github.com/topoteretes/cognee/blob/dev/examples/guides/fact_validity.py).
## What Just Happened
### Step 1: Remember the Original Fact
```python theme={null}
await cognee.forget(everything=True)
await cognee.remember("Alice works at Acme.", dataset_name="employment_facts")
```
`forget(everything=True)` starts from a clean slate, then `remember()` builds the fact into the knowledge graph the same way all your data gets in — no low-level ingestion needed. Every node it creates carries a `valid_to` stamp (`int | None`, ms epoch) that defaults to `None`, meaning the fact is still current.
### Step 2: Derive the Node Id from the Entity Name
```python theme={null}
acme_id = Entity.id_for("Acme")
```
`remember()` turned the sentence into entity nodes, and entity node ids are deterministic: `Entity.id_for(name)` applies the same normalization the ingestion pipeline uses (lowercase, spaces to underscores, apostrophes stripped) and returns the id the entity was stored under. No graph scan needed — this works the same on ten nodes or a hundred thousand.
### Step 3: Close the Superseded Fact
```python theme={null}
closed = await close_node(acme_id)
print("closed:", closed) # True — the node existed and valid_to was stamped
```
`close_node()` stamps `valid_to` on the stored node — "now" by default — marking the fact superseded without deleting it. It returns `True` only if the node existed and was patched, and `False` otherwise (for example when the id is not in the graph). Note the import path: `from cognee.tasks.storage.close_node import close_node, is_valid`.
### Step 4: Remember the Replacement Fact
```python theme={null}
await cognee.remember("Alice works at Globex.", dataset_name="employment_facts")
```
The new fact flows in through `remember()` like any other data and becomes its own nodes. Supersede instead of delete: the old node stays in the graph with `valid_to` set, so your memory keeps the history of what used to be true.
### Step 5: Check Staleness with is\_valid()
```python theme={null}
graph = await get_graph_engine()
node = await graph.get_node(str(acme_id))
print("is_valid:", is_valid(node)) # False — the fact was superseded
```
`is_valid(node, at_ms=None)` returns `True` while `valid_to` is `None` (never closed) or lies strictly in the future relative to `at_ms` (default: now). It accepts either a `DataPoint` instance (reads the attribute) or a plain graph-node dict (reads the key), so it works on records read back from the graph engine, as here.
## Behavior to Know About
* **Node-level granularity.** `valid_to` lives on nodes: closing marks the whole node stale, and the edges attached to it are not stamped.
* **Backdating.** `close_node(node_id, at_ms=...)` stamps a specific ms-epoch timestamp instead of "now", and `is_valid(node, at_ms=some_past_ms)` asks whether the fact was still current at that moment.
* **Not idempotent.** Closing is last-write-wins: re-closing an already-closed node overwrites `valid_to` with the new timestamp (earlier or later). Guard with `is_valid()` first if you need the first close to stick.
* **Two time axes.** `valid_to` records when a fact *stopped being true*. It is not the `time_to` on `Interval` nodes (the time range an `Event` points to via `during`), which records when an event *occurred* — that axis belongs to [Time Awareness](/guides/time-awareness).
* **Retrieval is not filtered yet.** Search and graph completion neither filter nor down-weight closed nodes, so a superseded fact can still surface in results. Applying `is_valid()` to what you retrieve is currently the caller's job; retrieval-side consumption is planned as a follow-up.
## Backend Support
`close_node()` persists `valid_to` through the graph adapter's optional `update_node` method — see [Adding a new graph database](/contributing/adding-providers/adding-new-graph-database) for the adapter contract.
<Warning>
Only the default Ladybug store implements `update_node` today. On every other backend (Neo4j, Kuzu, Postgres, Neptune, Turso), `close_node()` logs a warning and returns `False` — nothing is persisted, and no exception is raised. Check the return value rather than assuming the close landed.
</Warning>
<Columns>
<Card title="Time Awareness" icon="clock" href="/guides/time-awareness">
The other time axis: extract events and timestamps and run time-aware queries.
</Card>
<Card title="DataPoints" icon="circle" href="/core-concepts/building-blocks/datapoints">
The building block that carries `valid_to` and the rest of the node schema.
</Card>
</Columns>
# Feedback System
Source: https://docs.cognee.ai/guides/feedback-system
Step-by-step guide to using feedback with Cognee sessions
Feedback on recall answers is handled via **Sessions**: you record Q\&A in a session, then attach feedback to specific entries using `cognee.session.add_feedback` and `cognee.session.delete_feedback`.
**Before you start:**
* Complete [Quickstart](/getting-started/quickstart) and [Sessions](/guides/sessions)
* Run `recall()` with a `session_id` so that Q\&A entries are stored
* Ensure [caching](/core-concepts/sessions-and-caching) is enabled
## Record feedback on a session Q\&A
1. Run `recall()` with `session_id` so the interaction is stored.
2. Get the session history with `cognee.session.get_session` and identify the `qa_id` of the entry you want to rate.
3. Call `cognee.session.add_feedback` with that `qa_id`, and optionally `feedback_text` and `feedback_score` (1–5).
4. To make feedback influence future retrieval, run [`improve()`](/core-concepts/main-operations/improve) with the relevant `session_ids`. If you are already writing new content with [`remember()`](/core-concepts/main-operations/remember), `self_improvement=True` can trigger this in the background automatically.
5. To clear feedback, use `cognee.session.delete_feedback(session_id=..., qa_id=...)`. Both `add_feedback` and `delete_feedback` return `True` on success and `False` only when the entry was not found or caching is disabled — every other failure raises, see [Return contract and errors](#return-contract-and-errors).
`get_session` returns a list of `SessionQAEntry` objects. Each entry has: `qa_id`, `question`, `answer`, `context`, `time`, `feedback_text`, `feedback_score`. Entries are in chronological order (oldest first); use `entries[-1]` for the most recent. Pass optional `user` for multi-tenant or permission-scoped usage.
```python theme={null}
import cognee
from cognee import SearchType
# Run recall with a session so the Q&A is stored
results = await cognee.recall(
query_text="What are the main themes in my data?",
query_type=SearchType.GRAPH_COMPLETION,
session_id="my_session",
)
print(results)
# Get session history; entries are chronological, so latest is entries[-1]
entries = await cognee.session.get_session(session_id="my_session", last_n=5)
latest = entries[-1]
qa_id = latest.qa_id
# Add feedback for that Q&A (feedback_score must be 1–5)
ok = await cognee.session.add_feedback(
session_id="my_session",
qa_id=qa_id,
feedback_text="Very helpful and accurate",
feedback_score=5,
)
# ok is True if the entry was found and updated
# Push feedback into the graph
await cognee.improve(
dataset="main_dataset",
session_ids=["my_session"],
)
# Recall again with feedback_influence > 0 to surface the weighted results
improved_results = await cognee.recall(
query_text="What are the main themes in my data?",
query_type=SearchType.GRAPH_COMPLETION,
feedback_influence=0.5,
)
print(improved_results)
```
<Note>
The per-call `feedback_influence` defaults to the `DEFAULT_FEEDBACK_INFLUENCE`
environment variable, which is `0.0` (off) by default — so the learned feedback
signal does not change ranking until you opt in. Set `DEFAULT_FEEDBACK_INFLUENCE`
(e.g. `0.1`) to activate the signal globally, or pass `feedback_influence` per call
to `recall()` / `search()` to override it. Setting it back to `0.0` restores the
prior baseline.
</Note>
## Personalize ranking per user
The feedback weights above are a **global** signal: every user's ratings move the same
weights for everyone. To let one user's ratings nudge only that user's results, turn on
[per-user preference personalization](/core-concepts/further-concepts/user-preferences):
1. Enable it in your `.env` — it is off by default:
```dotenv theme={null}
PERSONALIZATION_ENABLED="true"
```
2. Rate answers. The same `add_feedback(..., feedback_score=1..5)` call from above feeds
personalization too; alternatively, with `AUTO_FEEDBACK` on, a rating is inferred when the
user's next message clearly judges the previous answer ("that was exactly right").
3. Run `improve()` with the session — the same call that applies global feedback weights also
folds the rated turns into that user's preference weights.
4. Recall again as the same user: ranking in graph, hybrid, and RAG completion is nudged toward
what they rated up, by at most `PERSONALIZATION_INFLUENCE` (default `0.3`, i.e. 30%).
```python theme={null}
# 2. Rate an answer (or let AUTO_FEEDBACK infer a rating from the conversation)
await cognee.session.add_feedback(
session_id="my_session",
qa_id=qa_id,
feedback_score=5,
)
# 3. Fold ratings into this user's preference weights
await cognee.improve(
dataset="main_dataset",
session_ids=["my_session"],
)
# 4. Recall as the same user — ranking now leans toward what they rated up
results = await cognee.recall(
query_text="What are the main themes in my data?",
query_type=SearchType.GRAPH_COMPLETION,
)
```
Unlike `feedback_influence`, there is no per-call knob: the nudge applies whenever a user is in
context and exactly one dataset resolves, and its strength is set by the
`PERSONALIZATION_INFLUENCE` environment variable. How ratings become weights, how they decay,
and how each retriever applies them is covered in
[User Preferences](/core-concepts/further-concepts/user-preferences).
## Feedback API Reference
### `add_feedback()`
Attach a rating and optional text comment to a stored Q\&A entry.
| Parameter | Type | Description |
| ---------------- | ---------------- | ------------------------------------------------------------------------------------------------------------- |
| `session_id` | `str` | Target session that contains the Q\&A entry. |
| `qa_id` | `str` | Target entry ID. You can get this value from `entry.qa_id` on a `SessionQAEntry` returned by `get_session()`. |
| `feedback_text` | `Optional[str]` | Optional free-form feedback comment. |
| `feedback_score` | `Optional[int]` | Optional integer rating from `1` to `5`. |
| `user` | `Optional[User]` | Optional session owner; resolves automatically when `None`. |
Returns `True` if feedback was stored successfully, `False` if the entry was not found or caching is disabled. See [Return contract and errors](#return-contract-and-errors).
### `delete_feedback()`
Clear both `feedback_text` and `feedback_score` for an existing Q\&A entry without deleting the entry itself.
| Parameter | Type | Description |
| ------------ | ---------------- | ----------------------------------------------------------- |
| `session_id` | `str` | Target session that contains the Q\&A entry. |
| `qa_id` | `str` | Target entry ID whose feedback should be cleared. |
| `user` | `Optional[User]` | Optional session owner; resolves automatically when `None`. |
Returns `True` if feedback was cleared, `False` if the entry was not found or caching is disabled. See [Return contract and errors](#return-contract-and-errors).
When calling `add_feedback()`, provide at least one of `feedback_text` or `feedback_score`. If you pass `feedback_score`, it must be an integer between `1` and `5`.
### Return contract and errors
`False` means exactly two things: **the Q\&A entry was not found**, or **caching is disabled**. It never means "something went wrong".
Everything else raises instead of being reported as `False` — an unreachable cache, a misconfigured backend, or invalid parameters (an empty `session_id` or `qa_id` raises `SessionParameterValidationError`). Earlier releases wrapped these calls in a catch-all that returned `False`, so a rating recorded while the cache hiccuped was silently dropped and looked identical to a bad `qa_id`. The same contract applies to `cognee.session.add_frequency_weights()`, which records the graph nodes and edges an answer used so [`improve()`](/core-concepts/main-operations/improve) can raise their frequency weights. Handle the two outcomes separately:
```python theme={null}
from cognee.infrastructure.databases.exceptions import SessionParameterValidationError
try:
ok = await cognee.session.add_feedback(
session_id="my_session",
qa_id=entry.qa_id,
feedback_score=5,
)
except SessionParameterValidationError:
# Empty session_id or qa_id — a caller bug, fix the ids.
raise
# Any other exception (unreachable cache, misconfigured backend) also propagates:
# the feedback was NOT recorded, and the call is safe to retry once the cause is fixed.
if not ok:
print("No such Q&A entry, or caching is disabled (check the CACHING setting).")
```
The `cognee-cli feedback add` and `cognee-cli feedback delete` commands report the two cases separately and **exit non-zero on both** — a not-found entry names the ids and points at the `CACHING` setting, while an infrastructure failure surfaces the underlying error. Earlier releases exited `0` with a generic message in both cases, so scripts that only checked the exit status silently accepted lost feedback.
### Example
```python theme={null}
cleared = await cognee.session.delete_feedback(
session_id="my_session",
qa_id="some-qa-id",
)
print("Feedback cleared:", cleared)
```
<Columns>
<Card title="Sessions" icon="message-square" href="/guides/sessions">
Enable conversation memory with sessions
</Card>
<Card title="Sessions and Caching" icon="brain" href="/core-concepts/sessions-and-caching">
How sessions and caching work
</Card>
<Card title="Improve" icon="sparkles" href="/core-concepts/main-operations/improve">
Enrich the graph and bridge session memory
</Card>
</Columns>
# Custom GLiNER Extraction
Source: https://docs.cognee.ai/guides/gliner-llm-free-cognify
Drive the GLiNER task list yourself to pass your own entity and relation labels and measure what extraction kept
A minimal guide to driving GLiNER extraction yourself instead of calling `cognify()`. You build the task list with `get_gliner_tasks()`, hand it the exact entity and relation labels you want, and read back a stats object reporting what the model proposed and what survived. A local GLiNER2 model does the extraction and writes the chunk summaries, so no `extract_content_graph` and no `extract_summary` calls are made — reach for this when the default labels are not the ones your graph needs, or when you want extraction loss measured rather than assumed.
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Install the extra: `pip install "cognee[gliner]"`. The first run downloads `fastino/gliner2.5-base-v1` (\~800 MB) into the Hugging Face cache
* Configure an [embedding provider](/setup-configuration/embedding-providers) — extraction and summaries are LLM-free, but `add_data_points` still embeds what it stores, and the default provider is OpenAI. To keep the text on your machine too, pair this with the local fastembed setup in [Recall Without an LLM Key](/guides/no-llm-remember-recall)
* Read [Pipelines](/core-concepts/building-blocks/pipelines) and [Tasks](/core-concepts/building-blocks/tasks) — this guide runs a task list directly instead of calling `cognify()`
* No data is required up front — the script ingests its own passage, but it starts with `prune_data()` and `prune_system()`, which wipe all existing Cognee data; run it against a setup you can afford to reset
* The optional search at the end runs only when `LLM_API_KEY` is set; everything before it does not need one
## Code in Action
```python theme={null}
import asyncio
import os
import cognee
from cognee.context_global_variables import set_database_global_context_variables
from cognee.infrastructure.databases.graph import get_graph_engine
from cognee.modules.users.methods import get_default_user
from cognee.tasks.graph.gliner import GlinerRunStats, get_gliner_tasks
TEXT = """
Tim Cook is the chief executive officer of Apple Inc., headquartered in Cupertino,
California. Before joining Apple in 1998 he worked at Compaq and IBM. Apple was
founded by Steve Jobs, Steve Wozniak and Ronald Wayne in 1976 and today produces
the iPhone, the Mac and the Apple Watch. In 2014 Apple acquired Beats Electronics,
the headphone company co-founded by Dr. Dre and Jimmy Iovine, for three billion
dollars. Apple Park, the company's campus, opened in 2017.
"""
DATASET = "gliner_demo"
async def main():
await cognee.prune.prune_data()
await cognee.prune.prune_system(metadata=True)
await cognee.add(TEXT, dataset_name=DATASET)
user = await get_default_user()
stats = GlinerRunStats()
tasks = await get_gliner_tasks(
entity_types={
"person": "Full name of a human being",
"organization": "Company or institution",
"location": "City, region, campus or country",
"product": "Commercial product",
"date": "Year or calendar date",
"money": "Monetary amount",
},
relation_types=["works_for", "headquartered_in", "founded_by", "produces", "acquired"],
stats=stats,
)
# Every task in this list declares needs_llm=False, so the first-run
# check probes only the embeddings it does use — no flag needed.
await cognee.run_custom_pipeline(
tasks=tasks,
user=user,
dataset=DATASET,
pipeline_name="cognify_pipeline",
)
print("\nschemas by document:")
for document_id, schema in stats.schemas_by_document.items():
print(f" {document_id} ({schema.source})")
print(f" entity types: {sorted(schema.entity_types)}")
print(f" relation types: {sorted(schema.relation_types)}")
print(f"\nchunks processed: {stats.chunks}")
print(f"nodes mapped: {stats.nodes}")
print(
f"relations: {stats.candidate_edges} proposed, "
f"{stats.kept_edges} kept, {stats.dropped_edges} dropped (endpoint did not resolve)"
)
datasets = await cognee.datasets.list_datasets(user)
dataset = next(d for d in datasets if d.name == DATASET)
async with set_database_global_context_variables(dataset.id, dataset.owner_id):
graph = await get_graph_engine()
nodes, edges = await graph.get_graph_data()
by_type: dict[str, int] = {}
for _node_id, props in nodes:
by_type[props.get("type", "?")] = by_type.get(props.get("type", "?"), 0) + 1
print(f"\nstored graph: {len(nodes)} nodes, {len(edges)} edges")
for type_name, count in sorted(by_type.items()):
print(f" {type_name}: {count}")
summaries = [props for _id, props in nodes if props.get("type") == "TextSummary"]
print(f"\nTextSummary nodes: {len(summaries)}")
for props in summaries:
print(" ---")
print(" " + (props.get("text") or "<empty>").replace("\n", "\n "))
structural = {"contains", "is_a", "is_part_of", "made_from", "belongs_to_set"}
entity_edges = sorted(
{(str(edge[0]), edge[2], str(edge[1])) for edge in edges if edge[2] not in structural}
)
print(f"\nentity relations stored: {len(entity_edges)}")
names = {str(node_id): props.get("name") for node_id, props in nodes}
for source, relation, target in entity_edges:
print(f" {names.get(source, source)} --{relation}--> {names.get(target, target)}")
if os.getenv("LLM_API_KEY"):
from cognee import SearchType
results = await cognee.search(
query_text="Who leads Apple and where is it based?",
query_type=SearchType.SUMMARIES,
datasets=[DATASET],
top_k=3,
)
print("\nSUMMARIES search (per dataset):")
for per_dataset in results:
for item in per_dataset.get("search_result", []):
print(" -", (item.get("text") or "<empty>").replace("\n", " | "))
if __name__ == "__main__":
asyncio.run(main())
```
## What Just Happened
### Step 1: Ingest the Text
```python theme={null}
await cognee.add(TEXT, dataset_name=DATASET)
user = await get_default_user()
```
`add()` stores the passage in the `gliner_demo` dataset exactly as it would on the LLM path — only the extraction step changes later. The pipeline is run explicitly rather than through `cognify()`, so it needs a user to run as; `get_default_user()` is the one `cognify()` would have used.
### Step 2: Declare the Extraction Schema
```python theme={null}
stats = GlinerRunStats()
tasks = await get_gliner_tasks(
entity_types={
"person": "Full name of a human being",
"organization": "Company or institution",
"location": "City, region, campus or country",
"product": "Commercial product",
"date": "Year or calendar date",
"money": "Monetary amount",
},
relation_types=["works_for", "headquartered_in", "founded_by", "produces", "acquired"],
stats=stats,
)
```
GLiNER extracts against a closed schema: it only finds the types you name, so the labels here decide what the graph can contain. `entity_types` accepts either a list of names or a `name -> description` mapping, and the descriptions help the model tell similar labels apart. `get_gliner_tasks()` returns the full task list — classify, prepare schema, chunk, extract and summarize, store — and fills `stats` in as the run progresses.
### Step 3: Run the Pipeline Without an LLM
```python theme={null}
# Every task in this list declares needs_llm=False, so the first-run
# check probes only the embeddings it does use — no flag needed.
await cognee.run_custom_pipeline(
tasks=tasks,
user=user,
dataset=DATASET,
pipeline_name="cognify_pipeline",
)
```
`run_custom_pipeline()` loads the dataset's records itself, the same way `cognify()` does, and runs the GLiNER task list over them. Because every task declares `needs_llm=False`, the first-run readiness check probes only the embedding provider the pipeline actually uses — a missing LLM key is not an error on this path.
### Step 4: Read the Run Stats
```python theme={null}
print(f"\nchunks processed: {stats.chunks}")
print(f"nodes mapped: {stats.nodes}")
print(
f"relations: {stats.candidate_edges} proposed, "
f"{stats.kept_edges} kept, {stats.dropped_edges} dropped (endpoint did not resolve)"
)
```
`GlinerRunStats` records what the model proposed against what survived mapping. A relation is dropped when one of its endpoints does not resolve to an extracted entity, so the kept-versus-dropped split measures extraction loss instead of leaving you to guess at it. `stats.schemas_by_document` additionally reports which schema each document was given and where it came from.
### Step 5: Inspect the Stored Graph
```python theme={null}
datasets = await cognee.datasets.list_datasets(user)
dataset = next(d for d in datasets if d.name == DATASET)
async with set_database_global_context_variables(dataset.id, dataset.owner_id):
graph = await get_graph_engine()
nodes, edges = await graph.get_graph_data()
```
Each dataset has its own graph database, so reading it back means entering that dataset's context first. The prints that follow count nodes by type, dump the `TextSummary` nodes — the summaries GLiNER wrote, with no `extract_summary` call behind them — and list the non-structural edges as `source --relation--> target`.
## Advanced Usage
<AccordionGroup>
<Accordion title="Let the Schema Resolve Itself">
Dropping `entity_types` and `relation_types` does not disable the schema — it changes where the schema comes from. `get_gliner_tasks()` then falls through to the configured OWL ontology (`ontology_file_path` overrides the `ONTOLOGY_FILE_PATH` setting), and if there is none, to frozen label banks probed once per document against a sketch of its text.
```python theme={null}
# Schema from the configured ontology, or the label banks if there is none
tasks = await get_gliner_tasks(stats=stats)
```
Read `stats.schemas_by_document` to see which of the three paths each document took — `schema.source` names the origin of the labels it was extracted with. This is the one place that answer is visible, so it is worth printing whenever you are not passing labels yourself.
</Accordion>
<Accordion title="Skip the Task List Entirely">
The task list above is what you want when you need the stats object or your own labels. For LLM-free extraction on its own, `cognify(extractor="gliner")` selects the same list for you — there is nowhere to pass labels on that path, so the schema always resolves itself as described above. See [LLM-free extraction with GLiNER](/python-api/cognify#llm-free-extraction-with-gliner) for the `GRAPH_EXTRACTOR` setting and the arguments the extractor rejects.
</Accordion>
</AccordionGroup>
<Columns>
<Card title="Recall Without an LLM Key" icon="unplug" href="/guides/no-llm-remember-recall">
Run the whole loop — embeddings included — with no API key configured at all.
</Card>
<Card title="Custom Tasks and Pipelines" icon="workflow" href="/guides/custom-tasks-pipelines">
Build and run your own task list, the mechanism this guide borrows.
</Card>
<Card title="Ontologies" icon="map-plus" href="/core-concepts/further-concepts/ontologies">
Where the schema comes from when you do not pass labels yourself.
</Card>
<Card title="Embedding Providers" icon="layers" href="/setup-configuration/embedding-providers">
Configure the one provider this pipeline still needs.
</Card>
</Columns>
# Building the Global Context Index
Source: https://docs.cognee.ai/guides/global-context-index
See how improve(build_global_context_index=True) builds a dataset-wide summary hierarchy, and how it extends that hierarchy incrementally as new data arrives
This guide shows you how to build a global context index over a dataset, inspect the summary hierarchy it produces, and then add more data and rebuild to see the index update incrementally instead of starting over.
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured
* Read [Global Context Index](/core-concepts/further-concepts/global-context-index) for the conceptual overview — this guide focuses on the runnable example, not the full parameter surface
* Be familiar with [`improve()`](/core-concepts/main-operations/improve)
## Code in Action
```python theme={null}
import asyncio
import cognee
from cognee.infrastructure.databases.graph import get_graph_engine
DATASET = "global_context_index_demo"
INITIAL_FACTS = [
"Alice hikes in the Alps every summer.",
"Bob sails along the Adriatic coast every summer.",
"Alice says hiking helps her disconnect from work.",
"Bob says sailing is the best way to unwind after a busy winter.",
"Last year Alice hiked a new trail near Lake Como.",
"Last year Bob sailed to a small island he had never visited.",
]
ADDITIONAL_FACT = (
"This year, Bob decided to join Alice's hiking trip to the Alps instead of sailing."
)
async def print_index_structure(label):
graph_engine = await get_graph_engine()
nodes_data, _edges_data = await graph_engine.get_graph_data()
root = None
buckets = []
text_summary_count = 0
for node_id, properties in nodes_data:
node_type = properties.get("type")
if node_type == "TextSummary":
text_summary_count += 1
elif node_type == "GlobalContextSummary":
if properties.get("is_root"):
root = (node_id, properties.get("text", ""))
else:
buckets.append((node_id, properties.get("text", "")))
print(f"\n{label}")
print(f"Source summaries: {text_summary_count} TextSummary nodes")
print(f"Buckets: {len(buckets)}")
for bucket_id, text in buckets:
print(f" - [{bucket_id}] {text[:60]}...")
if root:
print(f"Root [{root[0]}]: {root[1][:100]}...")
async def main():
await cognee.remember(
INITIAL_FACTS,
dataset_name=DATASET,
self_improvement=False,
)
await cognee.improve(dataset=DATASET, build_global_context_index=True)
await print_index_structure("Index structure after the initial build:")
await cognee.remember(
ADDITIONAL_FACT,
dataset_name=DATASET,
self_improvement=False,
)
await cognee.improve(dataset=DATASET, build_global_context_index=True)
await print_index_structure("Index structure after adding one more fact:")
if __name__ == "__main__":
asyncio.run(main())
```
## What Just Happened
### Step 1: Ingest the Initial Facts
```python theme={null}
INITIAL_FACTS = [
"Alice hikes in the Alps every summer.",
"Bob sails along the Adriatic coast every summer.",
"Alice says hiking helps her disconnect from work.",
"Bob says sailing is the best way to unwind after a busy winter.",
"Last year Alice hiked a new trail near Lake Como.",
"Last year Bob sailed to a small island he had never visited.",
]
await cognee.remember(
INITIAL_FACTS,
dataset_name=DATASET,
self_improvement=False,
)
```
Six short facts describe two people, the trips they take, and how they each feel about their favorite way to spend free time.
### Step 2: Build the Index
```python theme={null}
await cognee.improve(dataset=DATASET, build_global_context_index=True)
```
`improve()` runs its normal enrichment pass first, then builds the global context index on top: it groups the dataset's `TextSummary` nodes into **buckets**, and buckets into higher buckets, until everything fits under one **root** summary — see [Global Context Index](/core-concepts/further-concepts/global-context-index#how-it-works) for exactly what a bucket and a root are.
### Step 3: Inspect the Initial Structure
```python theme={null}
async def print_index_structure(label):
graph_engine = await get_graph_engine()
nodes_data, _edges_data = await graph_engine.get_graph_data()
root = None
buckets = []
text_summary_count = 0
for node_id, properties in nodes_data:
node_type = properties.get("type")
if node_type == "TextSummary":
text_summary_count += 1
elif node_type == "GlobalContextSummary":
if properties.get("is_root"):
root = (node_id, properties.get("text", ""))
else:
buckets.append((node_id, properties.get("text", "")))
print(f"\n{label}")
print(f"Source summaries: {text_summary_count} TextSummary nodes")
print(f"Buckets: {len(buckets)}")
for bucket_id, text in buckets:
print(f" - [{bucket_id}] {text[:60]}...")
if root:
print(f"Root [{root[0]}]: {root[1][:100]}...")
await print_index_structure("Index structure after the initial build:")
```
`print_index_structure()` reads the graph directly through `get_graph_engine().get_graph_data()` — see the **"Inspect extracted graph schema"** accordion (its "Python SDK" subsection) on the [Cognify](/core-concepts/main-operations/legacy-operations/cognify) page for exactly what it returns — a `(node_id, properties)` pair per node. The loop classifies each node by its `type`: a `TextSummary` just increments a counter, while a `GlobalContextSummary` is split into the single **root** (`is_root=True`) or a **bucket** (everything else), keeping each bucket's id alongside its text so you can compare it against the next build.
### Step 4: Add a Fact and Rebuild
```python theme={null}
await cognee.remember(
ADDITIONAL_FACT,
dataset_name=DATASET,
self_improvement=False,
)
await cognee.improve(dataset=DATASET, build_global_context_index=True)
```
The new fact mentions Bob, hiking, and the Alps — entities already grouped together in Alice's bucket from the first build. Running `improve(build_global_context_index=True)` again does not start over — it only places this one new `TextSummary` node.
### Step 5: Inspect the Updated Structure
```python theme={null}
await print_index_structure("Index structure after adding one more fact:")
```
Compare the two printouts: `Source summaries` goes up by one, but the bucket that already covered Alice's hiking trips keeps the **exact same id** it had after Step 3 — proof that fact was added to that bucket rather than triggering a full rebuild. The root also keeps the same id (it's derived only from the dataset, never from its children), though its text is regenerated to reflect the new fact.
## Under the Hood
<Accordion title="How the Index Is Actually Built and Updated">
* **How groupings are actually formed**: at level 0, `TextSummary` nodes land in the same bucket based on entity overlap — the more entities two summaries share, the more likely they're grouped together — weighted so entities that show up in almost every summary don't dominate the grouping. Every level above level 0 (bucket into higher bucket, up to the root) groups by **vector distance** between embeddings instead.
* **Bucket capacity**: `improve()` hardcodes `max_bucket_size=4`, not exposed as a configurable option.
* **Why ids stay stable**: a bucket's id is determined by the dataset, level, and its child ids *at the moment the bucket is created*. Adding a new child later mutates that bucket's contents in place — the id is never recomputed. The root's id depends only on the dataset id, never on its children, so it's stable for the life of the dataset even though its text gets regenerated whenever something changes underneath it.
* **Higher levels**: buckets group up to `max_bucket_size` buckets each. If that still leaves more than `max_bucket_size` buckets, another level is built on top, repeating until the topmost level fits under a single root.
</Accordion>
An advanced version of this guide is available as a smoke-check script on our [GitHub](https://github.com/topoteretes/cognee/blob/dev/examples/advanced_guides/global_context_index_smoke_demo.py). It ingests a multi-day scheduling conversation in which meetings are booked, moved, and cancelled, builds the index with `improve(build_global_context_index=True)`, then runs the same query with `include_global_context_index` off and on to verify the index's context prelude actually reaches `recall()` — and finishes by asking questions whose answers depend on the latest state of the whole thread.
<Columns>
<Card title="Global Context Index" icon="globe" href="/core-concepts/further-concepts/global-context-index">
The full concept, configuration options, and when to use it
</Card>
<Card title="Improve" icon="sparkles" href="/core-concepts/main-operations/improve">
Understand improve()'s other enrichment passes
</Card>
<Card title="Reading the Global Context Index" icon="search" href="/guides/global-context-index-recall">
See how include\_global\_context\_index changes retrieval, once the index above exists
</Card>
</Columns>
# Reading the Global Context Index
Source: https://docs.cognee.ai/guides/global-context-index-recall
See exactly what include_global_context_index adds to GRAPH_COMPLETION retrieval
This guide shows you what changes in `recall()`'s output when `include_global_context_index=True` is set — both in the retrieved context and in the final generated answer.
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured
* Complete [Building the Global Context Index](/guides/global-context-index) first — this guide assumes an index already exists and does not re-explain how it's built
* Be familiar with [`recall()`](/core-concepts/main-operations/recall)
## Code in Action
```python theme={null}
import asyncio
import cognee
from cognee import SearchType
DATASET = "global_context_index_recall_demo"
FACTS = [
"Alice hiked a new trail near Lake Como in winter.",
"Alice reached the summit of the peak she'd trained for all year in summer.",
"Bob sailed to a small island he had never visited in winter.",
"Bob completed his first solo overnight crossing in summer.",
"Alice started a sourdough starter in winter.",
"Alice baked her first focaccia for a dinner party in summer.",
"Bob took his first watercolor class in winter.",
"Bob sold a painting at a local market in summer.",
"Alice began German lessons in winter.",
"Alice had her first full conversation in German in summer.",
]
QUERY = "What changed across all of Alice and Bob's hobbies between winter and summer?"
async def main():
await cognee.remember(
FACTS,
dataset_name=DATASET,
self_improvement=False,
)
await cognee.improve(dataset=DATASET, build_global_context_index=True)
context_without = await cognee.recall(
query_text=QUERY,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET],
top_k=4,
only_context=True,
retriever_specific_config={"include_global_context_index": False},
)
context_with = await cognee.recall(
query_text=QUERY,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET],
top_k=4,
only_context=True,
retriever_specific_config={
"include_global_context_index": True,
"global_context_index_top_k": 3,
},
)
print("Context WITHOUT global context index:\n")
print(context_without[0].text)
print("\nContext WITH global context index:\n")
print(context_with[0].text)
answer_without = await cognee.recall(
query_text=QUERY,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET],
top_k=4,
retriever_specific_config={"include_global_context_index": False},
)
answer_with = await cognee.recall(
query_text=QUERY,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET],
top_k=4,
retriever_specific_config={
"include_global_context_index": True,
"global_context_index_top_k": 3,
},
)
print("\nAnswer WITHOUT global context index:\n")
print(answer_without[0].text)
print("\nAnswer WITH global context index:\n")
print(answer_with[0].text)
if __name__ == "__main__":
asyncio.run(main())
```
## What Just Happened
### Step 1: Ingest a Multi-Hobby Dataset
```python theme={null}
await cognee.remember(
FACTS,
dataset_name=DATASET,
self_improvement=False,
)
```
Ten facts, two per hobby across five hobbies (hiking, sailing, baking, watercolor painting, German lessons) shared between Alice and Bob, each with a winter and a summer update. This is enough that a single retrieval pass can't hold every hobby's triplets at once — exactly the condition where the global context index has real work to do.
### Step 2: Build the Index
```python theme={null}
await cognee.improve(dataset=DATASET, build_global_context_index=True)
```
Same mechanism as [Building the Global Context Index](/guides/global-context-index) — see that guide for how the bucket/root hierarchy actually forms. This guide only needs the finished index.
### Step 3: Compare the Retrieved Context
```python theme={null}
context_without = await cognee.recall(
query_text=QUERY,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET],
top_k=4,
only_context=True,
retriever_specific_config={"include_global_context_index": False},
)
context_with = await cognee.recall(
query_text=QUERY,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET],
top_k=4,
only_context=True,
retriever_specific_config={
"include_global_context_index": True,
"global_context_index_top_k": 3,
},
)
```
`top_k=4` deliberately restricts local retrieval to fewer triplets than the dataset has hobbies — low enough to make the effect in Step 4 obvious. Both calls retrieve the **exact same** `Nodes:` / `Connections:` block: `include_global_context_index` never changes which local triplets get selected, only what gets prepended before them. The only difference here is that `World summary:` and `Relevant areas:` appear only in the second call.
As [Global Context Index](/core-concepts/further-concepts/global-context-index) explains, this feature is meant for datasets large enough that local retrieval genuinely can't hold everything relevant — long documents, project memory spanning many updates, policy corpora with many sections. Ten facts across five hobbies is nowhere near that scale on its own; the only reason it behaves like a large dataset here is that we've deliberately shrunk `top_k` to `4`, well below what a real deployment would use. That's an artificial constraint chosen to make the effect visible in a guide-sized example, not a recommendation to run production workloads with `top_k` this low.
The root itself isn't a way around this at any scale, either: `World summary` is capped at a fixed token budget (the prompt that generates it caps output at 500 tokens), so on a genuinely large dataset it can't just list every fact losslessly — it has to compress. At real scale, expect the root to reliably preserve *which* hobbies and topics exist (cheap to list) while individual details, like a specific summer milestone, are more likely to survive in `Relevant areas` instead — the vector-matched, topic-specific bucket summaries have their own budget per bucket, so they carry more per-topic detail than one root summary stretched across everything.
### Step 4: Compare the Generated Answers
```python theme={null}
answer_without = await cognee.recall(
query_text=QUERY,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET],
top_k=4,
retriever_specific_config={"include_global_context_index": False},
)
answer_with = await cognee.recall(
query_text=QUERY,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET],
top_k=4,
retriever_specific_config={
"include_global_context_index": True,
"global_context_index_top_k": 3,
},
)
```
This is where the difference stops being cosmetic. With `top_k=4`, the local retrieval in Step 3 can only surface a fraction of the dataset's triplets — and since that local set is identical either way, the generated answer *without* the index inherits that gap directly:
| | Without the index | With the index |
| ------------------------ | ------------------------------- | -------------- |
| Hobbies covered (of 5) | 2 | 5 |
| Hobbies missing entirely | German lessons, baking, sailing | none |
The answer without the index only mentions hiking and watercolor painting — it never mentions Alice's German lessons, never mentions her sourdough baking, and never mentions Bob's sailing, three of the five hobbies vanish. The answer with the index names all five and still tracks each one's winter-to-summer progression, because `World summary` is built once over every `TextSummary` in the dataset — it doesn't compete for a spot in `top_k` the way local triplets do.
<Note>
This is real output from one run against an LLM, so if you run it yourself, expect the wording *and* the exact count of missing hobbies to differ — LLM generation isn't deterministic, and `top_k=4` is a hard cutoff on vector similarity scores, so a hobby hovering right at that boundary can land on either side of it from one call to the next. The *pattern* is what's stable and worth taking away: some hobbies reliably vanish without the index, and none do with it — not the specific count above. Raising `top_k` high enough eventually closes this particular gap on its own; the index is what keeps working when you can't or don't want to raise it that far.
</Note>
## Under the Hood
<Accordion title="How recall() Actually Reads the Index">
* **The root is loaded, never searched**: a dataset has at most one root `GlobalContextSummary`, and `recall()` loads it straight from the graph — filtering the dataset's `GlobalContextSummary` nodes for the one flagged `is_root` — instead of running a vector search for it. This becomes the `World summary:` line.
* **"Relevant areas" comes from one flat vector search**: every non-root bucket lives in the same vector collection, embedded when it was created. Finding the `global_context_index_top_k` "Relevant areas" is a single vector search across that whole collection, comparing every bucket directly against the query — not a walk down from the root through parent-child links.
* **Why `top_k` doesn't affect the index**: `top_k` only bounds the local triplet search (Step 3). The root load and the bucket vector search are separate lookups that always run in full, regardless of how restrictive `top_k` is — that decoupling is exactly why the index keeps covering every hobby in Step 4 even as local retrieval covers fewer and fewer.
* **`HYBRID_COMPLETION` honors this too**: the same `include_global_context_index` flag works with `SearchType.HYBRID_COMPLETION`, which places the same prelude under a `## Global context` heading at the top of its context block instead of a `World summary:` line.
</Accordion>
<Columns>
<Card title="Global Context Index" icon="globe" href="/core-concepts/further-concepts/global-context-index">
The full concept, configuration options, and when to use it
</Card>
<Card title="Building the Global Context Index" icon="globe" href="/guides/global-context-index">
Build the index and see it update incrementally
</Card>
<Card title="Recall" icon="search" href="/core-concepts/main-operations/recall">
Understand recall()'s full parameter surface and auto-routing behavior
</Card>
</Columns>
# Inspecting Graph Completion Context
Source: https://docs.cognee.ai/guides/graph-completion
See how GRAPH_COMPLETION turns retrieved graph triplets into an answer's context, and what a memory fragment is
This guide shows you how `SearchType.GRAPH_COMPLETION` — Cognee's graph-based search type — turns triplets (pairs of connected nodes plus the relationship between them) into the context behind an answer, before that context ever reaches the language model.
## Before You Start
* Complete the [Understand Recall with RAG Completion guide](/guides/rag-recall) to see the shared `recall()` parameters this guide builds on (`query_text`, `datasets`, `top_k`, `only_context`) — this guide does not repeat that explanation.
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured.
## Code in Action
```python theme={null}
import asyncio
import cognee
from cognee import SearchType
DATASET_NAME = "graph_completion_demo"
QUERY = "Who works on Cognee, and how do Alice and Bob collaborate?"
DOCUMENTS = [
"Alice is a Cognee engineer.",
"Bob is Cognee's product manager.",
"Cognee turns documents into AI memory.",
"Alice builds Cognee hybrid retrieval.",
"Alice and Bob meet weekly on Cognee demos.",
"Bob sends Cognee feedback to Alice.",
]
async def main():
await cognee.remember(
DOCUMENTS,
dataset_name=DATASET_NAME,
self_improvement=False,
)
context = await cognee.recall(
query_text=QUERY,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET_NAME],
top_k=20,
only_context=True,
)
print("Retrieved context:\n")
print(context[0].text)
narrow_context = await cognee.recall(
query_text=QUERY,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET_NAME],
top_k=20,
wide_search_top_k=1,
only_context=True,
)
print("\nNarrow candidate pool (wide_search_top_k=1):\n")
print(narrow_context[0].text)
wide_context = await cognee.recall(
query_text=QUERY,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET_NAME],
top_k=20,
wide_search_top_k=100,
only_context=True,
)
print("\nWide candidate pool (wide_search_top_k=100):\n")
print(wide_context[0].text)
answer = await cognee.recall(
query_text=QUERY,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET_NAME],
top_k=20,
)
print("\nGenerated answer:\n")
print(answer[0].text)
if __name__ == "__main__":
asyncio.run(main())
```
## What Just Happened
### Step 1: Ingest the Example Dataset
```python theme={null}
await cognee.remember(
DOCUMENTS,
dataset_name=DATASET_NAME,
self_improvement=False,
)
```
Six short sentences about Alice, Bob, and Cognee give the graph something to connect: two people, their roles, and how they interact with each other and with Cognee.
### Step 2: Retrieve Only the Graph Context
```python theme={null}
context = await cognee.recall(
query_text=QUERY,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET_NAME],
top_k=20,
only_context=True,
)
print("Retrieved context:\n")
print(context[0].text)
```
With `only_context=True`, `recall()` skips the completion step and returns the formatted graph context it would otherwise have sent to the language model.
`recall()` returns a list with one entry per query. This example passes a single query, so the list always has exactly one item: `context[0]`. By printing `context[0].text` instead of just `context[0]`, we make the output readable — otherwise it would print the full result object.
The printed text has two parts: a `Nodes:` section listing every node touched by a retrieved triplet, and a `Connections:` section rendering each triplet as `source --[relationship]--> target`. What ends up here is determined by `top_k=20`, which keeps the 20 most relevant triplets (two connected nodes plus the relationship between them) found by the search.
Lines like `alice --[works_at]--> cognee` or `alice --[attends_weekly_on]--> cognee demos` show relationships Graph Completion decided were relevant to the query.
### Step 3: Compare Narrow vs. Wide Candidate Pools
```python theme={null}
narrow_context = await cognee.recall(
query_text=QUERY,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET_NAME],
top_k=20,
wide_search_top_k=1,
only_context=True,
)
print("\nNarrow candidate pool (wide_search_top_k=1):\n")
print(narrow_context[0].text)
wide_context = await cognee.recall(
query_text=QUERY,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET_NAME],
top_k=20,
wide_search_top_k=100,
only_context=True,
)
print("\nWide candidate pool (wide_search_top_k=100):\n")
print(wide_context[0].text)
```
Unlike Step 2, both calls here add one more parameter: `wide_search_top_k`. We call `recall()` twice with the same `top_k=20`, but with two very different `wide_search_top_k` limits — `1` and `100` — to see its effect directly.
`wide_search_top_k` caps how many candidates the vector-search step contributes before the memory fragment is built (see [Under the Hood](#under-the-hood) below). Here, `wide_search_top_k=1` produces a fragment of only 19 nodes and 18 edges — short of the requested 20 triplets. `wide_search_top_k=100` produces a larger fragment of 33 nodes and 48 edges, enough to fill the full `top_k=20`. This is the performance/quality tradeoff: a narrower pool is cheaper but can miss relevant relationships; a wider pool costs more but is less likely to.
### Step 4: Generate the Final Answer
```python theme={null}
answer = await cognee.recall(
query_text=QUERY,
query_type=SearchType.GRAPH_COMPLETION,
datasets=[DATASET_NAME],
top_k=20,
)
print("\nGenerated answer:\n")
print(answer[0].text)
```
Dropping `only_context=True` lets `recall()` complete the answer using that same context — nothing changes between the two calls except whether the completion step runs. The context inspected in Step 2 is exactly what was sent to the LLM to produce this answer.
## Under the Hood
<AccordionGroup>
<Accordion title="How Graph Completion Selects Triplets">
Graph Completion does not just return the first nodes and edges it finds. Internally, it runs the same pipeline for every query:
1. **Vector search** — search the indexed node and edge collections (entity names, text summaries, document chunks, and relationship labels) for the query, scoring the nodes and relationship labels that matched. `wide_search_top_k` (default `100`) caps how many candidates come back from *each* collection at this stage — it is a separate knob from `top_k`, which only comes into play at the very end, in step 5.
2. **Project a memory fragment** — a memory fragment is a temporary, in-memory copy of the graph, built restricted to the node ids that matched in step 1 (see the next accordion for the full picture). An edge is only carried over if both of the nodes it connects were selected — edges are never chosen on their own.
3. **Map distances onto the fragment** — attach each match's vector distance from step 1 to the corresponding node or edge inside that fragment.
4. **Score each triplet** — a triplet's two endpoint nodes and its edge each carry their own importance weight (and, if configured, a feedback weight from past corrections); these are combined into one score for the triplet as a whole.
5. **Keep the top `top_k`** — the best-scoring triplets are what you saw resolved into the `Nodes:` / `Connections:` text above.
* **`wide_search_top_k` (performance vs. quality)** — a higher value scores more candidate nodes and edges for the memory fragment, making it more likely to catch the relevant relationship, but takes longer; a lower value is faster but more likely to miss one. `None` scores every node and edge in the graph — the best quality, and the slowest.
</Accordion>
<Accordion title="What a Memory Fragment Is">
A memory fragment is a temporary, in-memory graph object that Cognee builds fresh for one query and discards once triplets are selected — it is never read directly from, or written back to, the full persisted graph.
This guide's example passes a single `query_text` with no `node_name` or `neighborhood_depth` — so Cognee always builds the fragment the same way here: **ID-filtered**, using only the node ids the vector search already scored, plus the edges between them.
Three other projections exist, chosen by which parameters you pass to `recall()`:
1. **Full-graph** — every stored node and edge, used when there's no useful pre-filter (`wide_search_top_k=None`).
2. **Node-set** — restricted to nodes matching a given `node_name`.
3. **Neighborhood** — starts from a few nodes (via `neighborhood_depth`) and includes their direct connections, then those connections' own connections, up to a set limit. The nodes reached by traversal are not scored by the vector search that seeds the fragment, so they have no distance of their own; Cognee asks the vector store for those specific ids' distances to the query and maps them onto the fragment, so an expanded node competes in triplet scoring on its real similarity. This needs an adapter that implements the optional id-bounded scoring method (LanceDB — including its subprocess mode — PGVector, and Turso do); on an adapter that doesn't, expanded nodes keep the default distance penalty.
In every mode, an edge is included only if both of its endpoint nodes were — the node selection always comes first, and the edges follow from it.
</Accordion>
</AccordionGroup>
<Columns>
<Card title="Recall" icon="search" href="/core-concepts/main-operations/recall">
Understand recall()'s full parameter surface and auto-routing behavior
</Card>
<Card title="SearchType" icon="list" href="/python-api/search-type">
See every search type, including GRAPH\_COMPLETION's variants
</Card>
</Columns>
# Graph Engine and Adapters
Source: https://docs.cognee.ai/guides/graph-engine-adapters
Learn how GraphDBInterface, get_graph_engine(), and concrete adapters fit together through a small, offline Ladybug example
A minimal guide to how Cognee talks to a graph database without your code ever depending on which one is actually configured. You'll create two nodes and an edge, read them back, and clean up — using only interface methods, on an isolated local graph.
## Before You Start
* This is a low-level infrastructure exercise, not a replacement for `cognee.remember()` or Cognee pipelines — direct graph writes like this skip vector indexing, relational records, dataset provenance, and normal access-control behavior.
* This guide defines a custom `Person(DataPoint)` model — see [DataPoints](/core-concepts/building-blocks/datapoints) for the concept, or [Custom Data Models](/guides/custom-data-models) and [Custom Graph Model](/guides/custom-graph-model) to go deeper.
## Code in Action
```python theme={null}
import os
import asyncio
import cognee
from cognee.low_level import DataPoint
from cognee.infrastructure.databases.graph import get_graph_engine
class Person(DataPoint):
name: str
metadata: dict = {"index_fields": ["name"], "identity_fields": ["name"]}
async def main():
graph_db_path = os.path.join(os.path.dirname(__file__), ".artifacts", "graph_engine_guide_db")
cognee.config.set_graph_db_config(
{
"graph_database_provider": "ladybug",
"graph_file_path": graph_db_path,
"graph_database_subprocess_enabled": False,
}
)
graph_engine = await get_graph_engine()
alice = Person(name="Alice")
bob = Person(name="Bob")
try:
await graph_engine.add_nodes([alice, bob])
await graph_engine.add_edge(
str(alice.id),
str(bob.id),
"knows",
edge_properties={"since": "2020"},
)
stored_alice = await graph_engine.get_node(str(alice.id))
print(f"Alice node: {stored_alice}")
bob_neighbors = await graph_engine.get_neighbors(str(bob.id))
print(f"Bob's neighbors: {bob_neighbors}")
alice_knows_bob = await graph_engine.has_edge(str(alice.id), str(bob.id), "knows")
print(f"Alice knows Bob: {alice_knows_bob}")
finally:
await graph_engine.delete_nodes([str(alice.id), str(bob.id)])
if __name__ == "__main__":
asyncio.run(main())
```
## What Just Happened
### Step 1: Configure an Isolated Graph and Get the Engine
```python theme={null}
graph_db_path = os.path.join(os.path.dirname(__file__), ".artifacts", "graph_engine_guide_db")
cognee.config.set_graph_db_config(
{
"graph_database_provider": "ladybug",
"graph_file_path": graph_db_path,
"graph_database_subprocess_enabled": False,
}
)
graph_engine = await get_graph_engine()
```
`graph_file_path` points Ladybug at a throwaway directory so this exercise never touches your default Cognee graph. `graph_database_subprocess_enabled=False` keeps Ladybug running inside your Python process instead of as a separate background process — simpler for a short script like this. `get_graph_engine()` is `async` and returns whatever adapter the configuration selected — here, a Ladybug adapter — typed as `GraphDBInterface`. Everything from this point on is written against that interface, not against Ladybug specifically.
### Step 2: Define a Minimal Node Model
```python theme={null}
class Person(DataPoint):
name: str
metadata: dict = {"index_fields": ["name"], "identity_fields": ["name"]}
```
`Person` inherits from `DataPoint`, Cognee's base model for graph nodes. Declaring `"identity_fields": ["name"]` gives each `Person` a deterministic id based on its name, so `Person(name="Alice")` always resolves to the same node id instead of a random one.
### Step 3: Create Nodes and an Edge
```python theme={null}
await graph_engine.add_nodes([alice, bob])
await graph_engine.add_edge(
str(alice.id),
str(bob.id),
"knows",
edge_properties={"since": "2020"},
)
```
**`add_nodes(nodes)`** takes a list of `DataPoint` instances and writes them as one bulk operation. The Ladybug, Postgres, and Turso adapters split a large list into several idempotent bulk statements rather than sending one statement per call, so a single write cannot outgrow the backend's per-statement limits; the chunk size is an internal, per-adapter detail and is not configurable. **`add_edge(source_id, target_id, relationship_name, edge_properties)`** creates one directed edge — here, `Alice -[knows]-> Bob` with a small property dictionary. Both take plain string ids, so `DataPoint.id` (a `UUID`) is converted with `str(...)` before being passed in.
### Step 4: Read Data Back
```python theme={null}
stored_alice = await graph_engine.get_node(str(alice.id))
bob_neighbors = await graph_engine.get_neighbors(str(bob.id))
alice_knows_bob = await graph_engine.has_edge(str(alice.id), str(bob.id), "knows")
```
**`get_node(node_id)`** returns a single node's properties as a dictionary, or `None` if it does not exist. **`get_neighbors(node_id)`** returns the properties of every node connected to the given node — here, Bob's only neighbor is Alice. **`has_edge(source_id, target_id, relationship_name)`** checks whether a specific directed, labeled edge exists, returning a plain `bool`.
### Step 5: Clean Up
```python theme={null}
finally:
await graph_engine.delete_nodes([str(alice.id), str(bob.id)])
```
`delete_nodes(node_ids)` removes the listed nodes and any edges attached to them. Running it in a `finally` block ensures the example nodes are removed even if an earlier step raises, leaving the isolated graph empty again.
## One Interface, Many Databases
Cognee can store its graph in several different backends — Ladybug (the default local, file-based engine), Postgres, Neo4j, and others (see [Graph Stores](/setup-configuration/graph-stores) for how to configure each one). Application code that queries or writes to the graph should not need a different code path for each one. Cognee solves this with three pieces:
```text theme={null}
configuration -> get_graph_engine() -> GraphDBInterface -> concrete adapter
```
* **Configuration** picks *which* provider is active (e.g. `"ladybug"` or `"postgres_demo"`).
* **`get_graph_engine()`** is a factory function. It reads the configuration and returns an object implementing the interface — you never construct an adapter yourself.
* **`GraphDBInterface`** is an abstract base class that declares the methods every adapter must provide (`add_node`, `get_node`, `has_edge`, and so on).
* **Concrete adapters** (`LadybugAdapter`, `PostgresDemoAdapter`, ...) implement that interface against a specific database.
Because every adapter satisfies the same interface, code written against `get_graph_engine()` works unchanged no matter which provider is configured.
<Warning>
Do not import or instantiate `LadybugAdapter`, `PostgresDemoAdapter`, or any other adapter directly. Always go through `get_graph_engine()` — adapter modules and class names are internal and do move (the Postgres adapter was renamed to `PostgresDemoAdapter` and its old module removed), while code written against `get_graph_engine()` keeps working unchanged.
</Warning>
## Comparing Adapters
`LadybugAdapter` and `PostgresDemoAdapter` both implement `add_nodes`, `add_edge`, `get_node`, `get_neighbors`, and `has_edge` from `GraphDBInterface` — but the two implementations look nothing alike internally. Ladybug builds parameterized Cypher-style statements against an embedded Kuzu database; Postgres issues SQL against relational tables that model nodes and edges. Neither difference is visible to code written against the interface, which is the point: swapping `graph_database_provider` from `"ladybug"` to `"postgres_demo"` (with matching connection settings) does not require changing any of the code above.
<Warning>
Every adapter also has a raw `query()` method for running commands written directly in that database's own language — Cypher for Ladybug/Neo4j, SQL for Postgres. Using it ties your code to one specific database, which is exactly what this guide is trying to avoid. That's why it's left out here — everything above uses only the shared `GraphDBInterface` methods, which work the same way no matter which database is configured.
</Warning>
<Columns>
<Card title="DataPoints" icon="circle" href="/core-concepts/building-blocks/datapoints">
Learn more about DataPoint, the base model Person builds on
</Card>
<Card title="BaseRetriever Guide" icon="puzzle" href="/guides/base-retriever">
See the same abstract-contract pattern applied to retrievers
</Card>
</Columns>
# Graph Model from JSON
Source: https://docs.cognee.ai/guides/graph-model-from-json
Declare a custom graph model as plain JSON and compile it into a DataPoint model
A minimal guide to defining a custom graph model without writing model classes. Use it when the schema arrives as data — a config file, an API payload, or a document produced by the cognee UI graph-model editor — and you want the same extraction control that a hand-written `DataPoint` model gives you.
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — the cognify step extracts the graph with an LLM
* Read [Custom Graph Model](/guides/custom-graph-model) for the Python-class equivalent of what the JSON compiles to
* Read [DataPoints](/core-concepts/building-blocks/datapoints) for how graph nodes are identified and indexed
## Code in Action
```python theme={null}
import asyncio
import cognee
from cognee.low_level import graph_model_from_spec
PEOPLE_SPEC = {
"root": "Person",
"entities": [
{
"name": "Person",
"description": "A person mentioned in the text.",
"fields": [
{
"kind": "primitive",
"name": "role",
"primitive_type": "string",
"description": "What the person does.",
},
{
"kind": "relation",
"name": "works_at",
"relation": {"target_entity_name": "Organization", "cardinality": "one"},
},
{
"kind": "relation",
"name": "collaborates_with",
"relation": {"target_entity_name": "Person", "cardinality": "many"},
},
],
},
{
"name": "Organization",
"description": "A company, lab, or institution.",
"fields": [
{"kind": "primitive", "name": "field_of_work", "primitive_type": "string"},
],
},
],
}
TEXT = """
Ada Lovelace worked at the Analytical Engine project alongside Charles Babbage.
Grace Hopper worked at Remington Rand, where she collaborated with the UNIVAC team.
"""
async def main():
await cognee.forget(everything=True)
# JSON in, Pydantic model out.
PeopleGraph = graph_model_from_spec(PEOPLE_SPEC)
await cognee.add(TEXT, dataset_name="people_from_json")
await cognee.cognify(datasets=["people_from_json"], graph_model=PeopleGraph)
results = await cognee.search(
query_text="Who worked where, and with whom?",
datasets=["people_from_json"],
)
for result in results:
print(result)
if __name__ == "__main__":
asyncio.run(main())
```
## What Just Happened
### Step 1: Declare the Schema as JSON
```python theme={null}
PEOPLE_SPEC = {
"root": "Person",
"entities": [
{
"name": "Person",
"description": "A person mentioned in the text.",
"fields": [
{
"kind": "primitive",
"name": "role",
"primitive_type": "string",
"description": "What the person does.",
},
{
"kind": "relation",
"name": "works_at",
"relation": {"target_entity_name": "Organization", "cardinality": "one"},
},
{
"kind": "relation",
"name": "collaborates_with",
"relation": {"target_entity_name": "Person", "cardinality": "many"},
},
],
},
{
"name": "Organization",
"description": "A company, lab, or institution.",
"fields": [
{"kind": "primitive", "name": "field_of_work", "primitive_type": "string"},
],
},
],
}
```
A spec lists the `entities` in the graph and names one of them as the `root` — the entity the generated top-level model is built from (the first entity, when `root` is omitted). Each field is either a `primitive` value, an `enum`, or a `relation` pointing at another declared entity, and a relation's `cardinality` decides whether it holds one target or a list. `collaborates_with` targets `Person` itself, so an entity may relate to its own type. Every `description` is compiled into the schema and onto the generated model's fields, but cognee rebuilds that model into a plain extraction schema before the LLM call and drops them along the way — a `description` documents the spec, it does not steer the LLM. Use [`custom_prompt`](/guides/custom-prompts) for that.
### Step 2: Compile the Spec into a Model
```python theme={null}
# JSON in, Pydantic model out.
PeopleGraph = graph_model_from_spec(PEOPLE_SPEC)
```
`graph_model_from_spec` validates the spec, compiles it to the JSON Schema cognee's model generator accepts, and returns a `DataPoint`-derived Pydantic class. An invalid spec — an unknown relation target, a duplicate entity name, a field colliding with a `DataPoint` infrastructure field — raises `ValidationError` here, before any LLM call is made. `graph_spec_to_json_schema`, also exported from `cognee.low_level`, stops one step earlier and returns that JSON Schema, which is useful for inspecting what the spec compiled into.
### Step 3: Extract with the Generated Model
```python theme={null}
await cognee.add(TEXT, dataset_name="people_from_json")
await cognee.cognify(datasets=["people_from_json"], graph_model=PeopleGraph)
```
The compiled class is an ordinary `graph_model` argument: pass it to `cognify()` (or `remember()`) and it becomes the structured-output schema the LLM must fill, so the graph can only contain the entities and relations the JSON declared.
### Step 4: Search the Resulting Graph
```python theme={null}
results = await cognee.search(
query_text="Who worked where, and with whom?",
datasets=["people_from_json"],
)
for result in results:
print(result)
```
Nothing downstream changes because the model came from JSON — the graph is queried exactly like one built from hand-written classes.
## Advanced Usage
<Accordion title="Node identity and indexing">
Each entity compiles with a `metadata` default of `index_fields` and `identity_fields`, both defaulting to `["name"]`. `identity_fields` is what makes nodes extracted from different chunks and different runs merge into one graph node when their identity values match, so `Ada Lovelace` mentioned twice stays a single node. Set `"identity_fields": []` on an entity to opt out and give every extracted node a random id.
```python theme={null}
{
"name": "Person",
"index_fields": ["name"],
"identity_fields": [],
"fields": [...],
}
```
Both lists may only reference `name` or a declared primitive/enum field — never a relation, and so may `primary_label_field`, a third entity-level key the UI editor writes. `primary_label_field` is validated for that parity but never compiled into the generated model.
<Note>
`identity_fields` is a Python-side extension of the DSL. The cognee UI graph-model editor produces the same JSON shape but never emits it, so models built in the frontend do not merge nodes.
</Note>
</Accordion>
<Accordion title="Field kinds">
* **`primitive`** — a scalar value, with `primitive_type` one of `"string"` (the default), `"number"`, `"boolean"`, or `"date"` (an ISO date string).
* **`enum`** — a string restricted to the non-empty `enum_values` list.
* **`relation`** — an edge named after the field, pointing at `relation.target_entity_name` with `relation.cardinality` of `"one"` or `"many"`.
`primitive` and `enum` fields accept `"required": true` to force the LLM to supply a value; a relation accepts `required` too, for frontend parity, but it is never compiled. Field names are snake\_case with camelCase aliases accepted (`primitiveType`, `targetEntityName`), so one document works for both the UI editor and Python.
</Accordion>
<Accordion title="Validation limits">
Validation is also the safety gate — the generated model is built by executing generated code, so names are restricted and size is capped:
* Entity and field names must be plain identifiers: letters, digits, and underscores, starting with a letter.
* At most 50 entities per spec, and 40 fields per entity.
* Entity names must be unique, and may not collide with the `{Name}Type` marker the compiler generates for another entity.
* Field names must be unique within their entity.
* `root`, when given, must name a declared entity, and every relation must target one.
* A declared `name` field must be a string primitive — it is the node's primary identifier.
* Unknown keys are rejected rather than ignored, so a typo in a key name surfaces immediately.
</Accordion>
<Accordion title="What a custom graph model skips">
Extraction with any model other than the default `KnowledgeGraph` — whether written in Python or compiled from JSON — bypasses [ontology](/core-concepts/further-concepts/ontologies) grounding and the extra node/edge dedup passes cognee runs on the default path, and `functional_relationships` is not supported here: nothing raises if you pass it, but that pass is built for the default path's entity nodes and will not reliably act on a custom model's graph. Stay on the default path when you need those, and use a custom model when you need a predictable, domain-specific shape instead.
</Accordion>
<Columns>
<Card title="Custom Graph Model" icon="share-2" href="/guides/custom-graph-model">
Write the same schema as Python DataPoint classes
</Card>
<Card title="DataPoints" icon="circle" href="/core-concepts/building-blocks/datapoints">
How graph nodes are identified, indexed, and merged
</Card>
<Card title="Custom Prompts" icon="text-wrap" href="/guides/custom-prompts">
Tell the LLM what to look for inside the shape you declared
</Card>
</Columns>
# Graph Visualization
Source: https://docs.cognee.ai/guides/graph-visualization
Step-by-step guide to rendering interactive knowledge graphs
A minimal guide to rendering your current knowledge graph to an interactive HTML file. Use it when you want to see what your memory actually contains — a bounded, readable subgraph by default, or the whole graph on demand.
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) and [Embedding Providers](/setup-configuration/embedding-providers) configured
* Read [Core Concepts Overview](/core-concepts/overview) for how Cognee builds knowledge graphs
* No data is required up front — the script ingests its own sample passages, but it starts with `cognee.forget(everything=True)`, which wipes all existing Cognee data; run it against a setup you can afford to reset
## Code in Action
```python theme={null}
import asyncio
import os
import cognee
from cognee import visualize_graph
ARTIFACTS = os.path.join(os.path.dirname(__file__), ".artifacts", "graph_visualization")
DATASET = "graph_visualization_guide"
TEXT = [
"Python is a programming language. Guido van Rossum created Python.",
"Django is a web framework written in Python.",
"NLP is a subfield of AI. spaCy is an NLP library for Python.",
]
async def main():
os.makedirs(ARTIFACTS, exist_ok=True)
# Prune data and system metadata before running, only if we want "fresh" state.
await cognee.forget(everything=True)
await cognee.remember(TEXT, dataset_name=DATASET, self_improvement=False)
# 1. Bare call: highest-degree nodes seed a representative bounded subgraph.
await visualize_graph(os.path.join(ARTIFACTS, "default_degree_seeded.html"), dataset=DATASET)
# 2. Query-seeded: the query's nearest vector hits become the seeds.
await visualize_graph(
os.path.join(ARTIFACTS, "query_seeded.html"),
dataset=DATASET,
query="What is Python used for?",
)
# 3. Whole graph, unbounded.
await visualize_graph(os.path.join(ARTIFACTS, "full_graph.html"), dataset=DATASET, full=True)
# Two more seeding options, if you already have node ids or a recall result:
# await visualize_graph("explicit_seeds.html", dataset=DATASET, seed_node_ids=[...])
#
# result = await cognee.recall("What is Python?", datasets=[DATASET])
# await visualize_graph("recall_seeded.html", dataset=DATASET, recall_result=result)
# The second seeds the view from the answer's provenance (used_graph_element_ids),
# so you see the subgraph behind a specific answer.
print(f"Wrote visualizations to {ARTIFACTS}")
if __name__ == "__main__":
asyncio.run(main())
```
## What Just Happened
### Step 1: Create Your Knowledge Graph
```python theme={null}
# Prune data and system metadata before running, only if we want "fresh" state.
await cognee.forget(everything=True)
await cognee.remember(TEXT, dataset_name=DATASET, self_improvement=False)
```
This starts from a clean state, then uses `remember()` to ingest the passages into the `graph_visualization_guide` dataset and build the graph in one call. Every render below scopes itself to that same dataset.
### Step 2: Render the Default Bounded Subgraph
```python theme={null}
# 1. Bare call: highest-degree nodes seed a representative bounded subgraph.
await visualize_graph(os.path.join(ARTIFACTS, "default_degree_seeded.html"), dataset=DATASET)
```
A bare call no longer renders the whole graph: it seeds on the highest-degree nodes, expands their 2-hop neighborhood, and caps the result at 500 nodes. That keeps the HTML fast and readable no matter how large the graph grows.
### Step 3: Seed the View From a Query
```python theme={null}
# 2. Query-seeded: the query's nearest vector hits become the seeds.
await visualize_graph(
os.path.join(ARTIFACTS, "query_seeded.html"),
dataset=DATASET,
query="What is Python used for?",
)
```
Passing `query` seeds the subgraph from that query's nearest vector hits instead, so the render shows the neighborhood the question actually lands in.
### Step 4: Render the Whole Graph
```python theme={null}
# 3. Whole graph, unbounded.
await visualize_graph(os.path.join(ARTIFACTS, "full_graph.html"), dataset=DATASET, full=True)
```
`full=True` restores the legacy unbounded render of every node and edge. The script writes all three files side by side under `.artifacts/graph_visualization/` so you can open them and compare the seeding modes.
## What Graph Visualization Shows
* Nodes (entities, types, chunks, summaries) with color coding
* Edges with labels and weights; edge weights control line thickness, and tooltips show extra edge properties
* Interactive features: drag nodes, zoom/pan, hover edges for details
* Output is static, self-contained HTML — open it in any modern browser or share it as an artifact
## Tabs
Every rendered HTML file opens with a tab bar of four views — **Graph**, **Schema**, **Memory**, and **Semantic** — all computed from the same graph payload:
| Tab | What it shows |
| ------------ | ----------------------------------------------------------------------------- |
| **Graph** | Nodes and edges laid out by structure, with layout, label, and color controls |
| **Schema** | A by-type summary: instance counts per semantic type and how types connect |
| **Memory** | A deterministic map of how the memory was built, plus the run timeline |
| **Semantic** | Nodes placed by the 2-D projection of their embeddings |
For what each view shows in detail — layout modes, the label budget, search, the Schema tab's inspector and operations overlay, and the theme toggle — see [Reading the Visualization](/guides/reading-the-visualization).
## Advanced Usage
<Accordion title="Output location">
```python theme={null}
from cognee import visualize_graph
# Writes HTML to your home directory by default
await visualize_graph()
# Writes to the provided file path (created/overwritten)
await visualize_graph("./my_graph.html")
```
</Accordion>
<Accordion title="Bounded subgraph by default">
`visualize_graph()` renders a **bounded, relevant subgraph** by default instead of the entire graph: it picks a small set of seed nodes, expands their *k*-hop neighborhood, and caps the result at `max_nodes`. This keeps renders fast and readable on large graphs. Pass `full=True` to restore the legacy whole-graph render.
Seeds are resolved by priority — the first of these that produces nodes wins:
1. `seed_node_ids` — explicit node ids you pass.
2. `recall_result` — a `recall()` or search result whose graph provenance (`used_graph_element_ids`) seeds the subgraph, i.e. "show me the subgraph behind this answer".
3. `query` — a query string whose nearest vector hits (distance-ranked, nearest first) seed the subgraph.
4. Highest-degree nodes — the fallback so a bare `visualize_graph()` call still shows a representative view.
If none of these resolve any seeds, an empty graph is rendered.
The new parameters are **keyword-only**, so existing positional calls keep working unchanged:
```python theme={null}
from cognee import visualize_graph
# Default: bounded subgraph around the highest-degree nodes
await visualize_graph("./graph.html")
# Seed the subgraph from a query
await visualize_graph("./graph.html", query="natural language processing")
# Show the subgraph behind a recall answer
result = await cognee.recall("What does Alice know?")
await visualize_graph("./graph.html", recall_result=result)
# Legacy whole-graph render
await visualize_graph("./graph.html", full=True)
```
| Parameter | Default | Description |
| ------------------------- | ------- | ------------------------------------------------------------------------ |
| `full` | `False` | Render the entire graph (legacy behavior). |
| `query` | `None` | Query string; its nearest vector hits seed the subgraph. |
| `seed_node_ids` | `None` | Explicit seed node ids for neighborhood expansion. |
| `recall_result` | `None` | A recall/search result whose `used_graph_element_ids` seed the subgraph. |
| `neighborhood_depth` | `2` | *k*-hop expansion depth around the seeds (must be ≥ 1). |
| `neighborhood_seed_top_k` | `10` | Maximum number of seed nodes (must be ≥ 1). |
| `max_nodes` | `500` | Hard cap on rendered nodes after expansion (must be ≥ 1). |
When the neighborhood exceeds `max_nodes`, nodes are kept by hop distance from the seeds (seeds first) and edges survive only when both endpoints do, so no dangling edges remain.
**Over HTTP.** `GET /api/v1/visualize` exposes the same controls as query params: `full`, `query`, `seed_node_ids`, `neighborhood_depth`, `neighborhood_seed_top_k`, and `max_nodes` (`recall_result` is Python-only). For example, `GET /api/v1/visualize?dataset_id=<id>&full=true` returns the whole-graph render. [`GET /api/v1/visualize/json`](/python-api/visualize) accepts the same controls and returns the payload behind that render instead of HTML.
</Accordion>
## Troubleshooting
<Accordion title="Empty graph after cognify">
If `visualize_graph()` logs `No nodes found in the database` (or the HTML opens empty) even though `add()` and `cognify()` ran without errors, the most common causes are:
* **Graph path mismatch.** With the default Ladybug backend, the graph is stored on disk under `<SYSTEM_ROOT_DIRECTORY>/databases/`. By default, `SYSTEM_ROOT_DIRECTORY` is an absolute `.cognee_system` path under Cognee's package root. In notebooks like Colab, it is safer to set explicit absolute paths before running `add()`, `cognify()`, and `visualize_graph()` so every step uses the same persisted location across cells and runtime changes:
```python theme={null}
import cognee
cognee.config.system_root_directory("/content/cognee_system")
cognee.config.data_root_directory("/content/cognee_data")
```
* **`cognify()` produced no nodes.** A run can finish "successfully" yet extract nothing — for example if no data was actually ingested, or graph extraction silently returned empty results (often a misconfigured or failing LLM/embedding provider). Don't rely on the absence of an error; verify the graph was populated.
* **Data was pruned in between.** Calling `cognee.forget(everything=True)` (or `cognee.prune`) after `cognify()` clears the graph, so a later `visualize_graph()` sees nothing.
### Verify the graph was populated
Before visualizing, query the graph engine directly. `get_graph_data()` returns a `(nodes, edges)` tuple, and `get_graph_metrics()` reports the node/edge counts:
```python theme={null}
from cognee.infrastructure.databases.graph import get_graph_engine
graph_engine = await get_graph_engine()
nodes, edges = await graph_engine.get_graph_data()
print(f"nodes={len(nodes)}, edges={len(edges)}")
metrics = await graph_engine.get_graph_metrics()
print(metrics) # {'num_nodes': ..., 'num_edges': ..., ...}
```
If `len(nodes)` is `0` here, the problem is upstream in `add()`/`cognify()` (or a path mismatch), not in visualization. A non-zero count from the same process that then reports `No nodes found` points to a path/config mismatch between steps.
</Accordion>
## Related Projections
Two companion projections summarize your memory without rendering every node — both run end-to-end without an LLM:
* [Schema Inventory](/guides/schema-inventory) — `get_schema_inventory()` summarizes the graph by semantic type: per-type counts, sample names, and relationship distribution.
* [Memory Provenance](/guides/memory-provenance) — `visualize_memory_provenance()` renders the ownership and data-flow story (Tenant → User → Agent → Dataset → file) from the relational database.
## Full Examples
Additional examples about Cognee are available on our [github](https://github.com/topoteretes/cognee/tree/main/examples/guides).
<Accordion title="Semantic memory map">
```python theme={null}
import asyncio
import os
import cognee
from cognee.api.v1.visualize.visualize import visualize_graph
DEST = os.path.join(os.path.expanduser("~"), "semantic_memory_map.html")
# A few short, deliberately multi-topic passages so distinct clusters emerge:
# computing pioneers, jazz, and ocean science.
TEXT = """
Ada Lovelace worked with Charles Babbage on the Analytical Engine in London.
Alan Turing formalized computation and broke ciphers at Bletchley Park.
Grace Hopper built the first compiler and worked on the Harvard Mark I.
Miles Davis recorded Kind of Blue, a landmark modal jazz album, in New York.
John Coltrane played saxophone with the Miles Davis Quintet before A Love Supreme.
Bill Evans, the pianist on Kind of Blue, shaped its impressionistic harmony.
Marine biologists study coral reefs, which host a quarter of all ocean species.
Rising sea temperatures cause coral bleaching, threatening reef ecosystems.
Phytoplankton in the ocean produce a large share of the planet's oxygen.
"""
async def main():
await cognee.prune.prune_data()
await cognee.prune.prune_system(metadata=True)
await cognee.remember(TEXT, self_improvement=False)
html = await visualize_graph(destination_file_path=DEST)
has_semantic = 'data-view="semantic"' in html
has_positions = "window._semanticPositions = null" not in html
print(f"\nSaved: {DEST}")
print(f"Semantic tab present: {has_semantic}")
print(f"Semantic positions set: {has_positions}")
print("Open the file and click the Semantic tab (or append #semantic to the URL).")
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
<Columns>
<Card title="Reading the Visualization" icon="eye" href="/guides/reading-the-visualization">
What each tab shows, and which one to reach for.
</Card>
<Card title="Core Concepts Overview" icon="brain" href="/core-concepts/overview">
Understand how Cognee builds and stores knowledge graphs.
</Card>
<Card title="Visualization Payloads" icon="braces" href="/python-api/visualize">
The JSON payloads and HTTP endpoints behind the render.
</Card>
</Columns>
# Inspecting Hybrid Retrieval Context
Source: https://docs.cognee.ai/guides/hybrid-retrieval-recall
See exactly what HYBRID_COMPLETION sends to the completion step, and shape it with retriever_specific_config
This guide shows you how to look inside `SearchType.HYBRID_COMPLETION` — the default search type — before it turns into a final answer, and how to shape which parts of that context are included.
## Before You Start
* Be familiar with the main concepts of [Recall](/core-concepts/main-operations/recall).
* Complete the [Understand Recall with RAG Completion guide](/guides/rag-recall) to see the shared `recall()` parameters this guide builds on (`query_text`, `datasets`, `only_context`) in action.
## Code in Action
```python theme={null}
import asyncio
import cognee
from cognee import SearchType
QUERY = "What did Alice and Bob work on together?"
async def main():
await cognee.remember(
[
"Alice and Bob were PhD students in Berlin from 2021 to 2024.",
"Alice and Bob worked on a paper together in 2023.",
"Alice joined Cognee as a backend engineer in 2025.",
"Bob joined Cognee as a data scientist in 2026.",
"Alice and Bob worked together on some sections of the documentation of Cognee in 2026.",
"New sections of the documentation are available since July 2026.",
],
self_improvement=False,
)
context_passage_focused = await cognee.recall(
query_text=QUERY,
query_type=SearchType.HYBRID_COMPLETION,
only_context=True,
retriever_specific_config={
"chunks_top_k": 5,
"entities_top_k": 0,
"facts_top_k": 0,
},
)
print(context_passage_focused)
context_entity_focused = await cognee.recall(
query_text=QUERY,
query_type=SearchType.HYBRID_COMPLETION,
only_context=True,
retriever_specific_config={
"chunks_top_k": 0,
"entities_top_k": 5,
"max_edges_per_entity": 5,
"facts_top_k": 0,
},
)
print(context_entity_focused)
context_fact_focused = await cognee.recall(
query_text=QUERY,
query_type=SearchType.HYBRID_COMPLETION,
only_context=True,
retriever_specific_config={
"chunks_top_k": 0,
"entities_top_k": 5,
"max_edges_per_entity": 0,
"facts_top_k": 5,
},
)
print(context_fact_focused)
context_balanced = await cognee.recall(
query_text=QUERY,
query_type=SearchType.HYBRID_COMPLETION,
only_context=True,
retriever_specific_config={
"chunks_top_k": 2,
"entities_top_k": 2,
"max_edges_per_entity": 3,
"facts_top_k": 2,
},
)
print(context_balanced)
answer = await cognee.recall(
query_text=QUERY,
query_type=SearchType.HYBRID_COMPLETION,
retriever_specific_config={
"chunks_top_k": 2,
"entities_top_k": 2,
"max_edges_per_entity": 3,
"facts_top_k": 2,
},
)
print(answer)
if __name__ == "__main__":
asyncio.run(main())
```
The complete runnable script is on GitHub: [`examples/guides/hybrid_retrieval_recall.py`](https://github.com/topoteretes/cognee/blob/dev/examples/guides/hybrid_retrieval_recall.py).
## What Just Happened
Each `recall()` call below sets `only_context=True`, and Steps 2-4 also set one or two `retriever_specific_config` limits to `0`. A section's `*_top_k` at `0` removes that section from the context entirely; a non-zero limit controls how many items of that section are included — a larger number produces a more detailed (and larger) context.
### Step 1: Ingest the Example Dataset
```python theme={null}
await cognee.remember(
[
"Alice and Bob were PhD students in Berlin from 2021 to 2024.",
"Alice and Bob worked on a paper together in 2023.",
"Alice joined Cognee as a backend engineer in 2025.",
"Bob joined Cognee as a data scientist in 2026.",
"Alice and Bob worked together on some sections of the documentation of Cognee in 2026.",
"New sections of the documentation are available since July 2026.",
],
self_improvement=False,
)
```
A handful of sentences about Alice and Bob gives hybrid retrieval something to find — entities, a relationship between them, and the passages they come from.
### Step 2: Passage-Focused Context
```python theme={null}
context_passage_focused = await cognee.recall(
query_text=QUERY,
query_type=SearchType.HYBRID_COMPLETION,
only_context=True,
retriever_specific_config={
"chunks_top_k": 5,
"entities_top_k": 0,
"facts_top_k": 0,
},
)
print(context_passage_focused)
```
With `entities_top_k` and `facts_top_k` at `0`, `context_passage_focused` contains only matched passages — the raw text chunks that matched the query, combining lexical (keyword) and semantic (embedding) search, with no graph entities or derived facts.
Use this when you want to see raw supporting text without any graph-derived summarization — for example, to quote source text verbatim, or to debug retrieval quality without graph-derived noise in the way.
### Step 3: Entity-Focused Context
```python theme={null}
context_entity_focused = await cognee.recall(
query_text=QUERY,
query_type=SearchType.HYBRID_COMPLETION,
only_context=True,
retriever_specific_config={
"chunks_top_k": 0,
"entities_top_k": 5,
"max_edges_per_entity": 5,
"facts_top_k": 0,
},
)
print(context_entity_focused)
```
With `chunks_top_k` and `facts_top_k` at `0`, `context_entity_focused` contains only matched entities — nodes from the graph (like `Alice`, `Bob`, `Cognee`) — and the edges connected to each one (capped by `max_edges_per_entity`), rendered as short sentences such as `Alice contributed to documentation`.
Use this when you care about which entities are connected and how, more than the exact wording of the source text.
### Step 4: Fact-Focused Context
```python theme={null}
context_fact_focused = await cognee.recall(
query_text=QUERY,
query_type=SearchType.HYBRID_COMPLETION,
only_context=True,
retriever_specific_config={
"chunks_top_k": 0,
"entities_top_k": 5,
"max_edges_per_entity": 0,
"facts_top_k": 5,
},
)
print(context_fact_focused)
```
The fact budget is tied to the entity lane, so the lane has to stay on for `facts_top_k` to apply — with `entities_top_k` at `0` no facts are returned. Keeping `entities_top_k` at `5` and setting `max_edges_per_entity` to `0` drops the per-entity edge bullets from Step 3, so `context_fact_focused` contains the matched entity names plus the compact, fact-style statements derived from graph edges, with no raw passages.
Use this when you want a small context — for example to get a quick relationship summary without the full passage text or the per-entity edge listings.
### Step 5: Balanced Context
```python theme={null}
context_balanced = await cognee.recall(
query_text=QUERY,
query_type=SearchType.HYBRID_COMPLETION,
only_context=True,
retriever_specific_config={
"chunks_top_k": 2,
"entities_top_k": 2,
"max_edges_per_entity": 3,
"facts_top_k": 2,
},
)
print(context_balanced)
```
All three sections are present in `context_balanced`, each capped at a small limit. This mirrors the default behavior, just with tighter limits — useful when you want a compact context that still draws on passages, entities, and facts together.
## Generating the Final Answer
Every call above passes `only_context=True`, so `recall()` stops after assembling the context and never reaches the completion step. Drop `only_context=True` and `recall()` sends the assembled context to the LLM to generate a real answer instead:
```python theme={null}
answer = await cognee.recall(
query_text=QUERY,
query_type=SearchType.HYBRID_COMPLETION,
retriever_specific_config={
"chunks_top_k": 2,
"entities_top_k": 2,
"max_edges_per_entity": 3,
"facts_top_k": 2,
},
)
print(answer)
```
A typical result:
```text theme={null}
They worked on a paper in 2023 and on some sections of the Cognee documentation in 2026.
```
(The exact wording depends on the LLM provider you use.) This works the same way with any of the `retriever_specific_config` shapes from Steps 2-5 above — passage-focused, entity-focused, fact-focused, or balanced. The `*_top_k` limits keep their meaning in both cases: they are the per-section caps on what ends up in the context.
One difference between the two calls: dropping `only_context=True` also lets the call run as a full session turn. In the default `concurrent` session-search mode a session turn retrieves twice — once with your raw `query_text` and once with a rewrite that prefixes the last couple of question/answer turns in the session — and merges the two result sets per section under the same `chunks_top_k` / `entities_top_k` / `facts_top_k` caps. So on a follow-up question in an ongoing session, the answer's context can differ from what the equivalent `only_context=True` call would have surfaced; on the first turn of a session — as in this guide — the two calls retrieve identically. See [Session-context guidance](/core-concepts/sessions-and-caching#session-context-guidance-auto-feedback) for the full merge behavior and how to switch back to `sequential`.
<Columns>
<Card title="Recall" icon="search" href="/core-concepts/main-operations/recall">
Understand recall()'s full parameter surface and auto-routing behavior
</Card>
<Card title="SearchType" icon="list" href="/python-api/search-type">
See every search type and its retriever\_specific\_config options
</Card>
</Columns>
# Image OCR Extraction
Source: https://docs.cognee.ai/guides/image-ocr-extraction
Print the text an image becomes before you ingest it: the vision-model transcription plus appended OCR text
A minimal guide to inspecting what Cognee extracts from an image. Run it against a screenshot, a scanned page, or a dense chart to see exactly which text would reach the knowledge graph — before spending an ingestion run on it.
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured with a vision-capable `LLM_MODEL` — the transcription step calls it
* Install the OCR engine: `pip install "cognee[rapidocr]"`
* Read [Loaders](/core-concepts/further-concepts/loaders) for how Cognee picks a loader per file and for the full set of image environment variables
* Have an image file on disk. The script reads `revenue_chart.png` from the example's [`multimedia_audio_image_processing_example_data/`](https://github.com/topoteretes/cognee/tree/dev/examples/guides/multimedia_audio_image_processing_example_data) directory in the Cognee repo — download it from there to reproduce the run exactly, or point `image_path` at your own file to check that one instead
## Code in Action
```python theme={null}
import asyncio
import os
import pathlib
from cognee.infrastructure.loaders.core.image_loader import ImageLoader
os.environ.setdefault("IMAGE_EXTRACTION_ENABLED", "true")
os.environ.setdefault("IMAGE_OCR_ENABLED", "true")
async def main():
image_path = os.path.join(
pathlib.Path(__file__).parent,
"multimedia_audio_image_processing_example_data/revenue_chart.png",
)
text = await ImageLoader().load(image_path, persist=False)
print("=== Text extracted from the image ===")
print(text)
if __name__ == "__main__":
asyncio.run(main())
```
## What Just Happened
### Step 1: Enable Extraction and OCR
```python theme={null}
import asyncio
import os
import pathlib
from cognee.infrastructure.loaders.core.image_loader import ImageLoader
os.environ.setdefault("IMAGE_EXTRACTION_ENABLED", "true")
os.environ.setdefault("IMAGE_OCR_ENABLED", "true")
```
`ImageLoader` is the loader `remember()` would pick for an image, imported directly here so you can call it on its own. `IMAGE_EXTRACTION_ENABLED` asks the vision model for entities, relationships, and verbatim text rather than a short caption; `IMAGE_OCR_ENABLED` adds a local OCR pass on top. Both are read when the image loads, and `setdefault` leaves any value you already set in the environment or `.env` untouched.
### Step 2: Point at an Image File
```python theme={null}
image_path = os.path.join(
pathlib.Path(__file__).parent,
"multimedia_audio_image_processing_example_data/revenue_chart.png",
)
```
The path is resolved relative to the script so the example runs from any working directory. Swap in a path to your own screenshot or scan to check what Cognee makes of it.
### Step 3: Load the Image and Print the Text
```python theme={null}
text = await ImageLoader().load(image_path, persist=False)
print("=== Text extracted from the image ===")
print(text)
```
`load()` runs the transcription and, when OCR is on, appends the recognized text to it. `persist=False` returns that text directly; the default `persist=True` writes it into Cognee's data directory and returns the file path instead — useful in a pipeline, unhelpful when you just want to read the result.
## Advanced Usage
<AccordionGroup>
<Accordion title="Reading the two halves of the output">
The vision transcription comes first. If the OCR pass recognized anything, it follows under an `[OCR extracted text]` heading, so you can tell which half produced which text. OCR output is truncated at 8000 characters, and a failing OCR pass is logged and skipped rather than raising — so a result with no `[OCR extracted text]` block means OCR found nothing, was disabled, or failed.
</Accordion>
<Accordion title="Checking one flag at a time">
To see what each stage contributes, run the script twice. `IMAGE_OCR_ENABLED="false"` gives the vision transcription alone; `IMAGE_EXTRACTION_ENABLED="false"` restores the legacy short-caption prompt. Both flags, and the transcription prompt and token-cap settings around them, are documented in [Loaders](/core-concepts/further-concepts/loaders).
</Accordion>
<Accordion title="From check to ingestion">
Nothing here writes to memory. Once the extracted text looks right, pass the same image path to `remember()` — it selects `ImageLoader` for you and reads the same environment variables, so the text you just printed is the text that gets chunked and turned into graph memory.
</Accordion>
</AccordionGroup>
<Columns>
<Card title="Loaders" icon="download" href="/core-concepts/further-concepts/loaders">
See every format Cognee reads and each image transcription setting
</Card>
<Card title="Remember" icon="brain" href="/core-concepts/main-operations/remember">
Ingest the image once its extracted text looks right
</Card>
</Columns>
# Local Ollama Pipeline
Source: https://docs.cognee.ai/guides/local-ollama
Run remember and recall end to end against a local Ollama server and cognee's embedded stores.
A minimal guide to running a complete cognee pipeline on your own machine: Ollama for both generation and embeddings, and cognee's embedded stores for the graph, vectors, and metadata. Use it when you want the whole `remember` → `recall` round trip to run without any hosted API call, in a throwaway directory you can delete afterwards.
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Install [Ollama](https://ollama.ai), start it with `ollama serve`, and pull the two models the script uses:
```bash theme={null}
ollama pull llama3.1:8b
ollama pull nomic-embed-text
```
* Read [Local Setup (No API Key)](/guides/local-setup) for the equivalent `.env` configuration and the local-run troubleshooting list
* Check [LLM Providers](/setup-configuration/llm-providers) and [Embedding Providers](/setup-configuration/embedding-providers) if you want to point the script at a different local model
* Nothing else to install: the graph, vector, and relational stores this script selects are embedded and ship with cognee
## Code in Action
```python theme={null}
import asyncio
import os
import tempfile
from pathlib import Path
# Setup temp directory to keep this example self-contained
_DATA_DIR = tempfile.mkdtemp(prefix="cognee_ollama_example_")
os.environ["ENABLE_BACKEND_ACCESS_CONTROL"] = "false"
os.environ["CACHING"] = "false"
# Configure Ollama environment settings
os.environ["LLM_PROVIDER"] = "ollama"
os.environ["LLM_MODEL"] = "llama3.1:8b"
os.environ["LLM_ENDPOINT"] = "http://localhost:11434"
os.environ["LLM_API_KEY"] = "ollama"
os.environ["LLM_TEMPERATURE"] = "0.0"
os.environ["EMBEDDING_PROVIDER"] = "ollama"
os.environ["EMBEDDING_MODEL"] = "nomic-embed-text"
os.environ["EMBEDDING_ENDPOINT"] = "http://localhost:11434/api/embed"
os.environ["EMBEDDING_DIMENSIONS"] = "768"
os.environ["HUGGINGFACE_TOKENIZER"] = "nomic-ai/nomic-embed-text-v1.5"
import cognee # noqa: E402
from cognee.infrastructure.llm.config import get_llm_config # noqa: E402
from cognee.modules.search.types import SearchType # noqa: E402
# Force local embedded stack configuration
cognee.config.set_graph_database_provider("kuzu")
cognee.config.set_vector_db_provider("lancedb")
cognee.config.data_root_directory(str(Path(_DATA_DIR) / "data"))
cognee.config.system_root_directory(str(Path(_DATA_DIR) / "system"))
SAMPLE_TEXT = """\
Cognee is an open-source library that helps developers turn documents into AI memory.
It builds semantic graphs, indexes entities, and stores vectors to enable structured retrieval.
Cognee supports local execution via Ollama as well as hosted cloud providers.
"""
def banner(title: str) -> None:
print("\n" + "=" * 78)
print(title)
print("=" * 78)
async def main() -> None:
# Start from a clean slate in isolated directory
await cognee.prune.prune_data()
await cognee.prune.prune_system(metadata=True)
banner("LOCAL PIPELINE: REMEMBER USING OLLAMA")
llm_config = get_llm_config()
print(f"Using LLM: {llm_config.llm_model}")
print(f"Using Embeddings: {os.environ.get('EMBEDDING_MODEL')}")
# Ingest and build the knowledge graph (this will trigger a warning if an
# unvalidated model is used)
await cognee.remember(SAMPLE_TEXT, dataset_name="ollama_local_demo", self_improvement=False)
print("Local knowledge graph built successfully.")
banner("LOCAL RECALL")
query = "What does Cognee help developers do?"
results = await cognee.recall(
query_text=query,
query_type=SearchType.GRAPH_COMPLETION,
datasets=["ollama_local_demo"],
)
print(f"Query: {query}")
print("Recall Results:")
print(results[0].text if results else "<no results>")
if __name__ == "__main__":
asyncio.run(main())
```
## What Just Happened
### Step 1: Isolate the Example's Storage
```python theme={null}
# Setup temp directory to keep this example self-contained
_DATA_DIR = tempfile.mkdtemp(prefix="cognee_ollama_example_")
os.environ["ENABLE_BACKEND_ACCESS_CONTROL"] = "false"
os.environ["CACHING"] = "false"
```
A fresh temporary directory keeps this run's databases away from your default cognee directories, so the example leaves nothing behind. Access control stays off because the script runs as a plain single-user script, and caching stays off so every run really exercises the local models instead of replaying earlier answers.
### Step 2: Point Cognee at Ollama
```python theme={null}
# Configure Ollama environment settings
os.environ["LLM_PROVIDER"] = "ollama"
os.environ["LLM_MODEL"] = "llama3.1:8b"
os.environ["LLM_ENDPOINT"] = "http://localhost:11434"
os.environ["LLM_API_KEY"] = "ollama"
os.environ["LLM_TEMPERATURE"] = "0.0"
os.environ["EMBEDDING_PROVIDER"] = "ollama"
os.environ["EMBEDDING_MODEL"] = "nomic-embed-text"
os.environ["EMBEDDING_ENDPOINT"] = "http://localhost:11434/api/embed"
os.environ["EMBEDDING_DIMENSIONS"] = "768"
os.environ["HUGGINGFACE_TOKENIZER"] = "nomic-ai/nomic-embed-text-v1.5"
```
Both halves have to be set together: configuring only the LLM leaves embeddings falling back to OpenAI, which would need an API key. The two endpoints differ in shape because they are built by different layers: `LLM_ENDPOINT` is the bare host — on the default `litellm_native` backend LiteLLM appends Ollama's native `/api/generate` path, so a trailing `/v1` would 404 (see [LLM Providers → Ollama](/setup-configuration/llm-providers#ollama-local)) — while `EMBEDDING_ENDPOINT` is passed through as-is by Cognee's own embedding engine and needs the full `/api/embed` path. `LLM_API_KEY` is a placeholder Ollama never validates, and `EMBEDDING_DIMENSIONS` plus `HUGGINGFACE_TOKENIZER` describe `nomic-embed-text` so cognee sizes its vectors and counts tokens correctly. The environment is set before `import cognee` so the configuration is in place when cognee reads it — hence the `# noqa: E402` markers on the imports that follow.
### Step 3: Pin the Embedded Local Stack
```python theme={null}
# Force local embedded stack configuration
cognee.config.set_graph_database_provider("kuzu")
cognee.config.set_vector_db_provider("lancedb")
cognee.config.data_root_directory(str(Path(_DATA_DIR) / "data"))
cognee.config.system_root_directory(str(Path(_DATA_DIR) / "system"))
```
These four calls make the storage side local too: an embedded graph database, LanceDB for vectors, and SQLite metadata, all written under the temporary directory from Step 1. Both calls select what cognee already uses out of the box (`kuzu` is an accepted alias for the default embedded graph store, and LanceDB is the default vector store), so they are explicit rather than required — they keep the script behaving the same way even if `GRAPH_DATABASE_PROVIDER` or `VECTOR_DB_PROVIDER` is set in your environment. See [Graph Stores](/setup-configuration/graph-stores) for the other providers these names select between.
### Step 4: Remember the Sample Text
```python theme={null}
SAMPLE_TEXT = """\
Cognee is an open-source library that helps developers turn documents into AI memory.
It builds semantic graphs, indexes entities, and stores vectors to enable structured retrieval.
Cognee supports local execution via Ollama as well as hosted cloud providers.
"""
await cognee.remember(SAMPLE_TEXT, dataset_name="ollama_local_demo", self_improvement=False)
print("Local knowledge graph built successfully.")
```
`remember()` runs the whole extraction pipeline against the local models: `llama3.1:8b` pulls entities and relationships out of the text and `nomic-embed-text` embeds the chunks. Three sentences is deliberately small — an 8B model on CPU is far slower than a hosted one, so keep test inputs short. `llama3.1` is on cognee's recommended list for structured extraction, so this script runs without warnings; swap in a model cognee hasn't validated and the log opens with an advisory model-support warning instead — not a failure, extraction continues either way.
### Step 5: Recall From the Local Graph
```python theme={null}
query = "What does Cognee help developers do?"
results = await cognee.recall(
query_text=query,
query_type=SearchType.GRAPH_COMPLETION,
datasets=["ollama_local_demo"],
)
print(f"Query: {query}")
print("Recall Results:")
print(results[0].text if results else "<no results>")
```
`SearchType.GRAPH_COMPLETION` retrieves the triplets around the query and asks the local LLM to answer from them, which is the search type that proves the graph was actually populated. `datasets=["ollama_local_demo"]` scopes the search to the dataset built above, and printing `results[0].text` shows the generated answer rather than the raw result object.
<Columns>
<Card title="Local Setup (No API Key)" icon="computer" href="/guides/local-setup">
The same configuration as `.env` variables, plus local-run troubleshooting.
</Card>
<Card title="LLM Providers" icon="brain" href="/setup-configuration/llm-providers">
Swap in another local or hosted model for generation.
</Card>
<Card title="Embedding Providers" icon="layers" href="/setup-configuration/embedding-providers">
Pick a different embedding model and its matching dimensions.
</Card>
</Columns>
# Local Setup (No API Key)
Source: https://docs.cognee.ai/guides/local-setup
Run Cognee locally with Ollama and Fastembed.
Run Cognee entirely on your own machine — no cloud API key required. The key rule is that **both** the LLM provider **and** the embedding provider must be configured together to use a local backend; configuring only one will cause the other to fall back to OpenAI.
**Before you start:**
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Install [Ollama](https://ollama.ai) if using the Ollama options below
<Info>
After switching to a local provider for the first time, call `cognee.prune.prune_system(metadata=True)` before running `cognify` to ensure there are no stale vector collections from the previous (OpenAI) embedding dimensions.
</Info>
<Tabs>
<Tab title="Ollama (LLM + Embeddings)">
Fully local setup using [Ollama](https://ollama.ai) for both text generation and embeddings.
**Prerequisites**: Install Ollama and pull the required models:
```bash theme={null}
ollama pull llama3.1:8b
ollama pull nomic-embed-text:latest
```
**.env configuration:**
```dotenv theme={null}
# LLM — Ollama
LLM_PROVIDER="ollama"
LLM_MODEL="llama3.1:8b"
LLM_ENDPOINT="http://localhost:11434"
LLM_API_KEY="ollama"
# Embeddings — Ollama
EMBEDDING_PROVIDER="ollama"
EMBEDDING_MODEL="nomic-embed-text:latest"
EMBEDDING_ENDPOINT="http://localhost:11434/api/embed"
EMBEDDING_DIMENSIONS="768"
HUGGINGFACE_TOKENIZER="nomic-ai/nomic-embed-text-v1.5" # optional, recommended for accurate token counting
```
`LLM_API_KEY="ollama"` is a placeholder required by the client library — Ollama itself does not validate it.
`LLM_ENDPOINT` is the **bare host with no path**: on the default `litellm_native` structured output backend, LiteLLM routes Ollama over its native API and appends `/api/generate` itself, so a trailing `/v1` turns into `/v1/api/generate` and 404s. `EMBEDDING_ENDPOINT` is a separate, Cognee-owned path and keeps its `/api/embed` suffix. See [LLM Providers → Ollama](/setup-configuration/llm-providers#ollama-local) for the `instructor`-backend exception.
`HUGGINGFACE_TOKENIZER` is the HuggingFace repo ID of the tokenizer used for token counting when sending requests to the Ollama embedding endpoint. It is optional — Cognee no longer requires it at startup — but recommended for accurate token counting.
**Runnable version**: [`examples/guides/local_ollama_example.py`](https://github.com/topoteretes/cognee/blob/dev/examples/guides/local_ollama_example.py) sets this same configuration in Python, pins the embedded local stack (Ladybug graph, LanceDB vectors, SQLite metadata) into a temporary directory, and runs `remember` → `recall` over a short sample text.
<Note>
If your log shows a warning about the model name early in the run, that is Cognee's advisory Ollama model-support check, not a failure — extraction continues regardless. See [Model Support Warning](/setup-configuration/llm-providers#ollama-local) on the LLM Providers page for the recommended, problematic, and unvalidated model lists.
</Note>
</Tab>
<Tab title="Ollama LLM + Fastembed">
Uses [Ollama](https://ollama.ai) for text generation and [Fastembed](https://github.com/qdrant/fastembed) for CPU-friendly local embeddings (no Ollama embedding model required).
**Prerequisites**: Install Ollama and pull the LLM model:
```bash theme={null}
ollama pull llama3.1:8b
```
Install the Fastembed extra (not bundled with the base `cognee` package):
```bash theme={null}
pip install 'cognee[fastembed]'
```
See the [Fastembed setup notes](/setup-configuration/embedding-providers#fastembed-local) for supported models and dimensions.
**.env configuration:**
```dotenv theme={null}
# LLM — Ollama
LLM_PROVIDER="ollama"
LLM_MODEL="llama3.1:8b"
LLM_ENDPOINT="http://localhost:11434"
LLM_API_KEY="ollama"
# Embeddings — Fastembed (CPU, no API key)
EMBEDDING_PROVIDER="fastembed"
EMBEDDING_MODEL="sentence-transformers/all-MiniLM-L6-v2"
EMBEDDING_DIMENSIONS="384"
```
</Tab>
</Tabs>
<Note>
If you are running the Cognee backend locally or self-hosted, you can verify it with:
```bash theme={null}
curl http://localhost:8000/health
```
A `200 OK` confirms the backend is up. This is the local/self-hosted check only; it does not contact Cognee Cloud.
</Note>
## Troubleshooting
<AccordionGroup>
<Accordion title="`LLMAPIKeyNotSetError: LLM API key is not set` on a fully local setup">
Cognee is free and open source — running it locally with Ollama needs **no account, subscription, or paid API key**. You are not being asked to pay for anything.
The error appears because Ollama is one of the providers Cognee requires a **non-empty** `LLM_API_KEY` for, even though Ollama itself ignores the value. If `LLM_API_KEY` is unset or empty, Cognee raises `LLMAPIKeyNotSetError` before it ever contacts your local server.
The fix is to set any placeholder string — the convention is `ollama`:
```dotenv theme={null}
LLM_PROVIDER="ollama"
LLM_MODEL="llama3.1:8b"
LLM_ENDPOINT="http://localhost:11434"
LLM_API_KEY="ollama"
```
For the local examples above, keep `LLM_API_KEY="ollama"` in place. Fastembed does not need an embedding API key, and Ollama embeddings use the same local placeholder. Use the complete `.env` blocks in the tabs above so neither provider falls back to OpenAI.
If you hit this error while running the local UI (`cognee.start_ui()` or `cognee-cli -ui`), set `LLM_API_KEY` in the backend's environment or `.env` and restart it — the UI has no in-app field for the key. See [Run the UI Locally](/cognee-cloud/local-ui).
</Accordion>
<Accordion title="`Cannot connect to host` / connection refused with Ollama">
If Cognee can't reach Ollama, work through these checks:
1. **Ollama is running.** Start the server with `ollama serve`, or open the Ollama desktop app. Verify with:
```bash theme={null}
curl http://localhost:11434/api/tags
```
2. **Endpoints match Ollama's API surface.** The LLM endpoint is the bare host — no `/v1`, since Cognee's default backend appends Ollama's native path itself — while the embedding endpoint carries the full `/api/embed` path:
```dotenv theme={null}
LLM_ENDPOINT="http://localhost:11434"
EMBEDDING_ENDPOINT="http://localhost:11434/api/embed"
```
3. **The required models are pulled.** Cognee does not pull models on demand:
```bash theme={null}
ollama pull llama3.1:8b
ollama pull nomic-embed-text:latest
```
4. **Running Cognee in Docker?** `localhost` inside the container does not point at Ollama on the host. Use `host.docker.internal` instead:
```dotenv theme={null}
LLM_ENDPOINT="http://host.docker.internal:11434"
EMBEDDING_ENDPOINT="http://host.docker.internal:11434/api/embed"
```
5. **Repeated timeouts under load.** Ollama processes requests sequentially. If the default `EMBEDDING_BATCH_SIZE` of `36` overwhelms it, lower the batch size:
```dotenv theme={null}
EMBEDDING_BATCH_SIZE="5"
```
For more detail on embedding-side tuning, see [Embedding Providers → Ollama](/setup-configuration/embedding-providers#ollama-local).
</Accordion>
<Accordion title="Extraction output is inconsistent, or sampling changed after an upgrade">
On a local inference server — Ollama, llama.cpp, or LM Studio — an unset `LLM_TEMPERATURE` sends `temperature: 0.0`, so extraction runs deterministically by default. Previously Cognee sent no temperature at all on these providers and the model's own default applied, which is `1.0` for several Ollama models and produced varied, harder-to-parse extraction output.
Two things follow from that:
1. **Empty or malformed extraction results.** You no longer need to set `LLM_TEMPERATURE=0.0` yourself for deterministic output formatting — check instead that nothing has raised it. If you set `LLM_TEMPERATURE` to something higher, lower it back to `0.0`.
2. **Sampling that got less varied after upgrading.** If you were relying on the model's own default, set the value you want explicitly:
```dotenv theme={null}
LLM_TEMPERATURE=1.0
```
A `temperature` key in `LLM_ARGS` also still wins over `LLM_TEMPERATURE`, so `LLM_ARGS='{"temperature": 1.0}'` works too.
Hosted providers are unaffected, and so is vLLM — those still send no temperature when the variable is unset. See [Temperature and Seed](/setup-configuration/llm-providers#temperature-and-seed) for the full precedence rules.
</Accordion>
</AccordionGroup>
<Columns>
<Card title="LLM Providers" icon="brain" href="/setup-configuration/llm-providers">
Configure OpenAI, Azure, Gemini, Anthropic, Ollama, or custom LLM providers
</Card>
<Card title="Embedding Providers" icon="layers" href="/setup-configuration/embedding-providers">
Set up OpenAI, Mistral, Ollama, Fastembed, or custom embedding services
</Card>
<Card title="Setup Configuration" icon="settings" href="/setup-configuration/overview">
Full configuration reference for all backends
</Card>
</Columns>
# Low-Level LLM
Source: https://docs.cognee.ai/guides/low-level-llm
Step-by-step guide to using acreate_structured_output for direct LLM interaction
A minimal guide to the one function you can call directly to get Pydantic-validated structured output from an LLM.
**Before you start:**
* Complete [Quickstart](getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](setup-configuration/llm-providers) configured
* Have some text to process
## What It Is
* Single entrypoint: `LLMGateway.acreate_structured_output(text, system_prompt, response_model)`
* Returns an instance of your Pydantic `response_model` filled by the LLM
* Backend-agnostic: uses LiteLLM Native (the default), LiteLLM+Instructor, or BAML under the hood based on config — your code doesn't change
<Note>
This function is used by default during cognify via the extractor. The backend switch lives in `cognee/infrastructure/llm/LLMGateway.py`.
</Note>
## Code in Action
### Step 1: Define Your Schema
```python theme={null}
class MiniEntity(BaseModel):
name: str
type: str
class MiniGraph(BaseModel):
nodes: list[MiniEntity]
```
Create Pydantic models that define the structure you want the LLM to return. The LLM will fill these models with data extracted from your text.
### Step 2: Write a System Prompt
```python theme={null}
system_prompt = (
"Extract entities as nodes with name and type. "
"Use concise, literal values present in the text."
)
```
Write a clear prompt that tells the LLM what to extract and how to structure it. Short, explicit prompts work best.
### Step 3: Call the LLM
```python theme={null}
result = await LLMGateway.acreate_structured_output(text, system_prompt, MiniGraph)
```
This calls the LLM with your text and prompt, returning a Pydantic model instance with the extracted data.
<Note>
Passing `response_model=str` returns a plain string instead of a validated model. With the LiteLLM + Instructor backend, Cognee bypasses the Instructor structured-output pipeline for the default OpenAI, generic, and Ollama adapters and sends the prompt directly to the provider, returning the model's raw text content. With BAML, the call still routes through BAML and returns the generated text field. Pass a Pydantic model to get a validated instance back.
</Note>
<Tip>
A sync variant exists: `LLMGateway.create_structured_output(...)`.
</Tip>
## Custom Tasks
This function is often used when creating custom tasks for processing data with structured output. You'll see it in action when we cover custom task creation in a future guide.
## Backend Doesn't Matter
The config decides the engine:
* `STRUCTURED_OUTPUT_FRAMEWORK=instructor` → LiteLLM + Instructor
* `STRUCTURED_OUTPUT_FRAMEWORK=baml` → BAML client/registry
* `STRUCTURED_OUTPUT_FRAMEWORK=litellm_native` → LiteLLM native structured output
All three paths return the same Pydantic model instance to your code.
## Full Example
<Accordion title="Latest guide">
```python theme={null}
import asyncio
from pydantic import BaseModel
from cognee.infrastructure.llm.LLMGateway import LLMGateway
class MiniEntity(BaseModel):
name: str
type: str
class MiniGraph(BaseModel):
nodes: list[MiniEntity]
async def main():
system_prompt = (
"Extract entities as nodes with name and type. "
"Use concise, literal values present in the text."
)
text = "Apple develops iPhone; Audi produces the R8."
result = await LLMGateway.acreate_structured_output(text, system_prompt, MiniGraph)
print(result)
# MiniGraph(nodes=[MiniEntity(name='Apple', type='Organization'), ...])
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
<Note>
This simple example uses a basic schema for demonstration. In practice, you can define complex Pydantic models with nested structures, validation rules, and custom types.
</Note>
<Columns>
<Card title="Structured Output" icon="brackets" href="/setup-configuration/structured-output-backends">
Learn about structured output frameworks
</Card>
<Card title="Custom Prompts" icon="text-wrap" href="/guides/custom-prompts">
Control extraction with custom prompts
</Card>
<Card title="API Reference" icon="code" href="/api-reference/introduction">
Explore API endpoints
</Card>
</Columns>
# Entity Consolidation
Source: https://docs.cognee.ai/guides/memify-entity-consolidation
Rewrite fragmented entity descriptions using LLM analysis of graph neighborhoods
A minimal guide to consolidating entity descriptions in an existing knowledge graph. After ingestion, entity descriptions can be fragmented or repetitive because each one is derived from a single chunk — this lower-level Memify pipeline rewrites each entity's description using the LLM and the entity's full neighborhood context. Use `improve()` for the standard self-improvement flow; use this guide when you specifically want entity-description consolidation.
<Note>
This pipeline only rewrites description text — it never creates or deletes nodes. To merge near-duplicate `Entity` nodes into one, see [Entity Deduplication](/guides/memify-entity-deduplication).
</Note>
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured
* Have an existing knowledge graph with `Entity` nodes — the script below builds one with `remember()`
## Code in Action
```python theme={null}
import asyncio
import cognee
from os import path
from cognee.api.v1.visualize.visualize import visualize_graph
from cognee.memify_pipelines.consolidate_entity_descriptions import (
consolidate_entity_descriptions_pipeline,
)
custom_prompt = """
Extract only people and cities as entities.
Connect people to cities with the relationship "lives_in".
Ignore all other entities.
"""
graph_visualization_path_before_enrichment = path.join(
path.dirname(__file__), ".artifacts", "before_consolidate_enrichment_entity_descriptions.html"
)
graph_visualization_path_after_enrichment = path.join(
path.dirname(__file__), ".artifacts", "after_consolidate_enrichment_entity_descriptions.html"
)
async def main():
# Prune data and system metadata before running, only if we want "fresh" state.
await cognee.forget(everything=True)
await cognee.remember(
[
"Alice moved to Paris in 2010, while Bob has always lived in New York.",
"Andreas was born in Venice, but later settled in Lisbon.",
"Diana and Tom were born and raised in Helsinki. Diana currently resides in Berlin, while Tom never moved.",
],
custom_prompt=custom_prompt,
self_improvement=False,
)
await visualize_graph(graph_visualization_path_before_enrichment)
await consolidate_entity_descriptions_pipeline()
await visualize_graph(graph_visualization_path_after_enrichment)
if __name__ == "__main__":
asyncio.run(main())
```
## What Just Happened
### Step 1: Clear Existing Data
```python theme={null}
await cognee.forget(everything=True)
```
Start from a clean state so the before-and-after visualizations only reflect this example run.
### Step 2: Build and Visualize the Graph
```python theme={null}
custom_prompt = """
Extract only people and cities as entities.
Connect people to cities with the relationship "lives_in".
Ignore all other entities.
"""
await cognee.remember(
[
"Alice moved to Paris in 2010, while Bob has always lived in New York.",
"Andreas was born in Venice, but later settled in Lisbon.",
"Diana and Tom were born and raised in Helsinki. Diana currently resides in Berlin, while Tom never moved.",
],
custom_prompt=custom_prompt,
self_improvement=False,
)
await visualize_graph(graph_visualization_path_before_enrichment)
```
Create a focused graph with only people, cities, and `lives_in` relationships, then save a visualization of the graph before consolidation.
### Step 3: Consolidate and Visualize Again
```python theme={null}
await consolidate_entity_descriptions_pipeline()
await visualize_graph(graph_visualization_path_after_enrichment)
```
The pipeline rewrites each existing `Entity` node's `description` in place using LLM analysis of the entity's neighbors and edges — no nodes are created or deleted. Descriptions become more coherent because the LLM sees each entity in the context of its graph neighborhood, not just the original chunk text, and the before and after HTML files make the change easy to inspect.
## Additional Information
* Runnable guide script available on our [GitHub](https://github.com/topoteretes/cognee/blob/main/examples/guides/consolidate_entity_descriptions_example.py)
* Pipeline implementation: [consolidate\_entity\_descriptions.py](https://github.com/topoteretes/cognee/blob/main/cognee/memify_pipelines/consolidate_entity_descriptions.py)
<Accordion title="Under the hood">
Three tasks run in sequence:
1. **`get_entities_with_neighborhood`** — loads all `Entity` nodes and fetches their edges and neighbor nodes.
2. **`generate_consolidated_entities`** — sends each entity plus neighborhood to the LLM, which returns a refined description.
3. **`add_data_points`** — writes the updated `Entity` objects back to the graph and vector DB.
</Accordion>
<Accordion title="Legacy guide">
```python theme={null}
import asyncio
import cognee
from cognee.memify_pipelines.consolidate_entity_descriptions import (
consolidate_entity_descriptions_pipeline,
)
async def main():
await cognee.prune.prune_data()
await cognee.prune.prune_system(metadata=True)
await cognee.add(
[
"Alice moved to Paris in 2010, while Bob has always lived in New York.",
"Andreas was born in Venice, but later settled in Lisbon.",
"Diana and Tom were born and raised in Helsinki. Diana currently resides in Berlin, while Tom never moved.",
]
)
await cognee.cognify()
await consolidate_entity_descriptions_pipeline()
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
<Accordion title="Troubleshooting">
* **No entities found** — the graph must already contain `Entity` nodes. Run `cognee.remember()` first.
* **LLM errors** — verify that your LLM provider is configured. See [LLM Providers](/setup-configuration/llm-providers).
* **Permission errors** — the user must have write access to the target dataset. See [Permissions](/core-concepts/multi-user-mode/permissions-system/datasets).
</Accordion>
<Columns>
<Card title="Entity Deduplication" icon="merge" href="/guides/memify-entity-deduplication">
Merge near-duplicate entity nodes into one canonical node
</Card>
<Card title="Improve" icon="sparkles" href="/core-concepts/main-operations/improve">
Understand the current improvement workflow
</Card>
<Card title="Self-Improvement Quickstart" icon="brain" href="/guides/self-improvement-quickstart">
Bridge session memory and enrich a dataset
</Card>
<Card title="Search" icon="search" href="/core-concepts/main-operations/legacy-operations/search">
Query the enriched graph with specialized search types
</Card>
</Columns>
# Entity Deduplication
Source: https://docs.cognee.ai/guides/memify-entity-deduplication
Detect near-duplicate entity nodes and merge each group into a single canonical node
A minimal guide to merging near-duplicate `Entity` nodes. When the same real-world thing is extracted under several names (`New York City` and `NYC`), it becomes several `Entity` nodes and its edges are split across all of them — the `consolidate_entities` Memify pipeline detects those near-duplicates and collapses each group into a single canonical node.
<Warning>
This pipeline is **destructive**. Duplicate `Entity` nodes are deleted from the graph and their name embeddings are deleted from the vector store. Run it with `dry_run=True` first and back up your graph and vector stores before running it for real.
</Warning>
Deduplication is for entities that are the *same* thing under different names. For entities that are distinct but related — and should stay separate nodes — use the [cross-connect entities pipeline](/core-concepts/main-operations/legacy-operations/memify#built-in-pipelines) (`cross_connect_entities_pipeline`), which only adds inferred edges between existing `Entity` nodes and never creates, rewrites, or deletes a node. Note that its `dry_run` also defaults to `False`.
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — the script below builds a graph with `remember()`
* Ensure you have [Embedding Providers](/setup-configuration/embedding-providers) configured — detection embeds every entity name at run time
* Have an existing knowledge graph with `Entity` nodes, or let the script create one
* The acting user needs **write** access to the target dataset. See [Permissions](/core-concepts/multi-user-mode/permissions-system/datasets)
## Code in Action
```python theme={null}
import asyncio
from os import path
import cognee
from cognee.api.v1.visualize.visualize import visualize_graph
from cognee.memify_pipelines.consolidate_entities import consolidate_entities_pipeline
custom_prompt = """
Extract every place mentioned in the text as an entity, keeping the exact
surface form used in the text (so "NYC" stays "NYC").
Connect people to places with the relationship "visited".
Ignore all other entities.
"""
async def main():
# Prune data and system metadata before running, only if we want "fresh" state.
await cognee.forget(everything=True)
# Ingest the two texts separately: extracted together, the LLM resolves the
# abbreviation and emits a single entity, leaving nothing to merge.
await cognee.remember(
"Sara visited New York City last spring.",
custom_prompt=custom_prompt,
self_improvement=False,
)
await cognee.remember(
"Bob thinks NYC has the best bagels.",
custom_prompt=custom_prompt,
self_improvement=False,
)
await visualize_graph(
path.join(path.dirname(__file__), ".artifacts", "before_entity_deduplication.html")
)
# Preview the merge plan in the logs without touching the graph.
await consolidate_entities_pipeline(similarity_threshold=0.6, dry_run=True)
# Apply the merge for real.
await consolidate_entities_pipeline(similarity_threshold=0.6)
await visualize_graph(
path.join(path.dirname(__file__), ".artifacts", "after_entity_deduplication.html")
)
if __name__ == "__main__":
asyncio.run(main())
```
## What Just Happened
### Step 1: Build a Graph with Duplicates
```python theme={null}
await cognee.forget(everything=True)
await cognee.remember(
"Sara visited New York City last spring.",
custom_prompt=custom_prompt,
self_improvement=False,
)
await cognee.remember(
"Bob thinks NYC has the best bagels.",
custom_prompt=custom_prompt,
self_improvement=False,
)
await visualize_graph(
path.join(path.dirname(__file__), ".artifacts", "before_entity_deduplication.html")
)
```
Start from a clean state, then ingest two texts that mention the same city under different names — in two separate `remember()` calls. That separation matters: when the texts are ingested together, the LLM sees both mentions at once, resolves the abbreviation during extraction, and emits a single entity, leaving nothing to merge. Ingested independently, the graph ends up with separate `New York City` and `NYC` entities — visible in the before visualization — each holding only its own edges.
### Step 2: Preview the Merge Plan
```python theme={null}
await consolidate_entities_pipeline(similarity_threshold=0.6, dry_run=True)
```
`dry_run=True` runs detection and computes the full merge plan, then logs it — which clusters were found, which node becomes canonical, and how many edges would be re-pointed — without touching the graph. The threshold is lowered from the `0.85` default because an abbreviation like `NYC` is not embedding-close enough to `New York City` to clear it.
### Step 3: Apply the Merge
```python theme={null}
await consolidate_entities_pipeline(similarity_threshold=0.6)
await visualize_graph(
path.join(path.dirname(__file__), ".artifacts", "after_entity_deduplication.html")
)
```
Once the plan looks right, drop `dry_run` to execute it. The after visualization shows a single canonical city node carrying the edges from both duplicates.
## Options
`consolidate_entities_pipeline()` takes the following arguments:
| Argument | Type | Default | Description |
| ---------------------- | --------------------- | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `similarity_threshold` | `float` | `0.85` | Minimum cosine similarity between two entity-name embeddings for them to be treated as the same entity. Must be in the range `(0, 1]`. |
| `dry_run` | `bool` | `False` | When `True`, compute and log the merge plan but perform zero mutations. |
| `protect_node_types` | `Optional[List[str]]` | `None` | `EntityType` names that must never be merged. Must be a list of non-empty strings. |
| `name_match` | `bool` | `True` | Also merge entities whose normalized names are identical, where normalizing means lower-casing and stripping every non-alphanumeric character (so `U.S.A.` matches `USA`). This pass is not bounded by `top_k`. |
| `top_k` | `int` | `10` | Max neighbors considered per entity during similarity clustering. Must be a positive integer. |
| `allow_cross_type` | `bool` | `False` | When `False` (the default, conservative choice), only entities sharing the same `EntityType` are merged together. Set `True` to allow merging across types. |
| `user` | `Optional[User]` | `None` | Acting user. The default user is used when omitted. |
| `dataset` | `str` | `"main_dataset"` | Dataset name or id whose graph to consolidate. |
| `run_in_background` | `bool` | `False` | Forwarded to `memify`. |
Arguments are validated before anything runs. A `CogneeValidationError` is raised when `similarity_threshold` is outside `(0, 1]`, when `top_k` is not a positive integer, when `protect_node_types` is not a list of non-empty strings, or when the user has no write access to the requested dataset.
## What the Merge Changes
* Each detected cluster collapses into **one canonical `Entity` node**; the rest are deleted. The canonical is the most connected member of the cluster, with ties broken by oldest `created_at` and then by normalized name.
* The canonical **keeps its existing graph id**, so anything already pointing at it stays valid.
* Every edge on a duplicate is re-pointed onto the canonical with its direction preserved. Edges that would become a self-loop on the canonical are dropped.
* The canonical's `description` becomes the union of the distinct, non-empty descriptions across the cluster.
* The canonical's `belongs_to_set` becomes the union of every member's tags, so a merge never narrows a node's dataset / node-set scoping.
* Duplicate nodes are detach-deleted (which removes their old edges) and their name embeddings are purged from the `Entity_name` vector collection.
* A merge report listing each `canonical_id`, `canonical_name`, and the `merged_from` duplicates is written to the logs on every run, including dry runs.
## Additional Information
* Runnable guide script available on our [GitHub](https://github.com/topoteretes/cognee/blob/dev/examples/guides/entity_deduplication.py)
* Pipeline implementation: [consolidate\_entities.py](https://github.com/topoteretes/cognee/blob/dev/cognee/memify_pipelines/consolidate_entities.py)
<Accordion title="Tuning the matching">
Raise `similarity_threshold` to merge only very close names, exclude entity types you never want collapsed, and widen `top_k` if dense clusters of similar names are being missed.
```python theme={null}
await consolidate_entities_pipeline(
similarity_threshold=0.9,
protect_node_types=["Person"],
top_k=25,
dataset="my_dataset",
)
```
</Accordion>
<Accordion title="Under the hood">
Two tasks run in sequence, both exported from `cognee.tasks.memify`:
1. **`detect_entity_duplicates`** (extraction) — loads every `Entity` node from the graph, embeds its name, and clusters near-duplicates by cosine similarity and, when `name_match` is on, by normalized-name equality. Similarities are scanned in row-blocks and only each node's `top_k` nearest neighbors are inspected, so the full N×N similarity matrix is never materialized.
2. **`merge_entity_duplicates`** (enrichment) — picks the canonical for each cluster, re-points the duplicates' edges onto it in one batched call, writes back the canonical with unioned `description` and `belongs_to_set`, then deletes the duplicate nodes and their vectors.
The merge is backend-agnostic: it uses only `get_graph_data`, `add_edges`, `add_nodes`, `delete_nodes`, and the vector engine's `delete_data_points`. No per-edge delete primitive is required.
</Accordion>
<Accordion title="Troubleshooting">
* **Nothing was merged** — the graph must contain at least two candidate `Entity` nodes. Check the logs for the detected-cluster count, then lower `similarity_threshold` or raise `top_k`.
* **Entities that should match are not merging** — by default only entities sharing the same `EntityType` are merged. Set `allow_cross_type=True` if the duplicates were typed differently.
* **Too much was merged** — restore from backup, then raise `similarity_threshold` and add the affected types to `protect_node_types`.
* **Validation errors** — `similarity_threshold` must be in `(0, 1]`, `top_k` must be a positive integer, and `protect_node_types` must be a list of non-empty strings.
* **Permission errors** — the user must have write access to the target dataset. See [Permissions](/core-concepts/multi-user-mode/permissions-system/datasets).
</Accordion>
<Columns>
<Card title="Entity Consolidation" icon="sparkles" href="/guides/memify-entity-consolidation">
Rewrite fragmented entity descriptions with the LLM
</Card>
<Card title="Improve" icon="sparkles" href="/core-concepts/main-operations/improve">
Understand the current improvement workflow
</Card>
<Card title="Embedding Providers" icon="layers" href="/setup-configuration/embedding-providers">
Configure the embedder that duplicate detection relies on
</Card>
</Columns>
# Session Persistence
Source: https://docs.cognee.ai/guides/memify-session-persistence
Persist cached conversation sessions into the knowledge graph
## When to use this
You want to persist cached conversation sessions into the knowledge graph so the Q\&A history becomes part of the searchable graph. In the current API, the user-facing way to do this is `cognee.improve(dataset=..., session_ids=[...])`.
Use the lower-level Memify pipeline in this guide when you need direct control over only the session persistence step.
**Before you start:**
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured
* Have an existing knowledge graph created with `remember()`
* [Caching must be enabled](/core-concepts/sessions-and-caching#cache-adapters) and at least one session must exist (created by prior `cognee.recall()` calls with a `session_id`)
## Code in Action
```python theme={null}
import asyncio
import cognee
from cognee import SearchType
from cognee.memify_pipelines.persist_sessions_in_knowledge_graph import (
persist_sessions_in_knowledge_graph_pipeline,
)
from cognee.modules.users.methods import get_default_user
async def main():
await cognee.remember(
"Alice moved to Paris in 2010. She works as a software engineer.",
dataset_name="session_demo",
self_improvement=False,
)
# Build session history with recall
await cognee.recall(
query_type=SearchType.GRAPH_COMPLETION,
query_text="Where does Alice live?",
datasets=["session_demo"],
session_id="demo_session",
)
await cognee.recall(
query_type=SearchType.GRAPH_COMPLETION,
query_text="What does she do for work?",
datasets=["session_demo"],
session_id="demo_session",
)
# Persist the session into the graph
user = await get_default_user()
await persist_sessions_in_knowledge_graph_pipeline(
user=user,
session_ids=["demo_session"],
dataset="session_demo",
)
asyncio.run(main())
```
## What Just Happened
1. **Remember** — builds a knowledge graph from your text.
2. **Recall with `session_id`** — runs two recalls that accumulate Q\&A history in the session cache under `"demo_session"`.
3. **`get_default_user()`** — retrieves the authenticated user. This pipeline requires a `User` object with write access.
4. **`persist_sessions_in_knowledge_graph_pipeline(user, session_ids, dataset)`** — reads the cached session data and writes it into the knowledge graph.
## What Changed in Your Graph
* New nodes are created from the session Q\&A history, grouped under the `user_sessions_from_cache` node set.
* The session data is processed internally into graph memory, so entities and relationships from the session content become part of the graph.
* Persistence is incremental: each run only ingests the Q\&A entries added since the last successful persist for that session. Re-running on an unchanged session adds nothing new, so you can safely call it repeatedly as a session grows without re-embedding or duplicating the earlier history.
## Additional Information
You can find the runnable session persistence demo on our [GitHub](https://github.com/topoteretes/cognee/blob/dev/examples/advanced_guides/conversation_session_persistence_example.py). It builds two conversation sessions with `recall()`, persists both into the knowledge graph in one pipeline run, and renders a visualization of the resulting graph.
<Accordion title="Full example script">
```python theme={null}
import asyncio
import os
import cognee
from cognee import SearchType, visualize_graph
from cognee.memify_pipelines.persist_sessions_in_knowledge_graph import (
persist_sessions_in_knowledge_graph_pipeline,
)
from cognee.modules.users.methods import get_default_user
from cognee.shared.logging_utils import get_logger
logger = get_logger("conversation_session_persistence_example")
async def main():
# NOTE: CACHING has to be enabled for this example to work
await cognee.forget(everything=True)
text_1 = "Cognee is a solution that can build knowledge graph from text, creating an AI memory system"
text_2 = "Germany is a country located next to the Netherlands"
await cognee.remember([text_1, text_2], self_improvement=False)
question = "What can I use to create a knowledge graph?"
search_results = await cognee.recall(
query_type=SearchType.GRAPH_COMPLETION,
query_text=question,
session_id="first_session",
)
print("\nSession ID: first_session")
print(f"Question: {question}")
print(f"Answer: {search_results}\n")
question = "You sure about that?"
search_results = await cognee.recall(
query_type=SearchType.GRAPH_COMPLETION,
query_text=question,
session_id="first_session",
)
print("\nSession ID: first_session")
print(f"Question: {question}")
print(f"Answer: {search_results}\n")
question = "This is awesome!"
search_results = await cognee.recall(
query_type=SearchType.GRAPH_COMPLETION,
query_text=question,
session_id="first_session",
)
print("\nSession ID: first_session")
print(f"Question: {question}")
print(f"Answer: {search_results}\n")
question = "Where is Germany?"
search_results = await cognee.recall(
query_type=SearchType.GRAPH_COMPLETION,
query_text=question,
session_id="different_session",
)
print("\nSession ID: different_session")
print(f"Question: {question}")
print(f"Answer: {search_results}\n")
question = "Next to which country again?"
search_results = await cognee.recall(
query_type=SearchType.GRAPH_COMPLETION,
query_text=question,
session_id="different_session",
)
print("\nSession ID: different_session")
print(f"Question: {question}")
print(f"Answer: {search_results}\n")
question = "So you remember everything I asked from you?"
search_results = await cognee.recall(
query_type=SearchType.GRAPH_COMPLETION,
query_text=question,
session_id="different_session",
)
print("\nSession ID: different_session")
print(f"Question: {question}")
print(f"Answer: {search_results}\n")
session_ids_to_persist = ["first_session", "different_session"]
default_user = await get_default_user()
await persist_sessions_in_knowledge_graph_pipeline(
user=default_user,
session_ids=session_ids_to_persist,
)
visualize_graph_path = os.path.join(
os.path.dirname(__file__), ".artifacts/conversation_session_persistence.html"
)
await visualize_graph(visualize_graph_path)
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
<Accordion title="Parameters">
* **`user`** (`User`, required) — authenticated user with write access. Obtain via `await get_default_user()`.
* **`session_ids`** (`Optional[List[str]]`) — list of session IDs to persist. If `None`, no sessions are extracted.
* **`dataset`** (`str`, default: `"main_dataset"`) — the dataset to write session data into.
* **`run_in_background`** (`bool`, default: `False`) — run asynchronously and return immediately.
</Accordion>
<Accordion title="Under the hood">
Two tasks run in sequence, coordinated by a per-`(user, session)` persist watermark:
1. **`extract_user_sessions`** — reads Q\&A data from the `SessionManager` for the specified `session_ids`, then consults the persist watermark (the number of Q\&A entries already persisted for that session) and yields only the entries added since the last successful run. A session with no new entries yields nothing, so re-running on an unchanged session does no ingestion work.
2. **`cognify_session`** — calls `cognee.add` and `cognee.cognify` on the extracted entries, writing the results into the graph under the `user_sessions_from_cache` node set. Only after both succeed does it advance that session's watermark. If cognify fails, the watermark is left untouched and the same entries are retried on the next run (add-level content-hash deduplication keeps the retry safe).
The watermark is stored as an internal session-context row via the `SessionManager` — it reuses the existing session cache and introduces no new backend, configuration, or credentials.
</Accordion>
<Accordion title="Troubleshooting">
* **No sessions found** — caching must be enabled and `recall()` with a `session_id` must have been run first. See [Sessions and Caching](/core-concepts/sessions-and-caching).
* **Error: no graph data found** — run `cognee.remember(..., dataset_name=...)` before calling this pipeline.
* **LLM errors** — verify that your LLM provider is configured. See [LLM Providers](/setup-configuration/llm-providers).
* **Permission errors** — the user must have write access to the target dataset. See [Permissions](/core-concepts/multi-user-mode/permissions-system/datasets).
</Accordion>
<Columns>
<Card title="Improve" icon="sparkles" href="/core-concepts/main-operations/improve">
Bridge sessions with the current improvement workflow
</Card>
<Card title="Sessions Guide" icon="message-circle" href="/guides/sessions">
Learn how sessions and caching work in Cognee
</Card>
<Card title="Recall" icon="search" href="/core-concepts/main-operations/recall">
Query the enriched graph with v1.0 retrieval
</Card>
</Columns>
# Triplet Embeddings
Source: https://docs.cognee.ai/guides/memify-triplet-embeddings
Index graph triplets as embeddings to enable TRIPLET_COMPLETION search
## When to use this
You need `SearchType.TRIPLET_COMPLETION` with `recall()` to return results. This search type matches queries against text representations of graph triplets (source → relationship → target).
Most users should start with `improve()`, which runs Cognee's default enrichment pass. Use this lower-level Memify pipeline directly when you specifically want to build or rebuild triplet embeddings.
**Before you start:**
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured
* Have an existing knowledge graph created with `remember()`
## Code in Action
```python theme={null}
import asyncio
import cognee
from cognee import SearchType
from cognee.memify_pipelines.create_triplet_embeddings import create_triplet_embeddings
from cognee.modules.users.methods import get_default_user
DATASET = "triplet_demo"
async def main():
await cognee.remember(
"GraphRAG combines vector search with graph traversal for better context.",
dataset_name=DATASET,
self_improvement=False,
)
user = await get_default_user()
await create_triplet_embeddings(user=user, dataset=DATASET)
results = await cognee.recall(
query_type=SearchType.TRIPLET_COMPLETION,
query_text="How does GraphRAG work?",
datasets=[DATASET],
)
for result in results:
print(result)
asyncio.run(main())
```
## What Just Happened
1. **Remember** — builds a knowledge graph with entities and relationships from your text.
2. **`get_default_user()`** — retrieves the authenticated user. This pipeline requires a `User` object with write access to the dataset.
3. **`create_triplet_embeddings(user, dataset)`** — iterates over all graph triplets, converts each to a text representation, and indexes them in the vector DB.
4. **Recall with `TRIPLET_COMPLETION`** — queries the new `Triplet_text` collection by semantic similarity.
## What Changed in Your Graph
* The `Triplet_text` collection is populated in the vector DB. Each entry is a text representation of a graph triplet (source → relationship → target).
* `recall(..., query_type=SearchType.TRIPLET_COMPLETION)` queries now return results by matching your query against these triplet embeddings.
## Additional Information
<Accordion title="Parameters">
* **`user`** (`User`, required) — authenticated user with write access. Obtain via `await get_default_user()`.
* **`dataset`** (`str`, default: `"main_dataset"`) — the dataset whose graph triplets to index.
* **`run_in_background`** (`bool`, default: `False`) — run asynchronously and return immediately.
* **`triplets_batch_size`** (`int`, default: `100`) — how many triplets to index per batch. Lower values use less memory; higher values are faster. The value does not affect coverage: triplets are paginated in a stable order, so a full pass visits each triplet exactly once at any batch size.
</Accordion>
<Accordion title="Under the hood">
Two tasks run in sequence:
1. **`get_triplet_datapoints`** — iterates over graph triplets and yields `Triplet` objects with embeddable text.
2. **`index_data_points`** — indexes each triplet in the vector DB under the `Triplet_text` collection.
`get_triplet_datapoints` walks the whole graph with an offset loop, calling the graph adapter's `get_triplets_batch(offset, limit)` with `limit=triplets_batch_size` and advancing the offset until a batch comes back short or empty. Every built-in adapter that supports batched reads sorts the triplets on `(source node id, target node id, relationship name)` *before* applying the offset, so consecutive calls slice one stable sequence and the loop cannot skip or repeat a triplet. Changing `triplets_batch_size` changes how the triplets are grouped into batches, not which triplets are visited.
</Accordion>
<Accordion title="Troubleshooting">
* **Empty results from `TRIPLET_COMPLETION`** — ensure the graph has been built with `remember()` and that `create_triplet_embeddings` finished without errors.
* **Some triplets missing from `Triplet_text` on Neo4j or Ladybug** — on releases before the pagination fix, those two adapters applied `SKIP`/`LIMIT` to an unordered result set, so a multi-batch run could skip triplets and leave `Triplet_text` incomplete even though the pipeline reported success. Upgrade and re-run `create_triplet_embeddings` to index the missed triplets.
* **Error: no graph data found** — run `cognee.remember(..., dataset_name=...)` before calling this pipeline.
* **LLM errors** — verify that your LLM provider is configured. See [LLM Providers](/setup-configuration/llm-providers).
* **Permission errors** — the user must have write access to the target dataset. See [Permissions](/core-concepts/multi-user-mode/permissions-system/datasets).
</Accordion>
<Columns>
<Card title="Improve" icon="sparkles" href="/core-concepts/main-operations/improve">
Understand the current improvement workflow
</Card>
<Card title="Self-Improvement Quickstart" icon="brain" href="/guides/self-improvement-quickstart">
Bridge session memory and enrich a dataset
</Card>
<Card title="Recall" icon="search" href="/core-concepts/main-operations/recall">
Query the enriched graph with v1.0 retrieval
</Card>
</Columns>
# Memory Provenance
Source: https://docs.cognee.ai/guides/memory-provenance
Visualize the ownership and data-flow story behind your memory — tenants, users, agents, datasets, and files
`visualize_memory_provenance()` renders the *ownership and data-flow* story behind your memory — Tenant → User → Agent → Dataset → file, plus agent read/write access and agent-written sessions — to a self-contained HTML file.
**Before you start:**
* Complete [Quickstart](getting-started/quickstart) to understand basic operations
* Have some ingested data (the projection reads the relational database, so no graph backend is required)
## Code in Action
```python theme={null}
import os
import cognee
# Data must be remembered/loaded into cognee before projecting provenance,
# e.g. via cognee.remember() or cognee.add() + cognee.cognify().
dest = os.path.join(os.path.expanduser("~"), "memory_provenance.html")
await cognee.visualize_memory_provenance(
destination_file_path=dest,
include_memory=False, # when True, fold in extracted entities/relationships
)
```
This projection is read **purely from the relational database** — it does not require the graph backend or an LLM, so it works even when the graph database is unavailable. Set `include_memory=True` to also fold in the extracted entities/relationships (from the relational `nodes`/`edges` tables) and link them back to the files they were extracted from.
## Raw graph data
If you only need the projected graph data (in the same `(nodes, edges)` shape the renderer consumes) rather than HTML, call `get_memory_provenance_graph()`:
```python theme={null}
nodes, edges = await cognee.get_memory_provenance_graph(include_memory=True)
```
## Scoping in multi-tenant deployments
Both functions accept `scope_tenant_ids` and `scope_user_ids` to restrict the projection to a tenant or user. In **multi-tenant deployments you must pass a scope** — an unscoped read returns every tenant's actors, datasets, and files. When neither scope is given the read is global, which is the intended behavior only for single-user / OSS installs where one user owns everything.
## Over HTTP
The HTTP endpoint `GET /api/v1/schema/provenance` (query param `include_memory`) always scopes to the authenticated caller's tenant (or user).
## Full Example
A richer guide script ingests two documents into two datasets, then walks the ownership chain down to the `mentions` edges that tie each memory node back to the file it came from, prints it as a tree, and renders the same projection to a standalone HTML file:
* [`examples/guides/memory_provenance.py`](https://github.com/topoteretes/cognee/blob/dev/examples/guides/memory_provenance.py) → `.artifacts/memory_provenance.html`
The complete flow — remember data, render the provenance HTML, and read the raw projection is in the following code:
<Accordion title="Memory provenance">
```python theme={null}
import asyncio
import os
import cognee
async def main():
# Prune data and system metadata before running, only if we want "fresh" state.
await cognee.forget(everything=True)
# Data must be remembered/loaded into cognee before projecting provenance.
await cognee.remember(
["Alice knows Bob.", "NLP is a subfield of CS."],
self_improvement=False,
)
dest = os.path.join(
os.path.dirname(__file__), ".artifacts", "memory_provenance.html"
)
await cognee.visualize_memory_provenance(destination_file_path=dest)
# Raw (nodes, edges) projection, folding in the extracted memory.
nodes, edges = await cognee.get_memory_provenance_graph(include_memory=True)
print(f"provenance nodes={len(nodes)}, edges={len(edges)}")
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
<Columns>
<Card title="Graph Visualization" icon="network" href="/guides/graph-visualization">
Render your knowledge graph to an interactive HTML file
</Card>
<Card title="Schema Inventory" icon="table" href="/guides/schema-inventory">
Summarize the knowledge graph by semantic type
</Card>
</Columns>
# Multilingual Ingestion
Source: https://docs.cognee.ai/guides/multilingual-ingestion
Translate non-English content before building the knowledge graph
A minimal guide to enabling translation during ingestion. Cognee includes a built-in translation pipeline that detects languages and translates content before graph extraction, so non-English documents are indexed as English knowledge.
**Before you start:**
* Complete [Quickstart](getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](setup-configuration/llm-providers) configured
* Have non-English text or documents to process
## What Translation Does
* Detects language automatically using the `langdetect` library
* Skips chunks already in the target language
* Translates using one of three providers: `llm` (default), `google`, or `azure`
* Stores original text alongside the translation in the knowledge graph
## Configuration
Set these environment variables in your `.env` file:
```dotenv theme={null}
# Provider: "llm" (default), "google", or "azure"
TRANSLATION_PROVIDER=llm
# Target language ISO 639-1 code (default: "en")
TARGET_LANGUAGE=en
# Minimum detection confidence to trigger translation (default: 0.8)
CONFIDENCE_THRESHOLD=0.8
```
The `llm` provider uses your existing [LLM configuration](setup-configuration/llm-providers) — no additional keys needed.
## Using Translation in a Pipeline
Insert `translate_content` as a pipeline task between chunk extraction and graph building:
```python theme={null}
import asyncio
import os
import cognee
from cognee.infrastructure.llm import get_max_chunk_tokens
from cognee.tasks.documents import classify_documents, extract_chunks_from_documents
from cognee.shared.data_models import KnowledgeGraph
from cognee.tasks.translation import translate_content
from cognee.modules.pipelines import Task, run_pipeline
from cognee.tasks.graph import extract_graph_from_data
from cognee.tasks.storage import add_data_points
# the translated text is in data_chunks[].text,
async def drop_translation_metadata(data_chunks):
for chunk in data_chunks:
chunk.contains = None
return data_chunks
async def main():
await cognee.prune.prune_data()
await cognee.prune.prune_system(metadata=True)
text_fr = "La mémoire artificielle permet aux agents IA de retenir des informations complexes."
tasks = [
Task(classify_documents),
Task(extract_chunks_from_documents, max_chunk_size=get_max_chunk_tokens()),
Task(translate_content, target_language="en", translation_provider="llm"),
Task(drop_translation_metadata),
Task(extract_graph_from_data, graph_model=KnowledgeGraph),
Task(add_data_points),
]
async for _ in run_pipeline(tasks=tasks, datasets=["multilingual"]):
pass
visualize_graph_path = os.path.join(
os.path.dirname(__file__), ".artifacts", "multilingual.html"
)
await cognee.visualize_graph(visualize_graph_path)
asyncio.run(main())
```
<Note>
`translate_content` mutates chunks in-place: `chunk.text` is replaced with the translation and the original is preserved in a `TranslatedContent` data point attached to the chunk.
</Note>
## Additional Information
<AccordionGroup>
<Accordion title="How Language Detection Works">
Cognee detects language **per chunk** with the [`langdetect`](https://pypi.org/project/langdetect/) library. Each chunk produced by the chunker is analyzed independently, so a document that mixes languages has every chunk detected — and translated — on its own.
A chunk is translated only when **both** conditions hold: the detected language differs from `TARGET_LANGUAGE`, and the detection confidence is at least `CONFIDENCE_THRESHOLD` (default `0.8`). Otherwise the chunk is left untouched and only tagged with `LanguageMetadata`. Chunks shorter than 10 characters skip detection entirely.
`langdetect` recognizes 55 languages:
`af`, `ar`, `bg`, `bn`, `ca`, `cs`, `cy`, `da`, `de`, `el`, `en`, `es`, `et`, `fa`, `fi`, `fr`, `gu`, `he`, `hi`, `hr`, `hu`, `id`, `it`, `ja`, `kn`, `ko`, `lt`, `lv`, `mk`, `ml`, `mr`, `ne`, `nl`, `no`, `pa`, `pl`, `pt`, `ro`, `ru`, `sk`, `sl`, `so`, `sq`, `sv`, `sw`, `ta`, `te`, `th`, `tl`, `tr`, `uk`, `ur`, `vi`, `zh-cn`, `zh-tw`
Languages outside this set — for example Azerbaijani (`az`) — cannot be detected. `langdetect` either misclassifies them as a related language (Azerbaijani is often read as Turkish, `tr`) or returns low confidence, so such chunks may be skipped or translated from the wrong source language. Detection drives translation off the *detected* code, not the document's true language, so verify coverage before relying on it for an unsupported language.
</Accordion>
<Accordion title="Translating Individual Strings">
For one-off translation without a pipeline, use `translate_text`:
```python theme={null}
from cognee.tasks.translation import translate_text
result = await translate_text("Bonjour le monde!", target_language="en")
print(result.translated_text) # "Hello world!"
print(result.source_language) # "fr"
```
</Accordion>
<Accordion title="Choosing a Provider">
All three providers translate non-English chunks to your `TARGET_LANGUAGE`. Pick based on cost, setup, and quality trade-offs:
| Provider | Setup | Cost | Best for |
| --------------- | ------------------------------------------------------------------- | --------------------------------------------- | ----------------------------------------------------------------- |
| `llm` (default) | None — reuses your [LLM config](/setup-configuration/llm-providers) | Per-token LLM usage; higher quality, slower | Mixed/long-form documents where context-aware translation matters |
| `google` | Install `google-cloud-translate`, Google Cloud project | Per-character pricing; fast batch translation | High-volume ingestion across many languages |
| `azure` | Azure Cognitive Services key + region | Per-character pricing; fast batch translation | Enterprise deployments already on Azure |
**Supported languages:** detection uses `langdetect` (\~55 languages). The `llm` provider supports any language the underlying model handles. Google Translate and Azure Translator each support 130+ language codes, including locale-specific variants such as `zh-CN` and `zh-TW` — see the [Google Cloud Translation language list](https://cloud.google.com/translate/docs/languages) and [Azure Translator language list](https://learn.microsoft.com/azure/ai-services/translator/language-support) for the full set.
Set `TRANSLATION_PROVIDER` in `.env` to switch — no code changes required.
</Accordion>
<Accordion title="Provider-Specific Setup">
<AccordionGroup>
<Accordion title="LLM Provider (default)">
Uses your existing LLM — no extra configuration needed. Works with any provider configured via `LLM_PROVIDER` and `LLM_API_KEY`.
```dotenv theme={null}
TRANSLATION_PROVIDER=llm
```
</Accordion>
<Accordion title="Google Cloud Translation">
Requires the `google-cloud-translate` package and a Google Cloud project.
```bash theme={null}
pip install google-cloud-translate
```
```dotenv theme={null}
TRANSLATION_PROVIDER=google
GOOGLE_TRANSLATE_API_KEY=your_api_key
GOOGLE_PROJECT_ID=your_project_id
```
</Accordion>
<Accordion title="Azure Translator">
Requires an Azure Cognitive Services resource.
```dotenv theme={null}
TRANSLATION_PROVIDER=azure
AZURE_TRANSLATOR_KEY=your_key
AZURE_TRANSLATOR_REGION=eastus
# Endpoint defaults to https://api.cognitive.microsofttranslator.com
AZURE_TRANSLATOR_ENDPOINT=https://api.cognitive.microsofttranslator.com
```
</Accordion>
</AccordionGroup>
</Accordion>
<Accordion title="Advanced Options">
| Variable | Default | Description |
| ----------------------------- | ------- | ---------------------------- |
| `TRANSLATION_BATCH_SIZE` | `10` | Chunks per translation batch |
| `TRANSLATION_MAX_RETRIES` | `3` | Retry attempts on failure |
| `TRANSLATION_TIMEOUT_SECONDS` | `30` | Request timeout |
</Accordion>
</AccordionGroup>
<Columns>
<Card title="Custom Pipelines" icon="workflow" href="/guides/custom-tasks-pipelines">
Learn to build custom task pipelines
</Card>
<Card title="LLM Providers" icon="cpu" href="/setup-configuration/llm-providers">
Configure your LLM provider
</Card>
<Card title="Core Concepts" icon="brain" href="/core-concepts/overview">
Understand knowledge graph fundamentals
</Card>
</Columns>
# Multimedia Processing
Source: https://docs.cognee.ai/guides/multimedia-audio-image-processing
Build a knowledge graph from audio and image files and query their summaries
A minimal guide to ingesting non-text files — an audio recording and an image — into cognee. Use it when the knowledge you need lives in recordings, screenshots, charts, or scans rather than in documents.
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — the audio file needs a provider with a transcription endpoint, and the image needs a vision model
* Read [Loaders](/core-concepts/further-concepts/loaders) for how cognee turns audio and images into the text it then processes
* Have `text_to_speech.mp3` and `example.png` in a `multimedia_audio_image_processing_example_data/` directory next to the script — you can download the [example media files](https://github.com/topoteretes/cognee/tree/dev/examples/guides/multimedia_audio_image_processing_example_data) from the cognee GitHub repository
## Code in Action
```python theme={null}
import asyncio
import os
import pathlib
import cognee
from cognee import SearchType
from cognee.shared.logging_utils import ERROR, setup_logging
async def main():
# Create a clean slate for cognee -- reset data and system state
await cognee.forget(everything=True)
# cognee knowledge graph will be created based on the text
# and description of these files
mp3_file_path = os.path.join(
pathlib.Path(__file__).parent,
"multimedia_audio_image_processing_example_data/text_to_speech.mp3",
)
png_file_path = os.path.join(
pathlib.Path(__file__).parent,
"multimedia_audio_image_processing_example_data/example.png",
)
# Remember the files and create knowledge graph memory
await cognee.remember([mp3_file_path, png_file_path], self_improvement=False)
# Query cognee for summaries of the data in the multimedia files
search_results = await cognee.recall(
query_type=SearchType.SUMMARIES,
query_text="What is in the multimedia files?",
)
# Display search results
for result_text in search_results:
print(result_text)
if __name__ == "__main__":
logger = setup_logging(log_level=ERROR)
asyncio.run(main())
```
## What Just Happened
### Step 1: Locate the Media Files
```python theme={null}
# Create a clean slate for cognee -- reset data and system state
await cognee.forget(everything=True)
# cognee knowledge graph will be created based on the text
# and description of these files
mp3_file_path = os.path.join(
pathlib.Path(__file__).parent,
"multimedia_audio_image_processing_example_data/text_to_speech.mp3",
)
png_file_path = os.path.join(
pathlib.Path(__file__).parent,
"multimedia_audio_image_processing_example_data/example.png",
)
```
Forgetting everything first means the summaries you read at the end come only from this run. The two files are resolved relative to the script itself, inside its `multimedia_audio_image_processing_example_data/` directory — swap in your own paths to ingest different media.
### Step 2: Remember the Audio and the Image
```python theme={null}
# Remember the files and create knowledge graph memory
await cognee.remember([mp3_file_path, png_file_path], self_improvement=False)
```
`remember()` picks a loader per file from its type: the `.mp3` goes to the audio loader for transcription, the `.png` to the image loader for vision transcription. From there both are ordinary text, so the same chunking, extraction, and summarization steps build one graph across the two files.
### Step 3: Recall the Summaries
```python theme={null}
# Query cognee for summaries of the data in the multimedia files
search_results = await cognee.recall(
query_type=SearchType.SUMMARIES,
query_text="What is in the multimedia files?",
)
# Display search results
for result_text in search_results:
print(result_text)
```
`SearchType.SUMMARIES` returns the summaries generated during ingestion rather than asking an LLM to answer the question, which makes it a direct way to see what cognee understood from each file. Everything printed here comes from the transcriptions, so it is also the quickest check that the audio and image were read correctly.
## Advanced Usage
<AccordionGroup>
<Accordion title="Get More Out of Images">
Images are transcribed with an extraction-oriented prompt that asks for entities, their attributes, relationships, and any visible text — set `IMAGE_EXTRACTION_ENABLED="false"` to fall back to a short caption instead. With `IMAGE_OCR_ENABLED="true"` and `pip install cognee[rapidocr]`, a local OCR pass appends the text it recognizes to the transcription, which helps on dense screenshots and scans. See [Loaders](/core-concepts/further-concepts/loaders) for every image setting and its default.
</Accordion>
<Accordion title="Other Media Formats">
The same call handles the rest of the audio extensions (`.wav`, `.flac`, `.m4a`, and more) and the image formats the vision loader claims. Video files take the video loader, which transcribes the audio track with inline `[HH:MM:SS]` timestamps; `.mp4` and `.webm` work as-is, other containers need `ffmpeg` on your `PATH`.
</Accordion>
</AccordionGroup>
<Columns>
<Card title="Loaders" icon="download" href="/core-concepts/further-concepts/loaders">
How each file type is turned into text before ingestion.
</Card>
<Card title="remember()" icon="brain" href="/python-api/remember">
The full parameter surface of the ingestion call this guide uses.
</Card>
<Card title="SearchType" icon="list" href="/python-api/search-type">
Every search mode, including when to prefer SUMMARIES.
</Card>
</Columns>
# Neptune Analytics
Source: https://docs.cognee.ai/guides/neptune-analytics
Store your knowledge graph and its embeddings in Amazon Neptune Analytics and search them with recall
A minimal guide to using Amazon Neptune Analytics as cognee's graph **and** vector store. It needs an AWS account rather than a local server, and one Neptune Analytics graph holds both the entities and their embeddings — so there is no separate vector database to provision.
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured (`LLM_API_KEY` in `.env`)
* Provision a Neptune Analytics graph in your AWS account ([AWS instructions](https://docs.aws.amazon.com/neptune-analytics/latest/userguide/create-graph-using-console.html)), and give it a vector search dimension matching your [embedding model's](/setup-configuration/embedding-providers) dimension
* Install the Neptune extra: `uv pip install "cognee[neptune]"`
* Make AWS credentials authorized for that graph available to the standard AWS SDK chain — environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`, plus `AWS_SESSION_TOKEN` for temporary credentials), a shared profile, or an instance role. `load_dotenv()` makes `.env` values work as environment variables
* Set `GRAPH_ID` in `.env` to your graph's identifier; the script turns it into a `neptune-graph://<GRAPH_ID>` endpoint
* Read [Graph Stores](/setup-configuration/graph-stores) and [Vector Stores](/setup-configuration/vector-stores) for the rest of the backend settings
## Code in Action
```python theme={null}
import asyncio
import os
import pathlib
from dotenv import load_dotenv
import cognee
from cognee import SearchType
load_dotenv()
async def main():
# Set up Amazon credentials in .env file and get the values from environment variables
graph_endpoint_url = "neptune-graph://" + os.getenv("GRAPH_ID", "")
# Configure Neptune Analytics as the graph & vector database provider
cognee.config.set_graph_db_config(
{
"graph_database_provider": "neptune_analytics", # Specify Neptune Analytics as provider
"graph_database_url": graph_endpoint_url, # Neptune Analytics endpoint with the format neptune-graph://<GRAPH_ID>
}
)
cognee.config.set_vector_db_config(
{
"vector_db_provider": "neptune_analytics", # Specify Neptune Analytics as provider
"vector_db_url": graph_endpoint_url, # Neptune Analytics endpoint with the format neptune-graph://<GRAPH_ID>
}
)
# Set up data directories for storing documents and system files
# You should adjust these paths to your needs
current_dir = pathlib.Path(__file__).parent
data_directory_path = str(current_dir / "data_storage")
cognee.config.data_root_directory(data_directory_path)
cognee_directory_path = str(current_dir / "cognee_system")
cognee.config.system_root_directory(cognee_directory_path)
# Clean any existing data (optional)
# await cognee.forget(everything=True)
# Create a dataset
dataset_name = "neptune_example"
# Add sample text to the dataset
sample_text_1 = """Neptune Analytics is a memory-optimized graph database engine for analytics. With Neptune
Analytics, you can get insights and find trends by processing large amounts of graph data in seconds. To analyze
graph data quickly and easily, Neptune Analytics stores large graph datasets in memory. It supports a library of
optimized graph analytic algorithms, low-latency graph queries, and vector search capabilities within graph
traversals.
"""
sample_text_2 = """Neptune Analytics is an ideal choice for investigatory, exploratory, or data-science workloads
that require fast iteration for data, analytical and algorithmic processing, or vector search on graph data. It
complements Amazon Neptune Database, a popular managed graph database. To perform intensive analysis, you can load
the data from a Neptune Database graph or snapshot into Neptune Analytics. You can also load graph data that's
stored in Amazon S3.
"""
# Remember the sample text in the dataset
await cognee.remember(
[sample_text_1, sample_text_2],
dataset_name=dataset_name,
self_improvement=False,
)
# Now let's perform some searches
# 1. Search for insights related to "Neptune Analytics"
insights_results = await cognee.recall(
query_type=SearchType.GRAPH_COMPLETION, query_text="Neptune Analytics"
)
print("\n========Insights about Neptune Analytics========:")
for result in insights_results:
print(f"- {result}")
# 2. Search for text chunks related to "graph database"
chunks_results = await cognee.recall(
query_type=SearchType.CHUNKS, query_text="graph database", datasets=[dataset_name]
)
print("\n========Chunks about graph database========:")
for result in chunks_results:
print(f"- {result}")
# 3. Get graph completion related to databases
graph_completion_results = await cognee.recall(
query_type=SearchType.GRAPH_COMPLETION, query_text="database"
)
print("\n========Graph completion for databases========:")
for result in graph_completion_results:
print(f"- {result}")
# Clean up (optional)
await cognee.forget(everything=True)
if __name__ == "__main__":
asyncio.run(main())
```
<Warning>
The `cognee.forget(everything=True)` call at the end wipes the configured graph. Do not point this script at a Neptune Analytics graph holding data you want to keep — or delete that line before running it.
</Warning>
## What Just Happened
### Step 1: Build the Graph Endpoint
```python theme={null}
# Set up Amazon credentials in .env file and get the values from environment variables
graph_endpoint_url = "neptune-graph://" + os.getenv("GRAPH_ID", "")
```
Cognee addresses a Neptune Analytics graph as `neptune-graph://<GRAPH_ID>`, so the only cloud-specific value the script needs is the graph identifier. Reading it from the environment keeps the identifier — and the AWS credentials the SDK picks up alongside it — out of the code.
### Step 2: Use One Graph as Both Stores
```python theme={null}
cognee.config.set_graph_db_config(
{
"graph_database_provider": "neptune_analytics", # Specify Neptune Analytics as provider
"graph_database_url": graph_endpoint_url, # Neptune Analytics endpoint with the format neptune-graph://<GRAPH_ID>
}
)
cognee.config.set_vector_db_config(
{
"vector_db_provider": "neptune_analytics", # Specify Neptune Analytics as provider
"vector_db_url": graph_endpoint_url, # Neptune Analytics endpoint with the format neptune-graph://<GRAPH_ID>
}
)
```
Neptune Analytics supports vector search inside graph traversals, so the same endpoint is registered as both the graph store and the vector store. Both calls take the same URL on purpose — entities, relationships, and embeddings all live in the one graph you provisioned.
### Step 3: Point Cognee at Local Directories
```python theme={null}
current_dir = pathlib.Path(__file__).parent
data_directory_path = str(current_dir / "data_storage")
cognee.config.data_root_directory(data_directory_path)
cognee_directory_path = str(current_dir / "cognee_system")
cognee.config.system_root_directory(cognee_directory_path)
```
The graph and the embeddings are remote, but cognee still keeps ingested documents and its relational metadata on disk. Setting both roots next to the script keeps this example's local files separate from your default cognee directories.
### Step 4: Remember the Sample Text
```python theme={null}
# Create a dataset
dataset_name = "neptune_example"
# Add sample text to the dataset
sample_text_1 = """Neptune Analytics is a memory-optimized graph database engine for analytics. With Neptune
Analytics, you can get insights and find trends by processing large amounts of graph data in seconds. To analyze
graph data quickly and easily, Neptune Analytics stores large graph datasets in memory. It supports a library of
optimized graph analytic algorithms, low-latency graph queries, and vector search capabilities within graph
traversals.
"""
sample_text_2 = """Neptune Analytics is an ideal choice for investigatory, exploratory, or data-science workloads
that require fast iteration for data, analytical and algorithmic processing, or vector search on graph data. It
complements Amazon Neptune Database, a popular managed graph database. To perform intensive analysis, you can load
the data from a Neptune Database graph or snapshot into Neptune Analytics. You can also load graph data that's
stored in Amazon S3.
"""
# Remember the sample text in the dataset
await cognee.remember(
[sample_text_1, sample_text_2],
dataset_name=dataset_name,
self_improvement=False,
)
```
`remember()` ingests both passages, extracts entities and relationships, and writes the resulting nodes, edges, and embeddings into your Neptune Analytics graph. `dataset_name` groups everything this call produces so a later search can be scoped to it.
### Step 5: Query the Graph and Its Vectors
```python theme={null}
# Now let's perform some searches
# 1. Search for insights related to "Neptune Analytics"
insights_results = await cognee.recall(
query_type=SearchType.GRAPH_COMPLETION, query_text="Neptune Analytics"
)
print("\n========Insights about Neptune Analytics========:")
for result in insights_results:
print(f"- {result}")
# 2. Search for text chunks related to "graph database"
chunks_results = await cognee.recall(
query_type=SearchType.CHUNKS, query_text="graph database", datasets=[dataset_name]
)
print("\n========Chunks about graph database========:")
for result in chunks_results:
print(f"- {result}")
```
`SearchType.GRAPH_COMPLETION` retrieves the triplets stored around the query and asks the LLM to answer from them, which exercises the graph side of the store. `SearchType.CHUNKS` returns the raw text chunks behind a query instead, exercising the vector side — together they confirm both halves of the Neptune Analytics backend are populated. `datasets=[dataset_name]` limits the second search to the dataset created above.
<Columns>
<Card title="Graph Stores" icon="network" href="/setup-configuration/graph-stores">
Every graph backend cognee supports, and their settings.
</Card>
<Card title="Vector Stores" icon="database" href="/setup-configuration/vector-stores">
Per-provider vector settings, including the Neptune Analytics block.
</Card>
<Card title="Store Configurations" icon="database" href="/guides/store-configurations">
Copy-paste `.env` blocks for the other supported store combinations, including the local default.
</Card>
</Columns>
# Recall Without an LLM Key
Source: https://docs.cognee.ai/guides/no-llm-remember-recall
Run remember and recall end to end with local GLiNER extraction and fastembed embeddings, with no LLM API key configured at all.
A minimal guide to running the full `remember` → `recall` round trip with no LLM API key anywhere: GLiNER2 extracts the graph and writes the chunk summaries, and fastembed embeds them on CPU. Use it when you have no key to spend, when the data cannot leave the machine, or when you want a smoke test that proves the pipeline itself does not depend on an LLM.
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Install the two extras this script needs — GLiNER for extraction, fastembed for embeddings:
```bash theme={null}
pip install "cognee[gliner]" "fastembed<=0.8.0"
```
* Allow for the first-run downloads: the GLiNER model is about 800 MB and the `bge-small-en-v1.5` embedding model about 130 MB, both cached after the first run
* No LLM API key is needed, and none is used — the script removes `LLM_API_KEY` and `OPENAI_API_KEY` from the environment before importing cognee
* Read [Local Setup (No API Key)](/guides/local-setup) for the `.env` form of local provider configuration, and [Embedding Providers](/setup-configuration/embedding-providers) to swap in a different fastembed model with its matching dimensions
## Code in Action
```python theme={null}
import asyncio
import os
# Make sure no key leaks in from the shell: the point is to prove the pipeline
# runs without one.
for var in ("LLM_API_KEY", "OPENAI_API_KEY"):
os.environ.pop(var, None)
os.environ.update(
{
"GRAPH_EXTRACTOR": "gliner",
"EMBEDDING_PROVIDER": "fastembed",
"EMBEDDING_MODEL": "BAAI/bge-small-en-v1.5",
"EMBEDDING_DIMENSIONS": "384",
"EMBEDDING_MAX_COMPLETION_TOKENS": "512",
# Per-turn feedback analysis is an LLM call; without it recall is LLM-free.
"AUTO_FEEDBACK": "false",
}
)
import cognee # noqa: E402 (environment must be set before the import)
from cognee import SearchType # noqa: E402
TEXT = (
"Marie Curie was born in Warsaw and worked at the University of Paris. "
"She won the Nobel Prize in Physics in 1903 with Pierre Curie and Henri Becquerel."
)
async def main():
await cognee.prune.prune_data()
await cognee.prune.prune_system(metadata=True)
# remember() runs add + cognify and then improve(). Without session_ids,
# improve() only runs the default enrichment (triplet/vector indexing) —
# embeddings, no LLM — so it is safe to leave self_improvement on.
await cognee.remember(TEXT, dataset_name="no_llm")
# No query_type: on the gliner backend this is CHUNKS.
results = await cognee.recall("Where was Marie Curie born?", datasets=["no_llm"], top_k=3)
print(f"\ndefault ({results[0].search_type}): {len(results)} result(s)")
for item in results:
print(" -", item.text.replace("\n", " | "))
# The GLiNER-built summaries are searchable too.
results = await cognee.recall(
"Where was Marie Curie born?",
query_type=SearchType.SUMMARIES,
datasets=["no_llm"],
top_k=3,
)
print(f"\nSUMMARIES: {len(results)} result(s)")
for item in results:
print(" -", item.text.replace("\n", " | "))
if __name__ == "__main__":
asyncio.run(main())
```
## What Just Happened
### Step 1: Clear Any Inherited API Key
```python theme={null}
# Make sure no key leaks in from the shell: the point is to prove the pipeline
# runs without one.
for var in ("LLM_API_KEY", "OPENAI_API_KEY"):
os.environ.pop(var, None)
```
Cognee reads its LLM key from the environment, so a key exported in your shell or picked up from a `.env` file would quietly make this run an ordinary LLM run. Popping both variables first is what makes the result meaningful: everything after this line has no LLM to call.
### Step 2: Select the Local Extractor and Embedder
```python theme={null}
os.environ.update(
{
"GRAPH_EXTRACTOR": "gliner",
"EMBEDDING_PROVIDER": "fastembed",
"EMBEDDING_MODEL": "BAAI/bge-small-en-v1.5",
"EMBEDDING_DIMENSIONS": "384",
"EMBEDDING_MAX_COMPLETION_TOKENS": "512",
# Per-turn feedback analysis is an LLM call; without it recall is LLM-free.
"AUTO_FEEDBACK": "false",
}
)
import cognee # noqa: E402 (environment must be set before the import)
from cognee import SearchType # noqa: E402
```
`GRAPH_EXTRACTOR=gliner` swaps cognify's LLM task list for the GLiNER one, which extracts entities and relationships and writes each chunk's summary locally instead of prompting a model. Both halves have to be local: leaving `EMBEDDING_PROVIDER` at its default would send embedding requests to OpenAI and fail without a key. `EMBEDDING_DIMENSIONS` and `EMBEDDING_MAX_COMPLETION_TOKENS` describe `bge-small-en-v1.5` — 384-dimensional vectors and a 512-token input limit — so cognee sizes its vector collections and chunks correctly. The whole block runs before `import cognee` because cognee reads this configuration at import time, hence the `# noqa: E402` markers on the imports.
### Step 3: Remember the Text Locally
```python theme={null}
await cognee.prune.prune_data()
await cognee.prune.prune_system(metadata=True)
# remember() runs add + cognify and then improve(). Without session_ids,
# improve() only runs the default enrichment (triplet/vector indexing) —
# embeddings, no LLM — so it is safe to leave self_improvement on.
await cognee.remember(TEXT, dataset_name="no_llm")
```
Pruning first clears any vector collections left over from a previous embedding model — their dimensions would not match the 384 configured above. `remember()` then ingests the text, builds the graph with GLiNER, and embeds the results with fastembed. Because no LLM task is in the pipeline, cognee also skips the first-run LLM connection probe, so the missing key never becomes an error.
### Step 4: Recall With the Default Search Type
```python theme={null}
# No query_type: on the gliner backend this is CHUNKS.
results = await cognee.recall("Where was Marie Curie born?", datasets=["no_llm"], top_k=3)
print(f"\ndefault ({results[0].search_type}): {len(results)} result(s)")
for item in results:
print(" -", item.text.replace("\n", " | "))
```
With no `query_type`, `recall()` normally answers with an LLM-written completion. When no usable LLM key is configured it falls back to `SearchType.CHUNKS` instead — a pure vector search that returns the matching text chunks — which is why the call works here at all. The printed `search_type` on each result reports which type actually ran.
### Step 5: Search the GLiNER Summaries
```python theme={null}
# The GLiNER-built summaries are searchable too.
results = await cognee.recall(
"Where was Marie Curie born?",
query_type=SearchType.SUMMARIES,
datasets=["no_llm"],
top_k=3,
)
print(f"\nSUMMARIES: {len(results)} result(s)")
for item in results:
print(" -", item.text.replace("\n", " | "))
```
`SearchType.SUMMARIES` searches the per-chunk summaries rather than the raw chunks. On this run those summaries were written by GLiNER from the extracted entities and relationships, not by an LLM, and they are indexed and retrievable like any other summary. Both `CHUNKS` and `SUMMARIES` are retrieval-only search types, so requesting one explicitly stays LLM-free.
## What Still Needs an LLM
The graph, the summaries, and the two searches above run entirely on local models. Two things still do not:
* **Completion search types.** Anything ending in `_COMPLETION` — `GRAPH_COMPLETION`, `RAG_COMPLETION`, `HYBRID_COMPLETION` — retrieves context and then asks an LLM to write the answer. Requesting one without a key fails; see [Search Basics](/guides/search-basics) for the full list of types and what each returns.
* **Per-turn feedback analysis.** `AUTO_FEEDBACK` is disabled above because session feedback is itself an LLM call. For the same reason, `remember()` here is left without `session_ids`, so [`improve()`](/core-concepts/main-operations/improve) only runs the default triplet and vector enrichment.
<Columns>
<Card title="Local Setup (No API Key)" icon="computer" href="/guides/local-setup">
The `.env` form of local provider configuration, plus local-run troubleshooting.
</Card>
<Card title="Custom GLiNER Extraction" icon="tags" href="/guides/gliner-llm-free-cognify">
Pass your own entity and relation labels, and measure what extraction dropped.
</Card>
<Card title="Embedding Providers" icon="layers" href="/setup-configuration/embedding-providers">
Pick a different fastembed model and its matching dimensions.
</Card>
<Card title="Search Basics" icon="search" href="/guides/search-basics">
Every search type, and which ones need an LLM to answer.
</Card>
</Columns>
# NodeSet Grouping
Source: https://docs.cognee.ai/guides/nodeset-grouping
Tag each memory with node sets so one graph keeps several overlapping topics apart
A minimal guide to grouping memories with node sets. Use it when one dataset holds several topics and you want each memory labeled — including memories that belong to more than one group — so you can see and later query those slices separately.
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) and [Embedding Providers](/setup-configuration/embedding-providers) configured
* Read [NodeSets](/core-concepts/further-concepts/node-sets) for how tags become graph nodes
* No data is required up front — the script ingests its own passages, but it starts with `cognee.forget(everything=True)`, which wipes all existing Cognee data; run it against a setup you can afford to reset
## Code in Action
```python theme={null}
import asyncio
import os
import cognee
from cognee import visualize_graph
from cognee.shared.logging_utils import ERROR, setup_logging
text_a = """
AI is revolutionizing financial services through intelligent fraud detection
and automated customer service platforms.
"""
text_b = """
Advances in AI are enabling smarter systems that learn and adapt over time.
"""
text_c = """
MedTech startups have seen significant growth in recent years, driven by innovation
in digital health and medical devices.
"""
node_set_a = ["AI", "FinTech"]
node_set_b = ["AI"]
node_set_c = ["MedTech"]
async def main():
await cognee.forget(everything=True)
await cognee.remember(text_a, node_set=node_set_a, self_improvement=False)
await cognee.remember(text_b, node_set=node_set_b, self_improvement=False)
await cognee.remember(text_c, node_set=node_set_c, self_improvement=False)
visualization_path = os.path.join(
os.path.dirname(__file__), ".artifacts", "nodeset_grouping.html"
)
await visualize_graph(visualization_path)
if __name__ == "__main__":
logger = setup_logging(log_level=ERROR)
asyncio.run(main())
```
## What Just Happened
### Step 1: Plan the Groups
```python theme={null}
node_set_a = ["AI", "FinTech"]
node_set_b = ["AI"]
node_set_c = ["MedTech"]
```
Each list is the set of labels one passage carries. `text_a` sits in two groups at once (`AI` and `FinTech`), `text_b` only in `AI`, and `text_c` only in `MedTech` — node sets overlap freely, so a memory does not have to pick a single home.
### Step 2: Tag Each Memory on Ingest
```python theme={null}
await cognee.remember(text_a, node_set=node_set_a, self_improvement=False)
await cognee.remember(text_b, node_set=node_set_b, self_improvement=False)
await cognee.remember(text_c, node_set=node_set_c, self_improvement=False)
```
`node_set` is applied while the graph is built, so the labels are materialized as `NodeSet` nodes and attached to the derived chunks and entities with `belongs_to_set` edges. `self_improvement=False` keeps the run to plain ingestion instead of also running `improve()`.
### Step 3: Render the Grouped Graph
```python theme={null}
visualization_path = os.path.join(
os.path.dirname(__file__), ".artifacts", "nodeset_grouping.html"
)
await visualize_graph(visualization_path)
```
The render writes a self-contained HTML file next to the script under `.artifacts/`. Open it and recolor the nodes by **Node set**: each document contributes one key, so `text_a`'s nodes show as `AI, FinTech` while `text_b` shows as `AI` and `text_c` as `MedTech`. To see the overlap itself, follow the `belongs_to_set` edges out of the `AI` NodeSet node — they reach the entities extracted from both `text_a` and `text_b`.
## Advanced Usage
The labels the script wrote at ingest time are read back at query time. Each option below builds on the same `AI`, `FinTech`, and `MedTech` groups the script just created.
<AccordionGroup>
<Accordion title="Scope Recall to One Group">
The same names you passed to `remember()` scope a later `recall()` through `node_name`. This is the payoff of tagging: one dataset holds all three topics, but a query can be grounded in a single slice of it.
```python theme={null}
from cognee import SearchType
# Recall only within the FinTech subset
results = await cognee.recall(
"What is happening in financial services?",
query_type=SearchType.GRAPH_COMPLETION,
node_name=["FinTech"],
)
```
In practice this lets you keep one shared dataset while still asking targeted questions — "only the finance material", "just the MedTech passages" — without splitting everything into separate datasets.
</Accordion>
<Accordion title="Combine Several Node Sets">
Pass several names to widen the slice. `node_name_filter_operator` controls how they combine: the default `OR` returns results connected to any of the listed names, while `AND` requires results to belong to all of them at once.
```python theme={null}
from cognee import SearchType
# OR (default) — anything tagged AI or MedTech
results = await cognee.recall(
"What are the key topics?",
query_type=SearchType.GRAPH_COMPLETION,
node_name=["AI", "MedTech"],
node_name_filter_operator="OR",
)
# AND — only what sits in both AI and FinTech, i.e. text_a
results = await cognee.recall(
"How is AI used in finance?",
query_type=SearchType.GRAPH_COMPLETION,
node_name=["AI", "FinTech"],
node_name_filter_operator="AND",
)
```
<Note>
Node-set filtering works with graph-completion search types (`GRAPH_COMPLETION`, `GRAPH_COMPLETION_COT`, `GRAPH_COMPLETION_CONTEXT_EXTENSION`, `GRAPH_SUMMARY_COMPLETION`, `TEMPORAL`, `RAG_COMPLETION`, `TRIPLET_COMPLETION`, `CHUNKS`). It has no effect on `SUMMARIES`, `CYPHER`, or `NATURAL_LANGUAGE`.
</Note>
</Accordion>
<Accordion title="Navigate Data by Project or Domain">
Because node sets become first-class graph nodes, they can act as anchors for exploration as well as filtering. A project-level label like `project_alpha` or a domain-level label like `compliance` gives you a stable entry point into the related documents, chunks, and entities.
This makes node sets a lightweight way to organize one knowledge graph around the mental model your team already uses: project, customer, topic, workflow, or domain.
```python theme={null}
# Tag documents by project and domain during ingestion
await cognee.remember(
"Project Alpha must satisfy EU compliance requirements.",
node_set=["project_alpha", "compliance"],
)
await cognee.remember(
"Project Alpha rollout depends on infrastructure readiness.",
node_set=["project_alpha", "operations"],
)
```
After the `remember()` workflow finishes, `project_alpha`, `compliance`, and `operations` become graph anchors you can use to explore related information by project or by domain.
</Accordion>
</AccordionGroup>
<Columns>
<Card title="NodeSets" icon="tags" href="/core-concepts/further-concepts/node-sets">
How tags become first-class graph nodes you can filter on.
</Card>
<Card title="Graph Visualization" icon="network" href="/guides/graph-visualization">
Render your knowledge graph to an interactive HTML file.
</Card>
<Card title="remember()" icon="brain" href="/python-api/remember">
Every parameter `remember()` accepts, including `node_set`.
</Card>
</Columns>
# Ontology Quickstart
Source: https://docs.cognee.ai/guides/ontology-support
Step-by-step guide to using OWL ontologies to ground Cognee knowledge graphs
A minimal guide to using OWL ontologies to ground Cognee's knowledge graphs. The guide uses `remember()` so the data and ontology config are processed together in one call.
**Before you start:**
* Complete [Quickstart](getting-started/quickstart) to understand basic operations
* Read [Ontologies](../core-concepts/further-concepts/ontologies) to understand the concepts
* Ensure you have [LLM Providers](setup-configuration/llm-providers) configured
* Have an OWL ontology file (`.owl`) in RDF/XML format
* Have some text or files relevant to the ontology's domain
## What Ontology Support Does
* Grounds extracted entities and types to your OWL ontology (classes and individuals), renaming matches to the canonical term and marking them `ontology_valid=True`
* Extends the extracted graph with the matched term's parent classes and object-property links — in the default `annotate` mode unmatched entities are kept as-is, never rejected (see [Optional: drop ungrounded entities](#optional-drop-ungrounded-entities) to opt into dropping them)
* Improves graph completion answers for domain-specific queries
Grounding reads only `owl:Class`, `rdf:type`, `rdfs:subClassOf`, and `owl:ObjectProperty` — so an ontology can supply entity **types** and **relationships** between its own terms, but not node properties (`owl:DatatypeProperty` is ignored). See [Which OWL constructs grounding actually reads](/core-concepts/further-concepts/ontologies#additional-details-and-examples).
<Tip>
Unsure whether you need an ontology or a [custom graph model](/guides/custom-graph-model)? See "How does an ontology relate to a custom graph?" under [Additional details and examples](/core-concepts/further-concepts/ontologies#additional-details-and-examples).
</Tip>
## Step 1: Prepare an Ontology File
Start from a simple OWL file. Minimal ingredients:
* Classes (e.g., `TechnologyCompany`, `Car`)
* Individuals (e.g., `Apple`, `Audi`)
* Object properties with domain/range (e.g., `produces` with `domain=CarManufacturer`, `range=Car`)
Example ontology files:
* `examples/guides/ontology_input_example/basic_ontology.owl` ([GitHub](https://github.com/topoteretes/cognee/blob/dev/examples/guides/ontology_input_example/basic_ontology.owl))
<Tip>
Use any RDF/OWL editor (Protégé) to edit .owl files.
</Tip>
<Note>
This example uses a simple ontology for demonstration. In practice, you can work with larger, more complex ontologies - the same approach works regardless of ontology size or complexity.
</Note>
## Step 2: Prepare Your Data
Add either raw text or a directory. Keep it relevant to your ontology.
```python theme={null}
texts = [
"Audi produces the R8 and e-tron.",
"Apple develops iPhone and MacBook."
]
```
<Note>
This simple example uses a list of strings for demonstration. In practice, you can add multiple documents, files, or entire datasets - the ontology processing works the same way across all your data.
</Note>
## Step 3: Remember Your Data with the Ontology Config
Create the `config` that contains the ontology information, then pass it directly to `remember()`.
```python theme={null}
# Create full config structure manually
config: Config = {
"ontology_config": {
"ontology_resolver": RDFLibOntologyResolver(ontology_file=ontology_path)
}
}
await cognee.remember(texts, config=config, self_improvement=False)
```
<Info>
If omitted, Cognee builds memory without ontology grounding. With an ontology config, Cognee aligns nodes to classes and individuals while processing the remembered data.
</Info>
## Optional: Drop Ungrounded Entities
By default grounding only annotates, so entities missing from your ontology still end up in the graph. Add `"ontology_mode": "strict"` to the same `ontology_config` to drop them instead — a node survives only if the ontology matched either its type against a class or its name against an individual, and edges touching a dropped node go with it.
```python theme={null}
config: Config = {
"ontology_config": {
"ontology_resolver": RDFLibOntologyResolver(ontology_file=ontology_path),
"ontology_mode": "strict", # overrides the ONTOLOGY_MODE environment variable
}
}
await cognee.remember(texts, config=config, self_improvement=False)
```
<Warning>
Strict mode expects an ontology that covers your corpus's vocabulary. The two-class example above would drop most of what a real document set produces — start with a small run and read the aggregate dropped/retained warning Cognee logs (one per chunk batch). An ontology that resolves to no classes and no individuals at all (usually a mistyped path) raises `EmptyOntologyInStrictModeError` rather than quietly building an empty graph. Chunk text is unaffected either way: it is stored and embedded before grounding runs, so `CHUNKS` and `RAG_COMPLETION` still return passages mentioning dropped entities.
</Warning>
## Hot Reload and File Updates
Ontology files are parsed **once at initialization** — when `RDFLibOntologyResolver` is constructed, it reads and parses the file with RDFLib and caches the result in memory. There is no automatic file system monitoring, so changes to the `.owl` file on disk are not picked up while a session or server is running.
## Full Example
<Accordion title="Latest guide">
```python theme={null}
import asyncio
import os
import cognee
from cognee.modules.ontology.ontology_config import Config
from cognee.modules.ontology.rdf_xml.RDFLibOntologyResolver import RDFLibOntologyResolver
async def main():
# Prune data and system metadata before running, only if we want "fresh" state.
await cognee.forget(everything=True)
texts = ["Audi produces the R8 and e-tron.", "Apple develops iPhone and MacBook."]
ontology_path = os.path.join(
os.path.dirname(os.path.abspath(__file__)), "ontology_input_example/basic_ontology.owl"
)
# Create full config structure manually
config: Config = {
"ontology_config": {
"ontology_resolver": RDFLibOntologyResolver(ontology_file=ontology_path)
}
}
await cognee.remember(texts, config=config, self_improvement=False)
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
<Accordion title="Legacy guide">
```python theme={null}
import cognee
import os
from cognee.modules.ontology.ontology_config import Config
from cognee.modules.ontology.rdf_xml.RDFLibOntologyResolver import RDFLibOntologyResolver
ontology_path = os.path.join(
os.path.dirname(os.path.abspath(__file__)), "ontology_input_example/basic_ontology.owl"
)
# Create full config structure manually
config: Config = {
"ontology_config": {
"ontology_resolver": RDFLibOntologyResolver(ontology_file=ontology_path)
}
}
await cognee.cognify(config=config)
```
</Accordion>
* Basic ontology demo script can be found on the following [link](https://github.com/topoteretes/cognee/blob/dev/examples/guides/ontology_quickstart.py)
* Advanced ontology demo script can be found on the following [link](https://github.com/topoteretes/cognee/blob/dev/examples/advanced_guides/ontology_reference_vocabulary/ontology_as_reference_vocabulary_example.py)
<Columns>
<Card title="Core Concepts" icon="brain" href="/core-concepts/further-concepts/ontologies">
Understand ontology fundamentals
</Card>
<Card title="API Reference" icon="code" href="/api-reference/introduction">
Explore ontology API endpoints
</Card>
</Columns>
# Permission Snippets
Source: https://docs.cognee.ai/guides/permission-snippets
Practical code snippets and scenarios for Cognee's permission system
This guide provides practical code snippets demonstrating the permission system in action. These snippets show how to create users, tenants, roles, and datasets, and how to manage permissions effectively.
<Info>**Complete snippets** — All code snippets are complete and runnable, showing the full workflow from setup to permission management.</Info>
<Accordion title="Creating a User">
[Users](../core-concepts/multi-user-mode/permissions-system/users) are the foundation of the permission system. Here's how to create a new [user](../core-concepts/multi-user-mode/permissions-system/users):
```python theme={null}
from cognee.modules.users.methods import create_user
user = await create_user(
email="alice@company.com",
password="password123",
is_superuser=True
)
```
</Accordion>
<Accordion title="Creating a Tenant">
[Tenants](../core-concepts/multi-user-mode/permissions-system/tenants) group [users](../core-concepts/multi-user-mode/permissions-system/users) together and can receive permissions. Create a [tenant](../core-concepts/multi-user-mode/permissions-system/tenants) with an owner:
```python theme={null}
from cognee.modules.users.tenants.methods import create_tenant
# Assuming user is already created
await create_tenant("acme_corp", user.id)
```
</Accordion>
<Accordion title="Adding Users to a Tenant">
Add existing [users](../core-concepts/multi-user-mode/permissions-system/users) to a [tenant](../core-concepts/multi-user-mode/permissions-system/tenants). Only the [tenant](../core-concepts/multi-user-mode/permissions-system/tenants) owner can add [users](../core-concepts/multi-user-mode/permissions-system/users):
```python theme={null}
from cognee.modules.users.tenants.methods import add_user_to_tenant
# Assuming user2, tenant_id, and owner_id are already defined
await add_user_to_tenant(user2.id, tenant_id, owner_id)
```
</Accordion>
<Accordion title="Creating a Role">
[Roles](../core-concepts/multi-user-mode/permissions-system/roles) provide permission groups within a [tenant](../core-concepts/multi-user-mode/permissions-system/tenants). Create a [role](../core-concepts/multi-user-mode/permissions-system/roles) for the [tenant](../core-concepts/multi-user-mode/permissions-system/tenants):
```python theme={null}
from cognee.modules.users.roles.methods import create_role
# Assuming owner_id is the tenant owner
await create_role("editor", owner_id)
```
</Accordion>
<Accordion title="Creating a Dataset">
Datasets are the core data containers. Create a dataset with automatic permissions for the creator:
```python theme={null}
from cognee.modules.data.methods import create_authorized_dataset
# Assuming user is already created
dataset = await create_authorized_dataset("project_docs", user)
```
</Accordion>
<Accordion title="Granting Read Permission">
Grant specific permissions to principals. Give read access to a user:
```python theme={null}
from cognee.modules.users.permissions.methods import give_permission_on_dataset
# Assuming user2 and dataset are already created
await give_permission_on_dataset(user2, dataset.id, "read")
```
</Accordion>
<Accordion title="Granting Multiple Permissions">
Each call grants exactly one permission name, so write a separate call per permission. Give comprehensive access:
```python theme={null}
from cognee.modules.users.permissions.methods import give_permission_on_dataset
# Assuming user2 and dataset are already created
await give_permission_on_dataset(user2, dataset.id, "read")
await give_permission_on_dataset(user2, dataset.id, "write")
await give_permission_on_dataset(user2, dataset.id, "delete")
```
The four valid `permission_name` values are `"read"`, `"write"`, `"delete"`, and `"share"`. Passing any other value raises `PermissionNotFoundError`.
</Accordion>
<Accordion title="Owner Full Access + Read-Only Collaborator">
The most common pattern: the creator gets full control, a teammate gets read-only access.
`create_authorized_dataset` automatically grants the creator all four permissions (`read`, `write`, `delete`, `share`) on the new dataset — no extra calls are needed for the owner. To share with another user, call `give_permission_on_dataset` once per permission you want to grant.
```python theme={null}
from cognee.modules.data.methods import create_authorized_dataset
from cognee.modules.users.permissions.methods import give_permission_on_dataset
# Creator gets read + write + delete + share automatically
dataset = await create_authorized_dataset("research_docs", owner)
# Collaborator gets read-only — nothing else implied
await give_permission_on_dataset(reader, dataset.id, "read")
```
To upgrade the collaborator to read + write later, grant `"write"` as a separate call — it does not replace the existing `"read"` grant:
```python theme={null}
await give_permission_on_dataset(reader, dataset.id, "write")
# reader now has both read and write; the read entry is unchanged
```
Use [`revoke_permission_on_dataset`](../core-concepts/multi-user-mode/permissions-system/acl) to remove a specific permission later.
</Accordion>
<Accordion title="Repeated Grants and Idempotency">
`give_permission_on_dataset` is **insert-only and idempotent**. Calling it again with the same `(principal, dataset_id, permission_name)` does not create a duplicate ACL row, does not bump `updated_at`, and does not raise — the existing row is left as-is.
```python theme={null}
# First call: inserts an ACL row granting read.
await give_permission_on_dataset(user2, dataset.id, "read")
# Second call with the same arguments: no-op. No new row, no update.
await give_permission_on_dataset(user2, dataset.id, "read")
```
This means it is safe to re-run grant logic on startup or in setup scripts. It also means it cannot be used to "change" a permission — if `user2` already has `read` and you want them to have `write` *instead*, you must explicitly revoke `read` and grant `write`. See the `revoke_permission_on_dataset` function in [ACL](../core-concepts/multi-user-mode/permissions-system/acl).
</Accordion>
<Accordion title="Checking User Permissions">
Query what datasets a user can access. Check permissions by type:
```python theme={null}
from cognee.modules.users.permissions.methods import get_all_user_permission_datasets
# Assuming user is already created
# Get all datasets user can read
readable_datasets = await get_all_user_permission_datasets(user, "read")
# Get all datasets user can write
writable_datasets = await get_all_user_permission_datasets(user, "write")
```
</Accordion>
<Accordion title="Complete Permission Setup">
Set up a complete permission scenario from scratch. This example shows the full workflow:
```python theme={null}
from cognee.modules.users.methods import create_user, get_user
from cognee.modules.users.tenants.methods import create_tenant, add_user_to_tenant
from cognee.modules.data.methods import create_authorized_dataset
from cognee.modules.users.permissions.methods import give_permission_on_dataset
# 1. Create users
user1 = await create_user("alice@company.com", "password123", is_superuser=True)
user2 = await create_user("bob@company.com", "password456")
# 2. Create tenant and add users
await create_tenant("acme_corp", user1.id)
# Refresh user1 to get tenant_id
user1 = await get_user(user1.id)
await add_user_to_tenant(user2.id, user1.tenant_id, user1.id)
# 3. Create dataset
dataset = await create_authorized_dataset("confidential_docs", user1)
# 4. Grant different permissions
await give_permission_on_dataset(user2, dataset.id, "read") # Read-only access
```
</Accordion>
<Accordion title="Permission Inheritance Example">
Demonstrate how permissions flow through the hierarchy. Show tenant and role inheritance:
```python theme={null}
from cognee.modules.users.permissions.methods import give_permission_on_dataset
# Assuming tenant, role, and dataset are already created
# Grant permission to tenant (all users inherit)
await give_permission_on_dataset(tenant, dataset.id, "read")
# Grant permission to role (role members inherit)
await give_permission_on_dataset(role, dataset.id, "write")
# User gets both: read (from tenant) + write (from role)
```
</Accordion>
<Accordion title="Multi-tenant Organization Setup">
Create organization with multiple teams:
```python theme={null}
# Create organization with multiple teams
# 1. Create tenant
tenant = await create_tenant("tech_company", admin_user.id)
# 2. Create roles for different teams
dev_role = await create_role("developers", admin_user.id)
qa_role = await create_role("qa_team", admin_user.id)
pm_role = await create_role("product_managers", admin_user.id)
# 3. Create datasets for different projects
frontend_dataset = await create_authorized_dataset("frontend_docs", admin_user)
backend_dataset = await create_authorized_dataset("backend_docs", admin_user)
qa_dataset = await create_authorized_dataset("qa_docs", admin_user)
# 4. Grant role-based permissions
await give_permission_on_dataset(dev_role, frontend_dataset.id, "write")
await give_permission_on_dataset(dev_role, backend_dataset.id, "write")
await give_permission_on_dataset(qa_role, qa_dataset.id, "write")
await give_permission_on_dataset(pm_role, frontend_dataset.id, "read")
await give_permission_on_dataset(pm_role, backend_dataset.id, "read")
```
</Accordion>
<Accordion title="Temporary Access Management">
Grant temporary access to external contractor:
```python theme={null}
# Grant temporary access to external contractor
contractor = await create_user("contractor@external.com", "temp_password")
# Grant read access to specific dataset
await give_permission_on_dataset(contractor, project_dataset.id, "read")
# Later, revoke access by removing the permission
# (This would require a revoke_permission function)
```
</Accordion>
<Accordion title="Cross-team Collaboration">
Allow teams to collaborate on shared datasets:
```python theme={null}
# Allow teams to collaborate on shared datasets
shared_dataset = await create_authorized_dataset("shared_research", admin_user)
# Grant different levels of access to different teams
await give_permission_on_dataset(dev_role, shared_dataset.id, "read")
await give_permission_on_dataset(research_role, shared_dataset.id, "write")
await give_permission_on_dataset(management_role, shared_dataset.id, "read")
```
</Accordion>
<Accordion title="Best Practices">
Follow these best practices for permission management:
* **Start simple** — Begin with basic user and dataset creation
* **Use roles for teams** — Create roles for different job functions
* **Grant tenant permissions** — Use tenant-level permissions for organization-wide access
* **Regular audits** — Periodically review and update permissions
* **Document access patterns** — Keep clear records of who has access to what
* **Test permission changes** — Verify permissions work as expected after changes
</Accordion>
<Columns>
<Card title="Setup Configuration" icon="settings" href="/setup-configuration/permissions">
Learn how to configure the permission system
</Card>
<Card title="API Reference" icon="code" href="/api-reference/introduction">
Explore permission system API endpoints
</Card>
<Card title="Sharing Memory Across Users and Teams" icon="shield-check" href="/examples/multi-tenant-access-control">
See these calls in one runnable demo, from the first denial to a role-level grant
</Card>
</Columns>
# Folder Presort
Source: https://docs.cognee.ai/guides/presort-downloads
Scan a messy folder for junk, duplicates, and personal data before ingesting it as datasets
A minimal guide to pre-organizing a folder before it reaches your graph. Presort scans a directory without touching the files on disk, reports what is junk, duplicated, versioned, personal, or already in cognee, and then ingests the groups you approve — one dataset per group.
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — the analyze phase is deterministic, but the apply phase runs cognify
* Read [Datasets](/core-concepts/further-concepts/datasets) — presort ingests each proposed group into its own dataset
* Set `PRESORT_FOLDER` to the folder you want to scan, or let the script use `~/Downloads`
* Name that folder as a permitted scan root before running the script — the defaults are the working directory, the temp directory, and cognee's own storage, so a folder under `$HOME` is always outside them: `export COGNEE_ALLOWED_LOCAL_FILE_ROOTS="$HOME/Downloads"`. See [Presort scan roots](/setup-configuration/security#presort-scan-roots)
## Code in Action
```python theme={null}
import asyncio
import os
from pathlib import Path
import cognee
FOLDER = os.environ.get("PRESORT_FOLDER", str(Path.home() / "Downloads"))
async def main():
# Phase 1: analyze. Returns a PresortReport (also auto-saved under
# cognee's system directory as <scan-id>.presort.json).
report = await cognee.remember(FOLDER, dry_run="presort")
summary = report.summary()
print(f"Scanned {summary['files']} files ({summary['junk']} junk skipped)")
print(f"Already in cognee: {summary['cognee_status']}")
print(
f"Duplicate clusters: {summary['duplicate_clusters']} ({summary['wasted_bytes']} wasted bytes)"
)
print(f"Potential personal data: {summary['pii_findings']} findings")
for group in report.groups:
print(
f" group {group.name!r} -> dataset {group.dataset_name!r} ({len(group.file_paths)} files)"
)
# Review/adjust the apply decisions on the report itself.
report.exclude_pii = True # keep files with personal data out of the graph
report.skip_duplicates = True # ingest one copy per duplicate cluster
report.apply_groups = [group.name for group in report.groups if group.kind != "code_project"]
# Phase 2: apply. One dataset per proposed group; re-running is idempotent
# (already-cognified content is skipped by incremental loading).
results = await cognee.remember(report)
for dataset_name, result in results.items():
print(f"ingested dataset {dataset_name!r}: {result}")
# The presorted data is now queryable per dataset.
answers = await cognee.recall("What documents do I have?", datasets=list(results))
for answer in answers:
print(answer)
if __name__ == "__main__":
asyncio.run(main())
```
The complete runnable script is on GitHub: [`examples/guides/presort_downloads.py`](https://github.com/topoteretes/cognee/blob/dev/examples/guides/presort_downloads.py).
## What Just Happened
### Step 1: Scan the Folder
```python theme={null}
FOLDER = os.environ.get("PRESORT_FOLDER", str(Path.home() / "Downloads"))
# Phase 1: analyze. Returns a PresortReport (also auto-saved under
# cognee's system directory as <scan-id>.presort.json).
report = await cognee.remember(FOLDER, dry_run="presort")
```
`dry_run="presort"` turns `remember()` into an analyzer: it walks the folder, reads samples of the files, and returns a `PresortReport` instead of ingesting anything. Nothing on disk is moved, renamed, or deleted. The report is also saved to `<SYSTEM_ROOT_DIRECTORY>/presort/<scan_id>.presort.json`, so the analyze result survives a failed apply — `report.report_path` holds the exact location, or `None` when `SYSTEM_ROOT_DIRECTORY` is unset or on S3 and the report was not persisted.
### Step 2: Review the Report
```python theme={null}
summary = report.summary()
print(f"Scanned {summary['files']} files ({summary['junk']} junk skipped)")
print(f"Already in cognee: {summary['cognee_status']}")
print(
f"Duplicate clusters: {summary['duplicate_clusters']} ({summary['wasted_bytes']} wasted bytes)"
)
print(f"Potential personal data: {summary['pii_findings']} findings")
for group in report.groups:
print(
f" group {group.name!r} -> dataset {group.dataset_name!r} ({len(group.file_paths)} files)"
)
```
`report.summary()` is the at-a-glance view: file and junk counts, exact-duplicate clusters with the bytes they waste, version candidates, potential personal data, and a `cognee_status` breakdown of how many files are new, staged, or already cognified. `report.groups` holds the proposal itself — each group carries the `dataset_name` it would land in and the files it would take there.
### Step 3: Set the Apply Decisions
```python theme={null}
report.exclude_pii = True # keep files with personal data out of the graph
report.skip_duplicates = True # ingest one copy per duplicate cluster
report.apply_groups = [group.name for group in report.groups if group.kind != "code_project"]
```
The report is editable, and the apply phase reads these three fields off it. `exclude_pii` defaults to `False`, so setting it is what actually changes the run — it drops files with potential personal data. `skip_duplicates` is already on by default; it is spelled out to keep the decision visible. `apply_groups` narrows the run to the groups you name — here every group except code projects, which belong in a [code graph](/guides/code-graph) rather than a document dataset.
### Step 4: Apply the Report and Recall
```python theme={null}
results = await cognee.remember(report)
for dataset_name, result in results.items():
print(f"ingested dataset {dataset_name!r}: {result}")
answers = await cognee.recall("What documents do I have?", datasets=list(results))
for answer in answers:
print(answer)
```
Passing the report back to `remember()` runs the second phase: each approved group goes through the normal add → cognify chain into its own dataset, and you get back a `{dataset_name: result}` mapping. Ingestion is incremental, so re-running the script after adding a few files only processes what is new. From there the folder is ordinary cognee memory — `recall()` takes the dataset names as its scope.
## Advanced Usage
<AccordionGroup>
<Accordion title="Deterministic scan vs. LLM analysis">
The analyze phase needs no LLM or embedding configuration: junk filtering, duplicate detection, version candidates, and folder-based grouping are all deterministic. Pass `use_llm=True` to `remember(folder, dry_run="presort", use_llm=True)` for LLM content classification, deeper PII detection, and semantic grouping.
The apply phase runs cognify and does need a configured LLM. Without one, presort degrades rather than fails: the deterministic scan still runs, `use_llm` is downgraded, and apply stages files with `add()` only — each reported as a warning on the report.
</Accordion>
<Accordion title="Skipping the review step">
When you trust the defaults, `remember(FOLDER, dry_run="presort", auto_apply=True)` collapses both phases into one call instead of the two this guide makes — see [auto\_apply](/python-api/remember#folder-presort) for what comes back.
</Accordion>
<Accordion title="Running presort from the CLI">
`cognee-cli remember <folder> --presort` runs the same two phases from the shell, with `--allow-root` in place of the environment variable. The commands and their apply-time flags are in the [CLI reference](/cognee-cli/overview#remember-data).
</Accordion>
</AccordionGroup>
<Columns>
<Card title="Datasets" icon="database" href="/core-concepts/further-concepts/datasets">
How the datasets presort proposes organize documents, permissions, and processing.
</Card>
<Card title="remember()" icon="brain" href="/python-api/remember">
Every option the two presort phases accept, including the apply overrides.
</Card>
</Columns>
# Understand Recall with RAG Completion
Source: https://docs.cognee.ai/guides/rag-recall
Explore the RAG retrieval workflow with cognee.recall() and learn how its main parameters affect the results
This guide teaches `cognee.recall()` and `SearchType.RAG_COMPLETION` side by side, demonstrating `only_context`, `top_k`, `datasets`, `system_prompt`, and `include_references` along the way. `RAG_COMPLETION` always performs the same three steps: retrieve relevant information, build the context, and generate the answer.
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured
* Be familiar with the [`remember()`](/core-concepts/main-operations/remember) workflow
* See [Recall](/core-concepts/main-operations/recall) for the full definition of each parameter demonstrated here (`only_context`, `top_k`, `datasets`, `system_prompt`, `include_references`)
## Code in Action
```python theme={null}
import asyncio
import cognee
from cognee import SearchType
DATASET_NAME = "cognee_recall_demo"
DOCUMENTS = [
"""
Alice is learning how to use Cognee.
She stores her project documentation using cognee.remember().
""",
"""
Bob is helping Alice understand retrieval.
He explains that SearchType.RAG_COMPLETION retrieves relevant document chunks before generating an answer.
""",
"""
Bob also explains that only_context=True returns the retrieved context
without asking the language model to generate a response.
""",
]
QUERY = "What does only_context=True do?"
async def main():
await cognee.remember(
DOCUMENTS,
dataset_name=DATASET_NAME,
)
context = await cognee.recall(
QUERY,
query_type=SearchType.RAG_COMPLETION,
datasets=[DATASET_NAME],
top_k=2,
only_context=True,
)
print("Retrieved context:\n")
print(context)
answer = await cognee.recall(
QUERY,
query_type=SearchType.RAG_COMPLETION,
datasets=[DATASET_NAME],
top_k=2,
)
print("\nGenerated answer:\n")
print(answer)
answer1 = await cognee.recall(
QUERY,
query_type=SearchType.RAG_COMPLETION,
datasets=[DATASET_NAME],
top_k=2,
include_references=True,
)
print("\nAnswer with references:\n")
print(answer1)
answer2 = await cognee.recall(
QUERY,
query_type=SearchType.RAG_COMPLETION,
datasets=[DATASET_NAME],
top_k=2,
system_prompt="Answer in two sentences.",
include_references=True,
)
print("\nAnswer in two sentences:\n")
print(answer2)
answer3 = await cognee.recall(
QUERY,
query_type=SearchType.RAG_COMPLETION,
datasets=[DATASET_NAME],
top_k=2,
system_prompt="Answer with emojis and exclamation marks.",
include_references=True,
)
print("\nAnswer with emojis:\n")
print(answer3)
if __name__ == "__main__":
asyncio.run(main())
```
## What Just Happened
### Step 1: Ingest the Example Documents
```python theme={null}
await cognee.remember(
DOCUMENTS,
dataset_name=DATASET_NAME,
)
```
`remember()` ingests the three short documents into an isolated dataset, so `recall()` has something to search. `dataset_name` chooses which dataset the data goes into. Every `recall()` call below passes that same name via `datasets=[DATASET_NAME]`, so it searches only this dataset — not everything you may have stored.
### Step 2: Retrieve Only the Context
```python theme={null}
context = await cognee.recall(
QUERY,
query_type=SearchType.RAG_COMPLETION,
datasets=[DATASET_NAME],
top_k=2,
only_context=True,
)
print("Retrieved context:\n")
print(context)
```
With `only_context=True`, Cognee retrieves the two most relevant chunks (`top_k=2`) and assembles them into context — then stops. No language model is called, so this shows exactly what would be sent to it. `print(context)` lets you see exactly which chunks were considered most relevant to the query, before any answer is generated from them.
### Step 3: Generate the Answer
```python theme={null}
answer = await cognee.recall(
QUERY,
query_type=SearchType.RAG_COMPLETION,
datasets=[DATASET_NAME],
top_k=2,
)
print("\nGenerated answer:\n")
print(answer)
```
The same call without `only_context=True` retrieves the same chunks and builds the same context, but this time sends it to a language model along with the query. `print(answer)` lets you see this generated answer on its own, now that the context has been used to produce it. A typical result:
```text theme={null}
Setting only_context=True returns the retrieved context without prompting the language model to generate a response.
```
(The exact wording depends on the LLM provider you use.)
### Step 4: Include Supporting References
```python theme={null}
answer1 = await cognee.recall(
QUERY,
query_type=SearchType.RAG_COMPLETION,
datasets=[DATASET_NAME],
top_k=2,
include_references=True,
)
print("\nAnswer with references:\n")
print(answer1)
```
Same retrieval and context as Step 3 — `include_references=True` appends an Evidence section listing which retrieved chunks the answer was built from, without changing retrieval, context, or the answer's own wording. `print(answer1)` lets you see that Evidence section right below the answer text.
### Step 5: Customize the Answer with system\_prompt
```python theme={null}
answer2 = await cognee.recall(
QUERY,
query_type=SearchType.RAG_COMPLETION,
datasets=[DATASET_NAME],
top_k=2,
system_prompt="Answer in two sentences.",
include_references=True,
)
print("\nAnswer in two sentences:\n")
print(answer2)
answer3 = await cognee.recall(
QUERY,
query_type=SearchType.RAG_COMPLETION,
datasets=[DATASET_NAME],
top_k=2,
system_prompt="Answer with emojis and exclamation marks.",
include_references=True,
)
print("\nAnswer with emojis:\n")
print(answer3)
```
Same retrieval, same context as Step 3 for both calls — only `system_prompt` changes. One instructs the model to answer in two sentences; the other asks for emojis and exclamation marks instead. `system_prompt` only ever affects how the final answer is phrased, never what is retrieved or how the context is built.
`print(answer2)` and `print(answer3)` let you compare the two styles side by side — and with `include_references=True` on both, you can also see that the Evidence section lists the exact same retrieved chunks for each, confirming that only the answer's style changed, not what was retrieved. Possible answers are:
```text theme={null}
Setting only_context=True retrieves the relevant context without prompting the language model to generate a response. This allows users to access the pertinent information directly, without any additional output from the model.
```
for `answer2`, and:
```text theme={null}
Setting only_context=True retrieves the context directly, without generating a response! 🎉📚✨
```
for `answer3`. (The exact wording depends on the LLM provider you use.)
## Under the Hood
<Accordion title="How RAG_COMPLETION Works">
`SearchType.RAG_COMPLETION` always performs the same three steps:
1. **Retrieve relevant information** — Cognee searches the stored documents and selects the text chunks most relevant to your query.
2. **Build the context** — the retrieved chunks are combined into a single context containing the information needed to answer the question.
3. **Generate the answer** — Cognee sends the context, together with your query, to a language model, which uses it to generate the final answer.
`only_context=True` stops the process after Step 2 and returns the context instead of continuing to Step 3.
</Accordion>
<Columns>
<Card title="Recall" icon="search" href="/core-concepts/main-operations/recall">
Understand recall()'s full parameter surface and auto-routing behavior
</Card>
<Card title="Inspecting Hybrid Retrieval Context" icon="layers" href="/guides/hybrid-retrieval-recall">
Go deeper with SearchType.HYBRID\_COMPLETION once you're comfortable here
</Card>
</Columns>
# Reading the Visualization
Source: https://docs.cognee.ai/guides/reading-the-visualization
What each tab of a rendered graph shows, and which one to reach for
Every file `visualize_graph()` writes is a self-contained HTML page that opens with a tab bar of four views — **Graph**, **Schema**, **Memory**, and **Semantic** — all computed from the same graph payload. No server, no rebuild: switching tabs re-reads data the page already carries.
This page explains what each view shows and when to reach for it. To generate a file in the first place, start with [Graph Visualization](/guides/graph-visualization).
## The tab bar
| Tab | What it shows | Reach for it when |
| ------------ | ----------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| **Graph** | Nodes and edges laid out by structure, with layout, label, and color controls | You want to explore the graph itself — follow relationships, find a node, spot a cluster |
| **Schema** | A by-type summary: instance counts per semantic type and how types connect | You want to know what *kinds* of things are in memory, not which individual ones |
| **Memory** | A deterministic map of how the memory was built, plus the run timeline | You want to audit the pipeline — which document produced which chunks and entities |
| **Semantic** | Nodes placed by the 2-D projection of their embeddings | You want meaning-space neighborhoods rather than edge-based structure |
Everything below describes the rendered artifact. The graph's *content* — which nodes and edges make it into the file at all — is controlled when you render it; see [bounded subgraphs](/guides/graph-visualization#advanced-usage).
## Graph — classic topology
The default view: nodes and edges laid out by structure. Drag nodes, zoom/pan, and hover edges for details. A control bar at the bottom groups three sets of toggles, each with an explanatory tooltip on hover, plus zoom out / zoom in / **Fit to view** buttons.
### Layout modes
The three layout buttons change only where nodes are drawn — never which nodes or edges are in the render.
| Mode | How it positions nodes | Use it when |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Story** (default) | Fixed pipeline columns — Documents → Chunks → Entities → Types → Summaries (plus Context, Schema, and Other when present), each column labeled with its node count. Nodes are pinned on both axes in a readable grid. | You want to read how raw data became memory, stage by stage, or you need a stable layout that renders the same way every time (screenshots, comparing two runs). |
| **Flow** | Columns follow processing order, pulling nodes horizontally. Nothing is pinned — vertical position is only weakly anchored, so link forces let nodes settle next to whatever they connect to. | You want to keep the left-to-right pipeline reading order while letting related nodes drift together vertically. |
| **Force** | No columns. A physics simulation pulls connected nodes together: semantic edges pull tighter than structural ones (containment, `is_a`, summarization), and high-degree hubs repel more strongly so they stop occluding their neighbors. | You want to spot clusters, hubs, and unexpected connections rather than follow the pipeline. Drag nodes to pull a cluster apart and explore it. |
Story mode needs at least two pipeline stages in the render to draw its columns; on a graph with only one stage it falls back to Flow's column layout.
### Label budget
The label budget controls how many node labels are drawn at once — on a dense graph, labeling everything is unreadable.
* **Key** (default) — landmark nodes plus high-importance entities.
* **All** — every node.
* **Off** — labels hidden; hover a node to peek at its label.
### Node labels
Node labels are always human-readable — raw UUID- or content-hash-shaped values are never used as a display name. When a node has no readable `name`, the label falls back through its `title`, `text`, `summary`, `description`, or `content`. If none of those is usable it shows an explicit placeholder in the form `Unnamed <Type> (id8)` — for example `Unnamed Entity (a1b2c3d4)`, where `id8` is the first eight characters of the node id. Nodes that only have a placeholder name are never chosen as **Key**-mode label landmarks.
### Color by, legend, and stats
Nodes can be recolored by **Type** (default), **Node set**, or **User**.
A **Color by** mode is disabled, with an explanatory tooltip, when the graph carries no provenance for it — **Node set** shows *"No node sets in this graph"* and **User** shows *"No user provenance in this graph"* — rather than coloring every node the same "Unknown" gray. For the same reason the stats line counts only provenance that is present, so a graph with no node sets reports `0 node sets` instead of counting the absence as one.
In **Type** mode the legend swatches are sampled from the colors actually drawn on the nodes, so the legend can never disagree with the canvas. The legend lists at most the eight largest groups.
<Tip>
Coloring by **Node set** is the fastest way to see how a [node-set grouping](/guides/nodeset-grouping) actually partitioned your graph.
</Tip>
### Searching the graph
Type in the search box to highlight nodes whose name or type match. A live counter beside the box shows how many nodes matched (e.g. `18 matches`). Press **Enter** to jump to the best match and pan/zoom to it; each subsequent **Enter** cycles forward through the matches (the counter updates to `1 / 18`, `2 / 18`, …), and **Shift+Enter** cycles backward. Press **Escape** to clear the search and its highlights.
## Schema — types at a glance
A by-type summary of the rendered graph: instance counts per semantic type and the relationship distribution between types, computed at render time.
**Type boxes and instance mini-cards.** Each semantic type (e.g. *Person*, *Broker*, *Tool*) is shown as a box with its instance count, a few representative instance mini-cards, and the relationships connecting it to other types. When a type has more instances than fit, a **`+ N more`** toggle expands the full instance list, switching to **Show less** to collapse it again.
**Bounded Entity column.** Every distinct semantic entity type the LLM extracts becomes its own card, so on large graphs that column could grow without bound. Once a graph has more than **12** semantic entity types, the 11 most-populated keep their own cards and the remaining long tail collapses into a single **`Other entities`** rollup card. The rollup leads with the number of rolled-up types and its largest members (e.g. `8 rolled up: Animal (5), Vehicle (3), …`) and otherwise behaves like a normal type box — its instance count, drill-down, and relationship edges aggregate every rolled-up type. Graphs at or below 12 entity types are shown unchanged, with no rollup card.
**Instance inspector.** Click a type box to open the inspector side panel for that type, or an instance mini-card to drill into a single instance. The inspector then shows a breadcrumb (`‹ Type`) back to the type-level view and a close (`×`) control to dismiss the panel.
**Click-to-spotlight.** Clicking a type box, edge card, or operation chip spotlights the related elements on the canvas. The schema canvas supports pan/drag and mouse-wheel zoom, and the spotlight overlay tracks elements as you move around.
**Operations-impact overlay.** A rail of operation chips — `cognify`, `memify (triplets)`, `improve (self-improve)`, `feedback weighting`, `forget`, and others — maps cognee operations onto the schema types they touch. Clicking a chip highlights the affected types, color-coded by effect:
* **produces** — the operation creates instances of this type (`cognify` produces `TextDocument`, `DocumentChunk`, `Entity`, `EntityType`, and `TextSummary`).
* **enriches** — it augments existing instances (`memify (triplets)` enriches `Entity`).
* **modifies** — it changes a property on existing instances (feedback and frequency weighting update `feedback_weight` / `frequency_weight`).
* **removes** — it deletes instances of this type (`forget`).
The same projection is available as a standalone data API with its own HTTP endpoint — useful for driving dashboards without rendering HTML. See [Schema Inventory](/guides/schema-inventory).
## Memory — pipeline structure
A deterministic map of how the memory was built: documents, their chunks, and the entities extracted from them, plus the run timeline. Every list is ordered by keys intrinsic to the data, so the layout is reproducible and append-stable as the graph grows.
Not to be confused with [Memory Provenance](/guides/memory-provenance), which is a separate projection of the relational database — tenants, users, agents, datasets, files — rendered to its own HTML file rather than a tab in this one.
## Semantic — layout by meaning
Instead of laying nodes out by their edges, the Semantic tab places each node at the 2‑D projection of its embedding, so semantically similar nodes sit together and clusters of related entities become visible at a glance. It reuses the vectors Cognee already stored during `cognify()` — nothing is re‑embedded at render time on the default LanceDB backend, and only 2‑D positions and precomputed neighbor lists are sent to the browser.
Click **Semantic** in the tab bar, or append `#semantic` to the file URL to deep‑link straight to it. In the tab you can:
* **Cluster / Type** — toggle recoloring nodes by semantic cluster or by ontology type.
* **Hover** a node to light up its nearest neighbors and list its relations.
* **Legend** entries filter to a single cluster or type; scroll or use the on‑screen controls to zoom.
* **Semantic ⇄ Structural** — toggle between the pinned meaning‑space layout and a bounded force layout over the graph topology.
* **Recall overlay** — light up the nodes a past recall query retrieved.
**Choosing the projection.** By default the layout uses **PCA** (pure‑numpy, sign‑stabilized), which is deterministic — the same graph always renders the same layout. To use **UMAP** instead, install it and opt in with an environment variable:
```bash theme={null}
pip install umap-learn
export SEMANTIC_MAP_PROJECTION=umap
```
UMAP is an optional dependency and a lazy import — when `umap-learn` is not installed, the layout silently falls back to PCA.
<Note>
The Semantic tab is best‑effort: if embeddings can't be fetched or the projection fails, the tab shows a friendly empty state and the classic render is never affected. Nodes that have no stored vector are placed at the centroid of their positioned neighbors.
</Note>
**Behavior on large graphs.** The semantic layout and clustering are bounded to **2000 nodes** (`SEMANTIC_NODE_CAP`). Graphs above that are reduced with a deterministic seeded sample, so results are approximate at scale but stable across runs. When sampling kicks in, a warning is logged.
When vectors are fetched, an info‑level log reports the join hit‑rate, e.g. `resolved 128/150 node embeddings across 4 collection(s)`. If nothing resolves (a blank Semantic map), a warning names the missing collections and unmapped node types — the usual cause of a blank map is an id/collection‑name mismatch rather than a silent failure.
## Light and dark theme
A **Dark mode** toggle in the top-right corner switches between the light and dark themes. Toggling repaints the graph canvas immediately, and the Schema view re-renders its palette so cards, chips, and edges follow the active theme. Your choice is remembered across reloads — it is persisted in the browser's `localStorage` under the key `cognee-viz-theme` and applied before the first paint, so the visualization opens in the theme you last used, defaulting to light on first visit.
<Columns>
<Card title="Graph Visualization" icon="network" href="/guides/graph-visualization">
Render a file, seed the view from a query, and bound large graphs.
</Card>
<Card title="Schema Inventory" icon="table" href="/guides/schema-inventory">
The Schema tab's projection as a standalone data API.
</Card>
<Card title="Memory Provenance" icon="folder-tree" href="/guides/memory-provenance">
Tenants, users, agents, datasets, and files as their own HTML file.
</Card>
</Columns>
# S3 Storage
Source: https://docs.cognee.ai/guides/s3-storage
Step-by-step guide to using S3 for data ingestion and storage
A minimal guide to using S3 (or S3-compatible, e.g., MinIO) to ingest data and/or store Cognee's internal files.
**Before you start:**
* Complete [Quickstart](getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](setup-configuration/llm-providers) configured
* Have S3 credentials and access to an S3 bucket
## What S3 Storage Does
* **Ingest from S3**: Pass `s3://...` paths to `cognee.add()` to load data directly from S3
* **Store Cognee data on S3**: Set your data/system roots to S3 URLs to keep all files on S3
* **S3-compatible**: Works with MinIO and other S3-compatible services
## Prerequisites
Install with AWS extra if needed (boto3/s3fs) and add credentials to `.env`:
```dotenv theme={null}
aws_access_key_id=your_access_key
aws_secret_access_key=your_secret_key
aws_region=us-east-1
# Optional for S3-compatible endpoints (e.g., MinIO):
aws_endpoint_url=http://localhost:9000
```
## Option A: Ingest from S3
Pass S3 URIs (files or prefixes) directly to `remember()`. Directories/prefixes expand to files when credentials are set.
```python theme={null}
import asyncio
import cognee
async def main():
# Single file: ingest and build the graph in one call
await cognee.remember(
"s3://cognee-s3-small-test/Natural_language_processing.txt",
dataset_name="s3_single_demo",
self_improvement=False,
)
# Folder/prefix (recursively expands)
await cognee.remember(
"s3://cognee-s3-small-test",
dataset_name="s3_prefix_demo",
self_improvement=False,
)
# Mixed list
await cognee.remember(
[
"s3://cognee-s3-small-test/Natural_language_processing.txt",
"Some inline text to ingest",
],
dataset_name="s3_mixed_demo",
self_improvement=False,
)
if __name__ == "__main__":
asyncio.run(main())
```
<Accordion title="Legacy guide">
```python theme={null}
import asyncio
import cognee
async def main():
# Single file
await cognee.add("s3://my-bucket/docs/paper.pdf")
# Folder/prefix (recursively expands)
await cognee.add("s3://my-bucket/datasets/reports/")
# Mixed list
await cognee.add([
"s3://my-bucket/docs/paper.pdf",
"Some inline text to ingest",
])
# Process the data
await cognee.cognify()
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
This loads data directly from S3 using the `s3://` URI. `remember()` expands prefixes, reads the S3 objects, and builds retrieval-ready memory for each target dataset.
<Note>
This simple example uses S3 paths for demonstration. In practice, you can mix S3 files with local files, use dataset scoping, and apply custom loaders. The same `remember()` flow works with S3 paths.
</Note>
## Option B: Store Cognee Data on S3
Keep Cognee's generated files (text copies, system files) on S3 by pointing roots to S3 URLs.
Add this to your `.env`:
```dotenv theme={null}
DATA_ROOT_DIRECTORY="s3://my-bucket/cognee/data"
SYSTEM_ROOT_DIRECTORY="s3://my-bucket/cognee/system"
# Optional: force S3 backend detection
STORAGE_BACKEND="s3"
```
This configures Cognee to store all its internal files (processed data, system files) on S3 instead of locally.
<Info>
Cognee chooses S3 storage when roots start with `s3://` (or when `STORAGE_BACKEND=s3` and both roots are S3 URLs). If `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` are not set in `.env`, Cognee falls back to boto3/s3fs's default credential chain for native AWS S3 deployments instead of erroring. See [Object Storage](/setup-configuration/object-storage) for provider-specific setup details.
</Info>
## Performance notes
Ingesting a file over the S3 backend costs **4 S3 requests per file** (1 PUT, 2 HEAD, 1 GET), down from 13. Cognee now uploads the payload once instead of twice and hashes it from the bytes it already holds, rather than downloading the object back several times to recompute a hash of content it just wrote.
Uploads are written in 4 MB chunks rather than read into memory whole, so the peak memory one upload adds is bounded by the chunk size instead of the file size — large files no longer scale RAM with their length.
Inside `DATA_ROOT_DIRECTORY`, an uploaded source file is stored under a content-addressed key, `<content_md5>/<original_filename>`. Expect one hash-named prefix per distinct payload rather than a flat listing. Derived text keeps its existing flat `text_<md5>.txt` name. See [Hash-based file storage](/core-concepts/main-operations/legacy-operations/add) for what this means for deduplication.
Cognee raises botocore's per-client connection pool to 32, above the default per-dataset item concurrency of 20, so concurrent items are limited by the network rather than by the pool. This is a fixed internal value — there is no environment variable for it.
<Columns>
<Card title="Core Concepts" icon="brain" href="/core-concepts/overview">
Understand knowledge graph fundamentals
</Card>
<Card title="Setup Configuration" icon="settings" href="/setup-configuration/overview">
Configure providers and databases
</Card>
<Card title="API Reference" icon="code" href="/api-reference/introduction">
Explore API endpoints
</Card>
</Columns>
# Schema Inventory
Source: https://docs.cognee.ai/guides/schema-inventory
Summarize your knowledge graph by semantic type with per-type counts, samples, and relationships
`get_schema_inventory()` summarizes your knowledge graph by semantic type instead of rendering every node. It returns deterministic per-type instance counts, a bounded set of representative sample names, and the relationship distribution between types — useful for understanding the shape of a graph at a glance or for driving custom dashboards.
**Before you start:**
* Complete [Quickstart](getting-started/quickstart) to understand basic operations
* Have some remembered data or any existing knowledge graph
## Code in Action
```python theme={null}
import cognee
# Data must be remembered/loaded into cognee before building the inventory,
# e.g. via cognee.remember() or cognee.add() + cognee.cognify().
inventory = await cognee.get_schema_inventory(
dataset=None, # optional dataset UUID to scope the graph databases
samples_per_type=5, # max sample instance names per type (default 5)
sort="count", # "count" (default, descending) or "none" (discovery order)
)
```
Each list entry is a dict describing one semantic type:
```python theme={null}
{
"type": "Person", # semantic type name
"count": 42, # total instances of this type
"samples": ["Carlos", ...], # up to samples_per_type representative names
"sample_size": 5, # number of names actually returned
"relationships": [ # aggregated edges involving this type
{"to_type": "Broker", "relation": "works_at", "count": 12},
# incoming edges are shown with a "← " prefix on the relation name
],
}
```
## How types are resolved
Extracted entities are grouped under their resolved semantic type (the `EntityType` reached via the `is_a` edge) rather than the generic `"Entity"` label, and the internal `EntityType` taxonomy nodes are not surfaced as their own group. Passing a negative `samples_per_type`, or a `sort` value other than `"count"`/`"none"`, raises `ValueError`.
## Over HTTP
The same projection is available at `GET /api/v1/schema/inventory` (query params `dataset_id`, `samples_per_type`, `sort`). The endpoint is caller-scoped: it returns `403` when the caller is not authorized to read the dataset, and `409` if the inventory cannot be built.
## Full Example
A runnable guide script ingests a handful of sentences about one domain with `remember()` — so several distinct types show up — and renders the graph with the schema side panel to an HTML file next to the script. It needs a working `LLM_API_KEY`:
* [`examples/guides/schema_inventory.py`](https://github.com/topoteretes/cognee/blob/dev/examples/guides/schema_inventory.py) → `examples/guides/.artifacts/schema_inventory.html`
The complete flow — remember data, then summarize the graph by type is in the following example:
<Accordion title="Schema inventory">
```python theme={null}
import asyncio
import cognee
async def main():
# Prune data and system metadata before running, only if we want "fresh" state.
await cognee.forget(everything=True)
# Data must be remembered/loaded into cognee before building the inventory.
await cognee.remember(
["Alice knows Bob.", "NLP is a subfield of CS."],
self_improvement=False,
)
inventory = await cognee.get_schema_inventory(samples_per_type=5, sort="count")
for entry in inventory:
print(f"{entry['type']}: {entry['count']} instance(s), samples={entry['samples']}")
for rel in entry["relationships"]:
print(f" {rel['relation']} -> {rel['to_type']} ({rel['count']})")
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
<Columns>
<Card title="Graph Visualization" icon="network" href="/guides/graph-visualization">
Render your knowledge graph to an interactive HTML file
</Card>
<Card title="Reading the Visualization" icon="eye" href="/guides/reading-the-visualization">
What the rendered file's Schema tab shows, alongside its other views
</Card>
<Card title="Memory Provenance" icon="folder-tree" href="/guides/memory-provenance">
Visualize the ownership and data-flow story behind your memory
</Card>
</Columns>
# Search Basics
Source: https://docs.cognee.ai/guides/search-basics
Step-by-step guide to running your first Cognee search and understanding core parameters
A minimal guide to querying Cognee memory. The current top-level flow uses `cognee.recall()`, while the lower-level `cognee.search()` API remains available when you need direct retriever control.
**Before you start:**
* Complete [Quickstart](../getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](../setup-configuration/llm-providers) configured for LLM-backed retrieval types
* Run `cognee.remember(...)` or otherwise build the graph before querying it
* Keep at least one dataset with `read` permission for the user running the search
## Code in Action
```python theme={null}
await cognee.remember(
[
"Alice moved to Paris in 2010. She works as a software engineer.",
"Bob lives in New York. He is a data scientist.",
"Alice and Bob met at a conference in 2015.",
],
self_improvement=False,
)
answers = await cognee.recall(query_text="What are the main themes in my data?")
for answer in answers:
print(answer)
```
<Info>
When you call `recall()` without an explicit `query_type`, Cognee uses its default auto-routing behavior to choose the best retrieval strategy for the query. To learn more about that routing behavior and the available lower-level search types, see [Recall](/core-concepts/main-operations/recall) and [Search Types](/core-concepts/main-operations/legacy-operations/search).
</Info>
## Parameters Reference
All examples below assume you are inside an async function. Import helpers when needed:
```python theme={null}
from cognee import SearchType
from cognee.modules.engine.models.node_set import NodeSet
```
<AccordionGroup>
<Accordion title="Core and Prompt Parameters">
* **`query_text`** (str, required): The question or phrase to search for. On graph-completion, hybrid, and agentic searches (including the default `HYBRID_COMPLETION`), it must be a string that still has content after trimming, so a whitespace-only query such as `" "` or `"\t\n"` is rejected with `QueryValidationError` instead of being embedded and matched against nearest neighbours; other search types such as `CHUNKS` and `SUMMARIES` do not apply this check. Trimming is used only for that validity check — the string you pass is forwarded to retrieval unchanged.
```python theme={null}
answers = await cognee.recall(query_text="Who owns the rollout plan?")
```
* **`query_type`** (SearchType, optional): Sets the retrieval mode. With `recall()`, omitting it enables auto-routing by default; if you pass it explicitly, that strategy is used directly. See [Search Types](/core-concepts/main-operations/legacy-operations/search) for the full list and [Retrievers](/core-concepts/main-operations/legacy-operations/search#retrievers) for how each type maps to a retriever. To pick a type by latency and recall depth, see the [speed, cost, and recall depth comparison](/python-api/search-type#speed-cost-and-recall-depth).
```python theme={null}
await cognee.recall(
query_text="List coding guidelines",
query_type=SearchType.CODING_RULES,
)
```
* **`top_k`** (int, optional, default: 15): Maximum number of results to return. It must be a positive integer, or `None` — which falls back to the retriever's own default (`5`) for graph-completion searches but removes the result cap entirely for vector searches such as `CHUNKS` and `SUMMARIES`. `top_k=0` or a negative value raises `QueryValidationError` for every search type, before any retriever is constructed.
```python theme={null}
await cognee.recall(query_text="Summaries please", top_k=3)
```
* **`system_prompt_path`** (str, optional, default: `"answer_simple_question.txt"`): Path to a prompt file packaged with your project.
```python theme={null}
await cognee.recall(
query_text="Explain the roadmap in bullet points",
system_prompt_path="prompts/bullets.txt",
)
```
* **`system_prompt`** (Optional\[str]): Inline prompt string. Overrides `system_prompt_path` when set.
```python theme={null}
await cognee.recall(
query_text="Give me a confident answer",
system_prompt="Answer succinctly and state confidence at the end.",
)
```
* **`only_context`** (bool, optional, default: False): Skip the LLM completion step and return the retrieved context directly. This avoids the final LLM call and is useful when you want to inspect or reuse the context yourself.
```python theme={null}
import asyncio
import cognee
async def main():
results = await cognee.recall(
query_text="What did we promise the client?",
only_context=True,
)
if isinstance(results, str):
# Single-dataset searches may unwrap to a plain string.
print(results)
elif results and isinstance(results[0], dict):
# With access control enabled, each item is grouped by dataset.
for dataset_result in results:
print("Dataset:", dataset_result["dataset_name"])
print("Context:", dataset_result["search_result"])
else:
# Otherwise, results is typically a list of context strings.
for context_text in results:
print(context_text)
asyncio.run(main())
```
<Note>
`only_context=True` works with any search type. For LLM-completion types (`GRAPH_COMPLETION`, `RAG_COMPLETION`, etc.) it returns the retrieved context — not the whole prompt, which also carries session guidance, conversation history, and the rendered templates. Pass [`context_format="prompt"`](/python-api/search#the-prompt-envelope) to get that envelope instead. For retrieval-only types (`CHUNKS`, `SUMMARIES`) the behavior is effectively unchanged because no final LLM call is made.
</Note>
</Accordion>
<Accordion title="Advanced Parameters">
* **`wide_search_top_k`** (int, optional, default: `None` — graph retrievers use 100): Caps initial candidate retrieval for graph-completion retrievers before ranking. Increase for broader recall on large graphs.
* **`triplet_distance_penalty`** (float, optional, default: `None` — graph retrievers use 6.5): Penalty applied in graph retrieval ranking. Controls how triplet distance influences final result ordering.
<Warning>
Both are **graph-only knobs**: `HYBRID_COMPLETION` — the default for `search()`, and `recall()`'s fallback when auto-routing matches nothing — rejects any explicit value with a `CogneeValidationError` (HTTP 422):
```text theme={null}
InvalidHybridSearchConfig: wide_search_top_k requires query_type=SearchType.GRAPH_COMPLETION.
```
Pin a `query_type` that honors them: the `GRAPH_COMPLETION` family, `TEMPORAL`, or `AGENTIC_COMPLETION` (`RAG_COMPLETION` takes `wide_search_top_k` only). `None` is the same as omitting the parameter.
</Warning>
* **`retriever_specific_config`** (dict, optional): Per-retriever options. Examples: `response_model` for typed LLM output; `max_iter` for GRAPH\_COMPLETION\_COT; `context_extension_rounds` for GRAPH\_COMPLETION\_CONTEXT\_EXTENSION. For COT latency tuning, see [GRAPH\_COMPLETION\_COT](/core-concepts/main-operations/legacy-operations/search#graph_completion_cot) and the [SearchType speed comparison](/python-api/search-type#speed-cost-and-recall-depth).
```python theme={null}
await cognee.recall(
query_text="What is the current state of the project?",
query_type=SearchType.GRAPH_COMPLETION,
datasets=["project_memory"],
retriever_specific_config={
"include_global_context_index": True,
"global_context_index_top_k": 3,
},
)
```
Use `include_global_context_index` after building the index with `cognee.improve(..., build_global_context_index=True)`. See [Global Context Index](/core-concepts/further-concepts/global-context-index).
<Note>
For lower-level `search()` calls, a non-string `response_model` keeps the structured completion in the result payload as a model instance, plain `dict`, or list of those. Model instances serialize to dictionaries when the payload is dumped. With `recall()`, graph completions are normalized into `ResponseGraphEntry` objects: the validated payload lands in `result.structured` (mirrored in `result.raw`), and `result.text` stays renderable for display. `recall()` also accepts the model directly via its first-class [`response_model` parameter](/python-api/recall#structured-output-with-response_model).
</Note>
* **`verbose`** (bool, optional, default: False): When `true`, results include `text_result`, `context_result`, and `objects_result` fields alongside the answer.
* **`include_references`** (bool, optional, default: True): When `true`, completion-style answers (`GRAPH_COMPLETION`, `RAG_COMPLETION`, etc.) get a deterministic `Evidence:` block appended to the answer text, citing the source chunks or graph context. Set to `False` to restore the exact prior answer text. See [Citation and Source Tracking](#citation-and-source-tracking) for details.
```python theme={null}
# Disable the appended Evidence block
await cognee.recall(
query_text="What did we promise the client?",
include_references=False,
)
```
</Accordion>
<Accordion title="Node Sets & Filtering Parameters">
These options scope retrieval to specific node sets. With `recall()`, pass `node_name` and optionally `node_name_filter_operator` — use the same names you passed to `cognee.add(..., node_set=[...])`. See [NodeSets](/core-concepts/further-concepts/node-sets) for background.
**`node_name`** (Optional\[List\[str]]): Names of the node sets to include.
<Accordion title="`node_name` example">
```python theme={null}
await cognee.recall(
query_text="What discounts did TechSupply offer?",
node_name=["vendor_conversations"],
)
```
</Accordion>
**`node_name_filter_operator`** (str, optional, default: `"OR"`): Controls how multiple node-set names are combined. `"OR"` returns results connected to **any** of the listed node sets; `"AND"` returns results connected to **all** of them.
<Accordion title="`node_name_filter_operator` example">
```python theme={null}
# OR (default) — results touching any of the listed node sets
await cognee.recall(
query_text="Summarize procurement rules",
node_name=["procurement_policies", "purchase_history"],
node_name_filter_operator="OR",
)
# AND — results that belong to every listed node set
await cognee.recall(
query_text="What topics span both domains?",
node_name=["procurement_policies", "purchase_history"],
node_name_filter_operator="AND",
)
```
</Accordion>
<Note>
Node-set filtering applies to graph-completion search types (`GRAPH_COMPLETION`, `GRAPH_COMPLETION_COT`, `GRAPH_COMPLETION_CONTEXT_EXTENSION`, `GRAPH_SUMMARY_COMPLETION`, `TEMPORAL`, `RAG_COMPLETION`, `TRIPLET_COMPLETION`, `CHUNKS`). It has no effect on `SUMMARIES`, `CYPHER`, or `NATURAL_LANGUAGE`.
</Note>
</Accordion>
<Accordion title="Interaction & History Parameters">
* **`session_id`** (Optional\[str]): Links this recall to a conversation session. With `recall()`, passing `session_id` by itself makes Cognee search session cache entries first; if nothing matches, it falls through to graph retrieval. When `session_id` is reused across completion-style recalls, previous Q\&A turns can also be included in the prompt context. If omitted while caching is enabled, Cognee writes to the dataset-scoped default session — `default_session_<dataset_id>` when the dataset is known, falling back to the global `default_session` when it is not — so recalls against different datasets never share one session. See [What Is a Session?](/core-concepts/sessions-and-caching#what-is-a-session).
```python theme={null}
# Session-first recall: checks session cache before graph search
await cognee.recall(
query_text="Where does Alice live?",
session_id="conversation_1"
)
# Add datasets when you want to force graph-backed recall in session-aware flows
await cognee.recall(
query_text="What does she do for work?",
datasets=["people_demo"],
session_id="conversation_1"
)
```
See [Sessions Guide](/guides/sessions) for complete examples. To record feedback on answers, see the [Feedback System](/guides/feedback-system).
</Accordion>
<Accordion title="Datasets & Users">
* **`datasets`** (Optional\[Union\[list\[str], str]]): Limit search to specific dataset names.
```python theme={null}
await cognee.recall(
query_text="Key risks",
datasets=["risk_register", "exec_summary"],
)
```
* **`dataset_ids`** (Optional\[Union\[list\[UUID], UUID]]): Same as `datasets`, using UUIDs instead of names.
```python theme={null}
from uuid import UUID
await cognee.recall(
query_text="Customer feedback",
dataset_ids=[UUID("aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee")],
)
```
<Warning>
With backend access control enabled, `datasets=["name"]` resolves names only within datasets owned by the current user. If Bob is searching Alice's shared dataset `shared_dataset`, searching by name can fail even when Bob has read access. Use `dataset_ids=[shared_id]` for shared datasets the user did not create.
</Warning>
* **`user`** (Optional\[User]): The user to run the search as. Required for multi-tenant flows or background jobs.
```python theme={null}
from cognee.modules.users.methods import get_user
user = await get_user(user_id)
await cognee.recall(query_text="Team OKRs", user=user)
```
**When** `ENABLE_BACKEND_ACCESS_CONTROL=true`:
* **Result shape**: Searches run only on datasets the user can access. Results are returned as a list of per-dataset objects (`dataset_name`, `dataset_id`, `search_result`). Use `verbose=True` to include `text_result`, `context_result`, and `objects_result` in each item.
* **Parallel execution**: Multiple datasets are searched concurrently using `asyncio.gather()` — total time is roughly that of the slowest single-dataset search.
* If no `user` is given, `get_default_user()` is used (created if missing); an error is raised only if this user lacks dataset permissions.
* If `datasets` is not set, all datasets readable by the user are searched. An error is raised if none are accessible or if a requested dataset is forbidden.
<Warning>
`PermissionDeniedError` will be raised unless you search with the same user that added the data or grant access to the default user.
</Warning>
**When** `ENABLE_BACKEND_ACCESS_CONTROL=false`:
* Dataset filters (`datasets`, `dataset_ids`) are ignored — all data is searched.
* Results are returned as a plain list (e.g. `["answer1", "answer2"]`). If only one dataset is searched and the retriever returns a list, Cognee may unwrap one level for backwards compatibility.
</Accordion>
</AccordionGroup>
## Citation and Source Tracking
Provenance is available at two levels:
1. **Dataset level** — when `ENABLE_BACKEND_ACCESS_CONTROL=true`, results are wrapped with `dataset_name` and `dataset_id`.
2. **Chunk/summary level** — `CHUNKS` and `SUMMARIES` results include an `id` you can use to look up the item in the graph.
### Evidence block (`include_references`)
For completion-style answers (`GRAPH_COMPLETION`, `RAG_COMPLETION`, and similar), Cognee appends a short `Evidence:` block to the answer text by default (`include_references=True`). It lists the source chunks behind the answer — for example, `- chunk 2 of document policy.pdf: "…"`.
* **Deterministic and in-process**: the block is assembled locally, with **no extra LLM call** and no prompt injection — it is appended after completion generation. `RAG_COMPLETION` cites the retrieved chunks the LLM actually read. `GRAPH_COMPLETION` cites the chunks that support the graph edges placed in the LLM context, resolved through the [edge-evidence sidecar](/setup-configuration/overview#edge-evidence); it no longer runs the answer text as a vector query, so its bullets carry no text snippet.
* **Structured evidence too**: besides the text block, the result carries an `evidence` list (`metadata.evidence` on `recall()` results) of structured references to the chunks, graph nodes, and graph edges placed in context. Non-string `response_model` paths keep their answer untouched and still receive the list.
* **Graceful degradation**: `RAG_COMPLETION` chunk evidence needs the vector payload to carry `document_name`/`document_id`; `GRAPH_COMPLETION` evidence needs the sidecar table, which ships as an Alembic migration. When neither yields a citation the text block is omitted silently (no errors).
* **Opting out**: set `include_references=False` to restore the exact previous answer text. This is useful when you compare against stored snapshots or evaluation baselines, which will otherwise diff against the new Evidence block.
```python theme={null}
# Default: answer text includes an appended "Evidence:" block
await cognee.recall(query_text="What is the refund policy?")
# Restore the exact prior output (no Evidence block)
await cognee.recall(
query_text="What is the refund policy?",
include_references=False,
)
```
<Note>
**Search results do not include raw source file paths.** Evidence can show document names when reference metadata is available, but `raw_data_location` stays on the `Document` node. To trace a result back to the original stored file path, use dataset provenance or query the graph by `id`.
</Note>
<AccordionGroup>
<Accordion title="Chunk-level fields (`CHUNKS`)">
`SearchType.CHUNKS` returns a list of dicts with these fields:
| Field | Type | Description |
| ------------- | ----- | ------------------------------------------------------------------- |
| `id` | `str` | Chunk UUID |
| `text` | `str` | Raw chunk text |
| `chunk_index` | `int` | Position of this chunk within the source document |
| `chunk_size` | `int` | Token count of this chunk |
| `cut_type` | `str` | How the boundary was chosen (`sentence_end`, `paragraph_end`, etc.) |
```python theme={null}
import asyncio
import cognee
from cognee import SearchType
async def main():
await cognee.add("path/to/policy.pdf", dataset_name="docs")
await cognee.cognify(datasets=["docs"])
results = await cognee.recall(
query_text="What is the refund policy?",
query_type=SearchType.CHUNKS,
)
for chunk in results:
print("Text: ", chunk["text"])
print("Chunk index: ", chunk["chunk_index"])
print("Chunk ID: ", chunk["id"]) # UUID for graph lookups
print()
asyncio.run(main())
```
</Accordion>
<Accordion title="Summary-level fields (`SUMMARIES`)">
`SearchType.SUMMARIES` returns a list of dicts with these fields:
| Field | Type | Description |
| ------ | ----- | ------------ |
| `id` | `str` | Summary UUID |
| `text` | `str` | Summary text |
```python theme={null}
results = await cognee.recall(
query_text="What is the refund policy?",
query_type=SearchType.SUMMARIES,
)
for summary in results:
print("Summary:", summary["text"])
print("ID: ", summary["id"])
```
</Accordion>
<Accordion title="Look up a chunk or node directly by ID">
For `CHUNKS` and `SUMMARIES`, `recall()` (and the lower-level `cognee.search()`) use `query_text` as a retrieval query, not as an identifier lookup. Passing a chunk or summary `id` as the query searches for semantically similar content and can return unrelated results (or nothing) — not that specific item.
For most workflows, keep the `id` as provenance metadata and use normal `recall()` queries to retrieve related content. If you need to inspect a specific chunk, summary, or graph node during debugging, query the databases directly:
```python theme={null}
import asyncio
from cognee.infrastructure.databases.graph import get_graph_engine
from cognee.infrastructure.databases.vector import get_vector_engine_async
async def main():
item_id = "your-chunk-or-summary-uuid" # the `id` returned by a CHUNKS or SUMMARIES search
# From the graph: returns the node's properties, or None if not found
graph_engine = await get_graph_engine()
node = await graph_engine.get_node(item_id)
print(node)
# From the vector store: choose the collection for the search type
vector_engine = await get_vector_engine_async()
collection = "DocumentChunk_text" # CHUNKS results
# collection = "TextSummary_text" # SUMMARIES results
rows = await vector_engine.retrieve(collection, [item_id])
if rows:
print(rows[0].payload["text"])
asyncio.run(main())
```
Use `get_node(node_id)` / `get_nodes(node_ids)` for graph lookups and `retrieve(collection_name, data_point_ids)` for vector-store lookups. Use `DocumentChunk_text` for `CHUNKS` IDs and `TextSummary_text` for `SUMMARIES` IDs.
<Note>
`get_graph_engine()` and `get_vector_engine_async()` are async (use `await` for both); the vector engine's `retrieve` method is also async. The older synchronous `get_vector_engine()` still exists as a deprecated backward-compatibility shim — calling it emits a `DeprecationWarning`, so use `get_vector_engine_async()` in new code. With `ENABLE_BACKEND_ACCESS_CONTROL=true`, these getters return the default databases — direct lookups bypass the per-dataset isolation applied by `recall()`.
</Note>
</Accordion>
<Accordion title="Traverse from a chunk to its source document">
Each `DocumentChunk` has an `is_part_of` relationship to its parent document. The chunk also carries a flat `document_id` field, so use that field when you already have the chunk object and only need the parent document id.
For debugging or graph inspection, get the graph engine directly with `get_graph_engine()` and traverse from the chunk node.
```python theme={null}
from cognee.infrastructure.databases.graph import get_graph_engine
graph_engine = await get_graph_engine()
# get_connections returns (source, edge, target) triples for every edge on the node
connections = await graph_engine.get_connections(str(chunk_id))
parent_documents = [
target
for source, edge, target in connections
if edge["relationship_name"] == "is_part_of"
]
```
</Accordion>
<Accordion title="Dataset-level provenance (access control enabled)">
When `ENABLE_BACKEND_ACCESS_CONTROL=true`, every result is wrapped with dataset information:
```python theme={null}
import asyncio
import cognee
from cognee import SearchType
async def main():
results = await cognee.recall(
query_text="What is the refund policy?",
query_type=SearchType.CHUNKS,
datasets=["docs"],
)
for dataset_result in results:
print("Dataset: ", dataset_result["dataset_name"])
print("Dataset ID:", dataset_result["dataset_id"])
for chunk in dataset_result["search_result"]:
print(" Text:", chunk["text"])
print(" Chunk ID:", chunk["id"])
asyncio.run(main())
```
When `ENABLE_BACKEND_ACCESS_CONTROL=false`, results are a plain list with no `dataset_name` or `dataset_id` wrapper.
</Accordion>
<Accordion title="Raw source objects (LLM-completion modes)">
For modes that return a generated answer (`GRAPH_COMPLETION`, `RAG_COMPLETION`, etc.), use `verbose=True` to receive the raw retrieved objects alongside the answer:
```python theme={null}
results = await cognee.recall(
query_text="Summarize the launch timeline",
verbose=True,
)
for result in results:
print("Answer: ", result.get("text_result"))
print("Context passed: ", result.get("context_result"))
print("Source objects: ", result.get("objects_result"))
```
</Accordion>
</AccordionGroup>
## Writing Effective Queries
A good query does two jobs at once: it tells Cognee **what** you want and, through its wording, hints at **how** to retrieve it. Follow these practices to get better answers.
* **Ask a full, natural-language question.** `recall()` and the graph-completion retrievers are LLM-backed, so `"What discounts did TechSupply offer in 2023?"` works better than a bare keyword like `"discounts"`. Reserve short keyword strings for `CHUNKS_LEXICAL` or `CHUNKS`.
* **Let phrasing steer auto-routing, or set `query_type` yourself.** When you omit `query_type`, a rule-based router reads cue words in your query and picks a strategy — summary words (`summary`, `overview`, `key takeaways`) lean toward summary completion, reasoning words (`why`, `explain`, `step by step`) toward chain-of-thought, relationship words (`how are X and Y connected`, `path between`) toward context extension, and time words (`when`, `before`, `after`, or a 4-digit year) toward temporal search. If no cue matches, it defaults to `HYBRID_COMPLETION`. When you know the mode you want, pass `query_type` explicitly to bypass the router. See [Auto-routing behavior](/core-concepts/main-operations/recall#examples-and-details) and [Choosing a Search Type](/python-api/search-type#choosing-a-search-type).
* **Phrase positively.** A negation (`not`, `no`, `never`, `without`) within \~20 characters before a cue word suppresses that cue, so `"summary without dates"` will not route to summary. Rephrase or set `query_type` when a query needs negated wording.
* **Scope the query to narrow results.** Pass `datasets` to limit which knowledge base is searched and `node_name` to restrict retrieval to specific [node sets](/core-concepts/further-concepts/node-sets). Tighter scope means less noise and faster answers.
* **Tune the call to the task.** Lower `top_k` for concise answers, raise it for broad recall. Use `only_context=True` when you want the retrieved context without an LLM answer (useful for inspecting retrieval quality); add [`context_format="prompt"`](/python-api/search#the-prompt-envelope) when you are feeding your own model and want the session guidance and rendered prompt too. Match the search type to your latency and cost budget using the [speed, cost, and recall depth comparison](/python-api/search-type#speed-cost-and-recall-depth).
## Full Example
<Accordion title="Latest guide">
```python theme={null}
import asyncio
import cognee
async def main():
# Start clean (optional in your app)
await cognee.forget(everything=True)
# Prepare knowledge base
await cognee.remember(
[
"Alice moved to Paris in 2010. She works as a software engineer.",
"Bob lives in New York. He is a data scientist.",
"Alice and Bob met at a conference in 2015.",
],
self_improvement=False,
)
# Make sure you've already run cognee.remember(...) so the graph has content
answers = await cognee.recall(query_text="What are the main themes in my data?")
for answer in answers:
print(answer)
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
<Accordion title="Legacy guide">
```python theme={null}
import asyncio
import cognee
async def main():
# Start clean (optional)
await cognee.prune.prune_data()
await cognee.prune.prune_system(metadata=True)
await cognee.add(
[
"Alice moved to Paris in 2010. She works as a software engineer.",
"Bob lives in New York. He is a data scientist.",
"Alice and Bob met at a conference in 2015.",
]
)
await cognee.cognify()
answers = await cognee.recall(query_text="What are the main themes in my data?")
for answer in answers:
print(answer)
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
<Note>
`recall()` is the current high-level retrieval entry point. It routes the query to the best available retrieval strategy and can still use graph-backed search under the hood.
</Note>
## Additional Examples
Additional examples are available on our [GitHub](https://github.com/topoteretes/cognee/tree/main/examples/guides).
* An advanced script running this same `forget()` → `remember()` → `recall()` flow over a full real document is also on our [GitHub](https://github.com/topoteretes/cognee/blob/dev/examples/advanced_guides/simple_document_qa/simple_document_qa_demo.py). Instead of the three inline sentences used above, it passes a file path to `remember()` — the bundled full text of *Alice in Wonderland* — and then asks three plain `recall()` questions whose answers depend on reading across the whole book.
<Columns>
<Card title="Custom Prompts" icon="text-wrap" href="/guides/custom-prompts">
Learn about custom prompts for tailored answers
</Card>
<Card title="Permission Snippets" icon="shield" href="/guides/permission-snippets">
Multi-tenant deployment patterns
</Card>
<Card title="API Reference" icon="code" href="/api-reference/introduction">
Explore all search types and parameters
</Card>
<Card title="Sessions" icon="message-square" href="/guides/sessions">
Enable conversational memory with sessions
</Card>
<Card title="Agent Memory Decorator" icon="bot" href="/core-concepts/further-concepts/agent-memory-decorator">
Attach retrieval to an agent function boundary
</Card>
</Columns>
# Self-Improvement Quickstart
Source: https://docs.cognee.ai/guides/self-improvement-quickstart
Step-by-step guide to enriching memory and bridging session content with improve
A minimal guide to running a self-improvement pass over existing memory so session-only content becomes part of the permanent dataset. In the current API, this user-facing flow goes through `improve()`, which uses Memify-style enrichment under the hood.
**Before you start:**
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured
* Have an existing dataset or be ready to create one with `remember()`
* Have session content you want to bridge into permanent memory
## What Self-Improvement Does
* Enriches an existing dataset instead of re-ingesting all source data
* Bridges session memory into the permanent graph when you pass `session_ids`
* Distills accepted session guidance into `session_learnings` when the session has durable, gated lessons
* Improves later `recall()` results by adding retrieval-ready structures to the dataset
## Code in Action
### Step 1: Store Permanent Memory
```python theme={null}
await cognee.remember(
"Einstein developed general relativity.",
dataset_name=DATASET,
self_improvement=False,
)
```
This creates durable graph memory in `demo_dataset`. Setting `self_improvement=False` keeps the example focused on the explicit `improve()` call later.
### Step 2: Store Session-only Memory
```python theme={null}
await cognee.remember(
"Niels Bohr worked on atomic structure.",
dataset_name=DATASET,
session_id=SESSION,
self_improvement=False,
)
```
This writes the Bohr fact into session memory under `demo_session` instead of immediately pushing it into the permanent graph.
### Step 3: Recall Before Improvement
```python theme={null}
answer_before_improve = await cognee.recall(
"What did Bohr work on?",
datasets=[DATASET],
)
```
At this point, the permanent dataset may not yet know about the session-only Bohr fact, so the recall result can be empty or incomplete.
### Step 4: Bridge and Enrich with Improve
```python theme={null}
await cognee.improve(dataset=DATASET, session_ids=[SESSION])
```
This runs the improvement pass for `demo_dataset`, bridging the gap between short-term (session) memory and long term (permanent) memory by pulling in the specified session and enriching the graph.
If the session accumulated durable guidance during conversation, the same pass can also distill accepted lessons into `session_learnings`.
### Step 5: Recall After Improvement
```python theme={null}
answer_after_improve = await cognee.recall(
"What did Bohr work on?",
datasets=[DATASET],
)
```
After `improve()` finishes, the permanent dataset can answer from the newly bridged session content.
## What Changed in Your Graph
After `improve()` completes, the dataset can include:
* Session-derived content from `demo_session` persisted into the permanent graph
* Distilled session-learning documents tagged as `session_learnings`, when the session contains accepted guidance
* Additional derived retrieval structures created during the enrichment pass
* Better downstream recall for facts that were previously only available in the session
<Accordion title="Parameters">
- **`dataset`** (`str`, default: `"main_dataset"`) — the dataset to improve.
- **`session_ids`** (`Optional[List[str]]`) — session IDs whose cached memory should be bridged into the permanent graph.
- **`run_in_background`** (`bool`, default: `False`) — if `True`, returns immediately and runs improvement asynchronously.
- **`node_name`** (`Optional[List[str]]`) — narrows the improvement pass to specific named nodes or node sets.
- **`feedback_alpha`** (`float`, default depends on runtime config) — controls how strongly session feedback affects graph weighting when feedback data exists.
</Accordion>
## Customizing Tasks (Optional)
```python theme={null}
await cognee.improve(
dataset=DATASET,
session_ids=[SESSION],
extraction_tasks=[...],
enrichment_tasks=[...],
)
```
You can override the default extraction and enrichment tasks when you need domain-specific improvement behavior.
## What Happens Under the Hood
When `session_ids` are provided, the improvement flow can:
* apply feedback-based weighting updates to graph elements used during retrieval
* persist session memory into the permanent graph
* persist agent trace steps and distill accepted session guidance into `session_learnings`
* run the enrichment pass on the target dataset
* sync new graph context back into the session cache
`improve()` uses Memify for its enrichment stage, which is why this guide keeps its historical `memify-*` path while demonstrating the current way to trigger self-improvement.
## Additional Information
* Runnable guide script available on our [GitHub](https://github.com/topoteretes/cognee/blob/main/examples/guides/improve_quickstart.py)
* An advanced end-to-end script touring the whole memory API — `remember()`, `recall()`, `improve()`, and `forget()`, including session-aware `recall()` — is also on our [GitHub](https://github.com/topoteretes/cognee/blob/dev/examples/advanced_guides/remember_recall_improve_example.py). Unlike the examples above, it leaves `self_improvement` at its default (`True`), so its session `remember()` calls also bridge into the permanent graph in the background; its explicit `improve()` call is still what enriches its `scientists` dataset, because the session calls default to `main_dataset`.
<Accordion title="Latest guide">
```python theme={null}
import asyncio
import cognee
DATASET = "demo_dataset"
SESSION = "demo_session"
async def main():
await cognee.forget(everything=True)
await cognee.remember(
"Einstein developed general relativity.",
dataset_name=DATASET,
self_improvement=False,
)
await cognee.remember(
"Niels Bohr worked on atomic structure.",
dataset_name=DATASET,
session_id=SESSION,
self_improvement=False,
)
answer_before_improve = await cognee.recall(
"What did Bohr work on?",
datasets=[DATASET],
)
await cognee.improve(dataset=DATASET, session_ids=[SESSION])
answer_after_improve = await cognee.recall(
"What did Bohr work on?",
datasets=[DATASET],
)
print("Before improve:", answer_before_improve)
print("After improve:", answer_after_improve)
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
<Accordion title="Legacy guide">
```python theme={null}
import asyncio
import cognee
from cognee.modules.search.types import SearchType
async def main():
# 1) Add two short chats and build a graph
await cognee.add([
"We follow PEP8. Add type hints and docstrings.",
"Releases should not be on Friday. Susan must review PRs.",
], dataset_name="rules_demo")
await cognee.cognify(datasets=["rules_demo"]) # builds graph
# 2) Enrich the graph (uses default memify tasks)
await cognee.memify(dataset="rules_demo")
# 3) Query the new coding rules
rules = await cognee.search(
query_type=SearchType.CODING_RULES,
query_text="List coding rules",
node_name=["coding_agent_rules"],
)
print("Rules:", rules)
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
<Accordion title="Troubleshooting">
* **No visible change after `improve()`** — make sure the session ID you pass actually contains session memory and that `self_improvement=False` did not leave the content unbridged.
* **Empty recall results after improvement** — verify that you are recalling against the same dataset you improved.
* **Error: no graph data found** — create the base dataset first with `cognee.remember(..., dataset_name=...)`.
* **LLM errors** — verify that your LLM provider is configured correctly. See [LLM Providers](/setup-configuration/llm-providers).
* **Permission errors** — the user must have write access to the target dataset. See [Permissions](/core-concepts/multi-user-mode/permissions-system/datasets).
</Accordion>
<Note>
This updated example uses one permanent fact and one session-only fact for demonstration. In practice, you can bridge larger sessions and then run additional enrichment on the same dataset.
</Note>
<Columns>
<Card title="Improve" icon="sparkles" href="/core-concepts/main-operations/improve">
Understand the current improvement workflow
</Card>
<Card title="Remember" icon="brain" href="/core-concepts/main-operations/remember">
Store permanent and session memory
</Card>
<Card title="Sessions" icon="message-square" href="/guides/sessions">
Learn how session memory behaves before improvement
</Card>
</Columns>
# Session Distillation
Source: https://docs.cognee.ai/guides/session-distillation
Turn guidance stated in a session into permanent lessons in the knowledge graph so future recalls respect it
Guidance a user states during a session — preferences, corrections, durable instructions — is short-term by default: it lives in the session cache and expires with it. **Session distillation** bridges that guidance into long-term memory by turning a session's accepted, gated guidance into permanent lesson documents in the knowledge graph, tagged with the `session_learnings` node set. A brand-new session with no memory of the original conversation will then recall and respect what was learned.
**Before you start:**
* Complete [Quickstart](/getting-started/quickstart) or have Cognee installed and configured
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured
* Be familiar with [Sessions](/guides/sessions) and the `remember()` / `recall()` workflow
* [Caching must be enabled](/core-concepts/sessions-and-caching#cache-adapters) so session Q\&A history is retained
## How to Distill a Session
Sessions can be bridged into the knowledge graph with `improve()`, which can persist cached Q\&A, persist agent traces, and distill accepted session guidance into `session_learnings`:
```python theme={null}
await cognee.improve(
dataset="project_memory",
session_ids=["conversation_1"],
)
```
If you only want to distill one finished session's gated guidance, call `cognee.session.distill_session()` directly:
```python theme={null}
result = await cognee.session.distill_session(
"conversation_1",
dataset="project_memory",
)
```
The direct result includes `status` and `documents`. A completed run writes accepted lesson documents back into the dataset; statuses such as `no_gated_entries` or `no_accepted_lessons` mean there was nothing durable enough to persist.
## Example: Learning a Snack Preference
A minimal end-to-end demo of session distillation: state a preference in one session, distill that session into the knowledge graph, then confirm that a fresh session's recall now reflects the learned preference. Setting `AUTO_FEEDBACK=true` lets the session capture the user's stated preference as learned guidance during distillation.
### How It Works
**1. Remember two neutral snack facts.** The graph starts with one sweet and one savory snack — and no preference:
```python theme={null}
await cognee.remember(
[
"Oreos are a sweet snack: chocolate cookies with a sugary cream filling.",
"Doritos are a savory snack: salty, cheesy, seasoned tortilla chips.",
],
dataset_name="snack_preference_demo",
)
```
**2. Ask before distillation.** With no known preference, `recall()` (`RAG_COMPLETION`) makes an arbitrary pick:
```python theme={null}
before = await cognee.recall(
query_text="I want a snack. Should I get Oreos or Doritos? Recommend one.",
query_type=SearchType.RAG_COMPLETION,
datasets=["snack_preference_demo"],
session_id="snack_session",
)
```
**3. State the opposite preference in the same session** so the answer has to flip. Here the model picked Oreos first, so the user states a savory preference — the [full example](#full-example) below detects the first pick and states the opposite preference for either case:
```python theme={null}
await cognee.recall(
query_text="Just so you know, I always prefer savory snacks over sweet ones.",
query_type=SearchType.RAG_COMPLETION,
datasets=["snack_preference_demo"],
session_id="snack_session",
)
```
**4. Distill the session into long-term memory.** The curated preference lesson becomes a retrievable chunk in the graph:
```python theme={null}
result = await cognee.session.distill_session(
"snack_session", dataset="snack_preference_demo"
)
```
**5. Ask again in a fresh session.** A brand-new session with no memory of the conversation still recalls the preference, so the recommendation now respects it:
```python theme={null}
after = await cognee.recall(
query_text="I want a snack. Should I get Oreos or Doritos? Recommend one.",
query_type=SearchType.RAG_COMPLETION,
datasets=["snack_preference_demo"],
session_id="snack_verification_session",
)
```
### Full Example
The complete runnable script is on GitHub: [`examples/guides/session_distillation.py`](https://github.com/topoteretes/cognee/blob/dev/examples/guides/session_distillation.py). It picks the *opposite* of the model's first pick as the stated preference and then asserts the recommendation flips — a self-checking way to prove distillation changed the outcome.
<Accordion title="Full example script">
```python theme={null}
import asyncio
import os
import sys
# Let the session capture the user's stated preference as learned guidance.
os.environ["AUTO_FEEDBACK"] = "true"
os.environ.setdefault("LOG_LEVEL", "ERROR")
import cognee
from cognee import SearchType
from cognee.infrastructure.session.get_session_manager import get_session_manager
from cognee.modules.users.methods import get_default_user
SESSION_ID = "snack_session"
# flavor -> (snack that has it, statement of the preference)
SNACK_FOR_FLAVOR = {"savory": "Doritos", "sweet": "Oreos"}
def progress(message: str):
print(f"[snack-demo] {message}", file=sys.stderr, flush=True)
def answer_text(result) -> str:
"""recall() returns a list of response entries; join their text for parsing/printing."""
if isinstance(result, str):
return result
parts = []
for entry in result or []:
parts.append(getattr(entry, "text", None) or str(entry))
return " ".join(parts)
def recommended_snack(text: str) -> str:
"""Whichever snack the model recommends first in its answer."""
lowered = text.lower()
oreo_at = lowered.find("oreo")
dorito_at = lowered.find("dorito")
if oreo_at == -1 and dorito_at == -1:
return "Oreos" # fallback; shouldn't happen with the snack facts in context
if dorito_at == -1:
return "Oreos"
if oreo_at == -1:
return "Doritos"
return "Oreos" if oreo_at < dorito_at else "Doritos"
async def ask(message: str, user, session_id: str):
# RAG_COMPLETION answers from retrieved chunks. Before distillation only the two snack
# facts exist, so the model has no basis to prefer one. After distillation the curated
# preference lesson is a retrievable chunk, so it steers the pick.
return await cognee.recall(
query_text=message,
query_type=SearchType.RAG_COMPLETION,
datasets=["snack_preference_demo"],
session_id=session_id,
user=user,
)
async def main():
progress("Clearing previous demo state.")
await cognee.prune.prune_data()
await cognee.prune.prune_system(metadata=True)
progress("Ingesting the two snack facts.")
await cognee.remember(
[
"Oreos are a sweet snack: chocolate cookies with a sugary cream filling.",
"Doritos are a savory snack: salty, cheesy, seasoned tortilla chips.",
],
dataset_name="snack_preference_demo",
)
user = await get_default_user()
await get_session_manager().delete_session(user_id=str(user.id), session_id=SESSION_ID)
question = "I want a snack. Should I get Oreos or Doritos? Recommend one."
# 1) Before distillation: no preference known -> arbitrary pick.
progress("Asking BEFORE distillation (no preference known).")
before = answer_text(await ask(question, user, SESSION_ID))
first_pick = recommended_snack(before)
print("\n----- BEFORE distillation -----\n", file=sys.stderr)
print(f"picked: {first_pick}\n{before}", file=sys.stderr)
# 2) State the OPPOSITE preference so the answer has to flip to the other snack.
if first_pick == "Doritos":
preferred_flavor, opposite_flavor = "sweet", "savory"
else:
preferred_flavor, opposite_flavor = "savory", "sweet"
expected_after = SNACK_FOR_FLAVOR[preferred_flavor]
progress(
f"Model picked {first_pick}; telling it the user prefers {preferred_flavor} "
f"(expect it to flip to {expected_after})."
)
await ask(
f"Just so you know, I always prefer {preferred_flavor} snacks over {opposite_flavor} ones.",
user,
SESSION_ID,
)
# 3) Distill the session into long-term memory.
progress("Distilling the session into the graph.")
result = await cognee.session.distill_session(
SESSION_ID, dataset="snack_preference_demo", user=user
)
progress(f"Distillation status={result.status} documents={len(result.documents)}")
for doc in result.documents:
print("\n----- distilled lesson -----\n", file=sys.stderr)
print(doc, file=sys.stderr)
# 4) After distillation, in a FRESH session, ask the same question again.
progress("Asking AFTER distillation in a fresh session.")
after = answer_text(await ask(question, user, "snack_verification_session"))
second_pick = recommended_snack(after)
print(f"\n----- AFTER distillation (expected {expected_after}) -----\n", file=sys.stderr)
print(f"picked: {second_pick}\n{after}", file=sys.stderr)
flipped = second_pick == expected_after and second_pick != first_pick
progress(
f"RESULT: {first_pick} -> {second_pick} "
f"({'flipped as expected ✅' if flipped else 'did NOT flip ❌'})"
)
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
## Going Further
Distilled `session_learnings` are the foundation for other self-improvement features: [`improve()`](/core-concepts/main-operations/improve) bridges whole sessions (Q\&A, traces, and guidance) in one pass, and [truth-subspace reranking](/guides/truth-subspace-reranking) builds retrieval anchors from distilled lessons.
An advanced companion script is on GitHub: [`examples/advanced_guides/session_distillation_demo.py`](https://github.com/topoteretes/cognee/blob/dev/examples/advanced_guides/session_distillation_demo.py). It replays an eight-message session so Q\&A history accumulates, then shows which of those older turns hybrid vector recall brings back once a plain recency window would have dropped them, and distills only afterwards — covering the retrieval half of session memory that the before/after example above leaves out.
<Columns>
<Card title="Sessions Guide" icon="message-circle" href="/guides/sessions">
Learn how sessions and caching work in Cognee
</Card>
<Card title="Improve" icon="sparkles" href="/core-concepts/main-operations/improve">
Bridge sessions into the permanent graph with the improve pass
</Card>
</Columns>
# Sessions
Source: https://docs.cognee.ai/guides/sessions
Step-by-step guide to using sessions for conversational memory in Cognee
A minimal guide to enabling conversational memory with sessions. When you use the same `session_id` across `recall()` calls, Cognee remembers previous questions and answers, enabling contextually aware follow-up questions.
## Before You Start
* Complete [Quickstart](../getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](../setup-configuration/llm-providers) configured
* Read [Sessions and Caching](../core-concepts/sessions-and-caching) for conceptual overview
* Configure your cache adapter before using sessions. See [Cache Adapters](../core-concepts/sessions-and-caching#cache-adapters) for Redis and Filesystem setup instructions.
## Code in Action
```python theme={null}
import asyncio
import cognee
from cognee import SearchType
async def main():
# Start clean (optional in your app)
await cognee.forget(everything=True)
# Prepare knowledge base
await cognee.remember(
[
"Alice moved to Paris in 2010. She works as a software engineer.",
"Bob lives in New York. He is a data scientist.",
"Alice and Bob met at a conference in 2015.",
],
self_improvement=False,
)
# First recall - starts a new session (default user is used when none is passed)
result1 = await cognee.recall(
query_text="Where does Alice live?",
query_type=SearchType.GRAPH_COMPLETION,
session_id="conversation_1",
)
print("First answer:", result1[0].text)
# Follow-up recall - uses conversation history
result2 = await cognee.recall(
query_text="What does she do for work?",
query_type=SearchType.GRAPH_COMPLETION,
session_id="conversation_1", # Same session
)
print("Follow-up answer:", result2[0].text)
# The LLM knows "she" refers to Alice from previous context
# Different session - no memory of previous conversation
result3 = await cognee.recall(
query_text="What does she do for work?",
query_type=SearchType.GRAPH_COMPLETION,
session_id="conversation_2", # New session
)
print("New session answer:", result3[0].text)
# Without conversation history, the LLM cannot tell who "she" refers to
if __name__ == "__main__":
asyncio.run(main())
```
<Note>
This example works with either Redis or Filesystem adapter. Configure your chosen adapter in the [Before you start](#before-you-start) section above.
</Note>
## What Just Happened
### Step 1: Prepare Knowledge Base
```python theme={null}
await cognee.remember(
[
"Alice moved to Paris in 2010. She works as a software engineer.",
"Bob lives in New York. He is a data scientist.",
"Alice and Bob met at a conference in 2015.",
],
self_improvement=False,
)
```
Before you can use sessions, you need data in your knowledge base. `cognee.remember()` ingests the texts and builds the knowledge graph in one call.
### Step 2: Start a Session
```python theme={null}
result1 = await cognee.recall(
query_text="Where does Alice live?",
query_type=SearchType.GRAPH_COMPLETION,
session_id="conversation_1",
)
```
The `session_id` parameter on `cognee.recall()` creates or continues a conversation thread. All recalls with the same `session_id` share conversation history.
### Step 3: Ask Follow-up Questions
```python theme={null}
result2 = await cognee.recall(
query_text="What does she do for work?",
query_type=SearchType.GRAPH_COMPLETION,
session_id="conversation_1", # Same session
)
```
When you use the same `session_id`, Cognee automatically includes previous Q\&A turns in the LLM prompt, so the LLM resolves "she" to Alice from the earlier question.
### Step 4: Isolate Conversations
```python theme={null}
result3 = await cognee.recall(
query_text="What does she do for work?",
query_type=SearchType.GRAPH_COMPLETION,
session_id="conversation_2", # New session
)
```
Each `session_id` maintains its own conversation history. This recall runs in a fresh session, so no previous turns are sent to the LLM and "she" is ambiguous.
## Advanced Usage
<Accordion title="Custom Session IDs">
Use meaningful session IDs to organize conversations:
```python theme={null}
# User-specific sessions
await cognee.recall(query_text="...", session_id=f"user_{user_id}_chat")
# Topic-specific sessions
await cognee.recall(query_text="...", session_id="project_planning")
await cognee.recall(query_text="...", session_id="bug_discussion")
```
Session IDs are arbitrary strings—use whatever naming scheme fits your application.
</Accordion>
See [Sessions and Caching](/core-concepts/sessions-and-caching) for what happens when you omit `session_id` (the default-session behavior), reading session history with `get_session()`, the `include_context` flag and `SessionManager`, disabling sessions entirely, which search types are session-aware, session persistence/clearing, and token usage tracking.
## Legacy Guide
<Accordion title="Sessions with add(), cognify(), and search()">
If you are still on the pre-1.0 `add()` / `cognify()` / `search()` surface, the same session behavior is available through `cognee.search()`. The `session_id` parameter works exactly as described above. New projects should use `remember()` and `recall()` instead — see [Search (legacy)](/core-concepts/main-operations/legacy-operations/search) for how the legacy surface relates to `recall()`.
```python theme={null}
import asyncio
import cognee
from cognee import SearchType
async def main():
# Prepare knowledge base
await cognee.add([
"Alice moved to Paris in 2010. She works as a software engineer.",
"Bob lives in New York. He is a data scientist.",
"Alice and Bob met at a conference in 2015."
])
await cognee.cognify()
# First search - starts a new session (default user is used when none is passed)
result1 = await cognee.search(
query_type=SearchType.GRAPH_COMPLETION,
query_text="Where does Alice live?",
session_id="conversation_1"
)
print("First answer:", result1[0])
# Follow-up search - uses conversation history
result2 = await cognee.search(
query_type=SearchType.GRAPH_COMPLETION,
query_text="What does she do for work?",
session_id="conversation_1" # Same session
)
print("Follow-up answer:", result2[0])
# The LLM knows "she" refers to Alice from previous context
# Different session - no memory of previous conversation
result3 = await cognee.search(
query_type=SearchType.GRAPH_COMPLETION,
query_text="What does she do for work?",
session_id="conversation_2" # New session
)
print("New session answer:", result3[0])
# Without conversation history, the LLM cannot tell who "she" refers to
asyncio.run(main())
```
</Accordion>
<Columns>
<Card title="Sessions and Caching" icon="brain" href="/core-concepts/sessions-and-caching">
Understand how sessions work conceptually
</Card>
<Card title="Search Basics" icon="search" href="/guides/search-basics">
Learn about search parameters and types
</Card>
<Card title="Setup Configuration" icon="settings" href="/setup-configuration/overview">
Configure cache adapters and providers
</Card>
</Columns>
# Remember and Recall in One Script
Source: https://docs.cognee.ai/guides/simple-cognee
Store a single passage with remember() and query it back with recall() in the smallest complete Cognee script
A minimal guide to the smallest end-to-end Cognee script. Use it as the starting point for a new project: one passage of text goes in with `remember()`, one question comes back answered with `recall()`, and nothing else is configured.
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — the answer is generated by a language model
* Read [Remember](/core-concepts/main-operations/remember) and [Recall](/core-concepts/main-operations/recall) for what the two operations do
* No data is required up front — the script ingests its own passage, but it starts with `cognee.forget(everything=True)`, which wipes all existing Cognee data; run it against a setup you can afford to reset
## Code in Action
```python theme={null}
import asyncio
import cognee
from cognee import SearchType
from cognee.shared.logging_utils import ERROR, setup_logging
async def main():
# Start clean, then remember knowledge with the v1.0 memory API.
await cognee.forget(everything=True)
text = """
Natural language processing (NLP) is an interdisciplinary
subfield of computer science and information retrieval.
"""
await cognee.remember(text, self_improvement=False)
query_text = "Tell me about NLP"
print(f"Searching cognee for insights with query: '{query_text}'")
search_results = await cognee.recall(
query_type=SearchType.GRAPH_COMPLETION, query_text=query_text
)
for result_text in search_results:
print(result_text)
if __name__ == "__main__":
logger = setup_logging(log_level=ERROR)
asyncio.run(main())
```
## What Just Happened
### Step 1: Start From a Clean Slate
```python theme={null}
await cognee.forget(everything=True)
```
`forget(everything=True)` deletes every dataset, graph node, embedding, and session-cache entry the current user owns, so the run starts with an empty graph and the answer can only come from the passage below. Leave this line out of your own application — you rarely want to erase everything before storing something new.
### Step 2: Remember the Passage
```python theme={null}
text = """
Natural language processing (NLP) is an interdisciplinary
subfield of computer science and information retrieval.
"""
await cognee.remember(text, self_improvement=False)
```
`remember()` takes raw text and runs the full ingestion workflow on it: the passage is chunked, entities and relationships are extracted, and the result is written to the graph as memory. `self_improvement=False` keeps the run to plain ingestion instead of also running `improve()`.
### Step 3: Recall an Answer
```python theme={null}
query_text = "Tell me about NLP"
print(f"Searching cognee for insights with query: '{query_text}'")
search_results = await cognee.recall(
query_type=SearchType.GRAPH_COMPLETION, query_text=query_text
)
for result_text in search_results:
print(result_text)
```
`SearchType.GRAPH_COMPLETION` retrieves the graph triplets relevant to `query_text`, builds a context from them, and asks a language model to answer from that context. `recall()` returns a list, so the loop prints each result — with this small a graph, expect a single answer describing NLP.
### Step 4: Run the Script
```python theme={null}
if __name__ == "__main__":
logger = setup_logging(log_level=ERROR)
asyncio.run(main())
```
Both operations are asynchronous, so `main()` runs under `asyncio.run()`. The `setup_logging(log_level=ERROR)` call no longer changes the console level: logging is configured once per process, on the first call, and `import cognee` already made it. To keep Cognee's own progress logging out of the output, set `LOG_LEVEL=ERROR` in your `.env` or in the environment before importing Cognee — see [Setting the Log Level](/setup-configuration/logging#setting-the-log-level).
<Columns>
<Card title="Remember" icon="brain" href="/core-concepts/main-operations/remember">
Every way to get data into Cognee memory, and what each option changes.
</Card>
<Card title="Recall" icon="search" href="/core-concepts/main-operations/recall">
Understand recall()'s full parameter surface and auto-routing behavior.
</Card>
<Card title="Inspecting Graph Completion Context" icon="route" href="/guides/graph-completion">
See the triplets GRAPH\_COMPLETION retrieved before it answered.
</Card>
</Columns>
# Store Configurations
Source: https://docs.cognee.ai/guides/store-configurations
Copy-paste .env blocks for the supported relational, vector, and graph store combinations, and how to pair any of them with your LLM and embedding provider.
Cognee keeps memory in three stores: a **relational** database for metadata, a **vector** store for embeddings, and a **graph** store for entities and their relationships. Each is configured on its own, so a working setup is always a *combination* — and the combination is what the reference pages leave to you to assemble.
This page closes that gap. Every stack below is one complete `.env` block: paste it, fill in your LLM API key, install the listed extras, start the listed server, and run the same [verification script](#verify-your-stack). Nothing is left to look up on another page. The stacks assume OpenAI for the model side — [Configuring LLMs with stores](#configuring-llms-with-stores) shows how to pair any of them with a different [LLM](/setup-configuration/llm-providers) and [embedding](/setup-configuration/embedding-providers) provider.
**Before you start:**
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* These blocks change **stores only** — for a no-API-key setup on local models, see [Local Setup](/guides/local-setup)
* For per-option depth (tuning, pooling, managed providers), see [Relational Databases](/setup-configuration/relational-databases), [Vector Stores](/setup-configuration/vector-stores), and [Graph Stores](/setup-configuration/graph-stores)
<Info>
When you switch any store — or change embedding model or dimensions — run `cognee.prune.prune_system(metadata=True)` once before your next `cognify()` / `remember()`. Collections written under the previous embedding dimensions are not compatible with the new ones.
</Info>
## What you get by default
With no store variables set at all, Cognee runs entirely on embedded, file-based stores. No server, no extras, nothing to install:
| Layer | Default | Variable | Where it lives |
| ---------- | ------- | ------------------------- | --------------------------------------------- |
| Relational | SQLite | `DB_PROVIDER` | `<SYSTEM_ROOT_DIRECTORY>/databases/cognee_db` |
| Vector | LanceDB | `VECTOR_DB_PROVIDER` | `<SYSTEM_ROOT_DIRECTORY>/databases/` |
| Graph | Ladybug | `GRAPH_DATABASE_PROVIDER` | `<SYSTEM_ROOT_DIRECTORY>/databases/` |
`SYSTEM_ROOT_DIRECTORY` defaults to a folder inside the installed package, which is usually your virtual environment — pin it to an absolute path in your project if you want the stores to survive a reinstall. Because [per-dataset isolation](#multi-user-access-control-is-on-by-default) is on by default, the graph and vector layers write one file per dataset (`<user_id>/<dataset_id>.lbug` and `.lance.db`) rather than the single `cognee_graph_ladybug` / `cognee.lancedb` files you get with it switched off.
<Note>
`ladybug` and `kuzu` are the same embedded engine — Ladybug is the renamed Kuzu engine, and either provider value works. They do not share a file: the graph file is named `cognee_graph_<provider>`, so switching the value points Cognee at a different (initially empty) graph. The exception is upgrades — if a `cognee_graph_kuzu` file already exists, `ladybug` keeps using it instead of starting fresh.
</Note>
## Choose a stack
| Stack | Servers to run | Extras to install | Good for |
| -------------------------------------------------- | --------------------- | ---------------------------------------------- | ------------------------------------------------------------------ |
| [Embedded](#embedded-default) | none | none | Local development, single process, getting started |
| [One Postgres](#one-postgres-for-everything) | Postgres | `postgres` | One managed service for the whole memory layer |
| [Postgres and Neo4j](#postgres-and-neo4j) | Postgres, Neo4j | `postgres`, `neo4j` | Production shape: graph-native graph store, Postgres for the rest |
| [Neo4j only](#neo4j-graph-with-embedded-rest) | Neo4j | `neo4j` | Inspecting the graph in Neo4j Browser without standing up Postgres |
| [Turso](#turso-libsql) | none (or Turso cloud) | `turso` | A SQLite-compatible stack with a hosted option |
| [Custom or community](#beyond-the-built-in-stores) | your backend's | none — a separate package, or your own adapter | Qdrant, Redis, Memgraph, and other backends core does not ship |
The `docker compose` commands below come from the `docker-compose.yml` in the [cognee repository](https://github.com/topoteretes/cognee) — clone it, or point the blocks at your own instances.
## The stacks
### Embedded (default)
Everything file-based and in-process. This is what you get with no store configuration at all; the block is written out so you can see which variables the other stacks are overriding.
**Install:**
```bash theme={null}
pip install cognee
```
**Servers:** none.
**.env configuration:**
```dotenv theme={null}
# LLM — OpenAI by default (see "Configuring LLMs with stores" below to swap)
LLM_API_KEY="your_api_key"
# Relational — SQLite (embedded)
DB_PROVIDER="sqlite"
DB_NAME="cognee_db"
# Vector — LanceDB (embedded)
VECTOR_DB_PROVIDER="lancedb"
# Graph — Ladybug (embedded)
GRAPH_DATABASE_PROVIDER="ladybug"
# Optional: pin storage to your project instead of the install directory
# SYSTEM_ROOT_DIRECTORY="/absolute/path/to/project/.cognee_system"
# DATA_ROOT_DIRECTORY="/absolute/path/to/project/.data_storage"
```
<Warning>
The embedded graph store uses file-based locking and is not meant to be shared between processes or agents running at once. For concurrent access, use Neo4j or the Postgres graph store below.
</Warning>
### One Postgres for everything
Relational metadata, vectors (pgvector), and graph state all in a single Postgres database.
**Install:**
```bash theme={null}
pip install "cognee[postgres]"
# or, to avoid building psycopg2 from source:
pip install "cognee[postgres-binary]"
```
**Server:**
```bash theme={null}
docker compose --profile postgres up -d postgres
```
The bundled service is the `pgvector/pgvector:pg17` image on port `5432`, with user `cognee`, password `cognee`, and database `cognee_db` already created. Cognee issues `CREATE EXTENSION IF NOT EXISTS vector` itself, so nothing else needs preparing.
**.env configuration:**
```dotenv theme={null}
# LLM — OpenAI by default (see "Configuring LLMs with stores" below to swap)
LLM_API_KEY="your_api_key"
# Relational — Postgres
DB_PROVIDER="postgres"
DB_NAME="cognee_db"
DB_HOST="127.0.0.1"
DB_PORT="5432"
DB_USERNAME="cognee"
DB_PASSWORD="cognee"
# Vector — pgvector, in the same database
VECTOR_DB_PROVIDER="pgvector"
VECTOR_DB_NAME="cognee_db"
VECTOR_DB_HOST="127.0.0.1"
VECTOR_DB_PORT="5432"
VECTOR_DB_USERNAME="cognee"
VECTOR_DB_PASSWORD="cognee"
# Graph — Postgres tables, in the same database (demo, see the warning below)
GRAPH_DATABASE_PROVIDER="postgres_demo"
GRAPH_DATABASE_NAME="cognee_db"
GRAPH_DATABASE_HOST="127.0.0.1"
GRAPH_DATABASE_PORT="5432"
GRAPH_DATABASE_USERNAME="cognee"
GRAPH_DATABASE_PASSWORD="cognee"
```
<Warning>
**The Postgres graph store is a demo feature and is not production-ready.** It keeps nodes and edges in `graph_node` / `graph_edge` tables and does not support the Cypher search types. Postgres stays a good production choice for the relational and vector layers, but the graph layer belongs on a graph-native database — Neo4j (the [next stack](#postgres-and-neo4j)), or the embedded Ladybug/Kuzu default. A production-ready Postgres graph adapter is available as a licensed product; book a call at [cognee.ai](https://www.cognee.ai). See [Graph Stores](/setup-configuration/graph-stores) → *Postgres*.
</Warning>
The `VECTOR_DB_*` and `GRAPH_DATABASE_*` credentials repeat the relational ones on purpose. Cognee can fall back to `DB_*` for both layers, but **only** when [multi-user access control](#multi-user-access-control-is-on-by-default) is off — with it on (the default), the per-dataset engines need their own explicit values and fail without them. Spelling them out keeps the block working either way.
### Postgres and Neo4j
Postgres for metadata and vectors, Neo4j for the graph. This is the usual production shape.
**Install:**
```bash theme={null}
pip install "cognee[postgres,neo4j]"
```
**Servers:**
```bash theme={null}
docker compose --profile postgres up -d postgres
docker compose --profile neo4j up -d neo4j
```
The bundled Neo4j service is `neo4j:5.26` with the APOC and GDS plugins enabled, reachable at `bolt://localhost:7687` with user `neo4j` and password `pleaseletmein`.
**.env configuration:**
```dotenv theme={null}
# LLM — OpenAI by default (see "Configuring LLMs with stores" below to swap)
LLM_API_KEY="your_api_key"
# Relational — Postgres
DB_PROVIDER="postgres"
DB_NAME="cognee_db"
DB_HOST="127.0.0.1"
DB_PORT="5432"
DB_USERNAME="cognee"
DB_PASSWORD="cognee"
# Vector — pgvector, in the same Postgres database
VECTOR_DB_PROVIDER="pgvector"
VECTOR_DB_NAME="cognee_db"
VECTOR_DB_HOST="127.0.0.1"
VECTOR_DB_PORT="5432"
VECTOR_DB_USERNAME="cognee"
VECTOR_DB_PASSWORD="cognee"
# Graph — Neo4j
GRAPH_DATABASE_PROVIDER="neo4j"
GRAPH_DATABASE_URL="bolt://localhost:7687"
GRAPH_DATABASE_NAME="neo4j"
GRAPH_DATABASE_USERNAME="neo4j"
GRAPH_DATABASE_PASSWORD="pleaseletmein"
# Required on Neo4j Community, which allows only one database per server.
# Drop this line on Neo4j Enterprise or AuraDB to keep per-dataset isolation.
ENABLE_BACKEND_ACCESS_CONTROL="false"
```
<Note>
APOC is what gives Cognee's nodes their type-specific labels in Neo4j Browser. The bundled Docker service includes it; a self-hosted server needs the [APOC plugin](https://neo4j.com/docs/apoc/current/installation/) installed.
</Note>
For **Neo4j AuraDB**, keep everything above and swap the connection line for the `neo4j+s://` URI from your Aura console — see [Graph Stores](/setup-configuration/graph-stores) → *Neo4j Aura (Cloud)*:
```dotenv theme={null}
GRAPH_DATABASE_URL="neo4j+s://<your-instance-id>.databases.neo4j.io"
GRAPH_DATABASE_PASSWORD="<your-aura-password>"
```
### Neo4j graph with embedded rest
Neo4j for the graph, embedded defaults for everything else. The lightest way to get a browsable graph without running Postgres.
**Install:**
```bash theme={null}
pip install "cognee[neo4j]"
```
**Server:**
```bash theme={null}
docker compose --profile neo4j up -d neo4j
```
**.env configuration:**
```dotenv theme={null}
# LLM — OpenAI by default (see "Configuring LLMs with stores" below to swap)
LLM_API_KEY="your_api_key"
# Relational — SQLite (embedded)
DB_PROVIDER="sqlite"
DB_NAME="cognee_db"
# Vector — LanceDB (embedded)
VECTOR_DB_PROVIDER="lancedb"
# Graph — Neo4j
GRAPH_DATABASE_PROVIDER="neo4j"
GRAPH_DATABASE_URL="bolt://localhost:7687"
GRAPH_DATABASE_NAME="neo4j"
GRAPH_DATABASE_USERNAME="neo4j"
GRAPH_DATABASE_PASSWORD="pleaseletmein"
# Required on Neo4j Community, which allows only one database per server.
# Drop this line on Neo4j Enterprise or AuraDB to keep per-dataset isolation.
ENABLE_BACKEND_ACCESS_CONTROL="false"
```
Once a run finishes, open [http://localhost:7474](http://localhost:7474), log in with the same credentials, and inspect the graph. Neo4j Desktop works the same way — point `GRAPH_DATABASE_PASSWORD` at your Desktop database's password, and install APOC from the plugins panel. See [Graph Stores](/setup-configuration/graph-stores) → *Neo4j Desktop (Local Development)*.
### Turso (libSQL)
All three layers on libSQL. A libSQL file *is* a SQLite file, so this runs embedded with no server, and the relational layer can later sync against a hosted Turso primary.
**Install:**
```bash theme={null}
pip install "cognee[turso]"
```
**Servers:** none for the embedded setup.
**.env configuration:**
```dotenv theme={null}
# LLM — OpenAI by default (see "Configuring LLMs with stores" below to swap)
LLM_API_KEY="your_api_key"
# Relational — libSQL file (drop-in for SQLite)
DB_PROVIDER="turso"
DB_NAME="cognee_db"
# Vector — libSQL file
VECTOR_DB_PROVIDER="turso"
VECTOR_DB_URL="/absolute/path/to/cognee.turso.db"
# Graph — libSQL file (defaults under the system databases directory)
GRAPH_DATABASE_PROVIDER="turso"
# GRAPH_DATABASE_URL="/absolute/path/to/graph.db"
```
To point the **relational** layer at a hosted Turso database, add the remote credentials — Cognee then reads and writes a local replica and syncs it in the background:
```dotenv theme={null}
DB_TURSO_URL="libsql://<your-db>.turso.io"
DB_TURSO_AUTH_TOKEN="<your-token>"
```
<Note>
Remote mode is not available on all three layers. The vector layer accepts a `libsql://` URL with `VECTOR_DB_KEY` — see [Vector Stores](/setup-configuration/vector-stores) → *Turso (libSQL)* — and the graph layer is local-file only, so setting `GRAPH_DATABASE_KEY` raises an explicit "not supported yet" error.
</Note>
## Verify your stack
Every block above is checked the same way. Save the `.env` in your project root, then run this from the same directory:
```python theme={null}
import asyncio
import cognee
async def main():
# Clear anything written under a previous store or embedding configuration
await cognee.prune.prune_data()
await cognee.prune.prune_system(metadata=True)
await cognee.remember(
[
"Cognee keeps memory in three stores: a relational database for metadata, "
"a vector store for embeddings, and a graph store for entities and their relationships."
],
dataset_name="store_config_check",
)
print(await cognee.recall(query_text="Which stores does cognee keep memory in?"))
asyncio.run(main())
```
A working stack prints a `graph_completion` answer naming the three stores. If it prints an empty list, or raises before it gets there, work through [Troubleshooting](#troubleshooting) below.
To confirm *which* providers were actually resolved — useful when a variable is not being picked up — print the resolved configuration first:
```python theme={null}
from cognee.infrastructure.databases.relational import get_relational_config
from cognee.infrastructure.databases.vector.config import get_vectordb_config
from cognee.infrastructure.databases.graph.config import get_graph_config
print(get_relational_config().db_provider)
print(get_vectordb_config().vector_db_provider)
print(get_graph_config().graph_database_provider)
```
## Mix your own
The five stacks are combinations, not a fixed menu — any relational store works with any vector store and any graph store. These are the values Cognee supports out of the box:
| Layer | Variable | Built-in values | Extra required |
| ---------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
| Relational | `DB_PROVIDER` | `sqlite` (default), `postgres`, `turso` | `postgres` / `postgres-binary`, `turso` |
| Vector | `VECTOR_DB_PROVIDER` | `lancedb` (default), `pgvector`, `turso`, `neptune_analytics` | `postgres` / `postgres-binary`, `turso`, `neptune` |
| Graph | `GRAPH_DATABASE_PROVIDER` | `ladybug` (default), `kuzu`, `ladybug-remote`, `kuzu-remote`, `neo4j`, `postgres_demo`, `turso`, `neptune`, `neptune_analytics` | `neo4j`, `postgres` / `postgres-binary`, `turso`, `neptune` |
Two rules cover most of what can go wrong when you assemble your own:
* **Do not set the `*_DATASET_DATABASE_HANDLER` variables.** Cognee derives the per-dataset handler from the provider (`pgvector` → `pgvector`, `neo4j` → `neo4j`, `postgres_demo` → `postgres_graph`, `turso` → `turso_graph`). Set them only to pick a *different* isolation strategy, such as `pgvector_shared` and `postgres_graph_shared` (one schema per dataset instead of one database per dataset, which needs only `CREATE SCHEMA` rights) or `neo4j_community`. A handler that does not match its provider fails at startup with an explicit `EnvironmentError`. Configuring providers in Python is the one exception — see [Configure stores in code](#configure-stores-in-code).
* **Check the combination against access control** — see the next section.
The AWS options are the ones not shown as a stack above: `neptune` (graph) and `neptune_analytics` (hybrid graph + vector) both need `pip install "cognee[neptune]"`, a `neptune-graph://` URL, AWS credentials from the standard SDK chain, and `ENABLE_BACKEND_ACCESS_CONTROL="false"` — Cognee registers no per-dataset handler for either provider, so with access control on the first dataset access fails on the handler mismatch. See [Graph Stores](/setup-configuration/graph-stores) and [Vector Stores](/setup-configuration/vector-stores) for their blocks.
### Multi-user access control is on by default
Cognee isolates each dataset in its own database unless you turn that off, which is why several blocks above carry explicit per-layer credentials or an `ENABLE_BACKEND_ACCESS_CONTROL="false"` line. What changes with it on:
| Store | With access control on (default) | With `ENABLE_BACKEND_ACCESS_CONTROL="false"` |
| ------------------------------------------------- | -------------------------------------------------------------------------------- | ----------------------------------------------------------- |
| pgvector | Needs explicit `VECTOR_DB_HOST` / `PORT` / `NAME` / `USERNAME` / `PASSWORD` | Falls back to the relational `DB_*` values |
| `postgres_demo` graph | Needs explicit `GRAPH_DATABASE_HOST` / `PORT` / `NAME` / `USERNAME` / `PASSWORD` | Falls back to the relational `DB_*` values (logs a warning) |
| Neo4j | Creates one database per dataset — Enterprise or AuraDB only | All datasets share the one graph database |
| Embedded graph and vector (Ladybug/Kuzu, LanceDB) | One file per dataset, under a per-user directory | One shared file each |
| SQLite | One shared file either way | One shared file either way |
| Neptune, Neptune Analytics | Unsupported — no per-dataset handler exists, so the first dataset access raises | The only working mode; all datasets share the one graph |
## Configure stores in code
Every block above is a `.env` file, which is the path to prefer: the values are read once at startup and every process in your deployment sees the same stack. When the stack is decided at runtime instead — a script that targets a different database per run, a service with no `.env` on disk — the same choices are available as Python setters, one per layer. Call them before the first `remember()` or `recall()`, since that is when Cognee builds its engines.
This is the "Postgres and Neo4j" stack above, configured in code rather than in `.env`:
```python theme={null}
import cognee
cognee.config.set_relational_db_config(
{
"db_provider": "postgres",
"db_name": "cognee_db",
"db_host": "127.0.0.1",
"db_port": "5432",
"db_username": "cognee",
"db_password": "cognee",
}
)
cognee.config.set_vector_db_config(
{
"vector_db_provider": "pgvector",
# Required here, unlike in .env — see the warning below
"vector_dataset_database_handler": "pgvector",
"vector_db_name": "cognee_db",
"vector_db_host": "127.0.0.1",
"vector_db_port": "5432",
"vector_db_username": "cognee",
"vector_db_password": "cognee",
}
)
cognee.config.set_graph_db_config(
{
"graph_database_provider": "neo4j",
"graph_dataset_database_handler": "neo4j",
"graph_database_url": "bolt://localhost:7687",
"graph_database_name": "neo4j",
"graph_database_username": "neo4j",
"graph_database_password": "pleaseletmein",
}
)
```
Unlike the `.env` block it mirrors, this keeps per-dataset isolation on, so it wants Neo4j Enterprise or AuraDB. On Neo4j Community — including the bundled `docker compose --profile neo4j` service — add the `ENABLE_BACKEND_ACCESS_CONTROL="false"` line that block carries. That one has no setter: Cognee reads it from the environment on each check, so keep it in `.env`, or set `os.environ["ENABLE_BACKEND_ACCESS_CONTROL"] = "false"` before the first call.
Each setter takes the same field names the `.env` variables map to, lowercased: `VECTOR_DB_HOST` is `vector_db_host`. The full per-layer lists are in [Relational Databases](/setup-configuration/relational-databases), [Vector Stores](/setup-configuration/vector-stores), and [Graph Stores](/setup-configuration/graph-stores); a name that is not a config field raises `InvalidConfigAttributeError`, so a typo fails at the call rather than silently doing nothing.
<Warning>
**Name the dataset database handler when you set a provider in code.** The provider → handler derivation ([rule above](#mix-your-own)) is a validator that runs once, when the config object is built from the environment. The setters assign to the already-built object, so it does not run again and the handler keeps its default — `lancedb` for vectors, `ladybug` for the graph. With access control on, the first dataset access then raises `The selected vector dataset to database handler does not work with the configured vector database provider`. Naming the handler in the same call, as above, is the fix.
</Warning>
Mixing the two paths works and follows one rule: a setter overwrites whatever the environment supplied for that field, and leaves every field it does not name alone. Setting only `VECTOR_DB_PROVIDER="pgvector"` in `.env` and passing the credentials in code is a valid split — and in that direction the handler is derived correctly, because the provider was in the environment when the config was built.
## Configuring LLMs with stores
Every stack above carries a single `LLM_API_KEY` line because the model configuration is independent of the store configuration: Cognee defaults to OpenAI for both the LLM and embeddings, and the embedding key falls back to `LLM_API_KEY` when unset. To use a different provider, replace that one line with the provider's LLM and embedding blocks and keep the store lines exactly as written.
With Google Gemini, for example, any stack starts like this instead — one [Google AI Studio](https://aistudio.google.com/apikey) key covers both blocks:
```dotenv theme={null}
# LLM — Google Gemini
LLM_PROVIDER="gemini"
LLM_MODEL="gemini/gemini-2.0-flash"
LLM_API_KEY="AIza..."
# Embeddings — Google Gemini
EMBEDDING_PROVIDER="gemini"
EMBEDDING_MODEL="gemini/gemini-embedding-001"
EMBEDDING_API_KEY="AIza..."
EMBEDDING_DIMENSIONS="768"
# ...store lines from the stack you picked, unchanged
```
<Warning>
When you move the LLM off OpenAI, set the `EMBEDDING_*` block too. With it unset, embeddings still default to OpenAI's API and borrow `LLM_API_KEY` — so a non-OpenAI key is sent to `api.openai.com` and fails with an authentication error.
</Warning>
The per-provider blocks for every supported backend are in [LLM Providers](/setup-configuration/llm-providers) and [Embedding Providers](/setup-configuration/embedding-providers) — swap in any pair the same way. Remember that changing the embedding model or dimensions requires the one-time prune described in the note at the top of this page.
## Troubleshooting
<AccordionGroup>
<Accordion title="Neo4jMultiDatabaseSupportError on a Neo4j stack">
```text theme={null}
The configured Neo4j server reports the 'community' edition, which supports only a
single database. Per-dataset graph isolation on Neo4j requires multi-database support
(CREATE DATABASE), which is available on Neo4j Enterprise and AuraDB only.
```
Neo4j Community — including the bundled `docker compose --profile neo4j` service — allows exactly one database per server, but Cognee's default access-control mode wants one per dataset. Pick one of:
1. **Turn per-dataset isolation off** (what the Neo4j blocks above do): `ENABLE_BACKEND_ACCESS_CONTROL="false"`. All datasets then share one graph database.
2. **Keep isolation on Community** with `GRAPH_DATASET_DATABASE_HANDLER="neo4j_community"`, which runs one Neo4j container per dataset and needs a reachable Docker daemon. See [Neo4j Community handler](/core-concepts/multi-user-mode/dataset-database-handlers/existing-dataset-database-handlers/neo4j-community).
3. **Use Neo4j Enterprise or AuraDB**, where `CREATE DATABASE` is available and the default mode works unchanged.
</Accordion>
<Accordion title="Postgres graph store tries to connect to port 123">
```text theme={null}
OSError: Multiple exceptions: [Errno 61] Connect call failed ('127.0.0.1', 123)
```
Port `123` is the unset default for `GRAPH_DATABASE_PORT`. With access control on (the default), the `postgres_demo` graph store does **not** inherit the relational `DB_*` settings — it needs its own credentials. Add the full `GRAPH_DATABASE_HOST` / `PORT` / `NAME` / `USERNAME` / `PASSWORD` set, as in the [One Postgres](#one-postgres-for-everything) block, or set `ENABLE_BACKEND_ACCESS_CONTROL="false"` to use the fallback.
The related warning below is the same mechanism in its working case — the fallback ran, and naming the values explicitly silences it:
```text theme={null}
Postgres graph credentials are not fully configured; falling back to the relational
database configuration.
```
</Accordion>
<Accordion title="Empty results, or dimension errors, after switching stores">
Vector collections are written for a specific embedding model and dimension count. Switching store, embedding model, or `EMBEDDING_DIMENSIONS` leaves collections behind that no longer match, which surfaces as empty recalls or dimension-mismatch errors. Clear them once:
```python theme={null}
await cognee.prune.prune_data()
await cognee.prune.prune_system(metadata=True)
```
Then re-run your ingestion. Note this deletes everything Cognee has stored — on a shared server, point the new stack at a different database instead.
</Accordion>
<Accordion title="Postgres: the database does not exist">
Cognee creates its *tables*, but not the database named in `DB_NAME` — for Postgres that must already exist. The bundled Docker service creates `cognee_db` for you; on your own server, create it once:
```bash theme={null}
createdb -h 127.0.0.1 -U cognee cognee_db
```
See [Relational Databases](/setup-configuration/relational-databases#troubleshooting) for the related `DatabaseNotCreatedError` case.
</Accordion>
<Accordion title="Running Cognee inside Docker: connection refused">
`localhost` inside a container is the container itself. When Cognee runs in a container and the store runs on your host, use `host.docker.internal`:
```dotenv theme={null}
DB_HOST="host.docker.internal"
VECTOR_DB_HOST="host.docker.internal"
GRAPH_DATABASE_URL="bolt://host.docker.internal:7687"
```
When both run under the same Compose project, use the service names instead — `postgres` and `neo4j`.
</Accordion>
<Accordion title="A variable in .env seems to be ignored">
Cognee reads `.env` from the directory the process starts in, so run your script from the directory holding the file, or set the variables in the process environment instead. Values in `.env` are applied with `override=True`, so a `.env` entry wins over a variable already exported in your shell — if an old exported value is not taking effect, that is why.
To see what Cognee actually resolved, print the configuration with the snippet in [Verify your stack](#verify-your-stack).
</Accordion>
</AccordionGroup>
## Beyond the built-in stores
Qdrant, Redis, Pinecone, Turbopuffer, Milvus, Weaviate, FalkorDB, and Memgraph are available as community-maintained adapters. They install as separate packages and must be registered in your startup code before their provider value works — see [Community-Maintained Adapters](/setup-configuration/community-maintained/overview).
If your backend is on neither list, you can write a **custom adapter** for the vector or graph layer: implement the adapter class and register it in your startup code with `use_vector_adapter("your_name", YourAdapter)` or `use_graph_adapter(...)`, and the registered name then works as a `VECTOR_DB_PROVIDER` / `GRAPH_DATABASE_PROVIDER` value like any built-in one. With [access control](#multi-user-access-control-is-on-by-default) on, the adapter also needs its own dataset database handler registered via `use_dataset_database_handler(...)` — otherwise run it with `ENABLE_BACKEND_ACCESS_CONTROL="false"`. The walkthroughs are [Vector Database Integration](/contributing/adding-providers/adding-new-vector-database) and [Graph Database Integration](/contributing/adding-providers/adding-new-graph-database). The relational layer has no registration hook — it is limited to the built-in `sqlite`, `postgres`, and `turso`.
<Columns>
<Card title="Relational Databases" icon="database" href="/setup-configuration/relational-databases">
Pooling, SSL, managed Postgres, and migration sources
</Card>
<Card title="Vector Stores" icon="layers" href="/setup-configuration/vector-stores">
Per-provider settings, table layout, and subprocess tuning
</Card>
<Card title="Graph Stores" icon="network" href="/setup-configuration/graph-stores">
Every graph backend, including Neptune and remote Kuzu
</Card>
</Columns>
# Time Awareness
Source: https://docs.cognee.ai/guides/time-awareness
Step-by-step guide to using temporal mode for time-aware queries
A minimal guide to Cognee's temporal mode. Use it when your data contains dates and you want to ask time-scoped questions — before, after, or between two points in time — answered from an event timeline rather than embedding similarity alone.
## Before You Start
* Complete [Quickstart](/getting-started/quickstart) to understand basic operations
* Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured
* Read [Recall](/core-concepts/main-operations/recall) for how querying memory works
* No data is required up front — the script ingests its own dated sample text, but it starts with `cognee.forget(everything=True)`, which wipes all existing Cognee data; run it against a setup you can afford to reset
## Code in Action
```python theme={null}
import asyncio
import cognee
from cognee import SearchType
TEXT = """
In 1998 the project launched. In 2001 version 1.0 shipped. In 2004 the team merged
with another group. In 2010 support for v1 ended.
"""
QUERIES = [
"What happened before 2000?",
"What happened after 2004?",
"Events between 2001 and 2004",
]
async def main():
await cognee.forget(everything=True)
# temporal_cognify builds the event timeline alongside the usual graph.
await cognee.remember(
TEXT,
dataset_name="timeline_demo",
temporal_cognify=True,
self_improvement=False,
)
for query in QUERIES:
results = await cognee.recall(
query_text=query,
query_type=SearchType.TEMPORAL,
datasets=["timeline_demo"],
top_k=15,
)
print(f"\nQ: {query}")
print(f"A: {results[0].text}")
if __name__ == "__main__":
asyncio.run(main())
```
## What Just Happened
### Step 1: Remember Data with Temporal Mode
```python theme={null}
await cognee.forget(everything=True)
# temporal_cognify builds the event timeline alongside the usual graph.
await cognee.remember(
TEXT,
dataset_name="timeline_demo",
temporal_cognify=True,
self_improvement=False,
)
```
The script starts from a clean state, then ingests `TEXT` into the `timeline_demo` dataset. Because `temporal_cognify=True`, `remember()` extracts events and timestamps and builds the timeline during ingestion, so there is no separate `cognify()` step. This example uses one string treated as a single document; multiple documents, files, or entire datasets are processed the same way.
### Step 2: Ask Time-aware Questions
```python theme={null}
for query in QUERIES:
results = await cognee.recall(
query_text=query,
query_type=SearchType.TEMPORAL,
datasets=["timeline_demo"],
top_k=15,
)
print(f"\nQ: {query}")
print(f"A: {results[0].text}")
```
The loop runs the three query shapes temporal mode is built for — a before query, an after query, and one bounded by a pair of dates — using `SearchType.TEMPORAL`, imported from the `cognee` package at the top of the script. Each call is scoped with `datasets=["timeline_demo"]` so it only searches the timeline it just ingested; drop the argument to search every dataset you have access to. The answer for each query is `results[0].text`.
<Tip>
* If the query has clear dates, the retriever filters events by time and ranks them
* If no dates are detected, it falls back to event or entity retrieval and still answers
* Increase `top_k` to inspect more candidate events
</Tip>
## How Events and Timestamps Are Stored
Temporal ingestion adds three [DataPoint](/core-concepts/building-blocks/datapoints) types to the graph:
| Node | Fields | Role |
| ----------- | ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- |
| `Event` | `name`, `description`, `location`, `at`, `during` | One thing that happened. `name` is the embedded field, so events live in the `Event_name` vector collection. |
| `Timestamp` | `time_at`, `year`, `month`, `day`, `hour`, `minute`, `second`, `timestamp_str` | A single point in time. `time_at` is milliseconds since the Unix epoch in UTC; `timestamp_str` is the `YYYY-MM-DD HH:MM:SS` rendering. |
| `Interval` | `time_from`, `time_to` | A span, pointing at the two `Timestamp` nodes that bound it. |
An event is anything that happened in time: a dated milestone, but also any action or verb in the text — so a few sentences typically yield many events. Each extracted event is attached to its source chunk through the chunk's `contains` edge, and the entities mentioned in it become `Entity` nodes linked to the event by the relationship the extractor found (for example `transferred_to`).
How time attaches depends on what the text gives:
* **A single moment** → the event points at one `Timestamp` node through `at`.
* **A span** → the event points at an `Interval` through `during`, and the interval points at a start and an end `Timestamp`.
* Only the year is mandatory in a timestamp. Unknown month and day default to `1`, and unknown hour, minute, and second to `0`, so `"In 2001 version 1.0 shipped"` is stored as `2001-01-01 00:00:00`.
* A `Timestamp` node's id is derived from its `time_at` value, so every event resolving to the same instant shares one timestamp node. Resolved times are also appended to the event description as a `Time data: ...` line, which is what the answer is generated from.
At query time the retriever turns your question into a time range, matches `Timestamp` nodes whose `time_at` falls inside it, and collects the events within two hops of those nodes — one hop for `at`, two for `during`. Those candidates are then ranked by embedding similarity to your query and cut to `top_k`.
### Relative and vague time expressions
Extraction only timestamps events it can place on a calendar. A phrase such as *"when I was young, I loved running"* or *"when I was at primary 2, I transferred school"* still produces an `Event` node with its entities and its chunk link, but with no `at` or `during` — and therefore no `Timestamp` node:
* The event remains fully retrievable through event and entity search, including the fallback path inside `SearchType.TEMPORAL`.
* It is invisible to time-range filtering, so a "before 2005" query will not surface it.
On the query side, the time range is resolved against the current UTC date, so references like *"today"* or *"now"* resolve to a concrete bound. A phrase with no calendar anchor leaves both bounds unset; the retriever then logs that no timestamps were identified and falls back to triplet search over events and entities, which is also what happens when a range does resolve but no events fall inside it.
<Tip>
To make a relative phrase queryable by time, give the calendar date in the ingested text — `"In 2003, when I was in primary 2, I transferred school"` produces an event anchored to 2003, while the bare phrase does not.
</Tip>
## Using the HTTP API
If your server is running, you can run temporal search via the API by setting `search_type` to `"TEMPORAL"`:
```bash theme={null}
curl -X POST "http://localhost:8000/api/v1/search" \
-H "Content-Type: application/json" \
${TOKEN:+-H "Authorization: Bearer $TOKEN"} \
-d '{
"search_type": "TEMPORAL",
"query": "What happened between 2001 and 2004?",
"top_k": 10
}'
```
<Note>
The Python example above is still the easiest way to enable temporal ingestion because it lets you pass `temporal_cognify=True` directly to `remember()`.
</Note>
## Full Examples
Additional examples about temporal awareness are available on our [GitHub](https://github.com/topoteretes/cognee/tree/main/examples/guides).
* An advanced script running temporal search over real documents is on our [GitHub](https://github.com/topoteretes/cognee/blob/dev/examples/advanced_guides/temporal_awareness_example/temporal_awareness_example.py). Instead of the inlined four-sentence timeline above, it ingests two bundled biographies as separate documents with `temporal_cognify=True`, then mixes before / after / between range queries with person-centric questions that carry no dates — exercising the entity-retrieval fallback described in the tip above.
<Accordion title="Legacy guide">
```python theme={null}
import asyncio
import cognee
async def main():
text = """
In 1998 the project launched. In 2001 version 1.0 shipped. In 2004 the team merged
with another group. In 2010 support for v1 ended.
"""
await cognee.add(text, dataset_name="timeline_demo")
await cognee.cognify(datasets=["timeline_demo"], temporal_cognify=True)
from cognee import SearchType
# Before / after queries
await cognee.recall(
query_type=SearchType.TEMPORAL,
query_text="What happened before 2000?",
top_k=10,
)
await cognee.recall(
query_type=SearchType.TEMPORAL,
query_text="What happened after 2010?",
top_k=10,
)
# Between queries
await cognee.recall(
query_type=SearchType.TEMPORAL,
query_text="Events between 2001 and 2004",
top_k=10,
)
# Scoped descriptions
await cognee.recall(
query_type=SearchType.TEMPORAL,
query_text="Key project milestones between 1998 and 2010",
top_k=10,
)
await cognee.recall(
query_type=SearchType.TEMPORAL,
query_text="What happened after 2004?",
datasets=["timeline_demo"],
top_k=10,
)
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
<Columns>
<Card title="Fact Validity" icon="history" href="/guides/fact-validity">
Close superseded facts and check staleness with is\_valid()
</Card>
<Card title="Core Concepts Overview" icon="brain" href="/core-concepts/overview">
Understand how Cognee builds and stores knowledge graphs.
</Card>
<Card title="API Reference" icon="code" href="/api-reference/introduction">
Explore the search endpoint behind temporal queries.
</Card>
</Columns>
# Truth-Subspace Reranking
Source: https://docs.cognee.ai/guides/truth-subspace-reranking
Let finished sessions reshape retrieval ordering by reranking against learned 'truth' directions
<Note>
**Experimental, opt-in, and off by default.** Truth-subspace reranking only runs
when you build the subspace and pass `use_truth_weight=True`. With it off,
retrieval behaves exactly as before.
</Note>
Truth-subspace reranking turns what a finished session *learned* into a signal that
reshapes future retrieval ordering — without re-ingesting or re-embedding your data.
After a session is distilled into the `session_learnings` node set, Cognee builds a
small set of **anchor vectors** (a "truth subspace") from those lessons, projects each
chunk onto them, and stores the resulting **alignment coordinates** on the graph node.
At query time the **hybrid retriever** reads those coordinates and nudges its chunk
ranking toward the directions the session taught it to value.
## Truth Subspace vs. Reranking
The two terms name the **two halves of one feature**, not two alternatives:
* The **truth subspace** is the *learned structure* — the small set of anchor
vectors (up to `k = 8` deterministic centroid slots) built from a session's
distilled `session_learnings`. It is built once, offline, by
[`improve(..., build_truth_subspace=True)`](/core-concepts/main-operations/improve).
* **Truth-subspace reranking** is the *query-time process* that reads that
structure and reorders the retrieval candidates. It is switched on per query with
`use_truth_weight=True` on a `HYBRID_COMPLETION` search.
You build the subspace once (and rebuild it as new sessions add learnings); you
rerank against it on every query where you opt in. Reranking has nothing to reorder
until the subspace exists, so the build step always comes first.
**Before you start:**
* Complete [Quickstart](/getting-started/quickstart) and the [Self-Improvement Quickstart](/guides/self-improvement-quickstart)
* Have an existing dataset and at least one session whose learnings have been distilled (via [`improve()`](/core-concepts/main-operations/improve) with `session_ids`)
* Ensure [LLM Providers](/setup-configuration/llm-providers) are configured (the build pass embeds anchors and chunks)
## How It Works
The feature uses **two stores with two distinct roles**:
| Store | What it holds | Built when | Read when |
| ----------------------------------------- | ----------------------------------------------------------------- | ------------------ | ----------------------------------------------- |
| `TruthAnchor` vector collection | one row per accepted lesson; its embedding is a *truth direction* | post-session build | query time → gives the query's coordinates |
| `truth_alignment` property on chunk nodes | each chunk's cosine to every active anchor | post-session build | query time → gives each candidate's coordinates |
The rerank score is the **agreement** between the two, weighted by how strongly the
query itself aligns with each anchor. A chunk strongly aligned with the anchors the
query cares about is boosted; a chunk with no stored coordinates is left untouched
(neutral), so the feature can never penalize un-scored content.
<Note>
Coordinates live on the **graph node** (a cheap per-property write, no re-embedding),
mirroring how `feedback_weight` is stored. The reranker fetches them with a small
batched lookup for the candidate chunks only.
</Note>
## Code in Action
### Step 1: Build the truth subspace
Run `improve()` with `build_truth_subspace=True`. After the session's learnings are
distilled, this builds the `TruthAnchor` collection and writes `truth_alignment`
coordinates onto every chunk in the dataset.
```python theme={null}
import cognee
from cognee import SearchType
DATASET = "my_project"
await cognee.improve(
dataset=DATASET,
session_ids=["my_session"],
build_truth_subspace=True, # opt-in; default False
)
```
### Step 2: Retrieve with truth weighting on
Pass `use_truth_weight=True` through `retriever_specific_config` on a
`HYBRID_COMPLETION` search. Omit it (or set it `False`) for exact baseline ordering.
```python theme={null}
results = await cognee.search(
query_text="How should I prepare my morning drink at home?",
query_type=SearchType.HYBRID_COMPLETION,
datasets=[DATASET],
retriever_specific_config={"use_truth_weight": True},
)
print(results)
```
The same query run with `use_truth_weight` off vs on returns the same candidates in a
**different order** — chunks aligned with the session's learnings rise.
## Relationship to the Feedback Loop
Truth-subspace reranking is a *second*, complementary signal to the
[Feedback System](/guides/feedback-system):
* **`feedback_weight`** is a per-element scalar learned from session ratings. It
influences the graph/triplet retrievers via `feedback_influence`. Activate it by
setting `DEFAULT_FEEDBACK_INFLUENCE` (e.g. `0.1`); set it to `0.0` to restore the
prior baseline.
* **`truth_alignment`** is a per-chunk geometric signal learned from distilled
lessons. It influences the **hybrid** retriever's chunk lane via `use_truth_weight`.
Both are off (or neutral) by default and are stored on graph nodes the same way.
## When to Use It
Reach for truth-subspace reranking when **all** of the following hold:
* You have finished sessions whose distilled `session_learnings` encode a durable
preference or priority you want future retrieval to respect (a domain style, a
preferred source, a recurring correction).
* You retrieve with [`SearchType.HYBRID_COMPLETION`](/guides/search-basics) — the
subspace only reranks the hybrid chunk lane.
* You can rebuild the subspace after new sessions add learnings, so the anchors stay
current.
Skip it when there are no distilled learnings yet, when you use a non-hybrid
retriever, or when you need deterministic baseline ordering. Because the feature is
experimental and not yet validated by an automatic quality measurement, keep it
opt-in and evaluate its effect on your own data before relying on it.
<Accordion title="Parameters">
* **`improve(..., build_truth_subspace=True)`** — opt-in build stage. Runs after
distillation; reads `session_learnings`, upserts `TruthAnchor` rows, and writes
per-chunk `truth_alignment` coordinates. Default `False`.
* **`search(..., retriever_specific_config={"use_truth_weight": True})`** — enables
chunk-lane reranking for `HYBRID_COMPLETION`. Default `False`.
* **`DEFAULT_FEEDBACK_INFLUENCE`** (env, default `0.0` = off) — activates the separate
`feedback_weight` loop when set above `0` (e.g. `0.1`); `0.0` keeps the prior baseline.
</Accordion>
<Accordion title="Troubleshooting">
* **No change in ordering** — confirm `build_truth_subspace=True` ran and reported a
non-zero anchor / scored-node count, and that the session actually produced
`session_learnings`. With no anchors, reranking is a no-op.
* **All candidates ranked equally** — the subspace only reranks the **hybrid** chunk
lane; entity ordering is unaffected in this MVP.
* **LLM/embedding errors during build** — the build embeds anchors and chunk text;
verify your provider is configured. See [LLM Providers](/setup-configuration/llm-providers).
</Accordion>
## Additional Information
* A runnable end-to-end guide ships with the feature at
`examples/guides/truth_subspace_reranking.py`. Built entirely on the public API, it
ingests a small two-theme corpus (coffee vs. tea), remembers a couple of session
learnings into the `session_learnings` node set, builds the subspace, and then runs the
same ambiguous query twice — `use_truth_weight=False`, then `True` — printing both
retrieval contexts so the reordering is visible.
* An advanced script showing the machinery under that reordering is on our
[GitHub](https://github.com/topoteretes/cognee/blob/dev/examples/advanced_guides/truth_centroid_slots_demo.py).
Where the guide script stops at the changed ordering, this one prints the deterministic
centroid slots themselves — each slot's count and epoch, the nearest accepted learning,
and the `truth_alignment` coordinates stored on every `DocumentChunk` node — then adds a
second batch of learnings and rebuilds, so you can watch the anchors and the epoch move
as new lessons arrive. It reads those internals through APIs that are not part of the
public surface (`load_centroids`, `get_node_truth_state`, `align.cosine`), so treat it as
a look inside the feature rather than a pattern to build on.
<Note>
This MVP reranks the hybrid chunk lane only and is not yet validated by an automatic
quality measurement. Keep it opt-in until you have evaluated its effect on your data.
</Note>
<Columns>
<Card title="Self-Improvement Quickstart" icon="brain" href="/guides/self-improvement-quickstart">
Bridge and enrich memory with improve()
</Card>
<Card title="Feedback System" icon="brain-circuit" href="/guides/feedback-system">
The complementary per-element feedback signal
</Card>
<Card title="Improve" icon="sparkles" href="/core-concepts/main-operations/improve">
The operation that builds the subspace
</Card>
</Columns>
# Web URL Ingestion
Source: https://docs.cognee.ai/guides/web-url-ingestion
Step-by-step guide to ingesting web page content with custom extraction rules
A guide to building a knowledge graph straight from a web page: pass an `http(s)` URL to `remember()`, control what gets extracted with CSS-selector rules, and visualize the result.
**Before you start:**
* Complete [Quickstart](getting-started/quickstart) to understand basic operations
* Install the scraping extra: `pip install cognee[scraping]` (BeautifulSoup, Tavily, Playwright). Keenable needs no extra dependencies.
## Code in Action
### Step 1: Define Extraction Rules
Extraction rules tell the BeautifulSoup loader which parts of the page to keep. Each rule maps a name to a CSS selector (or XPath):
```python theme={null}
extraction_rules = {
"title": {"selector": "title"},
"headings": {"selector": "h1, h2, h3", "all": True},
"links": {
"selector": "a",
"attr": "href",
"all": True,
},
"paragraphs": {"selector": "p", "all": True},
}
```
Each rule supports:
| Key | Description |
| ----------- | ------------------------------------------------------------------------- |
| `selector` | CSS selector to match elements. |
| `xpath` | XPath expression, as an alternative to `selector`. |
| `attr` | HTML attribute to extract (e.g. `href`) instead of the element text. |
| `all` | `True` extracts every matching element; `False` (default) only the first. |
| `join_with` | String used to join multiple extracted elements (default `" "`). |
Rules are optional — without them the loader applies a comprehensive default set covering common HTML content areas (headings, paragraphs, articles, tables, code blocks, etc.). See [Loaders](/core-concepts/further-concepts/loaders) for more `preferred_loaders` examples.
### Step 2: Remember the URL
```python theme={null}
await cognee.remember(
"https://en.wikipedia.org/api/rest_v1/page/html/Large_language_model",
incremental_loading=False,
preferred_loaders={"beautiful_soup_loader": {"extraction_rules": extraction_rules}},
self_improvement=False,
)
```
`remember()` recognizes the URL, fetches the page, extracts content according to your rules, and builds the knowledge graph in one call.
### Step 3: Visualize the Result
```python theme={null}
await cognee.visualize_graph("./web_url_example.html")
```
Open the HTML file in a browser to explore what was extracted — see [Graph Visualization](/guides/graph-visualization).
## Choosing a crawler
Cognee can fetch pages with one of three backends:
* **Built-in BeautifulSoup crawler** (default) — asynchronous HTTP requests, robots.txt compliance, rate limiting, and Playwright rendering for JavaScript-heavy pages.
* **Tavily** — richer extraction from complex pages. Requires `TAVILY_API_KEY`.
* **Keenable** — clean markdown extraction via the [Keenable](https://docs.keenable.ai) API. Requires `KEENABLE_API_KEY`.
When you ingest a URL through `remember()` or `add()`, the backend is picked from the environment in this order:
1. Tavily, if `TAVILY_API_KEY` is set
2. Keenable, if `KEENABLE_API_KEY` is set
3. The built-in crawler
If both keys are set, Tavily wins — unset `TAVILY_API_KEY` to route through Keenable.
See [Python API: add()](/python-api/add) for the crawler configuration options (`tavily_config`, `soup_crawler_config`).
## Full Example
The script is available on our [github](https://github.com/topoteretes/cognee/blob/dev/examples/guides/web_url_content_ingestion_example.py).
The complete flow — forget, remember a URL with extraction rules, and visualize — is in the following example:
<Accordion title="Web URL content ingestion">
```python theme={null}
import asyncio
from os import path
import cognee
async def main():
await cognee.forget(everything=True)
print("Data forgotten.")
extraction_rules = {
"title": {"selector": "title"},
"headings": {"selector": "h1, h2, h3", "all": True},
"links": {
"selector": "a",
"attr": "href",
"all": True,
},
"paragraphs": {"selector": "p", "all": True},
}
await cognee.remember(
"https://en.wikipedia.org/api/rest_v1/page/html/Large_language_model",
incremental_loading=False,
preferred_loaders={"beautiful_soup_loader": {"extraction_rules": extraction_rules}},
self_improvement=False,
)
print("Knowledge graph created.")
graph_visualization_path = path.join(
path.dirname(__file__), ".artifacts", "web_url_example.html"
)
await cognee.visualize_graph(graph_visualization_path)
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
<Columns>
<Card title="Loaders" icon="download" href="/core-concepts/further-concepts/loaders">
How Cognee turns files and pages into ingestable content
</Card>
<Card title="Graph Visualization" icon="network" href="/guides/graph-visualization">
Render your knowledge graph to an interactive HTML file
</Card>
</Columns>
# Deploy Cognee on Coolify
Source: https://docs.cognee.ai/how-to-guides/cognee-sdk/deployment/coolify
Self-host the Cognee API on your own server with Coolify, an open-source PaaS, using Cognee's Docker Compose stack — from local to online.
[Coolify](https://coolify.io/) is an open-source, self-hostable PaaS — a Heroku/Netlify
alternative that runs on **your own server** and deploys anything Docker can run. Because
Cognee ships a `docker-compose.yml`, Coolify can build, run, route, and TLS-terminate the
Cognee API for you, so you go from "it works on my laptop" to a public HTTPS endpoint without
writing any infrastructure code.
This guide deploys the **Cognee API server** (the `cognee` service). It uses a slim,
Coolify-friendly compose so the deployment is reliable on a modest server, and explains how to
add the optional services (databases, MCP server, UI) when you need them.
## Architecture
```mermaid theme={null}
flowchart LR
client(["API client / browser"]) -->|HTTPS| traefik
subgraph server["Your server — managed by Coolify"]
traefik["Coolify proxy (Traefik)<br/>Let's Encrypt SSL"]
cognee["cognee API<br/>:8000 · /health"]
traefik --> cognee
cognee -.->|"file-based, default"| vol[("SQLite · LanceDB · Ladybug<br/>persisted on a named volume")]
cognee -.->|"optional"| ext[("Postgres · Neo4j · Redis")]
end
```
By default Cognee uses **file-based databases** (SQLite + LanceDB + Ladybug, Cognee's embedded
graph engine), so a working deployment needs **no external database** — just one API key.
## Prerequisites
* A **server** (VPS or bare metal) running a 64-bit Linux distro (Debian/Ubuntu recommended).
Coolify itself idles at \~0.6–1 GB RAM, so plan for **4 GB+** even though the documented
minimum is 2 GB.
* A running **Coolify** instance (v4). If you don't have one yet, follow **Step 1** below first.
* An **LLM API key**. Cognee defaults to OpenAI, so an OpenAI key works out of the box
(`LLM_API_KEY`); any [supported provider](/setup-configuration/llm-providers) can be configured later.
* *(Optional)* a **domain name** with a DNS `A` record pointing at your server, for a clean
HTTPS URL. Coolify can also hand out a free `sslip.io` hostname.
## Deployment at a glance
```mermaid theme={null}
flowchart TD
A["Install Coolify on a Linux server"] --> B["Create New Resource → Public Repository"]
B --> C["Build Pack: Docker Compose"]
C --> D["Set environment variables<br/>(LLM_API_KEY, ENV=prod, …)"]
D --> E["Assign a domain for HTTPS<br/>(SERVICE_FQDN_COGNEE_8000)"]
E --> F["Deploy"]
F --> G{"/health returns 200?"}
G -->|Yes| H["Live: Cognee API over HTTPS"]
G -->|No| I["Check Logs → Troubleshooting"]
I --> F
```
## Step 1 — Install Coolify (skip if you already have it)
On a fresh server, run the official one-line installer as root (or with `sudo`):
```bash theme={null}
curl -fsSL https://cdn.coollabs.io/coolify/install.sh | sudo bash
```
Open the dashboard at `http://<your-server-ip>:8000` and **register the first account** — the
first user to register owns the instance, so do this immediately. Ports used by Coolify:
**22** (SSH), **80** / **443** (proxy + SSL), and **8000 / 6001 / 6002** (dashboard, realtime,
terminal). Lock `8000` down to your own IP once you're set up.
<Note>
Coolify's proxy needs ports **80/443**. If another reverse proxy or PaaS already owns them on
the same host, free them first — two proxies can't bind the same ports.
</Note>
## Step 2 — Add Cognee as a Docker Compose resource
1. Open (or create) a **Project** and pick an environment (e.g. *production*).
2. Click **Create New Resource** and choose **Public Repository**.
3. Paste the repository URL (the upstream repo or your fork):
```
https://github.com/topoteretes/cognee
```
4. Coolify defaults the build pack to **Nixpacks** — open that dropdown and select
**Docker Compose**.
5. Set **Branch** to `main`, **Base Directory** to `/`, and **Docker Compose Location** to the
compose file you want to deploy (see below).
### Use a slim, Coolify-friendly compose
Cognee's root `docker-compose.yml` is tuned for **local development** and has three rough edges
on Coolify. Knowing them saves hours:
| Repo `docker-compose.yml` | Why it bites on Coolify |
| -------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Builds the image from source (`build:`) | The multi-stage `uv` build is heavy and can **OOM** a small server. Coolify's own docs recommend a **prebuilt image** on low-RAM hosts. |
| Declares a custom network (`cognee-network`) | Coolify warns that **custom networks cause intermittent 504s**; it auto-creates an isolated network per stack. |
| Gates optional services behind `profiles:` | Coolify **does not reliably honor compose profiles** ([issue #6395](https://github.com/coollabsio/coolify/issues/6395)) — it may start *every* service regardless. |
The fix is a small, production-oriented compose that uses the **official prebuilt image**
(`cognee/cognee:main`), declares no custom network, has a single service (no profiles), injects
config via Coolify environment variables, and persists data on named volumes. (It also drops the
repo compose's published `5678` debugger port.) Commit this file to your fork and point
**Docker Compose Location** at it, or paste it via Coolify's raw compose editor:
```yaml docker-compose.coolify.yml theme={null}
services:
cognee:
image: cognee/cognee:main
restart: always
ports:
- "8000:8000"
environment:
- LLM_API_KEY=${LLM_API_KEY}
# Canonical env var (ENVIRONMENT is a deprecated alias). Use exactly "prod":
# any other value enables FastAPI debug; "dev"/"local" also add Gunicorn auto-reload.
- ENV=prod
- DEBUG=false
- LOG_LEVEL=INFO
- CORS_ALLOWED_ORIGINS=${CORS_ALLOWED_ORIGINS:-*}
volumes:
# The image stores file-based databases under /cognee-storage and bakes
# in ownership for its non-root user (uid 1000), so fresh named volumes
# mounted here initialize writable.
- cognee_system:/cognee-storage/system
- cognee_data:/cognee-storage/data
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 40s
volumes:
cognee_system:
cognee_data:
```
<Tip>
The images `cognee/cognee` (API) and `cognee/cognee-mcp` (MCP server) are published to Docker Hub
on every push to `main`, so the prebuilt image skips the source build entirely. Note that `main`
is a moving tag that tracks the latest build — pin a specific tag if you need reproducible deploys.
</Tip>
### Reference: services in the full repo compose
If you do deploy the upstream `docker-compose.yml`, this is what it contains. Only services
**without** a `profiles:` key start under plain Docker Compose — the `cognee` API (`8000`) and
`redisinsight` (`5540`):
| Service | Profile | Default ports | Purpose |
| -------------- | ---------- | ------------- | ------------------------------- |
| `cognee` | *(none)* | 8000 | Core API server |
| `redisinsight` | *(none)* | 5540 | Redis GUI |
| `cognee-mcp` | `mcp` | 8001 | MCP server for IDE integrations |
| `frontend` | `ui` | 3000 | Experimental web UI |
| `postgres` | `postgres` | 5432 | PostgreSQL + pgvector |
| `neo4j` | `neo4j` | 7474 / 7687 | Neo4j graph database |
| `redis` | `redis` | 6379 | Redis cache |
*Ports are host mappings; `cognee-mcp` listens on `8000` inside its container (mapped to host `8001`).*
<Warning>
These are Docker Compose **profiles** (activated with `--profile` / `COMPOSE_PROFILES` locally),
**not** commented-out lines to uncomment. Because Coolify doesn't reliably honor profiles, the
dependable way to choose services on Coolify is to put exactly what you want in the compose —
which is what the slim file above does.
</Warning>
## Step 3 — Configure environment variables
Open the resource's **Environment Variables** tab. Add variables one-by-one in **Normal View**,
or switch to **Developer View** to paste a block of `KEY=VALUE` lines at once. Cognee reads
variables straight from the container environment (they take precedence over any `.env` file),
so whatever you set here is applied on the next deploy.
The only variable you **must** set is the API key:
| Variable | Required | Default | Notes |
| ---------------------- | -------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------- |
| `LLM_API_KEY` | **Yes** | — | OpenAI key by default; also used for embeddings if no separate key. |
| `LLM_MODEL` | No | `openai/gpt-5-mini` | Override the model. |
| `LLM_PROVIDER` | No | `openai` | `openai`, `anthropic`, `gemini`, `ollama`, … |
| `CORS_ALLOWED_ORIGINS` | No | `*` | `*` is the slim compose's fallback (the app alone defaults to `http://localhost:3000`). Lock down to your domain(s) in production. |
<Warning>
The repo's `docker-compose.yml` **hard-codes `ENV=local`** inline on the `cognee` service. Inline
compose values **override** anything you set in the UI, so adding `ENV=…` (or the deprecated
`ENVIRONMENT=…`) there has **no effect** with that file. Use the slim compose above (it sets
`ENV=prod`) or edit the service's `environment:` block. Use **exactly `ENV=prod`**: the app enables
FastAPI debug mode for any other value (including `production`), and the entrypoint additionally
turns on Gunicorn auto-reload + verbose logs when `ENV` is `dev` or `local`.
</Warning>
## Step 4 — Persist data and choose your databases
**File-based (default, zero setup).** Cognee defaults to SQLite (relational), LanceDB (vector)
and Ladybug (graph) — Ladybug is Cognee's embedded graph engine; Kuzu is also supported. To
survive redeploys, keep them on a **named volume** — Coolify persists
named volumes automatically (it appends the resource UUID to the name). The slim compose does
this with the `cognee_system` and `cognee_data` volumes mounted at `/cognee-storage/system` and
`/cognee-storage/data`, the storage roots baked into the image.
**External databases (optional).** To move onto Postgres or Neo4j, set the matching **provider**
variables — pointing only `DB_HOST` at a server is not enough. Run the database as a separate
Coolify resource, or add its service directly to your compose (declared normally, since Coolify
won't gate it behind a profile):
```env theme={null}
# PostgreSQL (relational) — the bundled image is pgvector/pgvector:pg17
DB_PROVIDER=postgres
DB_HOST=postgres # the compose service name, on the same Coolify network
DB_PORT=5432
DB_USERNAME=cognee
DB_PASSWORD=cognee
DB_NAME=cognee_db
# Vector store — lancedb (default), pgvector, …
VECTOR_DB_PROVIDER=pgvector
# Graph store — ladybug (default), kuzu, neo4j, …
GRAPH_DATABASE_PROVIDER=neo4j
GRAPH_DATABASE_URL=bolt://neo4j:7687
GRAPH_DATABASE_USERNAME=neo4j
GRAPH_DATABASE_PASSWORD=pleaseletmein
```
## Step 5 — Expose the API: domain, port & SSL
The `cognee` service listens on port **8000**. To publish it through Coolify's proxy with
automatic HTTPS, assign a domain to the service — either in its **Domains** field, or with
Coolify's magic variable in the compose:
```env theme={null}
SERVICE_FQDN_COGNEE_8000=https://cognee.example.com
```
The identifier is the service name (`COGNEE`) with the container port appended (`_8000`).
Entering an `https://` domain makes Coolify request and auto-renew a **Let's Encrypt**
certificate via Traefik. (Magic variables in a Git-sourced compose require Coolify
**v4.0.0-beta.411+**; on older builds, assign the domain in the service's **Domains** field instead.)
<Note>
Let's Encrypt needs the domain's DNS `A` record pointing at the server and ports **80/443**
open, and it won't validate behind Cloudflare's proxied ("orange cloud") mode. Publishing a
`ports:` mapping alone exposes plain HTTP on the host and **bypasses** the proxy/SSL — use a
domain assignment for managed TLS.
</Note>
## Step 6 — Deploy and verify
Click **Deploy** and open the **Logs**. A healthy start runs the database migrations and then
Gunicorn binds the server:
```text theme={null}
Debug mode: false
Environment: prod
Debug port: 5678
HTTP port: 8000
Bind address: 0.0.0.0
Running database migrations...
Database migrations done.
Starting server...
```
With `ENV=prod`, Gunicorn logs at error level, so the startup banner is quiet — rely on the
health check below to confirm readiness.
Check the health endpoint — a ready instance returns HTTP `200`:
```bash theme={null}
curl https://cognee.example.com/health
```
```json theme={null}
{ "status": "ready", "health": "healthy", "version": "x.y.z" }
```
If a critical database or storage check fails, `/health` returns HTTP `503` with
`{"status": "not ready", "health": "unhealthy", …}` — there is no separate "starting" state, and a
degraded-but-usable instance still returns `200`. For a component-by-component breakdown call
`GET /health/detailed`, and the interactive API docs live at `/docs`.
<Note>
With default settings Cognee runs in **multi-tenant mode** (`ENABLE_BACKEND_ACCESS_CONTROL=True`),
so API endpoints require an authenticated user — but `/health` and `/docs` stay open, which is
all you need to confirm a successful deployment.
</Note>
## Production hardening
Before exposing Cognee publicly, review these defaults:
* **Authentication.** `ENABLE_BACKEND_ACCESS_CONTROL=True` (default) **requires auth** on the API
— `REQUIRE_AUTHENTICATION=False` is ignored while access control is on. Create a user, or for
a single-user setup behind a token set `ENABLE_BACKEND_ACCESS_CONTROL=False`.
* **JWT secret.** Change `FASTAPI_USERS_JWT_SECRET` (default `super_secret`) to a long random
string, identical across replicas.
* **CORS.** Replace the default `*` in `CORS_ALLOWED_ORIGINS` with your real front-end origin(s).
* **Secrets.** Never commit API keys — set them only as Coolify environment variables.
## Troubleshooting
| Symptom | Likely cause | Fix |
| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------- |
| Build is slow / server runs out of memory | Building the image from source compiles many extras | Deploy the **prebuilt image** (`cognee/cognee:main`) via the slim compose (Step 2) |
| Intermittent `504 Gateway Timeout` | A custom `networks:` block in the compose | Remove custom networks; let Coolify manage the network (the slim compose has none) |
| Unwanted services (Postgres, Neo4j…) start anyway | Coolify doesn't honor compose `profiles:` ([#6395](https://github.com/coollabsio/coolify/issues/6395)) | Deploy a compose that contains only the services you want (the slim compose) |
| `ENV` / `ENVIRONMENT` changes have no effect | Repo compose hard-codes `ENV=local` inline | Edit the service `environment:`, or use the slim compose |
| Data disappears after a redeploy | File-based DBs weren't on a persistent volume | Mount **named** volumes at `/cognee-storage/system` and `/cognee-storage/data` (Step 2) |
| `/health` never turns ready | Can't reach a configured external DB, or migrations failed | Check **Logs**; verify `DB_*` / `GRAPH_*` values and that the DB is reachable |
| 401/403 on API calls (not `/health`) | Access control is on by default | Authenticate, or set `ENABLE_BACKEND_ACCESS_CONTROL=False` for single-user mode |
| Let's Encrypt certificate fails | DNS not pointing at server, ports 80/443 closed, or Cloudflare proxied | Fix DNS/ports; set Cloudflare DNS to "DNS only" |
## Cost estimate
Cognee runs comfortably on a small VPS; the main variable cost is LLM API usage.
| Tier | Specs | Approx. server cost |
| ------------------------------------ | -------------- | ------------------- |
| Minimal (file-based DBs, light use) | 2 vCPU / 4 GB | \~\$5–12 / month |
| Recommended (headroom for `cognify`) | 4 vCPU / 8 GB | \~\$15–30 / month |
| External DBs / heavier graphs | 8 vCPU / 16 GB | \~\$30–60 / month |
LLM token costs are separate and depend on your provider, model, and how much data you ingest.
## Next Steps
<CardGroup>
<Card title="API Reference" href="/api-reference/introduction" icon="book-open">
Explore the API you just deployed and build your graph with `add` → `cognify` → `search`.
</Card>
<Card title="MCP Server" href="/cognee-mcp/mcp-quickstart" icon="plug">
Connect Cognee to your IDE by also deploying the MCP server (`cognee/cognee-mcp`).
</Card>
</CardGroup>
<Card title="Need Help?" href="https://discord.gg/m63hxKsp4p" icon="discord">
Join our community for Coolify deployment support.
</Card>
# Deployment Options
Source: https://docs.cognee.ai/how-to-guides/cognee-sdk/deployment/deployment-options
Choose a Cognee deployment pattern based on writer ownership, storage, and read scaling
## Deployment facts
These facts anchor the rest of the guide:
| Common assumption | Correct model | Why it matters |
| ----------------------------------------------------- | -------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| The default graph backend is a separate named service | The default graph backend is embedded, file-backed Kuzu. NetworkX is the in-memory fallback. | The on-disk graph directory is `cognee_graph_kuzu`. |
| A shared service gives moderate write concurrency | A shared service over file-backed Kuzu is still single-writer. | The service boundary centralizes the writer; it does not make file-backed writes concurrent. |
| Helm includes an API, worker, and queue by default | Cognee ships one FastAPI image. Worker and queue splits are operator patterns. | Add a queue and worker deployment only when your write model needs them. |
## Executive summary
Cognee runs either embedded in a Python process or as a FastAPI service. Operationally, each deployment comes down to three independent choices:
* Writer ownership: who owns writes
* Storage location: where graph, vector, and relational state live
* Reader access path: how readers reach memory
The default embedded stack is file-backed: Kuzu for graph storage, LanceDB for vectors, and SQLite for relational metadata. In production, each tier can be externalized independently: Neo4j or the FalkorDB adapter for graph storage, Qdrant/pgvector/Pinecone/ChromaDB for vectors, and Postgres for relational metadata.
Deployment patterns are composable. Choose a base shape first, such as Embedded SDK, Compose, Helm, sidecar, or Lambda, then add the write model, read-scaling pattern, and storage backend that match your workload.
## Pick-a-path decision tree
Start with writer ownership. The first "no" on single-writer ownership pushes you toward queueing or external backends.
```text theme={null}
Single writer?
|-- yes
| `-- Few readers?
| |-- yes -> Embedded SDK
| `-- no -> Self-hosted FastAPI service
| `-- Add snapshot read replicas when reads need to scale
`-- no
`-- Concurrent or multi-agent writes?
|-- Bursty / event-driven -> Queue + single writer worker
`-- Sustained concurrency -> Externalized backends
Postgres + Neo4j/FalkorDB adapter + Qdrant/pgvector
```
## Storage model
Cognee has three independent storage layers:
| Layer | Selector | Default | Production options |
| ---------- | ------------------------- | --------- | -------------------------------------------- |
| Graph | `GRAPH_DATABASE_PROVIDER` | `kuzu` | `kuzu-remote`, `neo4j`, FalkorDB adapter |
| Vector | `VECTOR_DB_PROVIDER` | `lancedb` | `pgvector`, `qdrant`, `pinecone`, `chromadb` |
| Relational | `DB_PROVIDER` | `sqlite` | `postgres` |
Embedded storage is file-backed and easy to move. If `SYSTEM_ROOT_DIRECTORY` is unset, Cognee resolves it to `.cognee_system`, so the default on-disk layout still lands under `<SYSTEM_ROOT_DIRECTORY>/databases`:
```text theme={null}
<SYSTEM_ROOT_DIRECTORY>/
`-- databases/
|-- cognee_graph_kuzu
|-- cognee_graph_kuzu.wal
|-- lancedb/
`-- system.db
```
That makes backups simple, but the same file-backed layout is why one process should own writes. In production, replace each layer with service-backed systems by changing the provider and connection variables.
## Concurrency and the write model
This is the primary production decision. Packaging is secondary.
| Write model | Shape | Maps to | Concurrency |
| ---------------- | --------------------------------------------------------- | ------------------------------------------------- | ----------------------------- |
| Single process | One process owns local files | Embedded SDK | Lowest |
| Shared service | Many clients, one Cognee backend, one writer | Compose, Helm, sidecar | Centralized, still one writer |
| Queue-based | Producers enqueue, one worker consumes | SQS, Kafka, RabbitMQ, Redis | Good for bursty writes |
| Managed backends | Graph, vector, and relational tiers are external services | Neo4j/FalkorDB adapter, Qdrant/pgvector, Postgres | Highest |
<Warning>
For concurrent multi-agent writes, do not rely on shared file-backed Kuzu. Use a single writer service, a queue, or an external graph backend.
</Warning>
## 1. Embedded deployment
Cognee runs inside the calling Python process. Storage defaults to the local Cognee directories and can be moved with `DATA_ROOT_DIRECTORY` and `SYSTEM_ROOT_DIRECTORY`. One process owns the writer lock; there is no service boundary or cross-machine sharing unless the directory is mounted or copied.
```python theme={null}
import os
os.environ["DATA_ROOT_DIRECTORY"] = "/data/.cognee_data"
os.environ["SYSTEM_ROOT_DIRECTORY"] = "/data/.cognee_system"
os.environ["GRAPH_DATABASE_PROVIDER"] = "kuzu"
os.environ["VECTOR_DB_PROVIDER"] = "lancedb"
os.environ["DB_PROVIDER"] = "sqlite"
import cognee
await cognee.remember("...", dataset_name="docs")
await cognee.recall("...")
```
Best for notebooks, CLIs, local agents, and single-process jobs. Avoid it for cross-process concurrent writes.
| Pros | Cons |
| ------------------------------------------------------------- | ----------------------------------------------------- |
| Zero infrastructure; runs in a notebook, CLI, or local agent. | Single writer; no concurrent writes across processes. |
| Lowest latency because there is no network hop to storage. | No service boundary or shared multi-machine access. |
| Simple backup model: snapshot one data/system directory. | State is only as durable as the local disk or mount. |
<Accordion title="Full embedded script">
```python theme={null}
import asyncio
import os
os.environ["DATA_ROOT_DIRECTORY"] = "/data/.cognee_data"
os.environ["SYSTEM_ROOT_DIRECTORY"] = "/data/.cognee_system"
os.environ["GRAPH_DATABASE_PROVIDER"] = "kuzu"
os.environ["VECTOR_DB_PROVIDER"] = "lancedb"
os.environ["DB_PROVIDER"] = "sqlite"
import cognee
async def main():
await cognee.remember("Cognee turns documents into AI memory.", dataset_name="docs")
results = await cognee.recall("What does Cognee do?")
print(results)
if __name__ == "__main__":
asyncio.run(main())
```
</Accordion>
## 2. Self-hosted service
<Tabs>
<Tab title="2a. Docker Compose">
Use Compose to validate Cognee inside customer or on-prem infrastructure before moving to Kubernetes. Pin the image, persist data to a named volume, expose health checks, and load secrets from a managed source rather than plaintext `.env` files.
```yaml theme={null}
services:
cognee:
image: cognee/cognee:1.0.6
depends_on:
postgres:
condition: service_healthy
environment:
DB_PROVIDER: postgres
DB_HOST: postgres
DB_PORT: "5432"
DB_NAME: cognee
DB_USERNAME: cognee
DB_PASSWORD_FILE: /run/secrets/db_password
VECTOR_DB_PROVIDER: pgvector
GRAPH_DATABASE_PROVIDER: kuzu
DATA_ROOT_DIRECTORY: /data/.cognee_data
SYSTEM_ROOT_DIRECTORY: /data/.cognee_system
secrets:
- db_password
volumes:
- cognee_data:/data
ports:
- "8000:8000"
postgres:
image: pgvector/pgvector:pg17
environment:
POSTGRES_USER: cognee
POSTGRES_DB: cognee
POSTGRES_PASSWORD_FILE: /run/secrets/db_password
secrets:
- db_password
volumes:
- pg_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U cognee -d cognee"]
interval: 10s
retries: 5
secrets:
db_password:
file: ./secrets/db_password.txt
volumes:
cognee_data: {}
pg_data: {}
```
See [Docker Deployment](/how-to-guides/cognee-sdk/deployment/docker) for the full Compose workflow.
| Pros | Cons |
| ------------------------------------------------------------------------ | --------------------------------------------------------------------- |
| Fastest path to a persistent service on customer infrastructure. | Single host; vertical scaling only. |
| Easy to externalize one tier at a time, such as Postgres/pgvector first. | Compose secrets are weaker than a managed secret store. |
| Health checks and named volumes give practical readiness and durability. | Graph writes stay single-writer until the graph tier is externalized. |
<Accordion title="Full Compose skeleton">
```yaml theme={null}
services:
cognee:
image: cognee/cognee:1.0.6
depends_on:
postgres:
condition: service_healthy
environment:
DB_PROVIDER: postgres
DB_HOST: postgres
DB_PORT: "5432"
DB_NAME: cognee
DB_USERNAME: cognee
DB_PASSWORD_FILE: /run/secrets/db_password
VECTOR_DB_PROVIDER: pgvector
VECTOR_DB_HOST: postgres
VECTOR_DB_PORT: "5432"
VECTOR_DB_NAME: cognee
GRAPH_DATABASE_PROVIDER: kuzu
DATA_ROOT_DIRECTORY: /data/.cognee_data
SYSTEM_ROOT_DIRECTORY: /data/.cognee_system
LLM_PROVIDER: openai
LLM_API_KEY_FILE: /run/secrets/llm_api_key
secrets:
- db_password
- llm_api_key
volumes:
- cognee_data:/data
ports:
- "8000:8000"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 15s
retries: 5
postgres:
image: pgvector/pgvector:pg17
environment:
POSTGRES_USER: cognee
POSTGRES_DB: cognee
POSTGRES_PASSWORD_FILE: /run/secrets/db_password
secrets:
- db_password
volumes:
- pg_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U cognee -d cognee"]
interval: 10s
retries: 5
secrets:
db_password:
file: ./secrets/db_password.txt
llm_api_key:
file: ./secrets/llm_api_key.txt
volumes:
cognee_data: {}
pg_data: {}
```
</Accordion>
</Tab>
<Tab title="2b. Helm on Kubernetes">
Use Helm when the customer already operates Kubernetes. With embedded graph storage, encode the single-writer invariant with one replica, a `Recreate` strategy, and a read-write-once PVC. Queue and worker templates are add-ons, not defaults.
```text theme={null}
cognee/
|-- Chart.yaml
|-- values.yaml
`-- templates/
|-- deployment.yaml
|-- service.yaml
|-- pvc.yaml
|-- secret.yaml
|-- networkpolicy.yaml
|-- pdb.yaml
`-- ingress.yaml
```
```yaml theme={null}
image:
repository: cognee/cognee
tag: "1.0.6"
replicas: 1
strategy: Recreate
persistence:
storageClass: gp3
size: 8Gi
env:
GRAPH_DATABASE_PROVIDER: kuzu
VECTOR_DB_PROVIDER: pgvector
DB_PROVIDER: postgres
ENABLE_BACKEND_ACCESS_CONTROL: "true"
REQUIRE_AUTHENTICATION: "true"
externalPostgres:
host: cognee.example.rds.amazonaws.com
port: 5432
database: cognee
```
Production hardening should include pinned image tags, managed secrets, resource limits, readiness probes, network policies, and a PodDisruptionBudget for the writer. See [Kubernetes (Helm)](/how-to-guides/cognee-sdk/deployment/helm).
| Pros | Cons |
| --------------------------------------------------------------------------------- | ---------------------------------------------------------------- |
| Production-grade packaging with secrets, network policy, and observability hooks. | Requires a Kubernetes platform and team to operate it. |
| Each storage tier can be externalized through values. | Queue, worker split, and ExternalSecrets are operator additions. |
| `Recreate`, RWO PVC, and PDB encode the single-writer invariant. | Single-writer still holds until the graph tier is externalized. |
<Accordion title="Full Helm skeleton">
```yaml theme={null}
# templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: cognee
spec:
replicas: {{ .Values.replicas }}
strategy:
type: {{ .Values.strategy }}
selector:
matchLabels:
app: cognee
template:
metadata:
labels:
app: cognee
spec:
initContainers:
- name: wait-for-postgres
image: pgvector/pgvector:pg17
command:
- sh
- -c
- until pg_isready -h {{ .Values.externalPostgres.host }}; do sleep 2; done
containers:
- name: cognee
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
ports:
- containerPort: 8000
env:
- name: DB_PROVIDER
value: "{{ .Values.env.DB_PROVIDER }}"
- name: DB_HOST
value: "{{ .Values.externalPostgres.host }}"
- name: VECTOR_DB_PROVIDER
value: "{{ .Values.env.VECTOR_DB_PROVIDER }}"
- name: GRAPH_DATABASE_PROVIDER
value: "{{ .Values.env.GRAPH_DATABASE_PROVIDER }}"
- name: LLM_API_KEY
valueFrom:
secretKeyRef:
name: cognee-secrets
key: llmApiKey
volumeMounts:
- name: data
mountPath: /data
readinessProbe:
httpGet:
path: /health
port: 8000
resources:
{{- toYaml .Values.resources | nindent 12 }}
volumes:
- name: data
persistentVolumeClaim:
claimName: cognee-data
---
# templates/networkpolicy.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: cognee
spec:
podSelector:
matchLabels:
app: cognee
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
app: postgres
ports:
- port: 5432
- to:
- ipBlock:
cidr: 0.0.0.0/0
ports:
- port: 443
---
# templates/pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: cognee
spec:
maxUnavailable: 0
selector:
matchLabels:
app: cognee
```
</Accordion>
</Tab>
<Tab title="2c. Sidecar">
Run Cognee next to an agent in the same pod when one agent needs private, low-latency memory. The agent talks to Cognee over `localhost`; Cognee owns the storage connection and serializes writes.
```yaml theme={null}
containers:
- name: agent
image: my-agent:1.2.0
env:
- name: COGNEE_URL
value: "http://localhost:8000"
- name: cognee
image: cognee/cognee:1.0.6
ports:
- containerPort: 8000
volumeMounts:
- name: memory
mountPath: /data
volumes:
- name: memory
persistentVolumeClaim:
claimName: agent-memory
```
This isolates memory per pod. It does not create a shared multi-agent write path unless storage is externalized.
| Pros | Cons |
| ------------------------------------------------- | --------------------------------------------------------------------- |
| Lowest-latency service boundary over `localhost`. | Memory is scoped to one pod unless storage is shared or externalized. |
| Per-agent isolation by construction. | Scales with agent pods, not independently. |
| No separate Cognee service lifecycle to operate. | Wasteful if Cognee is mostly idle beside each agent. |
<Accordion title="Full sidecar pod skeleton">
```yaml theme={null}
apiVersion: v1
kind: Pod
metadata:
name: agent-with-cognee
spec:
containers:
- name: agent
image: my-agent:1.2.0
env:
- name: COGNEE_URL
value: "http://localhost:8000"
- name: cognee
image: cognee/cognee:1.0.6
ports:
- containerPort: 8000
env:
- name: DATA_ROOT_DIRECTORY
value: /data/.cognee_data
- name: SYSTEM_ROOT_DIRECTORY
value: /data/.cognee_system
volumeMounts:
- name: memory
mountPath: /data
volumes:
- name: memory
persistentVolumeClaim:
claimName: agent-memory
```
</Accordion>
</Tab>
</Tabs>
## 3. Scale-out patterns
Scale-out patterns are layered on top of a self-hosted deployment. They scale reads or isolate write jobs; they do not replace the write path.
<Tabs>
<Tab title="3a. Snapshot read replicas">
Use one writer to run `remember()`, publish a snapshot to object storage, and let many readers pull the latest snapshot at startup. This scales reads without a clustered graph database, at the cost of freshness.
```text theme={null}
writer -> snapshot.tar -> S3 / MinIO
|-> reader-1
|-> reader-2
`-> reader-N
```
```yaml theme={null}
initContainers:
- name: pull-snapshot
image: amazon/aws-cli
command:
- sh
- -c
- aws s3 cp s3://cognee-snapshots/latest.tar /data/latest.tar && tar -xf /data/latest.tar -C /data
containers:
- name: cognee-reader
image: cognee/cognee:1.0.6
env:
- name: COGNEE_READ_ONLY
value: "true"
```
Use this for heavy read traffic and static or slowly changing knowledge. Benchmark snapshot size because it drives reader cold-start time.
| Pros | Cons |
| ----------------------------------------------------------- | ---------------------------------------------------- |
| Horizontal read scaling without a clustered graph database. | Freshness is bounded by snapshot cadence. |
| Readers are stateless and replaceable. | Snapshot size drives cold-start time. |
| Cheap: object storage instead of a database fleet. | Not for collaborative or strict-freshness workloads. |
<Accordion title="Full reader deployment skeleton">
```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
name: cognee-reader
spec:
replicas: 6
selector:
matchLabels:
app: cognee-reader
template:
metadata:
labels:
app: cognee-reader
spec:
initContainers:
- name: pull-snapshot
image: amazon/aws-cli
command:
- sh
- -c
- aws s3 cp s3://cognee-snapshots/latest.tar /data/latest.tar && tar -xf /data/latest.tar -C /data
volumeMounts:
- name: snapshot
mountPath: /data
containers:
- name: cognee-reader
image: cognee/cognee:1.0.6
env:
- name: COGNEE_READ_ONLY
value: "true"
- name: DATA_ROOT_DIRECTORY
value: /data/.cognee_data
- name: SYSTEM_ROOT_DIRECTORY
value: /data/.cognee_system
volumeMounts:
- name: snapshot
mountPath: /data
volumes:
- name: snapshot
emptyDir: {}
```
</Accordion>
</Tab>
<Tab title="3b. Queue-based write path">
Use a durable queue when producers are bursty but Cognee still needs one writer. Producers submit jobs; one worker consumes them and owns the write path.
```hcl theme={null}
resource "aws_sqs_queue" "cognee_dlq" {
name = "cognee-ingest-dlq"
}
resource "aws_sqs_queue" "cognee_ingest" {
name = "cognee-ingest"
visibility_timeout_seconds = 900
redrive_policy = jsonencode({
deadLetterTargetArn = aws_sqs_queue.cognee_dlq.arn
maxReceiveCount = 3
})
}
```
Run the worker deployment with `replicas: 1` so it protects the single-writer path.
| Pros | Cons |
| ------------------------------------------------------------------------------ | ----------------------------------------------------------- |
| Absorbs write bursts; producers do not block on the writer. | One worker is a throughput ceiling by design. |
| DLQ and retry give durable, observable ingestion. | Adds a queue to operate and monitor. |
| Works naturally for event sources such as Jira, Confluence, S3, Kafka, or dlt. | End-to-end write latency becomes eventual, not synchronous. |
<Accordion title="Full queue and worker skeleton">
```hcl theme={null}
resource "aws_sqs_queue" "cognee_dlq" {
name = "cognee-ingest-dlq"
}
resource "aws_sqs_queue" "cognee_ingest" {
name = "cognee-ingest"
visibility_timeout_seconds = 900
redrive_policy = jsonencode({
deadLetterTargetArn = aws_sqs_queue.cognee_dlq.arn
maxReceiveCount = 3
})
}
output "ingest_queue_url" {
value = aws_sqs_queue.cognee_ingest.url
}
```
```yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
name: cognee-writer
spec:
replicas: 1
selector:
matchLabels:
app: cognee-writer
template:
metadata:
labels:
app: cognee-writer
spec:
containers:
- name: worker
image: cognee/cognee:1.0.6
env:
- name: INGEST_QUEUE_URL
valueFrom:
secretKeyRef:
name: cognee-queue
key: ingestQueueUrl
```
</Accordion>
</Tab>
</Tabs>
## 4. Serverless and managed
Serverless patterns are useful for HTTP-fronted memory APIs and scheduled jobs. They are rarely the primary on-prem pattern.
<Tabs>
<Tab title="4a. Lambda read-only artifact">
Run `remember()` offline, package the resulting Kuzu/LanceDB files into the deployment artifact or Lambda layer, and open them read-only at runtime.
```hcl theme={null}
resource "aws_lambda_function" "cognee_reader" {
function_name = "cognee-reader"
package_type = "Image"
image_uri = "${var.ecr_repo}:1.0.6-snapshot"
memory_size = 3008
timeout = 30
environment {
variables = {
COGNEE_READ_ONLY = "true"
DATA_ROOT_DIRECTORY = "/var/task/.cognee_data"
GRAPH_DATABASE_PROVIDER = "kuzu"
VECTOR_DB_PROVIDER = "lancedb"
}
}
}
```
This scales to zero and is rollback-friendly, but every knowledge update requires rebuilding and redeploying the snapshot.
| Pros | Cons |
| --------------------------------------------------------- | ----------------------------------------------------------- |
| No servers; scales to zero with per-request billing. | Read-only; every knowledge update needs a rebuilt artifact. |
| Immutable artifact is reproducible and rollback-friendly. | Cold start scales with artifact size. |
| No runtime database service to operate. | Bounded by Lambda image and runtime limits. |
<Accordion title="Full read-only Lambda skeleton">
```hcl theme={null}
resource "aws_lambda_function" "cognee_reader" {
function_name = "cognee-reader"
package_type = "Image"
image_uri = "${var.ecr_repo}:1.0.6-snapshot"
memory_size = 3008
timeout = 30
environment {
variables = {
COGNEE_READ_ONLY = "true"
DATA_ROOT_DIRECTORY = "/var/task/.cognee_data"
SYSTEM_ROOT_DIRECTORY = "/var/task/.cognee_system"
GRAPH_DATABASE_PROVIDER = "kuzu"
VECTOR_DB_PROVIDER = "lancedb"
DB_PROVIDER = "sqlite"
}
}
}
```
</Accordion>
</Tab>
<Tab title="4b. Lambda mutable graph on EFS">
Mount EFS and point Cognee at that mount for writable serverless memory. EFS gives shared storage, not write coordination, so concurrent writes still need a queue or external graph backend.
```hcl theme={null}
resource "aws_efs_file_system" "cognee" {
encrypted = true
}
resource "aws_lambda_function" "cognee_efs" {
function_name = "cognee-efs"
package_type = "Image"
image_uri = "${var.ecr_repo}:1.0.6"
timeout = 120
file_system_config {
arn = aws_efs_access_point.cognee.arn
local_mount_path = "/mnt/cognee"
}
environment {
variables = {
DATA_ROOT_DIRECTORY = "/mnt/cognee/.cognee_data"
SYSTEM_ROOT_DIRECTORY = "/mnt/cognee/.cognee_system"
}
}
}
```
Avoid EFS where POSIX locking matters under concurrency. Prefer block storage or an external graph backend for the hot graph path.
| Pros | Cons |
| ----------------------------------------------------- | --------------------------------------------------------------------------------------- |
| Writable persistent memory without managing a server. | No write coordination; concurrent writes still need queueing or external graph storage. |
| EFS survives Lambda redeploys. | EFS latency can hurt the hot graph path. |
| Shared across invocations. | Requires VPC wiring, security groups, and NAT or private LLM egress. |
<Accordion title="Full Lambda + EFS skeleton">
```hcl theme={null}
resource "aws_efs_file_system" "cognee" {
encrypted = true
}
resource "aws_efs_access_point" "cognee" {
file_system_id = aws_efs_file_system.cognee.id
posix_user {
gid = 1000
uid = 1000
}
root_directory {
path = "/cognee"
creation_info {
owner_gid = 1000
owner_uid = 1000
permissions = "0755"
}
}
}
resource "aws_lambda_function" "cognee_efs" {
function_name = "cognee-efs"
package_type = "Image"
image_uri = "${var.ecr_repo}:1.0.6"
memory_size = 3008
timeout = 120
vpc_config {
subnet_ids = var.private_subnets
security_group_ids = [var.lambda_sg]
}
file_system_config {
arn = aws_efs_access_point.cognee.arn
local_mount_path = "/mnt/cognee"
}
environment {
variables = {
DATA_ROOT_DIRECTORY = "/mnt/cognee/.cognee_data"
SYSTEM_ROOT_DIRECTORY = "/mnt/cognee/.cognee_system"
}
}
}
```
</Accordion>
</Tab>
</Tabs>
### Ephemeral cloud sandbox (Islo)
For a throwaway, HTTP-fronted API instance, Cognee ships a one-command deploy script that provisions an [Islo](https://islo.dev) cloud sandbox (2 vCPU / 4 GB / 10 GB), installs `cognee[api]` into a dedicated virtualenv, starts the FastAPI server on port 8000, gates on the internal `/health` endpoint, and then prints a public share URL that expires after 24 hours.
```bash theme={null}
pip install islo
# Mint a key with the Islo CLI, then export it:
export ISLO_API_KEY=... # islo api-key create cognee-deploy --expires 90 --show
export LLM_API_KEY=sk-...
python distributed/deploy/islo_sandbox.py
```
The Islo CLI is only used to mint the API key; the deployment itself is driven by the official Islo Python SDK. The sandbox name is fixed (`cognee-api`), so re-running the script while a previous deployment still exists fails with a name conflict — delete the old sandbox through the SDK first. You can still stop the sandbox separately when you want to pause it without recreating it. Best for demos and short-lived evaluation rather than durable state, since the share URL and sandbox are ephemeral. See [`distributed/deploy/README.md`](https://github.com/topoteretes/cognee/blob/main/distributed/deploy/README.md) for the full runbook, required environment variables, and cleanup commands.
## 5. Externalized backends
Use external services for sustained multi-agent writes and independent scaling of each storage layer. This removes the file-backed graph single-writer ceiling once the graph tier is externalized.
```hcl theme={null}
resource "aws_db_instance" "cognee" {
identifier = "cognee"
engine = "postgres"
engine_version = "17"
instance_class = "db.r6g.xlarge"
allocated_storage = 100
max_allocated_storage = 1000
storage_type = "gp3"
db_name = "cognee"
username = "cognee"
manage_master_user_password = true
multi_az = true
backup_retention_period = 14
storage_encrypted = true
}
```
Wire the external services into Helm:
```bash theme={null}
helm upgrade --install cognee ./cognee \
--set env.DB_PROVIDER=postgres \
--set env.VECTOR_DB_PROVIDER=pgvector \
--set env.GRAPH_DATABASE_PROVIDER=neo4j \
--set externalPostgres.host="$(terraform output -raw host)"
```
Typical production shape:
* Postgres or RDS for relational metadata
* pgvector, Qdrant, Pinecone, or ChromaDB for vectors
* Neo4j or the FalkorDB adapter for graph writes
* Cognee API pods configured as storage-backed application nodes with writable local paths for ingestion artifacts and caches
| Pros | Cons |
| -------------------------------------------------------------- | ----------------------------------------------------------------- |
| Highest write concurrency once graph storage is externalized. | Most infrastructure to provision, secure, and pay for. |
| Each tier scales, backs up, and fails over independently. | More moving parts means more failure modes and monitoring. |
| Managed services can provide HA, backups, and secret rotation. | Cross-service latency replaces local file access on the hot path. |
<Accordion title="Full externalized backend skeleton">
```hcl theme={null}
variable "name" {
default = "cognee"
}
variable "vpc_id" {}
variable "subnet_ids" {
type = list(string)
}
variable "app_sg" {}
resource "aws_db_subnet_group" "this" {
name = "${var.name}-db"
subnet_ids = var.subnet_ids
}
resource "aws_security_group" "db" {
name_prefix = "${var.name}-db-"
vpc_id = var.vpc_id
ingress {
from_port = 5432
to_port = 5432
protocol = "tcp"
security_groups = [var.app_sg]
}
}
resource "aws_db_instance" "cognee" {
identifier = var.name
engine = "postgres"
engine_version = "17"
instance_class = "db.r6g.xlarge"
allocated_storage = 100
max_allocated_storage = 1000
storage_type = "gp3"
db_name = "cognee"
username = "cognee"
manage_master_user_password = true
multi_az = true
backup_retention_period = 14
storage_encrypted = true
db_subnet_group_name = aws_db_subnet_group.this.name
vpc_security_group_ids = [aws_security_group.db.id]
}
# Create the pgvector extension once before using Postgres as a vector store.
# Run this through a migration, pre-deploy hook, or database init job:
# CREATE EXTENSION IF NOT EXISTS vector;
output "host" {
value = aws_db_instance.cognee.address
}
output "secret_arn" {
value = aws_db_instance.cognee.master_user_secret[0].secret_arn
}
```
```bash theme={null}
helm upgrade --install cognee ./cognee \
--set externalPostgres.host="$(terraform output -raw host)" \
--set env.DB_PROVIDER=postgres \
--set env.VECTOR_DB_PROVIDER=pgvector \
--set env.GRAPH_DATABASE_PROVIDER=neo4j
```
</Accordion>
## 6. Cloud service mapping
The patterns are cloud-agnostic. The concrete services differ by platform.
| Primitive | AWS | Azure |
| ------------------------ | --------------------------- | --------------------------------------------- |
| Managed Kubernetes | EKS | AKS |
| Writer block storage | EBS gp3 | Managed Disks / Premium SSD |
| Shared filesystem | EFS | Azure Files Premium NFS |
| Snapshots / object store | S3 | Blob Storage |
| Private registry | ECR | ACR |
| Secrets | Secrets Manager / SSM | Key Vault |
| Pod identity | IRSA | Workload Identity |
| Relational tier | RDS / Aurora Postgres | Azure Database for PostgreSQL Flexible Server |
| LLM endpoint | Bedrock or self-hosted vLLM | Azure OpenAI or self-hosted vLLM |
Keep the hot graph on block storage or an external graph service. Use shared filesystems only when the pattern truly requires cross-process file sharing.
## 7. Production readiness
### Schema migrations
* Relational: pin the Cognee version per environment and run migrations before deploying the live writer.
* Graph: additive model changes are safest. Renames, removals, and new required fields need a data migration or rebuild from source.
* Vector: rebuild collections when embedding dimension, distance metric, or metadata schema changes.
### Backups, restore, and DR
* Embedded: route writes through one writer, then snapshot the data directory to object storage.
* Externalized: back up each tier independently, such as Postgres dumps, managed snapshots, graph dumps, and vector snapshots.
* Region failure: use cross-region snapshot replication and warm standby. Active-active DR is not a good fit for file-backed Kuzu.
* Test restore on every release.
### Tenant isolation
| Level | Mechanism | Notes |
| -------------- | ----------------------------------------------- | -------------------------------------------------- |
| Logical | `dataset_name` and user filters | Cheap, but every query must apply the right scope. |
| Backend | `ENABLE_BACKEND_ACCESS_CONTROL=true` | Per-user and per-dataset storage isolation. |
| Infrastructure | Separate deployments, namespaces, and databases | Use for hard regulatory boundaries. |
### LLM egress and authentication
* Use self-hosted vLLM, customer-approved proxies, Bedrock private access, Azure OpenAI in-subscription, or fully air-gapped patterns when egress is restricted.
* Front the Cognee API with the customer gateway. Terminate OIDC or mTLS there rather than exposing the Cognee port directly.
* Use service-to-service mTLS or cluster-native identity.
* Propagate user identity through a gateway-validated header when Cognee needs user-scoped access.
## Appendix - options at a glance
| Option | Writer | Best for | Skeleton |
| ----------------- | ---------------------------- | ------------------------------------------------- | ------------------------------------------- |
| Embedded | Single process | Prototypes, notebooks, single-user agents | pip + env vars |
| Docker Compose | Single service | On-prem validation, first production step | Compose + pgvector |
| Coolify | Single service | Self-hosted PaaS on a single VPS with managed TLS | Compose + Coolify |
| Helm | Single service | Customers already on Kubernetes | Chart + values |
| Sidecar | Single service | Per-agent private memory | Pod spec |
| Snapshot replicas | One writer, many readers | Heavy read traffic, static knowledge | S3 + reader deployment |
| Queue | One worker | Bursty or event-driven ingestion | SQS + worker |
| Lambda read-only | Offline writer | HTTP memory API, scale-to-zero | Lambda image |
| Lambda + EFS | Single writer plus queue | Serverless writable memory | EFS + Lambda |
| Islo sandbox | Single process | Ephemeral demos, throwaway API instances | `python distributed/deploy/islo_sandbox.py` |
| Externalized | External graph-backed writes | Sustained multi-agent writes | RDS + Neo4j/FalkorDB adapter + vector DB |
For provider-specific configuration, see [Graph Stores](/setup-configuration/graph-stores), [Vector Stores](/setup-configuration/vector-stores), and [Relational Databases](/setup-configuration/relational-databases).
# Docker Deployment
Source: https://docs.cognee.ai/how-to-guides/cognee-sdk/deployment/docker
Deploy Cognee and its supporting services using Docker Compose profiles
Deploy Cognee locally or on a server with Docker Compose. The included `docker-compose.yml` uses **profiles** so you can start only the services you need.
## Prerequisites
* [Docker](https://docs.docker.com/get-docker/) and Docker Compose v2+
* Git — only for the build-from-source path; the minimal Compose file below needs no clone
## Quick Start
Two ways to start the API server — a prebuilt image for a quick try-out, or the repository compose file when you want profiles, the UI, MCP, or external databases:
<Tabs>
<Tab title="Minimal Compose (prebuilt image)">
To try the API server without cloning or building, save this single file as `docker-compose.yml` in an empty directory. It runs the prebuilt [`cognee/cognee:main`](https://hub.docker.com/r/cognee/cognee) image with the default local databases (SQLite, LanceDB, Ladybug), so an LLM API key is the only thing you supply:
```yaml theme={null}
services:
cognee:
image: cognee/cognee:main
ports:
- "8000:8000"
environment:
LLM_API_KEY: ${LLM_API_KEY:?set LLM_API_KEY to your OpenAI API key}
# Single-user try-out: no auth, shared local databases.
# Remove this line (or set it to true) for multi-tenant mode,
# which requires authentication on every API call.
ENABLE_BACKEND_ACCESS_CONTROL: "false"
```
Then start it:
```bash theme={null}
export LLM_API_KEY="sk-..." # your OpenAI API key
docker compose up
```
The `${LLM_API_KEY:?...}` guard is Compose variable interpolation: when `LLM_API_KEY` is unset, `docker compose up` aborts immediately and prints the message after `:?`, instead of starting a container that only fails later on the first LLM call.
<Warning>
`ENABLE_BACKEND_ACCESS_CONTROL: "false"` disables API authentication and per-user/dataset isolation so a first try-out needs no token. Use it for local experiments only — for anything shared or exposed, leave the flag at its `True` default and use the profile-based setup in the **Build from source** tab.
</Warning>
This file mounts nothing, so its data lives inside the container and is lost when the container is removed. See [Data Persistence and Host Files](#additional-information) for a named-volume variant. For other LLM providers, add the matching `LLM_PROVIDER` / `LLM_MODEL` / `LLM_ENDPOINT` variables — the repository `.env.template` lists them all.
</Tab>
<Tab title="Build from source">
```bash theme={null}
git clone https://github.com/topoteretes/cognee.git
cd cognee
cp .env.template .env
```
Edit `.env` and set your LLM API key:
```bash theme={null}
LLM_API_KEY="your_api_key"
```
Then start the Cognee API server (no profile needed):
```bash theme={null}
docker compose up --build cognee
```
</Tab>
</Tabs>
Either way, the API will be available at `http://localhost:8000`. Interactive docs at `http://localhost:8000/docs`.
## Verify Deployment
After the server starts, check that the API process is reachable:
```bash theme={null}
curl -f http://localhost:8000/health
```
This only proves that the server is alive. It does **not** prove that ingestion, graph building, vector search, or LLM-backed recall works.
### Container Health Status
The `cognee` and `cognee-mcp` images declare a Docker `HEALTHCHECK`, so Docker polls `/health` for you and tracks the result as container state. `docker ps` shows `(health: starting)`, `(healthy)`, or `(unhealthy)` in the `STATUS` column, and you can read the current state directly:
```bash theme={null}
docker inspect --format '{{.State.Health.Status}}' "$(docker compose ps -q cognee)"
```
| Image | Probe | Interval | Timeout | Start period | Retries |
| ------------------- | ----------------------------------------------------------------------- | -------- | ------- | ------------ | ------- |
| `cognee/cognee` | `curl -f http://localhost:8000/health` | `30s` | `10s` | `40s` | `3` |
| `cognee/cognee-mcp` | `GET http://localhost:8000/health`, skipped when `TRANSPORT_MODE=stdio` | `30s` | `10s` | `60s` | `3` |
In the MCP server's default `stdio` transport there is no HTTP server to probe, so the check reports healthy without touching the network. It only makes a real request under the HTTP/SSE transports.
Both probes target port `8000` **inside** the container, which is where the entrypoint binds by default — including the `mcp` profile, where `8001` is only the published host port. If you change `HTTP_PORT`, the baked-in healthcheck no longer matches the listening port; override `healthcheck.test` for that service in your compose file.
Because the health state is part of the image, other services can wait on Cognee the same way the [Postgres and Neo4j examples](#permissionerror-external-databases) wait on their databases:
```yaml theme={null}
services:
my-service:
depends_on:
cognee:
condition: service_healthy
```
<Note>
This metadata is baked in at build time. Images published before the `HEALTHCHECK` was added carry no health state at all: `docker ps` shows a plain `Up` status with no health annotation, and the `docker inspect` command above has nothing to report. If `condition: service_healthy` never becomes satisfiable, pull a newer tag or rebuild locally with `docker compose up --build cognee`.
</Note>
### Image Provenance and SBOM
Images built by the release pipeline carry in-toto provenance and SBOM attestations, pushed alongside the image manifest. To confirm an image was built by CI from `topoteretes/cognee` and to inspect its bill of materials:
```bash theme={null}
# Provenance attestation
docker buildx imagetools inspect cognee/cognee:latest --format '{{ json .Provenance }}'
# SBOM attestation
docker buildx imagetools inspect cognee/cognee:latest --format '{{ json .SBOM }}'
```
The same commands work for the MCP image (`cognee/cognee-mcp`). Like the healthcheck metadata above, attestations are attached at build time — images published before the release pipeline added them (August 2026) have nothing to report. For the full mechanism, see [Supply-chain provenance & release attestations](https://github.com/topoteretes/cognee/blob/dev/docs/supply_chain_provenance.md) in the cognee repo.
## Smoke Test Ingestion and Recall
Docker users often test API routes immediately after startup. Cognee API endpoints use the versioned `/api/v1` prefix, not plain `/api`; see [API Base URLs](/api-reference/introduction#api-base-urls) for the full API reference note.
By default, `ENABLE_BACKEND_ACCESS_CONTROL=True` makes API authentication required. For a local unauthenticated smoke test, set `ENABLE_BACKEND_ACCESS_CONTROL=false` in `.env` and restart the container, or include a valid Bearer token in the `curl` requests.
Create a small file, ingest it synchronously, then query the same dataset:
```bash theme={null}
printf "Cognee turns data into searchable AI memory." > /tmp/cognee-smoke.txt
curl -X POST http://localhost:8000/api/v1/remember \
-F "data=@/tmp/cognee-smoke.txt" \
-F "datasetName=smoke_test" \
-F "run_in_background=false"
curl -X POST http://localhost:8000/api/v1/recall \
-H "Content-Type: application/json" \
-d '{"query": "What does Cognee do?", "datasets": ["smoke_test"], "search_type": "GRAPH_COMPLETION", "top_k": 5}'
```
On the minimal Compose stack above, `ENABLE_BACKEND_ACCESS_CONTROL` is already `false`, so these calls work unauthenticated with no further changes. If you prefer the explicit three-step flow over `remember`/`recall`, the same result comes from `add` → `cognify` → `search`:
```bash theme={null}
echo "Cognee turns documents into AI memory." > note.txt
# Ingest a file — /api/v1/add takes a multipart upload, it does not accept inline text
curl -X POST http://localhost:8000/api/v1/add \
-F "data=@note.txt" \
-F "datasetName=main_dataset"
# Build the knowledge graph
curl -X POST http://localhost:8000/api/v1/cognify \
-H "Content-Type: application/json" \
-d '{"datasets": ["main_dataset"]}'
# Search it
curl -X POST http://localhost:8000/api/v1/search \
-H "Content-Type: application/json" \
-d '{"searchType": "GRAPH_COMPLETION", "query": "What does Cognee do?", "datasets": ["main_dataset"]}'
```
## Troubleshooting
<AccordionGroup>
<Accordion title="PermissionError with External Databases">
Even when Cognee is configured to use external databases (Postgres, pgvector, Neo4j, etc.), local writable paths are **still required**. `DATA_ROOT_DIRECTORY` (SDK default `.data_storage`) and `SYSTEM_ROOT_DIRECTORY` (SDK default `.cognee_system`) hold ingestion artifacts, file caches, and loader outputs — they are not bypassed by pointing the relational, vector, or graph backends elsewhere.
The `cognee/cognee` and `cognee/cognee-mcp` images override those defaults to `/cognee-storage/data` and `/cognee-storage/system`, and run as the non-root user `cognee` (**uid/gid 1000**). If the mounted path is read-only or owned by another user, ingestion fails with:
```
PermissionError: [Errno 13] Permission denied: '/cognee-storage/data/...'
```
The usual cause is a **host bind mount**: named volumes inherit the image's `cognee:cognee` ownership, but a bind-mounted host directory keeps the host's ownership, which is rarely uid 1000. Fix it on the host before starting the container:
```bash theme={null}
sudo chown -R 1000:1000 ./my-cognee-storage
```
**Fix — mount writable volumes at the image's storage roots**:
```yaml theme={null}
services:
cognee:
image: cognee/cognee:main
volumes:
- cognee_data:/cognee-storage/data
- cognee_system:/cognee-storage/system
environment:
DB_PROVIDER: postgres
# ... remaining DB / graph / vector settings
volumes:
cognee_data:
cognee_system:
```
If you relocate the storage paths with `DATA_ROOT_DIRECTORY` and `SYSTEM_ROOT_DIRECTORY`, mount the volumes at the same paths:
```yaml theme={null}
services:
cognee:
image: cognee/cognee:main
volumes:
- cognee_data:/var/cognee/data
- cognee_system:/var/cognee/system
environment:
DATA_ROOT_DIRECTORY: /var/cognee/data
SYSTEM_ROOT_DIRECTORY: /var/cognee/system
DB_PROVIDER: postgres
# ... remaining DB / graph / vector settings
volumes:
cognee_data:
cognee_system:
```
**Working Postgres + pgvector + Neo4j compose example** — includes healthchecks on both `postgres` and `neo4j` so Cognee does not start before either database is ready (Cognee otherwise races Neo4j's Bolt listener and exits with a connection error):
```yaml theme={null}
services:
postgres:
image: pgvector/pgvector:pg17
environment:
POSTGRES_USER: cognee
POSTGRES_PASSWORD: cognee
POSTGRES_DB: cognee_db
healthcheck:
test: ["CMD-SHELL", "pg_isready -U cognee -d cognee_db"]
interval: 10s
timeout: 5s
retries: 5
neo4j:
image: neo4j:5.26
environment:
NEO4J_AUTH: neo4j/pleaseletmein
healthcheck:
test: ["CMD-SHELL", "cypher-shell -u neo4j -p pleaseletmein 'RETURN 1'"]
interval: 10s
timeout: 5s
retries: 10
start_period: 30s
cognee:
image: cognee/cognee:main
depends_on:
postgres:
condition: service_healthy
neo4j:
condition: service_healthy
volumes:
- cognee_data:/cognee-storage/data
- cognee_system:/cognee-storage/system
environment:
DB_PROVIDER: postgres
DB_HOST: postgres
DB_PORT: 5432
DB_USERNAME: cognee
DB_PASSWORD: cognee
DB_NAME: cognee_db
VECTOR_DB_PROVIDER: pgvector
GRAPH_DATABASE_PROVIDER: neo4j
GRAPH_DATABASE_URL: bolt://neo4j:7687
GRAPH_DATABASE_USERNAME: neo4j
GRAPH_DATABASE_PASSWORD: pleaseletmein
volumes:
cognee_data:
cognee_system:
```
See [Storage & Logging](/setup-configuration/overview#storage-amp-logging) for the related env vars, or [S3 storage](/guides/s3-storage) if you want to point these directories at S3 instead of local volumes.
</Accordion>
<Accordion title="PostgreSQL Connection Refused">
When Cognee starts before PostgreSQL finishes initializing, the first API call triggers LLM/embedding connectivity checks (`setup_and_check_environment`) and may hit the database before it accepts connections, producing `[Errno 111] Connection refused` or `[Errno 99] Cannot assign requested address`.
**Recommended fix — add a healthcheck and `depends_on` condition to your `docker-compose.yml`.** The shipped compose file already carries this exact `pg_isready` healthcheck on the `postgres` service, so with it you only need to add the `depends_on` guard; the full example below is for hand-written compose files:
```yaml theme={null}
services:
postgres:
image: pgvector/pgvector:pg17
environment:
POSTGRES_USER: cognee
POSTGRES_PASSWORD: cognee
POSTGRES_DB: cognee_db
healthcheck:
test: ["CMD-SHELL", "pg_isready -U cognee -d cognee_db"]
interval: 10s
timeout: 5s
retries: 5
cognee:
image: cognee/cognee:main
depends_on:
postgres:
condition: service_healthy
environment:
DB_PROVIDER: postgres
DB_HOST: postgres
DB_PORT: 5432
DB_USERNAME: cognee
DB_PASSWORD: cognee
DB_NAME: cognee_db
```
This delays the `cognee` container until PostgreSQL passes its health check.
**Alternative fix — bypass the connectivity check:**
If you cannot modify the compose file (e.g. third-party orchestration), set `COGNEE_SKIP_CONNECTION_TEST=true` to skip the LLM/embedding startup probe entirely. The check is only performed once (on first run), so the trade-off is that misconfigured endpoints are not caught until the first real request.
```bash theme={null}
COGNEE_SKIP_CONNECTION_TEST=true
```
</Accordion>
<Accordion title="Migration Fails on First Boot">
The entrypoint runs [startup migrations](#database-migrations-on-startup) before the server starts, and a failed relational migration exits non-zero — so the container stops right after `Running database migrations...` instead of reaching `Starting server...`. Three causes account for most first-boot failures:
* **The storage directories are not writable.** The relational database (SQLite by default) and its parent directory live under `DATA_ROOT_DIRECTORY` / `SYSTEM_ROOT_DIRECTORY`. If those paths are read-only or owned by another user, the migration cannot create or open the database file. Mount writable volumes for both, as shown in [PermissionError with External Databases](#permissionerror-external-databases).
* **An external database is not reachable yet.** With `DB_PROVIDER=postgres`, the migration runs before the server would otherwise touch the database, so a Postgres container that is still initializing fails the boot. Add a `depends_on: condition: service_healthy` guard (the shipped compose file already has the healthcheck), as in [PostgreSQL Connection Refused](#postgresql-connection-refused).
* **The Postgres connection settings are incomplete.** With `DB_PROVIDER=postgres`, the port is cast with `int()` while the engine is built, and `DB_PORT` has no application-level default — the shipped compose file supplies `5432`, so this typically hits direct `docker run` or platform deployments. Leaving it unset aborts the boot with `TypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType'`, most often because the connection was supplied as a single platform-injected `DATABASE_URL`, which Cognee does not read. See [`DB_PORT` unset](/setup-configuration/relational-databases#troubleshooting) for the split `DB_*` variables to set instead.
The first two are safe to retry: restart the container once the volume or database is ready and the migration runs again from where it left off. The third needs the environment fixed first.
**Operator-driven alternative** — if you would rather migrate outside container startup (for example, from a one-shot job that runs before the app rolls out), disable the automatic run and invoke the CLI yourself:
```yaml theme={null}
services:
cognee:
image: cognee/cognee:main
environment:
ENABLE_AUTO_MIGRATIONS: "false"
```
```bash theme={null}
# Run once, before starting the app container
docker compose run --rm cognee cognee-cli upgrade
```
`cognee-cli upgrade` ignores `ENABLE_AUTO_MIGRATIONS` and always migrates. Leaving migrations disabled without running it means the schema is never brought to head — the server starts against whatever schema exists.
</Accordion>
<Accordion title="Web UI Login Loops Back to the Login Page">
On the `ui` profile, signing in returns `200` but the app immediately bounces back to `/local-login`, repeating on every attempt.
The auth cookie is host-scoped: it carries no `Domain` attribute, so a cookie set on `localhost` is not sent to `127.0.0.1` and vice versa. Older frontend builds always sent local API requests to a hard-coded `http://localhost:8000`, so opening the UI on `http://127.0.0.1:3000` stored the cookie for `localhost` while the page ran on `127.0.0.1`. The follow-up `GET /api/v1/users/me` check went out without the cookie, returned `401`, and the UI redirected back to the login page.
Current builds resolve the API host from the page you loaded, so `localhost` and `127.0.0.1` both work — the shipped compose file allows every origin (`CORS_ALLOWED_ORIGINS=${CORS_ALLOWED_ORIGINS:-*}`), so no extra configuration is needed. If you still see the loop:
* **Update the image.** The `frontend` service runs the published `cognee/cognee-ui` image, so an old local copy of a moving tag keeps serving old code: `docker compose --profile ui pull frontend && docker compose --profile ui up -d frontend`. On the `ui-dev` profile, which builds from `./cognee-frontend` and bind-mounts `src`, pull your checkout and rebuild instead: `docker compose --profile ui-dev up -d --build frontend-dev`.
* **Check any `COGNEE_BACKEND_URL` or `NEXT_PUBLIC_LOCAL_API_URL` you set.** Either one wins over the browser-derived host — `COGNEE_BACKEND_URL` first — so it reintroduces the mismatch if its hostname differs from the one in your address bar. `curl -s http://localhost:3000/api/runtime-config` shows what the container resolved.
* **Check any narrowed `CORS_ALLOWED_ORIGINS`.** If you replaced the default `*`, it must name the exact origin you browse to, port included.
* **Clear cookies** for both `localhost` and `127.0.0.1`, then sign in again.
</Accordion>
<Accordion title="Admin Can't See Agent Datasets in the Web UI">
A common solo-operator setup: one self-hosted Cognee, an agent connected over MCP writing to its own dataset, and you signing into the web UI as the admin superuser — where the datasets page is empty, even though the agent's `remember` calls succeed and its data is intact.
This is access control working as designed, not a failed ingestion. With `ENABLE_BACKEND_ACCESS_CONTROL=True` (the Docker default — see the Docker Environment Variables accordion under Additional Information), every dataset listing — including the UI's datasets page, backed by `GET /api/v1/datasets` — returns only datasets the authenticated user holds explicit `read` permission on. Superuser status does not bypass dataset permissions: it covers [user, tenant, and role management and system settings](/core-concepts/multi-user-mode/permissions-system/users#superuser-privileges), not dataset access. A dataset created by an agent user grants `read`/`write`/`delete`/`share` to that agent alone, so the admin account has no path to it.
Two fixes — one for datasets that already exist, one for every dataset created after it:
1. **Grant yourself access to the agent's existing dataset.** Only a principal holding `share` on a dataset can grant permissions on it, and that is the agent, not you. So make the request authenticated as the agent (its API key or credentials), granting your admin user id `read` — plus `write` and `delete` if you want to manage the dataset from the UI:
```bash theme={null}
curl -X POST "http://localhost:8000/api/v1/permissions/datasets/<admin-user-id>?permission_name=read" \
-H "X-Api-Key: <agent-api-key>" \
-H "Content-Type: application/json" \
-d '["<dataset-id>"]'
```
The agent's `GET /api/v1/datasets` gives the dataset id; your admin user id comes from `GET /api/v1/users/me`.
2. **Create agent identities as children of your admin user.** [`cognee.agents.create()`](/python-api/agents) — or `create_user(..., parent_user_id=<your-user-id>)` — mints the agent with [`parent_user_id`](/core-concepts/multi-user-mode/permissions-system/users) pointing at you, and every dataset the agent creates from then on automatically grants the parent full permissions. `parent_user_id` can only be set when the user is created — `PATCH /api/v1/users/{id}` does not accept it — so an existing standalone agent user cannot be re-parented: grant per-dataset access as above, or recreate the agent as a child (which rotates its API key).
The same grants are how two agents share one brain — the dataset's owner grants the second agent `read` and `write` on it. See [Permission Snippets](/guides/permission-snippets) for the full patterns.
</Accordion>
</AccordionGroup>
## Additional Information
<AccordionGroup>
<Accordion title="Docker Compose Services">
Each optional service is gated behind a profile. Use `--profile` to activate one or more:
| Profile | Service | Port(s) | Purpose |
| ---------- | -------------- | -------------- | -------------------------------------------------------------------------------------- |
| *(none)* | `cognee` | `8000`, `5678` | Core API server |
| *(none)* | `redisinsight` | `5540` | RedisInsight GUI for inspecting Redis; developer convenience only |
| `mcp` | `cognee-mcp` | `8001`, `5679` | MCP server for IDE integrations (host ports; container still listens on `8000`/`5678`) |
| `ui` | `frontend` | `3000` | Experimental web UI, pulled from the published `cognee/cognee-ui` image |
| `ui-dev` | `frontend-dev` | `3000` | The same UI built from `./cognee-frontend` with hot reload, for frontend development |
| `neo4j` | `neo4j` | `7474`, `7687` | Neo4j graph database |
| `postgres` | `postgres` | `5432` | PostgreSQL + pgvector |
| `redis` | `redis` | `6379` | Redis session cache |
Services with no profile start on a bare `docker compose up`, so `redisinsight` comes up alongside `cognee` unless you name the services you want (`docker compose up cognee`).
`ui` and `ui-dev` both publish host port `3000`, so they are alternatives rather than additions — activating both fails on the port collision.
For what each service does, whether you need it, and how `cognee-network`, `extra_hosts`, and the resource limits work, see the [Docker Compose Reference](/how-to-guides/cognee-sdk/deployment/docker-compose-reference).
</Accordion>
<Accordion title="Data Persistence and Host Files">
Both images store their data **outside the source tree**, under `/cognee-storage`. The `Dockerfile` and `cognee-mcp/Dockerfile` bake in these defaults:
```dockerfile theme={null}
ENV SYSTEM_ROOT_DIRECTORY=/cognee-storage/system
ENV DATA_ROOT_DIRECTORY=/cognee-storage/data
```
The compose file mounts the `cognee_system` and `cognee_data` named volumes at exactly those paths, on **both** the `cognee` and `cognee-mcp` services, so the API server and the MCP server share one memory store and it survives container recreation. The `./cognee` bind mount is for dev reload only — it is no longer where data is persisted — and `.env` is mounted read-only (`:ro`).
The database services each map to a distinct role, but only services with an active `volumes:` entry in `docker-compose.yml` persist data through container recreation by default:
| Storage area | Role | Persistence in the checked-in compose file |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------ |
| `redis` | [Session/conversation cache](/core-concepts/sessions-and-caching) | Uses the mounted `redis_data` named volume |
| `postgres` | Relational metadata/state ([SQLite](/setup-configuration/relational-databases) by default, or Postgres), and vector store when `VECTOR_DB_PROVIDER=pgvector` | Uses the mounted `postgres_data` named volume |
| Embedded graph store | [Knowledge graph](/setup-configuration/graph-stores) files under `SYSTEM_ROOT_DIRECTORY` (`/cognee-storage/system` in both images) | Uses the mounted `cognee_system` named volume, shared with `cognee-mcp` |
| Ingestion artifacts | Uploaded files, loader outputs, and caches under `DATA_ROOT_DIRECTORY` (`/cognee-storage/data` in both images) | Uses the mounted `cognee_data` named volume, shared with `cognee-mcp` |
| `neo4j` | Dedicated graph database when `GRAPH_DATABASE_PROVIDER=neo4j` | Runs in its own service; add a Neo4j data volume for durability across container recreation |
| ChromaDB | [Vector store](/setup-configuration/vector-stores) for embeddings when `VECTOR_DB_PROVIDER=chromadb` | Not in the shipped compose file; persist the Chroma data directory in the Chroma service you add |
If `GRAPH_DATABASE_PROVIDER` is unset, the application default graph provider is **Ladybug**. The repository `.env.template` currently sets **Kuzu** for Docker. Both are embedded file-based graph stores, so the graph files live under `SYSTEM_ROOT_DIRECTORY` unless you switch to a dedicated graph service.
The shipped compose file is therefore already persistent for the embedded stores. If you prefer an external graph database, run Neo4j with `--profile neo4j` and set `GRAPH_DATABASE_PROVIDER=neo4j`. See [Cognee + PostgreSQL + Neo4j](#postgresql-neo4j) and [PermissionError with External Databases](#permissionerror-external-databases) for volume examples.
The [minimal Compose file](#quick-start) mounts nothing, so an image-only run keeps everything inside the container and loses it on `docker compose down`. To keep data across container recreation, extend that file with named volumes at the image's default storage roots — no environment variables needed, since the image already points there:
```yaml theme={null}
services:
cognee:
# ...minimal file from Quick Start, plus:
volumes:
- cognee_system:/cognee-storage/system
- cognee_data:/cognee-storage/data
volumes:
cognee_system:
cognee_data:
```
<Note>
A fresh named volume mounted at `/cognee-storage/system` or `/cognee-storage/data` inherits the ownership Docker finds baked into the image at that path — `cognee:cognee` (uid/gid 1000), the user the container runs as — so it initializes writable with no `chown` on your part. A **host bind mount** does not: Docker uses the host directory's existing ownership, so `chown 1000:1000` it before starting the container. See [PermissionError with External Databases](#permissionerror-external-databases).
</Note>
To ingest files from your host machine, uncomment and update the volume in `docker-compose.yml`.
```yaml theme={null}
# - /path/to/your/data:/data
```
</Accordion>
<Accordion title="Docker Environment Variables">
The `cognee` container reads configuration from `.env` at startup. Key variables:
| Variable | Default | Description |
| ------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `LLM_API_KEY` | *(required)* | API key for your LLM provider |
| `LLM_MODEL` | `openai/gpt-5-mini` | LLM model to use |
| `DB_PROVIDER` | `sqlite` | Relational DB: `sqlite` or `postgres` |
| `GRAPH_DATABASE_PROVIDER` | `kuzu` in `.env.template` | Graph DB: `kuzu`, `neo4j`, etc. If unset, the application default is `ladybug`. |
| `VECTOR_DB_PROVIDER` | `lancedb` | Vector DB: `lancedb`, `chromadb`, `pgvector`, etc. |
| `SYSTEM_ROOT_DIRECTORY` | `/cognee-storage/system` (baked into both images) | Embedded graph/vector files and other system state. Mount a volume here to persist it |
| `DATA_ROOT_DIRECTORY` | `/cognee-storage/data` (baked into both images) | Ingested files, loader outputs, and caches. Mount a volume here to persist it |
| `CORS_ALLOWED_ORIGINS` | `*` in Docker Compose | Restrict to specific domains in production |
| `HTTP_PORT` | `8000` | Port the API server binds inside the container (entrypoint default) |
| `BIND_ADDRESS` | `0.0.0.0` | Address the API server binds inside the container (entrypoint default) |
| `ENABLE_BACKEND_ACCESS_CONTROL` | `True` | Enables per-user/dataset isolation. When this is `True`, authentication is required. |
| `REQUIRE_AUTHENTICATION` | Inherits from `ENABLE_BACKEND_ACCESS_CONTROL` when unset | Enable JWT auth for the API. Setting this to `False` is ignored when `ENABLE_BACKEND_ACCESS_CONTROL=True`. |
| `COGNEE_SKIP_CONNECTION_TEST` | `false` | Skip LLM/embedding connectivity checks on startup, and the [zero-network provider-consistency check](/setup-configuration/overview#configuration-workflow) that `add()` and `remember()` run. Accepts `true`, `1`, or `yes`. |
| `ENABLE_AUTO_MIGRATIONS` | `true` | Run database migrations automatically, including at container startup. Set to `false` (or `0`/`no`) to migrate explicitly with `cognee-cli upgrade` instead — see [Database Migrations on Startup](#database-migrations-on-startup). |
| `DEBUG` | `false` | When `true` and `ENV` is `dev` or `local`, the container entrypoint starts under `debugpy` listening on `DEBUG_PORT` |
| `DEBUG_PORT` | `5678` | Port `debugpy` listens on when `DEBUG=true` |
| `chunk_size` | `1500` | Max tokens per chunk during cognify (see [Chunkers](/core-concepts/further-concepts/chunkers)) |
| `chunk_overlap` | `10` | Overlap between chunks in words (only affects `LangchainChunker`) |
`ENVIRONMENT` is a deprecated alias for `ENV`, still accepted by the container entrypoints — prefer `ENV`.
See the full list of options in [Setup Configuration](/setup-configuration/overview).
</Accordion>
<Accordion title="Database Migrations on Startup">
Before the API server binds its port, the container entrypoint runs Cognee's own startup migrations — the same [`run_migrations()`](/python-api/run-migrations) path used by the API server's lifespan and by `cognee-cli`. You will see this in the container logs:
```
Running database migrations...
Database migrations done.
Starting server...
```
What runs depends on the state of the database it finds:
* **Fresh volume / empty database** — Cognee creates the missing directories and builds the schema by running the entire Alembic migration chain; nothing is stamped, so `alembic_version` only records revisions that actually executed. A database is only treated as empty when Cognee inspected it and found neither a `users` table nor an `alembic_version` table, or when it is a local SQLite database whose file does not exist yet — so a pre-Alembic legacy database is migrated rather than mistaken for an empty one.
* **Existing database** — Alembic applies the pending relational revisions, then the graph/vector data migration chain runs.
* **Database Cognee cannot inspect** — a database that is unreachable, still starting up, or rejecting the connection is **not** counted as empty. The inspection error aborts the boot instead of trying to build the schema from scratch against a database that may simply not be ready yet. Restart the container once the database is ready — see [Troubleshooting → Migration Fails on First Boot](#migration-fails-first-boot).
A failed **relational** migration aborts the boot with a non-zero exit rather than starting the server on an unmigrated schema. A per-dataset **data-chain** failure does not stop the boot: the server comes up, Cognee blocks writes to just those datasets, and the migration is retried on the next start. The entrypoint prints the affected datasets:
```
Data migrations failed for: <dataset ids>. Writes to those datasets are blocked until they migrate; retried on the next start.
```
Set `ENABLE_AUTO_MIGRATIONS=false` to turn off this automatic run and migrate explicitly instead — `cognee-cli upgrade` ignores the flag and always migrates. See [Troubleshooting → Migration Fails on First Boot](#migration-fails-first-boot) if the container exits during this step.
</Accordion>
<Accordion title="Common setups">
<AccordionGroup>
<Accordion title="Cognee + PostgreSQL">
PostgreSQL with pgvector is a good production choice for the relational database.
Add to your `.env`:
```bash theme={null}
DB_PROVIDER=postgres
DB_HOST=postgres
DB_PORT=5432
DB_USERNAME=cognee
DB_PASSWORD=cognee
DB_NAME=cognee_db
```
Start both services:
```bash theme={null}
docker compose --profile postgres up --build
```
</Accordion>
<Accordion title="Cognee + PostgreSQL + Neo4j">
For production deployments with a dedicated graph database:
Add to your `.env`:
```bash theme={null}
# Relational DB
DB_PROVIDER=postgres
DB_HOST=postgres
DB_PORT=5432
DB_USERNAME=cognee
DB_PASSWORD=cognee
DB_NAME=cognee_db
# Graph DB
GRAPH_DATABASE_PROVIDER=neo4j
GRAPH_DATABASE_URL=bolt://neo4j:7687
GRAPH_DATABASE_NAME=neo4j
GRAPH_DATABASE_USERNAME=neo4j
GRAPH_DATABASE_PASSWORD=pleaseletmein
```
The shipped `postgres` service already mounts the `postgres_data` volume. Neo4j does not, so add one for graph durability across container recreation:
```yaml theme={null}
services:
neo4j:
volumes:
- neo4j_data:/data
volumes:
neo4j_data:
```
Start the stack:
```bash theme={null}
docker compose --profile postgres --profile neo4j up --build
```
Neo4j browser is available at `http://localhost:7474`.
</Accordion>
<Accordion title="Cognee + ChromaDB">
Use ChromaDB as the vector store. The shipped `docker-compose.yml` has no `chromadb` service, so add one yourself:
```yaml theme={null}
services:
chromadb:
image: chromadb/chroma:latest
profiles:
- chromadb
ports:
- 8002:8000
networks:
- cognee-network
```
Add to your `.env`:
```bash theme={null}
VECTOR_DB_PROVIDER=chromadb
VECTOR_DB_URL=http://chromadb:8000
VECTOR_DB_KEY=your_chroma_token
```
Start:
```bash theme={null}
docker compose --profile chromadb up --build
```
</Accordion>
<Accordion title="Cognee + MCP Server">
Run the [MCP server](/cognee-mcp/mcp-overview) alongside the API:
```bash theme={null}
docker compose --profile mcp up --build cognee-mcp
```
The MCP server uses SSE transport and is published on host port `8001` (the container itself still listens on `8000`, so the `mcp` profile doesn't collide with the `cognee` API service when both run). Configure your IDE to point to `http://localhost:8001/sse`. The debugger is published on host port `5679`.
</Accordion>
<Accordion title="Cognee + Web UI">
The `ui` profile starts the same web interface that [`cognee.start_ui()`](/cognee-cloud/local-ui) launches locally — here it runs as a separate `frontend` container, pulled from the published `cognee/cognee-ui` image rather than built from your checkout:
```bash theme={null}
docker compose --profile ui up
```
The container waits for the `cognee` service's healthcheck to pass before it starts (`depends_on: condition: service_healthy`), so the UI does not come up against a backend that is still booting.
The backend API and the UI listen on **different ports**, so they don't conflict:
| Service | URL | Port |
| ---------------------- | ----------------------- | ------ |
| API backend (`cognee`) | `http://localhost:8000` | `8000` |
| Web UI (`frontend`) | `http://localhost:3000` | `3000` |
The compose file defaults to `cognee/cognee-ui:latest`. Override it with `COGNEE_UI_TAG` to pin a different tag:
```bash theme={null}
COGNEE_UI_TAG=main docker compose --profile ui up
```
Every push to `main` publishes `latest`, a `main` branch tag, and a `main-<short-sha>` tag; releases add the release version. The image is built for `linux/amd64` and `linux/arm64` and runs as a non-root user.
**Pointing the UI at a backend.** The `frontend` service passes `COGNEE_BACKEND_URL` through from your environment or `.env`, and the image reads it **at run time** on every request — which is what lets one prebuilt image serve any backend without a rebuild. Leave it unset for the default Compose setup: the browser then derives the backend host from the address you loaded the UI from, on port `8000`, so `localhost` and `127.0.0.1` both work. Set it when the API is reachable somewhere else:
```bash theme={null}
COGNEE_BACKEND_URL=https://cognee.example.com docker compose --profile ui up
```
The browser calls the backend directly, so this must be the address **as seen from the browser** — never the `cognee` service name, which only resolves inside `cognee-network`.
**Remote backend and CORS.** When the UI and the API run on different hosts, pointing the UI at the backend is only half the wiring — the backend must also allow the UI's origin through its CORS policy, or the browser refuses every call with the UI's generic "Cannot connect" error:
<Tabs>
<Tab title="On the UI host">
```yaml theme={null}
services:
frontend:
image: cognee/cognee-ui:latest
ports:
- "3000:3000"
environment:
COGNEE_BACKEND_URL: http://cognee-api.example.com:8000
```
</Tab>
<Tab title="On the backend host">
```yaml theme={null}
services:
cognee:
image: cognee/cognee:main
ports:
- "8000:8000" # must be published — the browser calls it directly
environment:
CORS_ALLOWED_ORIGINS: http://cognee-ui.example.com:3000
```
</Tab>
</Tabs>
`CORS_ALLOWED_ORIGINS` is a comma-separated list and must name the exact origin in your address bar, scheme and port included. The shipped compose file defaults it to `*`, so this only needs attention when you narrow it or start the backend some other way — a backend launched without it allows only `UI_APP_URL`, which defaults to `http://localhost:3000`, so a remote UI is refused until you set one of the two.
Behind a reverse proxy that terminates TLS, use the proxy's public URLs instead (`COGNEE_BACKEND_URL: https://cognee-api.example.com`, `CORS_ALLOWED_ORIGINS: https://cognee-ui.example.com`) and keep port `8000` unpublished on the backend host.
<Warning>
**The `frontend` container exits immediately.** The image validates `COGNEE_BACKEND_URL` in its entrypoint and refuses to start on a value it cannot use, rather than booting and answering every request with a `500`. `docker compose logs frontend` shows the reason:
```
[cognee-ui] COGNEE_BACKEND_URL must be an absolute http(s) URL, got "cognee:8000". Expected something like "http://localhost:8000".
```
The value must include an `http://` or `https://` scheme. An unset or empty value is fine — that is the default — so this only ever fires on a value you set. Because the service runs with `restart: always`, a bad value shows up as a crash loop.
</Warning>
The container's healthcheck probes `GET /api/runtime-config`, which also serves as a manual check that the backend URL resolved the way you expect:
```bash theme={null}
curl -s http://localhost:3000/api/runtime-config
# {"backendUrl":"https://cognee.example.com"} — or {"backendUrl":null} when unset
```
**Working on the frontend.** Use `ui-dev` instead, which builds the hot-reloading `dev` stage of `cognee-frontend/Dockerfile` from your checkout and bind-mounts `src` and `public`, so edits show up without a rebuild:
```bash theme={null}
docker compose --profile ui-dev up --build
```
It serves on the same host port `3000` and takes the same `COGNEE_BACKEND_URL`. Unlike `frontend`, it has no `depends_on` guard, so start the backend yourself if you need it up first.
<Note>
Don't also call `cognee.start_ui()` while the `ui` (or `ui-dev`) profile is running — both bind port `3000`, so the second will fail with a "port already in use" error. In a Docker deployment use the `ui` profile; reserve [`cognee.start_ui()`](/cognee-cloud/local-ui) for non-Docker, local Python setups.
</Note>
</Accordion>
</AccordionGroup>
</Accordion>
<Accordion title="Managing the Docker Deployment">
The `cognee` container reads `.env` **once at startup**, so edits to `.env` are not picked up by a running container. Restart the service to apply them:
```bash theme={null}
# Re-reads .env and restarts the cognee process
docker compose restart cognee
```
If you changed the `docker-compose.yml` definition itself (ports, volumes, `environment:`, profiles), recreate the container instead so the new settings take effect:
```bash theme={null}
docker compose up -d --force-recreate cognee
```
You only need `--build` when you change the `Dockerfile`, its dependencies, or the build arguments passed to it (for example, [adding optional extras](#additional-information) via `COGNEE_EXTRAS`) — not for `.env` edits:
```bash theme={null}
docker compose up --build cognee
```
<Note>
Your `.env` and the `cognee/` source directory are bind-mounted into the container, so a restart is enough to apply config changes — no rebuild required.
</Note>
Stop or remove containers with Docker Compose:
```bash theme={null}
# Stop containers (preserves volumes)
docker compose down
# Stop and remove volumes (deletes all data)
docker compose down --volumes
```
</Accordion>
<Accordion title="Optional Extras and Document Loaders">
The default Docker image includes a fixed set of extras from the repository `Dockerfile`: `fastembed`, `debug`, `api`, `postgres`, `neo4j`, `llama-index`, `aws`, `dlt`, `ollama`, `mistral`, `groq`, and `anthropic`. In particular, the `aws` extra (s3fs/boto3 for [S3 file storage](/guides/s3-storage)) is part of the defaults, so it does not need to be added at build time. The `fastembed` extra (`fastembed` plus a compatible `onnxruntime`) is also included, so [local CPU embeddings](/setup-configuration/embedding-providers#fastembed-local) work in the image without a custom build. To install additional optional dependencies, pass the `COGNEE_EXTRAS` build argument — a space-separated list of extra names, added on top of the defaults. No `Dockerfile` edit is required:
```bash theme={null}
docker build --build-arg COGNEE_EXTRAS="docs langchain" -t cognee-custom .
```
Because `dlt` is among the defaults, the image ingests `.csv` files through the structured dlt route: the [loader engine](/core-concepts/further-concepts/loaders) registers `dlt_csv_loader` above the plain-text `csv_loader` whenever `dlt` is importable, so a CSV added through the container is staged through dlt and its rows skip chunking and LLM entity extraction, as described under [CSV Files](/integrations/dlt-integration#csv-files), with no custom build. Flattening a CSV into plain text instead is possible only from the Python SDK, by requesting `csv_loader` explicitly for that call with `preferred_loaders=[{"csv_loader": {}}]`; `POST /api/v1/add` accepts no loader override, so every CSV uploaded to the image over HTTP takes the dlt route. The trade-off is footprint: the extra pulls `dlt[sqlalchemy]` and pandas into the image.
`COGNEE_EXTRAS` defaults to an empty string, so builds that don't pass it behave exactly as before. The `Dockerfile` applies it to **both** `uv sync` steps — the second sync is exact and would otherwise remove extras installed only in the dependency-cache layer — so the packages end up in the final runtime stage.
<Note>
Both `uv sync` invocations keep `--frozen`. `COGNEE_EXTRAS` selects from extras that are already resolved in `uv.lock`; it does not resolve new dependencies. Builds stay deterministic and no lockfile change is needed.
</Note>
The build argument is declared in the root `Dockerfile`, which builds the API image. The MCP and frontend images use their own Dockerfiles and do not accept it.
**Passing extras through Docker Compose.** The `cognee` service's `build:` block in `docker-compose.yml` declares only `context` and `dockerfile`, so `docker compose up --build cognee` does *not* forward `COGNEE_EXTRAS`. Add an `args:` entry first:
```yaml theme={null}
services:
cognee:
build:
context: .
dockerfile: Dockerfile
args:
COGNEE_EXTRAS: "docs scraping"
```
Then rebuild:
```bash theme={null}
docker compose up --build cognee
```
For a table of available extras and common combinations, see [Installation](/getting-started/installation#extras-and-common-installation-combinations).
For a table of supported file types and their loaders, see [Loaders](/core-concepts/further-concepts/loaders#supported-file-extensions).
For example, the `docs` extra adds [UnstructuredLoader](/core-concepts/further-concepts/loaders#external-loaders), office documents (`.docx`, `.pptx`, `.xlsx`, `.epub`, and similar formats), and `AdvancedPdfLoader`. Other commonly added extras include `scraping`, `redis`, `tracing`, and `docling`.
**System packages still require a `Dockerfile` edit.** `COGNEE_EXTRAS` only installs Python packages. For layout-aware or OCR-based PDF extraction with `AdvancedPdfLoader`, you also need `poppler-utils` and `tesseract-ocr` in the **runtime stage** of your `Dockerfile` (the second `FROM python:3.12-slim-bookworm` block):
```dockerfile theme={null}
RUN apt-get update && apt-get install -y \
libpq5 \
curl \
poppler-utils \
tesseract-ocr \
&& rm -rf /var/lib/apt/lists/*
```
Rebuild after updating the `Dockerfile`:
```bash theme={null}
docker compose up --build cognee
```
</Accordion>
<Accordion title="Bytecode Precompilation">
The repository `Dockerfile` sets `ENV UV_COMPILE_BYTECODE=1`, so `uv sync` compiles the virtual environment to `.pyc` bytecode at **build time** instead of leaving the interpreter to recompile each module from source on first import.
The effect is faster container cold starts: without it the shipped venv contains no `.pyc` files, so every cold start recompiles the dependency tree from source. On the `cognee-saas-pod` image this accounted for roughly 8s of a \~13s import — about half the startup time.
Trade-offs: builds take slightly longer and the image is marginally larger because the `.pyc` files are written into the venv layer.
To disable it (for example to debug or reproduce from-source import behavior), comment out or remove the line in the `Dockerfile` before building:
```dockerfile theme={null}
# ENV UV_COMPILE_BYTECODE=1
```
then rebuild:
```bash theme={null}
docker compose up --build cognee
```
</Accordion>
</AccordionGroup>
<Card title="Need help?" href="https://discord.gg/m63hxKsp4p" icon="discord">
Join our community for Docker deployment support.
</Card>
# Docker Compose Reference
Source: https://docs.cognee.ai/how-to-guides/cognee-sdk/deployment/docker-compose-reference
Annotated walkthrough of every service, port, and profile in the shipped docker-compose.yml
The `docker-compose.yml` at the root of the [cognee repository](https://github.com/topoteretes/cognee) defines eight services. Only two of them start by default — everything else is gated behind a [Compose profile](https://docs.docker.com/compose/how-tos/profiles/). This page explains what each service is for, so you can decide which ones you actually need.
For setup instructions and common stacks, see [Docker Deployment](/how-to-guides/cognee-sdk/deployment/docker).
## Service overview
| Service | Profile | Host ports | Required? | Purpose |
| -------------- | ---------- | -------------- | --------- | ------------------------------------------------------------------------------------------ |
| `cognee` | *(none)* | `8000`, `5678` | **Yes** | The Cognee REST API server |
| `redisinsight` | *(none)* | `5540` | No | Redis browser GUI (developer convenience) |
| `cognee-mcp` | `mcp` | `8001`, `5679` | No | [MCP server](/cognee-mcp/mcp-overview) for IDE integrations |
| `frontend` | `ui` | `3000` | No | Experimental [web UI](/cognee-cloud/local-ui), from the published `cognee/cognee-ui` image |
| `frontend-dev` | `ui-dev` | `3000` | No | The same UI built from source with hot reload |
| `postgres` | `postgres` | `5432` | No | PostgreSQL + pgvector |
| `neo4j` | `neo4j` | `7474`, `7687` | No | Neo4j graph database |
| `redis` | `redis` | `6379` | No | Redis, for the session cache |
<Note>
A service with no `profiles:` key starts whenever you run a bare `docker compose up`. That means `redisinsight` comes up alongside `cognee` even if you never configure Redis. Run `docker compose up cognee` to start only the API server.
</Note>
## Services in detail
<AccordionGroup>
<Accordion title="cognee — the API server">
Built from the repository `Dockerfile` and started by `docker compose up --build cognee`. This is the only service you need for a working deployment; the default [relational](/setup-configuration/relational-databases), [vector](/setup-configuration/vector-stores), and [graph](/setup-configuration/graph-stores) stores are all embedded and run inside this container.
* `8000` — the FastAPI app (`http://localhost:8000/docs` for interactive docs).
* `5678` — `debugpy`, used only when `DEBUG=true` and `ENV` is `dev` or `local`.
* Mounts `./cognee` from the host for **dev reload only**, and `.env` read-only (`:ro`), so config changes need only a restart, not a rebuild.
* Persists memory in the `cognee_system` and `cognee_data` named volumes, mounted at `/cognee-storage/system` and `/cognee-storage/data` — the image's baked-in `SYSTEM_ROOT_DIRECTORY` / `DATA_ROOT_DIRECTORY`. These are the same volumes `cognee-mcp` mounts, so both services read and write one memory store.
* Runs as the non-root user `cognee` (uid/gid 1000), matching `cognee-mcp` so the shared volumes have no ownership conflicts.
* Passes `DB_PROVIDER`, `DB_HOST`, `DB_PORT`, `DB_NAME`, `DB_USERNAME`, and `DB_PASSWORD` through from your environment or `.env`, defaulting to embedded SQLite; the `DB_HOST`/`DB_PORT` defaults (`host.docker.internal:5432`) only apply once you switch `DB_PROVIDER` to `postgres`. For `--profile postgres`, set `DB_PROVIDER=postgres` and `DB_HOST=postgres`.
* Has a `healthcheck` that polls `/health` every 30s after a 40s grace period.
</Accordion>
<Accordion title="cognee-mcp — MCP server for IDEs">
Enabled with `--profile mcp`. Exposes Cognee's tools over the Model Context Protocol so Cursor, Claude Desktop, VS Code, and similar clients can call them.
* Runs with `TRANSPORT_MODE=sse`, so clients connect to `http://localhost:8001/sse`.
* Host ports (`8001`, `5679`) differ from the `cognee` service, but inside the container it still listens on `8000`/`5678` — that's why both can run at once.
* Started with `--no-migration`, so it does not run Alembic migrations. Let the `cognee` service own the schema, and point both at the same database via the `DB_*` variables.
* Mounts the **same** `cognee_system` / `cognee_data` volumes as the `cognee` service, at `/cognee-storage/system` and `/cognee-storage/data`. Sharing one memory store between the API and the MCP server is the point of running both — the image runs as the same non-root user (uid/gid 1000) as `cognee`, so neither container can create files the other cannot write.
* Has a `healthcheck` that polls `/health` on the container's port `8000` every 30s after a 40s grace period. It probes with `python -c "import urllib.request..."` rather than `curl`, which the python-slim MCP runtime image does not ship.
* Optional. Skip it if you only use the REST API or the Python SDK.
</Accordion>
<Accordion title="frontend — experimental web UI">
Enabled with `--profile ui`. A Next.js app run from the published `cognee/cognee-ui` image — the same image CI builds, so this profile works from a bare `docker-compose.yml` with no checkout of `cognee-frontend`.
* Image tag comes from `COGNEE_UI_TAG`, defaulting to `latest`.
* Waits for the `cognee` service to report healthy (`depends_on: condition: service_healthy`) before starting.
* Passes `COGNEE_BACKEND_URL` through from your environment, defaulting to empty. It is read at **run time**, not baked in at build time, which is what lets one published image point at any backend. Empty means the browser derives the backend host from the address the UI was loaded from, on port `8000` — right for the default localhost setup. Because the browser calls the backend directly, a value you do set must be the address as seen from the browser, never the `cognee` service name.
* The image's entrypoint rejects a `COGNEE_BACKEND_URL` that is not an absolute `http(s)` URL and exits rather than starting; see [Cognee + Web UI](/how-to-guides/cognee-sdk/deployment/docker#additional-information).
* Has a baked-in `healthcheck` that fetches `/api/runtime-config` every 30s after a 20s grace period — the cheapest endpoint that proves both that the server is up and that the backend URL resolved.
Optional and explicitly a work in progress — it supports the minimum feature set needed to be functional. For a richer experience, use the `mcp` profile with your IDE or [Cognee Cloud](/cognee-cloud/overview).
</Accordion>
<Accordion title="frontend-dev — the web UI built from source">
Enabled with `--profile ui-dev`. Builds the `dev` target of `cognee-frontend/Dockerfile` — a hot-reloading `next dev` server — and bind-mounts `./cognee-frontend/src` and `./cognee-frontend/public`, so source edits are picked up without a rebuild. Never published; it exists for frontend work.
Takes the same `COGNEE_BACKEND_URL` as `frontend` and publishes the same host port `3000`, so the two profiles are alternatives, not additions. Unlike `frontend`, it declares no `depends_on`, so it starts without waiting for the API.
</Accordion>
<Accordion title="postgres — relational store and pgvector">
Enabled with `--profile postgres`. Uses the `pgvector/pgvector:pg17` image, so the same container can serve as the [relational database](/setup-configuration/relational-databases) (`DB_PROVIDER=postgres`) and the [vector store](/setup-configuration/vector-stores) (`VECTOR_DB_PROVIDER=pgvector`).
Ships with `cognee` / `cognee` / `cognee_db` as user, password, and database name. Optional — Cognee defaults to SQLite and LanceDB, which need no service at all.
* Mounts the `postgres_data` named volume at `/var/lib/postgresql/data`, so the database survives `docker compose up --force-recreate postgres` and `docker compose down`. Deleting the volume (`docker compose down -v`) still discards it. See [Data Persistence and Host Files](/how-to-guides/cognee-sdk/deployment/docker#additional-information).
* Has a `healthcheck` that runs `pg_isready -U cognee -d cognee_db` every 10s, 5 retries, after a 10s grace period.
<Note>
Nothing in the shipped file declares `depends_on` on `postgres` — only `frontend` uses one, to wait on `cognee` — so this healthcheck only reports state; it does not hold `cognee` back until Postgres accepts connections. Add a `depends_on: condition: service_healthy` guard yourself if startup ordering matters; see [PostgreSQL Connection Refused](/how-to-guides/cognee-sdk/deployment/docker#postgresql-connection-refused).
</Note>
</Accordion>
<Accordion title="neo4j — dedicated graph database">
Enabled with `--profile neo4j`. Pinned to `neo4j:5.26` for compatibility with the Neo4j Python driver, with the APOC and Graph Data Science plugins preloaded.
* `7474` — Neo4j Browser.
* `7687` — Bolt protocol; set `GRAPH_DATABASE_URL=bolt://neo4j:7687`.
* Default credentials are `neo4j` / `pleaseletmein`.
Optional. Use it instead of the embedded file-based graph store when the graph should live in an external database rather than in files owned by the Cognee container. No data volume is mounted, so add one for durability.
</Accordion>
<Accordion title="redis — session cache backend">
Enabled with `--profile redis`. Runs `redis:7-alpine` with append-only persistence into the `redis_data` volume.
Only needed when you set `CACHE_BACKEND=redis` for [sessions and caching](/core-concepts/sessions-and-caching); the default cache backend is SQLite. Point Cognee at it with `CACHE_HOST=redis` and `CACHE_PORT=6379`.
<Note>
This service backs the *session cache*. Using Redis as a [vector store](/setup-configuration/community-maintained/redis) is a separate, community-maintained adapter that needs the Redis Search module.
</Note>
</Accordion>
<Accordion title="redisinsight — Redis GUI">
[RedisInsight](https://redis.io/insight/) is Redis' official browser-based inspector. It is a **developer convenience only** — Cognee never talks to it, and nothing breaks if you remove it.
Open `http://localhost:5540` and connect to host `redis`, port `6379` to inspect cached sessions. Because it has no profile, it starts with a bare `docker compose up`; name the services you want (`docker compose up cognee`) or delete the block if you don't need it.
</Accordion>
</AccordionGroup>
## Shared building blocks
<AccordionGroup>
<Accordion title="cognee-network">
All services join the user-defined bridge network `cognee-network`. Inside it, containers reach each other by **service name** — that is why `DB_HOST=postgres` and `GRAPH_DATABASE_URL=bolt://neo4j:7687` work, while `localhost` would resolve to the container itself.
</Accordion>
<Accordion title="extra_hosts and host.docker.internal">
The `cognee` and `cognee-mcp` services map `host.docker.internal` to `host-gateway`:
```yaml theme={null}
extra_hosts:
- "host.docker.internal:host-gateway"
```
This lets a container reach services running directly on your machine — an Ollama server, or a Postgres instance you started outside Compose. Use `host.docker.internal` in place of `localhost` for those, for example `LLM_ENDPOINT=http://host.docker.internal:11434`.
</Accordion>
<Accordion title="Resource limits">
```yaml theme={null}
deploy:
resources:
limits:
cpus: "4.0"
memory: 8GB
```
Docker Compose v2 applies these as hard CPU and memory caps: `4.0` CPUs / 8 GB for `cognee`, `2.0` CPUs / 4 GB for `cognee-mcp`. Lower them if your machine has less headroom; raise them if large ingestion runs are being OOM-killed. On Docker Desktop the limits cannot exceed what the VM itself is allocated.
See **Memory sizing** below for what that 8 GB has to cover and which knobs move it.
</Accordion>
<Accordion title="Memory sizing">
The `cognee` container is not one process. With the default embedded stores, the graph engine (Ladybug/Kuzu) and the vector engine (LanceDB) each run in a **worker subprocess** — `GRAPH_DATABASE_SUBPROCESS_ENABLED` and `VECTOR_DB_SUBPROCESS_ENABLED` both default to `true` — so the figure `docker stats` reports is the sum for the whole container, and the 8 GB cap is shared across all of them.
**Cap the graph engine's buffer pool.** The largest single lever is the embedded graph database, which Cognee opens with a buffer-pool ceiling of **32 GB** (`KUZU_BUFFER_POOL_SIZE`) and an on-disk ceiling of 32 GB (`KUZU_MAX_DB_SIZE`). The buffer pool is a ceiling the engine grows into under load, not memory reserved at startup — but nothing bounds that growth to the container. Left alone the engine sizes the pool at roughly 80% of *system* memory; Cognee replaces that with a fixed 32 GB. Neither figure is derived from the container's limit, so a container capped at 8 GB still opens a graph engine that believes it may grow to 32 GB. On a memory-capped container, set the ceiling yourself — the value is a byte count:
```bash theme={null}
# Leaves room under an 8 GB container cap for the API process and the vector engine
KUZU_BUFFER_POOL_SIZE=2147483648 # 2 GB
```
This applies to `GRAPH_DATABASE_PROVIDER=kuzu` and to the default `ladybug` — both use the same embedded engine. It does not apply when the graph lives in [Neo4j](/setup-configuration/graph-stores) or another external service, where that memory is the database container's problem instead.
**Ingestion is the peak; recall is the steady state.** Memory is driven by how much Cognee has in flight, not by the size of the stored graph: `data_per_batch` (default `20`) bounds how many data items move through the pipeline at once, and `chunks_per_batch` (default `2000` in the standard Cognify pipeline) bounds the chunk-level batches handed to graph extraction and persistence. Lowering them is the first thing to try on a small container — see [Batching for faster processing](/core-concepts/main-operations/legacy-operations/cognify#batching-for-faster-processing) for the full tuning table.
**When the container is killed.** A container that exceeds its cap is killed by the kernel, not by Cognee, so the symptom is an abrupt exit rather than a Python traceback. The shipped compose sets `restart: always` on `cognee`, so the container is usually back up before you can inspect it — `.State.OOMKilled` and `.State.ExitCode` then describe the fresh, healthy run, not the killed one. A climbing restart count is the durable signal:
```bash theme={null}
docker inspect --format '{{.RestartCount}} {{.State.OOMKilled}} {{.State.ExitCode}}' cognee
# 4 false 0 <- restarted four times; the flags belong to the current run
```
To catch the kill itself, watch for the event while you reproduce the load:
```bash theme={null}
docker events --filter container=cognee --filter event=oom
```
Either give the container more headroom by raising the `memory` limit above, or reduce demand with `KUZU_BUFFER_POOL_SIZE` and the batching parameters. If the reported usage is far above the configured limit, the cap is not being enforced at all — `deploy.resources.limits` only applies under Compose v2.
</Accordion>
<Accordion title="Combining profiles">
Profiles are additive — repeat the flag to start several optional services together:
```bash theme={null}
docker compose --profile postgres --profile neo4j up --build
```
You can also set them once for the shell session:
```bash theme={null}
export COMPOSE_PROFILES=postgres,neo4j
docker compose up --build
```
</Accordion>
</AccordionGroup>
<Card title="Need help?" href="https://discord.gg/m63hxKsp4p" icon="discord">
Join our community for Docker deployment support.
</Card>
# EC2 Deployment
Source: https://docs.cognee.ai/how-to-guides/cognee-sdk/deployment/ec2
Deploy Cognee on Amazon EC2 for traditional cloud server deployments with custom configurations
# EC2 Deployment
Deploy Cognee on Amazon EC2 for traditional cloud server deployments with full control over the infrastructure and custom configurations.
<Info>
EC2 deployment is ideal for organizations that need direct server access, custom networking, or integration with existing AWS infrastructure.
</Info>
## Why EC2?
<CardGroup>
<Card title="Full Control" icon="settings">
Complete control over server configuration, networking, and security
</Card>
<Card title="AWS Integration" icon="aws">
Native integration with AWS services like RDS, S3, and VPC
</Card>
<Card title="Cost Predictable" icon="dollar-sign">
Fixed costs with reserved instances and predictable billing
</Card>
<Card title="Custom Networking" icon="network">
Advanced networking configurations and security groups
</Card>
</CardGroup>
## Prerequisites
<Steps>
<Step title="AWS Account">
* Active AWS account with EC2 permissions
* AWS CLI installed and configured
* Key pair created for SSH access
</Step>
<Step title="Network Setup">
* VPC with public/private subnets
* Security groups configured for HTTP/HTTPS traffic
* Internet Gateway for public access
</Step>
<Step title="Domain & SSL">
* Domain name (optional but recommended)
* SSL certificate (Let's Encrypt or AWS Certificate Manager)
</Step>
</Steps>
## Instance Configuration
<Tabs>
<Tab title="Development">
**Small Scale Setup**
* **Instance Type**: `t3.medium` (2 vCPU, 4GB RAM)
* **Storage**: 20GB GP3 SSD
* **OS**: Ubuntu 22.04 LTS
* **Databases**: Local SQLite, embedded vector DB
</Tab>
<Tab title="Production">
**Production Ready**
* **Instance Type**: `m5.xlarge` (4 vCPU, 16GB RAM)
* **Storage**: 100GB GP3 SSD + EBS volumes for data
* **OS**: Ubuntu 22.04 LTS
* **Databases**: External RDS, managed vector DB
</Tab>
<Tab title="High Performance">
**Large Scale Processing**
* **Instance Type**: `c5.4xlarge` (16 vCPU, 32GB RAM)
* **Storage**: 500GB GP3 SSD + dedicated EBS volumes
* **OS**: Ubuntu 22.04 LTS
* **Databases**: Multi-AZ RDS, clustered databases
</Tab>
</Tabs>
## Quick Deployment
<Steps>
<Step title="Launch EC2 Instance">
```bash theme={null}
# Using AWS CLI
aws ec2 run-instances \
--image-id ami-0c02fb55956c7d316 \
--instance-type t3.medium \
--key-name your-key-pair \
--security-group-ids sg-12345678 \
--subnet-id subnet-12345678 \
--block-device-mappings '[{
"DeviceName": "/dev/sda1",
"Ebs": {
"VolumeSize": 20,
"VolumeType": "gp3"
}
}]' \
--tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=cognee-server}]'
```
<Note>
Replace the AMI ID, security group, and subnet with your specific values.
</Note>
</Step>
<Step title="Connect to Instance">
```bash theme={null}
# Get instance public IP
aws ec2 describe-instances --filters "Name=tag:Name,Values=cognee-server" \
--query 'Reservations[*].Instances[*].PublicIpAddress'
# SSH into the instance
ssh -i /path/to/your-key.pem ubuntu@YOUR-INSTANCE-IP
```
</Step>
<Step title="Install Dependencies">
```bash theme={null}
# Update system
sudo apt update && sudo apt upgrade -y
# Install Python and pip
sudo apt install python3 python3-pip python3-venv git curl -y
# Install uv for faster Python package management
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.cargo/env
```
</Step>
<Step title="Deploy Cognee">
```bash theme={null}
# Clone repository
git clone https://github.com/topoteretes/cognee.git
cd cognee
# Run automated setup script
chmod +x deployment/setup_ubuntu_instance.sh
source deployment/setup_ubuntu_instance.sh
```
</Step>
</Steps>
## Manual Setup Process
<AccordionGroup>
<Accordion title="Environment Setup">
```bash theme={null}
# Create virtual environment
python3 -m venv cognee-env
source cognee-env/bin/activate
# Install Cognee with all dependencies
uv sync --dev --all-extras --reinstall
# Set up environment variables
cat > .env << EOF
OPENAI_API_KEY=your-openai-api-key
POSTGRES_URL=postgresql://user:pass@localhost:5432/cognee
NEO4J_URL=bolt://neo4j:password@localhost:7687
QDRANT_URL=http://localhost:6333
COGNEE_HOST=0.0.0.0
COGNEE_PORT=8000
EOF
```
</Accordion>
<Accordion title="Database Installation">
```bash theme={null}
# Install PostgreSQL
sudo apt install postgresql postgresql-contrib -y
sudo -u postgres createuser --interactive cognee
sudo -u postgres createdb cognee
# Install Neo4j
wget -O - https://debian.neo4j.com/neotechnology.gpg.key | sudo apt-key add -
echo 'deb https://debian.neo4j.com stable 4.4' | sudo tee /etc/apt/sources.list.d/neo4j.list
sudo apt update && sudo apt install neo4j -y
# Install and configure Qdrant
docker run -d --name qdrant -p 6333:6333 qdrant/qdrant
```
</Accordion>
<Accordion title="Service Configuration">
```bash theme={null}
# Create systemd service
sudo tee /etc/systemd/system/cognee.service << EOF
[Unit]
Description=Cognee Knowledge Graph Service
After=network.target postgresql.service neo4j.service
[Service]
Type=simple
User=ubuntu
WorkingDirectory=/home/ubuntu/cognee
Environment=PATH=/home/ubuntu/cognee/cognee-env/bin
ExecStart=/home/ubuntu/cognee/cognee-env/bin/python -m cognee.api.server
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
# Enable and start service
sudo systemctl daemon-reload
sudo systemctl enable cognee
sudo systemctl start cognee
```
</Accordion>
</AccordionGroup>
## AWS Service Integration
<CardGroup>
<Card title="RDS Integration" icon="database">
**Managed PostgreSQL**
```bash theme={null}
# Connect to RDS instance
POSTGRES_URL=postgresql://user:pass@your-rds-endpoint:5432/cognee
```
</Card>
<Card title="S3 Storage" icon="hard-drive">
**Object Storage**
```bash theme={null}
# Configure S3 for file storage
AWS_S3_BUCKET=your-cognee-bucket
AWS_ACCESS_KEY_ID=your-access-key
AWS_SECRET_ACCESS_KEY=your-secret-key
```
</Card>
</CardGroup>
## Security Configuration
<Steps>
<Step title="Security Groups">
```bash theme={null}
# Create security group
aws ec2 create-security-group \
--group-name cognee-sg \
--description "Security group for Cognee server"
# Allow SSH (port 22)
aws ec2 authorize-security-group-ingress \
--group-id sg-12345678 \
--protocol tcp \
--port 22 \
--cidr 0.0.0.0/0
# Allow HTTP/HTTPS (ports 80/443)
aws ec2 authorize-security-group-ingress \
--group-id sg-12345678 \
--protocol tcp \
--port 80 \
--cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress \
--group-id sg-12345678 \
--protocol tcp \
--port 443 \
--cidr 0.0.0.0/0
```
</Step>
<Step title="SSL/TLS Setup">
```bash theme={null}
# Install Nginx
sudo apt install nginx certbot python3-certbot-nginx -y
# Configure Nginx reverse proxy
sudo tee /etc/nginx/sites-available/cognee << EOF
server {
listen 80;
server_name your-domain.com;
location / {
proxy_pass http://localhost:8000;
proxy_set_header Host \$host;
proxy_set_header X-Real-IP \$remote_addr;
}
}
EOF
# Enable site and get SSL certificate
sudo ln -s /etc/nginx/sites-available/cognee /etc/nginx/sites-enabled/
sudo certbot --nginx -d your-domain.com
```
</Step>
<Step title="Firewall Configuration">
```bash theme={null}
# Configure UFW firewall
sudo ufw allow OpenSSH
sudo ufw allow 'Nginx Full'
sudo ufw --force enable
```
</Step>
</Steps>
## Monitoring & Maintenance
<Tabs>
<Tab title="System Monitoring">
```bash theme={null}
# Install monitoring tools
sudo apt install htop iotop nethogs -y
# Check system resources
htop
df -h
free -m
# Monitor Cognee service
sudo systemctl status cognee
sudo journalctl -u cognee -f
```
</Tab>
<Tab title="Log Management">
Cognee already rotates its own logs: the active file is rolled over at `COGNEE_LOG_MAX_BYTES` (default 50 MB) into numbered `*.log.1` … backups, and the 10 most recent startups' `.log` files are kept. Do **not** point `logrotate` at `*.log` — renaming the active file out from under the running handler means Cognee keeps writing to the renamed inode. What the built-in cleanup never removes is the numbered backups, so that is what an external policy should compress and expire:
```bash theme={null}
# Compress and expire the rotation backups Cognee leaves behind
sudo tee /etc/logrotate.d/cognee << EOF
/home/ubuntu/cognee/logs/*.log.* {
daily
rotate 30
compress
missingok
notifempty
nocreate
}
EOF
```
See [Size-Based Rotation](/setup-configuration/logging#size-based-rotation) for the built-in behavior and its variables.
</Tab>
<Tab title="Backup Strategy">
```bash theme={null}
# Create backup script
cat > backup.sh << EOF
#!/bin/bash
# Database backup
pg_dump cognee > /backup/cognee-$(date +%Y%m%d).sql
# Upload to S3
aws s3 cp /backup/cognee-$(date +%Y%m%d).sql s3://your-backup-bucket/
# Clean old backups
find /backup -name "cognee-*.sql" -mtime +7 -delete
EOF
# Schedule with cron
echo "0 2 * * * /home/ubuntu/backup.sh" | crontab -
```
</Tab>
</Tabs>
## Scaling & Performance
<AccordionGroup>
<Accordion title="Vertical Scaling">
```bash theme={null}
# Stop instance
aws ec2 stop-instances --instance-ids i-1234567890abcdef0
# Change instance type
aws ec2 modify-instance-attribute \
--instance-id i-1234567890abcdef0 \
--instance-type Value=m5.xlarge
# Start instance
aws ec2 start-instances --instance-ids i-1234567890abcdef0
```
</Accordion>
<Accordion title="Load Balancing">
```bash theme={null}
# Create Application Load Balancer
aws elbv2 create-load-balancer \
--name cognee-alb \
--subnets subnet-12345678 subnet-87654321 \
--security-groups sg-12345678
# Create target group
aws elbv2 create-target-group \
--name cognee-targets \
--protocol HTTP \
--port 8000 \
--vpc-id vpc-12345678
```
</Accordion>
<Accordion title="Auto Scaling">
```bash theme={null}
# Create launch template
aws ec2 create-launch-template \
--launch-template-name cognee-template \
--launch-template-data '{
"ImageId": "ami-0c02fb55956c7d316",
"InstanceType": "t3.medium",
"KeyName": "your-key-pair",
"SecurityGroupIds": ["sg-12345678"],
"UserData": "base64-encoded-startup-script"
}'
```
</Accordion>
</AccordionGroup>
## Troubleshooting
<Tabs>
<Tab title="Common Issues">
**Service Won't Start**
```bash theme={null}
# Check service status
sudo systemctl status cognee
sudo journalctl -u cognee --no-pager
# Check port availability
sudo netstat -tlnp | grep :8000
# Verify environment variables
cat .env
```
</Tab>
<Tab title="Database Issues">
**Connection Problems**
```bash theme={null}
# Test PostgreSQL connection
psql -h localhost -U cognee -d cognee -c "SELECT 1;"
# Check Neo4j status
sudo systemctl status neo4j
curl http://localhost:7474
# Verify Qdrant
curl http://localhost:6333/collections
```
</Tab>
<Tab title="Performance Issues">
**Resource Monitoring**
```bash theme={null}
# Check CPU and memory usage
top
free -m
# Monitor disk I/O
iotop
# Check network connections
ss -tuln
```
</Tab>
</Tabs>
## Cost Optimization
<CardGroup>
<Card title="Reserved Instances" icon="dollar-sign">
**Save up to 75%**
Purchase reserved instances for predictable workloads to reduce costs significantly.
</Card>
<Card title="Spot Instances" icon="trending-down">
**Development/Testing**
Use spot instances for non-critical workloads to save up to 90% on compute costs.
</Card>
</CardGroup>
<Tip>
Use AWS Cost Explorer to monitor your EC2 spending and optimize instance types based on actual usage patterns.
</Tip>
## Next Steps
<CardGroup>
<Card title="High Availability" icon="shield">
**Multi-AZ Setup**
Deploy across multiple availability zones for improved resilience.
</Card>
<Card title="Monitoring Stack" icon="bar-chart">
**CloudWatch Integration**
Set up comprehensive monitoring with CloudWatch and custom metrics.
</Card>
</CardGroup>
<Card title="Need Help?" href="https://discord.gg/m63hxKsp4p" icon="discord">
Join our community for EC2 deployment support and AWS best practices.
</Card>
# Kubernetes (Helm)
Source: https://docs.cognee.ai/how-to-guides/cognee-sdk/deployment/helm
Deploy Cognee on Kubernetes with the in-repo Helm chart for enterprise-grade, production-ready deployments
# Kubernetes Deployment with Helm
Deploy Cognee on Kubernetes using the chart in `deployment/helm` for enterprise-grade deployments with full control over configuration and resources.
Kubernetes deployment provides container orchestration, auto-healing, and declarative configuration for production workloads.
The chart deploys the Cognee API together with a bundled **PostgreSQL + pgvector** database. Relational metadata and vectors both live in that Postgres instance — the chart does not bundle Neo4j or Qdrant.
<Warning>
Cognee runs as a **single-replica** deployment. The API pod is a single process with process-local locks and caches, so running multiple replicas against the same stores is not supported — do not enable horizontal autoscaling or set `replicaCount` above 1. The chart prints a warning after install when `replicaCount > 1`. Scale vertically (more CPU/memory per pod) instead.
</Warning>
## Why Kubernetes + Helm?
<CardGroup>
<Card title="Enterprise Ready" icon="shield-check">
Resource requests/limits, a restricted security context, and a dedicated ServiceAccount ship as defaults
</Card>
<Card title="Auto-Healing" icon="server">
Startup and readiness probes keep traffic away from a pod until its dependencies are reachable
</Card>
<Card title="Validated Values" icon="settings">
`values.schema.json` validates your values before anything is applied to the cluster
</Card>
<Card title="GitOps Integration" icon="git-branch">
Version-controlled infrastructure with automated deployment pipelines
</Card>
</CardGroup>
## Prerequisites
<Steps>
<Step title="Kubernetes Cluster">
You need a running Kubernetes cluster:
* **Local**: Minikube, Kind, or Docker Desktop
* **Cloud**: GKE, EKS, AKS, or DigitalOcean Kubernetes
* **On-premise**: Self-managed Kubernetes cluster
<Note>
The chart targets Kubernetes 1.25+ and Helm 3.10+.
</Note>
</Step>
<Step title="Install Tools">
```bash theme={null}
# Install kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install kubectl /usr/local/bin/
# Install Helm
curl https://get.helm.sh/helm-v3.12.0-linux-amd64.tar.gz | tar xz
sudo mv linux-amd64/helm /usr/local/bin/
# Verify installations
kubectl version --client
helm version
```
</Step>
<Step title="Configure Access">
```bash theme={null}
# Test cluster connectivity
kubectl cluster-info
kubectl get nodes
```
</Step>
</Steps>
## Quick Deployment
<Steps>
<Step title="Clone Repository">
```bash theme={null}
git clone https://github.com/topoteretes/cognee.git
cd cognee
```
</Step>
<Step title="Create the credentials Secret">
Credentials are kept out of `values.yaml`. Create a Secret holding `LLM_API_KEY` and `DB_PASSWORD`, and point the chart at it with `existingSecret`:
```bash theme={null}
kubectl create namespace cognee
kubectl create secret generic cognee-credentials \
--namespace cognee \
--from-literal=LLM_API_KEY="sk-..." \
--from-literal=DB_PASSWORD="strongpassword"
```
<Note>
If you skip this step, the chart renders its own Secret from `postgres.auth.password` with an **empty** `LLM_API_KEY`. That is a development convenience only — the post-install notes print the `kubectl patch` command to fill the key in.
</Note>
</Step>
<Step title="Configure Values">
Create a `values.yaml` file to customize your deployment. Only keys that exist in the chart are accepted:
```yaml theme={null}
# values.yaml
replicaCount: 1 # Cognee is single-process; keep a single replica
image:
repository: cognee/cognee
tag: main
pullPolicy: IfNotPresent
service:
type: ClusterIP
port: 8000
cognee:
env: local
llmProvider: openai
llmModel: openai/gpt-4o-mini
vectorDbProvider: pgvector
enableBackendAccessControl: false
# Secret holding LLM_API_KEY and DB_PASSWORD
existingSecret: cognee-credentials
postgres:
auth:
username: cognee
database: cognee_db
password: "" # unused when existingSecret is set
storage: 2Gi
```
<Warning>
The chart ships a `values.schema.json` with `additionalProperties: false`, so **unknown keys fail the install** before anything reaches the cluster. Keys such as `cognee.image`, `cognee.replicas`, `cognee.env.OPENAI_API_KEY`, `postgresql`, `neo4j`, `qdrant`, `monitoring`, and `networkPolicy` are not part of this chart and will be rejected with a `values don't meet the specifications of the schema` error. The schema also constrains `image.pullPolicy` to `Always`/`Never`/`IfNotPresent`, `service.type` to `ClusterIP`/`NodePort`/`LoadBalancer`, and ports to 1–65535.
</Warning>
</Step>
<Step title="Deploy with Helm">
```bash theme={null}
# Install the Helm chart
helm upgrade --install cognee ./deployment/helm \
--namespace cognee --create-namespace \
-f values.yaml
# Check deployment status
kubectl get pods -n cognee -l app.kubernetes.io/instance=cognee
kubectl get services -n cognee
```
</Step>
<Step title="Verify Deployment">
```bash theme={null}
# Check pod status
kubectl get pods -n cognee
# View logs
kubectl logs -n cognee -l app.kubernetes.io/name=cognee-chart -f
# Test connectivity
kubectl port-forward svc/cognee-cognee-chart -n cognee 8000:8000
curl http://localhost:8000/health
```
<Note>
Resource names are `<release>-<chart>` unless you set `nameOverride` / `fullnameOverride`, so a release named `cognee` produces `cognee-cognee-chart` and `cognee-cognee-chart-postgres`.
</Note>
</Step>
</Steps>
## Values Reference
| Key | Default | Description |
| --------------------------------------------- | ------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| `replicaCount` | `1` | Number of Cognee replicas. Keep at 1 — see the single-replica warning above. |
| `nameOverride` / `fullnameOverride` | `""` | Override generated resource names |
| `image.repository` | `cognee/cognee` | Cognee image repository |
| `image.tag` | `main` | Image tag |
| `image.pullPolicy` | `IfNotPresent` | `Always`, `Never`, or `IfNotPresent` |
| `service.type` | `ClusterIP` | `ClusterIP`, `NodePort`, or `LoadBalancer` |
| `service.port` | `8000` | Service and container port |
| `cognee.env` | `local` | Runtime environment (`ENV`) |
| `cognee.llmProvider` | `openai` | LLM provider (`LLM_PROVIDER`) |
| `cognee.llmModel` | `openai/gpt-4o-mini` | LLM model (`LLM_MODEL`) |
| `cognee.vectorDbProvider` | `pgvector` | Vector store (`VECTOR_DB_PROVIDER`) |
| `cognee.enableBackendAccessControl` | `false` | Multi-tenant access control (`ENABLE_BACKEND_ACCESS_CONTROL`) |
| `existingSecret` | `""` | Name of an existing Secret with `LLM_API_KEY` and `DB_PASSWORD` |
| `resources.requests` | `500m` CPU / `512Mi` | Cognee container requests |
| `resources.limits` | `4000m` CPU / `2Gi` | Cognee container limits |
| `serviceAccount.create` | `true` | Create a dedicated ServiceAccount |
| `serviceAccount.name` | `""` | Override the ServiceAccount name |
| `serviceAccount.automountServiceAccountToken` | `false` | Mount the API token into the pod |
| `podSecurityContext` | `{}` | Pod-level security context |
| `securityContext` | `allowPrivilegeEscalation: false`, `capabilities.drop: [ALL]` | Container security context |
| `startupProbe.enabled` | `true` | HTTP `GET /health`, `failureThreshold: 30`, `periodSeconds: 10` |
| `readinessProbe.enabled` | `true` | HTTP `GET /health`, `initialDelaySeconds: 10`, `periodSeconds: 10` |
| `livenessProbe.enabled` | `false` | Intentionally disabled — see [Health & Probes](#health--probes) |
| `postgres.image.repository` / `.tag` | `pgvector/pgvector` / `pg17` | Bundled Postgres image |
| `postgres.port` | `5432` | Postgres port |
| `postgres.auth.username` / `.database` | `cognee` / `cognee_db` | Postgres user and database |
| `postgres.auth.password` | `""` | Dev-only password; ignored when `existingSecret` is set |
| `postgres.storage` | `2Gi` | PVC size for Postgres data |
| `postgres.resources.requests` | `250m` CPU / `256Mi` | Postgres requests |
| `postgres.resources.limits` | `1000m` CPU / `1Gi` | Postgres limits |
<Note>
If you change `service.port`, set `startupProbe.httpGet.port` and `readinessProbe.httpGet.port` to match — the probe ports are separate values and default to `8000`.
</Note>
## Architecture Components
<Tabs>
<Tab title="Application Tier">
**Cognee Services**
* **Cognee API**: single-replica Deployment running `image.repository:image.tag`
* **Service**: `ClusterIP` on `service.port` (8000) by default
* **ServiceAccount**: created by default with `automountServiceAccountToken: false`
</Tab>
<Tab title="Data Tier">
**Database Services**
* **PostgreSQL + pgvector** (`pgvector/pgvector:pg17`): a single-replica Deployment backed by a PersistentVolumeClaim (`postgres.storage`), serving both relational metadata and vectors
* **Probes**: `pg_isready` drives the Postgres startup, readiness, and liveness probes
* No graph-database provider is set in the ConfigMap, so Cognee falls back to its embedded default (Ladybug) inside the API container
</Tab>
<Tab title="Infrastructure">
**Supporting Services**
* **ConfigMap**: non-sensitive runtime configuration, including `DB_PROVIDER: postgres`, `DB_HOST`/`DB_PORT`/`DB_NAME`/`DB_USERNAME` for the bundled database, and `VECTOR_DB_PROVIDER`
* **Secret**: `LLM_API_KEY` and `DB_PASSWORD`, either pre-created (`existingSecret`) or rendered by the chart for development
* **Checksum annotations**: the Deployment carries `checksum/config` and `checksum/secret`, computed from the chart-rendered ConfigMap and Secret, so values changes that alter either one trigger a rolling restart. An `existingSecret` is not rendered by the chart, so rotating it requires a manual `kubectl rollout restart`
</Tab>
</Tabs>
## Database Configuration
The ConfigMap pins `DB_PROVIDER: postgres` explicitly, so the API always talks to the bundled Postgres service rather than silently falling back to SQLite. `DB_HOST` is derived from the chart's Postgres service name, and `DB_PORT`, `DB_NAME`, and `DB_USERNAME` follow `postgres.port` and `postgres.auth.*`. `DB_PASSWORD` comes only from the Secret.
`VECTOR_DB_PROVIDER` defaults to `pgvector`, which is why the bundled image is `pgvector/pgvector:pg17` — embeddings are stored in the same database.
<Note>
To run against an external database instead of the bundled one, override the environment through your own ConfigMap/Secret layer or a chart fork; the shipped chart always deploys and points at its own Postgres.
</Note>
## Production Configuration
<AccordionGroup>
<Accordion title="Credentials with existingSecret">
Create the Secret outside Helm (kubectl, External Secrets, Vault, …), then reference it:
```bash theme={null}
kubectl create secret generic cognee-credentials \
--namespace cognee \
--from-literal=LLM_API_KEY="sk-..." \
--from-literal=DB_PASSWORD="strongpassword"
helm upgrade --install cognee ./deployment/helm \
--namespace cognee --create-namespace \
--set existingSecret="cognee-credentials" \
--set postgres.auth.password=""
```
Both the Cognee Deployment and the Postgres Deployment read their credentials from this Secret, so the database password and the API's `DB_PASSWORD` cannot drift apart. Helm never renders this Secret, so the checksum annotations do not notice updates to it — after rotating it, restart the pods explicitly (see Rotating credentials below).
</Accordion>
<Accordion title="Resources">
Defaults are set for both containers; raise them for production workloads:
```yaml theme={null}
resources:
requests:
cpu: "1000m"
memory: "2Gi"
limits:
cpu: "4000m"
memory: "4Gi"
postgres:
storage: 100Gi
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "2000m"
memory: "4Gi"
```
</Accordion>
<Accordion title="Security Context & ServiceAccount">
The container security context defaults to `allowPrivilegeEscalation: false` with all Linux capabilities dropped, and the chart's ServiceAccount does not mount an API token. Add pod-level settings as needed:
```yaml theme={null}
podSecurityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 1000
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: false
capabilities:
drop:
- ALL
serviceAccount:
create: true
name: ""
automountServiceAccountToken: false
```
Set `serviceAccount.create: false` and `serviceAccount.name` to reuse an existing account (for example one bound to a cloud IAM role).
</Accordion>
<Accordion title="Probe tuning">
```yaml theme={null}
startupProbe:
enabled: true
httpGet:
path: /health
port: 8000
failureThreshold: 30 # up to 5 minutes at periodSeconds: 10
periodSeconds: 10
readinessProbe:
enabled: true
httpGet:
path: /health
port: 8000
initialDelaySeconds: 10
periodSeconds: 10
```
Increase `startupProbe.failureThreshold` if first boot (image pull plus database initialization) regularly takes longer than five minutes.
</Accordion>
</AccordionGroup>
## Health & Probes
The chart wires `GET /health` to a **startup probe** and a **readiness probe**. The startup probe allows up to 30 attempts, 10 seconds apart, before the pod is failed; the readiness probe then polls every 10 seconds after a 10 second delay and pulls the pod out of the Service whenever its dependencies are unhealthy.
<Warning>
`livenessProbe` is **disabled by default and should stay that way** with the current endpoint. `/health` verifies external dependencies — database, vector store, graph store, filesystem — so wiring it to liveness makes Kubernetes restart the pod during transient dependency outages, producing `CrashLoopBackOff` instead of letting the pod recover. Enable liveness only against a process-only endpoint that reports whether the process itself is alive.
</Warning>
Postgres uses `pg_isready` for its startup, readiness, and liveness probes, so the API's readiness naturally follows the database coming up.
## Upgrading from an Earlier Chart
<Steps>
<Step title="Move credentials into a Secret">
Inline API keys in `values.yaml` are no longer part of the values surface. Create a Secret with `LLM_API_KEY` and `DB_PASSWORD` and set `existingSecret`.
</Step>
<Step title="Rename your values keys">
The values structure is flat: `image.*`, `service.*`, `replicaCount`, `resources.requests/limits`, and `postgres.auth.*`. Older layouts such as `cognee.image`, `cognee.port`, `cognee.env.<VAR>`, and `cognee.resources.cpu` are rejected by the schema.
</Step>
<Step title="Review database behavior">
`DB_PROVIDER` is now pinned to `postgres` in the ConfigMap. Deployments that previously ran on the implicit SQLite fallback will start against Postgres and see an empty knowledge base.
</Step>
<Step title="Dry-run before applying">
```bash theme={null}
helm upgrade cognee ./deployment/helm -n cognee -f values.yaml --dry-run
```
Schema violations surface here rather than mid-rollout.
</Step>
</Steps>
## Scaling & Performance
Cognee scales **vertically**: give the single API pod more CPU and memory rather than adding replicas. Horizontal Pod Autoscaling is not supported — multiple Cognee pods writing to the same stores is not a supported configuration, and the chart warns after install when `replicaCount > 1`.
<Steps>
<Step title="Vertical Pod Autoscaler">
```yaml theme={null}
# VPA configuration
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
name: cognee-vpa
spec:
targetRef:
apiVersion: apps/v1
kind: Deployment
name: cognee-cognee-chart
updatePolicy:
updateMode: "Auto"
```
</Step>
<Step title="Scale the database, not the app">
Give the bundled Postgres more CPU, memory, and PVC space through `postgres.resources` and `postgres.storage`, or run a managed Postgres alongside a chart fork.
</Step>
</Steps>
## Maintenance Operations
<AccordionGroup>
<Accordion title="Updates & Rollbacks">
```bash theme={null}
# Update deployment
helm upgrade cognee ./deployment/helm -n cognee -f values.yaml
# Check rollout status
kubectl rollout status deployment/cognee-cognee-chart -n cognee
# Rollback if needed
helm rollback cognee 1 -n cognee
```
</Accordion>
<Accordion title="Rotating credentials">
```bash theme={null}
# Update the Secret in place
kubectl create secret generic cognee-credentials -n cognee \
--from-literal=LLM_API_KEY="sk-new..." \
--from-literal=DB_PASSWORD="newpassword" \
--dry-run=client -o yaml | kubectl apply -f -
# Restart the pods — the checksum annotations track only the
# chart-rendered Secret, not an existingSecret managed outside Helm
kubectl rollout restart -n cognee \
deployment/cognee-cognee-chart \
deployment/cognee-cognee-chart-postgres
```
<Note>
Changing `DB_PASSWORD` after Postgres has been initialized does not change the password already stored in the database volume — rotate it inside Postgres as well, or reinitialize the volume.
</Note>
</Accordion>
<Accordion title="Health Monitoring">
```bash theme={null}
# Check pod health and probe events
kubectl describe pods -n cognee -l app.kubernetes.io/instance=cognee
# View resource usage
kubectl top pods -n cognee
kubectl top nodes
# Check service endpoints
kubectl get endpoints -n cognee
```
</Accordion>
</AccordionGroup>
## Troubleshooting
<Tabs>
<Tab title="Values Rejected">
**`values don't meet the specifications of the schema`**
The chart validates values against `values.schema.json` before rendering. The message names the offending path — most often an unknown top-level key or a key from an older values layout.
```bash theme={null}
# Validate without touching the cluster
helm template cognee ./deployment/helm -f values.yaml
# Compare against the shipped defaults
helm show values ./deployment/helm
```
</Tab>
<Tab title="Pod Not Ready">
**Startup or readiness probe failing**
```bash theme={null}
# Probe failures show up as events
kubectl describe pod <pod-name> -n cognee
kubectl logs <pod-name> -n cognee --previous
```
A pod that never becomes ready usually means `/health` cannot reach Postgres. Check that the Postgres pod is ready and that `DB_PASSWORD` in the Secret matches the password Postgres was initialized with.
</Tab>
<Tab title="LLM Calls Failing">
**Empty `LLM_API_KEY`**
Without `existingSecret`, the chart-rendered Secret contains an empty `LLM_API_KEY`. Patch it, or reinstall with `existingSecret`:
```bash theme={null}
kubectl patch secret cognee-cognee-chart-secret -n cognee \
--type=merge \
-p '{"data":{"LLM_API_KEY":"'$(echo -n "sk-..." | base64)'"}}'
```
</Tab>
<Tab title="Database Issues">
**Connection Problems**
```bash theme={null}
# Test database connectivity — psql ships in the Postgres container,
# not in the API image
kubectl exec -it deploy/cognee-cognee-chart-postgres -n cognee -- \
psql -U cognee -d cognee_db -c "SELECT 1;"
# Check service DNS resolution from the API pod
kubectl exec -it <cognee-pod> -n cognee -- \
sh -c 'getent hosts cognee-cognee-chart-postgres'
```
</Tab>
</Tabs>
## Uninstalling
<Steps>
<Step title="Remove Helm Release">
```bash theme={null}
helm uninstall cognee -n cognee
```
</Step>
<Step title="Clean Up Resources">
```bash theme={null}
# Remove the Postgres volume (if desired)
kubectl delete pvc cognee-cognee-chart-postgres-pvc -n cognee
# Remove a pre-created credentials Secret
kubectl delete secret cognee-credentials -n cognee
```
</Step>
</Steps>
<Warning>
Uninstalling will permanently delete all data unless you have backups. Ensure you have proper backup procedures in place.
</Warning>
## Next Steps
<CardGroup>
<Card title="Monitoring Setup" icon="bar-chart">
**Observability Stack**
The chart does not ship ServiceMonitor or dashboard resources — wire Prometheus and Grafana in through your own manifests.
</Card>
<Card title="CI/CD Integration" icon="git-branch">
**GitOps Deployment**
Set up automated deployments with ArgoCD or Flux.
</Card>
</CardGroup>
<Card title="Need Help?" href="https://discord.gg/m63hxKsp4p" icon="discord">
Join our community for Kubernetes deployment support and production best practices.
</Card>
# Deployment Overview
Source: https://docs.cognee.ai/how-to-guides/cognee-sdk/deployment/index
Deploy Cognee with flexible data storage options for any scale
Cognee is designed for flexible deployment across development and production environments, with configurable data storage backends that scale with your needs.
## Data Storage Architecture
Cognee operates on a three-tier data storage model, each optimized for specific data types and query patterns:
<CardGroup>
<Card title="Graph Database" icon="network">
**Relationships & Entities**
Stores knowledge graph structure, entity relationships, and semantic connections.
</Card>
<Card title="Vector Database" icon="brain">
**Embeddings & Search**
Handles semantic embeddings for similarity search and content retrieval.
</Card>
<Card title="Relational Database" icon="database">
**Metadata & State**
Manages datasets, user permissions, pipeline state, and operational data.
</Card>
</CardGroup>
<Info>
Each storage layer can be deployed as managed services, self-hosted servers, or file-based systems (like S3 buckets), giving you complete flexibility over your infrastructure.
</Info>
## Deployment Options
Choose the deployment strategy that matches your requirements:
<Tabs>
<Tab title="Development">
**Local & Testing**
* **Docker**: Containerized local deployment with embedded databases
* **MCP**: Direct integration with code editors and IDEs
* **File-based**: SQLite, local files, and embedded vector stores
</Tab>
<Tab title="Production">
**Scalable & Managed**
* **Coolify**: Self-hosted PaaS on your own VPS with managed HTTPS
* **Islo**: Ephemeral cloud sandbox for demos and short-lived API instances
* **Kubernetes**: Container orchestration with Helm charts
* **EC2**: Traditional cloud server deployment
* **Cloud Services**: Managed databases (RDS, Neo4j Aura, Qdrant)
</Tab>
<Tab title="Hybrid">
**Flexible Storage**
* **S3 + Servers**: File storage in S3 with managed database services
* **Multi-cloud**: Different storage tiers across cloud providers
* **Edge**: Local processing with cloud storage backends
</Tab>
</Tabs>
## Storage Configuration Examples
<AccordionGroup>
<Accordion title="Local Development">
**Embedded & File-based**
```bash theme={null}
# All data stored locally
GRAPH_DATABASE=networkx
VECTOR_DATABASE=lancedb
RELATIONAL_DATABASE=sqlite://./cognee.db
```
<Warning>
**Multi-Agent Limitation**: Default Kuzu graph store uses file-based locking and is not suitable for concurrent access from multiple agents. Use Neo4j or FalkorDB for multi-agent deployments.
</Warning>
</Accordion>
<Accordion title="Cloud Production">
**Managed Services**
```bash theme={null}
# Fully managed cloud services
GRAPH_DATABASE=neo4j://your-aura-instance
VECTOR_DATABASE=pinecone://your-index
RELATIONAL_DATABASE=postgresql://your-rds-instance
```
</Accordion>
<Accordion title="Hybrid S3">
**S3 + Managed Databases**
```bash theme={null}
# Vector data in S3, databases managed
VECTOR_DATABASE=s3://your-bucket/vectors/
GRAPH_DATABASE=neo4j://managed-instance
RELATIONAL_DATABASE=postgresql://rds-instance
```
</Accordion>
<Accordion title="Migrating to Another Instance">
Cognee stores all persistent data under `SYSTEM_ROOT_DIRECTORY` (default: `.cognee_system`). There is no dedicated export API; migration works by either copying the database files or switching to shared external databases.
<Tabs>
<Tab title="Option 1: Copy database files">
Stop Cognee on the source instance, copy the `databases/` folder to the destination, then set `SYSTEM_ROOT_DIRECTORY` to the new path:
```bash theme={null}
# All three databases live under $SYSTEM_ROOT_DIRECTORY/databases/
cp -r /old-instance/.cognee_system/databases /new-instance/.cognee_system/databases
# Optionally copy raw data files
cp -r /old-instance/.data_storage /new-instance/.data_storage
```
File paths inside `databases/`:
* `cognee_graph_kuzu` — Kuzu graph database
* `cognee.lancedb` — LanceDB vector store
* `cognee_db` — SQLite relational database
On the new instance, configure:
```dotenv theme={null}
SYSTEM_ROOT_DIRECTORY=/new-instance/.cognee_system
DATA_ROOT_DIRECTORY=/new-instance/.data_storage
```
</Tab>
<Tab title="Option 2: Shared external databases">
Point both instances to the same managed services — no file copying required:
```dotenv theme={null}
GRAPH_DATABASE_PROVIDER=neo4j
GRAPH_DATABASE_URL=bolt://your-neo4j-host:7687
GRAPH_DATABASE_USERNAME=neo4j
GRAPH_DATABASE_PASSWORD=your-password
VECTOR_DB_PROVIDER=qdrant
VECTOR_DB_URL=http://your-qdrant-host:6333
DB_PROVIDER=postgres
DB_HOST=your-postgres-host
DB_USERNAME=cognee
DB_PASSWORD=your-password
```
</Tab>
</Tabs>
See [Graph Stores](/setup-configuration/graph-stores) and [Vector Stores](/setup-configuration/vector-stores) for all supported external providers.
</Accordion>
</AccordionGroup>
## Quick Start Guide
<Steps>
<Step title="Choose Deployment">
Select your deployment method based on scale and requirements
</Step>
<Step title="Configure Storage">
Set up your preferred combination of graph, vector, and relational databases
</Step>
<Step title="Deploy & Test">
Launch Cognee and verify connectivity to all storage backends
</Step>
<Step title="Scale">
Adjust storage and compute resources based on usage patterns
</Step>
</Steps>
## Deployment Methods
<CardGroup>
<Card title="Docker Deployment" href="/how-to-guides/cognee-sdk/deployment/docker" icon="container">
**Local & Server**
Start Cognee with optional databases using compose profiles.
</Card>
<Card title="Coolify Deployment" href="/how-to-guides/cognee-sdk/deployment/coolify" icon="anchor">
**Self-hosted PaaS**
Run the API on your own VPS with a prebuilt image and automatic Let's Encrypt SSL.
</Card>
<Card title="Islo Sandbox" href="/how-to-guides/cognee-sdk/deployment/deployment-options#ephemeral-cloud-sandbox-islo" icon="cloud">
**Ephemeral & Shareable**
Launch a temporary public Cognee API for demos and short-lived evaluation.
</Card>
<Card title="Kubernetes (Helm)" href="/how-to-guides/cognee-sdk/deployment/helm" icon="ship">
**Enterprise & Production**
Container orchestration with full control and high availability.
</Card>
<Card title="EC2 Deployment" href="/how-to-guides/cognee-sdk/deployment/ec2" icon="server">
**Traditional Cloud**
Standard server deployment with custom configurations.
</Card>
</CardGroup>
## Self-hosted vs Cognee Cloud
Cognee can run fully self-hosted without [Cognee Cloud](/cognee-cloud/overview). The open-source package works as an embedded Python SDK, a Docker/Compose service, or a server deployment on Kubernetes or a VM.
| | Self-hosted open source | Cognee Cloud |
| -------------- | ------------------------------------------------------------ | ----------------------------------------------------- |
| Account | Not required | Required |
| Infrastructure | Your laptop, server, VPC, or cloud account | Managed by Cognee |
| Data location | Your configured local or external databases | Cognee-managed storage |
| Best for | Custom infrastructure, air-gapped environments, full control | Hosted UI, collaboration, and lower operations burden |
The same core memory operations are available in both paths. [`cognee.serve()`](/cognee-cloud/connections/syncing-local-instance) can point the local SDK at Cognee Cloud or at your own self-hosted API backend; it does not copy local datasets by itself. To move an already-built local graph, use [`cognee.push()`](/core-concepts/main-operations/push).
Self-hosted data is organized by [datasets](/core-concepts/further-concepts/datasets), not Cloud projects. Without Cognee Cloud, datasets live in the storage backends you configure, such as local Kuzu/LanceDB/SQLite files, or external Postgres, Neo4j, Qdrant, and related services.
## Architecture Benefits
<Tip>
**Flexible Data Tiers**: Each storage layer can be independently scaled, managed, or migrated without affecting others.
</Tip>
<Note>
**Cost Optimization**: Use file-based storage (S3) for archival data and managed services for active workloads.
</Note>
<Warning>
**Security**: Ensure proper network security and access controls across all storage tiers in production deployments.
</Warning>
## Need Help?
<CardGroup>
<Card title="Setup Troubleshooting" href="/how-to-guides/cognee-sdk/deployment/docker#troubleshooting" icon="wrench">
Storage `PermissionError`, database connection refused, failed migrations, UI login loops, and agent datasets missing from the admin dashboard — with the cause and fix for each.
</Card>
<Card title="Join Our Community" href="https://discord.gg/m63hxKsp4p" icon="discord">
Get deployment support, share configurations, and connect with other Cognee users.
</Card>
</CardGroup>
# AWS Bedrock Integration
Source: https://docs.cognee.ai/integrations/aws-bedrock-integration
Use AWS Bedrock models with Cognee's native Bedrock provider.
AWS Bedrock is a **first-class LLM provider** in Cognee. You configure it directly with `LLM_PROVIDER="bedrock"` and a few AWS environment variables — Cognee talks to Bedrock natively through its built-in Bedrock adapter.
## Prerequisites
* AWS account with Bedrock model access enabled for the models you want to use
* Python 3.10+
* Cognee installed with the AWS extra (see below)
## Setup (Native Provider)
### 1. Install Cognee with the AWS extra
```bash theme={null}
pip install cognee[aws]
```
### 2. Configure your `.env`
Set `LLM_PROVIDER="bedrock"` and provide your AWS details:
```dotenv theme={null}
LLM_PROVIDER="bedrock"
LLM_MODEL="eu.amazon.nova-lite-v1:0"
LLM_API_KEY="<your_bedrock_api_key>"
LLM_MAX_COMPLETION_TOKENS="16384"
AWS_REGION="<your_aws_region>"
AWS_ACCESS_KEY_ID="<your_aws_access_key_id>"
AWS_SECRET_ACCESS_KEY="<your_aws_secret_access_key>"
AWS_SESSION_TOKEN="<your_aws_session_token>"
# Optional parameters
# AWS_BEDROCK_RUNTIME_ENDPOINT="bedrock-runtime.eu-west-1.amazonaws.com"
# AWS_PROFILE_NAME="<your_aws_profile_name>"
```
### 3. Choose an authentication method
The Bedrock adapter supports four ways to authenticate (it uses the first one it finds, in this order):
1. **API key** — generate a Bedrock API key on AWS and set it in `LLM_API_KEY`.
2. **AWS credentials** — set `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` (you can leave `LLM_API_KEY` unset). If you use temporary credentials (an access key ID starting with `ASIA...`), you must also set `AWS_SESSION_TOKEN`.
3. **AWS profile** — set `AWS_PROFILE_NAME` to use a profile from your AWS credentials file (the standard boto3 credential chain).
4. **Ambient IAM role** — set none of the above. Cognee then sends no credentials at all and boto3 resolves them from the default AWS credential chain, so an EC2 instance profile, ECS task role, or EKS IRSA service-account role with Bedrock permissions is enough.
`AWS_REGION` is applied with any of these methods, and `AWS_BEDROCK_RUNTIME_ENDPOINT` optionally overrides the Bedrock runtime endpoint.
### Running without explicit credentials
Explicit credentials are optional: `bedrock` is one of the providers for which Cognee does not require `LLM_API_KEY`, so leaving it unset does not raise a missing-key error. On AWS-hosted infrastructure, `LLM_PROVIDER`, `LLM_MODEL`, and `AWS_REGION` are enough to authenticate to Bedrock:
```dotenv theme={null}
LLM_PROVIDER="bedrock"
LLM_MODEL="us.amazon.nova-lite-v1:0"
AWS_REGION="us-east-1"
```
Because the order above is a first-match, make sure no stale `LLM_API_KEY`, `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY`, or `AWS_PROFILE_NAME` values are left in your `.env` or shell — any of them will be used instead of the instance role.
Embeddings are configured separately: the default embedding provider is OpenAI and falls back to `LLM_API_KEY` for its key, so with no keys set you still need to configure an [embedding provider](/setup-configuration/embedding-providers) before `remember()` works.
### Model naming
Use the Bedrock model ID directly in `LLM_MODEL` — no `bedrock/` prefix is needed because the provider is already set to `bedrock`. Model IDs are region-scoped, so the prefix depends on your region (`eu.` for Europe, `us.` for the US, etc.):
* Amazon Nova: `eu.amazon.nova-lite-v1:0`, `us.amazon.nova-pro-v1:0`
* Anthropic Claude: `anthropic.claude-3-5-sonnet-20240620-v1:0`
* OpenAI GPT-OSS: `openai.gpt-oss-120b-1:0`
<Info>
The exact model ID (and whether it needs a region prefix) varies by AWS region. Check the [AWS Bedrock model catalog](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html) for the ID that applies to your region.
</Info>
## Usage Example
```python theme={null}
import cognee
import asyncio
async def main():
# Remember text with Cognee
await cognee.remember(
"Natural language processing (NLP) is an interdisciplinary subfield of computer science and information retrieval."
)
# Query the knowledge graph
results = await cognee.recall("Tell me about NLP")
# Display the results
for result in results:
print(result)
if __name__ == '__main__':
asyncio.run(main())
```
## Optional: Using a LiteLLM Proxy
A LiteLLM proxy is **not required** for Bedrock. Use this approach only if you already run a LiteLLM proxy to centralize model routing, credentials, or logging across multiple services.
<Warning>
Use **LiteLLM Proxy** (not the SDK) for this approach. The proxy runs as a server that Cognee connects to over HTTP.
</Warning>
Install and configure the proxy:
```bash theme={null}
pip install litellm[proxy]
```
Create a `config.yaml`:
```yaml theme={null}
model_list:
- model_name: bedrock-claude-3-5-sonnet
litellm_params:
model: bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0
aws_access_key_id: your_aws_id
aws_secret_access_key: your_aws_key
aws_region_name: your_aws_region_name
drop_params: true
```
Start the proxy (it runs on `http://localhost:4000` by default):
```bash theme={null}
litellm --config config.yaml
```
Then point Cognee at the proxy by treating it as an OpenAI-compatible endpoint:
```dotenv theme={null}
LLM_PROVIDER="openai"
LLM_MODEL="litellm_proxy/bedrock-claude-3-5-sonnet"
LLM_ENDPOINT="http://localhost:4000"
LLM_API_KEY="doesn't matter"
```
<Note>
For detailed proxy setup, see the [official LiteLLM Bedrock tutorial](https://docs.litellm.ai/docs/providers/bedrock).
</Note>
## Troubleshooting
1. **Authentication Errors**: Verify your AWS credentials, region, and that Bedrock model access is enabled in the AWS console. For temporary (`ASIA...`) credentials, ensure `AWS_SESSION_TOKEN` is set **alongside** `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` — a session token is not used on its own.
2. **Model Not Found**: Confirm the model ID matches your region exactly (including any `eu.`/`us.` prefix).
3. **Rate Limit / Throttling Errors (`BedrockException`)**: When Bedrock returns throttling errors (e.g. `Too many requests`, `ThrottlingException`), Cognee automatically retries them with exponential backoff (up to 5 retries). If errors persist, request a quota increase in the AWS console, or enable client-side rate limiting by setting `LLM_RATE_LIMIT_ENABLED="true"` and tuning `LLM_RATE_LIMIT_REQUESTS` to stay within your account's requests-per-minute limit. See [Rate Limiting](/setup-configuration/llm-providers) for details.
4. **Connection Issues (proxy only)**: Check that the LiteLLM proxy is running on the expected port.
To enable verbose LLM logging, set `LITELLM_LOG="DEBUG"` in your `.env`.
## Resources
<CardGroup>
<Card title="LLM Providers" href="/setup-configuration/llm-providers" icon="brain">
**Cognee LLM Configuration**
Full reference for configuring AWS Bedrock and other LLM providers.
</Card>
<Card title="AWS Bedrock Models" href="https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html" icon="cloud">
**Available Models**
Browse all Bedrock models and find the model ID for your region.
</Card>
</CardGroup>
# Band
Source: https://docs.cognee.ai/integrations/band-integration
Give any Band AI agent persistent memory by wrapping its adapter with Cognee.
Wrap any [Band](https://band.ai) adapter in `CogneeMemoryAdapter` and every room your agent joins gets persistent, shared memory. Context is recalled before your adapter sees a message, the question and reply are stored after it answers, and a closed room is promoted into the permanent knowledge graph. Your adapter, prompts, and framework do not change.
<Info>
The `cognee-integration-band` package is not published on PyPI yet. It ships from the open draft [pull request #313](https://github.com/topoteretes/cognee-integrations/pull/313) in [cognee-integrations](https://github.com/topoteretes/cognee-integrations), so install it from that branch and expect details to move until the PR merges.
</Info>
## Why Use This Integration
* **One line of code**: `CogneeMemoryAdapter(inner)` wraps an adapter you already built
* **Framework agnostic**: works with every Band adapter — Anthropic, Claude SDK, LangGraph, CrewAI, Pydantic AI — because it imports nothing from `band`
* **Rooms become memory**: Band room `r1` maps to Cognee session `band-r1` instead of a transcript that disappears
* **Shared brain**: the default dataset is the one the Claude Code and Codex plugins use, so a Band agent can recall what a terminal session learned
* **Never breaks a turn**: every memory failure is logged and swallowed
## Install
You need a Band agent from the [agent console](https://app.band.ai/agents), a reachable Cognee server, and Python 3.11+. The package declares 3.10, but `band-sdk` sets the real floor. Cognee can be Cloud, self-hosted, or local — this is a thin HTTP client and never starts a server for you.
```bash theme={null}
git clone https://github.com/topoteretes/cognee-integrations.git
cd cognee-integrations
git checkout feat-band-integration
pip install "band-sdk[anthropic]" # or [langgraph], [crewai], ...
pip install -e integrations/band
```
Pick the `band-sdk` extra matching the adapter you plan to wrap. The memory package adds no runtime dependencies of its own.
## Configure
Point at your Cognee server once in `~/.cognee/.env` — the same file the [Claude Code](/integrations/claude-code-integration) and [Codex](/integrations/codex-integration) plugins read, so if you run either of those you are already done:
```bash theme={null}
mkdir -p ~/.cognee
cat >> ~/.cognee/.env <<'EOF'
COGNEE_BASE_URL="https://your-tenant.aws.cognee.ai"
COGNEE_API_KEY="ck_..."
EOF
chmod 600 ~/.cognee/.env
```
Precedence is exported environment variables, then this file, then defaults. Keep your Band and model credentials — `BAND_AGENT_ID`, `BAND_AGENT_API_KEY`, `ANTHROPIC_API_KEY` — exported in the shell instead.
| Variable | Default | Description |
| ----------------------- | ----------------------- | ------------------------------------------------------------------ |
| `COGNEE_BASE_URL` | `http://localhost:8011` | Cognee server URL |
| `COGNEE_API_KEY` | unset | Sent as the `X-Api-Key` header |
| `COGNEE_PLUGIN_DATASET` | `agent_sessions` | Dataset for both writes and recall |
| `COGNEE_RECALL_TOP_K` | `5` | Results per recall; ignored unless it parses as a positive integer |
| `COGNEE_ENV_FILE` | `~/.cognee/.env` | Read the configuration file from somewhere else |
<Note>
The default URL is Cognee's [agent-mode](/guides/deploy-rest-api-server#agent-mode) port, where the Claude Code and Codex plugins bootstrap a local API. A standard local server listens on `8000`, so set `COGNEE_BASE_URL` to match. Those plugins also mint an API key for you, while this integration only ever sends the `COGNEE_API_KEY` you give it.
</Note>
Timeouts and the session prefix have no environment variables. Override those, or any other field, in code with `CogneeSettings.resolve(recall_timeout=5.0, session_prefix="prod")`.
## Quick Start
Wrap the adapter you already pass to `Agent.create`:
```python theme={null}
import os
from band import Agent
from band.adapters import AnthropicAdapter
from cognee_band import CogneeMemoryAdapter
agent = Agent.create(
adapter=CogneeMemoryAdapter(AnthropicAdapter(prompt="...")),
agent_id=os.environ["BAND_AGENT_ID"],
api_key=os.environ["BAND_AGENT_API_KEY"],
)
await agent.run()
```
That is the whole integration. The wrapper forwards every attribute and lifecycle call to the adapter underneath, so the inner adapter never needs to know memory exists.
To confirm it works, store a fact in one room and ask for it back in a **different** room — same-room recall could just be conversation history. For the demo worth showing someone, tell the Cognee Claude Code plugin a fact in a terminal, let it sync, then ask your Band agent. Both write to and recall from the same `agent_sessions` dataset, so the answer comes back.
## Explicit Memory Tools
Automatic recall runs on every text message. To also let the model make deliberate memory calls, build one client and share it, which keeps the tools and the wrapper on one dataset:
```python theme={null}
from cognee_band import CogneeClient, CogneeMemoryAdapter, CogneeSettings, cognee_tools
settings = CogneeSettings.resolve()
client = CogneeClient(settings)
inner = AnthropicAdapter(
prompt=(
"Blocks labeled 'Cognee memory' contain context recalled from past "
"sessions, so treat them as your own memory. Use cognee_search for "
"explicit lookups and cognee_remember to store facts worth keeping."
),
additional_tools=cognee_tools(client),
)
agent = Agent.create(
adapter=CogneeMemoryAdapter(inner, settings=settings, client=client),
agent_id=os.environ["BAND_AGENT_ID"],
api_key=os.environ["BAND_AGENT_API_KEY"],
)
```
This gives the model `cognee_search(query)`, which searches the whole dataset rather than just the current room's session, and `cognee_remember(content)`, which writes durably under the `band_memory` node set. Include the prompt guidance above: without it the model may read the injected block as something the user said rather than as its own memory. `cognee_tools` works with any adapter that accepts `additional_tools`.
<Note>
`integrations/band/examples/memory_agent.py` is a complete agent built this way. Its inline dependency header resolves `cognee-integration-band` from PyPI, so until the package is published, run it with the interpreter where you installed the package rather than through `uv run`.
</Note>
## How It Works
| When | What happens |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| The agent starts | Logs the active dataset and server, so a wrong endpoint surfaces at once |
| A text message arrives | [Recalls](/core-concepts/main-operations/recall) context and injects it above the message as a labeled block |
| Your adapter replies | Stores the question and reply as a QA pair in the [session cache](/core-concepts/sessions-and-caching), in the background |
| A room closes | Bridges that room's session into the graph with [`improve`](/core-concepts/main-operations/improve) |
| The agent stops | Drains pending writes, then bridges every room still open |
Recall is the only memory operation on the critical path, since the result is needed before your adapter runs. Writes never block the event loop. Non-text events pass through untouched and are never stored, so a room that only carried those is never bridged. The stored question is always the original one, so recalled context is not re-ingested turn after turn, and capture also runs when the inner adapter raises.
This is what the model receives, and when recall finds nothing, nothing is injected:
```
Cognee memory — context recalled from shared memory, possibly relevant to the message below:
- the deployment target is EKS, decided last Tuesday
- Ravit owns the migration checklist
---
what's left on the migration?
```
## Choosing Your Memory Scope
`COGNEE_PLUGIN_DATASET` is the isolation control. Recall reads exactly one [dataset](/core-concepts/further-concepts/datasets), so this is a hard boundary and not a ranking preference. Leave the default `agent_sessions` for one shared brain across every agent and coding harness, or set one dataset name per team or per agent to isolate them. Decide before your agents start writing, since moving content between datasets afterwards is a Cognee-side job.
<Warning>
Recalled memory is injected into the prompt sent to your model provider, so whatever sits in the dataset can reach that provider. The default is shared with the Claude Code, Codex, and OpenClaw plugins. Choose a shared dataset deliberately.
</Warning>
## Troubleshooting
Memory degrades to a no-op and never breaks a turn, which means failures are quiet by design. Adapter problems are logged on the `cognee_band` logger and transport problems also write a `[cognee-band]` line to standard error, so configure logging before debugging anything below. Start with the line the wrapper logs at startup, `cognee memory active: dataset=... server=...`, which catches a wrong endpoint before you chase anything else.
<AccordionGroup>
<Accordion title="Nothing is recalled">
Look for a warning on the `cognee_band` logger or a `[cognee-band]` line on standard error. No warning means the server searched and found nothing, which is expected on a cold dataset. An empty result is never confused with a failure.
</Accordion>
<Accordion title="Server unreachable, or HTTP 401 and 403">
For an unreachable server, check that `COGNEE_BASE_URL` works from the agent process, remembering that the default points at the agent-mode port. For an authorization failure, set `COGNEE_API_KEY` — this integration never mints one for you, localhost included.
</Accordion>
<Accordion title="Configuration seems ignored">
Parsing never raises, so a malformed file is skipped silently. Check the format is `KEY=VALUE`, one per line, with an optional `export ` prefix and optional quotes, and no interpolation. Check nothing in your shell already exports the same variable, since exports win. The file is read once per process, so restart the agent after editing it.
</Accordion>
<Accordion title="Questions stored with empty answers">
Replies are captured by proxying the adapter's `send_message` call, so an adapter that emits output only through `send_event` is not captured.
</Accordion>
<Accordion title="Memory from a closed room never shows up">
Consolidation runs through `improve` in the background on the server, so give it time. Also confirm the agent shut down cleanly, since the shutdown path is what bridges rooms that were still open.
</Accordion>
</AccordionGroup>
## Current Limits
* **Only text messages are remembered.** Tool calls and reasoning events are excluded, so a fact surfaces only if it reaches the reply text. The client has a trace-storing method for this, but the adapter does not drive it yet.
* **Session names can collide.** Two agents on the same dataset serving the same room ID write to the same session. That is usually intended, but change `session_prefix` if you need them apart.
* **Recall adds latency to every text turn**, up to the 20 second default timeout.
<Tip>
Band's Jam Desktop bridges local Claude Code sessions onto Band. Those are real Claude Code sessions, so the [Cognee plugin for Claude Code](/integrations/claude-code-integration) already gives them memory. This package is for agents you build on the Band SDK directly.
</Tip>
***
<CardGroup>
<Card title="Integration Source" icon="github" href="https://github.com/topoteretes/cognee-integrations/pull/313">
Read the adapter, client, and examples on the open pull request
</Card>
<Card title="Band" icon="book" href="https://band.ai">
Learn about the Band agent platform
</Card>
</CardGroup>
# Cognee Plugin for Claude Code
Source: https://docs.cognee.ai/integrations/claude-code-integration
Give Claude Code persistent memory with the Cognee plugin.
Add persistent memory to [Claude Code](https://www.anthropic.com/claude-code) with the **Cognee memory plugin** — no code and no `pip install`. The plugin hooks into Claude Code's lifecycle, so it:
* captures your prompts, tool traces, and assistant responses into session memory
* injects relevant context on every prompt submit
* syncs the session into your knowledge graph on session end
Sessions are disposable; your memory isn't.
## Install
Install from the Claude Code marketplace. Run these in your terminal (or type the equivalent `/plugin …` slash commands directly in the Claude Code chat):
```bash theme={null}
claude plugin marketplace add topoteretes/cognee-integrations
claude plugin install cognee-memory@cognee
```
On startup you'll see a **"Cognee Memory Connected"** message, and the status line shows `cognee: <dataset> · <mode>`.
## Configure your backend
Configure the plugin **once** in `~/.cognee/.env`. The file is created with a commented template on the first session start, and its values act exactly like shell exports — a real `export` in your shell still wins, per terminal. It is shared with the Codex plugin, so both read the same configuration.
<Tabs>
<Tab title="Cognee Cloud / remote">
Point the plugin at [Cognee Cloud](/cognee-cloud/overview) or a remote server by setting both:
```bash theme={null}
mkdir -p ~/.cognee
cat >> ~/.cognee/.env <<'EOF'
COGNEE_BASE_URL="https://your-tenant.aws.cognee.ai"
COGNEE_API_KEY="ck_..."
EOF
chmod 600 ~/.cognee/.env
```
Cloud mode is a pure thin client: it talks to your remote server over HTTP only and does **not** install a local Cognee runtime.
</Tab>
<Tab title="Local (default)">
When `COGNEE_BASE_URL` is unset, the plugin bootstraps a local Cognee API at `http://localhost:8011`. Only an LLM key is required — `COGNEE_API_KEY` is auto-minted if absent:
```bash theme={null}
mkdir -p ~/.cognee
cat >> ~/.cognee/.env <<'EOF'
LLM_API_KEY="sk-..."
EOF
chmod 600 ~/.cognee/.env
```
</Tab>
<Tab title="Windows (PowerShell)">
Same file, same keys:
```powershell theme={null}
New-Item -ItemType Directory -Force "$env:USERPROFILE\.cognee" | Out-Null
@'
COGNEE_BASE_URL="https://your-tenant.aws.cognee.ai"
COGNEE_API_KEY="ck_..."
'@ | Add-Content "$env:USERPROFILE\.cognee\.env"
```
</Tab>
</Tabs>
Re-running any of these blocks is safe: when a key appears more than once the **last value wins**, so pasting again with a new value updates the setting. Editing the file directly (`nano ~/.cognee/.env`) works too. Either way, changes apply on the next launch. The file format:
* Comments are whole lines starting with `#` — a trailing `# note` after a value becomes part of the value.
* Quotes around values are optional, and a leading `export ` is tolerated so existing shell profile lines paste verbatim.
* Every variable in the [Configuration Reference](#configuration-reference) can live here, so there is nothing else to persist. The one exception is `COGNEE_ENV_FILE` itself: the plugin reads it before it opens the file, so it only works as a shell export.
### Which mode wins, and how to switch
You can configure **both modes at once** — keep `COGNEE_BASE_URL` + `COGNEE_API_KEY` *and* `LLM_API_KEY` in the file together. The mode is then decided per terminal, by three rules in order:
1. **A `COGNEE_BACKEND` export wins.** `export COGNEE_BACKEND=local` (or `=cloud`) pins that terminal to that mode.
2. **Otherwise, cloud wins when configured.** If `COGNEE_BASE_URL` is set — in the file or the shell — the plugin connects to it.
3. **Otherwise, local.** With no URL anywhere, the plugin boots the local server.
So with both modes in `~/.cognee/.env`:
```bash theme={null}
claude # → cloud: a configured URL selects it
COGNEE_BACKEND=local claude # → local, this launch only
export COGNEE_BACKEND=local # → local for every launch from this shell
```
<Warning>
**To go local, use the switch — not `unset COGNEE_BASE_URL`.** Unsetting does not work: the env file re-injects the URL at the next launch. Export `COGNEE_BACKEND=local` instead. (Deleting the line from the file works too, but that changes the default for *every* terminal.)
</Warning>
Details worth knowing:
* **The switch is pinned.** `COGNEE_BACKEND=cloud` with no `COGNEE_BASE_URL` configured still counts as cloud — the plugin does **not** silently fall back to local, and the status line shows `✕ (missing_cognee_base_url)` so you know exactly what to fix.
* A forced-local switch **blanks** `COGNEE_BASE_URL` and `COGNEE_API_KEY` in the process environment, so the per-prompt hooks and every spawned worker resolve the same local endpoint — not just `SessionStart`. They are emptied rather than deleted on purpose: a child process that reloads the env file must not re-inject the cloud values.
* The shared `COGNEE_BACKEND` flips both the Claude Code **and** Codex plugins in that terminal. To flip only one, use the plugin-specific name — `COGNEE_CLAUDE_BACKEND` or `COGNEE_CODEX_BACKEND` — which beats the shared one.
* Accepted values: `local` (aliases `native`, `sdk`) and `cloud` (aliases `http`, `api`, `server`). Anything else is ignored.
* `COGNEE_BACKEND` can also live in `~/.cognee/.env` to make a mode the durable default; a shell export still overrides it per terminal.
* Not sure what a terminal resolved? The status line's mode field shows it live.
<Note>
**Cognee's LLM calls do not run through your coding agent.** Your Claude Code or Codex plan pays only for your conversation with the model. Everything Cognee does on its own — entity and relationship extraction during cognify, summarization, embeddings, and search-time completions — happens inside the Cognee backend against the LLM provider configured *there*, and is billed by that provider. In local mode you configure it with `LLM_API_KEY`; in Cloud/remote mode your tenant holds the key server-side, so no local LLM key is needed.
</Note>
In local mode, the single `LLM_API_KEY` above covers extraction, summarization, and embeddings: Cognee defaults to `openai/gpt-5-mini` for the LLM and `openai/text-embedding-3-large` for embeddings, and embeddings reuse `LLM_API_KEY` when `EMBEDDING_API_KEY` is unset. To use another provider, set `LLM_PROVIDER`, `LLM_MODEL`, and — for Azure, Ollama, or OpenAI-compatible endpoints — `LLM_ENDPOINT`. Changing only the LLM leaves embeddings on OpenAI, so also set the `EMBEDDING_*` variables or set `EMBEDDING_API_KEY` to an OpenAI key so the default embeddings keep working. See [LLM providers](/setup-configuration/llm-providers) and [embedding providers](/setup-configuration/embedding-providers).
## Use it
Just use Claude Code as usual — memory is captured and recalled automatically. You can also invoke the skills explicitly:
| Skill | Purpose |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `/cognee-memory:cognee-remember` | Store something in memory now |
| `/cognee-memory:cognee-search` | Query your memory |
| `/cognee-memory:cognee-sync` | Persist the current session into the graph |
| `/cognee-memory:cognee-forget` | Delete memory you ask to forget — the raw data, its derived graph knowledge, and, best-effort, the session Q\&A that cited it. Irreversible |
| `/cognee-memory:cognee-code` | Index a repository into the code graph and query it (callers, impact analysis, paths, endpoints) |
| `/cognee-memory:cognee-switch-datasets` | Move the running session to another dataset, syncing the current one first |
To verify the connection, open a fresh session and ask: *"What do you know from cognee?"*
<Info>
With the plugin active, Cognee is the **preferred** memory: the `SessionStart` hook steers Claude to treat Cognee as authoritative over Claude Code's built-in `MEMORY.md`. Set `COGNEE_PREFER_MEMORY=false` to turn the steer off.
</Info>
## Sessions & datasets
* **Sessions** — by default the plugin derives the Cognee session id from the Claude Code session, so a new conversation (or `/clear`) starts a new one and `claude --resume` continues the same one automatically. Set `COGNEE_SESSION_ID` before launching to pin a named session, or to deliberately share one live session across two terminals.
* **Datasets** — all writes and recall are scoped to one dataset (`agent_sessions` by default). Set `COGNEE_PLUGIN_DATASET` to use a custom one. The Claude Code and Codex plugins share the default dataset, so memory carries across both.
To move a running session to another dataset, run `/cognee-memory:cognee-switch-datasets` (optionally with a name). Without a name it lists the datasets you can write to; a name that is not listed is created for you. Because a Cognee session never spans two datasets, the switch first syncs the current session into its dataset — and aborts if that fails, changing nothing — then registers a fresh session on the chosen one. The choice lives in the launch record, so it survives `--resume` and beats `COGNEE_PLUGIN_DATASET` for the rest of the launch.
## How It Works
The plugin registers Claude Code lifecycle hooks:
| Hook | Fires | What it does |
| ------------------ | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `SessionStart` | Session launch | Selects local or remote mode, sets up session identity, prepares the dataset, and starts the background idle watcher |
| `UserPromptSubmit` | Each prompt you submit | Recalls relevant, dataset-scoped context from Cognee and injects it, then stages the prompt in session memory in the background |
| `PostToolUse` | After each tool call | Stores the tool call as a trace entry in session memory, without blocking the tool |
| `Stop` | Turn end | Stores the assistant's answer in session memory, paired with the prompt it answered |
| `PreCompact` | Before context compaction | Builds a memory anchor so relevant context survives the compacted history |
| `SessionEnd` | Session exit | Starts a detached worker that runs the final sync of the session into your knowledge graph |
A background idle watcher persists the session cache after periods of inactivity, and a final sync on session end bridges the session into the permanent graph.
## Session distillation (self-improvement)
The Cognee coding-agent plugins (Claude Code, Codex) run session distillation for you — you never call `improve()` by hand. A distillation pass fires on three triggers:
| Trigger | When it fires | How |
| ---------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Session end** | You quit the agent | The `SessionEnd` hook drains buffered turns into the server session cache, then fires `POST /api/v1/improve` for the session id in the background. A launch-exit watcher covers exits where the hook never fires — a hard exit, or Codex CLI shutdowns that skip `SessionEnd`. |
| **Every N tool calls** | Long sessions, incrementally | A per-session counter fires an improve every `COGNEE_AUTO_IMPROVE_EVERY` stored tool calls/stops (default **150**), so a long session bridges into the graph without waiting for the end. |
| **On idle** | The session goes quiet | A background idle watcher polls every `COGNEE_IDLE_POLL` seconds and fires an improve after `COGNEE_IDLE_THRESHOLD` seconds of inactivity, then waits at least `COGNEE_IMPROVE_COOLDOWN` seconds before the next idle run. |
<Note>
Overlapping triggers are safe. A per-session improve **lock** on the server serializes concurrent runs, and unchanged session content **dedups server-side by content hash** — so a repeat improve over content that hasn't changed is a cheap no-op, not duplicated work.
</Note>
### Configuration
All triggers are tuned through environment variables read by the plugin. The defaults are chosen so distillation stays out of your way; you rarely need to change them.
| Variable | Default | What it controls |
| ------------------------------- | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `COGNEE_AUTO_IMPROVE_EVERY` | `150` | Stored tool calls/stops between automatic mid-session improves |
| `COGNEE_IDLE_THRESHOLD` | `60` | Seconds of inactivity before an idle improve fires |
| `COGNEE_IMPROVE_COOLDOWN` | `600` | Minimum seconds between idle improve runs |
| `COGNEE_IDLE_POLL` | `10` | How often the idle watcher checks for inactivity |
| `COGNEE_IMPROVE_SUBMIT_TIMEOUT` | `420` (Claude Code) / `180` (Codex) | Read timeout for the improve POST (distillation runs inside the request). The Claude Code plugin raises this to `420` at startup to clear cognee's LLM-retry floor; Codex uses the `180` fallback. |
| `COGNEE_IDLE_DISABLED` | *unset* | Set to `1` / `true` to turn off the idle-watcher trigger entirely |
<Tip>
The plugin READMEs document additional advanced knobs — timing (poll deadlines, busy-retry intervals for a held session lock), session-sync retries, and the update-notification variables (`COGNEE_UPDATE_CHECK`, `COGNEE_UPDATE_CHECK_INTERVAL`). You almost never need them — reach for the table above first.
</Tip>
### Turning it down or off
* **Stop idle-triggered improves:** set `COGNEE_IDLE_DISABLED=1` before launching the agent. Session-end and per-turn improves still run.
* **Reduce mid-session improves:** raise `COGNEE_AUTO_IMPROVE_EVERY` to a large value so the per-turn trigger effectively never fires within a session.
* **Session-end distillation always runs** when the plugin is active — it's how a finished session reaches permanent memory.
### Confirming it happened
* **Cloud UI:** the **Self-improvement** card at the top of a session on the [Sessions page](/cognee-cloud/ui/sessions#self-improvement) shows the status of the last graph enrichment and the dataset it wrote to.
* **Plugin hook log:** each automatic run emits an `improve_fired` event you can grep for when debugging (in local SDK mode, where the plugin calls the library directly instead of the HTTP endpoint, look for `auto_improve_fired` instead).
* **`improve-unsupported.json` marker:** if this file appears in the plugin's shared state directory (24h TTL), the server rejected the improve endpoint and the plugin fell back to the legacy `remember` bridge for that window — a signal the server predates session-aware improve.
## Code graph
Repositories can be indexed into a deterministic **code graph** — symbols, calls, imports, endpoints, dependencies. Indexing makes **no LLM or embedding calls** by default, so it is fast and costs no tokens. It requires a Cognee server ≥ 1.5.3.
Starting a session inside a git repository indexes it automatically — **but only when the server is local**, so a private checkout is never shipped to a hosted tenant on the plugin's own initiative. Auto-indexing runs in the background, never blocks the first prompt, and re-indexes after any turn that changed the working tree.
Index explicitly when automation won't: against a Cloud tenant, for a different repo, or for a git URL. The simplest route is to ask the agent to index the repository — its code skill (`/cognee-memory:cognee-code` in Claude Code, `codebase` in Codex) runs this command:
```bash theme={null}
${CLAUDE_PLUGIN_ROOT}/scripts/cognee-index-repo.sh <repo-path-or-git-url> [--dataset <name>] [--index-vectors] [--wait <seconds>]
```
The plugin-root variable is expanded only inside the plugin's skills, so from your own terminal substitute the plugin's install path. An explicit request is its own consent, so it bypasses every auto-index gate below. Add `--index-vectors` to also embed the extracted code facts so semantic search can see them — that is the one flag that makes embedding calls.
Query the graph through the same code skill. Prompts that mention an identifier-shaped token from an indexed repo also get code facts injected automatically by the per-prompt recall hook.
Each indexed repository gets its own dataset, named `codebase-<repo-name>-<digest>`, where the digest identifies the indexed path or git URL — two checkouts that share a basename would otherwise share one graph and delete each other's nodes. Code-graph searches resolve the dataset from the current checkout, so the generated name rarely needs typing. Indexing writes its snapshot into the repository itself at `<repo>/.enola/` (untracked) — add `.enola/` to the repository's `.gitignore` or your global excludes.
<Warning>
**Freshness depends on where the server runs.** A **local** server indexes the repository path on this machine, so the graph reflects your working tree, including uncommitted and untracked changes. A **cloud or remote** server clones a git URL — it cannot read your disk, so the graph reflects only the **last pushed commit**, and the plugin does not re-submit URL-indexed repositories after local edits. The output looks identical either way, so push before relying on code answers about work in progress, or use a local server for branches you are actively editing.
</Warning>
| Env var | Default | Effect |
| ----------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `COGNEE_CODE_AUTOINDEX` | `auto` | `auto`: auto-index new repositories only when the server is local, so code never leaves the machine · `always`: also auto-index against a remote server — note that it submits the local path, which only a server sharing your filesystem can read, so a hosted tenant still needs an explicit git URL · `off`: never auto-index new repositories (explicitly indexed ones still refresh) |
`always` also accepts `1`, `true`, `yes`, and `on`; `off` also accepts `0`, `false`, and `no`. Any other value means `auto`. Automatic indexing skips directories that are not git repositories, hold no source files, or exceed 3000 source files. Explicit indexing has no size cap.
## Debugging & Resuming Sessions
Hooks are callbacks from Claude Code, not a durable job queue: events that happen while the plugin is uninstalled, failing, or unable to reach the backend are not replayed later. When memory does not appear, check these layers first:
| Layer | What it controls | How to verify |
| ----------------- | ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Claude Code hooks | Whether Claude Code calls the plugin on prompt submit, tool use, stop, compaction, and session end | Confirm the plugin is installed and the startup message says **"Cognee Memory Connected"** |
| Cognee backend | Where the plugin writes session memory and recalls graph context | In Cloud/remote mode, confirm `COGNEE_BASE_URL` and `COGNEE_API_KEY`; in local mode, confirm `LLM_API_KEY` is set so the plugin-managed local API can start |
| Session identity | Which live session receives staged prompts, traces, and answers | A resume keeps the session on its own; when you expect *two* terminals to share one, confirm both set the same `COGNEE_SESSION_ID` and `COGNEE_PLUGIN_DATASET` |
| Mode | Which backend this terminal writes to and recalls from — memory written in one mode is not visible from the other | Read the mode field in the status line; if it is not what you expect, check for a forgotten `COGNEE_BACKEND` / `COGNEE_CLAUDE_BACKEND` export in the shell or in `~/.cognee/.env`, and see [Which mode wins](#which-mode-wins-and-how-to-switch) |
A `claude --resume` of the same conversation continues the same live session on its own — the session id is derived from the Claude Code session, and a dataset chosen with `/cognee-memory:cognee-switch-datasets` is remembered too. `COGNEE_SESSION_ID` is what you need for the other case: making a *second* terminal, or a different conversation, write into one shared live session. Set it in both terminals and keep `COGNEE_PLUGIN_DATASET` the same, otherwise each lands in its own session. After changing credentials, dataset, or session id, restart Claude Code so `SessionStart` can run with the new state.
<Note>
Hook commands run with `python3`, falling back to `python` if `python3` isn't found. If neither resolves on `PATH` — most commonly on Windows, where the python.org installer doesn't always register a `python3` alias — every hook fails and no memory is ever captured. Run `python3 --version` or `python --version` in the same shell that launches Claude Code to confirm one is available, or reinstall Python with "Add python.exe to PATH" checked.
</Note>
For a clean handoff into long-term memory, run `/cognee-memory:cognee-sync` or exit Claude Code normally so `SessionEnd` can trigger the final graph sync. If the process is killed instead, recent session cache entries may exist, but the final session-to-graph sync may not have run yet.
## Configuration Reference
Precedence:
1. Environment variables (shell exports)
2. `~/.cognee/.env` — the one-time setup file, shared with the Codex plugin; loaded into the environment at process start, so every variable below except `COGNEE_ENV_FILE` itself can live in it
3. Defaults
The `COGNEE_BACKEND` / `COGNEE_CLAUDE_BACKEND` [mode switch](#which-mode-wins-and-how-to-switch) follows the same precedence — it is an ordinary variable, and a shell export of it beats a value in the file. What makes it special is its effect: wherever it is set, it pins the mode regardless of where the connection variables are defined.
<Note>
**There is no `config.json`.** Older plugin versions wrote `~/.cognee-plugin/claude-code/config.json`, and `SessionStart` read a `base_url` from it while the per-turn hooks did not — so a stale URL there could point the two halves of the plugin at different servers. `SessionStart` now deletes a leftover file. Put everything in `~/.cognee/.env` instead.
</Note>
| Setting | Env var | Default | Notes |
| ------------------------------ | -------------------------- | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------- |
| Dataset | `COGNEE_PLUGIN_DATASET` | `agent_sessions` | Dataset for writes and recall at launch; `/cognee-memory:cognee-switch-datasets` changes it mid-session |
| Session ID | `COGNEE_SESSION_ID` | derived from the Claude Code session | Override to pin a named session, or share one across terminals |
| Session prefix | `COGNEE_SESSION_PREFIX` | `claude` | Prefix on auto-derived session ids (`<prefix>_<claude-session-id>`) |
| Session strategy | `COGNEE_SESSION_STRATEGY` | `per-directory` | Currently inert — the plugin derives session ids from the Claude Code session and never reads this |
| Base URL | `COGNEE_BASE_URL` | unset | Set to use a managed/remote endpoint |
| API key | `COGNEE_API_KEY` | unset | Auto-minted if absent in local mode |
| Mode switch | `COGNEE_BACKEND` | unset | `local` or `cloud` — pins the terminal's mode, overriding the URL rule; flips the Claude Code **and** Codex plugins |
| Mode switch (this plugin only) | `COGNEE_CLAUDE_BACKEND` | unset | Same, for this plugin only; beats `COGNEE_BACKEND` |
| Env file location | `COGNEE_ENV_FILE` | `~/.cognee/.env` | Read the one-time setup file from somewhere else |
| Local API URL | `COGNEE_LOCAL_API_URL` | `http://localhost:8011` | Local API base URL |
| Local LLM | `LLM_API_KEY`, `LLM_MODEL` | unset | Required for local mode |
| Code auto-indexing | `COGNEE_CODE_AUTOINDEX` | `auto` | `auto`, `always`, or `off` — see [Code graph](#code-graph) |
| Prefer Cognee memory | `COGNEE_PREFER_MEMORY` | `true` | Inject the SessionStart memory steer |
## Update or Remove
There's no automatic update — reinstall to pull a new plugin version:
```bash theme={null}
claude plugin uninstall cognee-memory@cognee
claude plugin install cognee-memory@cognee
```
***
<CardGroup>
<Card title="GitHub Repository" icon="github" href="https://github.com/topoteretes/cognee-integrations/tree/main/integrations/claude-code">
View source code and the full configuration reference
</Card>
<Card title="Codex plugin" icon="terminal" href="/integrations/codex-integration">
The same memory plugin for the Codex CLI
</Card>
</CardGroup>
# Cognee Plugin for Codex
Source: https://docs.cognee.ai/integrations/codex-integration
Give Codex persistent memory with the Cognee plugin.
Add persistent memory to Codex with the **Cognee memory plugin** — no code and no `pip install`. It works in the Codex CLI and can also be activated through the Codex IDE plugin. The plugin hooks into Codex's lifecycle, so it:
* captures your prompts, tool traces, and assistant responses into session memory
* injects relevant context on every prompt submit
* syncs the session into your knowledge graph on session end
Sessions are disposable; your memory isn't.
## Install
The Cognee memory plugin depends on Codex lifecycle hooks. Enable hooks before installing it.
<Tabs>
<Tab title="CLI">
Enable hooks, then install from the Codex marketplace with the Codex CLI:
```bash theme={null}
codex features enable hooks
codex plugin marketplace add topoteretes/cognee-integrations --ref main
codex plugin add cognee@cognee
```
</Tab>
<Tab title="Manual">
If you prefer to manage Codex config directly, add this to `~/.codex/config.toml`:
```toml theme={null}
[features]
hooks = true
[marketplaces.cognee]
source_type = "git"
source = "https://github.com/topoteretes/cognee-integrations.git"
ref = "main"
[plugins."cognee@cognee"]
enabled = true
```
</Tab>
</Tabs>
<Info>
Make sure Cognee hooks are enabled for both the Codex CLI and the Codex IDE plugin. If Codex asks you to review hooks, open `/hooks` and allow or trust the Cognee hooks. Until hooks are enabled and trusted, Codex will not call the plugin on prompt submit, tool use, stop, compaction, or session end.
</Info>
On startup the status line shows `cognee: <dataset> · <mode>` to confirm the plugin is active.
## Configure your backend
Configure the plugin **once** in `~/.cognee/.env`. The file is created with a commented template on the first session start, and its values act exactly like shell exports — a real `export` in your shell still wins, per terminal. It is shared with the Claude Code plugin, so both read the same configuration.
<Tabs>
<Tab title="Cognee Cloud / remote">
Point the plugin at [Cognee Cloud](/cognee-cloud/overview) or a remote server by setting both:
```bash theme={null}
mkdir -p ~/.cognee
cat >> ~/.cognee/.env <<'EOF'
COGNEE_BASE_URL="https://your-tenant.aws.cognee.ai"
COGNEE_API_KEY="ck_..."
EOF
chmod 600 ~/.cognee/.env
```
Cloud mode is a pure thin client: it talks to your remote server over HTTP only and does **not** install a local Cognee runtime.
</Tab>
<Tab title="Local (default)">
When `COGNEE_BASE_URL` is unset, the plugin bootstraps a local Cognee API at `http://localhost:8011`. Only an LLM key is required — `COGNEE_API_KEY` is auto-minted if absent:
```bash theme={null}
mkdir -p ~/.cognee
cat >> ~/.cognee/.env <<'EOF'
LLM_API_KEY="sk-..."
EOF
chmod 600 ~/.cognee/.env
```
</Tab>
<Tab title="Windows (PowerShell), either mode">
Same file, same keys — swap in `LLM_API_KEY="sk-..."` for local mode:
```powershell theme={null}
New-Item -ItemType Directory -Force "$env:USERPROFILE\.cognee" | Out-Null
@'
COGNEE_BASE_URL="https://your-tenant.aws.cognee.ai"
COGNEE_API_KEY="ck_..."
'@ | Add-Content "$env:USERPROFILE\.cognee\.env"
```
</Tab>
</Tabs>
Re-running any of these blocks is safe: when a key appears more than once the **last value wins**, so pasting again with a new value updates the setting. Editing the file directly (`nano ~/.cognee/.env`) works too. Either way, changes apply on the next launch. The file format:
* Comments are whole lines starting with `#` — a trailing `# note` after a value becomes part of the value.
* Quotes around values are optional, and a leading `export ` is tolerated so existing shell profile lines paste verbatim.
* Every variable in the [Configuration Reference](#configuration-reference) can live here, so there is nothing else to persist. The one exception is `COGNEE_ENV_FILE` itself: the plugin reads it before it opens the file, so it only works as a shell export.
### Which mode wins, and how to switch
You can configure **both modes at once** — keep `COGNEE_BASE_URL` + `COGNEE_API_KEY` *and* `LLM_API_KEY` in the file together. The mode is then decided per terminal, by three rules in order:
1. **A `COGNEE_BACKEND` export wins.** `export COGNEE_BACKEND=local` (or `=cloud`) pins that terminal to that mode.
2. **Otherwise, cloud wins when configured.** If `COGNEE_BASE_URL` is set — in the file or the shell — the plugin connects to it.
3. **Otherwise, local.** With no URL anywhere, the plugin boots the local server.
So with both modes in `~/.cognee/.env`:
```bash theme={null}
codex # → cloud: a configured URL selects it
COGNEE_BACKEND=local codex # → local, this launch only
export COGNEE_BACKEND=local # → local for every launch from this shell
```
<Warning>
**To go local, use the switch — not `unset COGNEE_BASE_URL`.** Unsetting does not work: the env file re-injects the URL at the next launch. Export `COGNEE_BACKEND=local` instead. (Deleting the line from the file works too, but that changes the default for *every* terminal.)
</Warning>
Details worth knowing:
* **The switch is pinned.** `COGNEE_BACKEND=cloud` with no `COGNEE_BASE_URL` configured still counts as cloud — the plugin does **not** silently fall back to local, and the status line shows `✕ (missing_cognee_base_url)` so you know exactly what to fix.
* A forced-local switch **blanks** `COGNEE_BASE_URL` and `COGNEE_API_KEY` in the process environment, so the per-prompt hooks and every spawned worker resolve the same local endpoint — not just `SessionStart`. They are emptied rather than deleted on purpose: a child process that reloads the env file must not re-inject the cloud values.
* The shared `COGNEE_BACKEND` flips both the Claude Code **and** Codex plugins in that terminal. To flip only one, use the plugin-specific name — `COGNEE_CLAUDE_BACKEND` or `COGNEE_CODEX_BACKEND` — which beats the shared one.
* Accepted values: `local` (aliases `native`, `sdk`) and `cloud` (aliases `http`, `api`, `server`). Anything else is ignored.
* `COGNEE_BACKEND` can also live in `~/.cognee/.env` to make a mode the durable default; a shell export still overrides it per terminal.
* Not sure what a terminal resolved? The status line's mode field shows it live.
<Note>
**Cognee's LLM calls do not run through your coding agent.** Your Claude Code or Codex plan pays only for your conversation with the model. Everything Cognee does on its own — entity and relationship extraction during cognify, summarization, embeddings, and search-time completions — happens inside the Cognee backend against the LLM provider configured *there*, and is billed by that provider. In local mode you configure it with `LLM_API_KEY`; in Cloud/remote mode your tenant holds the key server-side, so no local LLM key is needed.
</Note>
In local mode, the single `LLM_API_KEY` above covers extraction, summarization, and embeddings: Cognee defaults to `openai/gpt-5-mini` for the LLM and `openai/text-embedding-3-large` for embeddings, and embeddings reuse `LLM_API_KEY` when `EMBEDDING_API_KEY` is unset. To use another provider, set `LLM_PROVIDER`, `LLM_MODEL`, and — for Azure, Ollama, or OpenAI-compatible endpoints — `LLM_ENDPOINT`. Changing only the LLM leaves embeddings on OpenAI, so also set the `EMBEDDING_*` variables or set `EMBEDDING_API_KEY` to an OpenAI key so the default embeddings keep working. See [LLM providers](/setup-configuration/llm-providers) and [embedding providers](/setup-configuration/embedding-providers).
## Use it
Use Codex as usual — memory is captured and recalled automatically. To verify, end a session with `/exit` (which syncs it into Cognee), then start a fresh session and ask: *"What do you know from cognee?"* Answering from a clean session proves it's recalling from your memory.
The plugin also ships skills for explicit requests. Ask for what you want and Codex picks the matching skill:
| Skill | Purpose |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `memory` | Remember something now, search or recall memory, and improve existing memory |
| `cognee-forget` | Delete memory you ask to forget — the raw data, its derived graph knowledge, and, best-effort, the session Q\&A that cited it. Irreversible |
| `codebase` | Index a repository into the code graph and query it (callers, impact analysis, paths, endpoints) |
| `cognee-switch-datasets` | Move the running session to another dataset, syncing the current one first |
| `setup` | Configure and check Cognee through the `cognee-cli` |
| `local-ui` | Launch and health-check the local Cognee UI and backend |
## Sessions & datasets
* **Sessions** — by default the plugin derives the Cognee session id from the Codex thread, so a new conversation starts a new one and `codex resume` continues the same one automatically. (Should a launch report no thread id at all, the plugin falls back to a fresh per-launch id.) Set `COGNEE_SESSION_ID` before launching to pin a named session, or to deliberately share one live session across two terminals.
* **Datasets** — all writes and recall are scoped to one dataset (`agent_sessions` by default). Set `COGNEE_PLUGIN_DATASET` to use a custom one. The Codex and Claude Code plugins share the default dataset, so memory carries across both.
To move a running session to another dataset, ask Codex to switch datasets (the `cognee-switch-datasets` skill), optionally naming the dataset. Without a name it lists the datasets you can write to as a numbered list; a name that is not listed is created for you. Because a Cognee session never spans two datasets, the switch first syncs the current session into its dataset — and aborts if that fails, changing nothing — then registers a fresh session on the chosen one. The choice lives in the launch record, so it survives a resume and beats `COGNEE_PLUGIN_DATASET` for the rest of the launch.
## How It Works
The plugin registers Codex lifecycle hooks:
| Hook | Fires | What it does |
| ------------------ | ------------------------- | -------------------------------------------------------------------------------------------------------------------- |
| `SessionStart` | Session launch | Selects local or remote mode, sets up session identity, prepares the dataset, and starts the background idle watcher |
| `UserPromptSubmit` | Each prompt you submit | Recalls relevant context from Cognee and injects it, then stages the prompt in session memory in the background |
| `PostToolUse` | After each tool call | Stores the tool call as a trace entry in session memory, without blocking the tool |
| `Stop` | Turn end | Stores the assistant's answer in session memory, paired with the prompt it answered |
| `PreCompact` | Before context compaction | Builds a memory anchor so relevant context survives the compacted history |
| `SessionEnd` | Session exit | Starts a detached worker that runs the final sync of the session into your knowledge graph |
A background idle watcher persists the session cache after periods of inactivity, and a final sync on session end bridges the session into the permanent graph.
## Session distillation (self-improvement)
The Cognee coding-agent plugins (Claude Code, Codex) run session distillation for you — you never call `improve()` by hand. A distillation pass fires on three triggers:
| Trigger | When it fires | How |
| ---------------------- | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Session end** | You quit the agent | The `SessionEnd` hook drains buffered turns into the server session cache, then fires `POST /api/v1/improve` for the session id in the background. A launch-exit watcher covers exits where the hook never fires — a hard exit, or Codex CLI shutdowns that skip `SessionEnd`. |
| **Every N tool calls** | Long sessions, incrementally | A per-session counter fires an improve every `COGNEE_AUTO_IMPROVE_EVERY` stored tool calls/stops (default **150**), so a long session bridges into the graph without waiting for the end. |
| **On idle** | The session goes quiet | A background idle watcher polls every `COGNEE_IDLE_POLL` seconds and fires an improve after `COGNEE_IDLE_THRESHOLD` seconds of inactivity, then waits at least `COGNEE_IMPROVE_COOLDOWN` seconds before the next idle run. |
<Note>
Overlapping triggers are safe. A per-session improve **lock** on the server serializes concurrent runs, and unchanged session content **dedups server-side by content hash** — so a repeat improve over content that hasn't changed is a cheap no-op, not duplicated work.
</Note>
### Configuration
All triggers are tuned through environment variables read by the plugin. The defaults are chosen so distillation stays out of your way; you rarely need to change them.
| Variable | Default | What it controls |
| ------------------------------- | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `COGNEE_AUTO_IMPROVE_EVERY` | `150` | Stored tool calls/stops between automatic mid-session improves |
| `COGNEE_IDLE_THRESHOLD` | `60` | Seconds of inactivity before an idle improve fires |
| `COGNEE_IMPROVE_COOLDOWN` | `600` | Minimum seconds between idle improve runs |
| `COGNEE_IDLE_POLL` | `10` | How often the idle watcher checks for inactivity |
| `COGNEE_IMPROVE_SUBMIT_TIMEOUT` | `420` (Claude Code) / `180` (Codex) | Read timeout for the improve POST (distillation runs inside the request). The Claude Code plugin raises this to `420` at startup to clear cognee's LLM-retry floor; Codex uses the `180` fallback. |
| `COGNEE_IDLE_DISABLED` | *unset* | Set to `1` / `true` to turn off the idle-watcher trigger entirely |
<Tip>
The plugin READMEs document additional advanced knobs — timing (poll deadlines, busy-retry intervals for a held session lock), session-sync retries, and the update-notification variables (`COGNEE_UPDATE_CHECK`, `COGNEE_UPDATE_CHECK_INTERVAL`). You almost never need them — reach for the table above first.
</Tip>
### Turning it down or off
* **Stop idle-triggered improves:** set `COGNEE_IDLE_DISABLED=1` before launching the agent. Session-end and per-turn improves still run.
* **Reduce mid-session improves:** raise `COGNEE_AUTO_IMPROVE_EVERY` to a large value so the per-turn trigger effectively never fires within a session.
* **Session-end distillation always runs** when the plugin is active — it's how a finished session reaches permanent memory.
### Confirming it happened
* **Cloud UI:** the **Self-improvement** card at the top of a session on the [Sessions page](/cognee-cloud/ui/sessions#self-improvement) shows the status of the last graph enrichment and the dataset it wrote to.
* **Plugin hook log:** each automatic run emits an `improve_fired` event you can grep for when debugging (in local SDK mode, where the plugin calls the library directly instead of the HTTP endpoint, look for `auto_improve_fired` instead).
* **`improve-unsupported.json` marker:** if this file appears in the plugin's shared state directory (24h TTL), the server rejected the improve endpoint and the plugin fell back to the legacy `remember` bridge for that window — a signal the server predates session-aware improve.
## Code graph
Repositories can be indexed into a deterministic **code graph** — symbols, calls, imports, endpoints, dependencies. Indexing makes **no LLM or embedding calls** by default, so it is fast and costs no tokens. It requires a Cognee server ≥ 1.5.3.
Starting a session inside a git repository indexes it automatically — **but only when the server is local**, so a private checkout is never shipped to a hosted tenant on the plugin's own initiative. Auto-indexing runs in the background, never blocks the first prompt, and re-indexes after any turn that changed the working tree.
Index explicitly when automation won't: against a Cloud tenant, for a different repo, or for a git URL. The simplest route is to ask the agent to index the repository — its code skill (`/cognee-memory:cognee-code` in Claude Code, `codebase` in Codex) runs this command:
```bash theme={null}
${CODEX_PLUGIN_ROOT}/scripts/cognee-index-repo.sh <repo-path-or-git-url> [--dataset <name>] [--index-vectors] [--wait <seconds>]
```
The plugin-root variable is expanded only inside the plugin's skills, so from your own terminal substitute the plugin's install path. An explicit request is its own consent, so it bypasses every auto-index gate below. Add `--index-vectors` to also embed the extracted code facts so semantic search can see them — that is the one flag that makes embedding calls.
Query the graph through the same code skill. Prompts that mention an identifier-shaped token from an indexed repo also get code facts injected automatically by the per-prompt recall hook.
Each indexed repository gets its own dataset, named `codebase-<repo-name>-<digest>`, where the digest identifies the indexed path or git URL — two checkouts that share a basename would otherwise share one graph and delete each other's nodes. Code-graph searches resolve the dataset from the current checkout, so the generated name rarely needs typing. Indexing writes its snapshot into the repository itself at `<repo>/.enola/` (untracked) — add `.enola/` to the repository's `.gitignore` or your global excludes.
<Warning>
**Freshness depends on where the server runs.** A **local** server indexes the repository path on this machine, so the graph reflects your working tree, including uncommitted and untracked changes. A **cloud or remote** server clones a git URL — it cannot read your disk, so the graph reflects only the **last pushed commit**, and the plugin does not re-submit URL-indexed repositories after local edits. The output looks identical either way, so push before relying on code answers about work in progress, or use a local server for branches you are actively editing.
</Warning>
| Env var | Default | Effect |
| ----------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `COGNEE_CODE_AUTOINDEX` | `auto` | `auto`: auto-index new repositories only when the server is local, so code never leaves the machine · `always`: also auto-index against a remote server — note that it submits the local path, which only a server sharing your filesystem can read, so a hosted tenant still needs an explicit git URL · `off`: never auto-index new repositories (explicitly indexed ones still refresh) |
`always` also accepts `1`, `true`, `yes`, and `on`; `off` also accepts `0`, `false`, and `no`. Any other value means `auto`. Automatic indexing skips directories that are not git repositories, hold no source files, or exceed 3000 source files. Explicit indexing has no size cap.
## Debugging & Resuming Sessions
Hooks are callbacks from Codex, not a durable job queue: events that happen while hooks are disabled or untrusted, or while the plugin cannot reach the backend, are not replayed later. When memory does not appear, check these layers first:
| Layer | What it controls | How to verify |
| ---------------- | ----------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Codex hooks | Whether Codex calls the plugin on prompt submit, tool use, stop, compaction, and session end | Run `codex features enable hooks`, then open `/hooks` if Codex asks you to review or trust hooks |
| Cognee backend | Where the plugin writes session memory and recalls graph context | In Cloud/remote mode, confirm `COGNEE_BASE_URL` and `COGNEE_API_KEY`; in local mode, confirm `LLM_API_KEY` is set so the plugin-managed local API can start |
| Session identity | Which live session receives staged prompts, traces, and answers | A resume keeps the session on its own; when you expect *two* terminals to share one, confirm both set the same `COGNEE_SESSION_ID` and `COGNEE_PLUGIN_DATASET` |
| Mode | Which backend this terminal writes to and recalls from — memory written in one mode is not visible from the other | Read the mode field in the status line; if it is not what you expect, check for a forgotten `COGNEE_BACKEND` / `COGNEE_CODEX_BACKEND` export in the shell or in `~/.cognee/.env`, and see [Which mode wins](#which-mode-wins-and-how-to-switch) |
A resume keeps both the session and any dataset you switched to on its own. `COGNEE_SESSION_ID` is what you need for the other case: making a *second* terminal, or a different conversation, write into one shared live session. Set it in both terminals and keep `COGNEE_PLUGIN_DATASET` the same, otherwise each lands in its own session. After changing hook trust, credentials, dataset, or session id, restart Codex so `SessionStart` can run with the new state.
<Note>
Hook commands run with `python3`, falling back to `python` if `python3` isn't found. If neither resolves on `PATH` — most commonly on Windows, where the python.org installer doesn't always register a `python3` alias — every hook fails with a "hook failure" error and no session is ever created. Run `python3 --version` or `python --version` in the same shell that launches Codex to confirm one is available, or reinstall Python with "Add python.exe to PATH" checked.
</Note>
Exit Codex normally (for example with `/exit`) when you want `SessionEnd` to trigger the final graph sync. If the process is killed instead, recent session cache entries may exist, but the final session-to-graph sync may not have run yet.
## Configuration Reference
Precedence:
1. Environment variables (shell exports)
2. `~/.cognee/.env` — the one-time setup file, shared with the Claude Code plugin; loaded into the environment at process start, so every variable below except `COGNEE_ENV_FILE` itself can live in it
3. Defaults
The `COGNEE_BACKEND` / `COGNEE_CODEX_BACKEND` mode switch follows the same precedence; its effect is described in [Which mode wins](#which-mode-wins-and-how-to-switch).
<Note>
**There is no `config.json`.** Older plugin versions wrote `~/.cognee-plugin/config.json`, and `SessionStart` read a `base_url` from it while the per-turn hooks did not — so a stale URL there could point the two halves of the plugin at different servers. `SessionStart` now deletes a leftover file. Put everything in `~/.cognee/.env` instead.
</Note>
| Setting | Env var | Default | Notes |
| ------------------------------ | -------------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| Dataset | `COGNEE_PLUGIN_DATASET` | `agent_sessions` | Dataset for writes and recall at launch; the `cognee-switch-datasets` skill changes it mid-session |
| Session ID | `COGNEE_SESSION_ID` | derived from the Codex thread | Override to pin a named session, or share one across terminals |
| Session prefix | `COGNEE_SESSION_PREFIX` | `codex` | Prefix on auto-derived session ids (`<prefix>_<codex-thread-id>`) |
| Session strategy | `COGNEE_SESSION_STRATEGY` | `per-directory` | Currently inert — the plugin derives session ids from the Codex thread and never reads this |
| Base URL | `COGNEE_BASE_URL` | unset | Set to use a managed/remote endpoint |
| API key | `COGNEE_API_KEY` | unset | Auto-minted if absent in local mode |
| Mode switch | `COGNEE_BACKEND` | unset | `local` or `cloud` — pins the terminal's mode, overriding the URL rule; flips the Codex **and** Claude Code plugins |
| Mode switch (this plugin only) | `COGNEE_CODEX_BACKEND` | unset | Same, for this plugin only; beats `COGNEE_BACKEND` |
| Env file location | `COGNEE_ENV_FILE` | `~/.cognee/.env` | Read the one-time setup file from somewhere else |
| Local API URL | `COGNEE_LOCAL_API_URL` | `http://localhost:8011` | Local API base URL |
| Local LLM | `LLM_API_KEY`, `LLM_MODEL` | unset | Required for local mode |
| Code auto-indexing | `COGNEE_CODE_AUTOINDEX` | `auto` | `auto`, `always`, or `off` — see [Code graph](#code-graph) |
## Update or Remove
The `cognee` marketplace tracks the repository's `main` branch, so updates arrive as new commits and are not automatic. Pull the latest with:
```bash theme={null}
codex plugin marketplace upgrade cognee
```
If a stale cached copy persists, remove and re-add the plugin:
```bash theme={null}
codex plugin remove cognee@cognee
codex plugin add cognee@cognee
```
***
<CardGroup>
<Card title="GitHub Repository" icon="github" href="https://github.com/topoteretes/cognee-integrations/tree/main/integrations/codex">
View source code and the full configuration reference
</Card>
<Card title="Claude Code plugin" icon="bot" href="/integrations/claude-code-integration">
The same memory plugin for Claude Code
</Card>
</CardGroup>
# CrewAI
Source: https://docs.cognee.ai/integrations/crewai-integration
Add persistent memory to CrewAI agents with Cognee.
Give your [CrewAI](https://www.crewai.com/) agents a shared, persistent knowledge base powered by cognee. Agents store information with `add_tool` and retrieve it via natural language with `search_tool` — memory that survives across crews and runs.
## Why Use This Integration
* **Shared crew memory**: Multiple agents read and write the same knowledge graph
* **Semantic recall**: Retrieve information using natural language queries
* **Session isolation**: Multi-tenant support with per-user/session data separation
* **Drop-in tools**: Add `add_tool` and `search_tool` to any CrewAI agent
## Installation
```bash theme={null}
pip install cognee-integration-crewai
```
<Info>
Requires Python 3.10+. Pins `cognee>=0.4.0`, `crewai>=1.5.0`, and `crewai-tools>=1.5.0`.
</Info>
## Quick Start
Set your LLM key, then attach the tools to a CrewAI agent:
```bash theme={null}
export LLM_API_KEY="your-openai-api-key-here" # or OPENAI_API_KEY
```
```python theme={null}
import asyncio
from dotenv import load_dotenv
import cognee
from crewai import Agent
from cognee_integration_crewai import add_tool, search_tool
load_dotenv()
async def main():
# Optional: start from a clean knowledge base
await cognee.prune.prune_data()
await cognee.prune.prune_system(metadata=True)
agent = Agent(
role="Research Analyst",
goal="Find and analyze information using the knowledge base",
backstory="You are an expert analyst with access to a comprehensive knowledge base.",
tools=[add_tool, search_tool],
verbose=True,
)
# Store information
response = agent.kickoff(
"Remember that our company signed a contract with HealthBridge Systems "
"in the healthcare industry, starting Feb 2023, ending Jan 2026, worth £2.4M"
)
print(response.raw)
# Query the stored information
response = agent.kickoff("What contracts do we have in the healthcare industry?")
print(response.raw)
if __name__ == "__main__":
asyncio.run(main())
```
## Tools
| Tool | Signature | Description |
| ------------- | ----------------------------------------------------------- | ------------------------------------------------------------ |
| `add_tool` | `add_tool(data: str, node_set: Optional[List[str]] = None)` | Store information in the knowledge base for later retrieval |
| `search_tool` | `search_tool(query_text: str)` | Search and retrieve stored information with natural language |
<Info>
`add_tool` queues writes and runs `cognee.cognify()` after each batch, so data is indexed before searches. Async operations have a default 120-second timeout.
</Info>
## Session Management
Use `get_sessionized_cognee_tools(session_id)` to isolate data per user or session:
```python theme={null}
from cognee_integration_crewai import get_sessionized_cognee_tools
user1_add, user1_search = get_sessionized_cognee_tools("user-123")
user2_add, user2_search = get_sessionized_cognee_tools("user-456")
agent1 = Agent(
role="Assistant",
goal="Help user 1",
backstory="You are a helpful assistant.",
tools=[user1_add, user1_search],
)
```
<Info>
If you omit `session_id`, a UUID-based session ID is generated automatically. Session isolation is implemented by injecting the session ID into the `node_set` of `add_tool`.
</Info>
## How It Works
1. **Add Tool**: Stores data in cognee's knowledge graph with embeddings, then cognifies it
2. **Search Tool**: Retrieves relevant information via `cognee.search()`
3. **Auto-Processing**: cognee extracts entities, relationships, and context automatically
4. **Background loop**: cognee's async API runs on a dedicated background event loop with safe concurrent writes
***
<CardGroup>
<Card title="GitHub Repository" icon="github" href="https://github.com/topoteretes/cognee-integrations/tree/main/integrations/crewai">
View source code and examples
</Card>
<Card title="Examples" icon="book" href="https://github.com/topoteretes/cognee-integrations/tree/main/integrations/crewai/examples">
Runnable example scripts
</Card>
</CardGroup>
# Evaluation with DeepEval
Source: https://docs.cognee.ai/integrations/deepeval-integration
Evaluate Cognee retrieval pipelines with DeepEval.
## Why DeepEval?
[DeepEval](https://deepeval.com/) is an open-source evaluation framework that provides ready-made metrics (both traditional and LLM-as-a-judge) for LLM pipelines. Compared with hand-rolled evaluation scripts, DeepEval lets you:
* Track **Contextual Relevancy**, **Contextual Precision/Recall**, **Coverage** and more.
* Swap between automatic string-based metrics (EM/F1) and LLM-based scoring with a single flag.
* Re-use the same metrics across different projects and datasets.
> DeepEval stores no data – it simply runs metrics locally or via your preferred LLM. That makes it a perfect drop-in evaluator for Cognee’s pipelines.
## DeepEval inside Cognee
Cognee ships with a dedicated **`DeepEvalAdapter`**. When enabled, every answer produced by your pipeline is scored with the metrics you choose.
```python theme={null}
evaluating_answers: bool = True
evaluating_contexts: bool = True
evaluation_engine: str = "DeepEval" # Options: 'DeepEval', 'DirectLLM'
evaluation_metrics: list[str] = [
"correctness", # LLM-based correctness
"EM", # Exact-Match
"f1", # Token-level precision / recall
]
deepeval_model: str = "gpt-4o-mini" # Any OpenAI-compatible LLM
```
Behind the scenes the adapter:
1. Transforms Cognee’s `Answer` objects into DeepEval’s `LLMTestCase` format.
2. Runs the selected metrics.
3. Stores the raw scores alongside rationales so they appear in Cognee’s HTML dashboard.
## Quick Start
1. **Install Cognee** (DeepEval is declared in `pyproject.toml` so you automatically get the dependency).
2. Set your LLM API key so DeepEval can run LLM-based metrics:
```python theme={null}
import os
os.environ["LLM_API_KEY"] = "<YOUR_OPENAI_API_KEY>"
```
You can also export the variable in your shell (`export LLM_API_KEY=...`).
3\. (Optional) Configure the model DeepEval should call:
```bash theme={null}
export DEEPEVAL_MODEL=gpt-4o
```
4. Run a standard Cognee pipeline (add → cognify → search). The evaluation executor will automatically invoke DeepEval.
## Useful Links
* DeepEval integration guide – [deepeval.com » Cognee](https://deepeval.com/integrations/vector-databases/cognee)
* DeepEval docs – [deepeval.com/docs](https://deepeval.com/docs/getting-started)
***
Join the conversation on [Discord](https://discord.gg/m63hxKsp4p) and let us know how DeepEval works for you!
# Dify
Source: https://docs.cognee.ai/integrations/dify-integration
Add Cognee memory tools to your Dify apps and workflows.
Connect [Dify](https://dify.ai/) to cognee's AI memory engine with a marketplace tool plugin. Add data, build a knowledge graph with cognify, and run semantic search directly from your Dify apps and workflows — no code required.
There are two plugins depending on where your Cognee runs:
| Plugin | Backend | Auth |
| --------------------------------------- | -------------------------- | ---------------- |
| **Cognee** (`cognee`) | Cognee Cloud (hosted) | API key |
| **Cognee (Self-Hosted)** (`cognee-sdk`) | Your own Cognee OSS server | Email + password |
## Cognee (Cloud)
Targets the hosted [Cognee Cloud](/cognee-cloud/overview) API.
### Setup
1. Get your **Base URL** and **API key** from the [Cognee Cloud dashboard](https://platform.cognee.ai/).
2. Install the **Cognee** plugin from the Dify Marketplace.
3. Configure the plugin credentials:
| Field | Value |
| ---------- | -------------------------------------- |
| `base_url` | `https://tenant-xxx.aws.cognee.ai/api` |
| `api_key` | Your Cognee API key |
### Tools
| Tool | Description |
| ---------------- | ---------------------------------------------------------------------------- |
| Create Dataset | Create a new dataset (or return an existing one with the same name) |
| Add Data | Ingest text content into a dataset |
| Add File | Upload files (PDF, DOCX, TXT, …) into a dataset |
| Cognify | Process dataset(s) into a searchable knowledge graph |
| Search | Query the knowledge graph (14 search strategies, `top_k`, `only_context`, …) |
| Get Datasets | List all datasets for the authenticated user |
| Get Dataset Data | List all data items in a dataset |
| Delete Dataset | Permanently delete a dataset and its data |
| Delete Data | Delete a specific data item from a dataset |
## Cognee (Self-Hosted)
Targets a local or self-hosted Cognee OSS server. Tested with Cognee `v0.5.5`.
### Run a Cognee server
<CodeGroup>
```yaml docker-compose.yml theme={null}
services:
cognee:
image: cognee/cognee:0.5.5
container_name: cognee-local
ports:
- "8000:8000"
environment:
- HOST=0.0.0.0
- ENVIRONMENT=local
volumes:
- .env:/app/.env
```
```bash pip theme={null}
pip install cognee==0.5.5
LLM_API_KEY=sk-your-openai-key-here python -m cognee.api.client
```
</CodeGroup>
Provide your `LLM_API_KEY` (via `.env` for Docker) and start the server, then verify it:
```bash theme={null}
curl http://localhost:8000/health # should return HTTP 200
```
### Setup
In the Dify plugins page, find **Cognee (Self-Hosted)** and configure:
| Field | Value |
| --------------- | -------------------------- |
| `base_url` | `http://localhost:8000` |
| `user_email` | `default_user@example.com` |
| `user_password` | `default_password` |
The plugin validates the configuration with a health check and login.
<Info>
The plugin runs on your host (not inside Docker), so use `localhost`. If you run the plugin inside Docker too, use `host.docker.internal`. Change the default credentials before production use.
</Info>
### Tools
| Tool | Description |
| -------------- | ------------------------------------------------------------- |
| Add Data | Add text data to a dataset |
| Cognify | Build memory from one or more datasets |
| Search | Search the knowledge graph for relevant information |
| Update Data | Replace an existing data item and re-integrate it into memory |
| Delete Dataset | Delete a dataset and all its data |
| Delete Data | Delete a specific data item |
<Info>
The self-hosted plugin authenticates with email + password (not an API key), adds an **Update Data** tool, and does not include the cloud-only **Add File** / **Create Dataset** tools.
</Info>
## Typical Workflow
Both plugins follow the same pattern inside a Dify app or workflow:
1. **Add Data** (or **Add File**) to ingest content into a dataset
2. **Cognify** to build the knowledge graph
3. **Search** before LLM calls to pull relevant context from memory
4. **Delete Dataset** / **Delete Data** to clean up
## Trigger Cognee over HTTP (no plugin)
If you already run the conversational layer in Dify and just want to pull memory into a prompt with minimal setup, skip the marketplace plugin and call Cognee's REST API directly from a Dify **HTTP Request** node. This works against any self-hosted Cognee OSS server — all routes live under `/api/v1/*`. See the [HTTP API reference](/api-reference/introduction) and the [Deploy a REST API server](/guides/deploy-rest-api-server) guide.
### 1. Get a token
Unless the server runs with auth off (`ENABLE_BACKEND_ACCESS_CONTROL=false`), first exchange credentials for a JWT. `POST /api/v1/auth/login` expects form-encoded fields:
```http theme={null}
POST http://localhost:8000/api/v1/auth/login
Content-Type: application/x-www-form-urlencoded
username=default_user@example.com&password=default_password
```
The response is `{ "access_token": "...", "token_type": "bearer" }`. Store `access_token` in a Dify variable and send it as `Authorization: Bearer {{access_token}}` on every later call. Tokens expire after `JWT_LIFETIME_SECONDS` (default `3600`).
### 2. Search from your workflow
Add an HTTP Request node that queries memory before your LLM node, passing the user's message as the `query`:
```http theme={null}
POST http://localhost:8000/api/v1/search
Authorization: Bearer {{access_token}}
Content-Type: application/json
{
"search_type": "GRAPH_COMPLETION",
"datasets": ["my-dataset"],
"query": "{{sys.query}}",
"top_k": 10
}
```
Set `"only_context": true` to get back just the retrieved context (instead of a Cognee-generated answer) and feed it into your own prompt. See [search basics](/guides/search-basics) for the available `search_type` values.
<Info>
To ingest and process data over HTTP too, use `POST /api/v1/add` (multipart form: `data` files + `datasetName`) followed by `POST /api/v1/cognify` (JSON: `{ "datasets": ["my-dataset"] }`). Point the URL at whichever backend you deploy — `http://localhost:8000` for a local server, or your hosted host name.
</Info>
***
<CardGroup>
<Card title="Cloud plugin source" icon="github" href="https://github.com/topoteretes/cognee-integrations/tree/main/integrations/dify">
View the Cognee Cloud Dify plugin
</Card>
<Card title="Self-hosted plugin source" icon="github" href="https://github.com/topoteretes/cognee-integrations/tree/main/integrations/dify-sdk">
View the self-hosted Dify plugin
</Card>
</CardGroup>
# dlt (Data Load Tool)
Source: https://docs.cognee.ai/integrations/dlt-integration
Ingest structured data into Cognee with dlt.
Ingest structured relational data — databases, CSV files, and [dlt](https://dlthub.com/) resources — directly into cognee's knowledge graph. Foreign keys become graph edges, tables become schema nodes, and each row becomes a searchable document, all built deterministically from the schema without LLM extraction.
## Why Use This Integration
* **Schema-Aware Graphs**: Foreign key relationships are preserved as first-class edges in the knowledge graph
* **Deterministic Graph Construction**: Structured data bypasses LLM entity extraction — no hallucination risk
* **Mixed Ingestion**: Combine structured (dlt) and unstructured (text, PDF) data in the same dataset
* **Multiple Input Modes**: Pass explicit dlt resources, CSV file paths, or database connection strings
* **Write Dispositions**: Control how data is synced — merge (upsert), append, or replace
## Installation
```bash theme={null}
pip install 'cognee[dlt]'
```
Or with uv:
```bash theme={null}
uv pip install 'cognee[dlt]'
```
<Note>
The official `cognee/cognee` Docker image ships the `dlt` extra by default, so a containerised deployment needs no extra install — CSVs added there follow the `dlt_csv_loader` path described under [CSV Files](#csv-files). See [Optional Extras and Document Loaders](/how-to-guides/cognee-sdk/deployment/docker).
</Note>
## Quick Start
### 1. Ingest a dlt Resource
Define a dlt resource and pass it to `cognee.remember()`. The dlt-specific structured-ingestion options `primary_key`, `write_disposition`, SQL `query`, and `max_rows_per_table` are accepted by `cognee.remember()` and forwarded to the underlying ingestion step. After ingestion, use `cognee.recall(...)` to query the graph.
```python theme={null}
import dlt
import cognee
import asyncio
@dlt.resource()
def users_and_pets():
yield [
{
"id": 1,
"name": "Alice",
"pets": [
{"id": 1, "name": "Fluffy", "type": "cat"},
{"id": 2, "name": "Spot", "type": "dog"},
],
},
{
"id": 2,
"name": "Bob",
"pets": [{"id": 3, "name": "Fido", "type": "dog"}],
},
]
async def main():
await cognee.remember(
users_and_pets,
dataset_name="users_and_pets",
primary_key="id",
)
results = await cognee.recall(
query_text="Which pet does Alice have?",
datasets=["users_and_pets"],
)
print(results)
asyncio.run(main())
```
dlt automatically detects nested structures (like `pets` inside each user) and creates separate tables with foreign key relationships.
<Note>
The lower-level `cognee.add(...)` + `cognee.cognify(...)` pair still accepts the same dlt kwargs and remains useful when you need to run ingestion and graph building as separate steps. For the runnable end-to-end version of this walkthrough, see [`examples/demos/ingestion_and_migration/dlt_ingestion_example.py`](https://github.com/topoteretes/cognee/blob/dev/examples/demos/ingestion_and_migration/dlt_ingestion_example.py).
</Note>
### 2. Build and Query the Graph
Once `remember()` finishes ingesting and building the graph, use `cognee.recall(...)` to query it.
## Other Input Modes
### CSV Files
CSV files are handled by the **`dlt_csv_loader`**, which the [loader engine](/core-concepts/further-concepts/loaders) registers above the plain-text `csv_loader` whenever the `dlt` extra is installed. Selection happens inside the ingest pipeline rather than before it, so the same route applies to every way a CSV can arrive — a local path, a `file://` or `s3://` location, or an uploaded file:
```python theme={null}
await cognee.remember(
"/path/to/employees.csv",
dataset_name="employees",
)
```
Each CSV is staged through dlt and becomes **one manifest record per original file**. The manifest's identity is derived from the *original* file name — not the temporary copy an `s3://` download or an upload lands in — so it stays stable across runs and a re-add updates the existing record instead of creating a second one. As with any dlt source, the rows skip chunking and LLM entity extraction.
#### Delimiter detection
The delimiter is detected per file, so semicolon-, tab-, and pipe-separated exports load as proper tables instead of collapsing into a single column — or, when a text field contains commas of its own, failing outright on a ragged split. No configuration is required and there is no knob to set.
Cognee reads up to the first 20 lines of the file, ignoring blank ones, and parses them with each of `,`, `;`, `\t`, `|` in that order, picking the first delimiter that splits every sampled line into the *same* number of columns, with more than one column. Parsing (rather than counting characters) means quoted fields are respected, so a comma-heavy text column — prose, an embedded JSON list — no longer forces a semicolon-delimited file onto commas. Candidates are tried in order, so a comma wins ties, and `,` is also the fallback when the file is empty or no candidate produces a consistent column count.
<Note>
If a genuinely ambiguous file — one where a second candidate also yields a consistent column count — loads with the wrong delimiter, convert it to comma-separated before ingesting, or request the plain-text `csv_loader` for that call (below).
</Note>
Per-call dlt options for CSVs travel through the loader-config channel rather than as `remember()` keyword arguments. `dlt_csv_loader` accepts `primary_key`, `write_disposition`, `max_rows_per_table`, and `column_value_columns`:
```python theme={null}
await cognee.remember(
"/path/to/employees.csv",
dataset_name="employees",
preferred_loaders=[
{"dlt_csv_loader": {"primary_key": "id", "write_disposition": "merge"}}
],
)
```
To flatten a CSV into plain text instead — the behavior you get automatically on installs without the `dlt` extra — request `csv_loader` explicitly for that call:
```python theme={null}
await cognee.remember(
"/path/to/employees.csv",
dataset_name="employees",
preferred_loaders=[{"csv_loader": {}}],
)
```
<Note>
A CSV that yields no rows now fails loudly with an `IngestionError` naming the source, rather than being ingested as an empty record. `dlt_csv_loader` likewise requires the dataset and user context that the ingest pipeline supplies, so it cannot be invoked outside of `remember()` / `add()`.
</Note>
### Database Connection String
Ingest tables directly from an existing database:
```python theme={null}
await cognee.remember(
"postgresql://user:pass@host/db",
dataset_name="company_db",
primary_key="id",
)
```
<Note>
The connection string is a **source cognee reads from**, never a destination it writes to — ingested rows land in cognee's own configured stores, including vector embeddings of each row, so they're searchable by semantic similarity as well as graph traversal. To change where cognee stores memory, configure the [graph](/setup-configuration/graph-stores), [vector](/setup-configuration/vector-stores), and [relational](/setup-configuration/relational-databases) providers.
</Note>
Supported databases via auto-detection: SQLite, PostgreSQL, MySQL, MSSQL, Oracle. Hosted Postgres providers such as Neon work with their standard `postgresql://` connection strings; keep provider-required SSL parameters such as `?sslmode=require`. Amazon Redshift is also compatible since it speaks the PostgreSQL wire protocol — use a standard `postgresql://` connection string pointing to your Redshift endpoint.
For Snowflake and Google BigQuery, construct a dlt source directly and pass it to `cognee.remember()` (see the [Cloud Data Warehouses](#cloud-data-warehouses) accordion below).
You can optionally filter with a SQL WHERE clause:
```python theme={null}
await cognee.remember(
"postgresql://user:pass@host/db",
dataset_name="engineering_team",
primary_key="id",
query="SELECT * FROM employees WHERE department = 'Engineering'",
)
```
The `FROM` target may also be schema-qualified. Cognee splits `schema.table` before building the source — the schema is handed to dlt and the bare table name is what the `WHERE` clause is matched against — so the filter applies to the qualified table just as it does to a bare one:
```python theme={null}
await cognee.remember(
"postgresql://user:pass@host/db",
dataset_name="engineering_team",
primary_key="id",
query="SELECT * FROM public.employees WHERE department = 'Engineering'",
)
```
The `FROM` target may also carry a table alias, with or without `AS`: `query="SELECT * FROM employees e WHERE department = 'Engineering'"` filters exactly like the unaliased form above; previously the alias broke the parse and the whole table was loaded unfiltered. The `WHERE` clause is applied as written as long as it does not reference the alias — leave columns unqualified, or qualify them with the table name.
Two query shapes cannot be replayed against the single-table select cognee hands to dlt, so they raise a `ValueError` instead of loading the table unfiltered:
* **A `WHERE` that references the table alias** — `SELECT * FROM employees e WHERE e.department = 'Engineering'`. The filter is replayed against the bare table, where the alias does not exist. Rewrite the columns bare (`department = 'Engineering'`) or qualify them with the table name (`employees.department = 'Engineering'`). The check is a textual scan for `<alias>.<column>` across the whole clause, so a string literal that happens to match trips it too — under the alias `f`, `WHERE path = 'f.txt'` raises even though no column is qualified. A longer alias (`emp` rather than `e`) avoids the collision.
* **A `JOIN` in the `FROM` clause** — `SELECT o.id FROM orders o JOIN customers c ON o.customer_id = c.id WHERE c.active`. The source targets a single table, so a join across tables has no representation. Create a database view over the joined result and ingest that view, or ingest the table without a filter. Only the `JOIN` keyword is recognized: a comma-style join (`SELECT * FROM orders o, customers c WHERE c.active`) still parses as a plain select on the first table with its `WHERE` dropped, loading that table unfiltered — write joins with `JOIN` so the error fires, or ingest a view.
<Note>
`query` accepts a `SELECT` with an unquoted `FROM` target — either a bare table name (`employees`) or a dot-qualified one (`public.employees`) — an optional table alias (`employees e` / `employees AS e`), and an optional `WHERE` clause. Quoted identifiers such as `"public"."employees"` are not parsed and raise a `ValueError`. Omitting `WHERE` loads the whole table.
</Note>
### Mixed Structured + Unstructured
Combine dlt resources with unstructured text in a single dataset:
```python theme={null}
text = """Alice has two pets: a cat named Fluffy and a dog named Spot.
Bob has a dog named Fido, who is friendly with both Fluffy and Spot."""
await cognee.remember(
[text, users_and_pets],
dataset_name="users_and_pets_with_text",
primary_key="id",
)
```
<Info>
Structured data creates deterministic graph nodes from the schema, while unstructured text goes through LLM-based entity extraction. Both are combined in the same knowledge graph.
</Info>
## Write Dispositions
Control how data is synced on repeated runs using the `write_disposition` parameter:
* **`replace`** (default): Drop and recreate tables on each run. Use for full snapshot refreshes.
* **`merge`**: Upsert by primary key — updates existing rows, inserts new ones. Best for data that changes over time.
* **`append`**: Always insert without deduplication. Use for time-series data and event logs.
```python theme={null}
# Append mode — every call adds new rows, no dedup
await cognee.remember(
event_resource,
dataset_name="events",
primary_key="id",
write_disposition="append",
)
```
<Note>
`write_disposition` only controls dlt's staging snapshot. It does not by itself make a re-run pick up rows a source has gained, because the ingestion skip in `add()` short-circuits an already-ingested source first — see [Re-Ingesting a Source](#re-ingesting-a-source) and [Re-ingesting a source that keeps growing](/core-concepts/main-operations/legacy-operations/cognify#examples-and-details).
</Note>
## How It Works
1. **Source Detection**: cognee identifies dlt resources and connection strings in the input. `.csv` files are not detected here — they are routed by the loader engine's `dlt_csv_loader` inside the ingest pipeline (see [CSV Files](#csv-files)) and join the flow from step 2 onward
2. **Pipeline Execution**: A dlt pipeline loads data into a per-dataset staging database
3. **Schema Extraction**: Table schemas, primary keys, and foreign keys are extracted
4. **Graph Construction**: Each row becomes a document node; foreign keys become edges between nodes
5. **LLM Bypass**: Structured rows skip chunking, entity extraction, and summarization — the graph is built entirely from schema metadata
<Info>
The `primary_key` parameter controls upsert behavior when you use `write_disposition="merge"`. If not specified, cognee auto-detects from an `id` column or falls back to the first column.
</Info>
## Bounding Ingestion
Two opt-in caps guard against large sources; both default to `0`, meaning no cap.
* **Rows per table**: pass the `max_rows_per_table` kwarg to `remember()` / `add()` to bound the per-table row count for a single call, or set the `DLT_MAX_ROWS_PER_TABLE` environment variable to change the process-wide default.
* **Column-value length**: `DLT_MAX_COLUMN_VALUE_LENGTH` bounds the length of the cell values that become shared `ColumnValue` nodes — cell-level graph nodes that link rows sharing the same value in a selected column. (Columns are selected with the `column_value_columns` kwarg — `add()` only — or `DLT_COLUMN_VALUE_COLUMNS`; nothing is selected by default.) A positive value **skips** selected cells longer than that many characters, dropping rather than truncating them — worthwhile for free-text-heavy columns or wildcard (`"*"`) selection, since long one-off values make poor shared nodes and each unique value costs one embedding. Unlike `max_rows_per_table`, this cap has no per-call kwarg: set the environment variable or the `dlt_max_column_value_length` ingestion config field. Cognee previously applied a fixed 256-character cap unconditionally; set `DLT_MAX_COLUMN_VALUE_LENGTH=256` to keep that behavior.
## Re-Ingesting a Source
Each relational dlt source is stored as a **single record** whose identity is stable: it is derived from the dataset name and the source name (`dlt_source:{dataset_name}:{source_name}`) and does not depend on the data. Re-running `remember()` / `add()` on the same source therefore never creates a second copy — but what happens to the existing record depends on how you call it:
* **Plain re-add (the default)**: `add()` is idempotent. A source that has already been ingested keeps its record, content hash, and cognify status as-is — **whether or not the upstream data changed** — and nothing is reprocessed.
* **Explicit re-ingest**: to pick up upstream changes, call `add(..., incremental_loading=False, data_cache=False)` — the completed-skip runs whenever either flag is on, so both must be off — or use `update()` with the record's UUID. The record then updates in place under the same identity (a content hash over the source's tables and rows tracks the change), so the source is never absent from the store, and the next `cognify()` **purges the source's previously derived artifacts** from the graph and vector stores before re-emitting the current rows. Rows deleted upstream disappear, and changed rows do not keep their stale values alongside the new ones. Between the re-ingesting `add()` and `cognify()`, searches still return the source's previous rows; they are replaced only once the re-cognify completes.
<Warning>
The purge is a real delete, re-authorized as one: re-ingesting a **changed** dlt source requires `delete` permission on the dataset. If that permission is missing, the run fails instead of continuing — silently skipping the purge would leave stale rows in the graph and present them as current.
</Warning>
The purge also reaches the session cache: cached session turns whose answers used the purged graph elements are removed, along with the feedback and session-context lessons distilled from them, so a re-ingest cannot replay the source's old rows out of session memory. This part is best-effort — a session-cache error is logged as a warning and never fails the run. See [Invalidation when the underlying data is deleted](/core-concepts/sessions-and-caching#additional-information) for how the targeted pass decides what is contaminated.
### Renaming a Source
Because the identity is keyed on the dataset and source names, renaming either one is a **remove + add**, not an in-place rename. The new name ingests from scratch as a fresh source, and the records under the old name stay in the dataset — a re-ingest only reconciles the source names it just ingested, so it will not treat the old name's records as orphans. Delete them explicitly (for example with `cognee.forget(...)`) if you do not want both.
For the same reason, two dlt sources that share a name within a single `add()` call would resolve to the same identity. cognee raises an error rather than letting one silently overwrite the other, so give each source a distinct name.
CSVs are keyed the same way, but through `dlt_csv_loader` rather than that check: a CSV's source name is its file-name stem, with runs of characters other than letters, digits, and underscores replaced by `_`, leading and trailing separators stripped, and the result lowercased. Two CSVs whose stems normalize to the same string (for example `Employees 2024.csv` and `employees_2024.csv`) therefore share one manifest record within a dataset, with the later ingest updating the earlier one. Give CSV files names that stay distinct under that normalization.
## Foreign Key Resolution
A foreign key becomes a graph edge only when **both** the source row and the target row are loaded in the same ingestion run. Two edge cases are worth knowing about — cognee now logs a warning in each so they are diagnosable rather than silent:
* **Target row not loaded**: if a foreign key points at a row that wasn't ingested — most commonly because the target table hit a `max_rows_per_table` cap you set — the reference is dropped and no edge is created. The warning identifies the dropped references as `source_table.column -> ref_table:value`. If you see missing edges, raise `max_rows_per_table` so the referenced rows are included.
* **Duplicate primary keys within a table**: if multiple rows in a table share the same primary key, foreign key edges that target that key resolve to the **last** such row loaded; earlier rows with the same key are shadowed for FK targeting. The warning names the affected `table` and `pk`.
## Connectors
Connectors are dlt sources for a specific system. The list below keeps the current connector packages visible; the routing details are tucked away for reference.
<AccordionGroup>
<Accordion title="Connector Modes">
Cognee supports two DLT connector modes:
* **Relational connectors** take the default dlt path described above: each row becomes a schema-context document and foreign keys become edges, all built deterministically without LLM extraction.
* **Document-mode connectors** opt each row into normal cognify instead: the row is turned into a text document that goes through LLM entity extraction, just like unstructured text passed to `remember()`.
A dlt source opts into document mode by setting the `cognee_document_source` attribute (via the `document_source_tag()` helper in `cognee.tasks.ingestion.dlt_utils`) to a short source tag. cognee then routes every row from that source through cognify rather than the relational schema-context path:
* Each row is built from its `title` and `content` columns (rendered as `# {title}\n\n{content}`, or just the content when there is no title), with optional `url` and `id` columns preserved in metadata.
* `external_metadata["source"]` is set to the connector's own tag (for example `"notion"`) instead of `"dlt"`, alongside `title` and, when present, `url` and `external_id` (from the row's `id`).
Because the tag is connector-provided, the shared ingestion engine stays connector-agnostic: a connector declares its own nature rather than being hard-coded by name.
**Sync and orphan cleanup.** Document sources read back their full current snapshot (`max_rows_per_table=0`) and honor the `write_disposition` you pass: use `replace` for snapshot sources with no delete feed and `merge` with a hard-delete tombstone column for incremental sources that emit real deletions. `primary_key` defaults to `id`. Orphan cleanup is scoped to the source tag, so reconciling a document source only removes that source's rows, and relational (`"dlt"`) rows and other sources' rows are never cross-deleted in a mixed dataset. Cleanup is skipped when the fresh read-back is empty, so an empty snapshot is treated as a failed sync rather than a signal to delete everything. Deleting an orphan also invalidates the cached session turns that used that row's graph elements, so a row removed upstream stops surfacing from session memory too; that step is best-effort and logged as a warning if it fails.
<Note>
Orphan cleanup now runs in the foreground of `add()` / `remember()`: blocking runs execute it synchronously after the fresh rows are committed, so upstream deletions are reflected within the same call. Background runs (`run_in_background=True`) perform it up front instead.
</Note>
</Accordion>
<Accordion title="Gmail connector">
## Gmail Connector
The Gmail connector is a first-class dlt source that turns your inbox into cognee memory. It reuses the same `remember()` + dlt path described above, so it gets incremental re-sync and forget-on-delete for free. `gmail_source()` returns a dlt resource that you hand directly to `cognee.remember()`.
<Warning>
This connector reads the **content** of your email. It is strictly opt-in — nothing is fetched until you construct a source and call `remember()`. Scope what you ingest with `label_ids` / `query`, keep the OAuth token file (`token.json`) private, and prefer a dedicated dataset so you can wipe it with a single `cognee.forget(...)`.
</Warning>
<Note>
The Gmail connector ships as the standalone community package **`cognee-community-connector-gmail`**, maintained in the [cognee-community](https://github.com/topoteretes/cognee-community) repository, so core stays free of the Google client SDKs. Install it with `pip install cognee-community-connector-gmail`, then import `gmail_source` from `cognee_community_connector_gmail` as shown below.
</Note>
### Installation
```bash theme={null}
pip install cognee-community-connector-gmail
```
Or with uv:
```bash theme={null}
uv pip install cognee-community-connector-gmail
```
The standalone package pulls in `dlt[sqlalchemy]`, `google-api-python-client`, `google-auth`, and `google-auth-oauthlib`. The Google client libraries are imported lazily, so the core cognee install stays slim.
### One-Time OAuth Setup
The connector authenticates with Gmail via the OAuth2 **installed-app** (Desktop app) flow using the read-only scope `https://www.googleapis.com/auth/gmail.readonly` — it never modifies your mailbox.
1. In the [Google Cloud Console](https://console.cloud.google.com/), enable the **Gmail API**, configure an OAuth consent screen (add yourself as a test user), and create an **OAuth 2.0 Client ID** of type **Desktop app**.
2. Download the client-secret JSON and save it as `credentials.json` (or point `credentials_path` at it).
3. The first run opens a browser to consent and caches the resulting user token at `token.json` (`token_path`). Later runs reuse and silently refresh that token.
### Usage
```python theme={null}
import cognee
from cognee_community_connector_gmail import gmail_source
source = gmail_source(
credentials_path="credentials.json",
token_path="token.json",
label_ids=["INBOX"],
)
await cognee.remember(
source,
dataset_name="gmail_inbox",
primary_key="id",
write_disposition="merge", # incremental upsert by message id
max_rows_per_table=0, # 0 = no row cap (see note below)
)
```
`gmail_source()` accepts these keyword-only parameters:
| Parameter | Default | Description |
| -------------------- | -------------------- | ---------------------------------------------------------------------------------------------- |
| `credentials_path` | `"credentials.json"` | Path to the OAuth client-secret JSON (Desktop app). |
| `token_path` | `"token.json"` | Where the cached user token is read/written. |
| `label_ids` | `None` | Restrict to these Gmail label ids (e.g. `["INBOX"]`). |
| `query` | `None` | Gmail search query (e.g. `"from:boss@x.com newer_than:30d"`). |
| `include_spam_trash` | `False` | Include SPAM/TRASH in the backfill listing. |
| `max_results` | `None` | Cap the number of messages pulled in a full backfill (handy for demos/tests). `None` = no cap. |
The returned resource (`gmail_messages`) is preconfigured with `primary_key="id"`, `write_disposition="merge"`, and an `_deleted` hard-delete column, so combined with `primary_key="id"` on `remember()` it performs idempotent upserts by Gmail message id.
<Note>
cognee's dlt ingestion reads at most `max_rows_per_table` rows from the dlt destination, and the default is `0` — no cap. For a real inbox, keep it unlimited so forget-on-delete compares against the **whole** synced corpus rather than a truncated window.
</Note>
### How It Works
* **Incremental sync**: The first run does a full (label-scoped) backfill and records the mailbox `historyId`. This cursor is persisted in dlt's per-resource state, so re-running `remember()` on the same dataset resumes where it left off — subsequent runs call `users.history.list(startHistoryId=...)` and emit only the delta (added / changed / deleted messages).
* **Forget-on-delete**: Messages reported as deleted or trashed by the History API are emitted with the `_deleted` hard-delete marker. dlt removes those rows from its destination on `merge`, and cognee's existing `orphan_cleanup` then purges them from the graph, vector, and relational stores.
* **History expiry**: Gmail expires history after roughly a week. If the stored `historyId` is too old, the History API returns a 404; the connector detects this and falls back to a full backfill so memory re-syncs rather than silently stalling.
For a runnable end-to-end walkthrough that demonstrates the initial backfill followed by an incremental sync, see the [`cognee-community-connector-gmail`](https://github.com/topoteretes/cognee-community) package in the cognee-community repository.
</Accordion>
</AccordionGroup>
## Use Cases
<AccordionGroup>
<Accordion title="CRM and Relational Data">
Load customer, order, and product tables from a database. Foreign keys between tables (e.g., `order.customer_id → customer.id`) become graph edges, enabling cross-table queries like "Which customers ordered product X?"
</Accordion>
<Accordion title="CSV Analytics Pipeline">
Point cognee at CSV exports from analytics tools. Each row becomes a searchable node in the graph, and you can combine them with unstructured reports in the same dataset.
</Accordion>
<Accordion title="Event Log Ingestion">
Use `write_disposition="append"` to stream event batches into cognee without deduplication. Query across the full event history with natural language.
</Accordion>
<Accordion title="Database Mirroring">
Use `write_disposition="merge"` to keep cognee's graph in sync with a live database. Rows that are removed upstream are cleaned up best-effort; any orphaned rows that fail to delete are logged and retried on the next ingest.
</Accordion>
<Accordion title="Cloud Data Warehouses (Snowflake, Redshift, BigQuery)">
**Amazon Redshift** speaks the PostgreSQL wire protocol, so the standard connection string auto-detection works:
```python theme={null}
await cognee.remember(
"postgresql://user:pass@my-cluster.us-east-1.redshift.amazonaws.com:5439/mydb",
dataset_name="redshift_data",
primary_key="id",
)
```
**Snowflake** requires constructing a dlt `sql_database` source manually (install `snowflake-sqlalchemy` first):
```bash theme={null}
pip install 'cognee[dlt]' snowflake-sqlalchemy
```
```python theme={null}
from dlt.sources.sql_database import sql_database
import cognee
source = sql_database(
credentials="snowflake://user:password@account_identifier/database/schema?warehouse=MY_WH",
table_names=["orders", "customers"],
)
await cognee.remember(source, dataset_name="snowflake_data", primary_key="id")
```
The `account_identifier` is the part before `.snowflakecomputing.com` in your Snowflake URL (e.g. `myorg-myaccount`). Omit `table_names` to ingest all tables in the schema.
**Google BigQuery** works the same way using dlt's BigQuery connector — construct the source and pass it directly to `cognee.remember()`. See the [dlt sql\_database docs](https://dlthub.com/docs/dlt-ecosystem/verified-sources/sql_database) for connector-specific setup.
</Accordion>
</AccordionGroup>
<CardGroup>
<Card title="Remember Operation" icon="brain" href="/core-concepts/main-operations/remember">
Learn more about data ingestion in cognee
</Card>
<Card title="dlt Documentation" icon="book" href="https://dlthub.com/docs">
Official dlt documentation and guides
</Card>
</CardGroup>
# Docker Sandboxes
Source: https://docs.cognee.ai/integrations/docker-sandboxes-kit
Give any sandboxed coding agent persistent memory with the cognee-memory Docker Sandboxes kit.
The **cognee-memory** kit is a [Docker Sandboxes](https://docs.docker.com/ai/sandboxes/customize/kits/) mixin that gives any sandboxed coding agent persistent, self-improving memory backed by cognee. Everything runs embedded inside the sandbox — no external services, no database to provision.
```bash theme={null}
sbx run claude --kit ./cognee-memory
```
The kit lives in the cognee repository at [`examples/integrations/docker-sandbox-kit`](https://github.com/topoteretes/cognee/tree/main/examples/integrations/docker-sandbox-kit). Clone it (or copy the `cognee-memory/` directory) and point `--kit` at it.
<Info>
Verified end-to-end with **sbx v0.39.0** under a `deny-all` network policy. The kit does not declare a minimum `sbx` version.
</Info>
## What the kit does
`cognee-memory/spec.yaml` is a stackable mixin kit (`schemaVersion: "2"`, `kind: mixin`) that:
* installs the CLI with `uv tool install cognee` at sandbox creation — this assumes `uv` is present in the base image, which Docker's default sandbox images ship;
* pins all memory state to `/home/agent/.cognee` (`DATA_ROOT_DIRECTORY=/home/agent/.cognee/data`, `SYSTEM_ROOT_DIRECTORY=/home/agent/.cognee/system`), so memory survives sandbox restarts and is easy to inspect or back up;
* defaults to `LLM_MODEL=openai/gpt-5-mini` and sets `TELEMETRY_DISABLED=1`;
* pins `ENABLE_BACKEND_ACCESS_CONTROL=true` — cognee's default, made explicit so the kit's multi-agent behavior is unambiguous: multi-tenant ACLs and per-user+dataset database isolation;
* declares a proxy-managed OpenAI credential so the real key never enters the sandbox VM;
* allowlists only the domains cognee actually needs under `deny-all`;
* appends usage instructions to the agent's memory file (`kits-memory/cognee-memory.md`): recall at task start → work → remember durable learnings, plus the multi-agent handover pattern.
Because it is a mixin, it stacks onto any agent sandbox — `claude`, `shell`, `opencode`, and so on.
## Prerequisites
Install and start `sbx`, then set the baseline policy and the OpenAI secret. The install line below is macOS/Homebrew; use whatever install path `sbx` documents for your platform.
```bash theme={null}
brew trust docker/tap && brew install docker/tap/sbx
sbx daemon start # own terminal, or: nohup sbx daemon start &
sbx login # browser OAuth
sbx policy init deny-all # strictest baseline; the kit's allowlist is the only egress
sbx secret set-custom --host api.openai.com --env LLM_API_KEY --value "$LLM_API_KEY"
```
`set-custom` prints a placeholder (`sbx-cs-…`, retrievable later with `sbx secret ls`). Sandboxes only ever see the placeholder; the proxy substitutes the real key on requests to `api.openai.com`.
### The `cognee-openai` credential
The kit declares its credential service as **`cognee-openai`**, not `openai`:
```yaml theme={null}
credentials:
- service: cognee-openai
description: OpenAI API key used by cognee for entity extraction and embeddings
required: false
apiKey:
name: LLM_API_KEY
proxyManaged: true
inject:
- domain: api.openai.com
scheme: bearer
```
The rename is deliberate: built-in agent kits (`shell`, `claude`, …) already declare the common LLM services, and composition fails when two kits define the same service. `required: false` means sandbox creation is never blocked when no key is bound.
There are two ways to supply the key, both proxy-side:
1. **Bind the service** — interactively at run time, or via `~/.config/sbx/credentials.yaml`. The agent sees `LLM_API_KEY=proxy-managed` and the `inject` rule rewrites the `Authorization` header for `api.openai.com`.
2. **Headless** — `sbx secret set-custom --host api.openai.com --env LLM_API_KEY --value <key>`, as in the prerequisites above.
<Warning>
The kit's `proxy-managed` environment value wins over the custom secret's placeholder. For headless `sbx exec` commands, export the printed placeholder explicitly:
```bash theme={null}
sbx exec <sandbox> -- sh -lc 'export LLM_API_KEY=<placeholder> LOG_LEVEL=ERROR; \
cognee-cli remember "fact worth keeping"'
```
</Warning>
### Network allowlist
Under `deny-all`, the kit's allowlist is the sandbox's only egress. Each domain was discovered by running under the deny-all policy and reading `sbx policy log` — the recommended way to derive a kit allowlist:
| Domain | Why cognee needs it |
| ------------------------------------ | ------------------------------------------------------------------------ |
| `api.openai.com` | LLM and embedding calls |
| `pypi.org`, `files.pythonhosted.org` | `uv tool install cognee` |
| `extension.ladybugdb.com` | Ladybug, cognee's embedded graph DB, fetches its extensions on first use |
| `raw.githubusercontent.com` | litellm's model-cost map |
## Single-agent usage
```bash theme={null}
sbx run claude --kit ./cognee-memory # agent with persistent memory
sbx run shell --kit ./cognee-memory # or a plain shell sandbox
```
Inside the sandbox the agent has the full [memory CLI](/cognee-cli/overview):
| Command | Purpose |
| --------------------------------------------------- | ------------------------------------------ |
| `cognee-cli remember "text, a file path, or a URL"` | Store knowledge |
| `cognee-cli recall "your question"` | Query memory |
| `cognee-cli improve` | Enrich/index the graph |
| `cognee-cli forget --all` | Delete everything (no confirmation prompt) |
The first `remember` builds a knowledge graph with a few LLM calls, so it takes noticeably longer than a plain key-value write; `recall` then answers from the graph.
To use a provider other than OpenAI, edit `environment.variables`, the `credentials` and `permissions.network` blocks, and the stored secret to match — see [LLM providers](/setup-configuration/llm-providers).
## Multi-agent demo: supervisor to worker handover
[`demo/handover.sh`](https://github.com/topoteretes/cognee/blob/main/examples/integrations/docker-sandbox-kit/demo/handover.sh) runs a round-trip memory handover between **two real sandboxes**, `cognee-supervisor` and `cognee-worker`, both created from this kit and sharing the `demo/` directory as their workspace. The supervisor and worker are separate cognee **users**, so ACLs gate what each one can read and write.
```bash theme={null}
export LLM_API_KEY=sk-... # only needed the first time, for the secret
./demo/handover.sh
sbx policy log # the audit trail: per-domain allow/deny
```
The three phases run sequentially:
<Steps>
<Step title="brief — supervisor sandbox">
Stores a private note (dataset `supervisor_private`) and a handover briefing (dataset `handover`) in its own datasets, grants the worker `read` and `write` on the briefing, and writes a handover token to `demo/handover-out/handover_token.json` carrying the briefing's dataset **UUID**.
```python theme={null}
for permission in ("read", "write"):
await authorized_give_permission_on_datasets(
principal_id=worker.id,
dataset_ids=[handover_id],
permission_name=permission,
owner_id=supervisor.id,
)
```
The creator automatically holds `share`, so no extra grant is needed to hand access over.
</Step>
<Step title="work — worker sandbox">
Redeems the token by UUID, proves the two boundaries, then writes its completion report back into the shared dataset:
```python theme={null}
await cognee.recall("What is my task and how do I deploy?",
dataset_ids=[dataset_id], user=worker)
...
await cognee.remember(WORKER_REPORT, dataset_id=dataset_id, user=worker)
```
Negative checks: recalling the supervisor's private dataset raises `PermissionDeniedError`, and recalling the shared dataset **by name** (`datasets=["handover"]`) fails to resolve.
</Step>
<Step title="review — supervisor sandbox">
Recalls the worker's report from the shared dataset.
</Step>
</Steps>
<Note>
**Share by UUID, never by name.** Dataset names are namespaced per user — a name maps to `uuid5(name + user.id + tenant_id)` — so a name never crosses a user boundary. The dataset UUID is the only cross-user address, for reads and for cross-owner writes alike.
</Note>
Permission management is Python-SDK/REST-only: `cognee-cli` has no user or permission commands, so the multi-agent pattern requires the SDK (`create_user`, `get_datasets`, `authorized_give_permission_on_datasets`). On a fresh install, call `create_db_and_tables()` before touching users or ACLs — the CLI does this implicitly, the raw SDK path does not.
The payload, [`demo/supervisor_worker_handover.py`](https://github.com/topoteretes/cognee/blob/main/examples/integrations/docker-sandbox-kit/demo/supervisor_worker_handover.py), is self-contained: drop it into any repository with cognee installed and run `python supervisor_worker_handover.py` to execute all phases in one process, or `--phase brief|work|review` to split them across environments.
Clean up with:
```bash theme={null}
sbx rm -f cognee-supervisor cognee-worker && rm -rf demo/cognee-state demo/handover-out
```
### How the memory snapshot moves
Embedded LanceDB cannot operate on the shared virtiofs workspace mount. The demo works around this by running cognee state on each VM's **local disk** during a phase and handing it between sandboxes as a snapshot with `sbx cp` — the host keeps the canonical copy in `demo/cognee-state/` between phases:
```bash theme={null}
sbx exec "$1" -- sudo rm -rf /home/agent/cognee-state
sbx cp cognee-state "$1":/home/agent/
sbx exec "$1" -- sudo chown -R agent:agent /home/agent/cognee-state
# ... run the phase against DATA_ROOT_DIRECTORY / SYSTEM_ROOT_DIRECTORY on local disk ...
sbx cp "$1":/home/agent/cognee-state .
```
The `chown` step is required because `sbx cp` preserves the host uid. Phases run sequentially — the snapshot moves, it is never shared live.
<Tip>
For always-on cross-sandbox memory (concurrent agents, no shared workspace), run a central [cognee API server](/how-to-guides/cognee-sdk/deployment/index) and point sandboxes at it over the network allowlist instead of sharing embedded storage.
</Tip>
### Backends that support user permissioning
Permissions are `read` / `write` / `delete` / `share` per dataset. The kit's defaults — Ladybug/Kuzu for the graph, LanceDB for vectors — support per-user+dataset database isolation, as do Neo4j (including the `neo4j_community` container-per-dataset handler), Postgres (demo), Turso, and PGVector. Neptune, Neptune Analytics, `ladybug-remote`, and community vector adapters that do not register a dataset-database handler are **not** supported — see the [permissions system](/core-concepts/multi-user-mode/permissions-system/overview) for how users, datasets, and ACLs fit together.
## Inspecting memory and security
```bash theme={null}
sbx exec cognee-supervisor -- sh -lc 'echo $LLM_API_KEY' # "proxy-managed" — never a real key
sbx exec cognee-supervisor -- curl -s -o /dev/null -w "%{http_code}" https://example.com # 403: deny-all
sbx policy log # every allow/deny decision
ls demo/cognee-state/system/databases/<owner-user-uuid>/ # <dataset-uuid>.lbug + .lance.db per dataset
```
The relational database (`demo/cognee-state/system/databases/cognee_db`, SQLite) holds users, datasets, and the ACL rows. After the demo, the worker holds exactly two grants — `read` and `write` — on the shared dataset, and nothing on the private one.
***
<CardGroup>
<Card title="Kit source" icon="github" href="https://github.com/topoteretes/cognee/tree/main/examples/integrations/docker-sandbox-kit">
`spec.yaml`, the handover demo, and the README
</Card>
<Card title="Docker Sandboxes kits" icon="container" href="https://docs.docker.com/ai/sandboxes/customize/kits/">
Docker's documentation for authoring kits
</Card>
</CardGroup>
# Built-in Evaluation Framework
Source: https://docs.cognee.ai/integrations/eval-framework
Benchmark Cognee retrieval with the built-in evaluation framework.
Cognee ships a self-contained evaluation framework at `cognee/eval_framework/` that lets you benchmark retrieval quality on standard multi-hop QA datasets, compare different search strategies, and inspect results in an interactive HTML dashboard — all without any third-party evaluation service.
<Note>
This page covers the **built-in** evaluation framework. For the DeepEval integration (LLM-as-a-judge scoring), see [Evaluation with DeepEval](/integrations/deepeval-integration).
</Note>
## Overview
The pipeline has four sequential stages:
```
Corpus Builder → Answer Generation → Evaluation → Dashboard
```
| Stage | What it does |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| **Corpus Builder** | Downloads a benchmark dataset, ingests the corpus into Cognee's processing pipeline, and persists Q\&A pairs to disk. |
| **Answer Generation** | Runs each question through your chosen retriever and records the generated answer alongside the golden answer. |
| **Evaluation** | Scores every answer with the metrics you select (EM, F1, correctness, contextual relevancy, context coverage). |
| **Dashboard** | Produces a standalone `dashboard.html` with per-metric histograms, 95% confidence-interval bar charts, and a details table. |
## Quick Start
Before you start, complete [Quickstart](/getting-started/quickstart), make sure your corpus is already processed and indexed or let the corpus builder create it from scratch, and set `LLM_API_KEY`.
### Basic Usage
This minimal example runs the built-in evaluation pipeline and writes an HTML dashboard:
```python theme={null}
import asyncio
from cognee.eval_framework.run_eval import main
from cognee.eval_framework.eval_config import EvalConfig
config = EvalConfig(
benchmark="HotPotQA",
number_of_samples_in_corpus=50,
qa_engine="cognee_graph_completion",
evaluation_engine="DeepEval",
evaluation_metrics=["EM", "f1", "correctness"],
dashboard=True,
dashboard_path="my_dashboard.html",
)
asyncio.run(main(params=config.to_dict()))
```
This runs all four stages in sequence and writes `my_dashboard.html` to the current directory.
### What just happened
* **Benchmark setup** — `EvalConfig` defines which benchmark to use, how many samples to ingest, and which retrieval and evaluation engines to run.
* **Full evaluation run** — `main(params=config.to_dict())` executes corpus building, answer generation, evaluation, and dashboard generation in one flow.
* **Output artifacts** — the run produces metrics files and an HTML dashboard you can open locally to inspect results.
Open `my_dashboard.html` in any browser. You will see:
* **Distribution histograms** — score distributions per metric (10 bins).
* **Confidence-interval bar chart** — mean score ± 95% CI for each metric.
* **Details table** — per-question breakdown with generated answer, golden answer, retrieved context, score, and LLM rationale.
<Note>
For containerized runs, a `Dockerfile` is included at `cognee/eval_framework/Dockerfile`.
</Note>
***
## Token Usage Analysis
The framework ships a standalone CLI utility at `cognee/eval_framework/token_usage_analysis/` that estimates the token cost of **Cognee persistent memory** versus **full-context prompting**. It chunks an input text, measures the real ingestion token usage of running a few representative chunks through Cognee, and reports the break-even query count — after how many repeated queries full-context prompting has spent more tokens than Cognee's one-time ingestion plus per-query recall. It can optionally write cumulative-cost plots.
Install the eval dependencies and run the tool from its own directory:
```bash theme={null}
uv sync --dev --all-extras
cd cognee/eval_framework/token_usage_analysis
# one representative file (the input is treated as the corpus)
uv run python analyze.py --file data/wikipedia_article.txt --plot
# a folder of .txt files, pooled then sampled
uv run python analyze.py --dir some_corpus/ --out report.json
# a single representative chunk (corpus size must be given explicitly)
uv run python analyze.py --text "$(cat one_chunk.txt)" --corpus-tokens 200000
```
The script loads the repo-root `.env`, so it must contain a working `LLM_PROVIDER`, `LLM_MODEL`, and API key. If `--llm-models` is omitted, the configured `LLM_MODEL` is used. It always writes a JSON report; `--plot` additionally writes the cumulative-cost figure.
You must supply exactly one input form (`--file`, `--dir`, or `--text`). Key options:
| Flag | Default | Meaning |
| ----------------------------- | ------------------------- | ------------------------------------------------------- |
| `--file` / `--dir` / `--text` | — | input form (exactly one, required) |
| `--samples` | `3` | chunks to measure |
| `--max-chunk-size` | `4095` | chunk size (pass `8191` for Cognee's default) |
| `--llm-models` | the `.env` model | comma list; runs each, switching Cognee's config |
| `--corpus-tokens` | token count of the input | corpus size for the comparison (required with `--text`) |
| `--retrieved-context` | `1118` | recall context per query |
| `--query-overhead` | `32` | instruction + question tokens per query |
| `--reduction-factors` | `1,2,7,10` | milestones to report (`1` = parity/break-even) |
| `--out` | `token_usage_report.json` | JSON report output path |
| `--plot` / `--plot-dir` | off / `.` | also write the cross-over figure |
<Note>
`--plot` requires `matplotlib`, which is included in the `evals` extra (installed by `uv sync --dev --all-extras`).
</Note>
For the full cost model, sample corpora, and precomputed results, see the in-repo `cognee/eval_framework/token_usage_analysis/README.md` and the `results/chunk_4095/` and `results/chunk_8191/` folders.
***
## Code
<AccordionGroup>
<Accordion title="Running Individual Stages">
Each stage exposes a standalone async function you can call from your own scripts:
```python theme={null}
import asyncio
from cognee.eval_framework.corpus_builder.run_corpus_builder import run_corpus_builder
from cognee.eval_framework.answer_generation.run_question_answering_module import run_question_answering
from cognee.eval_framework.evaluation.run_evaluation_module import run_evaluation
from cognee.eval_framework.analysis.dashboard_generator import create_dashboard
from cognee.eval_framework.eval_config import EvalConfig
async def custom_eval():
config = EvalConfig().to_dict()
# Stage 1 – ingest corpus
await run_corpus_builder(config)
# Stage 2 – generate answers
await run_question_answering(config)
# Stage 3 – score answers
await run_evaluation(config)
# Stage 4 – build dashboard
create_dashboard(
metrics_path=config["metrics_path"],
aggregate_metrics_path=config["aggregate_metrics_path"],
output_file=config["dashboard_path"],
benchmark=config["benchmark"],
)
asyncio.run(custom_eval())
```
</Accordion>
<Accordion title="Filtering Benchmark Instances">
You can restrict which instances are evaluated using `INSTANCE_FILTER`:
```python theme={null}
from cognee.eval_framework.corpus_builder.run_corpus_builder import run_corpus_builder
# By integer indices (0-based)
await run_corpus_builder(config, instance_filter=[0, 1, 2, 10, 42])
# By string IDs (HotPotQA uses "_id" keys)
await run_corpus_builder(config, instance_filter=["5a7a06935542990198eaf050", ...])
# By path to a JSON file containing a list of IDs or indices
await run_corpus_builder(config, instance_filter="my_instance_ids.json")
```
</Accordion>
<Accordion title="Comparing Search Strategies">
A typical workflow for comparing two retrieval strategies:
```python theme={null}
import asyncio
from cognee.eval_framework.run_eval import main
from cognee.eval_framework.eval_config import EvalConfig
import os
for engine in ["cognee_graph_completion", "cognee_completion"]:
os.environ["QA_ENGINE"] = engine
os.environ["DASHBOARD_PATH"] = f"dashboard_{engine}.html"
os.environ["METRICS_PATH"] = f"metrics_{engine}.json"
os.environ["ANSWERS_PATH"] = f"answers_{engine}.json"
asyncio.run(main())
```
Open both HTML files side-by-side to compare F1 and exact-match scores across retrieval strategies.
</Accordion>
</AccordionGroup>
***
## Further details
All options are read from an `.env` file (or environment variables) via a Pydantic `BaseSettings` class (`EvalConfig`).
<AccordionGroup>
<Accordion title="Corpus Builder">
| Variable | Default | Description |
| ------------------------------ | --------- | ---------------------------------------------------------------------------- |
| `BENCHMARK` | `Dummy` | Which dataset to load (`Dummy`, `HotPotQA`, `Musique`, `TwoWikiMultiHop`). |
| `NUMBER_OF_SAMPLES_IN_CORPUS` | `1` | How many corpus paragraphs to ingest. |
| `BUILDING_CORPUS_FROM_SCRATCH` | `True` | Re-ingest the corpus on every run, or reuse an existing Cognee index. |
| `TASK_GETTER_TYPE` | `Default` | Cognify pipeline variant described in the Pipeline Strategies section below. |
</Accordion>
<Accordion title="Answer Generation">
| Variable | Default | Description |
| --------------------- | ------------------------- | --------------------------------------------------------------------- |
| `ANSWERING_QUESTIONS` | `True` | Run the QA stage. |
| `QA_ENGINE` | `cognee_graph_completion` | Which retriever to use, as described in the QA Engines section below. |
| `QUESTIONS_PATH` | `questions_output.json` | Where to save generated answers. |
</Accordion>
<Accordion title="Evaluation">
| Variable | Default | Description |
| --------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `EVALUATING_ANSWERS` | `True` | Run the metrics stage. |
| `EVALUATING_CONTEXTS` | `True` | Also compute `contextual_relevancy` and `context_coverage`. |
| `EVALUATION_ENGINE` | `DeepEval` | `DeepEval`, `DirectLLM`, or `BeamEval` (async evaluator for the BEAM `beam_rubric` / `kendall_tau` metrics). |
| `EVALUATION_METRICS` | `["correctness", "EM", "f1"]` | Any combination of the five metrics. |
| `DEEPEVAL_MODEL` | `gpt-4o-mini` | LLM used by DeepEval for `correctness` and `contextual_relevancy`. |
| `ANSWERS_PATH` | `answers_output.json` | Path to the answers file produced by the QA stage. |
| `METRICS_PATH` | `metrics_output.json` | Where per-sample metric results are written. |
</Accordion>
<Accordion title="Dashboard">
| Variable | Default | Description |
| ------------------------ | ------------------------ | -------------------------------------------- |
| `CALCULATE_METRICS` | `True` | Compute aggregate statistics (mean, 95% CI). |
| `DASHBOARD` | `True` | Generate the HTML report. |
| `AGGREGATE_METRICS_PATH` | `aggregate_metrics.json` | Where aggregate stats are written. |
| `DASHBOARD_PATH` | `dashboard.html` | Output path for the HTML dashboard. |
</Accordion>
<Accordion title="Supported Benchmarks">
| Benchmark | Adapter key | Description |
| ----------------- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **HotPotQA** | `HotPotQA` | \~90 K multi-hop Q\&A pairs from CMU; includes supporting-fact indices. |
| **MuSiQue** | `Musique` | Multi-step reasoning with question decompositions (Google Drive, JSONL). |
| **2WikiMultiHop** | `TwoWikiMultiHop` | Fact-triplet-style multi-hop QA from HuggingFace. |
| **Dummy** | `Dummy` | One hard-coded Q\&A pair — useful for smoke-testing the pipeline. |
| **BEAM** | `BEAM` | Long-context conversational-memory benchmark ([paper](https://arxiv.org/abs/2510.27246)) of synthetic multi-session conversations with rubric-based, LLM-judged questions. Preliminary support — evaluated with the `BeamEval` engine and the `beam_rubric` / `kendall_tau` metrics. |
</Accordion>
<Accordion title="Available Metrics">
| Metric key | Type | Description |
| ---------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `EM` | String | **Exact Match** — 1 if the generated answer exactly equals the golden answer (case-insensitive, whitespace-normalized). |
| `f1` | String | **Token-level F1** — precision/recall over word tokens between generated and golden answer. |
| `correctness` | LLM | **GEval correctness** via DeepEval (uses `DEEPEVAL_MODEL`). |
| `contextual_relevancy` | LLM | DeepEval's `ContextualRelevancyMetric` — how relevant the retrieved context is to the question. |
| `context_coverage` | LLM | Custom metric — fraction of the golden context covered by the retrieved context. |
| `beam_rubric` | LLM | **BEAM rubric** metric — an LLM judge scores each rubric criterion `0.0`, `0.5`, or `1.0`; the metric score is the mean across the question's criteria. Requires the `BeamEval` engine. |
| `kendall_tau` | LLM | **BEAM event-ordering** metric — Kendall's tau-b rank correlation (normalized to `[0, 1]`) between predicted and reference event orderings, multiplied by event-coverage F1. Applies only to `event_ordering` questions; returns `None` for all other question types. Requires the `BeamEval` engine. |
All metrics return a `{"score": float, "reason": str}` dict. Aggregate statistics include mean and a 95% confidence interval computed with 10 000 bootstrap samples.
</Accordion>
<Accordion title="Pipeline Strategies">
The `TASK_GETTER_TYPE` variable controls how each corpus document is processed during the corpus-building stage:
| Strategy | Description |
| -------------- | ------------------------------------------------------------------------------------------------------------------ |
| `Default` | Full pipeline: classify → chunk → extract graph → summarize → add data points. |
| `CascadeGraph` | Same as Default but processes documents in batches of 10. |
| `NoSummaries` | Skips the summary step; applies ontology grounding during graph extraction when an ontology file path is provided. |
| `JustChunks` | Minimal pipeline: classify → chunk → add data points (no graph). |
</Accordion>
<Accordion title="QA Engines">
| Engine key | Description |
| ------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| `cognee_graph_completion` | Graph traversal followed by LLM completion. |
| `cognee_graph_completion_cot` | Chain-of-thought reasoning over the graph. |
| `cognee_graph_completion_context_extension` | Graph traversal with extended context retrieval. |
| `cognee_completion` | Direct LLM completion without graph traversal. |
| `graph_summary_completion` | Uses pre-computed graph summaries for retrieval. |
| `beam_router` | Routes each question through the BEAM answer router. Preliminary — intended for BEAM benchmark runs. |
</Accordion>
</AccordionGroup>
# GitHub
Source: https://docs.cognee.ai/integrations/github-integration
Connect a GitHub organization and index every covered repository into the code graph.
Connect a GitHub **organization** (or user account) to Cognee by installing a GitHub App into it. Every repository the installation covers is cloned and indexed into the code graph under a single dataset, and stays fresh through webhooks — a push to a default branch re-indexes that repository, and repositories added to the installation are picked up automatically.
<Info>
This integration is for self-hosted Cognee deployments — you create and own the GitHub App, so it lives in your own GitHub organization and talks only to your own Cognee backend.
</Info>
**Ships in `1.5.4`.** The connector landed after `1.5.3` and is first tagged in v1.5.4, so `cognee>=1.5.4` carries it — no `dev` checkout is needed. See the [changelog](/changelog).
## What It Does
* **One install covers the whole org.** The org admin installs the app once and picks which repositories it can see. Cognee indexes all of them — there is no per-repository connect step.
* **Code graph, not documents.** Repositories go through the deterministic [code-graph pipeline](/guides/code-graph): typed module/symbol/route/storage/service facts and their edges, with **no LLM or embedding calls**. Results are reachable through [`SearchType.CODE`](/python-api/search-type) only — the route produces no chunks and no embeddings, so `GRAPH_COMPLETION`, `CHUNKS`, and `RAG_COMPLETION` do not cover this content.
* **No stored GitHub tokens.** The credential Cognee persists for a GitHub installation carries an **empty token payload**. Access tokens (\~1 hour) are minted on demand from the app's private key when a sync runs, used, and discarded.
* **Read-only.** The app needs `Contents: Read-only` and nothing else. Cognee never writes to GitHub.
## Prerequisites
* Cognee **`>= 1.5.4`** — see the version note above.
* A running Cognee backend reachable over **HTTPS from the public internet**: GitHub delivers webhooks and the OAuth redirect to it and will not call `localhost`. For local development use a tunnel (ngrok or similar) — see the [Slack integration's ngrok section](/integrations/slack-integration#local-development-ngrok-tunnel), which applies verbatim; for production this is your deployment's normal public domain.
* Permission to create and install a GitHub App in the target organization (org owner, or a repository admin if the org allows it).
* `git` available on the Cognee host — repositories are shallow-cloned into `COGNEE_REPOS_DIR` (default `~/.cognee/repos`).
## Setup
### 1. Create the GitHub App
Go to [github.com/settings/apps](https://github.com/settings/apps) → **New GitHub App** and configure:
| Setting | Value |
| ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Callback URL** | `https://<public-host>/api/v1/integrations/github/callback` |
| **Request user authorization (OAuth) during installation** | **Enabled** — required. Cognee never trusts the `installation_id` in the redirect on its own; it exchanges the OAuth `code` for a user token and confirms that user actually has access to the installation before binding it. Without this the install always fails. |
| **Webhook URL** | `https://<public-host>/api/v1/integrations/github/events` |
| **Webhook secret** | Any high-entropy string — this becomes `GITHUB_WEBHOOK_SECRET`. |
| **Repository permissions** | `Contents: Read-only`. |
| **Subscribe to events** | **Push** and **Installation repositories**. |
Then, from the app's settings page, collect: the numeric **App ID**, the app's **slug** (the `<slug>` in `github.com/apps/<slug>`), a generated **private key** (`.pem` download), and the **Client ID** / **Client secret**.
<Warning>
Create a **separate app per environment** (local dev, staging, production). One app has one callback URL and one webhook URL, so pointing a shared app at a laptop routes another environment's deliveries there.
</Warning>
### 2. Configure Cognee
Copy the app's values into your backend's `.env` (the same block is in `.env.template`):
| Variable | Value |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GITHUB_APP_ID` | The app's numeric id, from its settings page. Used to sign the app JWT that mints installation tokens. |
| `GITHUB_APP_SLUG` | The app's URL slug (`github.com/apps/<slug>`) — used to build the installation URL the connect flow redirects to. |
| `GITHUB_APP_PRIVATE_KEY` | The downloaded PEM private key. Literal `\n` escapes are accepted, so the whole key fits on one line in an env file. Must be an **RSA** key (GitHub's generated keys are). |
| `GITHUB_CLIENT_ID` | From the app's settings page. |
| `GITHUB_CLIENT_SECRET` | From the app's settings page. |
| `GITHUB_WEBHOOK_SECRET` | The webhook secret you set on the app. It verifies every inbound delivery's `X-Hub-Signature-256` **and** signs the OAuth `state` parameter — both server-side-only uses of the same secret. |
| `GITHUB_FRONTEND_BASE_URL` | Your frontend's origin, e.g. `http://localhost:3000` — where the callback redirects the browser when the install finishes. |
| `INTEGRATION_CREDENTIALS_KEY` | Not GitHub-specific — the AES-256-GCM key (32 raw bytes, base64-encoded) the integrations framework encrypts stored credentials under. GitHub's payload is empty, but the row still goes through the encrypted store, so a missing key fails the connect. Generate one with `openssl rand -base64 32`. `INTEGRATION_CREDENTIALS_KEYS` (a JSON keyring) is the preferred, rotation-capable form. |
<Note>
There is no `GITHUB_REDIRECT_URI`. Unlike Slack, the GitHub App install flow takes its callback URL from the app's own settings rather than from a query parameter, so the redirect URL is configured in one place only — the app.
</Note>
Every one of these is checked at use time, not at boot: a deployment with GitHub unconfigured still starts normally, and the first call that needs a missing value fails with `GITHUB_<NAME> is not configured` rather than a confusing downstream error. `POST /authorize` turns that into a **503** (`github integration is not configured on this server`) instead of a 500.
Restart the backend after any `.env` change.
### 3. Install and Connect
The backend ships without frontend wiring for GitHub in this release — the aggregate `GET /api/v1/integrations/status` reports the provider generically, but there is no **Connect** button yet. Drive the flow through the API:
```bash theme={null}
# 1. Mint the install URL (authenticated as the Cognee user who will own the data)
curl -X POST https://<public-host>/api/v1/integrations/github/authorize \
-H "Authorization: Bearer <your-cognee-token>"
# {"authorizeUrl":"https://github.com/apps/<slug>/installations/new?state=..."}
```
Open that URL in a browser, pick the organization, choose which repositories the app may access, and approve. GitHub redirects back to `/api/v1/integrations/github/callback`, which binds the installation to your Cognee account and then sends the browser to `<GITHUB_FRONTEND_BASE_URL>/integrations?github=connected`.
<Warning>
**Install from the minted URL, not from the app's public install page.** The signed `state` is what binds the installation to a Cognee account, and it expires **10 minutes** after `/authorize` returns. An install started without it (or after it expires) lands on `?github=error_invalid_state` and stores nothing.
</Warning>
The initial sync of every covered repository is kicked off **detached**, after the redirect. The browser returns immediately; repositories appear in the graph as the clones and pipeline runs finish, which for a large org can take a while.
## What Gets Indexed
Every repository the installation covers lands in **one dataset per installation**, named `github_<account>`: the org (or user) login lowercased, with every run of non-alphanumeric characters collapsed to `_`. `Acme-Corp` becomes `github_acme_corp`. One dataset per org rather than per repository means [backend access control](/core-concepts/multi-user-mode/multi-user-mode-overview) isolates at the org boundary — per-repo datasets would mean one isolated database per repository.
The dataset is owned by the Cognee user who completed the install. Query it with `SearchType.CODE`:
```python theme={null}
import cognee
from cognee import SearchType
results = await cognee.search(
query_type=SearchType.CODE,
query_text="",
datasets=["github_acme_corp"],
code_query={"operation": "query_facts", "kinds": ["module", "symbol", "route"], "limit": 20},
)
```
See [SearchType — CODE](/python-api/search-type#per-search-type-parameters) for the full set of `code_query` operations (`explore`, `traverse`, `find_path`, `impact_analysis`, `architecture`, `insights`, `delta`) and their arguments, and the [Code Graph guide](/guides/code-graph) for a walkthrough.
## Keeping It in Sync
After the initial sync, webhook deliveries to `POST /api/v1/integrations/github/events` keep the graph fresh. Coverage is deliberately narrow in this first cut:
| Event | Behavior |
| -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `push` | Re-indexes that repository — but only for pushes to its **default branch**. The clone is a shallow copy of the default branch, so other refs change nothing Cognee holds. |
| `installation_repositories` | Repositories **added** to the installation are indexed. Repositories **removed** are logged only — indexed data is retained, because deleting it on a webhook would be a silent, destructive surprise. Drop it yourself with [`forget()`](/python-api/forget). |
| `installation` (`deleted` / `suspend`) | Revokes the stored credential, so no further tokens are minted. |
Every delivery is authenticated by an HMAC-SHA256 over the raw request body, keyed with `GITHUB_WEBHOOK_SECRET` and compared against `X-Hub-Signature-256`. That signature is the entire auth model for the route — it is unauthenticated by design, since GitHub cannot send a bearer token. A bad or missing signature is a **401**. Cognee acks the delivery as soon as the signature checks out and handles it detached, so GitHub's delivery timeout is never in play no matter how long a re-index takes.
Deliveries are idempotent and safe to redeliver: unchanged repository snapshots are skipped, and revokes are no-ops on already-revoked rows. A delivery for an installation Cognee doesn't know (or has revoked) is logged and dropped rather than erroring — GitHub gives no ordering guarantee, so an `installation.created` arriving before the callback stored the credential is a normal, self-healing state.
<Note>
The installation token minted for a sync lives about an hour and repositories are cloned sequentially. A very large installation can outlive its token mid-batch; the affected repositories surface as per-repo errors, and the next push (or a manual re-sync) picks them up with a fresh token.
</Note>
## How Access Works
* **Nothing durable is stored but the installation id.** Tokens are minted on demand: the app's private key signs a \~9-minute RS256 JWT, which mints a \~1-hour installation token scoped to that installation's repositories.
* **The redirect's `installation_id` is never trusted on its own.** It arrives on an unauthenticated endpoint and is a small, guessable integer. The OAuth `code` is exchanged for a user token, that user's access to the installation is confirmed against `GET /user/installations`, and only then is the authoritative installation record fetched with the app JWT. Without the check, anyone could bind another org's installation — and read access to its private repositories — to their own account.
* **One installation, one Cognee account.** An installation already connected to a different user is refused (`?github=error_already_connected`) rather than silently reassigned.
* **Tokens never ride a URL.** Clone URLs stay credential-free (`https://github.com/org/repo.git`); the token reaches `git` through environment-level config instead. The persisted git remote, the recorded `source` on each result item, log lines, and git error output all use the credential-free form, so a token cannot leak through any of them.
## Disconnect vs. Uninstall
They are not the same, and neither deletes indexed data:
* **Disconnect in Cognee** — `DELETE /api/v1/integrations/github/connection` — marks the stored credential revoked. No further installation tokens are minted, so syncs stop. The app stays installed on GitHub, and the `github_<org>` dataset stays exactly as it is.
* **Uninstalling the app on GitHub** fires the `installation` `deleted` webhook, which revokes the credential server-side — the same end state, reached from the other side. This is what actually cuts off GitHub's access.
Cognee deliberately does **not** uninstall the app on your behalf during a disconnect: the GitHub-side equivalent of a token revoke is deleting the installation for the entire org, which is too destructive for a per-account action. To remove the indexed data itself, use [`forget()`](/python-api/forget).
## Troubleshooting
1. **`POST /authorize` returns 503 "github integration is not configured on this server"**: `GITHUB_APP_SLUG` or `GITHUB_WEBHOOK_SECRET` (it signs the state) is unset. The backend log names the exact variable. A missing `GITHUB_FRONTEND_BASE_URL` surfaces later, as the same 503 from the callback instead.
2. **Redirected to `?github=error_invalid_state`**: the install didn't start from a freshly minted `/authorize` URL, or more than 10 minutes passed between minting it and approving the install. Mint a new one.
3. **Redirected to `?github=error_exchange_failed`**: the callback never 500s — it catches everything and redirects, and the traceback is in the **backend log**. Usual causes: "Request user authorization (OAuth) during installation" is off on the app (so no `code` arrives and the exchange has nothing to trade), mismatched `GITHUB_CLIENT_ID`/`GITHUB_CLIENT_SECRET`, a `GITHUB_APP_PRIVATE_KEY` that isn't an RSA PEM, or a missing `INTEGRATION_CREDENTIALS_KEY`.
4. **Redirected to `?github=error_already_connected`**: this installation is bound to another Cognee user. Disconnect it there first.
5. **Connect succeeds but no repositories appear**: the initial sync runs detached and can take a long time on a large org — check the backend log for `Syncing N GitHub repositories for <org> into dataset github_<org>`. `N` is what the app can actually see, so if it's `0` or too low, widen the installation's repository access on GitHub (that fires `installation_repositories` and syncs the additions).
6. **Webhook deliveries show 401 in GitHub's "Recent Deliveries"**: `GITHUB_WEBHOOK_SECRET` doesn't match the secret set on the app — easy to hit when juggling more than one app. Re-copy it and restart the backend.
7. **Webhook deliveries show 404**: the provider name in the URL is wrong, or the backend is running a build without the GitHub connector registered. The route is `/api/v1/integrations/github/events`.
8. **A push doesn't re-index anything**: only pushes to a repository's **default branch** trigger a re-index. Pushes to other branches are ignored by design.
9. **`GRAPH_COMPLETION` or `CHUNKS` search finds nothing from a connected repo**: expected — the code route stores graph facts only, with no chunks or embeddings. Use `SearchType.CODE`.
## Related
<CardGroup>
<Card title="Code Graph" icon="code" href="/guides/code-graph">
The pipeline this connector feeds, and every `SearchType.CODE` operation.
</Card>
<Card title="Slack" icon="messages-square" href="/integrations/slack-integration">
The other first-party OAuth connector, sharing the same credential store and connect flow.
</Card>
</CardGroup>
# Google ADK
Source: https://docs.cognee.ai/integrations/google-adk-integration
Add persistent memory to Google ADK agents with Cognee.
Give your [Google ADK](https://google.github.io/adk-docs/) agents cognee's memory. Store structured knowledge, retrieve via natural language, and maintain context across sessions - all through two async tools.
## Why Use This Integration
* **Advanced Memory Features**: Structured knowledge graphs with embeddings for multi-hop reasoning
* **Cross-Session Persistence**: Memory survives agent restarts and conversation threads
* **Semantic Recall**: Natural language queries with graph traversal
* **Session Isolation**: Multi-tenant support with per-user data boundaries
* **Native ADK Tools**: Works as `LongRunningFunctionTool`—async-first, production-ready
## Installation
```bash theme={null}
pip install cognee-integration-google-adk
```
## Quick Start
Before using the integration, configure your environment variables:
```bash theme={null}
export GOOGLE_API_KEY="your-google-api-key-here" # for Google ADK
export LLM_API_KEY="your-openai-api-key-here" # for cognee
```
Add memory tools to your Google ADK agent:
```python theme={null}
import asyncio
from google.adk.agents import Agent
from google.adk.runners import InMemoryRunner
from cognee_integration_google_adk import add_tool, search_tool
async def main():
# Create agent with memory
agent = Agent(
model="gemini-2.5-flash",
name="assistant",
description="An assistant with persistent memory",
instruction="You are a helpful assistant with access to a knowledge base.",
tools=[add_tool, search_tool],
)
runner = InMemoryRunner(agent=agent)
# Store information
await runner.run_debug(
"Remember: Acme Corp, healthcare, $1.2M contract"
)
# Retrieve information
events = await runner.run_debug(
"What healthcare contracts do we have?"
)
for event in events:
if event.is_final_response() and event.content:
for part in event.content.parts:
if part.text:
print(part.text)
asyncio.run(main())
```
## Cross-Session Persistence
Memory persists across different agent instances:
```python theme={null}
from cognee_integration_google_adk import get_sessionized_cognee_tools
# Session 1: Store information
add_tool, search_tool = get_sessionized_cognee_tools("user-123")
agent_1 = Agent(
model="gemini-2.5-flash",
name="assistant",
description="Assistant with memory",
instruction="You are a helpful assistant.",
tools=[add_tool, search_tool],
)
runner_1 = InMemoryRunner(agent=agent_1)
await runner_1.run_debug("I'm working on authentication")
# Session 2: Different instance, same memory
add_tool, search_tool = get_sessionized_cognee_tools("user-123")
agent_2 = Agent(
model="gemini-2.5-flash",
name="assistant",
description="Assistant with memory",
instruction="You are a helpful assistant.",
tools=[add_tool, search_tool],
)
runner_2 = InMemoryRunner(agent=agent_2)
events = await runner_2.run_debug("What was I working on?")
# Returns: "authentication"
```
## Custom Session Management
Control session isolation with custom session IDs:
```python theme={null}
# User-specific memory
add_tool, search_tool = get_sessionized_cognee_tools(session_id="user_123")
# Org-specific memory
add_tool, search_tool = get_sessionized_cognee_tools(session_id="org_acme")
# Generate unique session automatically
add_tool, search_tool = get_sessionized_cognee_tools() # Uses UUID-based session ID
```
<Info>
Sessionized tools scope data to their session ID. Use non-sessionized `add_tool` and `search_tool` to access data across all sessions.
</Info>
## How It Works
1. **Add Tool**: Stores data in cognee's knowledge graph with embeddings
2. **Search Tool**: Retrieves relevant information via cognee's recall pipeline
3. **Auto-Processing**: cognee extracts entities, relationships, and context automatically
4. **Session Scoping**: Sessionized tools filter data by session ID; non-sessionized tools access all data
## Use Cases
<AccordionGroup>
<Accordion title="Knowledge Accumulation">
Build domain knowledge incrementally over multiple sessions:
```python theme={null}
# Pre-load documents into cognee
import cognee
for doc_path in document_paths:
with open(doc_path, 'r') as f:
content = f.read()
await cognee.remember(content)
# Query across all documents
events = await runner.run_debug(
"Find information about contract terms"
)
```
</Accordion>
<Accordion title="Context-Aware Assistance">
Maintain user context across work sessions:
```python theme={null}
# Monday
await runner.run_debug("Debugging payment flow")
# Wednesday
events = await runner.run_debug("What was I debugging?")
```
</Accordion>
<Accordion title="Multi-Tenant Applications">
Isolate data per user/organization while sharing global knowledge:
```python theme={null}
# Per-user isolation
add_tool, search_tool = get_sessionized_cognee_tools(session_id=user_id)
agent = Agent(
model="gemini-2.5-flash",
name="assistant",
description="User assistant",
instruction="You are a helpful assistant.",
tools=[add_tool, search_tool]
)
```
</Accordion>
<Accordion title="Multi-Agent Workflows">
Share knowledge between specialized agents:
```python theme={null}
# Data collector agent
data_agent = Agent(
model="gemini-2.5-flash",
name="data_collector",
description="Collects and stores information",
instruction="You collect and store important information.",
tools=[add_tool]
)
# Research agent
research_agent = Agent(
model="gemini-2.5-flash",
name="researcher",
description="Searches stored information",
instruction="You search and analyze the knowledge base.",
tools=[search_tool]
)
```
</Accordion>
</AccordionGroup>
***
<CardGroup>
<Card title="GitHub Repository" icon="github" href="https://github.com/topoteretes/cognee-integrations/tree/main/integrations/google-adk">
View source code and examples
</Card>
<Card title="Examples" icon="book" href="https://github.com/topoteretes/cognee-integrations/tree/main/integrations/google-adk/examples">
Runnable example scripts
</Card>
</CardGroup>
# Cognee Plugin for Hermes Agent
Source: https://docs.cognee.ai/integrations/hermes-agent-integration
Add persistent memory to Hermes Agent with a Cognee plugin.
Give [Hermes Agent](https://github.com/NousResearch/hermes-agent) persistent memory with a drop-in Cognee memory provider plugin. Each completed turn is stored in Cognee's session cache and automatically promoted into the permanent knowledge graph at session end — no code required.
## Why Use This Integration
* **Zero code**: Install, run `hermes memory setup`, and memory works automatically
* **Two memory tiers**: Turns land in a session cache, then `improve()` promotes them into the permanent graph
* **Three connection modes**: Local server (default), remote/cloud, or in-process embedded
* **Resilient**: Built-in circuit breaker prevents cascading failures when Cognee is unreachable
## Installation
The `cognee-integration-hermes-agent` package is not yet published on PyPI. Install the plugin locally from the [cognee-integrations](https://github.com/topoteretes/cognee-integrations) repository:
```bash theme={null}
git clone https://github.com/topoteretes/cognee-integrations.git
cd cognee-integrations
mkdir -p ~/.hermes/plugins/cognee
cp -R integrations/hermes-agent/. ~/.hermes/plugins/cognee/
hermes memory setup # select "cognee" in the memory provider picker
```
<Info>
Requires Python 3.10+. The plugin declares `cognee>=1.0.0,<2.0.0` as a pip dependency in its `plugin.yaml`. Once a PyPI release is available, `pip install cognee-integration-hermes-agent` will register the plugin via the `hermes_agent.plugins` entry point.
</Info>
## Quick Start
Configure your LLM key, then run Hermes as usual — memory is captured and recalled automatically:
```bash theme={null}
export LLM_API_KEY="your-openai-api-key-here" # cognee extracts knowledge with an LLM
hermes memory setup # one-time: pick "cognee"
hermes # start a session
```
During a session, just talk to the agent:
```
You: Remember that Alice works in engineering.
(Cognee stores this turn in the session cache)
You: What does Alice do?
(Hermes recalls from Cognee memory)
```
At session end, if `COGNEE_IMPROVE_ON_END=true` (the default), `cognee.improve(...)` runs to promote the session cache into the permanent graph.
## Connection Modes
The plugin connects to Cognee in one of three modes. There are **no silent fallbacks** — if the configured mode fails, the failure surfaces.
| Mode | How to enable | Notes |
| -------------------------- | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| **Local server** (default) | *(nothing)* | Hermes is a thin HTTP client to a single-writer local Cognee server. Safest for concurrent/background work. |
| **Remote / Cloud** | Set `COGNEE_BASE_URL` (and `COGNEE_API_KEY`) | Thin HTTP client to a managed or cloud Cognee instance. |
| **Embedded** | Set `COGNEE_EMBEDDED=true` | Runs Cognee in-process. Single-process/offline only; unsafe for concurrency. |
## Authentication
Cognee authenticates with its own credentials — Hermes' model credentials are not reused. Everything Cognee does on its own — entity and relationship extraction, summarization, embeddings, and search-time completions — runs against the provider Cognee itself is configured with, billed by that provider. Cognee's LLM layer authenticates by API key only: a host agent's subscription or OAuth sign-in (for example a Codex-style ChatGPT login) cannot be handed to it, and Cognee's own OAuth flows sign you in to Cognee Cloud or to data-source integrations such as Slack, never to an LLM provider. So in local or embedded mode you need your own `LLM_API_KEY` even when your agent's model is already signed in.
Which credentials you need depends on the connection mode:
| Mode | Credentials Cognee needs |
| -------------------------- | ---------------------------------------------------------------------------------------------- |
| **Local server** (default) | `LLM_API_KEY` — the local server does extraction and embeddings itself |
| **Remote / Cloud** | `COGNEE_BASE_URL` + `COGNEE_API_KEY`; no LLM key, since the server side holds the provider key |
| **Embedded** | `LLM_API_KEY` — Cognee runs in-process and calls the provider directly |
In local and embedded mode one `LLM_API_KEY` covers everything: Cognee defaults to `openai/gpt-5-mini` for the LLM and `openai/text-embedding-3-large` for embeddings, and embeddings reuse `LLM_API_KEY` when `EMBEDDING_API_KEY` is unset. For another provider, set `LLM_PROVIDER`, `LLM_MODEL`, and — for Azure, Ollama, or OpenAI-compatible endpoints — `LLM_ENDPOINT`, plus the matching `EMBEDDING_*` variables. See [LLM providers](/setup-configuration/llm-providers) and [embedding providers](/setup-configuration/embedding-providers).
<Tip>
To avoid a separate LLM key altogether, run [Cognee as an MCP server](/cognee-mcp/mcp-local-setup) in a host that grants the MCP `sampling` capability and set `LLM_PROVIDER="mcp-sampling"` — completions are delegated to the host's own model. That is a different setup from this memory provider plugin, and embeddings still need their own provider.
</Tip>
## Configuration
Set these as environment variables. Non-secret settings are also saved to `$HERMES_HOME/cognee.json`; secrets go to `$HERMES_HOME/.env`.
| Variable | Default | Description |
| ----------------------- | ---------------- | ----------------------------------------- |
| `LLM_API_KEY` | — | LLM API key used by Cognee |
| `LLM_MODEL` | provider default | LLM model name |
| `COGNEE_DATASET` | `hermes` | Dataset name for stored memory |
| `COGNEE_BASE_URL` | — | Cognee API base URL (enables remote mode) |
| `COGNEE_API_KEY` | — | Cognee service API key (remote mode) |
| `COGNEE_EMBEDDED` | `false` | Run Cognee in-process |
| `COGNEE_TOP_K` | `5` | Max results per recall |
| `COGNEE_IMPROVE_ON_END` | `true` | Run `improve()` at session end |
| `COGNEE_SESSION_PREFIX` | `hermes` | Session ID prefix |
<Info>
Manage the plugin with `hermes cognee status`, `hermes cognee setup`, `hermes cognee config`, and `hermes cognee install`.
</Info>
## Tools
The plugin exposes three tools to the agent:
| Tool | Description |
| ----------------- | -------------------------------------------------------------------------------------------------- |
| `cognee_recall` | Search session memory and the persistent graph (`query`, optional `scope`, `search_type`, `top_k`) |
| `cognee_remember` | Persist important content into the knowledge graph (`content`, optional `dataset`) |
| `cognee_forget` | Delete memory (optional `dataset`, `everything=true`, or `memory_only`) |
## How It Works
1. **Prefetch**: Before each turn, `cognee_recall` runs in the background to populate memory context
2. **Capture**: Each completed turn is synced to the session cache automatically
3. **Promote**: At session end (with `COGNEE_IMPROVE_ON_END=true`), `cognee.improve(session_ids=[...])` promotes the session cache into the permanent graph
4. **Resilience**: After repeated failures the provider trips a circuit breaker, pausing briefly before retrying so errors don't cascade
***
<CardGroup>
<Card title="GitHub Repository" icon="github" href="https://github.com/topoteretes/cognee-integrations/tree/main/integrations/hermes-agent">
View source code and examples
</Card>
<Card title="Hermes Agent" icon="book" href="https://github.com/NousResearch/hermes-agent">
Learn about Hermes Agent
</Card>
</CardGroup>
# Integrations
Source: https://docs.cognee.ai/integrations/index
Connect Cognee to agent frameworks, observability tools, and evaluation tools.
Cognee works seamlessly with AI Agent frameworks and popular tools in the AI ecosystem.
These integrations help you **observe**, **evaluate**, **use** your AI Memory.
<Info>
All integrations are designed to be lightweight and easy to configure, requiring minimal setup to get started.
</Info>
## Observability & Monitoring
Track performance, debug issues, and monitor your knowledge graph operations in production.
<CardGroup>
<Card title="OpenTelemetry" href="/integrations/opentelemetry-tracing" icon="chart-bar">
**Vendor-neutral observability**
Export OTEL traces, metrics, and logs for Cognee's memory operations to any OTLP-compatible backend (Grafana, Jaeger, Dash0, Dynatrace, and more).
</Card>
<Card title="Keywords AI" href="/integrations/keywordsai-integration" icon="tag">
**LLM application tracing**
Span-level tracing across tasks and workflows with minimal code using Cognee's observe abstraction.
</Card>
</CardGroup>
## Evaluation & Testing
Measure and improve the quality of your knowledge graph outputs with comprehensive evaluation frameworks.
<CardGroup>
<Card title="DeepEval" href="/integrations/deepeval-integration" icon="test-tube">
**Comprehensive RAG evaluation**
Run QA & RAG metrics including Contextual Relevancy, Precision/Recall, and Coverage using LLM-as-a-judge workflows.
</Card>
</CardGroup>
## Cloud LLM Providers
Connect to enterprise-grade LLM services through Cognee's flexible integration layer.
<CardGroup>
<Card title="AWS Bedrock" href="/integrations/aws-bedrock-integration" icon="cloud">
**Enterprise LLM access**
Use AWS Bedrock models including Claude, Nova/Titan, GPT-OSS, and others through Cognee's native Bedrock provider. LiteLLM proxy setup is optional.
</Card>
</CardGroup>
## Data Ingestion
Scrape, extract, and ingest web content directly into Cognee's knowledge graph.
<CardGroup>
<Card title="GitHub" href="/integrations/github-integration" icon="github">
**Org repositories to code graph**
Install a GitHub App into an org and index every covered repository into one `github_<org>` dataset, kept fresh by webhooks and searchable with `SearchType.CODE`.
</Card>
<Card title="Linear" href="/integrations/linear-integration" icon="square-kanban">
**Workspace issues + an agent that answers**
Install Cognee as a Linear agent: @mention it or delegate an issue and it answers from memory, while the workspace's issues sync into one `linear_<workspace>` dataset via webhooks.
</Card>
<Card title="ScrapeGraphAI" href="/integrations/scrapegraphai-integration" icon="globe">
**Web scraping to knowledge graph**
Scrape URLs with natural language prompts and ingest the results directly into cognee.
</Card>
<Card title="Gmail" href="/integrations/dlt-integration#gmail-connector" icon="envelope">
**Inbox to knowledge graph**
Ingest Gmail messages via dlt with OAuth read-only access, incremental sync, and forget-on-delete. Ships as the standalone `cognee-community-connector-gmail` package.
</Card>
</CardGroup>
## Agent Frameworks
Build stateful AI agents with persistent semantic memory that endures across sessions.
<CardGroup>
<Card title="LangGraph" href="/integrations/langgraph-integration" icon="bot">
`add_tool` / `search_tool` for `create_agent`—no manual state management.
</Card>
<Card title="Google ADK" href="/integrations/google-adk-integration" icon="bot">
Native `LongRunningFunctionTool` for async-first Gemini agents.
</Card>
<Card title="CrewAI" href="/integrations/crewai-integration" icon="bot">
Shared crew memory via `add_tool` and `search_tool`.
</Card>
<Card title="Strands" href="/integrations/strands-integration" icon="bot">
`cognee_tools()` exposing `remember` and `recall` (cognee v1.0).
</Card>
<Card title="Hermes Agent" href="/integrations/hermes-agent-integration" icon="bot">
Drop-in memory provider plugin—no code required.
</Card>
<Card title="Band" href="/integrations/band-integration" icon="bot">
Wrap any Band adapter in `CogneeMemoryAdapter` — rooms become memory.
</Card>
<Card title="Vellum" href="/integrations/vellum-integration" icon="bot">
Persistent memory for Vellum agents with the Cognee plugin.
</Card>
<Card title="OpenAI Agent SDK" href="/integrations/openai-agents-sdk-integration" icon="bot">
Native `function_tool` pattern with agent handoffs support.
</Card>
<Card title="OpenClaw" href="/integrations/openclaw-integration" icon="bot">
Multi-scope auto-index and recall for your personal AI agent.
</Card>
</CardGroup>
## Coding Agents
Give your terminal coding agents persistent memory with the Cognee plugin—no code, no `pip install`. It hooks into the agent's lifecycle to capture and recall memory automatically.
<CardGroup>
<Card title="Claude Code" href="/integrations/claude-code-integration" icon="bot">
`claude plugin install cognee-memory@cognee` — memory across every session.
</Card>
<Card title="Codex" href="/integrations/codex-integration" icon="terminal">
`codex plugin add cognee@cognee` — memory across every session.
</Card>
<Card title="Pi" href="/integrations/pi-integration" icon="pi">
`pi install npm:@kerryhatcher/pi-cognee` — community extension with SDK and MCP modes.
</Card>
<Card title="Docker Sandboxes" href="/integrations/docker-sandboxes-kit" icon="container">
`sbx run claude --kit ./cognee-memory` — a mixin kit that stacks memory onto any sandboxed agent.
</Card>
</CardGroup>
## No-Code & Low-Code
Build memory pipelines visually—no Python required.
<CardGroup>
<Card title="n8n" href="/integrations/n8n-integration" icon="workflow">
No-code memory workflows with all n8n integrations.
</Card>
<Card title="Dify" href="/integrations/dify-integration" icon="workflow">
Cognee memory tools for Dify apps and workflows (Cloud or self-hosted).
</Card>
</CardGroup>
## Team Chat
Query and grow your memory without leaving a conversation.
<CardGroup>
<Card title="Slack" href="/integrations/slack-integration" icon="messages-square">
Ask with `/cognee-ask`, save with `/cognee-remember` or the **Remember this** shortcut, and link your account — or run the standalone channel-memory bot that answers `@cognee` with cited source messages.
</Card>
</CardGroup>
## Agent IDEs & Development Tools
Integrate Cognee directly into your development workflow with MCP-compatible tools and AI assistants.
<Accordion title="MCP Integrations">
<CardGroup>
<Card title="Cursor" href="/cognee-mcp/integrations/cursor" icon="code">
AI-powered code editor integration
</Card>
<Card title="Continue" href="/cognee-mcp/integrations/continue" icon="play">
VS Code AI assistant plugin
</Card>
<Card title="Claude Code via MCP" href="/cognee-mcp/integrations/claude-code" icon="brain">
Anthropic's Claude integration
</Card>
</CardGroup>
<CardGroup>
<Card title="Cline" href="/cognee-mcp/integrations/cline" icon="terminal">
VS Code AI coding extension
</Card>
<Card title="Codex via MCP" href="/cognee-mcp/integrations/codex" icon="terminal">
OpenAI coding agent with MCP support
</Card>
</CardGroup>
</Accordion>
<Note>
Cognee's Model Context Protocol (MCP) adapter enables seamless integration with AI assistants, providing direct access to your knowledge graphs for in-context assistance.
</Note>
## Contributing Integrations
Don't see your favorite tool? Cognee is open and extensible—help us grow the ecosystem!
<Info>
All community database integrations are maintained in the [cognee-community](https://github.com/topoteretes/cognee-community) repository, keeping the core Cognee package lean while providing extensibility.
</Info>
# Observability with Keywords AI
Source: https://docs.cognee.ai/integrations/keywordsai-integration
Trace Cognee workflows with Keywords AI.
## Observability with Keywords AI
[Keywords AI](https://www.keywordsai.co/) provides observability and tracing for LLM-powered and agentic applications. In Cognee, it captures spans for tasks and workflows via the same `@observe` decorator used across providers.
## Keywords AI inside Cognee
Cognee exposes a single abstraction for observability: `get_observe()` returning the `@observe` decorator. The Keywords AI integration is provided via the `cognee-community` extension hub as `cognee-community-observability-keywordsai`.
* **About `cognee-community`**: This is Cognee’s extension hub. Adapters for third‑party databases, pipelines, and community-contributed tasks live here, evolving independently from core. Installs stay slim (only what you need), and a predictable layout under `packages/*` keeps providers consistent.
* **Drop-in**: Importing the package patches Cognee’s `get_observe()` to return a Keywords AI–backed decorator when `MONITORING_TOOL=keywordsai`.
* **Decorator mapping**: `@observe` and `@observe(workflow=True)` map to Keywords AI’s `task()` and `workflow()` decorators from `keywordsai-tracing`.
### Installation and Configuration
```bash theme={null}
pip install cognee-community-observability-keywordsai
# Required
export MONITORING_TOOL=keywordsai
export KEYWORDSAI_API_KEY=<your_KeywordsAI_key>
# If your pipeline elsewhere calls LLMs (optional)
export LLM_API_KEY=<your_OpenAI_key>
```
### Minimal Example
```python theme={null}
# 1) Import to patch Cognee
import cognee_community_observability_keywordsai # noqa: F401
# 2) Use Cognee's abstraction
from cognee.modules.observability.get_observe import get_observe
observe = get_observe() # returns Keywords AI decorator when MONITORING_TOOL=keywordsai
# 3) Decorate a task
@observe
def ingest_files(data: list[dict]):
...
# 4) Decorate a workflow
@observe(workflow=True)
async def main():
...
```
Behind the scenes, Cognee’s community adapter:
1. Patches `get_observe()` so it returns a Keywords AI–aware decorator when configured.
2. Initializes telemetry via `KeywordsAITelemetry()` once on import.
3. Wraps tasks with `task()` and workflows with `workflow()` from `keywordsai-tracing`.
## Quick Start
1. Install the integration:
```bash theme={null}
pip install cognee-community-observability-keywordsai
```
2. Configure your environment:
```bash theme={null}
export MONITORING_TOOL=keywordsai
export KEYWORDSAI_API_KEY=<your_KeywordsAI_key>
export LLM_API_KEY=<your_OpenAI_key>
```
3. Import the package early (so `get_observe()` is patched), decorate your tasks/workflows, and run your pipeline.
4. Open your Keywords AI dashboard to inspect spans across tasks and workflows.
## Useful Links
* Get a Keywords AI API key: [Keywords AI Platform](https://platform.keywordsai.co/)
* Community package: [cognee-community/packages/observability/keywordsai](https://github.com/topoteretes/cognee-community/tree/main/packages/observability/keywordsai)
***
Join the conversation on [Discord](https://discord.gg/m63hxKsp4p) and let us know how the Keywords AI integration works for you!
# LangGraph
Source: https://docs.cognee.ai/integrations/langgraph-integration
Add persistent memory to LangGraph agents with Cognee.
Give your [LangGraph](https://langchain-ai.github.io/langgraph/) agents persistent semantic memory that survives across sessions. Store data in cognee's knowledge graph and retrieve it via natural language—no manual state management required.
## Why Use This Integration
* **Cross-Session Memory**: Context persists across agent instances and conversation sessions
* **Semantic Recall**: Retrieve information using natural language queries
* **Session Isolation**: Multi-tenant support with per-user data separation
* **Drop-in tools**: `add_tool` and `search_tool` work with LangGraph agents out of the box
## Installation
```bash theme={null}
pip install cognee-integration-langgraph
```
## Quick Start
Before using the integration, configure your environment variables:
```bash theme={null}
export OPENAI_API_KEY="your-openai-api-key-here" # for LangGraph
export LLM_API_KEY="your-openai-api-key-here" # for cognee
export LLM_MODEL="gpt-4o-mini"
```
Add memory tools to your LangGraph agent. Tools are built per session and the agent must be invoked asynchronously with `ainvoke()`:
```python theme={null}
import asyncio
from langchain.agents import create_agent
from langchain_core.messages import HumanMessage
from cognee_integration_langgraph import get_sessionized_cognee_tools
async def main():
# Build sessionized memory tools (omit the arg to auto-generate a session ID)
add_tool, search_tool = get_sessionized_cognee_tools("user-123")
# Create an agent with memory
agent = create_agent(
"openai:gpt-4o-mini",
tools=[add_tool, search_tool],
)
# Store and retrieve information (note: must use await with .ainvoke())
response = await agent.ainvoke({
"messages": [
HumanMessage(content="Remember: I like pizza and coding in Python")
]
})
print(response["messages"][-1].content)
if __name__ == "__main__":
asyncio.run(main())
```
## Tools
`get_sessionized_cognee_tools(session_id=None, include_persist_tool=False, user=None)` returns the memory tools:
| Tool | Description |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `add_tool` | Stores data in the knowledge base (`data`, optional `node_set`) |
| `search_tool` | Retrieves stored information with natural language (`query_text`, optional `session_id`, `query_type`); defaults to `SearchType.GRAPH_COMPLETION` |
| `persist_sessions_tool` | Promotes conversation sessions into the permanent graph (`session_ids`); returned only when `include_persist_tool=True` |
## Session Management
Pass a `session_id` to isolate memory per user or organization:
```python theme={null}
# User-specific memory
user_tools = get_sessionized_cognee_tools(session_id="user_123")
# Org-specific memory
org_tools = get_sessionized_cognee_tools(session_id="org_acme")
# Generate a unique session automatically
auto_tools = get_sessionized_cognee_tools() # Uses a UUID-based session ID
```
<Info>
Session isolation is implemented by scoping data with `node_set=[session_id]` for `add_tool`, and injecting the `session_id` into `search_tool`. Data added outside a session forms separate clusters.
</Info>
## How It Works
1. **Add Tool**: Stores data in cognee's knowledge graph with embeddings
2. **Search Tool**: Retrieves relevant information via cognee's recall pipeline
3. **Auto-Processing**: cognee extracts entities, relationships, and context automatically
4. **Session Scoping**: Data is organized by session clusters but globally accessible
## Use Cases
<AccordionGroup>
<Accordion title="Knowledge Accumulation">
Build domain knowledge incrementally over multiple sessions:
```python theme={null}
# Add knowledge from sessions
for doc in knowledge_base:
await agent.ainvoke({"messages": [HumanMessage(content=f"Learn: {doc}")]})
# Query across all
response = await agent.ainvoke({
"messages": [HumanMessage(content="Find information about contract terms")]
})
```
</Accordion>
<Accordion title="Context-Aware Assistance">
Maintain user context across work sessions:
```python theme={null}
# Monday
await agent.ainvoke({"messages": [HumanMessage(content="Debugging payment flow")]})
# Wednesday
await agent.ainvoke({"messages": [HumanMessage(content="What was I debugging?")]})
```
</Accordion>
<Accordion title="Multi-Tenant Applications">
Isolate data per user/organization while sharing global knowledge:
```python theme={null}
# Per-user isolation
add_tool, search_tool = get_sessionized_cognee_tools(session_id=user_id)
agent = create_agent("openai:gpt-4o-mini", tools=[add_tool, search_tool])
```
</Accordion>
</AccordionGroup>
***
<CardGroup>
<Card title="GitHub Repository" icon="github" href="https://github.com/topoteretes/cognee-integrations/tree/main/integrations/langgraph">
View source code and examples
</Card>
<Card title="Examples" icon="book" href="https://github.com/topoteretes/cognee-integrations/tree/main/integrations/langgraph/examples">
Runnable example scripts
</Card>
</CardGroup>
# Linear
Source: https://docs.cognee.ai/integrations/linear-integration
Install Cognee as a Linear agent that answers @mentions and delegated issues from memory, and index the workspace's issues.
Connect a Linear **workspace** to Cognee by installing a Linear OAuth app into it as an **agent**. Workspace members can then @mention the agent or delegate an issue to it and get an answer drawn from your Cognee memory, while the workspace's issues flow into a dataset of their own and stay fresh through webhooks.
<Info>
This integration is for self-hosted Cognee deployments — you create and own the Linear OAuth app, so it lives in your own Linear workspace settings and talks only to your own Cognee backend.
</Info>
**Ships in `1.5.4`.** The connector landed after `1.5.3` and is first tagged in v1.5.4, so `cognee>=1.5.4` carries it — no `dev` checkout is needed. See the [changelog](/changelog).
## What It Does
* **Installs as an agent, not a bot user.** The authorize URL carries `actor=app`, which puts an *app user* — the agent's identity — into the workspace. The `app:assignable` and `app:mentionable` scopes are what let members delegate issues to it and @mention it. Requested scopes are `read,write,app:assignable,app:mentionable`.
* **Answers agent sessions from memory.** A mention or a delegation opens an agent session; Cognee runs a [`HYBRID_COMPLETION`](/python-api/search-type) search across **every dataset the connecting user can read** — not just the Linear one — and replies inside the session.
* **Indexes the workspace's issues.** Issue creates and updates are remembered as plain text in one dataset per workspace, plus a backfill of recent issues at install time.
* **One workspace, one Cognee user.** The credential is owned by whoever completed the install, and answers come from that user's memory.
## Prerequisites
* Cognee **`>= 1.5.4`** — see the version note above.
* A running Cognee backend reachable over **HTTPS from the public internet**: Linear delivers webhooks and the OAuth redirect to it and will not call `localhost`. For local development use a tunnel (ngrok or similar) — see the [Slack integration's ngrok section](/integrations/slack-integration#local-development-ngrok-tunnel), which applies verbatim; for production this is your deployment's normal public domain.
* Permission to create an OAuth application in the target Linear workspace.
## Setup
### 1. Create the Linear OAuth app
Go to [linear.app/settings/api/applications](https://linear.app/settings/api/applications) → create a new application and configure:
| Setting | Value |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Callback URL** | `https://<public-host>/api/v1/integrations/linear/callback` — must match `LINEAR_REDIRECT_URI` exactly. |
| **Agent capabilities** | **Enabled.** Cognee installs with `actor=app`, so the app must be allowed to install an app user. Without it the `app:assignable` / `app:mentionable` scopes are not grantable. |
| **Webhook URL** | `https://<public-host>/api/v1/integrations/linear/events` |
| **Webhook events** | **Agent session events** (the point of the integration), **Issues**, and **App revoked (OAuth)**. |
| **Webhook signing secret** | Generated by Linear — this becomes `LINEAR_WEBHOOK_SECRET`. |
Then collect the app's **Client ID** and **Client secret**.
<Warning>
Create a **separate app per environment** (local dev, staging, production). One app has one webhook URL, so pointing a shared app at a laptop routes another environment's deliveries there.
</Warning>
### 2. Configure Cognee
Copy the app's values into your backend's `.env` (the same block is in `.env.template`):
| Variable | Value |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `LINEAR_CLIENT_ID` | From the app's settings page. |
| `LINEAR_CLIENT_SECRET` | From the app's settings page. |
| `LINEAR_WEBHOOK_SECRET` | The app's webhook signing secret. It verifies every inbound delivery's `Linear-Signature` **and** signs the OAuth `state` parameter — both server-side-only uses of the same secret. |
| `LINEAR_REDIRECT_URI` | The callback URL registered on the app, e.g. `http://localhost:8000/api/v1/integrations/linear/callback`. It is sent on both the authorize request and the token exchange, so it must match the app's registered value exactly. |
| `LINEAR_FRONTEND_BASE_URL` | Your frontend's origin, e.g. `http://localhost:3000` — where the callback redirects the browser when the install finishes. |
| `INTEGRATION_CREDENTIALS_KEY` | Not Linear-specific — the AES-256-GCM key (32 raw bytes, base64-encoded) the integrations framework encrypts stored credentials under. Unlike GitHub, Linear stores a real access token in that payload, so this key is what protects it at rest. Generate one with `openssl rand -base64 32`. `INTEGRATION_CREDENTIALS_KEYS` (a JSON keyring) is the preferred, rotation-capable form. |
Every one of these is checked at use time, not at boot: a deployment with Linear unconfigured still starts normally, and the first call that needs a missing value fails with `LINEAR_<NAME> is not configured` rather than a confusing downstream error. `POST /authorize` turns that into a **503** (`linear integration is not configured on this server`) instead of a 500.
Restart the backend after any `.env` change.
### 3. Install and connect
The backend ships without frontend wiring for Linear in this release — the aggregate `GET /api/v1/integrations/status` reports the provider generically, but there is no **Connect** button yet. Drive the flow through the API:
```bash theme={null}
# Mint the authorize URL (authenticated as the Cognee user who will own the data)
curl -X POST https://<public-host>/api/v1/integrations/linear/authorize \
-H "Authorization: Bearer <your-cognee-token>"
# {"authorizeUrl":"https://linear.app/oauth/authorize?...&actor=app&scope=read,write,app:assignable,app:mentionable"}
```
Open that URL in a browser, pick the workspace, and approve. Linear redirects back to `/api/v1/integrations/linear/callback`, which binds the workspace to your Cognee account and then sends the browser to `<LINEAR_FRONTEND_BASE_URL>/integrations?linear=connected`.
<Warning>
**Install from the minted URL.** The signed `state` is what binds the install to a Cognee account, and it expires **10 minutes** after `/authorize` returns. An install started without it (or after it expires) lands on `?linear=error_invalid_state` and stores nothing.
</Warning>
The callback does one extra round trip before it redirects: Linear's token response says nothing about *which* workspace authorized the app, but every webhook delivery routes by its `organizationId`, so the fresh token is immediately spent on a GraphQL query for `viewer` (the newly installed app user) and `organization`. The organization id becomes the credential's account id; the organization name becomes its display label.
The initial issue backfill is kicked off **detached**, after the redirect — the browser returns immediately.
## What Gets Indexed
Issues land in **one dataset per workspace**, named `linear_<url_key>`: the workspace's Linear URL key (the slug in `linear.app/<url_key>`) lowercased, with every run of characters outside `A-Z a-z 0-9 _` collapsed to `_`. A workspace at `linear.app/acme-corp` becomes `linear_acme_corp`. One dataset per workspace rather than per team or per issue means [backend access control](/core-concepts/multi-user-mode/multi-user-mode-overview) isolates at the workspace boundary — per-issue datasets would mean one isolated database per issue.
Each issue is stored as a deterministic plain-text rendering — identifier, title, URL, state, description, and nothing volatile — so re-syncing an unchanged issue produces byte-identical text instead of churning the graph.
```
Linear issue ENG-42: Rate limiter drops bursts
URL: https://linear.app/acme-corp/issue/ENG-42
State: In Progress
Description: ...
```
At install time Cognee backfills the workspace's **50 most recently updated issues** in a single [`remember()`](/python-api/remember) call — webhooks only cover changes from then on, so this is what gives the agent something to recall from on day one. Because it is ordinary text ingestion, the content is reachable through the normal search types, and you can query the dataset directly:
```python theme={null}
import cognee
results = await cognee.recall(
"What is the status of the rate limiter work?",
datasets=["linear_acme_corp"],
)
```
<Note>
Both the backfill and per-issue webhook syncs run with `self_improvement=False`, so they never trigger an [`improve()`](/python-api/improve) pass. Whole-graph enrichment carries LLM cost that is far too heavy to fire on every issue edit — it stays a human or scheduled decision.
</Note>
## Answering Agent Sessions
An agent session opens when a member **@mentions** the agent (`AgentSessionEvent` / `created`) or **delegates an issue** to it, and continues when they reply inside the session (`prompted`). For each turn Cognee:
1. **Posts a `thought` activity immediately** — "Searching cognee memory…" — before any search or LLM work. Linear requires an activity within **10 seconds** of a `created` event or it marks the session unresponsive, and a `HYBRID_COMPLETION` search routinely takes longer than that.
2. **Builds the query.** Linear's own `promptContext` (its digest of the issue and prior comments) and workspace-level `guidance` are prepended to the user's question, so retrieval is grounded in what the session is about. On a `prompted` event the question is the new message; on a `created` event it is the triggering comment, or — for a delegation, which has no comment — the issue's title and description.
3. **Searches memory** with `HYBRID_COMPLETION` across every dataset the connecting user can read, keeping the top three answers and discarding per-chunk completions that are really refusals ("no relevant information…").
4. **Posts a `response` activity** with the answer, which completes the turn.
Every failure path ends in an **`error` activity** rather than silence, so a session never sits showing the agent as working forever. Two cases answer politely instead of erroring: no question could be found in the session, and nothing relevant in memory ("No relevant information found in cognee memory.").
## Keeping It in Sync
Webhook deliveries to `POST /api/v1/integrations/linear/events` drive everything after the install. Coverage is deliberately narrow in this first cut:
| Event | Behavior |
| ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `AgentSessionEvent` (`created`, `prompted`) | Answers the session from memory — see above. |
| `Issue` (`create`, `update`) | Remembers the issue's current text into the workspace dataset. |
| `OAuthApp` (`revoked`) | Revokes the stored credential: the workspace removed the app on Linear's side, so the local token stops being used. Idempotent, so a redelivery is harmless. |
Everything else — **issue deletions included** — is logged and dropped. Deleting indexed data on a webhook would be a silent, destructive surprise, so [`forget()`](/python-api/forget) stays a human decision.
A delivery for a workspace Cognee doesn't know (or has revoked) is logged and dropped rather than erroring — Linear gives no ordering guarantee, so a delivery racing ahead of the callback storing the credential is a normal, self-healing state.
### Webhook Security
Every delivery is authenticated by a hex **HMAC-SHA256 over the raw request body**, keyed with `LINEAR_WEBHOOK_SECRET` and compared against the `Linear-Signature` header with a constant-time comparison. That signature is the entire auth model for the route — it is unauthenticated by design, since Linear cannot send a bearer token. A bad or missing signature is a **401**.
Linear also carries replay protection, which Cognee enforces: the payload's `webhookTimestamp` (Unix milliseconds) must be within **60 seconds** of the server's clock, or the delivery is rejected as `Stale Linear webhook timestamp` — also a **401**. A missing or malformed timestamp is rejected the same as a stale one, and the check runs **only after the HMAC passes**, since a timestamp read from unverified bytes proves nothing.
Two consequences worth planning for:
* **Keep the Cognee host's clock in sync** (NTP). More than a minute of drift rejects every delivery even though its signature is perfect.
* **Rotating the signing secret is a two-sided change.** Deliveries signed with the old secret fail the HMAC, so rotate on Linear and update `LINEAR_WEBHOOK_SECRET` together and restart the backend. Because the same secret also signs the OAuth `state`, an install that is mid-flight during a rotation lands on `?linear=error_invalid_state`; just mint a fresh authorize URL.
Cognee acks the delivery as soon as both checks pass and handles it detached, so Linear's delivery timeout is never in play no matter how long a search takes. The 10-second rule is separate — it is Linear's clock on the *session*, which only the `thought` activity satisfies.
## Disconnect vs. Uninstall
Neither deletes indexed data:
* **Disconnect in Cognee** — `DELETE /api/v1/integrations/linear/connection` — marks the stored credential revoked, so agent sessions and issue syncs stop. It also makes a **best-effort remote revoke**: Linear exposes a cheap token-revoke endpoint that kills only Cognee's own token without touching the app install, so unlike the GitHub connector this is actually called. It is best effort by design — the local revoke is the real cut-off, and a network blip must never block a disconnect.
* **Removing the app in Linear** fires the `OAuthApp` `revoked` webhook, which revokes the credential server-side — the same end state, reached from the other side.
The `linear_<url_key>` dataset stays exactly as it is in both cases. To remove the indexed data itself, use [`forget()`](/python-api/forget).
## Troubleshooting
1. **`POST /authorize` returns 503 "linear integration is not configured on this server"**: `LINEAR_CLIENT_ID`, `LINEAR_REDIRECT_URI`, or `LINEAR_WEBHOOK_SECRET` (it signs the state) is unset. The backend log names the exact variable. A missing `LINEAR_FRONTEND_BASE_URL` surfaces later, as the same 503 from the callback instead.
2. **Redirected to `?linear=error_invalid_state`**: the install didn't start from a freshly minted `/authorize` URL, more than 10 minutes passed before approving it, or `LINEAR_WEBHOOK_SECRET` changed in between. Mint a new one.
3. **Redirected to `?linear=error_exchange_failed`**: the callback never 500s — it catches everything and redirects, and the traceback is in the **backend log**. Usual causes: mismatched `LINEAR_CLIENT_ID`/`LINEAR_CLIENT_SECRET`, a `LINEAR_REDIRECT_URI` that doesn't exactly match the app's registered callback URL, a missing `INTEGRATION_CREDENTIALS_KEY`, or the identity query failing (`Linear token response carries no organization id`).
4. **Redirected to `?linear=error_already_connected`**: this workspace is bound to another Cognee user. Disconnect it there first.
5. **Redirected to `?linear=cancelled`**: the consent screen was cancelled, or Linear rejected the request. Nothing was stored.
6. **Connect succeeds but no issues appear**: the backfill runs detached — check the backend log for `Syncing N Linear issues for organization <id> into dataset linear_<url_key>`, or `has no issues to sync`. It covers the 50 most recently updated issues only; everything older arrives when those issues are next touched.
7. **Webhook deliveries return 401**: either `LINEAR_WEBHOOK_SECRET` doesn't match the app's signing secret (easy to hit when juggling more than one app), or the host clock has drifted more than 60 seconds. The response detail distinguishes the two — `Invalid Linear signature` versus `Stale Linear webhook timestamp`.
8. **Webhook deliveries return 404**: the provider name in the URL is wrong, or the backend is running a build without the Linear connector registered. The route is `/api/v1/integrations/linear/events`.
9. **The agent posts an error activity in the session**: the search or the reply failed — the backend log carries the traceback under `Linear agent session <id>: answering failed`. If the answer instead says the connection "is not fully configured", the Cognee user who installed the integration was deleted; disconnect and reconnect.
10. **The agent replies "No relevant information found in cognee memory."**: the search returned nothing usable for that question. Remember that it searches the installing user's datasets — content ingested by a different Cognee user is not visible to it.
## Related
<CardGroup>
<Card title="Slack" icon="messages-square" href="/integrations/slack-integration">
Ask and save memory from a conversation, sharing the same credential store and connect flow.
</Card>
<Card title="GitHub" icon="github" href="/integrations/github-integration">
The other first-party webhook connector — an org's repositories into the code graph.
</Card>
</CardGroup>
# n8n
Source: https://docs.cognee.ai/integrations/n8n-integration
Give your n8n workflows memory with Cognee — no code required.
Cognee gives your [n8n](https://n8n.io/) workflows a memory. Store notes, documents, and messages, let Cognee organize them, and then ask questions about them in plain English — all from n8n's visual editor, without writing any code.
A few things people build with it:
* A **support bot** that answers questions from your help docs
* A **research assistant** that remembers everything you feed it from RSS feeds or web pages
* A **sales helper** that recalls what you know about a customer before a call
## Install the Cognee node
The Cognee node is a free n8n community node:
1. Log in to your self-hosted n8n
2. Go to **Settings → Community Nodes**
3. Click **Install**, search for `n8n-nodes-cognee`, and confirm
<Info>
Don't have n8n yet? Follow n8n's [Docker setup guide](https://docs.n8n.io/hosting/installation/docker/#starting-n8n), then open n8n at `localhost:5678`.
</Info>
## Connect your Cognee account
1. Sign in to [Cognee Cloud](https://platform.cognee.ai) and copy your **Base URL** and **API key** from the API Keys page
2. In n8n, add the Cognee node to a workflow (click **+** and search for "cognee")
3. When asked for credentials, create a new **Cognee API** credential and paste in your Base URL and API key
<Info>
Enter the Base URL exactly as shown in your dashboard (for example `https://tenant-xxx.aws.cognee.ai`) — don't add anything after it.
</Info>
## Your first memory workflow
A memory workflow has three steps: store something, let Cognee organize it, then ask a question about it. Each step is one Cognee node.
### 1. Add Data — store something
* **Resource**: Add Data
* **Dataset Name**: `my-dataset` (a dataset is like a folder for related content)
* **Text Data**: any text you want Cognee to remember
### 2. Cognify — organize it into memory
* **Resource**: Cognify
* **Datasets**: `my-dataset`
This is where Cognee reads what you stored and connects the facts inside it, so it can answer questions later.
### 3. Search — ask a question
* **Resource**: Search
* **Search Type**: GraphCompletion
* **Datasets**: `my-dataset`
* **Query**: a plain-English question, like "Tell me about NLP"
Run the workflow, and the Search node returns an answer based on what you stored.
<Info>
Everything you add from n8n also appears in your [Cognee Cloud](/cognee-cloud/overview) workspace, and n8n can search data you've already added there.
</Info>
## Start from a template
Prefer not to build it by hand? Copy the template below, click an empty spot on your n8n canvas, and paste (**Cmd+V** / **Ctrl+V**). The three connected nodes appear, ready to run — you don't need to read or edit the JSON, just copy and paste it.
1. Copy the template (use the copy button in the corner of the block)
2. Paste it onto your n8n canvas
3. Open each Cognee node once and select your **Cognee API** credential
4. Click **Execute workflow**, then check the Search node's output for the answer
<Accordion title="Copy-paste workflow template">
```json theme={null}
{
"name": "Cognee memory quickstart",
"nodes": [
{
"name": "Start",
"type": "n8n-nodes-base.manualTrigger",
"typeVersion": 1,
"position": [0, 0],
"parameters": {}
},
{
"name": "Add Data",
"type": "n8n-nodes-cognee.cognee",
"typeVersion": 1,
"position": [220, 0],
"parameters": {
"resource": "addData",
"operation": "add",
"datasetName": "my-dataset",
"textData": [
"FAQ: You can reset your password from the account settings page.",
"Guide: To export your data, open the dashboard and choose Export as CSV."
]
}
},
{
"name": "Cognify",
"type": "n8n-nodes-cognee.cognee",
"typeVersion": 1,
"position": [440, 0],
"parameters": {
"resource": "cognify",
"operation": "cognify",
"datasets": ["my-dataset"]
}
},
{
"name": "Search",
"type": "n8n-nodes-cognee.cognee",
"typeVersion": 1,
"position": [660, 0],
"parameters": {
"resource": "search",
"operation": "search",
"searchType": "GRAPH_COMPLETION",
"datasets": ["my-dataset"],
"query": "How do I export my data?",
"topK": 10
}
}
],
"connections": {
"Start": {
"main": [[{ "node": "Add Data", "type": "main", "index": 0 }]]
},
"Add Data": {
"main": [[{ "node": "Cognify", "type": "main", "index": 0 }]]
},
"Cognify": {
"main": [[{ "node": "Search", "type": "main", "index": 0 }]]
}
}
}
```
</Accordion>
The template stores two short support answers, organizes them into memory, and asks "How do I export my data?" — swap in your own text and questions once it runs.
## What each operation does
| Operation | What it does |
| --------- | ------------------------------------------------------------------------------------ |
| Add Data | Stores text in a dataset |
| Cognify | Organizes stored data into searchable memory |
| Search | Answers questions about your data in plain English |
| Delete | Removes a dataset or a single item |
| Skill | Advanced: manages self-improving agent skills (requires a self-hosted Cognee server) |
## Picking a search type
| Search type | When to use it |
| --------------- | ------------------------------------------------------------------------------ |
| GraphCompletion | The best all-round choice — start here |
| ChainOfThought | Complicated questions that need step-by-step reasoning (slower, more thorough) |
| RagCompletion | Quick lookups of simple facts |
Learn more about how search works in the [search basics guide](/guides/search-basics).
## Troubleshooting
| Problem | What to check |
| ---------------------- | -------------------------------------------------------------------------------------- |
| "Unauthorized" error | Your API key is copied correctly and still active |
| Can't connect | The Base URL matches your dashboard exactly, and your n8n instance has internet access |
| Search returns nothing | You ran Cognify after adding data |
***
<CardGroup>
<Card title="GitHub Repository" icon="github" href="https://github.com/topoteretes/cognee-integrations/tree/main/integrations/n8n">
View source code and contribute
</Card>
<Card title="n8n Documentation" icon="book" href="https://docs.n8n.io/integrations/community-nodes/">
Learn more about n8n community nodes
</Card>
</CardGroup>
# OpenAI Agents SDK
Source: https://docs.cognee.ai/integrations/openai-agents-sdk-integration
Add persistent memory to OpenAI Agents SDK agents with Cognee.
Give your [OpenAI Agents SDK](https://github.com/openai/openai-agents-python) agents structured memory that persists across sessions.
## Why Use This Integration
* **Persistent Memory**: Knowledge survives agent restarts and session boundaries
* **Advanced Retrieval**: Natural language queries backed by graph + vector search
* **Sessionized tools**: Built in session data organization in memory
* **Native SDK Tools**: Works directly with OpenAI's `function_tool` pattern
* **Async-First**: Built for high-performance applications with proper concurrency handling
## Installation
```bash theme={null}
pip install cognee-integration-openai-agents
```
## Quick Start
Configure your environment variables:
```bash theme={null}
export OPENAI_API_KEY="your-openai-api-key-here" # for OpenAI Agents SDK
export LLM_API_KEY="your-openai-api-key-here" # for cognee
```
Add memory tools to your agent:
```python theme={null}
import asyncio
from agents import Agent, Runner
from cognee_integration_openai_agents import add_tool, search_tool
async def main():
# Create agent with memory
agent = Agent(
name="assistant",
instructions="You are a helpful assistant with access to a knowledge base.",
tools=[add_tool, search_tool],
)
# Store information
await Runner.run(
agent,
"Remember: Acme Corp, healthcare, $1.2M contract"
)
# Retrieve information
result = await Runner.run(
agent,
"What healthcare contracts do we have?"
)
print(result.final_output)
asyncio.run(main())
```
## Cross-Session Persistence
Memory persists across different agent instances:
```python theme={null}
from agents import Agent, Runner
from cognee_integration_openai_agents import get_sessionized_cognee_tools
# Session 1: Store information
add_tool, search_tool = get_sessionized_cognee_tools("user-123")
agent_1 = Agent(
name="assistant",
instructions="You are a helpful assistant.",
tools=[add_tool, search_tool],
)
await Runner.run(agent_1, "I'm working on authentication")
# Session 2: Different instance, same memory
add_tool, search_tool = get_sessionized_cognee_tools("user-123")
agent_2 = Agent(
name="assistant",
instructions="You are a helpful assistant.",
tools=[add_tool, search_tool],
)
result = await Runner.run(agent_2, "What was I working on?")
# Returns: "authentication"
```
## Custom Session Management
Control session isolation with custom session IDs:
```python theme={null}
# User-specific memory
add_tool, search_tool = get_sessionized_cognee_tools(session_id="user_123")
# Org-specific memory
add_tool, search_tool = get_sessionized_cognee_tools(session_id="org_acme")
# Generate unique session automatically
add_tool, search_tool = get_sessionized_cognee_tools() # Uses UUID-based session ID
```
<Info>
Sessionized tools scope data to their session ID. Use non-sessionized `add_tool` and `search_tool` to access data across all sessions.
</Info>
## How It Works
1. **Add Tool**: Stores data in cognee's knowledge graph with embeddings
2. **Search Tool**: Retrieves relevant information via cognee's recall pipeline
3. **Auto-Processing**: cognee extracts entities, relationships, and context automatically
4. **Session Scoping**: Sessionized tools filter by session; non-sessionized tools access everything
## Use Cases
<AccordionGroup>
<Accordion title="Knowledge Accumulation">
Pre-load documents into cognee before creating agents:
```python theme={null}
import cognee
from agents import Agent, Runner
from cognee_integration_openai_agents import search_tool
# Pre-load documents
for doc_path in document_paths:
with open(doc_path, 'r') as f:
await cognee.remember(f.read())
# Query across all documents
agent = Agent(
name="analyst",
instructions="You have access to our knowledge base.",
tools=[search_tool]
)
result = await Runner.run(agent, "Find information about contract terms")
```
</Accordion>
<Accordion title="Context-Aware Assistance">
Maintain user context across work sessions:
```python theme={null}
# Monday
await Runner.run(agent, "Debugging payment flow")
# Wednesday
result = await Runner.run(agent, "What was I debugging?")
```
</Accordion>
<Accordion title="Multi-Tenant Applications">
Isolate data per user or organization:
```python theme={null}
add_tool, search_tool = get_sessionized_cognee_tools(session_id=user_id)
agent = Agent(
name="assistant",
instructions="You are a helpful assistant.",
tools=[add_tool, search_tool]
)
```
</Accordion>
<Accordion title="Multi-Agent Workflows">
Share knowledge between specialized agents:
```python theme={null}
from cognee_integration_openai_agents import add_tool, search_tool
# Data collector agent
data_agent = Agent(
name="data_collector",
instructions="You collect and store important information.",
tools=[add_tool]
)
# Research agent
research_agent = Agent(
name="researcher",
instructions="You search and analyze the knowledge base.",
tools=[search_tool]
)
```
</Accordion>
<Accordion title="Agent Handoffs">
Route requests to specialized agents with shared memory:
```python theme={null}
storage_agent = Agent(
name="storage_specialist",
instructions="You specialize in storing information.",
tools=[add_tool]
)
search_agent = Agent(
name="search_specialist",
instructions="You specialize in finding information.",
tools=[search_tool]
)
triage_agent = Agent(
name="triage",
instructions="Route to storage_specialist for saving, search_specialist for queries.",
handoffs=[storage_agent, search_agent]
)
result = await Runner.run(triage_agent, "Find our healthcare contracts")
```
</Accordion>
</AccordionGroup>
***
<CardGroup>
<Card title="PyPI Package" icon="python" href="https://pypi.org/project/cognee-integration-openai-agents/">
Install the integration package
</Card>
<Card title="Integrations Overview" icon="book" href="/integrations/index">
Explore other cognee integrations
</Card>
</CardGroup>
# Cognee Plugin for OpenClaw
Source: https://docs.cognee.ai/integrations/openclaw-integration
Add persistent memory to OpenClaw agents with a Cognee plugin.
Give your [OpenClaw](https://github.com/openclaw/openclaw) agents Cognee-backed memory with **multi-scope support** (company / user / agent), session tracking, and automatic recall. The plugin indexes your Markdown memory files, recalls relevant context before each run, and searches across sessions with natural language.
## Why Use This Integration
* **Multi-scope memory**: Separate datasets for company-wide knowledge, per-user preferences, and per-agent context, routed automatically by file path
* **Auto-recall**: Relevant memories are injected as labeled `<cognee_memories>` context before every prompt
* **Auto-index**: Memory files sync to Cognee (add new, update changed, forget removed, skip unchanged)
* **Session tracking**: Each turn is captured in Cognee's session cache and bridged into the graph on session end
* **14 search types**: From semantic vector search to chain-of-thought graph reasoning
* **One-command setup**: `openclaw cognee setup` configures Cognee as the memory provider
## Installation
Once published, pin to an exact version (supply-chain best practice):
```bash theme={null}
openclaw plugins install @cognee/cognee-openclaw@2026.6.11
```
Or install locally for development:
```bash theme={null}
cd integrations/openclaw
npm install && npm run build
openclaw plugins install -l .
```
## Quick Start
### 1. Start Cognee
Run a local Cognee server with this [minimal Docker Compose file](https://github.com/topoteretes/cognee-integrations/blob/main/integrations/openclaw/cognee-docker-compose.yaml):
```bash theme={null}
export LLM_API_KEY="your-openai-api-key"
docker compose -f ./integrations/openclaw/cognee-docker-compose.yaml up -d
curl http://localhost:8000/health
```
### 2. Run setup
```bash theme={null}
openclaw cognee setup # Cognee only (replaces built-in memory)
openclaw cognee setup --hybrid # Keep built-in memory enabled in config
```
### 3. Configure the connection
Add the Cognee connection to `~/.openclaw/openclaw.json`:
```yaml theme={null}
plugins:
entries:
cognee-openclaw:
enabled: true
hooks:
allowConversationAccess: true # required for after-run file sync — see note
config:
baseUrl: "http://localhost:8000"
apiKey: "${COGNEE_API_KEY}"
datasetName: "my-project"
```
<Warning>
OpenClaw ≥ 2026.4.27 blocks non-bundled plugins from registering the `agent_end` hook unless `hooks.allowConversationAccess: true` is set. Without it, file sync after each agent turn is silently disabled until the next manual `openclaw cognee index` or gateway start. Restart the gateway after adding the flag: `openclaw gateway stop && openclaw gateway start`.
</Warning>
That's it. Your OpenClaw memory files are now backed by Cognee's knowledge graph. If you omit `apiKey`, the plugin auto-logs in with the default local credentials (`default_user@example.com` / `default_password`).
## Cognee Cloud
To use [Cognee Cloud](/cognee-cloud/overview) instead of a local instance, set `mode` to `"cloud"`:
```yaml theme={null}
plugins:
entries:
cognee-openclaw:
enabled: true
config:
mode: "cloud"
baseUrl: "https://tenant-xxx.aws.cognee.ai/api"
apiKey: "${COGNEE_API_KEY}"
```
<Info>
Cloud mode supports `remember` (new files), `recall`, and per-item `forget`. Updating an existing file in place is **not** supported in cloud mode (`PATCH /update` is self-hosted only) — delete and re-add the file instead.
</Info>
## Multi-Scope Memory
For production use, enable multi-scope mode by setting any scope-specific dataset name. Memory files are routed to the right dataset by path.
```yaml theme={null}
plugins:
entries:
cognee-openclaw:
enabled: true
config:
baseUrl: "http://localhost:8000"
apiKey: "${COGNEE_API_KEY}"
# Multi-scope datasets
companyDataset: "acme-shared"
userDatasetPrefix: "acme-user"
agentDatasetPrefix: "acme-agent"
userId: "${OPENCLAW_USER_ID}"
agentId: "code-assistant"
# Search all scopes during recall (in priority order)
recallScopes:
- agent
- user
- company
defaultWriteScope: "agent"
```
| Scope | Purpose | Example files |
| ----------- | ------------------------------------------- | ------------------------------ |
| **Company** | Shared knowledge across all users/agents | `memory/company/policies.md` |
| **User** | Per-user preferences, feedback, corrections | `memory/user/preferences.md` |
| **Agent** | Per-agent learned behaviors, tool outputs | `memory/tools.md`, `MEMORY.md` |
Default routing: `memory/company/**` → company, `memory/user/**` → user, `memory/**` and `MEMORY.md` → agent (catch-all). Override it with a `scopeRouting` list of `{ pattern, scope }` rules.
During recall, each scope is searched independently and injected with labels:
```xml theme={null}
<cognee_memories>
<agent_memory>[agent-specific results]</agent_memory>
<user_memory>[user preference results]</user_memory>
<company_memory>[shared knowledge results]</company_memory>
</cognee_memories>
```
## How It Works
1. **On startup**: Health check, then scan `memory/` and call `/remember` (one batched upload per scope). Cognee runs add + cognify + improve server-side.
2. **Before each prompt**: Call `/recall` for each configured scope in parallel, merge results with scope labels, and inject them as context. The session id is passed through so Cognee captures the turn as a session QA.
3. **After each agent run**: Re-scan memory files — new files batch into `/remember`, changed files go through `PATCH /update` (self-hosted), removed files are dropped via `/forget`.
4. **On session end**: Final sync sweep; with `improveOnSessionEnd` on, dispatches `/improve` for the ended session to bridge session QAs into the permanent graph.
State is tracked under `~/.openclaw/memory/cognee/` (`datasets.json`, `scoped-sync-indexes.json`, legacy `sync-index.json`). Files are stored using sanitized relative paths (e.g. `MEMORY.md.txt`, `memory.tools.md.txt`).
## Configuration Reference
### Connection
| Option | Type | Default | Description |
| --------- | ------ | ----------------------- | ----------------------------------------------- |
| `baseUrl` | string | `http://localhost:8000` | Cognee API base URL |
| `apiKey` | string | `$COGNEE_API_KEY` | API key (optional; falls back to default login) |
| `mode` | string | `local` | `local` or `cloud` |
### Search
| Option | Type | Default | Description |
| -------------- | ------ | ------------------ | ---------------------------------------- |
| `searchType` | string | `GRAPH_COMPLETION` | Search strategy (see below) |
| `maxResults` | number | `3` | Max memories per scope (sent as `top_k`) |
| `minScore` | number | `0.3` | Minimum relevance score filter |
| `searchPrompt` | string | `""` | System prompt to guide search |
### Automation
| Option | Type | Default | Description |
| --------------------- | ------- | ------- | ------------------------------------------------------------ |
| `autoRecall` | boolean | `true` | Inject memories before agent runs |
| `autoIndex` | boolean | `true` | Sync memory files on startup, after runs, and on session end |
| `improveOnSessionEnd` | boolean | `true` | Bridge session-cache QAs into the graph on `session_end` |
### Timeouts
| Option | Type | Default | Description |
| -------------------- | ------ | -------- | ------------------------------------ |
| `requestTimeoutMs` | number | `60000` | HTTP timeout for general requests |
| `ingestionTimeoutMs` | number | `300000` | HTTP timeout for add/update requests |
<Info>
**Deprecated options** (silently ignored if set): `maxTokens` — use `maxResults`; `autoCognify` and `autoMemify` — now run server-side via `/remember`; `deleteMode` — `/forget` is always a soft delete.
</Info>
### Search Types
The plugin supports all of Cognee's search types, including `GRAPH_COMPLETION` (default), `GRAPH_COMPLETION_COT`, `GRAPH_COMPLETION_CONTEXT_EXTENSION`, `GRAPH_SUMMARY_COMPLETION`, `RAG_COMPLETION`, `TRIPLET_COMPLETION`, `CHUNKS`, `CHUNKS_LEXICAL`, `SUMMARIES`, `TEMPORAL`, `NATURAL_LANGUAGE`, `CYPHER`, `CODING_RULES`, and `FEELING_LUCKY` (auto-selects per query). See [Search Types](/core-concepts/main-operations/legacy-operations/search) for details.
## CLI Commands
```bash theme={null}
openclaw cognee setup # configure Cognee as the memory provider
openclaw cognee setup --hybrid # keep built-ins enabled in config
openclaw cognee index # manually sync memory files
openclaw cognee status # files indexed, dataset info, per-scope breakdown
openclaw cognee health # verify Cognee API connectivity
openclaw cognee scopes # show scope routing for current workspace files
openclaw cognee forget --dataset <name> # wipe a dataset
openclaw cognee forget --everything --confirm # wipe all of this user's data
openclaw cognee improve # bridge captured QAs into the permanent graph
openclaw cognee improve --session-id <id> # scope to one session
```
## Troubleshooting
<AccordionGroup>
<Accordion title="File sync not running after each agent turn">
OpenClaw ≥ 2026.4.27 requires `hooks.allowConversationAccess: true` under the plugin entry for the `agent_end` hook to register. Add it to `~/.openclaw/openclaw.json`, then restart the gateway:
```bash theme={null}
openclaw gateway stop && openclaw gateway start
```
</Accordion>
<Accordion title="Verifying the plugin can reach Cognee">
Check connectivity and sync state:
```bash theme={null}
openclaw cognee health
openclaw cognee status
```
Healthy status shows the dataset ID, indexed file count, and a recent last-sync timestamp. On the Cognee side, confirm the server logs show `Backend server has started` (Docker: `docker compose logs cognee`) before starting OpenClaw.
</Accordion>
<Accordion title="Manual fallback">
Trigger the same sync `autoIndex` runs at any time:
```bash theme={null}
openclaw cognee index
```
If this fails, verify `baseUrl` points to a reachable Cognee server and that your `apiKey` (or default credentials) are valid.
</Accordion>
</AccordionGroup>
***
<CardGroup>
<Card title="GitHub Repository" icon="github" href="https://github.com/topoteretes/cognee-integrations/tree/main/integrations/openclaw">
View source code and examples
</Card>
<Card title="Blog Post" icon="newspaper" href="https://www.cognee.ai/blog/integrations/what-is-openclaw-ai-and-how-we-give-it-memory-with-cognee">
Deep dive into building this plugin
</Card>
<Card title="OpenClaw Docs" icon="book" href="https://docs.openclaw.ai/concepts/memory">
Learn about OpenClaw's memory system
</Card>
</CardGroup>
# OpenTelemetry Tracing
Source: https://docs.cognee.ai/integrations/opentelemetry-tracing
Export Cognee traces, metrics, and logs to OpenTelemetry backends.
Cognee ships a built-in OpenTelemetry (OTEL) layer. It creates OTEL spans for every `@observe`-decorated function, emits [metrics](#metrics) and [log records](#logs) for the core memory operations, and can export all three to any OTLP-compatible backend — Grafana Tempo, Jaeger, Dash0, Dynatrace, Datadog, Honeycomb, and others.
## How It Works
In Cognee core, the `@observe` decorator maps only to OpenTelemetry. When tracing is enabled, it wraps each decorated function in an OTEL span; when tracing is disabled it is a no-op that passes the call straight through. There is no separate Sentry backend to configure. [Langfuse](#langfuse) is supported not as a separate SDK but as one more OTLP destination on this same pipeline — set your Langfuse keys and Cognee derives the OTLP endpoint and auth for you.
The single switch turns on all three signals: enabling tracing also configures a `MeterProvider` for the [memory operation metrics](#metrics) and attaches an [OTel log bridge](#logs) to Cognee's loggers. Failures while setting up metrics or the log bridge are swallowed, so a missing or partial OpenTelemetry install degrades to traces-only rather than breaking your application.
| Feature | OTEL |
| ---------------- | ------------------------------------------ |
| Enabled by | `COGNEE_TRACING_ENABLED=true` |
| Signals | Traces, metrics, logs |
| Data destination | Any OTLP backend (or in-memory, for spans) |
| Span type | OTEL span |
## Installation
OTEL support requires OpenTelemetry dependencies. Install them with the `tracing` extra:
```bash theme={null}
pip install 'cognee[tracing]'
```
The extra installs `opentelemetry-sdk` plus the OTLP gRPC and HTTP exporters. No additional packages are needed for the in-memory buffer.
## Quick Start
### 1. Enable tracing via environment variable
```dotenv theme={null}
COGNEE_TRACING_ENABLED=true
```
That's all that is required to activate in-memory span collection. Spans are buffered in a ring buffer (last 50 traces) and can be read programmatically.
### 2. Export to an OTLP backend (optional)
Point cognee at an OTLP-compatible collector:
```dotenv theme={null}
COGNEE_TRACING_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=https://your-collector:4317
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <token>
OTEL_SERVICE_NAME=my-cognee-service # default: "cognee"
```
<AccordionGroup>
<Accordion title="Grafana Tempo / Grafana Cloud">
```dotenv theme={null}
COGNEE_TRACING_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=https://tempo-us-central1.grafana.net:443
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic <base64(instanceId:token)>
OTEL_SERVICE_NAME=cognee
```
</Accordion>
<Accordion title="Jaeger (local)">
```dotenv theme={null}
COGNEE_TRACING_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
OTEL_SERVICE_NAME=cognee
```
Start Jaeger with the all-in-one image and its OTLP gRPC port (`4317`) exposed.
</Accordion>
<Accordion title="Dash0">
```dotenv theme={null}
COGNEE_TRACING_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=https://ingress.eu-west-1.aws.dash0.com:4317
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <dash0_auth_token>
OTEL_SERVICE_NAME=cognee
```
</Accordion>
<Accordion title="Dynatrace">
```dotenv theme={null}
COGNEE_TRACING_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=https://<environment-id>.live.dynatrace.com/api/v2/otlp/v1/traces
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Api-Token <dynatrace_api_token>
OTEL_SERVICE_NAME=cognee
```
Dynatrace ingests OTLP over HTTP only. Cognee recognizes Dynatrace endpoints and forces the OTLP **HTTP** exporter for them, so you do not need to uninstall the gRPC exporter or set any additional variable (see the note under [Environment Variables Reference](#environment-variables-reference)).
</Accordion>
</AccordionGroup>
### 3. Using an auto-instrumentation agent
If you launch your application with `opentelemetry-instrument` or an APM agent (Datadog, Dash0, Elastic), it configures its own `TracerProvider` before your code runs. Cognee detects this and **attaches its in-memory exporter to the existing provider** instead of creating a new one — so cognee spans appear inside your existing trace alongside other spans from your application.
```bash theme={null}
opentelemetry-instrument python my_app.py
```
No additional Cognee configuration is required beyond enabling tracing. Your external agent or APM still needs its own exporter configuration if you want spans sent to a remote backend; in this mode Cognee attaches its in-memory exporter to the existing provider so `get_last_trace()` and related helpers still work.
## Langfuse
[Langfuse](https://langfuse.com) is wired into the same OTLP pipeline as any other backend — there is no separate Langfuse SDK. Setting your Langfuse keys is enough: Cognee derives the OTLP endpoint and Basic-auth header, and enables tracing for you. LLM calls emitted as **generation** spans render as generations in the Langfuse dashboard without extra instrumentation.
```dotenv theme={null}
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
# Optional; defaults to https://cloud.langfuse.com
LANGFUSE_HOST=https://us.cloud.langfuse.com
```
You do **not** need to set `COGNEE_TRACING_ENABLED` — providing the keys turns tracing on automatically. This is fully opt-in: with no `LANGFUSE_*` keys set, nothing changes.
To keep the keys in place but stop tracing, set `COGNEE_TRACING_ENABLED` to an explicit off value (`false`, `0`, or `no`). That is authoritative: the keys no longer auto-enable tracing, and set before tracing initializes it means zero OTLP traffic. Flipping it off after tracing was already enabled in the process stops new spans and metric recordings, but the already-attached log bridge and periodic metric reader keep exporting until you call `disable_tracing()`.
<Warning>
`LANGFUSE_PUBLIC_KEY` and `LANGFUSE_SECRET_KEY` must be provided together. If only one is set, Cognee raises a `ValueError` at startup to prevent a misconfigured exporter.
</Warning>
When the keys are set, Cognee:
* Derives the OTLP endpoint as `{LANGFUSE_HOST}/api/public/otel/v1/traces`.
* Derives the auth header `Authorization=Basic <base64(public_key:secret_key)>`.
* Resolves the host from `LANGFUSE_HOST`, falling back to `LANGFUSE_BASE_URL` when `LANGFUSE_HOST` is unset, and finally to `https://cloud.langfuse.com`.
* Forces the OTLP **HTTP** exporter, because Langfuse ingests OTLP over HTTP only (see the note under [Environment Variables Reference](#environment-variables-reference)).
<Note>
Explicit `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_EXPORTER_OTLP_HEADERS` take precedence: if either is already set, Cognee does not overwrite it with the Langfuse-derived value. This lets you route Langfuse traffic through a collector or override the endpoint while still using the `LANGFUSE_*` keys.
</Note>
Langfuse support requires the `tracing` extra (`pip install 'cognee[tracing]'`). A runnable example lives at `examples/guides/langfuse_telemetry.py`.
## Programmatic API
You can also control tracing from Python:
```python theme={null}
from cognee.modules.observability.trace_context import (
enable_tracing,
disable_tracing,
is_tracing_enabled,
get_last_trace,
get_all_traces,
clear_traces,
)
# Enable with optional console output for debugging
enable_tracing(console_output=True)
# ... run cognee operations ...
trace = get_last_trace()
if trace:
print(trace.summary())
# {'operation': 'cognee.observe.main', 'total_duration_ms': 1234.5,
# 'span_count': 12, 'breakdown': {...}, 'errors': []}
for span in trace.spans():
print(span["name"], span["duration_ms"])
disable_tracing()
```
`enable_tracing()` sets up all three signals — the `TracerProvider` with the in-memory span buffer, the `MeterProvider` for [metrics](#metrics), and the [log bridge](#logs) — and `console_output=True` routes spans, metrics, and log records to the console. `disable_tracing()` shuts all three down. You do not need to call either when `COGNEE_TRACING_ENABLED=true`: the first traced operation initializes them lazily.
### `CogneeTrace` API
| Method | Returns | Description |
| ----------- | ------------ | ---------------------------------------------------------- |
| `spans()` | `list[dict]` | Flat list of span dicts sorted by start time |
| `summary()` | `dict` | Root operation, total duration, per-span breakdown, errors |
| `tree()` | `dict` | Hierarchical span tree as nested dicts |
Each span dict contains: `name`, `trace_id`, `span_id`, `parent_span_id`, `start_time_ns`, `end_time_ns`, `duration_ms`, `status`, `attributes`.
## Memory Operation Spans
Beyond the `@observe`-derived spans, the four core memory operations emit spans named after the memory-semconv v0.1.0 operations, so a single dashboard can chart Cognee alongside any other memory system that follows the same convention:
| Span | Emitted by | `memory.operation` |
| ----------------- | ----------- | ------------------ |
| `memory.store` | `add()` | `store` |
| `memory.process` | `cognify()` | `process` |
| `memory.retrieve` | `search()` | `retrieve` |
| `memory.delete` | `forget()` | `delete` |
<Warning>
These span names replace the previous `cognee.api.search`, `cognee.api.cognify`, and `cognee.api.forget` names. Dashboards, saved queries, and alerts that match on the old names must be updated. Span *attributes* are backward compatible — the `cognee.*` keys are still set alongside the new `memory.*` keys.
</Warning>
The `memory.process`, `memory.retrieve`, and `memory.delete` spans wrap their operation, so their duration is the operation's duration. On `add()` the `memory.store` span records the operation's attributes only; use the [`memory.operation.duration`](#metrics) metric for end-to-end `add()` latency.
## Span Attributes
### memory-semconv attributes
Set on the [memory operation spans](#memory-operation-spans) above:
| Attribute | Description |
| --------------------- | -------------------------------------------------------------------------------------------- |
| `memory.system` | Always `cognee` |
| `memory.operation` | `store`, `process`, `retrieve`, or `delete` |
| `memory.collection` | Dataset name on `memory.store`, defaulting to `main_dataset` when no dataset name was passed |
| `memory.query.text` | Search query text, truncated to the first 500 characters |
| `memory.query.type` | Search type enum value used by retrieval |
| `memory.result.count` | Number of results the search returned after permission filtering |
<Note>
Both query-text attributes — `memory.query.text` and `cognee.search.query` — carry only the first 500 characters of the query. This caps attribute cardinality and limits how much user input reaches your telemetry backend, but it is a truncation, not a redaction: whatever appears in the first 500 characters of a query is exported verbatim. Treat trace data as containing user input and apply your backend's retention and access controls accordingly.
</Note>
### Cognee attributes
Cognee sets the following semantic attributes on spans:
| Attribute | Description |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `cognee.span.category` | Value of `as_type=` passed to `@observe` |
| `cognee.llm.model` | LLM model name |
| `cognee.llm.provider` | LLM provider |
| `cognee.pipeline.stage` | Pipeline stage that issued the LLM call — `extraction`, `summarization`, or `query`; present on LLM spans made within a stage boundary regardless of whether [per-stage routing](/setup-configuration/llm-providers#per-stage-model-routing) overrides are set |
| `cognee.search.type` | Search type enum value used by retrieval |
| `cognee.search.query` | Search query text used by retrieval |
| `cognee.pipeline.task_name` | Pipeline task name |
| `cognee.vector.collection` | Vector collection name |
| `cognee.db.system` | Database backend identifier |
### Generation spans
Spans for LLM calls (functions decorated with `@observe(as_type="generation")`) additionally carry the OTel-GenAI semantic conventions plus Langfuse's observation attributes, so any OTLP backend — Langfuse included — renders them as generations. These spans are also marked `SpanKind.CLIENT` (other spans are `SpanKind.INTERNAL`).
| Attribute | Description |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `gen_ai.request.model` | Requested LLM model name (omitted when the adapter reports no model, e.g. local llama.cpp) |
| `gen_ai.system` | LLM provider name, lowercased |
| `langfuse.observation.type` | Always `generation` on these spans, so they are classified correctly even when the model name is unavailable |
| `langfuse.observation.input` | JSON of the call's string prompt arguments; secret-redacted and truncated to 8000 characters |
| `langfuse.observation.output` | The LLM response; secret-redacted and truncated to 8000 characters |
## Metrics
Enabling tracing also registers memory-semconv v0.1.0 metric instruments on a `cognee` meter. Cognee records the following instruments from the core memory operations:
| Instrument | Type | Unit | Recorded by |
| --------------------------- | --------- | ---------- | ----------------------------------------------------------- |
| `memory.operation.duration` | Histogram | `ms` | `add()`, `cognify()`, `search()`, `forget(everything=True)` |
| `memory.items.stored` | Counter | `{item}` | `add()` |
| `memory.items.retrieved` | Counter | `{item}` | `search()` |
| `memory.query.result.count` | Histogram | `{item}` | `search()` |
| `memory.vector.searches` | Counter | `{search}` | `search()` |
| `memory.items.deleted` | Counter | `{item}` | `forget(everything=True)` |
Every recording carries `memory.system` and `memory.operation` as attributes; `add()` also attaches `memory.collection`, and the `search()` instruments also attach `memory.query.type` — so you can break latency and result counts down per search type without touching span data.
<Note>
Two counting details are worth knowing before you build alerts on these. `memory.items.stored` is derived from the length of the `data` argument when it is sized, and `1` otherwise — pass a list to `add()` to get a meaningful per-item count. `forget()` records metrics only on the delete-everything path, where `memory.items.deleted` counts datasets removed; dataset-scoped forget calls record no metrics at all, not even a duration.
</Note>
Four further instruments are registered and exported but not yet recorded by Cognee itself — `memory.data.bytes.stored` (`By`), `memory.graph.nodes.added` (`{node}`), `memory.graph.edges.added` (`{edge}`), and `memory.operation.errors` (`{error}`). They exist so custom instrumentation can feed the same metric names via the helpers in `cognee.modules.observability` (`increment_bytes_stored`, `increment_graph_nodes`, `increment_graph_edges`, `increment_operation_errors`). Expect no data on them out of the box.
### Exporting metrics
Unlike spans, metrics have no in-memory buffer: they are only collected when a reader is attached. Set `OTEL_EXPORTER_OTLP_ENDPOINT` to export them, or pass `console_output=True` to `enable_tracing()` to print them locally. With neither set, the instruments are registered but nothing is collected.
* The metrics endpoint defaults to your traces endpoint with `/v1/traces` rewritten to `/v1/metrics`; if the endpoint contains no `/v1/traces` path it is used as-is. Override it with `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`.
* OTLP metrics are exported every 30 seconds; the console reader exports every 60 seconds.
* Exporter selection follows the same rules as traces — HTTP for Dynatrace, Langfuse, and the other HTTP-only endpoints, gRPC otherwise.
* If an external `MeterProvider` is already configured (for example by `opentelemetry-instrument`), Cognee reuses it instead of creating its own, exactly as it does for the `TracerProvider`.
## Logs
Enabling tracing attaches an OTel `LoggingHandler` (at `DEBUG` level) to the `cognee`, `cognee.api`, `cognee.modules`, `cognee.tasks`, and `cognee.infrastructure` loggers, so Cognee's log output is emitted as OTel log records. Records produced inside a memory operation carry the active span's trace and span IDs, letting your backend pivot from a slow `memory.retrieve` span straight to the log lines it produced.
The bridge is idempotent — repeated enablement will not attach duplicate handlers — and `disable_tracing()` detaches it. Your existing Python logging configuration is untouched: the handler is added alongside whatever handlers you already have, so console and file logging continue to work.
Log export follows the same endpoint rules as metrics: records are exported only when `OTEL_EXPORTER_OTLP_ENDPOINT` is set (or `console_output=True`), and they go to your traces endpoint with `/v1/traces` rewritten to `/v1/logs`, overridable with `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT`.
OTLP log records are batched and exported on a background thread (`BatchLogRecordProcessor`), on both the gRPC and the HTTP exporter path, so emitting a log line never waits on a network round trip. Console output (`console_output=True`) is the exception — it still exports each record as it is emitted.
## Environment Variables Reference
| Variable | Default | Description |
| ------------------------------------- | ---------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `COGNEE_TRACING_ENABLED` | *(unset)* | Set to `true` (or `1`/`yes`) to enable OTEL traces, metrics, and the log bridge. An explicit off value — `false`, `0`, or `no` — is authoritative: it wins over `LANGFUSE_*` keys and stops new spans and metric recordings even when tracing was already switched on in the process (already-attached exporters keep running until `disable_tracing()`). Leaving the variable unset is *not* a veto, so Langfuse keys still auto-enable tracing on their own. |
| `OTEL_SERVICE_NAME` | `cognee` | Service name attached to all spans, metrics, and log records |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | *(none)* | OTLP collector endpoint for all three signals. Cognee prefers the OTLP gRPC exporter when both gRPC and HTTP exporters are installed, except for the HTTP-only endpoints listed in the note below. |
| `OTEL_EXPORTER_OTLP_HEADERS` | *(none)* | Auth headers (e.g. `Authorization=Bearer <token>`), applied to the trace, metric, and log exporters |
| `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` | *(derived)* | Overrides the metrics endpoint. Defaults to `OTEL_EXPORTER_OTLP_ENDPOINT` with `/v1/traces` rewritten to `/v1/metrics`. |
| `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | *(derived)* | Overrides the logs endpoint. Defaults to `OTEL_EXPORTER_OTLP_ENDPOINT` with `/v1/traces` rewritten to `/v1/logs`. |
| `OTEL_RESOURCE_ATTRIBUTES` | *(none)* | Comma-separated extra resource attributes attached to the tracer provider resource (e.g. `service.namespace=my-team,service.version=1.0,deployment.environment.name=prod`). Read by the OpenTelemetry SDK itself. |
| `LANGFUSE_PUBLIC_KEY` | *(none)* | Langfuse public key. Must be set together with `LANGFUSE_SECRET_KEY`; enables tracing and auto-derives the OTLP endpoint and auth header (see [Langfuse](#langfuse)). |
| `LANGFUSE_SECRET_KEY` | *(none)* | Langfuse secret key. Must be set together with `LANGFUSE_PUBLIC_KEY`. |
| `LANGFUSE_HOST` | `https://cloud.langfuse.com` | Langfuse base URL (e.g. a region or self-hosted instance). Falls back to `LANGFUSE_BASE_URL` when unset. |
| `LANGFUSE_BASE_URL` | *(none)* | Accepted as an alias for `LANGFUSE_HOST` when `LANGFUSE_HOST` is not set. |
Cognee reads `OTEL_EXPORTER_OTLP_ENDPOINT` directly and passes it to the OTLP exporter. Other standard `OTEL_EXPORTER_OTLP_*` settings such as headers are honored by the underlying exporter library. `OTEL_RESOURCE_ATTRIBUTES` is read by the OpenTelemetry SDK and merged into the provider resource. When Cognee creates its own `TracerProvider`, that includes `service.name`, `service.version`, and `deployment.environment`; in auto-instrumented setups, the external provider's resource configuration applies instead.
<Note>
Cognee tries the OTLP gRPC exporter first and only falls back to the OTLP HTTP exporter if the gRPC exporter package is unavailable. Because the shipped extras install both exporters, the practical default is gRPC. Use a gRPC-compatible OTLP endpoint unless your endpoint matches one of the HTTP-only patterns below.
Endpoints known to accept OTLP over HTTP only always use the HTTP exporter, for traces, metrics, and logs alike. Pointing a gRPC exporter at one of these fails silently — the connection is accepted and then closed, so telemetry disappears with no visible error. An endpoint is treated as HTTP-only when it contains any of:
| Pattern | Backend |
| ---------------------------------------------------------------- | --------------------------------------------------- |
| `/api/public/otel` | Langfuse (including self-hosted on a custom domain) |
| `dynatrace.com`, `live.dynatrace.com`, `otel.live.dynatrace.com` | Dynatrace |
| `/api/v2/otlp` | Dynatrace OTLP path |
| `:4318` | Standard OTLP HTTP port, any backend |
| `:443/` | HTTPS with a path — almost always HTTP OTLP |
Note the trailing slash in `:443/`: `https://collector:443` still uses gRPC, while `https://collector:443/v1/traces` uses HTTP. If you were relying on gRPC for an HTTPS endpoint that includes a path, that endpoint now exports over HTTP.
</Note>
Span and log export to an OTLP backend is batched — a `BatchSpanProcessor` for spans and a `BatchLogRecordProcessor` for [log records](#logs) — so both are flushed asynchronously on a background thread rather than one request per span or per record. Call `disable_tracing()` before your process exits to flush pending spans, metrics, and log records; because log records are buffered too, skipping it can drop the last batch of them, and the call itself takes as long as the force-flush needs. The in-memory buffer that backs `get_last_trace()` is unaffected and still receives spans as they complete.
<Tip>
If you are running Cognee as an HTTP server and want to inspect traces or pipeline activity over HTTP, see **Activity and Observability** in [Deploy REST API Server](/guides/deploy-rest-api-server#activity-and-observability).
</Tip>
# Pi
Source: https://docs.cognee.ai/integrations/pi-integration
Give the Pi coding agent persistent memory with the community pi-cognee extension.
Add persistent, queryable knowledge-graph memory to [Pi](https://github.com/earendil-works/pi) with the community **pi-cognee** extension. Your agent can remember facts, recall them in later sessions, forget what's outdated, and manage datasets — all from inside Pi.
<Info>
pi-cognee is a **community extension** built and maintained by [Kerry Hatcher](https://github.com/kerryhatcher) (MIT licensed). It is not maintained by the Cognee team — report issues on the [pi-cognee repository](https://github.com/kerryhatcher/pi-cognee/issues).
</Info>
The extension offers two backends behind one interface:
| Mode | How it runs | Best for |
| ----------------- | ------------------------------------------------------------------------- | ----------------------------------------------- |
| **SDK** (default) | In-process via the [`@cognee/cognee-ts`](/typescript/getting-started) SDK | Zero infrastructure — works out of the box |
| **MCP** | Connects to a remote [Cognee MCP server](/cognee-mcp/mcp-overview) | Shared memory across agents, self-hosted setups |
All memory tools work identically in both modes, so you can start with SDK mode and switch to a shared MCP server later without changing how you use the agent.
## Install
```bash theme={null}
pi install npm:@kerryhatcher/pi-cognee
```
## Quick Start (SDK mode)
SDK mode runs Cognee in-process — no server needed. Set your LLM API key inside Pi:
```bash theme={null}
/cognee-config llmApiKey sk-...
/cognee-config llmModel gpt-5-mini
```
Then start using memory in conversation:
```
> Remember this: our staging environment runs in eu-central-1
```
Later — even in a new session — ask for it back:
```
> Which region does staging run in? Check your cognee memory.
```
Recall isn't automatic — Pi decides when to call `cognee_recall`, and for questions it can answer from project files, a coding agent will usually read the files instead. Test with facts that aren't in your codebase, or nudge it with "check your cognee memory". Memory is also stored per working directory in SDK mode, so recall it from the directory where you stored it.
<Note>
In SDK mode, Cognee's own LLM calls (entity extraction, embeddings, search-time completions) run against the provider you configure with `llmApiKey` / `llmModel` and are billed by that provider — they don't go through Pi's model.
</Note>
## MCP Mode
To share memory across agents or use a self-hosted Cognee deployment, point the extension at a running [Cognee MCP server](/cognee-mcp/mcp-quickstart) (HTTP transport):
```bash theme={null}
/cognee-mode mcp
/cognee-config mcpUrl http://localhost:8001/mcp
```
In MCP mode, LLM and database configuration lives on the server — only `mcpUrl` is needed locally.
## Commands
| Command | Description |
| ------------------------------ | --------------------------------------- |
| `/cognee-mode [sdk\|mcp]` | Switch backend mode |
| `/cognee-config` | Show all config (API keys are redacted) |
| `/cognee-config <key>` | Show one config value |
| `/cognee-config <key> <value>` | Set a config value |
## Tools
The extension registers these tools with Pi; they behave the same in both modes:
| Tool | Description |
| ----------------------- | ----------------------------- |
| `cognee_health` | Check connectivity |
| `cognee_remember` | Store text in memory |
| `cognee_recall` | Search memory |
| `cognee_forget` | Delete datasets or all memory |
| `cognee_datasets` | List datasets |
| `cognee_dataset_data` | List items in a dataset |
| `cognee_create_dataset` | Create a new dataset |
| `cognee_client_info` | Show client identity and mode |
| `cognee_cognify_file` | Ingest a file (base64) |
## Configuration Reference
Set values with `/cognee-config <key> <value>`:
| Key | Default | Description |
| ------------------- | --------------------------- | ----------------------------- |
| `mode` | `sdk` | Backend mode: `sdk` or `mcp` |
| `mcpUrl` | `http://localhost:8001/mcp` | MCP server URL (MCP mode) |
| `llmModel` | — | LLM model (SDK mode) |
| `llmApiKey` | — | LLM API key (SDK mode) |
| `embeddingProvider` | — | Embedding provider (SDK mode) |
| `embeddingModel` | — | Embedding model (SDK mode) |
| `vectorDbProvider` | — | Vector DB provider (SDK mode) |
| `graphDbProvider` | — | Graph DB provider (SDK mode) |
The SDK-mode keys map to the [`@cognee/cognee-ts` constructor options](/typescript/configuration) — see that page for supported providers and models.
***
<CardGroup>
<Card title="GitHub Repository" icon="github" href="https://github.com/kerryhatcher/pi-cognee">
View the pi-cognee source code
</Card>
<Card title="npm Package" icon="box" href="https://www.npmjs.com/package/@kerryhatcher/pi-cognee">
@kerryhatcher/pi-cognee on npm
</Card>
<Card title="TypeScript SDK" icon="square-js" href="/typescript/getting-started">
The @cognee/cognee-ts SDK powering SDK mode
</Card>
</CardGroup>
# ScrapeGraphAI
Source: https://docs.cognee.ai/integrations/scrapegraphai-integration
Scrape web pages into Cognee with ScrapeGraphAI.
Scrape web pages and feed the extracted content into cognee using [ScrapeGraphAI](https://scrapegraphai.com). The [`cognee-community-tasks-scrapegraph`](https://github.com/topoteretes/cognee-community/tree/main/packages/task/scrapegraph_tasks) package provides two async tasks: `scrape_urls` for extraction and `scrape_and_add` for end-to-end scrape-to-graph ingestion.
## Why Use This Integration
* **Prompt-Based Extraction**: Describe what you want in natural language — no CSS selectors or scraper maintenance
* **Single Function Pipeline**: `scrape_and_add` scrapes, ingests, and builds the graph in one call
* **Structured Output**: Optionally pass a Pydantic schema for domain-specific extraction
* **Source Attribution**: Each scraped page is tagged with its origin URL in the knowledge graph
* **JavaScript Rendering**: Handles JS-rendered pages and common bot protection
## Installation
```bash theme={null}
pip install cognee-community-tasks-scrapegraph cognee
```
Or with uv:
```bash theme={null}
uv pip install cognee-community-tasks-scrapegraph cognee
```
## Requirements
You need two API keys:
| Variable | Description |
| -------------- | ------------------------------------------------------------ |
| `LLM_API_KEY` | OpenAI (or other LLM provider) API key used by cognee |
| `SGAI_API_KEY` | [ScrapeGraphAI](https://dashboard.scrapegraphai.com) API key |
```bash theme={null}
export LLM_API_KEY="sk-..."
export SGAI_API_KEY="sgai-..."
```
<Info>
See [LLM Providers](/setup-configuration/llm-providers) and [Embedding Providers](/setup-configuration/embedding-providers) if you want to use a provider other than OpenAI.
</Info>
## Quick Start
### 1. Scrape and Inspect
Use `scrape_urls` to verify what ScrapeGraphAI extracts before building a graph:
```python theme={null}
import asyncio
from cognee_community_tasks_scrapegraph import scrape_urls
async def main():
results = await scrape_urls(
urls=[
"https://cognee.ai",
"https://docs.cognee.ai",
],
user_prompt="Extract the product description, key features, and target use cases",
)
for item in results:
if item.get("error"):
print(f"[!] {item['url']}: {item['error']}")
else:
print(f"\n=== {item['url']} ===")
print(str(item["content"])[:500])
asyncio.run(main())
```
The `user_prompt` tells ScrapeGraphAI what to focus on when extracting content from each page.
### 2. Build the Knowledge Graph
Use `scrape_and_add` to scrape, ingest, and build the graph in one call:
```python theme={null}
import asyncio
import cognee
from cognee_community_tasks_scrapegraph import scrape_and_add
async def main():
await cognee.prune.prune_data()
await cognee.prune.prune_system(metadata=True)
await scrape_and_add(
urls=[
"https://cognee.ai",
"https://docs.cognee.ai",
"https://github.com/topoteretes/cognee",
],
user_prompt="Extract the product description, key features, and target use cases",
dataset_name="cognee_research",
)
results = await cognee.recall(
query_text="What is Cognee and what problems does it solve?",
datasets=["cognee_research"],
)
for r in results:
print(r)
asyncio.run(main())
```
<Info>
The `prune` calls reset the local database. Skip them when building incrementally on top of an existing graph.
</Info>
## Structured Extraction
When you know the shape of the data you need, pass a Pydantic schema to ScrapeGraphAI's `smartscraper` directly. This bypasses the integration's `scrape_urls` and gives you full control over the output structure:
```python theme={null}
import asyncio
import os
import cognee
from pydantic import BaseModel, Field
from scrapegraph_py import Client
class ProductPage(BaseModel):
name: str = Field(description="Product or company name")
tagline: str = Field(description="One-line value proposition")
features: list[str] = Field(description="List of key features or capabilities")
pricing_model: str = Field(description="How the product is priced")
target_audience: str = Field(description="Who the product is primarily aimed at")
async def scrape_structured(urls: list[str]) -> list[str]:
client = Client(api_key=os.environ["SGAI_API_KEY"])
texts = []
try:
for url in urls:
response = client.smartscraper(
website_url=url,
user_prompt="Extract product information",
output_schema=ProductPage,
)
result = response.get("result", {})
text = f"""Source: {url}
Product: {result.get('name', 'N/A')}
Tagline: {result.get('tagline', 'N/A')}
Features: {', '.join(result.get('features', []))}
Pricing: {result.get('pricing_model', 'N/A')}
Audience: {result.get('target_audience', 'N/A')}"""
texts.append(text)
finally:
client.close()
return texts
async def main():
await cognee.prune.prune_data()
await cognee.prune.prune_system(metadata=True)
urls = ["https://cognee.ai", "https://scrapegraphai.com", "https://firecrawl.dev"]
texts = await scrape_structured(urls)
combined = "\n\n".join(texts)
await cognee.remember(combined, dataset_name="product_landscape")
results = await cognee.recall(
query_text="How do these products differ in their approach?",
datasets=["product_landscape"],
)
for r in results:
print(r)
asyncio.run(main())
```
## Querying the Graph
Once the graph is built, use `cognee.recall(...)` to query it.
## Use Cases
<AccordionGroup>
<Accordion title="Competitive Intelligence">
Scrape competitor product and pricing pages, build a knowledge graph, then query across all of them:
1. Gather competitor URLs (product pages, pricing, docs)
2. Use `scrape_and_add` with a prompt focused on pricing, features, and positioning
3. Query with synthesis questions like "Which product is best for enterprise use cases?"
Scope queries to the dataset with `datasets=["competitive_intel"]`.
</Accordion>
<Accordion title="News Monitoring">
Scrape news sources on a schedule and add to the existing graph incrementally:
1. Set up a list of news/blog URLs
2. Run `scrape_and_add` daily (skip the `prune` calls to accumulate data)
3. Query across the full timeline: "What are the biggest trends this week?"
The graph gets better over time as entities and relationships accumulate.
</Accordion>
<Accordion title="Research Aggregation">
Collect and correlate information from many sources:
1. Scrape documentation, blog posts, and GitHub READMEs for a topic
2. Build the graph with `scrape_and_add`
3. Ask cross-source questions: "How does library X compare to library Y?"
Use structured extraction with Pydantic schemas for consistent input.
</Accordion>
<Accordion title="Product Landscape Analysis">
Map out an entire product category:
1. Scrape product pages with a schema targeting name, features, pricing, and audience
2. Ingest into cognee
3. Query for patterns: "Which products target developers?" or "What pricing models are most common?"
</Accordion>
</AccordionGroup>
<CardGroup>
<Card title="GitHub Repository" icon="github" href="https://github.com/topoteretes/cognee-community/tree/main/packages/task/scrapegraph_tasks">
View source code and examples
</Card>
<Card title="Blog Post" icon="newspaper" href="https://scrapegraphai.com/blog/scrapegraphai-cognee">
Read full tutorial on ScrapeGraph website
</Card>
</CardGroup>
# Slack
Source: https://docs.cognee.ai/integrations/slack-integration
Ask, save, and recall your Cognee memory from Slack.
Connect Slack to Cognee so your team can query and grow the knowledge graph without leaving a channel or DM. Cognee ships more than one way to reach Slack. Pick the one that matches what you run:
| You want | Use | Where it lives |
| ----------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- | --------------------------------------------------------------------- |
| A managed connector for Cognee Cloud, connected from the SaaS UI | The cloud **Data sources** Slack connector | [Cloud Integrations UI](/cognee-cloud/ui/integrations) |
| Slash commands (`/cognee-ask`, `/cognee-remember`) against your own self-hosted Cognee backend, per-person memory | The **built-in Slack app** in the `cognee` package | [Built-in Slack App](#built-in-slack-app) below |
| A bot that passively remembers whole channels and answers `@cognee` / `/recall` with cited source messages | The **standalone channel-memory bot**, `cognee-integration-slack` | [Standalone Channel-Memory Bot](#standalone-channel-memory-bot) below |
| Bulk-loading channel history in code | dlt's `slack_source` | [dlt integration](/integrations/dlt-integration) |
<Info>
The built-in app, the standalone bot, and the cloud connector are three **separate Slack apps** (different manifest, different scopes, different transport) — do not point one at another's server.
</Info>
## Built-in Slack App
The Slack app that ships inside the `cognee` package: slash commands and a message shortcut, served by your own self-hosted backend, with memory scoped to each linked member. Once a workspace is connected, anyone in it can search memory with `/cognee-ask`, save new facts with `/cognee-remember`, capture existing messages with the **Remember this** shortcut, and review every answer privately before sharing it with the channel.
**Requires `cognee >= 1.5.0`.** On older releases `/cognee-remember` falls through to *"Command `/cognee-remember` is not yet supported."* while `/cognee-ask` and **Remember this** work.
```bash theme={null}
# consuming cognee as a dependency
pip install -U cognee
# working on cognee itself — run the checkout, not a release
git checkout dev && uv sync
```
### Why Use This Integration
* **Ask from anywhere in Slack**: `/cognee-ask <question>` searches your memory and replies privately, with a **Share** button to post the answer to the channel once you've checked it, or **Discard** to drop it.
* **Save what Slack never saw**: `/cognee-remember <text>` stores free text nobody typed into Slack yet — a decision from a call, a conclusion recorded after the fact.
* **Save what Slack already has**: the **Remember this** message shortcut (in a message's ⋯ menu) stores any Slack message into Cognee memory, tagged with who said it and where.
* **Per-user, not per-workspace**: each member runs `/cognee-link` once so their asks and saves use *their own* Cognee account — without it, Cognee refuses to act for them rather than falling back to anyone else's memory.
* **Self-hosted, no middleman**: your Slack app talks directly to your Cognee backend. No data passes through a third-party relay.
<Note>
One Slack **app** is installed once per workspace, but memory is per-**person** — that's what `/cognee-link` exists to resolve. A member who skips it is **refused, not silently attributed**: the only unlinked member Cognee will act for is the one who completed the workspace's OAuth Connect (their own account is used), and everyone else gets *"I don't know which Cognee account you are yet. Run `/cognee-link` to connect yours, then try again."* — for `/cognee-ask`, `/cognee-remember`, and **Remember this** alike. Nothing is written into someone else's memory as a fallback. Workspaces connected before the installer's Slack id was recorded have no installer exception at all, so every member there must link.
</Note>
All replies are ephemeral (visible only to the person who triggered them) except a shared answer, which posts to the channel.
### Prerequisites
* Cognee **`>= 1.5.0`** or a `dev` checkout — see the version note above.
* A running Cognee backend (and frontend, for the `/cognee-link` confirmation page) reachable over HTTPS from the public internet — Slack will not call `localhost`. For local development, use a tunnel like ngrok (below); for production, this is your deployment's normal public domain.
* A Cognee account for each Slack member who wants to link their own memory.
### Local Development: ngrok Tunnel
Slack requires a public HTTPS URL for every request URL. ngrok tunnels a public HTTPS host to your local port 8000 — that host is what every `<public-host>` placeholder below refers to. Skip this section if your backend already has a public domain.
1. Install ngrok: `brew install ngrok` on macOS, or download the binary from [ngrok.com/download](https://ngrok.com/download).
2. Authenticate once: sign up for a free account, copy your authtoken from the dashboard, and run `ngrok config add-authtoken <token>`.
3. Start the tunnel:
```bash theme={null}
ngrok http 8000
```
ngrok prints a line like `Forwarding https://<random>.ngrok-free.app -> http://localhost:8000`. That `https://<random>.ngrok-free.app` host goes into the manifest and `.env` below. Leave the process running the whole time you're testing — closing it kills the tunnel and every request URL stops resolving.
<Tip>
A free plan's random host changes on every restart, which means re-editing the Slack app's request URLs and `.env` each time. A free account includes one **reserved domain** that fixes this permanently: create it under **Domains** in the ngrok dashboard, then run `ngrok http 8000 --domain=your-name.ngrok-free.app`. Set the manifest and `SLACK_REDIRECT_URI` to that fixed host once and it never changes across restarts.
</Tip>
<Info>
Free-tier ngrok shows a "Visit Site" interstitial page to **browsers** opening the tunnel URL directly. It does **not** apply to Slack's server-to-server requests (commands, events, interactivity) — those go straight through, so it never blocks the integration itself.
</Info>
### Setup
#### 1. Create the Slack App from the Manifest
<Warning>
Create a **separate** Slack app per environment (local dev, staging, production) — and per developer. Never point the production app's manifest at your laptop: reusing one app means whoever else is testing has their events routed to your tunnel instead of the real server.
Also prefer a workspace where **only your app is installed**. Slack lets multiple apps register the same slash command and shows a disambiguation dropdown — so in a shared workspace with several Cognee apps, `/cognee-remember` may silently go to someone else's backend, and the reply looks like your server misbehaving.
</Warning>
1. Go to [api.slack.com/apps](https://api.slack.com/apps) → **Create New App** → **From an app manifest**.
2. Paste the manifest below, replacing every `<public-host>` with your backend's public host.
3. On the app's **Basic Information** page, copy **Client ID**, **Client Secret**, and **Signing Secret** — you'll need them for `.env` in step 2.
4. Install the app to your workspace when prompted.
```yaml theme={null}
display_information:
name: Cognee
description: Ask questions, save messages, and recall your team's shared memory — right from Slack.
background_color: "#6510F4"
features:
bot_user:
display_name: cognee
always_online: true
app_home:
home_tab_enabled: true
messages_tab_enabled: true
slash_commands:
- command: /cognee-ask
url: https://<public-host>/api/v1/slack/commands
description: Ask your Cognee memory a question — reviewed privately before you share it
usage_hint: what do we know about the Q3 launch?
- command: /cognee-remember
url: https://<public-host>/api/v1/slack/commands
description: Save a decision or fact worth keeping — for things Slack never saw
usage_hint: we chose Neon for v2, branching is cheaper
- command: /cognee-link
url: https://<public-host>/api/v1/slack/commands
description: Get a private link to connect this Slack account to your own Cognee memory
shortcuts:
- name: Remember this
type: message
callback_id: remember_this
description: Save this message to your own Cognee memory, tagged with who said it and where
oauth_config:
redirect_urls:
- https://<public-host>/api/v1/integrations/slack/callback
scopes:
bot:
- commands
- chat:write
- im:write
- channels:read
settings:
event_subscriptions:
request_url: https://<public-host>/api/v1/slack/events
bot_events:
- app_uninstalled
- tokens_revoked
- app_home_opened
interactivity:
is_enabled: true
request_url: https://<public-host>/api/v1/slack/interactive
org_deploy_enabled: false
socket_mode_enabled: false
token_rotation_enabled: false
```
<Warning>
**Adding a slash command to an existing app is not enough on its own.** Slack does not grant a new command to an already-installed app — it refuses the command client-side and nothing ever reaches your server, which looks exactly like a broken backend. After editing the manifest to add a command, **reinstall the app** to the workspace, then reload the Slack client (`Cmd+R`) so autocomplete picks it up.
</Warning>
<Info>
Deliberately no `channels:history` — bulk channel ingestion isn't part of this integration (see the [dlt integration](/integrations/dlt-integration) for pulling Slack history via dlt's `slack_source` instead). Slack's non-Marketplace rate limits on history reads make it impractical here anyway.
</Info>
#### 2. Configure Cognee
Copy the app's credentials into your backend's `.env`:
| Variable | Value |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `SLACK_CLIENT_ID` | From the app's **Basic Information** page. |
| `SLACK_CLIENT_SECRET` | From the app's **Basic Information** page. |
| `SLACK_SIGNING_SECRET` | From the app's **Basic Information** page — verifies every inbound request's `X-Slack-Signature` and signs the OAuth `state` and link codes. |
| `SLACK_REDIRECT_URI` | `https://<public-host>/api/v1/integrations/slack/callback` — must **byte-match** the manifest's `redirect_urls` entry. Slack re-validates it during the token exchange, so a mismatch fails *after* the user has already approved the app, as a generic callback error. |
| `SLACK_FRONTEND_BASE_URL` | Your frontend's origin, e.g. `http://localhost:3000` — used to build the `/cognee-link` confirmation URL. |
| `INTEGRATION_CREDENTIALS_KEY` | Not Slack-specific — the AES-256-GCM key (32 raw bytes, base64-encoded) the integrations framework encrypts every stored OAuth token under. If it's missing, the first credential write raises `RuntimeError`, so `/cognee-link` and the OAuth Connect step both hard-fail. Generate one with `openssl rand -base64 32`. `INTEGRATION_CREDENTIALS_KEYS` (a JSON keyring) is the preferred, rotation-capable form — the single-key variable is the legacy fallback. |
Then bring the stack up **from source** — it runs exactly the checkout you have, with no image layer in between to go stale:
```bash theme={null}
# backend, on :8000
cd cognee
uv sync # re-run after every branch switch
uv run python cognee/api/client.py
# frontend, on :3000 (separate terminal)
cd cognee-frontend
npm install
npm run dev
```
<Info>
`uv sync` matters more than it looks: branches move `pyproject.toml`/`uv.lock`, and a venv left over from another branch fails in ways that read as application bugs rather than a dependency mismatch. Restart the backend after any `.env` change.
</Info>
#### 3. Install and Connect
1. Installing the app from the manifest (step 1) prompts the workspace install once.
2. In the Cognee frontend, go to **Integrations** → **Connect** under Slack. This runs the OAuth exchange and stores the workspace-level bot credential.
3. In Slack, run `/cognee-link` and open the link it replies with in a browser where you're already logged in to Cognee, then click **Confirm**. The link is a short-lived signed code (10-minute expiry), not an API key — nothing secret is ever typed into Slack. This is what points `/cognee-ask` and the save actions at **your** memory — and for anyone who didn't run Connect, what makes them work at all (see the per-person note above).
4. Try `/cognee-ask What does Cognee do?` — you should get a private reply with **Share** / **Discard** buttons.
<Warning>
**Use the frontend's Connect button, not Slack's "Install to Workspace".** The OAuth `state` is what binds the install to a Cognee account. Slack's own install button sends no signed state, so the callback rejects it and you land on `/integrations?slack=error_invalid_state`.
</Warning>
### Using It
| Command / action | Behavior |
| ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `/cognee-ask <question>` | Ephemeral ack (`🔎 Recalling: ...`) → searches memory → private reply with **Share** (posts to the channel) / **Discard** buttons. |
| `/cognee-remember <text>` | Ephemeral ack (`💾 Remembering: ...`) → saves the text with provenance (`In Slack, @user noted: ...`) → the ack is replaced by a confirmation (`✅ Remembered — it'll be recallable shortly.`) or by a failure message (`Could not save that. Please try again.`). |
| **Remember this** (message ⋯ menu) | Saves the message text with provenance (`In #channel, @user said: ...`). Both save paths land in the same `slack` dataset and node set — one Slack-origin body of memory, not two the graph can't relate. |
| `/cognee-link` | Ephemeral, one-time per member. The link expires in 10 minutes if unused. |
<Note>
**Ingest something before you ask.** `/cognee-ask` only searches — it never writes, so it cannot create the `slack` dataset. On a completely empty memory it fails with `CollectionNotFoundError: Collection 'DocumentChunk_text' not found`, surfaced as *"Search failed. Please try again."* — which reads like a broken integration but just means there is nothing to search. Run `/cognee-remember` (or ingest via the CLI) first.
</Note>
**@-mentioning the bot does nothing.** The SDK app subscribes to `app_home_opened`, `app_uninstalled`, and `tokens_revoked` only — there is no `app_mention` handler. Use the slash commands. If you want `@cognee <question>` answered from channel memory, that is what the [standalone channel-memory bot](#standalone-channel-memory-bot) does.
No `/invite @cognee` is needed for any of the above either. Every reply — including Share, which posts to the channel — is delivered by Slack on the app's behalf regardless of channel membership.
#### Restricting Which Channels Can Run Commands
An opt-in, per-workspace channel allowlist is available from the Integrations page, backed by two authenticated endpoints: `GET /api/v1/slack/channels` lists the workspace's public channels and flags which are currently allowed, `PUT /api/v1/slack/channels` sets the list. An empty list (the default) means unrestricted — a workspace that never touches this setting keeps working everywhere.
This requires the `channels:read` scope (already in the manifest above). If a workspace connected before that scope existed, the channel list call fails — the fix is to disconnect and reconnect the workspace, not to re-save the allowlist.
Once a non-empty allowlist exists, every command in every other channel returns *"Cognee isn't enabled in this channel."* — worth remembering when testing from a scratch channel.
#### Removing the App from a Channel vs. Uninstalling
* **Removing/kicking the bot from a channel** only stops it from posting there — it does not revoke anything; the workspace connection and everyone's `/cognee-link`s stay intact.
* **Uninstalling the app** (or a token revoke) fires `app_uninstalled`/`tokens_revoked`, which revokes the stored workspace credential server-side — commands then fail with "not connected" until reconnected from Integrations.
### Multiple Environments
Slack apps are tied to one set of OAuth credentials and one redirect URL each. If you run more than one Cognee deployment (local dev, staging, production), create a **separate Slack app per environment** rather than editing one app's config back and forth — each environment gets its own `SLACK_CLIENT_ID`/`SLACK_CLIENT_SECRET`/`SLACK_SIGNING_SECRET`/`SLACK_REDIRECT_URI`. This avoids local testing ever breaking a shared staging or production connection.
<Info>
This is separate from how many Slack **workspaces** one deployment can serve: any number of teams can each connect their own workspace to the same Cognee deployment through the same Connect flow — that's already multi-tenant by design and needs no extra setup per workspace.
</Info>
### Troubleshooting
1. **``Command `/cognee-remember` is not yet supported``**: most likely your backend is on a Cognee release older than `1.5.0`, which has the integration but not this command. Run `pip install -U cognee`, or run the `dev` checkout directly.
2. **Same message, correct version installed**: either the manifest you installed from predates the command and the app was never **reinstalled** (see the reinstall warning above — this is the most common cause), or you are running stale code — restart the backend process (or rebuild the container if containerized).
3. **Same message, but a *different app name* answers**: another Cognee app in the workspace owns the command. Slack shows all of them in autocomplete — pick yours explicitly, or test in a workspace where only your app is installed.
4. **Typing the command produces nothing at all, no request reaches your server**: the command is not registered on your app. Check **api.slack.com/apps → your app → Slash Commands**; if it's missing, the manifest save silently failed. Add it via **Create New Command**, then reinstall.
5. **"Something went wrong when authorizing Cognee"** during Connect: the `redirect_uri` your backend sent (from `SLACK_REDIRECT_URI`) isn't byte-identical to an entry under the app's redirect URLs, or `.env` still holds a different app's `SLACK_CLIENT_ID`/`SLACK_CLIENT_SECRET` than the workspace you're installing into. Re-sync `.env` to the exact app you're testing and restart.
6. **Callback redirects to `?slack=error_exchange_failed`**: the callback never 500s — it catches everything and redirects. The traceback is in the **backend log**. Usual causes: `SLACK_REDIRECT_URI` mismatch, or a missing/malformed `INTEGRATION_CREDENTIALS_KEY`.
7. **Signature verification fails on every request (401)**: `SLACK_SIGNING_SECRET` in `.env` doesn't match the app you actually installed — easy to hit when juggling more than one test app. Copy it again from **Basic Information** and restart the backend.
8. **Request URL mismatch after restarting ngrok**: free-tier ngrok mints a new random host every restart. Update every request URL and redirect URL in the app's Slack settings — or use a reserved/static domain so this never happens.
9. **Slack shows "dispatch\_failed" instead of a reply**: the slash command's request URL is missing or wrong — it must point at `/api/v1/slack/commands`, not `/api/v1/slack/events` or `/api/v1/slack/interactive`.
10. **"This Slack workspace is not connected to Cognee"**: the OAuth install (Connect in the Cognee UI) hasn't completed for this workspace yet, or the connection was later disconnected/revoked.
11. **"I don't know which Cognee account you are yet"**: the member running the command has no active `/cognee-link`, and isn't the member who completed the workspace's Connect — so the command is refused before it reads or writes anything, rather than falling back to the installer's memory. Run `/cognee-link` and confirm the link in a browser where you're logged in to Cognee. **No API key is involved**; if the reply instead tells you to run `` `/cognee-link <api_key>` ``, you're on code from before this was fixed — that argument never existed.
12. **Can't delete a dataset — deletion hangs or errors**: a running Cognee backend holds an **exclusive lock** on the graph database via a worker subprocess. Stop the server first, then delete.
13. **Asking returns an "I don't know"-phrased answer as if it were real**: known limitation — the ask handler substring-matches common refusal phrasings ("cannot answer", "no relevant information", "does not contain", …) before forwarding an answer, but new phrasings can still slip through.
## Standalone Channel-Memory Bot
`cognee-integration-slack` is a separate Slack bot from the [cognee-integrations](https://github.com/topoteretes/cognee-integrations/tree/main/integrations/slack) repository. Instead of per-person slash commands, it gives a workspace **per-channel memory**: once a channel opts in, the bot silently ingests its messages into that channel's own Cognee dataset, and anyone can ask `@cognee <question>` or `/recall <question>` to get an answer with **cited links back to the source Slack messages**.
It is a thin client to a **running Cognee server** over HTTP (`POST /api/v1/add`, `/cognify`, `/search`, `/forget`) — no in-process Cognee and no LLM key of its own, since the server holds those. Slack I/O runs over **Socket Mode**, so it needs no public URL, no tunnel, and none of the manifest or OAuth setup above.
### Install
The package is not published on PyPI yet. Install it from the repository:
```bash theme={null}
git clone https://github.com/topoteretes/cognee-integrations.git
cd cognee-integrations/integrations/slack
uv sync # or: pip install -e .
```
Requires Python 3.10+. The only runtime dependencies are `slack-bolt`, `aiohttp`, and `httpx`. `uv sync` creates a project venv without activating it, so run the bot through `uv run` (below) or activate `.venv` first; `pip install -e .` into an already active environment needs neither.
### Create the Slack App
In [api.slack.com/apps](https://api.slack.com/apps), create an app with **Socket Mode** enabled, then:
1. Copy the **bot token** (`xoxb-…`) and the app-level **app token** (`xapp-…`).
2. Subscribe the bot to the `message.channels` and `app_mention` events.
3. Add the slash commands `/recall`, `/cognee-optin`, `/cognee-optout`, and `/cognee-forget`.
4. Install the app to the workspace and invite the bot to the channels it should remember.
### Configure and Run
The bot reads everything from environment variables (a `.env.example` ships in the package directory):
| Variable | Required | Meaning |
| -------------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `SLACK_BOT_TOKEN` | yes | Bot token (`xoxb-…`) for Web API calls — posting replies and resolving message permalinks. |
| `SLACK_APP_TOKEN` | yes | App token (`xapp-…`) for the Socket Mode connection. |
| `COGNEE_BASE_URL` | no | The running Cognee server. Default `http://localhost:8000`. |
| `COGNEE_API_KEY` | no | Sent as the `X-Api-Key` header. Leave unset for a local server with `ENABLE_BACKEND_ACCESS_CONTROL=false`; otherwise create a key for the Cognee user the bot should act as with [`POST /api/v1/auth/api-keys`](/api-reference/auth/create-api-key-for-user). Every channel dataset is owned by that user. |
| `COGNEE_SLACK_OPTED_IN_CHANNELS` | no | Comma-separated channel IDs that are opted in at startup. |
| `COGNEE_SLACK_COGNIFY_BATCH` | no | Buffered messages per channel before the graph is rebuilt. Default `10`. |
| `COGNEE_SLACK_DEFAULT_TEAM_ID` | no | Fallback workspace ID for events that carry no `team` field. |
```bash theme={null}
cp .env.example .env # then edit
export SLACK_BOT_TOKEN="xoxb-..." SLACK_APP_TOKEN="xapp-..."
export COGNEE_BASE_URL="http://localhost:8000"
uv run python -m cognee_integration_slack # or the installed script: uv run cognee-slack
```
<Warning>
**Opt-in state lives in the bot's process memory.** `/cognee-optin` and `/cognee-optout` mutate a set that starts from `COGNEE_SLACK_OPTED_IN_CHANNELS` and is never written back anywhere. After a restart, only the channels listed in that variable are remembered — put every channel that should stay opted in there, or opt them in again.
</Warning>
### Commands
| Trigger | Effect |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `@cognee <question>` | Answers from the channel's memory with sources. Asked inside a thread, it replies in that thread. |
| `/recall <question>` | Same answer as a slash command, posted to the channel. |
| `/cognee-optin` | Starts capturing this channel. The first opt-in posts a disclosure telling members that messages are now remembered and how to stop or erase. |
| `/cognee-optout` | Stops ingesting new messages. Existing memory is kept. |
| `/cognee-forget` | Deletes the channel's entire dataset — messages, graph, and citations. |
An empty question gets a usage hint back. A channel with no dataset yet — never opted in, or opted in with nothing cognified — returns *"I couldn't find anything about that in this channel's memory yet."*: the server answers the search with a `404` for the unknown dataset, which the bot treats as an empty result. Once the channel has memory, an unanswerable question gets whatever the LLM produces from the available context; unlike the built-in app, the bot does not filter refusal phrasings. A server-side failure (LLM auth, rate limit, a failed cognify) returns *"Sorry — I hit an error answering that."* rather than silence; the cause is in the bot's logs.
### How It Works
* **One dataset per channel.** Messages land in the dataset `slack_<channel_id>`, tagged with a node set of `[<channel_id>]`. The dataset is the boundary `/cognee-forget` clears, so forgetting is per channel, never per person.
* **Batched graph building.** Each message is added with a cheap `add` call; `cognify` runs once a channel has `COGNEE_SLACK_COGNIFY_BATCH` pending messages, and always before answering a question, so an answer reflects everything said so far. Concurrent cognify runs on one channel are serialized.
* **Cited answers.** Answering runs two searches: `GRAPH_COMPLETION` for the prose, and `CHUNKS` filtered to the channel's node set for the source messages. Each stored message carries a one-line provenance header (`[cognee-slack] channel=… ts=… author=… permalink=…`) inside its text, which is parsed back out of the retrieved chunks to build the links. The reply is a Block Kit message: the answer, then up to five sources as links labelled `#<channel ID> · <user ID> · <UTC time>` (raw Slack IDs, not names — the link itself opens the original message) with a `+N more` note for the rest. A message whose permalink could not be resolved shows as plain text, never a broken link.
* **What gets ingested.** Messages and thread replies in opted-in channels. Edits, deletions, joins, bot messages, the bot's own posts, messages that mention the bot, and empty messages are skipped. Public channels only, via `message.channels`. Edited or deleted Slack messages are not re-synced.
### Tests
The suite runs without Slack, Cognee, or LLM credentials — the adapter runs against a fake HTTP client, the handlers against mocks:
```bash theme={null}
uv run pytest tests/ -v
```
## Related
<CardGroup>
<Card title="Channel-Memory Bot Source" icon="github" href="https://github.com/topoteretes/cognee-integrations/tree/main/integrations/slack">
Source and tests for `cognee-integration-slack`.
</Card>
<Card title="dlt Integration" icon="database" href="/integrations/dlt-integration">
Pull Slack channel history in bulk via dlt, instead of a live bot.
</Card>
<Card title="Cloud Integrations UI" icon="puzzle" href="/cognee-cloud/ui/integrations">
Where "Connect Slack" lives in the Cognee UI.
</Card>
<Card title="Node Sets" icon="tags" href="/core-concepts/further-concepts/node-sets">
How per-channel node-set tags scope retrieval.
</Card>
</CardGroup>
# Strands
Source: https://docs.cognee.ai/integrations/strands-integration
Add persistent memory to Strands agents with Cognee.
Give your [Strands](https://strandsagents.com/) agents persistent memory powered by cognee. Built on cognee v1.0, the integration exposes two tools — `remember` and `recall` — that write to a permanent knowledge graph or a lightweight session cache, and survive across agent instances.
## Why Use This Integration
* **Two memory tiers**: Write straight to the permanent knowledge graph, or to a cheap per-session cache that you promote later
* **Natural language recall**: Retrieve stored knowledge with graph traversal and vector similarity
* **Cross-session memory**: Context persists across agent instances and restarts
* **Drop-in tools**: `cognee_tools()` returns ready-to-use Strands tools
## Installation
```bash theme={null}
pip install cognee-integration-strands
```
<Info>
Requires Python 3.10+. Pins `cognee>=1.0.0,<=1.1.2` and `strands-agents>=1.42.0,<2.0.0`.
</Info>
## Quick Start
Set your LLM key (cognee extracts knowledge with an LLM), then attach the tools:
```bash theme={null}
export LLM_API_KEY="your-openai-api-key-here" # used by cognee
```
```python theme={null}
import os
import cognee
from cognee_integration_strands import cognee_tools, run_cognee_task
from strands import Agent
from strands.models.openai import OpenAIModel
run_cognee_task(cognee.forget(everything=True)) # optional: start fresh
model = OpenAIModel(client_args={"api_key": os.getenv("LLM_API_KEY")}, model_id="gpt-4o")
agent = Agent(model=model, tools=cognee_tools())
# Store information
agent("Remember that we signed a contract with Meditech Solutions for £1.2M.")
# Retrieve it (even from a fresh agent — memory is persistent)
print(agent("What is the value of the Meditech Solutions contract?"))
```
## Tools
`cognee_tools(session_id=None, *, remember_kwargs=None, recall_kwargs=None)` returns a list of two tools:
| Tool | Description |
| ---------- | ------------------------------------------------------------------------------------ |
| `remember` | Store information in memory for later retrieval (wraps `cognee.remember`) |
| `recall` | Search and retrieve stored information with natural language (wraps `cognee.recall`) |
## Session Memory
By default, `remember` writes straight to the **permanent knowledge graph**. Pass a `session_id` to write to that session's lightweight cache instead, then promote the cache into the graph when you're ready:
```python theme={null}
SESSION_ID = "mission-briefing"
session_agent = Agent(
model=model,
tools=cognee_tools(session_id=SESSION_ID, remember_kwargs={"self_improvement": False}),
)
# ... use session_agent throughout the session ...
# Promote the session cache into the permanent graph
run_cognee_task(cognee.improve(session_ids=[SESSION_ID]))
```
<Info>
Passing `remember_kwargs={"self_improvement": False}` keeps session writes in cache-only mode until you call `cognee.improve(...)`. Without a `session_id`, writes go directly to the permanent graph.
</Info>
## How It Works
1. **Remember**: Stores data in cognee's memory — the permanent graph, or a session cache when `session_id` is set
2. **Recall**: Retrieves relevant information via cognee's recall pipeline
3. **Auto-Processing**: cognee extracts entities, relationships, and context automatically
4. **Background loop**: cognee's async API runs on a dedicated background event loop; `run_cognee_task()` handles this transparently
***
<CardGroup>
<Card title="GitHub Repository" icon="github" href="https://github.com/topoteretes/cognee-integrations/tree/main/integrations/strands">
View source code and examples
</Card>
<Card title="Examples" icon="book" href="https://github.com/topoteretes/cognee-integrations/tree/main/integrations/strands/examples">
Runnable example scripts
</Card>
</CardGroup>
# Vellum
Source: https://docs.cognee.ai/integrations/vellum-integration
Add persistent memory to Vellum agents with Cognee.
Connect Vellum agents to Cognee so prompts, tool calls, and responses are captured automatically and persisted as memory.
## Why Use This Integration
* **Zero-code memory**: the plugin captures prompts, tool traces, and responses automatically
* **Cross-session memory**: context persists across agent instances and conversation sessions
* **Flexible backend**: use a plugin-managed local Cognee server, your own self-hosted server, or Cognee Cloud
## Prerequisites
* A Vellum account and the Vellum CLI
* A Cognee API key if you are using Cognee Cloud
* An LLM API key if you are using local Cognee and want graph sync to work
* An agent already hatched in Vellum
<Note>
Vellum plugins are configured per agent, so you need to hatch the agent before installing Cognee.
</Note>
## Quick Start
Use the local flow if you want Cognee to run with a managed local server. Use the cloud flow if you want to connect Vellum to Cognee Cloud. You can also install the plugin from the Vellum web app instead of the CLI: open your assistant, go to the **Plugins** tab, search for **Cognee**, and install it there — the credentials shown below still need to be configured.
<Info>
The examples below use `my-assistant` as the agent name. Use the same name in every command.
</Info>
<Note>
**Two separate API keys are involved.** The Vellum provider key (set with `vellum setup`) is what the agent itself uses to chat — hatch configures Anthropic as the default provider. The Cognee `llm_api_key` credential is used by the Cognee server to build the knowledge graph during sync. Cognee does not reuse the Vellum provider key, so configure both — they can hold the same value if both use the same LLM provider.
</Note>
<Tabs>
<Tab title="Local">
Set up a local agent, configure both keys, and install the plugin:
```bash theme={null}
vellum hatch --name my-assistant --remote docker
vellum setup --provider anthropic
vellum exec my-assistant -- assistant credentials set sk-... --service cognee --field llm_api_key
vellum exec my-assistant -- assistant plugins install cognee
```
After setup, you can talk to the agent as usual, for example with:
```bash theme={null}
vellum client my-assistant
```
<Note>
If the agent replies with `provider_connection "anthropic-personal" has no API key stored`, the Vellum provider key is missing — run `vellum setup --provider anthropic`, or store one directly with `vellum exec my-assistant -- assistant keys set anthropic sk-ant-...`. See the [Vellum docs](https://www.vellum.ai/docs) for advanced agent setup options.
</Note>
</Tab>
<Tab title="Cognee Cloud">
Set the Cognee Cloud credentials for the agent and install the plugin:
```bash theme={null}
vellum hatch --name my-assistant --remote docker
vellum setup --provider anthropic
vellum exec my-assistant -- assistant credentials set your-cognee-api-key --service cognee --field api_key
vellum exec my-assistant -- assistant credentials set https://your-cognee-server-url --service cognee --field base_url
vellum exec my-assistant -- assistant plugins install cognee
```
The Cognee Cloud backend provides its own provider configuration, so you do not need to set an LLM API key for graph sync in this mode. The Vellum provider key is still required for the agent to chat.
</Tab>
</Tabs>
## How It Works
The plugin hooks into the Vellum agent lifecycle to capture and recall memory automatically:
| Hook | Fires | What it does |
| -------------------- | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- |
| `init` | Plugin load | Disables Vellum's default memory so Cognee is the sole memory provider, resolves the Cognee backend, and mints an API key for local servers |
| `user-prompt-submit` | Each user turn | Recalls relevant context from Cognee and injects it into the conversation |
| `post-tool-use` | After each tool call | Stores the tool call as a trace entry in session memory |
| `stop` | Turn end | Pairs the user prompt with the assistant response as a QA entry and triggers graph sync when the threshold is reached |
| `post-compact` | After context compaction | Injects a memory anchor (recent QA pairs, traces, graph context) into the compacted history |
| `shutdown` | Plugin unload | Runs a final graph sync and unregisters the agent connection |
See the [hook mapping in the integration README](https://github.com/topoteretes/cognee-integrations/tree/main/integrations/vellum-assistant#hook-mapping) for implementation details.
## Update or Remove
Upgrade the plugin to the current marketplace pin:
```bash theme={null}
vellum exec my-assistant -- assistant plugins upgrade cognee
```
Remove it from the agent:
```bash theme={null}
vellum exec my-assistant -- assistant plugins uninstall cognee
```
***
<CardGroup>
<Card title="GitHub Repository" icon="github" href="https://github.com/topoteretes/cognee-integrations/tree/main/integrations/vellum-assistant">
View the Vellum integration source code
</Card>
<Card title="Vellum GitHub" icon="github" href="https://github.com/vellum-ai/vellum-assistant">
Learn more about the Vellum assistant project
</Card>
</CardGroup>
# add()
Source: https://docs.cognee.ai/python-api/add
Ingest data into the Cognee knowledge base
# cognee.add()
```python theme={null}
async def add(
data: Union[BinaryIO, list[BinaryIO], str, list[str], DataItem, list[DataItem]],
dataset_name: str = 'main_dataset',
user: User = None,
node_set: Optional[List[str]] = None,
vector_db_config: dict = None,
graph_db_config: dict = None,
dataset_id: Optional[UUID] = None,
preferred_loaders: Optional[List[Union[str, dict[str, dict[str, Any]]]]] = None,
incremental_loading: bool = True,
data_per_batch: Optional[int] = 20,
llm_config: Optional[LLMConfig] = None,
embedding_config: Optional[EmbeddingConfig] = None,
)
```
## Description
Add data to Cognee for knowledge graph processing.
This is the first step in the Cognee workflow - it ingests raw data and prepares it
for processing. The function accepts various data formats including text, files, urls and
binary streams, then stores them in a specified dataset for further processing.
Prerequisites:
* **LLM\_API\_KEY**: Must be set in environment variables for content processing. The
provider preflight skips its LLM half when the call runs no LLM task — an `add()` of
plain text — so a missing key is not an error there; the graph build that follows
still needs one unless it runs on the
[`gliner` extractor](/python-api/cognify#llm-free-extraction-with-gliner). Embedding
configuration is checked either way.
* **Database Setup**: Relational and vector databases must be configured
* **User Authentication**: Uses default user if none provided (created automatically)
Supported Input Types:
* **Text strings**: Direct text content (str) - any string that is not a `file://`, `s3://`,
`http://` or `https://` URL and does not point to an existing local file. An absolute-looking
string that is not an existing file (for example `"/remember to call the dentist"`) is ingested
as text content on every platform, including Windows.
* **File paths**: Local file paths as strings in these formats:
* Absolute paths: "/path/to/document.pdf" (treated as a file reference only when the file exists)
* File URLs: "file:///path/to/document.pdf" or "file://relative/path.txt"
* S3 paths: "s3://bucket-name/path/to/file.pdf"
* **Binary file objects**: File handles/streams (BinaryIO)
* **Code repository URLs**: `"https://github.com/<owner>/<repo>"` (or a `gitlab.com` project,
or any `http(s)` URL ending in `.git`) is shallow-cloned and ingested as ONE code-repo item
that `cognify()` runs through the enola code graph pipeline (cross-file edges), plus the
repository's documents — the same result as adding a local code project directory.
Requires `ALLOW_HTTP_REQUESTS` and `git` on `PATH`. See
[Code repository URLs](#code-repository-urls) below.
* **Lists**: Multiple files or text strings in a single call
Supported File Formats:
* Text files (.txt, .md, .csv)
* PDFs (.pdf)
* Images (.png, .jpg, .jpeg) - transcribed via vision models, with an optional local OCR pass
* Audio files (.mp3, .wav) - transcribed to text
* Code files (.py, .js, .ts, etc.) - parsed for structure and content
* Office documents (.docx, .pptx)
See the [Supported File Formats](#supported-file-formats) table below for the full list grouped by loader,
including which formats require optional extras.
Workflow:
1. **Data Resolution**: Resolves file paths and validates accessibility
2. **Content Extraction**: Extracts text content from various file formats
3. **Dataset Storage**: Stores processed content in the specified dataset
4. **Metadata Tracking**: Records file metadata, timestamps, and user permissions
5. **Permission Assignment**: Grants user read/write/delete/share permissions on dataset
Args:
data: The data to ingest. Can be:
* Single text string: "Your text content here"
* Absolute file path: "/path/to/document.pdf" (must exist; otherwise the string is ingested as text)
* File URL: "file:///absolute/path/to/document.pdf" or "file://relative/path.txt"
* S3 path: "s3://my-bucket/documents/file.pdf"
* List of mixed types: \["text content", "/path/file.pdf", "file://doc.txt", file\_handle]
* Binary file object: open("file.txt", "rb")
* url: A web link url (https or http); a GitHub/GitLab repository URL is
cloned and indexed as a code graph instead of fetched as a page
dataset\_name: Name of the dataset to store data in. Defaults to "main\_dataset".
Create separate datasets to organize different knowledge domains.
user: User object for authentication and permissions. Uses default user if None.
Default user: "[default\_user@example.com](mailto:default_user@example.com)" (created automatically on first use).
Users can only access datasets they have permissions for.
node\_set: Optional list of node identifiers for graph organization and access control.
Used for grouping related data points in the knowledge graph.
vector\_db\_config: Optional configuration for vector database (for custom setups).
graph\_db\_config: Optional configuration for graph database (for custom setups).
dataset\_id: Optional specific dataset UUID to use instead of dataset\_name.
extraction\_rules: Optional dictionary of rules (e.g., CSS selectors, XPath) for extracting specific content from web pages using BeautifulSoup
tavily\_config: Optional configuration for Tavily API, including API key and extraction settings
soup\_crawler\_config: Optional configuration for BeautifulSoup crawler, specifying concurrency, crawl delay, and extraction rules.
Returns:
PipelineRunInfo: Information about the ingestion pipeline execution including:
* Pipeline run ID for tracking
* Dataset ID where data was stored
* Processing status and any errors
* Execution timestamps and metadata
Next Steps:
After successfully adding data, call `cognify()` to process the ingested content:
```python theme={null}
import cognee
# Step 1: Add your data (text content or file path)
await cognee.add("Your document content") # Raw text
# OR
await cognee.add("/path/to/your/file.pdf") # File path
# Step 2: Process into knowledge graph
await cognee.cognify()
# Step 3: Search and query
results = await cognee.search("What insights can you find?")
```
Example Usage:
```python theme={null}
# Add a single text document
await cognee.add("Natural language processing is a field of AI...")
# Add multiple files with different path formats
await cognee.add([
"/absolute/path/to/research_paper.pdf", # Absolute path to an existing file
"file://relative/path/to/dataset.csv", # Relative file URL
"file:///absolute/path/to/report.docx", # Absolute file URL
"s3://my-bucket/documents/data.json", # S3 path
"Additional context text", # Raw text content
"/remember to call the dentist" # No such file -> stored as raw text
])
# Add to a specific dataset
await cognee.add(
data="Project documentation content",
dataset_name="project_docs"
)
# Add a single file
await cognee.add("/home/user/documents/analysis.pdf")
# Add a single url and bs4 extract ingestion method
extraction_rules = {
"title": "h1",
"description": "p",
"more_info": "a[href*='more-info']"
}
await cognee.add("https://example.com",extraction_rules=extraction_rules)
# Add a single url and tavily extract ingestion method
Make sure to set TAVILY_API_KEY = YOUR_TAVILY_API_KEY as a environment variable
await cognee.add("https://example.com")
# Add a single url and keenable extract ingestion method
Make sure to set KEENABLE_API_KEY = YOUR_KEENABLE_API_KEY as a environment variable
(Tavily takes precedence if both keys are set.)
await cognee.add("https://example.com")
# Add multiple urls
await cognee.add(["https://example.com","https://books.toscrape.com"])
```
Environment Variables:
Required:
* LLM\_API\_KEY: API key for your LLM provider (OpenAI, Anthropic, etc.)
Optional:
* LLM\_PROVIDER: "openai" (default), "anthropic", "gemini", "ollama", "mistral", "bedrock"
* LLM\_MODEL: Model name (default: "gpt-5-mini")
* DEFAULT\_USER\_EMAIL: Custom default user email
* DEFAULT\_USER\_PASSWORD: Custom default user password
* VECTOR\_DB\_PROVIDER: "lancedb" (default), "chromadb", "pgvector"
* GRAPH\_DATABASE\_PROVIDER: "kuzu" (default), "neo4j"
* TAVILY\_API\_KEY: YOUR\_TAVILY\_API\_KEY
* KEENABLE\_API\_KEY: YOUR\_KEENABLE\_API\_KEY
* COGNEE\_REPOS\_DIR: Directory that repository URLs are shallow-cloned into
(default: `~/.cognee/repos`)
## Parameters
<ParamField type="Union[BinaryIO, list[BinaryIO], str, list[str], DataItem, list[DataItem]]">
Data to ingest. Accepts text strings, file paths (local, S3, or URLs), binary file objects, `DataItem` objects, or lists of any of these.
**`DataItem`** is a lightweight wrapper that lets you attach per-item metadata, human-readable label, and an optional stable `data_id`. Import it from `cognee.tasks.ingestion.data_item`:
```python theme={null}
from cognee.tasks.ingestion.data_item import DataItem
item = DataItem(
data="/path/to/report.pdf",
label="q4-earnings-report",
external_metadata={"title": "Q4 Financial Report", "author": "Jane Smith"},
)
await cognee.add(item)
# Mix items with different labels / metadata in a list
await cognee.add([
DataItem("Contract text …", label="contract-2024"),
DataItem("Meeting notes …", external_metadata={"source": "CRM"}),
])
```
`label` and `external_metadata` are stored on the relational `Data` record. They are not propagated into the knowledge graph automatically and are not searchable via `cognee.search()`. Use `node_set` when you need tags that flow into the graph and can be used for scoped queries.
</ParamField>
<ParamField type="str">Name of the dataset to add data to.</ParamField>
<ParamField type="User">User performing the operation. Uses default user if not provided.</ParamField>
<ParamField type="Optional[List[str]]">List of node set names to associate with the data.</ParamField>
<ParamField type="dict">Override vector database configuration for this operation.</ParamField>
<ParamField type="dict">Override graph database configuration for this operation.</ParamField>
<ParamField type="Optional[UUID]">UUID of an existing dataset to add data to. Alternative to dataset\_name.</ParamField>
<ParamField type="Optional[List[Union[str, dict[str, dict[str, Any]]]]]">Custom loader configuration for specific file types.</ParamField>
<ParamField type="bool">If true, skip data that has already been ingested. The skip runs whenever this **or** `data_cache` is true, so re-ingesting an already-stored item requires both to be `False`. See [Incremental loading and deduplication](/core-concepts/main-operations/legacy-operations/cognify#examples-and-details).</ParamField>
<ParamField type="bool">Companion flag to `incremental_loading` — either one being true enables the already-processed skip for a data item.</ParamField>
<ParamField type="Optional[int]">Number of data items to process per batch.</ParamField>
<ParamField type="Optional[LLMConfig]">LLM settings to install into the current async context for this ingestion operation. When omitted, Cognee uses the active context config or global LLM config. Import `LLMConfig` from `cognee.infrastructure.llm.config`.</ParamField>
<ParamField type="Optional[EmbeddingConfig]">Embedding settings to install into the current async context for this ingestion operation. When omitted, Cognee uses the active context config or global embedding config. Import `EmbeddingConfig` from `cognee.infrastructure.databases.vector.embeddings.config`.</ParamField>
<ParamField type="Optional[float]">Floating-point score stored on the `Data` record for retrieval ranking. Applied uniformly to all items in the batch. Use a higher value to make items more likely to surface in ranked results.</ParamField>
<ParamField type="str">Column name for primary key when ingesting dlt resources or database connection strings. Auto-detected if not specified. For CSV files, pass it through `preferred_loaders=[{"dlt_csv_loader": {"primary_key": "..."}}]` instead — this kwarg does not reach the CSV loader.</ParamField>
<ParamField type="str">How to handle existing data for dlt sources: "replace" (drop and recreate), "merge" (upsert on `primary_key`), or "append" (always insert). Any other value raises `InvalidDLTArgumentError`. This controls dlt's staging snapshot only — a growing source also needs the ingestion skip turned off, see [Re-ingesting a source that keeps growing](/core-concepts/main-operations/legacy-operations/cognify#examples-and-details).</ParamField>
<ParamField type="str">SQL query that filters what is ingested when using a database connection string as input. Accepts `SELECT ... FROM <table> [WHERE ...]`, where the `FROM` target may be schema-qualified and may carry a table alias. A `WHERE` clause that references the alias, and a `FROM` clause containing `JOIN`, raise a `ValueError` — see [Database Connection String](/integrations/dlt-integration#database-connection-string) for the accepted shapes, the rewrites, and the two detection caveats.</ParamField>
## Supported Input Types
| Type | Example |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| Text string | `"Cognee is a knowledge graph platform."` |
| File path | `"/path/to/document.pdf"` |
| S3 path | `"s3://bucket/file.txt"` |
| URL | `"https://example.com/article"` |
| Code repository URL | `"https://github.com/owner/repo"` (cloned and indexed as a code graph — see [below](#code-repository-urls)) |
| Binary file | `open("file.pdf", "rb")` |
| Mixed list | `["text", "/path/file.pdf", open("f.txt", "rb")]` |
| dlt resource | `@dlt.resource()` decorated generator |
| CSV file | `"/path/to/data.csv"` (routed through dlt structured ingestion by the loader engine when `cognee[dlt]` is installed; flattened to text otherwise) |
| DB connection | `"postgresql://user:pass@host/db"` |
## Code Repository URLs
A string that names a whole git repository is recognised by shape and handled as code rather than as a web page: Cognee shallow-clones it (`git clone --depth 1`) and hands `cognify()` one code-repo item, which takes the CODE\_REPO route (a single enola pass with cross-file edges) alongside the repository's own documents. This is the same graph you get from adding a local code project directory.
```python theme={null}
# Indexed as a code graph — no content_type needed
await cognee.add("https://github.com/owner/repo", dataset_name="my_repo")
await cognee.cognify(datasets=["my_repo"])
# Same for remember()
await cognee.remember("https://github.com/owner/repo", dataset_name="my_repo")
```
Recognised as a repository:
* `https://github.com/<owner>/<repo>` — a repository root on `github.com`
* `https://gitlab.com/<group>/<repo>` — a project on `gitlab.com`, nested groups included
* Any `http(s)` URL whose last path segment ends in `.git`, on any host
An optional `.git` suffix, trailing slash, `www.` prefix, query string, or fragment is tolerated and stripped before cloning.
Not recognised (these stay on the ordinary web-page path):
* Deeper forge URLs such as `/blob/`, `/tree/`, `/issues`, `/pull/`, and GitLab's `/-/` pages
* Forge site pages such as `https://github.com/topics/rust` or `https://gitlab.com/explore`
* `git@…` and `ssh://…` specs — pass these explicitly through `remember(..., content_type="code")`
Requirements and behaviour:
* `git` must be on the server's `PATH`, and `ALLOW_HTTP_REQUESTS` must not be `false`
* Clones land under `COGNEE_REPOS_DIR` (default `~/.cognee/repos`), are reused across calls, and are refreshed with a best-effort `git pull` — budget disk space accordingly
* Cloning is credential-free. For a private repository, use `remember(url, content_type="code", repo_credentials="<token>")`, described in [Code Graph](/guides/code-graph)
## Supported File Formats
| Loader | Extensions | Install extra |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ |
| **CodeLoader** | `.py` `.js` `.ts` `.go` `.rs` `.java` and 25 more source-code extensions — see [Loaders](/core-concepts/further-concepts/loaders); these take the code graph pipeline instead of LLM extraction | — (built-in) |
| **TextLoader** | `.txt` `.md` `.json` `.xml` `.yaml` `.yml` `.log` | — (built-in) |
| **DltCsvLoader** | `.csv` — takes precedence over `CsvLoader` when installed; routes rows through [dlt structured ingestion](/integrations/dlt-integration) | `pip install cognee[dlt]` |
| **CsvLoader** | `.csv` | — (built-in) |
| **PyPdfLoader** | `.pdf` | — (built-in) |
| **ImageLoader** | `.png` `.jpg` `.jpe` `.jpeg` `.gif` `.webp` `.bmp` `.tif` `.tiff` `.heic` `.avif` `.ico` `.psd` `.apng` `.cr2` `.dwg` `.xcf` `.jxr` `.jpx` | — (built-in) |
| **AudioLoader** | `.mp3` `.wav` `.aac` `.flac` `.ogg` `.m4a` `.mid` `.amr` `.aiff` | — (built-in) |
| **UnstructuredLoader** | `.docx` `.doc` `.odt` `.xlsx` `.xls` `.ods` `.pptx` `.ppt` `.odp` `.rtf` `.html` `.htm` `.eml` `.msg` `.epub` | `pip install cognee[docs]` |
| **AdvancedPdfLoader** | `.pdf` (layout-aware, preserves tables) | `pip install cognee[docs]` |
| **BeautifulSoupLoader** | `.html` | `pip install cognee[scraping]` |
See [Loaders](/core-concepts/further-concepts/loaders) for how to override the default loader selection or register custom loaders.
## Examples
```python theme={null}
import cognee
# Add text
await cognee.add("Cognee builds knowledge graphs from your data.")
# Add a file
await cognee.add("/path/to/report.pdf", dataset_name="reports")
# Add multiple items to a named dataset
await cognee.add(
["First document text", "/path/to/second.pdf"],
dataset_name="my_project",
)
# Add with custom node set
await cognee.add("Technical spec content", node_set=["engineering"])
# Add a public repository — cloned and indexed as a code graph, not scraped
await cognee.add("https://github.com/owner/repo", dataset_name="my_repo")
# Add structured data via dlt resource
import dlt
@dlt.resource()
def my_data():
yield [{"id": 1, "name": "Alice"}, {"id": 2, "name": "Bob"}]
await cognee.add(my_data, dataset_name="people", primary_key="id")
# Add a CSV file (with cognee[dlt] installed, the loader engine routes it
# through dlt structured ingestion; dlt options travel via preferred_loaders)
await cognee.add(
"/path/to/data.csv",
dataset_name="csv_data",
preferred_loaders=[{"dlt_csv_loader": {"primary_key": "id"}}],
)
# Add from a database with filtering
await cognee.add(
"postgresql://user:pass@host/db",
dataset_name="db_data",
primary_key="id",
query="SELECT * FROM users WHERE active = true",
)
```
<Note>
For a complete guide on structured data ingestion with dlt, see the [dlt integration page](/integrations/dlt-integration).
</Note>
# agents
Source: https://docs.cognee.ai/python-api/agents
Agent management: create, list, inspect, and delete agents, and manage their connections
# cognee.agents
Static class for managing agents and their connections. Each agent is backed by
its own agent user with a one-time API key, and every operation is scoped to the
acting user — permissions are enforced per acting user before any agent is
created or any dataset is granted.
```python theme={null}
import cognee
# Create an agent with read/write on a dataset
agent = await cognee.agents.create("my-agent", datasets=["my_project"])
# List your agents
all_agents = await cognee.agents.list()
```
<Note>
Agent emails are displayed in their `{slug}@cognee.agent` form. Internally each
agent user is minted with a unique `{slug}+{parent_id}@cognee.agent` address; the
SDK strips the `+{parent_id}` segment before returning the email.
</Note>
## Methods
### agents.create()
```python theme={null}
await cognee.agents.create(
name: str,
datasets: Optional[list[str | UUID]] = None,
user: Optional[User] = None,
) -> dict
```
Create a new agent and, optionally, grant it `read` and `write` access to one or
more datasets. Each requested dataset is resolved and the **calling** user's
`read` access to it is verified *before* the agent user is minted, so a failed
authorization never leaves an orphaned agent user with a live API key.
| Parameter | Type | Default | Notes |
| ---------- | ----------------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------- |
| `name` | `str` | required | Display name for the agent. |
| `datasets` | `Optional[list[str \| UUID]]` | `None` | Dataset names or UUIDs to grant the agent `read`/`write` on. The calling user must already have `read` access to each. |
| `user` | `Optional[User]` | `None` | Acting user. If omitted, Cognee resolves the default user. |
Returns a dict with the agent id, display email, and the **one-time** API key:
```python theme={null}
{
"agent_id": "…",
"agent_email": "my-agent@cognee.agent",
"agent_api_key": "…", # shown once — store it now
}
```
<Warning>
`agent_api_key` is returned only on creation and cannot be retrieved again. Store
it at creation time.
</Warning>
### agents.list()
```python theme={null}
await cognee.agents.list(user=None) -> list[dict]
```
List the agents owned by the resolved user. Each entry contains `agent_id`,
`agent_email`, and `api_key_label`.
### agents.get()
```python theme={null}
await cognee.agents.get(agent_id: str | UUID, user=None) -> dict
```
Return details (`agent_id`, `agent_email`, `api_key_label`) for a single agent.
Raises `ValueError` if the agent does not exist, or `PermissionDeniedError` if the
acting user is not authorized to view it.
### agents.delete()
```python theme={null}
await cognee.agents.delete(agent_id: str | UUID, user=None) -> None
```
Delete an agent by id. Raises `ValueError` if the agent does not exist, or
`PermissionDeniedError` if the acting user is not authorized to delete it.
### agents.register()
```python theme={null}
await cognee.agents.register(
agent_session_name: str,
user: Optional[User] = None,
type: AgentConnectionType = "api",
memory_mode: AgentMemoryMode = "unknown",
session_id: Optional[str] = None,
dataset_ids: Optional[list[str]] = None,
dataset_names: Optional[list[str]] = None,
source: AgentSource = "api",
origin_function: Optional[str] = None,
metadata: Optional[dict] = None,
) -> dict
```
Register an agent connection (session). The acting user's `read` access to every
supplied dataset (by id and by name) is validated *before* the connection is
created. Returns the connection as a JSON-serializable dict.
| Parameter | Type | Default | Notes |
| -------------------- | --------------------- | ----------- | ----------------------------------------------------------------------------- |
| `agent_session_name` | `str` | required | Name of the agent session/connection. |
| `type` | `AgentConnectionType` | `"api"` | One of `sdk`, `api`, `mcp`, `claude_code`, `opencode`, `workflow`, `unknown`. |
| `memory_mode` | `AgentMemoryMode` | `"unknown"` | One of `session`, `cognee`, `hybrid`, `none`, `unknown`. |
| `session_id` | `Optional[str]` | `None` | Optional session id to associate. |
| `dataset_ids` | `Optional[list[str]]` | `None` | Dataset UUIDs to associate; calling user must have `read` on each. |
| `dataset_names` | `Optional[list[str]]` | `None` | Dataset names to associate; calling user must have `read` on each. |
| `source` | `AgentSource` | `"api"` | One of `agent_memory`, `session_trace`, `serve`, `api_key`, `mcp`, `api`. |
| `origin_function` | `Optional[str]` | `None` | Optional originating function label. |
| `metadata` | `Optional[dict]` | `None` | Arbitrary metadata stored with the connection. |
| `user` | `Optional[User]` | `None` | Acting user. If omitted, Cognee resolves the default user. |
### agents.unregister()
```python theme={null}
await cognee.agents.unregister(agent_session_name: str, user=None) -> int
```
Unregister an agent connection by session name. Returns the number of remaining
active connections.
### agents.list\_connections()
```python theme={null}
await cognee.agents.list_connections(
user: Optional[User] = None,
agent_id: Optional[str | UUID] = None,
range_key: RangeLiteral = "30d",
status_filter: Optional[str] = None,
include_sources: bool = True,
active_only: bool = True,
limit: int = 50,
offset: int = 0,
) -> dict
```
List active agent connections and their memory sources, scoped to the acting
user. Connections that are bound neither to an owning user nor to any dataset are
filtered out of the result so a caller never sees connections outside their own
scope, and `total`/`has_more` are adjusted accordingly.
| Parameter | Type | Default | Notes |
| ----------------- | ----------------------- | ------- | ---------------------------------------------------------- |
| `agent_id` | `Optional[str \| UUID]` | `None` | Filter to a single agent. |
| `range_key` | `RangeLiteral` | `"30d"` | Time window: one of `24h`, `7d`, `30d`, `all`. |
| `status_filter` | `Optional[str]` | `None` | Filter by connection status. |
| `include_sources` | `bool` | `True` | Include memory sources in the response. |
| `active_only` | `bool` | `True` | Return only active connections. |
| `limit` | `int` | `50` | Maximum number of connections to return. |
| `offset` | `int` | `0` | Result offset for pagination. |
| `user` | `Optional[User]` | `None` | Acting user. If omitted, Cognee resolves the default user. |
### agents.get\_connection()
```python theme={null}
await cognee.agents.get_connection(
agent_id: str | UUID,
user: Optional[User] = None,
agent_session_name: Optional[str] = None,
) -> Optional[dict]
```
Return the detail for a single agent connection, or `None` if no matching
connection is found.
## CLI
The same operations are available from the command line via the
[`cognee-cli agents`](/cognee-cli/overview#manage-agents) command.
# cognify()
Source: https://docs.cognee.ai/python-api/cognify
Transform raw data into a structured knowledge graph
# cognee.cognify()
```python theme={null}
async def cognify(
datasets: Union[str, list[str], list[UUID]] = None,
user: User = None,
graph_model: BaseModel = KnowledgeGraph,
chunker = TextChunker,
chunk_size: int = None,
chunks_per_batch: int = None,
config: Config = None,
vector_db_config: dict = None,
graph_db_config: dict = None,
run_in_background: bool = False,
incremental_loading: bool = True,
custom_prompt: Optional[str] = None,
temporal_cognify: bool = False,
data_per_batch: int = 20,
llm_config: Optional[LLMConfig] = None,
embedding_config: Optional[EmbeddingConfig] = None,
dry_run: bool = False,
raise_on_error: bool = True,
chunk_attachment: Optional[Literal["direct", "all"]] = None,
extractor: Optional[Literal["llm", "gliner"]] = None,
)
```
## Description
Transform ingested data into a structured knowledge graph.
This is the core processing step in Cognee that converts raw text and documents
into an intelligent knowledge graph. It analyzes content, extracts entities and
relationships, and creates semantic connections for enhanced search and reasoning.
Prerequisites:
* **LLM\_API\_KEY**: Must be configured (required for entity extraction and graph generation
on the default `llm` extractor; `extractor="gliner"` extracts the graph and the chunk
summaries with a local GLiNER2 model instead — see
[LLM-free extraction with GLiNER](#llm-free-extraction-with-gliner))
* **Data Added**: Must have data previously added via `cognee.add()`
* **Vector Database**: Must be accessible for embeddings storage
* **Graph Database**: Must be accessible for relationship storage
Input Requirements:
* **Datasets**: Must contain data previously added via `cognee.add()`
* **Content Types**: Works with any text-extractable content including:
* Natural language documents
* Structured data (CSV, JSON)
* Code repositories
* Academic papers and technical documentation
* Mixed multimedia content (with text extraction)
Processing Pipeline:
1. **Document Classification**: Identifies document types and structures
2. **Text Chunking**: Breaks content into semantically meaningful segments
3. **Entity Extraction**: Identifies key concepts, people, places, organizations
4. **Relationship Detection**: Discovers connections between entities
5. **Graph Construction**: Builds semantic knowledge graph with embeddings
6. **Content Summarization**: Creates hierarchical summaries for navigation
Graph Model Customization:
The `graph_model` parameter allows custom knowledge structures:
* **Default**: General-purpose KnowledgeGraph for any domain
* **Custom Models**: Domain-specific schemas (e.g., scientific papers, code analysis)
* **Ontology Integration**: Pass an ontology resolver via `config` (or set the `ONTOLOGY_FILE_PATH` environment variable) for predefined vocabularies.
An optional `"ontology_mode"` key in the same `ontology_config` picks how strictly the ontology is applied — `"annotate"` (default: enrich only) or
`"strict"` (drop extracted entities the ontology does not ground, and their edges). It overrides the `ONTOLOGY_MODE` environment variable for this
call; unrecognized values warn and fall back to `"annotate"`.
Args:
datasets: Dataset name(s) or dataset uuid to process. Processes all available data if None.
* Single dataset: "my\_dataset"
* Multiple datasets: \["docs", "research", "reports"]
* None: Process all datasets for the user
user: User context for authentication and data access. Uses default if None.
graph\_model: Pydantic model defining the knowledge graph structure.
Defaults to KnowledgeGraph for general-purpose processing.
chunker: Text chunking strategy (TextChunker, LangchainChunker).
* TextChunker: Paragraph-based chunking (default, most reliable)
* LangchainChunker: Recursive character splitting with overlap
Determines how documents are segmented for processing.
chunk\_size: Maximum tokens per chunk. Auto-calculated based on LLM if None.
Formula: min(embedding\_max\_completion\_tokens, llm\_max\_completion\_tokens // 2)
Default limits: \~512-8192 tokens depending on models.
Smaller chunks = more granular but potentially fragmented knowledge.
chunks\_per\_batch: Number of chunks to be processed in a single batch in Cognify tasks.
vector\_db\_config: Custom vector database configuration for embeddings storage.
graph\_db\_config: Custom graph database configuration for relationship storage.
run\_in\_background: If True, starts processing asynchronously and returns immediately.
If False, waits for completion before returning.
Background mode recommended for large datasets (>100MB).
Use pipeline\_run\_id from return value to monitor progress.
custom\_prompt: Optional custom prompt string to use for entity extraction and graph generation.
If provided, this prompt will be used instead of the default prompts for
knowledge graph extraction. The prompt should guide the LLM on how to
extract entities and relationships from the text content.
dry\_run: If True, return a stage-level estimate of LLM token usage and rough cost
without making LLM calls or writing graph results. The estimate covers all
data in the selected dataset(s); an incremental run may process fewer items.
chunk\_attachment: How widely each chunk links into the graph extracted from it.
Accepts `"direct"`, `"all"`, or `None`; omitting it is the same as `"direct"`.
Requires a custom `DataPoint` `graph_model`. See
[Chunk attachment](#chunk-attachment).
extractor: Which implementation fills the extract-and-summarize step of the standard
pipeline. Accepts `"llm"`, `"gliner"`, or `None`; the explicit argument wins
over the `GRAPH_EXTRACTOR` setting and `None` resolves to `"llm"`. See
[LLM-free extraction with GLiNER](#llm-free-extraction-with-gliner).
Returns:
Union\[dict, list\[PipelineRunInfo], DryRunEstimate]:
* **Blocking mode**: Dictionary mapping dataset\_id -> PipelineRunInfo with:
* Processing status (completed/failed/in\_progress)
* Extracted entity and relationship counts
* Processing duration and resource usage
* Error details if any failures occurred
* **Background mode**: List of PipelineRunInfo objects for tracking progress
* Use pipeline\_run\_id to monitor status
* Check completion via pipeline monitoring APIs
Next Steps:
After successful cognify processing, use search functions to query the knowledge:
```python theme={null}
import cognee
from cognee import SearchType
# Process your data into knowledge graph
await cognee.cognify()
# Query for insights using different search types:
# 1. Natural language completion with graph context
insights = await cognee.search(
"What are the main themes?",
query_type=SearchType.GRAPH_COMPLETION
)
# 2. Get entity relationships and connections
relationships = await cognee.search(
"connections between concepts",
query_type=SearchType.GRAPH_COMPLETION
)
# 3. Find relevant document chunks
chunks = await cognee.search(
"specific topic",
query_type=SearchType.CHUNKS
)
```
Advanced Usage:
```python theme={null}
# Custom domain model for scientific papers
class ScientificPaper(DataPoint):
title: str
authors: List[str]
methodology: str
findings: List[str]
await cognee.cognify(
datasets=["research_papers"],
graph_model=ScientificPaper,
)
# Ground extraction in an ontology (there is no `ontology_file_path` argument;
# pass a resolver through `config` instead).
from cognee.modules.ontology.rdf_xml.RDFLibOntologyResolver import RDFLibOntologyResolver
from cognee.modules.ontology.ontology_config import Config
config: Config = {
"ontology_config": {
"ontology_resolver": RDFLibOntologyResolver(ontology_file="scientific_ontology.owl"),
# Optional; overrides ONTOLOGY_MODE. "annotate" (default) enriches only,
# "strict" drops extracted entities the ontology does not ground.
"ontology_mode": "annotate",
}
}
await cognee.cognify(datasets=["research_papers"], config=config)
# Background processing for large datasets
run_info = await cognee.cognify(
datasets=["large_corpus"],
run_in_background=True
)
# Check status later with run_info.pipeline_run_id
```
Environment Variables:
Required:
* LLM\_API\_KEY: API key for your LLM provider
Optional (same as add function):
* LLM\_PROVIDER, LLM\_MODEL, VECTOR\_DB\_PROVIDER, GRAPH\_DATABASE\_PROVIDER
* AUTO\_RATE\_LIMIT: Turn the rate limiter on automatically when the provider shows overload evidence (default: True)
* LLM\_RATE\_LIMIT\_ENABLED: Enable rate limiting from the first request (default: False)
* LLM\_RATE\_LIMIT\_REQUESTS: Max requests per interval (default: 60; 10 for local inference servers)
Optional (contradiction detection — see [Contradiction detection](#contradiction-detection)):
* CONTRADICTION\_DETECTION: Append the opt-in contradiction check to the pipeline (default: False)
* CONTRADICTION\_CONFIDENCE\_THRESHOLD: Minimum LLM confidence for a pair to be flagged (default: 0.5)
* CONTRADICTION\_MAX\_FACTS: Cap on the facts sent to the LLM in a single check (default: 500)
Optional (provenance ledger — see [Provenance ledger](#provenance-ledger)):
* PROVENANCE\_TRACKING: Append the opt-in provenance-ledger task to the pipeline (default: False)
Optional (extractor — see [LLM-free extraction with GLiNER](#llm-free-extraction-with-gliner)):
* GRAPH\_EXTRACTOR: Which implementation fills the extract-and-summarize step, `llm` or `gliner` (default: `llm`)
## Parameters
<ParamField type="Union[str, list[str], list[UUID]]">Dataset name(s) or UUID(s) to process. Processes all datasets if not specified.</ParamField>
<ParamField type="User">User performing the operation.</ParamField>
<ParamField type="BaseModel">Pydantic model defining the knowledge graph schema. Defaults to KnowledgeGraph.</ParamField>
<ParamField type="Any">Text chunking strategy class.</ParamField>
<ParamField type="int">Maximum size of text chunks in tokens.</ParamField>
<ParamField type="int">Number of chunks to process per LLM batch.</ParamField>
<ParamField type="Config">Override the full Cognee config for this run.</ParamField>
<ParamField type="dict">Override vector database configuration.</ParamField>
<ParamField type="dict">Override graph database configuration.</ParamField>
<ParamField type="bool">If true, return immediately and process in background.</ParamField>
<ParamField type="bool">If true, skip already-processed data. The skip runs whenever this **or** `data_cache` is true, so a full reprocess requires both to be `False`. See [Incremental loading and deduplication](/core-concepts/main-operations/legacy-operations/cognify#examples-and-details).</ParamField>
<ParamField type="bool">Companion flag to `incremental_loading` — either one being true enables the already-processed skip for a data item.</ParamField>
<ParamField type="Optional[str]">Custom system prompt for entity/relationship extraction.</ParamField>
<ParamField type="bool">Enable temporal-aware processing.</ParamField>
<ParamField type="int">Number of data items per processing batch.</ParamField>
<ParamField type="Optional[LLMConfig]">LLM settings to install into the current async context for this graph-building operation. When omitted, Cognee uses the active context config or global LLM config. Import `LLMConfig` from `cognee.infrastructure.llm.config`.</ParamField>
<ParamField type="Optional[EmbeddingConfig]">Embedding settings to install into the current async context for this graph-building operation. When omitted, Cognee uses the active context config or global embedding config. Import `EmbeddingConfig` from `cognee.infrastructure.databases.vector.embeddings.config`.</ParamField>
<ParamField type="bool">If true, return a `DryRunEstimate` of LLM token usage and rough cost instead of running the pipeline. No LLM calls are made and no graph results are written. See [Dry-run cost estimation](#dry-run-cost-estimation).</ParamField>
<ParamField type="bool">If true, a blocking run whose pipeline errored raises `CognifyFailedError` instead of returning an errored `PipelineRunInfo`. Set `False` to keep the previous swallow-and-return behavior. Ignored when `run_in_background=True`. See [Failed runs](#failed-runs).</ParamField>
<ParamField type="Optional[Literal["direct", "all"]]">How widely each chunk links into the graph extracted from it. `"all"` links the chunk to every stored node reachable from its extracted root; `"direct"` and omitting it keep the single-root linkage. Requires a custom `DataPoint` `graph_model`. See [Chunk attachment](#chunk-attachment).</ParamField>
<ParamField type="Optional[Literal["llm", "gliner"]]">Which implementation fills the extract-and-summarize step of the standard pipeline. `"gliner"` builds the graph and the chunk summaries with a local GLiNER2 model instead of the LLM. Omitting it (or passing `None`) falls back to the `GRAPH_EXTRACTOR` setting, which itself defaults to `"llm"`; any other value raises `ValueError`. See [LLM-free extraction with GLiNER](#llm-free-extraction-with-gliner).</ParamField>
## Failed runs
A blocking `cognify()` whose pipeline ends in an error raises `CognifyFailedError` instead of returning a run info with status `PipelineRunErrored`:
```python theme={null}
import cognee
from cognee.modules.pipelines.exceptions import CognifyFailedError
try:
await cognee.cognify(datasets=["my_dataset"])
except CognifyFailedError as error:
print(error) # Cognify failed for dataset 'my_dataset': AuthenticationError: … | Pass raise_on_error=False …
error.dataset_name # "my_dataset"
error.error_class # class name of the underlying exception, e.g. "AuthenticationError"
error.error_message # PII-scrubbed message of the underlying error
```
The same exception type covers both failure paths: a run that finished with an errored run info, and a run-level exception that escaped the pipeline (an authentication error from the LLM provider, for example). Already-typed Cognee API errors — `PermissionDeniedError`, `DatasetNotFoundError`, and other `CogneeApiError` subclasses — pass through unwrapped, so existing `except` clauses keep matching. When several datasets are processed, the first errored one raises, and its name is in `dataset_name`. On the run-level-exception path there is no run info to read the name from, so `dataset_name` is the stringified `datasets` argument (`"['my_dataset']"`) rather than a single resolved name.
Pass `raise_on_error=False` to get the previous behavior:
```python theme={null}
from cognee.modules.pipelines.models.PipelineRunInfo import get_errored_run_info
result = await cognee.cognify(datasets=["my_dataset"], raise_on_error=False)
errored = get_errored_run_info(result)
if errored:
print(errored.error_class, errored.error_message)
```
On the errored-run-info path `raise_on_error=False` hands back the `{dataset_id: PipelineRunInfo}` mapping as before; when the failure was a run-level exception, it re-raises the original exception untouched.
<Note>
Background runs (`run_in_background=True`) never raise here — the call returns as soon as the run starts. Their failures land on the pipeline run record, and a later `recall()` surfaces them through the [`build_failed` marker](/python-api/recall#warming-up-marker).
The REST `/cognify` route and `update()` call `cognify()` with `raise_on_error=False`, and the `/remember` route passes the same flag through `remember()`, so their responses (including the errored run-info bodies and the 409 behavior) are unchanged.
</Note>
## Chunk attachment
With a custom `DataPoint` `graph_model`, a chunk gets exactly one `contains` edge into the graph extracted from it — to the root the LLM filled in — and every other extracted entity is reachable only by walking down from that root. (The default `KnowledgeGraph` path already links each extracted entity to its chunk, which is why this argument does not apply to it.) Pass `chunk_attachment="all"` to link the chunk once to **every** stored node reachable from that root instead, so any extracted entity is one hop from the chunk it came from:
```python theme={null}
from typing import List
import cognee
from cognee.low_level import DataPoint
class Activity(DataPoint):
name: str
metadata: dict = {"index_fields": ["name"], "identity_fields": ["name"]}
class Person(DataPoint):
name: str
likes: List[Activity] | None = None
metadata: dict = {"index_fields": ["name"], "identity_fields": ["name"]}
class PeopleGraph(DataPoint):
people: List[Person]
await cognee.cognify(graph_model=PeopleGraph, chunk_attachment="all")
# Every Person *and* every Activity now sits one `contains` edge from its chunk,
# instead of only the PeopleGraph root.
```
| Value | Chunk links to |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `None` (omitted) | Same as `"direct"` — existing behavior, so existing runs are unaffected. |
| `"direct"` | The extracted root; or, when that root is a [transparent container](/core-concepts/building-blocks/datapoints#transparent-containers-nodes-that-group-rather-than-describe), the children that replaced it. |
| `"all"` | Every node the run stores from that root — the root itself included, unless it is a transparent container and so never stored. Each node is linked once. |
Behavior notes:
* **Requires a custom `DataPoint` `graph_model`.** Passing it with the default `KnowledgeGraph` (or a subclass of it) raises a `ValueError` — that path already attaches every extracted entity to its chunk, so there is nothing to widen.
* **Rejected up front.** An invalid value, a non-`DataPoint` `graph_model`, `temporal_cognify=True`, or a connection to a remote instance via `serve()` each raise a `ValueError` before any pipeline work starts. `cognify()` forwards unknown keyword arguments into the LLM call, so an unusable value has to raise rather than be silently dropped.
* **Permitted with `dry_run=True`.** The estimate covers the two LLM-heavy stages only and is genuinely unchanged by attachment, so a dry run behaves exactly as it does without the argument.
* **`cognify()`-only.** It is not a `remember()` keyword and not a field on the REST `POST /api/v1/cognify` body.
* **Standard-routed items only**, exactly like `graph_model` — DLT-source manifests and code files run their own task lists and ignore both.
* **Independent of `metadata["transparent"]`.** Transparency is a property of the model; attachment is a property of the run. They compose.
* **Cost of `"all"`.** Edge indexing embeds one `EdgeType` per distinct edge text, and a `contains` edge's text is `"<chunk label> contains <node label>."` — so a model that yields N nodes per chunk means roughly N extra embedded rows per chunk.
## LLM-free extraction with GLiNER
By default, `cognify()` calls the LLM twice per chunk batch: once to extract entities and relationships, once to summarize. Pass `extractor="gliner"` to fill that same extract-and-summarize step with the local [GLiNER2](https://huggingface.co/fastino/gliner2.5-base-v1) model instead — **no LLM call is made for extraction or for summaries**. Everything else about the run is unchanged: documents are still classified and chunked the same way, and `add_data_points` still embeds what it writes, so an embedding provider is still required.
Install the extra first:
```bash theme={null}
uv pip install "cognee[gliner]"
```
The first run downloads `fastino/gliner2.5-base-v1` (\~800 MB) into the Hugging Face cache.
```python theme={null}
import cognee
await cognee.add("Tim Cook is the chief executive officer of Apple Inc.")
# Per call — the explicit argument wins over GRAPH_EXTRACTOR
await cognee.cognify(extractor="gliner")
```
Or select it for the whole process:
```bash theme={null}
GRAPH_EXTRACTOR=gliner # default: llm
```
<Note>
The cognify config is cached for the lifetime of the process, so set `GRAPH_EXTRACTOR` in your `.env` or environment **before** the first `cognify()` call. The per-call `extractor` argument is read on every call and always wins; passing anything other than `"llm"`, `"gliner"`, or `None` raises `ValueError`.
</Note>
### What GLiNER extracts
GLiNER is a span-labelling model, not a generative one, so the extraction schema is a closed list of entity and relation labels rather than a free-form prompt. Cognee builds that list from your ontology when one is configured (`ONTOLOGY_FILE_PATH`, or a resolver passed through `config`), and otherwise from built-in label banks. Expect the resulting graph to differ from an LLM-extracted one, and the summaries to be short and deterministic rather than written prose.
### Limitations
`extractor="gliner"` produces the generic `KnowledgeGraph`, and it is rejected up front — before any pipeline task runs — wherever the run could not honour it:
| Combination | Behavior |
| -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Custom `graph_model` | Raises `ValueError` — only the `llm` extractor fills a custom schema. |
| `temporal_cognify=True` | Raises `ValueError` — the temporal pipeline extracts events with the LLM. |
| `dry_run=True` | Raises `ValueError` — the estimator prices the LLM extraction pipeline and has no cost model for GLiNER. |
| Connected to a remote instance via `serve()` | Passing `extractor` at all — including `"llm"` — raises `ValueError`, because the remote call carries no extractor field. A `GRAPH_EXTRACTOR` setting is not forwarded either, but it raises nothing: the remote silently runs its own default. Call `cognee.disconnect()` to choose the extractor locally. |
| Unrecognized `**kwargs` | Raises `ValueError` naming them, instead of forwarding them to an LLM call that no longer happens. |
| `custom_prompt` | Accepted but **ignored**, with a `custom_prompt is ignored when extractor='gliner'.` warning — there is no prompt to replace. |
Every row except the `serve()` one reads the *resolved* extractor, so a `GRAPH_EXTRACTOR=gliner` setting raises the same errors as the explicit argument. The `serve()` check is the exception: it looks only at whether you passed `extractor`.
<Note>
`CONTRADICTION_DETECTION=true` appends an LLM task to the pipeline whichever extractor you choose, so a run that must make no LLM call at all needs that flag off (it is off by default).
</Note>
### Running with no LLM key at all
Two related behaviors make a fully LLM-free `remember()` → `recall()` loop possible once extraction is LLM-free:
* **The provider preflight skips its LLM half** when the pipeline ahead runs no LLM task, so `add()` / `remember()` no longer fail on a missing `LLM_API_KEY`. Embedding configuration is still checked. See [Provider consistency preflight](/setup-configuration/overview#provider-consistency-preflight).
* **`recall()` without a `query_type` resolves to `SearchType.CHUNKS`** when no usable LLM is configured, because nothing could write a completion answer. See [Auto-routing](/python-api/recall#parameters).
Pair the extractor with a local embedding provider (for example `fastembed`) and `AUTO_FEEDBACK=false` — the per-turn feedback analysis is itself an LLM call. Anything ending in `_COMPLETION` still needs an LLM to write the answer, so those search types remain unavailable without a key.
Two guides walk through this end to end: [Custom GLiNER Extraction](/guides/gliner-llm-free-cognify) drives the extractor's task list directly, with your own entity and relation labels and a stats object reporting what it kept, and [Recall Without an LLM Key](/guides/no-llm-remember-recall) runs a full `remember()` → `recall()` loop with no key present — including the concrete `EMBEDDING_MODEL` and `EMBEDDING_DIMENSIONS` values for a CPU-only `fastembed` setup.
## Dry-run cost estimation
Pass `dry_run=True` to preview the LLM token usage and rough USD cost of a `cognify()` run **without making any LLM calls or writing graph results**. This is useful for budgeting a large dataset before committing to the run.
```python theme={null}
import cognee
estimate = await cognee.cognify(datasets=["my_dataset"], dry_run=True)
print(estimate) # human-readable summary table
print(estimate.estimated_cost_usd) # e.g. 0.012345
print(estimate.total_tokens) # input + output tokens across stages
print(estimate.to_dict()) # JSON-serializable dict
```
The estimate covers the two LLM-heavy stages of the default pipeline — `structured_graph_extraction` and `chunk_summarization` — reusing the real document classifier, chunker, and prompt templates so chunk and call counts track an actual run.
The returned `DryRunEstimate` exposes:
| Field | Type | Description |
| ------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------- |
| `operation` | `str` | `"cognify"` for this call. |
| `model` | `str` | The configured LLM model the estimate is priced against. |
| `chunks` | `int` | Number of chunks that would trigger LLM calls. |
| `chunk_tokens` | `int` | Total input tokens across those chunks. |
| `input_tokens` / `output_tokens` / `total_tokens` | `int` | Aggregate token counts across all stages. |
| `estimated_cost_usd` | `float` | Rough total cost across all stages. |
| `skipped_items` | `int` | Items excluded from estimation (e.g. audio/image items, DLT row chunks, code files). |
| `warnings` | `list[str]` | Notes about approximations (reasoning-model output allowance, skipped items, or missing pricing entries). |
| `stages` | `list` | Per-stage breakdown (`name`, `calls`, `input_tokens`, `output_tokens`, `total_tokens`, `estimated_cost_usd`). |
Behavior notes:
* **Datasets are resolved read-only.** Unlike a normal run, a dry run never creates a missing dataset, so estimating a typo'd dataset name fails loudly instead of silently creating one.
* **Estimates are upper bounds for re-runs.** With `incremental_loading=True`, a real run skips already-processed documents, so a dry run may over-estimate a re-run.
* **Not supported with `temporal_cognify=True`** (only the default pipeline is estimated) or while connected to a remote Cognee instance via `serve()` — both raise a `ValueError`.
* **`chunk_attachment` is allowed alongside `dry_run`.** It changes only how the graph is wired at write time, and the estimated stages make no extra LLM calls because of it, so the estimate is the same with or without it. See [Chunk attachment](#chunk-attachment).
* **Unknown models emit a warning** rather than reporting a `$0` cost when no pricing entry exists for the configured model.
* **Code files are counted as skipped, at zero cost.** Items that `cognify()` would route down the code graph pipeline are separated out before any document is read, so they are never chunked and contribute no tokens or cost. They are folded into `skipped_items` and reported in `warnings` as `Skipped N code file(s) because they run the deterministic code graph pipeline — no LLM calls.` See [Loaders](/core-concepts/further-concepts/loaders) for which extensions take that route.
## Processing Pipeline
When you call `cognify()`, data goes through these stages:
1. **Document classification** — identify content type
2. **Text chunking** — split into manageable segments
3. **Entity extraction** — identify entities using the LLM
4. **Relationship detection** — find connections between entities
5. **Graph construction** — build the knowledge graph
6. **Summarization** — generate summaries of content
7. **Provenance recording** *(opt-in, off by default)* — append an audit-ledger entry for every document, chunk, entity, and relationship this run produced. Enabled with `PROVENANCE_TRACKING=true`; when off, the task list is identical to the pipeline above. See [Provenance ledger](#provenance-ledger).
8. **Contradiction detection** *(opt-in, off by default)* — compare the facts this run touched against the facts already stored around them and record each conflict as a `contradicts` edge. Enabled with `CONTRADICTION_DETECTION=true`; when off, the task list is identical to the pipeline above. See [Contradiction detection](#contradiction-detection).
## Provenance ledger
When `PROVENANCE_TRACKING` is enabled, `cognify()` splices one extra task (`record_provenance`) into the default pipeline. It runs immediately after the graph and embeddings have been written — so node ids are persisted and stable — and before the contradiction check, so the ledger never depends on contradiction edges. For each item the run produced it appends document → chunk → entity → relationship lineage entries to the append-only `provenance_entries` table in the relational database, committing all entries for one task invocation as a single chained transaction.
There is **no `cognify()` argument** for this feature; like contradiction detection it is configured entirely through `CognifyConfig`:
```bash theme={null}
PROVENANCE_TRACKING=true # default: false
```
<Note>
The cognify config is cached for the lifetime of the process, so set this in your `.env` or environment **before** the first `cognify()` call. Changing it mid-process has no effect.
</Note>
Behavior notes:
* **Failure is never fatal.** The task returns its input unchanged and swallows all of its own errors, logging a warning (`Provenance recording failed; ingestion unaffected: ...`) instead of failing the run. Missing pipeline context or ids degrade to entries with a `source_ref_key` of `None` rather than raising. A swallowed failure rolls the whole batch back, so those entries are simply absent — they never claimed sequence numbers, the hash chain stays intact, and `verify_chain()` still reports `valid`. Verification proves the stored entries were not tampered with, not that everything an ingestion produced was recorded, so watch that warning if you depend on the ledger being complete.
* **Ledger keys are dataset-scoped.** Cognee entity ids are deterministic and the ledger lives in the shared relational database, so every node-derived ledger id is prefixed with the dataset id (`"{dataset_id}:{raw_id}"`), and relationship ids are built from the prefixed endpoints. Two datasets mentioning the same entity name keep separate version chains.
* **There is one chain, and one writer at a time.** Dataset scoping applies to ledger keys and version chains, not to the hash chain itself: every entry in the table shares a single `sequence_id` sequence, and each commit takes a ledger-wide write lock (a Postgres advisory lock, so it serializes across processes too). Concurrent `cognify()` runs — several datasets, several workers — therefore queue behind each other for their ledger commits, which is why entries are batched one transaction per task invocation.
* **Custom `graph_model` schemas are covered.** DataPoints that do not follow the default `made_from` / `is_part_of` / `contains` shape are walked with the same traversal `add_data_points` uses, so every node and edge is still recorded.
* **The table must exist.** It ships as Alembic revision `b8c1d3e5f7a9`; run migrations (`cognee.run_migrations()`, or `alembic upgrade head`) before enabling the flag on an existing deployment. The migration is idempotent — it is a no-op if the table is already present.
Entries are read back programmatically through `ProvenanceManager`, which exposes `track_entity`, `track_chunk`, `track_relationship`, `get_provenance`, `get_lineage`, `trace_lineage`, `revision_history`, `invalidate`, `verify_chain`, `check`, and `get_statistics`:
```python theme={null}
from cognee.modules.provenance import get_provenance_manager
manager = get_provenance_manager()
report = await manager.verify_chain()
print(report["valid"], report["total_entries"], report["broken_links"])
```
Each entry carries a SHA-256 checksum over its canonical JSON plus the previous entry's checksum, linked by a unique `sequence_id`, so `verify_chain()` detects deletion, reordering, and single-field edits. It streams the ledger in sequence order by keyset pagination rather than materializing it client-side, so it is safe to run against a large table.
## Contradiction detection
When `CONTRADICTION_DETECTION` is enabled, `cognify()` appends one extra task to the end of the default pipeline. It runs after the graph has been written, so both the new facts and the pre-existing ones are persisted and comparable. For each pair of facts the LLM judges to be in conflict, Cognee logs a warning and writes a `contradicts` edge into the graph — nothing is rewritten or deleted.
There is **no `cognify()` argument** for this feature; it is configured entirely through `CognifyConfig`, which reads these environment variables:
```bash theme={null}
CONTRADICTION_DETECTION=true # default: false
CONTRADICTION_CONFIDENCE_THRESHOLD=0.5 # minimum LLM confidence to flag a pair
CONTRADICTION_MAX_FACTS=500 # cap on facts sent to the LLM in one check
```
<Note>
The cognify config is cached for the lifetime of the process, so set these in your `.env` or environment **before** the first `cognify()` call. Changing them mid-process has no effect.
</Note>
Because entity node ids are deterministic (`Entity:<name>`), a re-mentioned entity keeps the id it was first stored under — so a new fact and the stored fact it contradicts share a subject and land in the same neighbourhood:
```python theme={null}
import cognee
await cognee.add("Alice was born in 1985.")
await cognee.cognify()
# Later ingestion, with CONTRADICTION_DETECTION=true
await cognee.add("Alice was born in 1990.")
await cognee.cognify()
```
The second run emits a `WARNING` of the form:
```text theme={null}
Contradiction detected (confidence 0.95): 'alice born in 1985' contradicts 'alice born in 1990' — <reason>
```
followed by an `INFO` line reporting how many contradictions were flagged in the graph.
### The `contradicts` edge
Each flagged pair is written as a single edge with `relationship_name` `"contradicts"` and these properties:
| Property | Type | Description |
| ----------------------------------- | ------- | ---------------------------------------------------------------------------------------------------- |
| `relationship_name` | `str` | Always `"contradicts"`. |
| `source_node_id` / `target_node_id` | `str` | The two nodes the edge links. |
| `first_fact` | `str` | Rendered text of the first conflicting fact, e.g. `"alice born in 1985"`. |
| `second_fact` | `str` | Rendered text of the second conflicting fact. |
| `reason` | `str` | Short LLM explanation of why the two facts are incompatible. |
| `confidence` | `float` | The LLM's confidence, in `[0.0, 1.0]`. Pairs below `CONTRADICTION_CONFIDENCE_THRESHOLD` are dropped. |
`first_fact` and `second_fact` are rendered by Cognee from the graph itself (`<source name> <relationship name> <target name>`, with underscores in the relationship name replaced by spaces), not taken from the model output, so the stored text always matches the graph. Node names are stored normalized (lowercase), so the rendered facts are lowercase too.
The edge connects the two nodes that actually differ: the two subjects when the facts have different subjects, otherwise the two objects. If both facts reference exactly the same pair of nodes, no edge is written.
Since it is an ordinary edge, you can read it back with the graph engine:
```python theme={null}
from cognee.infrastructure.databases.graph import get_graph_engine
graph_engine = await get_graph_engine()
_, edges = await graph_engine.get_graph_data()
for source, target, relationship_name, properties in edges:
if relationship_name == "contradicts":
print(properties["first_fact"], "<>", properties["second_fact"])
print(properties["reason"], properties["confidence"])
```
### Scope, cost, and limits
<Note>
* **Scoped to what you just ingested.** Only the 1-hop neighbourhood of the entities the current run touched is inspected — not the whole graph.
* **Structural edges are ignored.** `contains`, `is_part_of`, `made_from`, `exists_in`, and `contradicts` itself are skipped when building the candidate fact list, as are edges whose endpoints are unnamed (chunks, documents).
* **Fact cap.** At most `CONTRADICTION_MAX_FACTS` facts are compared per check; when the cap is hit, the remainder are skipped and an `INFO` line is logged. Very large neighbourhoods may therefore be only partially compared.
* **Cost.** One additional LLM call per chunk batch, and only when at least two candidate facts were found.
* **Fail-safe and non-destructive.** The task only adds edges, returns its input unchanged, and swallows its own errors (logging a warning), so it can never break ingestion.
</Note>
<Warning>
Contradiction detection needs a graph backend that supports neighbourhood reads. The default provider (and Neo4j, Neptune, the Postgres graph adapter **(demo)**, and Turso) support it; with `GRAPH_DATABASE_PROVIDER=kuzu` the check currently logs a warning and skips silently, so no `contradicts` edges are written.
</Warning>
## Examples
```python theme={null}
import cognee
# Process all datasets
await cognee.cognify()
# Process a specific dataset
await cognee.cognify(datasets=["my_dataset"])
# Process in background
await cognee.cognify(datasets=["large_dataset"], run_in_background=True)
# Use a custom graph model
from pydantic import BaseModel
class MyGraph(BaseModel):
nodes: list
edges: list
await cognee.cognify(graph_model=MyGraph)
# Custom extraction prompt
await cognee.cognify(
custom_prompt="Extract all technical concepts and their relationships."
)
# Pin specific entity and relationship types
custom_prompt = """
Extract only people and cities as entities.
Connect people to cities with the relationship "lives_in".
Ignore all other entities.
"""
await cognee.cognify(custom_prompt=custom_prompt)
```
When `custom_prompt` is set, it fully **replaces** the default graph extraction prompt (see [`GRAPH_PROMPT_PATH`](/core-concepts/main-operations/legacy-operations/cognify#default-extraction-prompts)) for the entity/relationship extraction step, so you can constrain exactly which entity types and relationship labels the LLM produces. For a step-by-step walkthrough, see the [Custom Prompts guide](/guides/custom-prompts).
<Note>
`custom_prompt` is ignored when `temporal_cognify=True`.
</Note>
## Further details
<AccordionGroup>
<Accordion title="Background Execution">
When `run_in_background=True`, `cognify()` starts the processing pipeline as an async background task and **returns immediately**. The return shape is the same as blocking mode — a dict mapping `dataset_id` → `PipelineRunInfo` — but each entry has status `PipelineRunStarted` instead of `PipelineRunCompleted`, and the knowledge graph construction continues in the background.
```python theme={null}
import cognee
# Start processing without waiting for completion
run_info = await cognee.cognify(
datasets=["large_corpus"],
run_in_background=True
)
# run_info is a dict of {dataset_id: PipelineRunInfo}
for dataset_id, info in run_info.items():
print(info.pipeline_run_id) # UUID to track this run
print(info.dataset_id) # Dataset being processed
print(info.status) # Initial status (e.g. "PipelineRunStarted")
```
The returned `PipelineRunInfo` fields relevant for monitoring:
| Field | Type | Description |
| ----------------- | ------ | --------------------------------------- |
| `pipeline_run_id` | `UUID` | Unique identifier for this pipeline run |
| `dataset_id` | `UUID` | The dataset being processed |
| `dataset_name` | `str` | Name of the dataset |
| `status` | `str` | Current status of the run |
Possible status values: `PipelineRunStarted`, `PipelineRunYield`, `PipelineRunCompleted`, `PipelineRunAlreadyCompleted`, `PipelineRunErrored`.
A `PipelineRunErrored` entry additionally carries `error_class` and `error_message` (PII-scrubbed) describing the root cause. These are the fields a `raise_on_error=False` caller reads — see [Failed runs](#failed-runs). `payload` is still a `repr()` string, but it now holds the repr of that same root cause instead of the generic `PipelineRunFailedError` it carried before; unlike `error_message` it is **not** PII-scrubbed, so log `error_class` / `error_message` rather than `payload`.
A sixth status, `PipelineRunProgress`, exists only on the WebSocket channel described below. `cognify()` never returns or yields it, so SDK callers that want in-flight progress should poll `GET /api/v1/datasets/status/progress` instead.
</Accordion>
<Accordion title="Monitoring progress via WebSocket (REST API)">
When using the REST API, subscribe to real-time pipeline updates with the WebSocket endpoint:
```
WebSocket: /cognify/subscribe/{pipeline_run_id}
```
**Authentication**: The handshake accepts exactly the credentials the HTTP API accepts — an API key header (`X-Api-Key`), a bearer `Authorization` header, or the auth cookie. The configured authentication backends are tried in registration order and the first one that resolves an active user wins. When `REQUIRE_AUTHENTICATION` is off, an unauthenticated handshake falls back to the default user, so single-user deployments can connect without credentials just as their HTTP routes do.
Browsers cannot set custom headers when opening a WebSocket, so both the bearer and API key schemes also accept the credential as a `?token=` query parameter. The fallback applies only to WebSocket connections — plain HTTP requests still require the header.
<Warning>
A WebSocket handshake is itself an HTTP request, so a `?token=` query string can end up in access logs. Uvicorn's own access/error logs redact it automatically, but a reverse proxy or load balancer in front of Cognee (nginx, AWS ALB) logs the full request path by default — redact the `token` query parameter there if you terminate WebSocket traffic through one. Prefer a header wherever your client can set one.
</Warning>
**Usage example (JavaScript)**:
```javascript theme={null}
const pipelineRunId = "your-pipeline-run-id-uuid";
// Same-origin browser client with an auth cookie — nothing extra needed:
const ws = new WebSocket(`ws://your-server/cognify/subscribe/${pipelineRunId}`);
// Otherwise pass the bearer token or API key as a query parameter:
// const ws = new WebSocket(
// `ws://your-server/cognify/subscribe/${pipelineRunId}?token=${token}`
// );
ws.onmessage = (event) => {
const data = JSON.parse(event.data);
console.log("Status:", data.status);
console.log("Run ID:", data.pipeline_run_id);
// data.payload contains the current graph data for the dataset
};
ws.onclose = () => {
// Server closes the connection when processing completes (status: PipelineRunCompleted)
console.log("Pipeline run finished");
};
```
Most WebSocket messages have this shape:
```json theme={null}
{
"pipeline_run_id": "uuid-string",
"status": "PipelineRunYield",
"payload": { /* current graph data for the dataset */ }
}
```
**In-flight progress messages.** While a run is executing, the connection also carries `PipelineRunProgress` messages — one each time a processed result exits the run's task chain — so a backgrounded run signals it is alive and moving instead of going quiet until the terminal event:
```json theme={null}
{
"pipeline_run_id": "uuid-string",
"status": "PipelineRunProgress",
"completed_items": null,
"total_items": null,
"current_stage": "add_data_points",
"stage_index": 4,
"stage_total": 4
}
```
| Field | Type | Description |
| ----------------- | ---------------- | ------------------------------------------------------------------------ |
| `current_stage` | `string \| null` | Name of the task most recently entered in the emitting item's task chain |
| `stage_index` | `int \| null` | 1-based position of that task in the chain |
| `stage_total` | `int \| null` | Total number of tasks in the chain |
| `completed_items` | `null` | Always `null` on this channel — see below |
| `total_items` | `null` | Always `null` on this channel — see below |
Read the stage fields for what they are, not as a stage-by-stage tracker: tasks stream results through the chain, so a message only fires once a result has passed through **every** stage. By then the whole chain has been entered, which means `current_stage` in practice always names the chain's *final* task (`add_data_points` on the default cognify pipeline) and `stage_index` equals `stage_total`. Treat these messages as a liveness heartbeat that also tells you the chain's length, and get progress numbers from the polling endpoint below.
Two more things to handle in a client:
* Progress messages carry **no `payload` key**. Ticks are frequent, and the graph snapshot that the other statuses attach is deliberately not recomputed for each one. Read `data.payload` only when `data.status` is not `PipelineRunProgress`.
* `completed_items` and `total_items` arrive as **`null`** here. The N-of-M file counts are the polling endpoint's signal. For a "3 of 10 files" bar, read `GET /api/v1/datasets/status/progress`, which returns `{status, progress}` per dataset with `completed_items`, `total_items`, and `current_stage`.
The server closes the WebSocket with one of these codes:
| Code | Meaning |
| ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `1000` | The run reached `PipelineRunCompleted` and its final payload was sent |
| `1008` | Rejected: not authenticated, `pipeline_run_id` is not a valid UUID, no such run, or no read permission on the run's dataset. Also sent mid-stream if the dataset is deleted or read access is revoked while streaming |
| `1011` | The server hit an internal failure while streaming the run |
Each `1008` close carries a `reason` naming which of those it was. A retry replays the same rejection, so clients should stop rather than reconnect.
<Note>
The run's update queue is consumed, not observed: there is one subscriber per run. A second client subscribing to the same `pipeline_run_id` steals events from the first. Authorization is checked before the queue is touched at all, so a rejected caller cannot disturb the real subscriber.
</Note>
</Accordion>
<Accordion title="When to use background mode">
* **Large datasets** (>100 MB) where blocking would time out HTTP connections
* **API integrations** where you want to return a job ID to the caller immediately
* **Parallel processing** of multiple datasets without waiting for each
For small datasets or scripts, the default blocking mode (`run_in_background=False`) is simpler and returns the final result directly.
</Accordion>
</AccordionGroup>
# config
Source: https://docs.cognee.ai/python-api/config
Configure LLM providers, databases, chunking, and more
# cognee.config
Static class for configuring Cognee's runtime settings. All setters persist for the duration of the process (or until overridden). Use `cognee.config.set(...)` for supported runtime-safe settings inside a Python process.
Cognee can also be configured through `.env` or process environment variables before import. Use those for process-level settings such as auth, logging, cache backend, storage backend, telemetry, API server settings, and deployment credentials.
For the full environment variable reference and precedence rules, see [Setup Configuration](/setup-configuration/overview).
## Configuration Types
<AccordionGroup>
<Accordion title="LLM Configuration">
```python theme={null}
cognee.config.set_llm_provider("openai") # "openai", "anthropic", "ollama", "gemini", "mistral", "bedrock"
cognee.config.set_llm_model("gpt-4o-mini")
cognee.config.set_llm_api_key("sk-...")
cognee.config.set_llm_endpoint("https://custom-endpoint.example.com")
# Or set all at once — keys must match LLMConfig attribute names exactly
cognee.config.set_llm_config({
"llm_provider": "openai",
"llm_model": "gpt-4o",
"llm_api_key": "sk-...",
})
```
`set_llm_config()` keys must match the internal attribute names on `LLMConfig`. The exact internal attributes name are displayed in the table below.
<Accordion title="Internal LLM Configuration attributes">
| Key | Type | Default | Description |
| ----------------------------- | ------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `structured_output_framework` | `str` | `"litellm_native"` | Structured output backend: `"litellm_native"`, `"instructor"`, or `"baml"`. See [Structured Output Backends](/setup-configuration/structured-output-backends) |
| `llm_instructor_mode` | `str` | `""` | Instructor `Mode` string (e.g. `"json_schema_mode"`, `"json_mode"`, `"tool_call"`). Empty = provider default |
| `llm_provider` | `str` | `"openai"` | Provider: `"openai"`, `"anthropic"`, `"ollama"`, `"gemini"`, `"mistral"`, `"bedrock"`, `"azure"`, `"custom"` |
| `llm_model` | `str` | `"openai/gpt-5-mini"` | Model identifier |
| `llm_api_key` | `str` | `None` | API key for the provider |
| `llm_endpoint` | `str` | `""` | Custom endpoint URL (required for Ollama, vLLM, etc.) |
| `llm_api_version` | `str` | `None` | API version (required for Azure) |
| `llm_temperature` | `float` | unset | Response temperature (0.0–2.0). Takes effect via the `LLM_TEMPERATURE` env var **when you set it explicitly** — leave it unset and nothing is sent, so the provider's default applies rather than `0.0`. Local inference servers (`ollama`, `llama_cpp`, and LM Studio models, identified by an `lm_studio/` prefix) are the exception: there an unset value is folded in as `0.0`. Setting it through `set_llm_config()` has no effect; at runtime use `llm_args` instead. gpt-5 models (including the default `openai/gpt-5-mini`) reject any temperature but their own default |
| `llm_seed` | `int` | `None` | Sampling seed for reproducible outputs. Takes effect via the `LLM_SEED` env var; setting it through `set_llm_config()` has no effect — at runtime use `llm_args` instead. Provider support varies |
| `llm_streaming` | `bool` | `False` | Legacy flag, and **not** the one that streams recall's answer. It reaches only the OpenAI, Azure, and Bedrock adapters on the `instructor` path, where nothing consumes a token stream — and because it is part of the adapter cache key, flipping it rebuilds the adapter. For a streamed answer use `llm_answer_streaming` |
| `llm_answer_streaming` | `bool` | `False` | Stream the answer tokens of recall's final answer call to a client reading the [REST API's SSE response](/guides/deploy-rest-api-server#http-api-examples) (env `LLM_ANSWER_STREAMING`). Inert unless a client is reading that stream — the returned value is identical either way. Supported on the default `litellm_native` backend; on `instructor` only by the OpenAI-compatible adapters (`openai`, `custom`, and Azure without managed identity), and never by `baml` |
| `llm_max_completion_tokens` | `int` | `16384` | Maximum tokens in the response |
| `llm_args` | `dict` | `{}` | Arbitrary provider-specific kwargs merged into every LLM call. Set as a JSON string in `.env` (e.g. `LLM_ARGS='{"top_p": 0.9}'`). The `LLM_TEMPERATURE` and `LLM_SEED` env vars are folded in here at startup, and a `temperature`/`seed` key set directly in `llm_args` takes precedence over them — this is also the key to use for temperature/seed with `set_llm_config()` |
| `llm_rate_limit_enabled` | `bool` | `False` | Enable client-side rate limiting for LLM calls |
| `llm_rate_limit_requests` | `int` | `60` | Max LLM requests allowed per interval |
| `llm_rate_limit_interval` | `int` | `60` | Duration of the rate limit window in seconds |
| `llm_rate_limit_tokens` | `int` | `0` | Max tokens per interval (`0` = disabled) |
</Accordion>
</Accordion>
<Accordion title="Embedding Configuration">
```python theme={null}
cognee.config.set_embedding_provider("fastembed")
cognee.config.set_embedding_model("BAAI/bge-small-en-v1.5")
cognee.config.set_embedding_dimensions(384)
# Or set all at once — keys must match EmbeddingConfig attribute names exactly
cognee.config.set_embedding_config({
"embedding_provider": "fastembed",
"embedding_model": "BAAI/bge-small-en-v1.5",
"embedding_dimensions": 384,
})
```
<Accordion title="Internal Embedding Configuration attributes">
| Key | Type | Default | Description |
| --------------------------------- | ------ | --------------------------------- | ------------------------------------------------------------------------------------------------- |
| `embedding_provider` | `str` | `"openai"` | Provider: `"openai"`, `"ollama"`, `"fastembed"`, `"gemini"`, `"mistral"`, `"bedrock"`, `"custom"` |
| `embedding_model` | `str` | `"openai/text-embedding-3-large"` | Model identifier |
| `embedding_dimensions` | `int` | `3072` | Vector dimension size (must match your vector store) |
| `embedding_api_key` | `str` | `None` | API key (falls back to `LLM_API_KEY` if unset) |
| `embedding_endpoint` | `str` | `None` | Custom endpoint URL |
| `embedding_api_version` | `str` | `None` | API version |
| `embedding_max_completion_tokens` | `int` | `8191` | Maximum tokens for embedding input |
| `embedding_batch_size` | `int` | `36` | Batch size for embedding requests |
| `huggingface_tokenizer` | `str` | `None` | HuggingFace Hub model ID for token counting with Ollama |
| `embedding_rate_limit_enabled` | `bool` | `False` | Enable client-side rate limiting for embedding calls |
| `embedding_rate_limit_requests` | `int` | `60` | Max embedding requests allowed per interval |
| `embedding_rate_limit_interval` | `int` | `60` | Duration of the rate limit window in seconds |
| `embedding_rate_limit_tokens` | `int` | `0` | Max tokens per interval (`0` = disabled) |
</Accordion>
</Accordion>
<Accordion title="Graph Database Configuration">
```python theme={null}
cognee.config.set_graph_database_provider("kuzu") # "kuzu", "neo4j", "kuzu-remote", "neptune", "neptune_analytics"
# Keys must match GraphConfig attribute names exactly
cognee.config.set_graph_db_config({
"graph_database_provider": "neo4j",
"graph_database_url": "bolt://localhost:7687",
"graph_database_username": "neo4j",
"graph_database_password": "password",
})
```
</Accordion>
<Accordion title="Vector Database Configuration">
```python theme={null}
cognee.config.set_vector_db_provider("lancedb") # "lancedb", "pgvector", "qdrant", "chromadb"
cognee.config.set_vector_db_url("http://localhost:6333")
cognee.config.set_vector_db_key("your-key")
# Keys must match VectorConfig attribute names exactly
cognee.config.set_vector_db_config({
"vector_db_provider": "qdrant",
"vector_db_url": "http://localhost:6333",
"vector_db_key": "...",
})
```
</Accordion>
<Accordion title="Chunking Configuration">
```python theme={null}
cognee.config.set_chunk_size(1024)
cognee.config.set_chunk_overlap(128)
cognee.config.set_chunk_strategy("PARAGRAPH") # "EXACT", "PARAGRAPH", "SENTENCE", "CODE"
cognee.config.set_chunk_engine("DEFAULT_ENGINE") # "DEFAULT_ENGINE", "LANGCHAIN_ENGINE"
```
</Accordion>
<Accordion title="Model Configuration">
```python theme={null}
cognee.config.set_classification_model(MyClassifier)
cognee.config.set_summarization_model(MySummarizer)
cognee.config.set_graph_model(MyGraphModel)
```
</Accordion>
<Accordion title="Other Settings">
```python theme={null}
cognee.config.system_root_directory("/custom/path")
cognee.config.data_root_directory("/data/path")
cognee.config.set_translation_provider("google") # "llm", "google", "azure"
cognee.config.set_translation_target_language("en")
# Generic setter for supported keys
cognee.config.set("llm_model", "openai/gpt-5-mini")
# Generic getters for the same keys
cognee.config.get("llm_model") # "openai/gpt-5-mini"
cognee.config.get_all() # dict of every known setting
```
### Reading configuration
`get(key)` returns the current value of any key `set()` accepts, and `get_all()` returns
all of them as a dict. Both mask secret values (`llm_api_key`, `embedding_api_key`,
`vector_db_key`) by default, keeping only the first three and last four characters as a
hint (values of eight characters or fewer are masked entirely); pass `reveal_secrets=True`
to get the raw value. `get()` raises
`InvalidConfigAttributeError` for an unrecognized key.
```python theme={null}
cognee.config.get("llm_api_key") # "sk-...c3d4"
cognee.config.get("llm_api_key", reveal_secrets=True) # "sk-proj-9f2a7b41c3d4"
cognee.config.get_all(reveal_secrets=True) # unmasked dict
```
### Persisting a value to `.env`
`set()` normally only updates the in-process configuration. Pass `persist=True` to also
write the resolved value into a `.env` file in the current working directory (created if
missing), so it is picked up by the next process started from that directory. This is
what `cognee-cli config set` does. When persisting, `set()` returns a dict describing
where the value was written:
```python theme={null}
cognee.config.set("llm_model", "openai/gpt-5-mini", persist=True)
# {"path": "/home/you/project/.env", "created": True, "env_var": "LLM_MODEL"}
```
Without `persist=True`, `set()` returns `None` and changes nothing on disk.
<Warning>
Persisting a secret writes it in plaintext to `.env` in whatever directory the process is
running from. Keep that file out of version control and restrict its permissions.
</Warning>
<Warning>
`cognee.config.set(key, value)` is not a free-form setter. Use it for supported runtime-safe settings such as LLMs, embeddings, graph/vector databases, chunking, model overrides, and root directories. Use `.env`, shell variables, deployment variables, or pre-import `os.environ` for process-level settings such as auth, logging, cache backend, storage backend, API server settings, telemetry, and cloud credentials.
</Warning>
</Accordion>
</AccordionGroup>
## All Configuration Methods
<Accordion title="Configuration Methods">
| Method | Description |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| `set_llm_provider(provider)` | Set the LLM provider |
| `set_llm_model(model)` | Set the LLM model name |
| `set_llm_api_key(key)` | Set the LLM API key |
| `set_llm_endpoint(url)` | Set a custom LLM endpoint |
| `set_llm_config(dict)` | Set all LLM config at once |
| `set_embedding_provider(provider)` | Set the embedding provider |
| `set_embedding_model(model)` | Set the embedding model name |
| `set_embedding_dimensions(dimensions)` | Set embedding vector dimensions |
| `set_embedding_endpoint(url)` | Set a custom embedding endpoint |
| `set_embedding_api_key(key)` | Set the embedding API key |
| `set_embedding_config(dict)` | Set all embedding config at once |
| `set_graph_database_provider(provider)` | Set the graph DB provider |
| `set_relational_db_config(dict)` | Set relational DB config |
| `set_migration_db_config(dict)` | Set migration DB config |
| `set_graph_db_config(dict)` | Set all graph DB config |
| `set_vector_db_provider(provider)` | Set the vector DB provider |
| `set_vector_db_url(url)` | Set the vector DB URL |
| `set_vector_db_key(key)` | Set the vector DB API key |
| `set_vector_db_config(dict)` | Set all vector DB config |
| `set_chunk_size(size)` | Set chunk size in tokens |
| `set_chunk_overlap(overlap)` | Set chunk overlap |
| `set_chunk_strategy(strategy)` | Set chunking strategy |
| `set_chunk_engine(engine)` | Set chunking engine |
| `set_classification_model(model)` | Set classification model |
| `set_summarization_model(model)` | Set summarization model |
| `set_graph_model(model)` | Set graph extraction model |
| `system_root_directory(path)` | Set system root directory |
| `data_root_directory(path)` | Set data root directory |
| `monitoring_tool(tool)` | Set the monitoring tool |
| `set_translation_provider(provider)` | Set translation provider |
| `set_translation_target_language(lang)` | Set translation target language |
| `set_translation_config(dict)` | Set translation config |
| `set(key, value, persist=False)` | Generic config setter. With `persist=True`, also writes the value to `.env` in the current directory and returns `{"path", "created", "env_var"}` |
| `get(key, reveal_secrets=False)` | Generic config getter. Secret values are masked unless `reveal_secrets=True` |
| `get_all(reveal_secrets=False)` | Return every known setting as a dict. Secret values are masked unless `reveal_secrets=True` |
</Accordion>
# run_custom_pipeline()
Source: https://docs.cognee.ai/python-api/custom-pipeline
Run your own sequence of tasks as a Cognee pipeline with run_custom_pipeline().
# cognee.run\_custom\_pipeline()
```python theme={null}
async def run_custom_pipeline(
tasks: Union[List[Task], List[str]] = None,
data: Any = None,
dataset: Union[str, UUID] = 'main_dataset',
user: User = None,
vector_db_config: Optional[dict] = None,
graph_db_config: Optional[dict] = None,
use_pipeline_cache: bool = False,
incremental_loading: bool = False,
data_per_batch: int = 20,
run_in_background: bool = False,
pipeline_name: str = 'custom_pipeline',
)
```
## Description
Custom pipeline in Cognee, can work with already built graphs. Data needs to be provided which can be processed
with provided tasks.
Provided tasks and data will be arranged to run the Cognee pipeline and execute graph enrichment/creation.
This is the core processing step in Cognee that converts raw text and documents
into an intelligent knowledge graph. It analyzes content, extracts entities and
relationships, and creates semantic connections for enhanced search and reasoning.
Args:
tasks: List of Cognee Tasks to execute.
data: The data to ingest. Can be anything when custom extraction and enrichment tasks are used.
Data provided here will be forwarded to the first extraction task in the pipeline as input.
dataset: Dataset name or dataset uuid to process.
user: User context for authentication and data access. Uses default if None.
vector\_db\_config: Custom vector database configuration for embeddings storage.
graph\_db\_config: Custom graph database configuration for relationship storage.
use\_pipeline\_cache: If True, pipelines with the same ID that are currently executing and pipelines with the same ID that were completed won't process data again.
Pipelines ID is created based on the generate\_pipeline\_id function. Pipeline status can be manually reset with the reset\_dataset\_pipeline\_run\_status function.
incremental\_loading: If True, only new or modified data will be processed to avoid duplication. (Only works if data is used with the Cognee python Data model).
The incremental system stores and compares hashes of processed data in the Data model and skips data with the same content hash.
data\_per\_batch: Number of data items to be processed in parallel.
run\_in\_background: If True, starts processing asynchronously and returns immediately.
If False, waits for completion before returning.
Background mode recommended for large datasets (>100MB).
Use pipeline\_run\_id from return value to monitor progress.
## Parameters
<ParamField type="Union[List[Task], List[str]]">List of Task objects or task names defining the pipeline steps.</ParamField>
<ParamField type="Any">Input data for the pipeline.</ParamField>
<ParamField type="Union[str, UUID]">Dataset name or UUID.</ParamField>
<ParamField type="User">User performing the operation.</ParamField>
<ParamField type="Optional[dict]">Override vector database configuration.</ParamField>
<ParamField type="Optional[dict]">Override graph database configuration.</ParamField>
<ParamField type="bool">Cache intermediate pipeline results.</ParamField>
<ParamField type="bool">Skip already-processed data.</ParamField>
<ParamField type="int">Number of data items per batch.</ParamField>
<ParamField type="bool">If true, return immediately and process in background.</ParamField>
<ParamField type="str">Name identifier for the pipeline run.</ParamField>
## Examples
```python theme={null}
import cognee
from cognee.modules.pipelines import Task
# Define custom tasks
async def my_extractor(data):
# Custom extraction logic
yield extracted_data
async def my_enricher(data):
# Custom enrichment logic
yield enriched_data
# Run a custom pipeline
await cognee.run_custom_pipeline(
tasks=[
Task(my_extractor),
Task(my_enricher),
],
data="Input data for the pipeline",
dataset="my_dataset",
pipeline_name="my_custom_pipeline",
)
```
See [Custom Tasks & Pipelines](/guides/custom-tasks-pipelines) for a full guide.
# Data Models
Source: https://docs.cognee.ai/python-api/data-models
Pydantic models and types accepted or returned by the cognee Python API.
# Data Models
Key Pydantic models and types used in the cognee Python API.
## SearchResult
Returned by `cognee.search()`.
```python theme={null}
class SearchResult:
search_result: Any # The actual result content
dataset_id: UUID | None # Associated dataset UUID
dataset_name: str | None # Associated dataset name
```
## PipelineRunInfo
Returned by `cognee.add()`, `cognee.cognify()`, and pipeline functions.
```python theme={null}
class PipelineRunInfo:
status: str # Pipeline status
pipeline_run_id: UUID # Unique run identifier
dataset_id: UUID # Associated dataset
dataset_name: str # Dataset name
payload: Any | None # Execution payload
data_ingestion_info: list | None # Ingestion details
```
**Status values:** `PipelineRunStarted`, `PipelineRunYield`, `PipelineRunCompleted`, `PipelineRunAlreadyCompleted`, `PipelineRunErrored`
A `PipelineRunErrored` run info additionally carries `error_class: str | None` and `error_message: str | None` (PII-scrubbed) naming the root cause of the failure. See [Failed runs](/python-api/cognify#failed-runs) for how to read them.
## Task
Wraps a callable for use in pipelines.
```python theme={null}
from cognee.modules.pipelines import Task
# Wrap any async generator, generator, coroutine, or function
task = Task(my_function, task_config={"batch_size": 10})
```
<ParamField type="Callable">The function to execute. Can be an async generator, generator, coroutine, or regular function.</ParamField>
<ParamField type="dict">Task configuration, primarily batch size.</ParamField>
## DataPoint
The **public base class** for all user-defined graph entities. Extend `DataPoint` to create custom node types that Cognee can index, search, and connect in the knowledge graph.
```python theme={null}
class DataPoint:
id: UUID # Unique identifier
created_at: int # Creation timestamp (ms)
updated_at: int # Update timestamp (ms)
version: int # Version number
type: str # Data type name
ontology_valid: bool # Ontology validation status
topological_rank: int | None # Topological rank
belongs_to_set: list[str] | None # Node set membership
source_pipeline: str | None # Source pipeline name
source_node_set: str | None # Source node set
```
**Key methods:** `to_json()`, `from_json()`, `to_dict()`, `from_dict()`, `update_version()`
See [DataPoints](/core-concepts/building-blocks/datapoints) and [Custom Data Models](/guides/custom-data-models) for usage details.
## KnowledgeGraph
Default graph model used by `cognify()` as an **internal LLM extraction format**. The LLM populates this structure while processing documents; it is not exported from the top-level `cognee` package and is not intended for user extension.
```python theme={null}
class KnowledgeGraph(BaseModel):
nodes: list[Node] # Graph nodes
edges: list[Edge] # Graph edges
summary: str # Graph summary
description: str # Graph description
```
<Note>
`Node` and the `Edge` type nested inside `KnowledgeGraph` are used as internal pipeline types during extraction, rather than as user-facing extension points. For custom entities and application models, use [`DataPoint`](#datapoint) subclasses instead.
</Note>
### Node (internal)
```python theme={null}
class Node:
id: str
name: str
type: str
description: str
```
### Edge (internal)
```python theme={null}
class Edge:
source_node_id: str
target_node_id: str
relationship_name: str
```
## Exceptions
All cognee exceptions inherit from `CogneeApiError`:
| Exception | Status Code | Use |
| -------------------------- | ----------- | -------------------------- |
| `CogneeSystemError` | 500 | Internal system errors |
| `CogneeValidationError` | 422 | Invalid input/parameters |
| `CogneeConfigurationError` | 500 | Misconfiguration |
| `CogneeTransientError` | 503 | Temporary failures (retry) |
# cognee.datasets
Source: https://docs.cognee.ai/python-api/datasets
Dataset management: list, create, fetch, and delete datasets
# cognee.datasets
Static class for managing datasets and their data. Methods that target a specific dataset identify it by `UUID`, never by name — see [dataset name vs dataset id](/core-concepts/further-concepts/datasets#dataset-name-vs-dataset-id) for how the two relate and which identifier each operation accepts.
For lower-level helpers such as `get_dataset()`, `get_datasets_by_name()`, and `create_authorized_dataset()`, see [Dataset helper methods](#dataset-helper-methods) below.
## Methods
### datasets.list\_datasets()
```python theme={null}
await cognee.datasets.list_datasets(user=None)
```
Returns all datasets accessible to the resolved user.
| Parameter | Type | Default | Notes |
| --------- | ---------------- | ------- | --------------------------------------------- |
| `user` | `Optional[User]` | `None` | If omitted, Cognee resolves the default user. |
### datasets.discover\_datasets()
```python theme={null}
cognee.datasets.discover_datasets(directory_path: str)
```
Discover dataset names from a local directory layout.
| Parameter | Type | Default | Notes |
| ---------------- | ----- | -------- | --------------------------------------------------------- |
| `directory_path` | `str` | required | Local directory to scan for dataset-style subdirectories. |
### datasets.list\_data()
```python theme={null}
await cognee.datasets.list_data(dataset_id, user=None)
```
Returns all `Data` records in a dataset.
This is the API to use when you want to read back `DataItem` fields stored during `cognee.add()`, such as `label` and `external_metadata`.
| Parameter | Type | Default | Notes |
| ------------ | ---------------- | -------- | ----------------------------------------------------------------------------------------------------------- |
| `dataset_id` | `UUID` | required | Dataset UUID to inspect. |
| `user` | `Optional[User]` | `None` | If omitted, Cognee resolves the default user before permission checks. Not used in remote mode — see below. |
#### Remote mode (`serve()`)
While a [`serve()`](/python-api/serve) connection is active, `list_data()` reads the remote dataset over `GET /api/v1/datasets/{dataset_id}/data` instead of the local store, so it works for datasets that exist only on the remote instance. The `user` argument is ignored on that path: the API key used by `serve()` identifies the caller, and the server runs its own permission check.
Remote rows are parsed back through `DataDTO`, the same wire model the server serializes them with, so the fields it returns carry the same attribute names and types as their local counterparts — `row.id` is a `UUID` (not a string), and the field is `row.mime_type` (not camelCase `mimeType`). `label` and `external_metadata` survive the round trip, so the `DataItem` fields stored during `cognee.add()` are readable remotely too.
A remote row exposes exactly these fields: `id`, `name`, `created_at`, `updated_at`, `extension`, `mime_type`, `raw_data_location`, `dataset_id`, `label`, and `external_metadata`. A local `Data` record carries more — `content_hash`, `owner_id`, `tenant_id`, `loader_engine`, `system_metadata`, and others — so code that reads those attributes works locally and raises `AttributeError` over a connection.
<Warning>
`list_datasets()` is **not** routed to the remote instance — it still reads the local store. The common pattern of listing datasets and then reading one's data breaks over a `serve()` connection, because the dataset ids come from the local store and the data lookup goes to the remote one. Get remote dataset ids from the instance itself (`GET /api/v1/datasets`) or from the Cognee Cloud UI, and pass them to `list_data()` directly.
</Warning>
### datasets.has\_data()
```python theme={null}
await cognee.datasets.has_data(dataset_id, user=None) -> bool
```
Check whether a dataset contains any data.
| Parameter | Type | Default | Notes |
| ------------ | ---------------- | -------- | ---------------------------------------------------------------------- |
| `dataset_id` | `str` | required | Dataset identifier to check. |
| `user` | `Optional[User]` | `None` | If omitted, Cognee resolves the default user before permission checks. |
### datasets.get\_status()
```python theme={null}
await cognee.datasets.get_status(
dataset_ids: list[UUID],
pipeline_names: list[str] | None = None,
) -> dict
```
Get pipeline status for one or more datasets.
When `pipeline_names` is omitted, this method keeps the legacy flat shape and returns the status of `cognify_pipeline` only.
| Parameter | Type | Default | Notes |
| ---------------- | --------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `dataset_ids` | `list[UUID]` | required | Dataset UUIDs to check. |
| `pipeline_names` | `Optional[list[str]]` | `None` | Pipeline names to query. If omitted, defaults to `cognify_pipeline`. Duplicate names are deduplicated while preserving order. |
With no `pipeline_names` or a single pipeline name, the method returns `{str(dataset_id): PipelineRunStatus}`.
With multiple pipeline names, it returns `{str(dataset_id): {pipeline_name: PipelineRunStatus}}`.
Possible values:
| Value | Meaning |
| ------------------------------ | ----------------------------------- |
| `DATASET_PROCESSING_INITIATED` | Pipeline queued but not yet started |
| `DATASET_PROCESSING_STARTED` | Pipeline is running |
| `DATASET_PROCESSING_COMPLETED` | Indexing finished successfully |
| `DATASET_PROCESSING_ERRORED` | Processing failed |
Datasets with no recorded run for the requested pipeline are absent from the result.
```python theme={null}
status = await cognee.datasets.get_status([dataset.id])
# {"<dataset-uuid>": "DATASET_PROCESSING_COMPLETED"}
```
<AccordionGroup>
<Accordion title="Troubleshooting UUID errors">
`get_status()` expects `dataset_ids` to be a list of dataset **UUIDs**, not dataset names or string ids. Internally the values are bound against the `pipeline_runs.dataset_id` UUID column, so passing a plain string raises a SQLAlchemy `StatementError` wrapping one of:
* `AttributeError: 'str' object has no attribute 'hex'`
* `ValueError: badly formed hexadecimal UUID string`
```python theme={null}
# ❌ Wrong — passing a dataset name (or string id)
await cognee.datasets.get_status(["my_dataset"])
# ✅ Right — resolve the name to its UUID first
datasets = await cognee.datasets.list_datasets()
dataset_id = next(ds.id for ds in datasets if ds.name == "my_dataset")
status = await cognee.datasets.get_status([dataset_id])
```
If you already hold a string id (for example one read back from the HTTP API), wrap it in `UUID` before calling:
```python theme={null}
from uuid import UUID
status = await cognee.datasets.get_status([UUID(dataset_id_str)])
```
</Accordion>
</AccordionGroup>
### datasets.empty\_dataset()
```python theme={null}
await cognee.datasets.empty_dataset(dataset_id, user=None)
```
Delete all data in a dataset and remove the dataset itself.
| Parameter | Type | Default | Notes |
| ------------ | ---------------- | -------- | ---------------------------------------------------------------------------- |
| `dataset_id` | `UUID` | required | Dataset UUID to empty. |
| `user` | `Optional[User]` | `None` | If omitted, Cognee resolves the default user and checks `delete` permission. |
<AccordionGroup>
<Accordion title="Notes">
<Note>
Despite the name, `empty_dataset()` does not leave an empty dataset record behind. It deletes graph content, data records, and the dataset entity itself.
</Note>
</Accordion>
</AccordionGroup>
### datasets.delete\_data()
```python theme={null}
await cognee.datasets.delete_data(
dataset_id,
data_id,
user=None,
mode="soft",
delete_dataset_if_empty=False,
)
```
Delete a specific data item from a dataset.
| Parameter | Type | Default | Notes |
| ------------------------- | ---------------- | -------- | -------------------------------------------------------------------------------------- |
| `dataset_id` | `UUID` | required | Dataset UUID containing the target data item. |
| `data_id` | `UUID` | required | Data item UUID to delete. |
| `user` | `Optional[User]` | `None` | If omitted, Cognee resolves the default user and checks `delete` permission. |
| `mode` | `str` | `soft` | Kept for backward compatibility. The implementation warns against using `"hard"`. |
| `delete_dataset_if_empty` | `bool` | `False` | If `True`, deletes the dataset when the removed item was its last remaining data item. |
<AccordionGroup>
<Accordion title="Notes">
<Warning>
`mode="hard"` is preserved for backward compatibility, but the implementation explicitly warns not to use it.
</Warning>
</Accordion>
<Accordion title="What delete_data() removes across stores">
`delete_data()` is not a relational-only operation. It cleans up every backend that holds memory derived from the targeted data item:
| Store | What is removed |
| ------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Graph](/setup-configuration/graph-stores) (Kuzu, Neo4j, Neptune, FalkorDB, …) | The nodes and edges Cognee recorded as derived from that data item. |
| [Vector](/setup-configuration/vector-stores) (LanceDB, Qdrant, PGVector, …) | The matching points in every `{NodeType}_{indexed_field}` collection (for example `Entity_name`, `DocumentChunk_text`), plus the `EdgeType_relationship_name` and `Triplet_text` entries belonging to the removed edges. |
| [Relational](/setup-configuration/relational-databases) (SQLite, Postgres) | The per-document node/edge ownership records and the `Data` row itself. A `Data` row belongs to exactly one dataset, so there is no membership to unlink and nothing to keep alive on behalf of another dataset — the identical content in another dataset is a separate row that is left untouched. |
| File storage (local disk, S3) | The raw file — but only when this was the last `Data` row (in any dataset) pointing at that location **and** the file lives under `DATA_ROOT_DIRECTORY` (that is, Cognee copied it in). Files referenced from outside that directory are left in place. |
| [Session cache](/core-concepts/sessions-and-caching) (SQL, Redis, filesystem) | Only the contaminated entries, across both the sessions attributed to this dataset and the sessions carrying no dataset attribution (the global `default_session` an unscoped or cross-dataset recall runs in): turns whose recorded graph elements overlap the nodes and edges just deleted, plus what they propagated to — feedback referencing such a turn, the distilled session-context lesson it fed, and later turns that consumed that lesson. Untouched turns, and the sessions themselves, are kept. |
Session cleanup is **best-effort**: a cache failure is logged as a warning and never fails `delete_data()`, so a `{"status": "success"}` result does not guarantee every entry was removed. It is also the fine-grained variant — `empty_dataset()` deletes every session attributed to the dataset outright (and only then applies this same targeted pass to unattributed sessions), and [`forget(everything=True)`](/core-concepts/main-operations/forget) prunes the cache wholesale.
What is intentionally left behind:
* **Shared nodes.** An entity such as `"New York"` that another data item also references stays in the graph and vector store; only nodes unique to the deleted item are dropped. Relationships between surviving shared nodes are not deleted either.
* **The dataset.** The dataset record survives unless you pass `delete_dataset_if_empty=True` and the removed item was the last one.
See [Delete](/core-concepts/main-operations/legacy-operations/delete) for the same flow in narrative form, and [`forget()`](/core-concepts/main-operations/forget) for a scope matrix across all deletion modes.
</Accordion>
<Accordion title="Cascading changes when a source document changes">
`delete_data()` only removes; it never re-extracts. To propagate an edit to a source document through the graph and vector stores, use [`update()`](/python-api/update), which by default diffs the new content against the stored text and replaces only the chunks the edit touched — falling back to calling `delete_data()` for the old item, re-adding the new content, and re-running `cognify` on the dataset when the [chunk-level preconditions](/python-api/update#how-it-works) are not met:
```python theme={null}
await cognee.update(
data_id=item.id,
data="Updated document content.",
dataset_id=ds.id,
)
```
Relationships that existed only in the old version disappear with the deleted nodes; relationships found in the new content are created by the cognify step. `incremental_loading=True` (the default) keeps the other, unchanged documents in the dataset from being reprocessed — pass `incremental_loading=False` only when the whole dataset should be rebuilt, for example after changing your graph model or prompts.
To clear a document's derived memory while keeping its record and raw file, use [`forget(..., memory_only=True)`](/core-concepts/main-operations/forget) and re-run `cognify`.
</Accordion>
</AccordionGroup>
### datasets.delete\_all()
```python theme={null}
await cognee.datasets.delete_all(user=None)
```
Delete all datasets the user has permission to delete.
| Parameter | Type | Default | Notes |
| --------- | ---------------- | ------- | --------------------------------------------- |
| `user` | `Optional[User]` | `None` | If omitted, Cognee resolves the default user. |
## Dataset helper methods
`cognee.datasets` covers the common cases. Underneath it, Cognee exports a set of dataset helpers importable from `cognee.modules.data.methods`. Use them when you need to resolve a dataset by name or id, create one explicitly, or apply a permission type other than `read`. All of them are async except `check_dataset_name()`.
### Fetching datasets
Ownership-scoped lookups (they match on `Dataset.owner_id` only, ignoring [ACLs](/core-concepts/multi-user-mode/permissions-system/acl)):
| Method | Signature | Returns |
| ------------------------ | ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| `get_dataset()` | `get_dataset(user_id: UUID, dataset_id: UUID)` | The `Dataset`, or `None` if it does not exist or is not owned by `user_id`. |
| `get_datasets()` | `get_datasets(user_id: UUID)` | All `Dataset` rows owned by `user_id`. |
| `get_datasets_by_name()` | `get_datasets_by_name(dataset_names: str \| list[str], user_id: UUID)` | Datasets owned by `user_id` whose name is in `dataset_names`. A single string is treated as a one-item list. |
| `get_dataset_data()` | `get_dataset_data(dataset_id: UUID)` | The `Data` records in a dataset, largest first. Backs [`datasets.list_data()`](#datasets-list_data). |
| `has_dataset_data()` | `has_dataset_data(dataset_id: UUID)` | `True` if the dataset has at least one data record. Backs [`datasets.has_data()`](#datasets-has_data). |
Permission-aware lookups (they take a `User` object and go through the [permissions system](/core-concepts/multi-user-mode/permissions-system/overview), so they also return datasets shared with the user):
| Method | Signature | Returns |
| ------------------------------------ | --------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `get_authorized_dataset()` | `get_authorized_dataset(user: User, dataset_id: UUID, permission_type: str = "read")` | The `Dataset` if the user holds `permission_type` on it, otherwise `None`. |
| `get_authorized_dataset_by_name()` | `get_authorized_dataset_by_name(dataset_name: str, user: User, permission_type: str)` | The first authorized dataset with that name, otherwise `None`. |
| `get_authorized_existing_datasets()` | `get_authorized_existing_datasets(datasets: list[str] \| list[UUID] \| None, permission_type: str, user: User)` | All datasets the user holds `permission_type` on, filtered to `datasets` when that argument is non-empty. Backs [`datasets.list_datasets()`](#datasets-list_datasets). |
| `get_dataset_ids()` | `get_dataset_ids(datasets: list[str] \| list[UUID], user: User)` | Dataset UUIDs for the given identifiers. Names are only resolved against datasets the user *owns* in their tenant — to target a dataset owned by someone else, pass its UUID. Raises `DatasetTypeError` on mixed or unsupported types. |
`permission_type` is one of `read`, `write`, `delete`, or `share`.
### Creating and deleting datasets
| Method | Signature | Notes |
| ----------------------------- | --------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `create_dataset()` | `create_dataset(dataset_name: str, user: User)` | Returns the existing dataset with that name for the user's owner/tenant pair, or creates it. The id is derived deterministically from the name and owner, so different users can reuse the same dataset name. Grants no permissions. |
| `create_authorized_dataset()` | `create_authorized_dataset(dataset_name: str, user: User)` | `create_dataset()` plus `read`, `write`, `delete`, and `share` permissions for `user` — and for their parent user when `parent_user_id` is set. Use this one in multi-user setups. |
| `load_or_create_datasets()` | `load_or_create_datasets(dataset_names: list[str \| UUID], existing_datasets: list[Dataset], user: User)` | Reuses matching datasets from `existing_datasets` and creates the rest via `create_authorized_dataset()`. Raises `DatasetNotFoundError` if a UUID has no match. |
| `delete_dataset()` | `delete_dataset(dataset: Dataset)` | Deletes the dataset row, its [dedicated graph and vector databases](/core-concepts/multi-user-mode/dataset-database-handlers/dataset-database-handlers-what-are-they) if it has any, and the `Data` rows the dataset owns. Takes a `Dataset` object, not an id. |
| `check_dataset_name()` | `check_dataset_name(dataset_name: str)` | Synchronous. Raises `ValueError` if the name contains a space or a dot. |
<Note>
Because a `Data` row belongs to exactly one dataset, `delete_dataset()` also deletes that dataset's `Data` rows. Each removal goes through the same raw-file rule as [`delete_data()`](#datasets-delete_data): the file on disk is only removed when no other `Data` row still points at that `raw_data_location` and it lives under `DATA_ROOT_DIRECTORY`. Use [`datasets.empty_dataset()`](#datasets-empty_dataset) when you want to clear a dataset's data but keep the dataset itself.
</Note>
## Examples
<AccordionGroup>
<Accordion title="Basic dataset operations">
```python theme={null}
import cognee
# List all datasets
datasets = await cognee.datasets.list_datasets()
for ds in datasets:
print(ds.name, ds.id)
# Check dataset contents
data = await cognee.datasets.list_data(dataset_id=ds.id)
# Delete a specific item
await cognee.datasets.delete_data(
dataset_id=ds.id,
data_id=item.id,
)
# Wipe everything
await cognee.datasets.delete_all()
```
</Accordion>
<Accordion title="Poll for indexing completion across parallel datasets">
Use `get_status()` in a wait loop to confirm all datasets in a parallel batch have finished indexing before querying.
```python theme={null}
import asyncio
import cognee
from cognee.modules.pipelines.models import PipelineRunStatus
TERMINAL = {
PipelineRunStatus.DATASET_PROCESSING_COMPLETED,
PipelineRunStatus.DATASET_PROCESSING_ERRORED,
}
async def wait_for_indexing(dataset_ids, poll_interval=3, timeout=120):
for _ in range(timeout // poll_interval):
statuses = await cognee.datasets.get_status(dataset_ids)
if all(s in TERMINAL for s in statuses.values()):
return statuses
await asyncio.sleep(poll_interval)
raise TimeoutError("Indexing did not complete in time")
async def main():
batches = {
"batch_a": ["doc1.pdf", "doc2.pdf"],
"batch_b": ["doc3.pdf", "doc4.pdf"],
}
# Add and index multiple datasets in parallel
await asyncio.gather(*[
cognee.add(files, dataset_name=name) for name, files in batches.items()
])
await asyncio.gather(*[
cognee.cognify(datasets=[name]) for name in batches
])
# Confirm all datasets reached a terminal status
all_datasets = await cognee.datasets.list_datasets()
dataset_ids = [ds.id for ds in all_datasets if ds.name in batches]
statuses = await wait_for_indexing(dataset_ids)
for ds_id, status in statuses.items():
if status == PipelineRunStatus.DATASET_PROCESSING_COMPLETED:
print(f"{ds_id}: indexed successfully")
else:
print(f"{ds_id}: error — {status.value}")
asyncio.run(main())
```
The same pattern works when indexing is triggered via the [HTTP API](/api-reference/introduction) — poll `get_status()` from a separate process until all datasets reach `DATASET_PROCESSING_COMPLETED` or `DATASET_PROCESSING_ERRORED`.
</Accordion>
<Accordion title="Read back DataItem metadata">
```python theme={null}
import cognee
from cognee.tasks.ingestion.data_item import DataItem
await cognee.add(
DataItem(
"/path/to/report.pdf",
label="q4-report",
external_metadata={"author": "Jane Smith", "quarter": "Q4-2024"},
),
dataset_name="finance",
)
datasets = await cognee.datasets.list_datasets()
data_items = await cognee.datasets.list_data(dataset_id=datasets[0].id)
for item in data_items:
print(item.label, item.external_metadata)
# q4-report {"author": "Jane Smith", "quarter": "Q4-2024"}
```
`external_metadata` is stored on the relational `Data` record only. It is not placed into the vector store or knowledge graph and is not returned by `cognee.search()`. If you need metadata to be vector-searchable, define a custom `DataPoint` subclass and list the fields to embed in `metadata.index_fields`. See [DataPoints](/core-concepts/building-blocks/datapoints#indexing--embeddings).
</Accordion>
</AccordionGroup>
# delete()
Source: https://docs.cognee.ai/python-api/delete
Delete data from the knowledge base (deprecated)
# cognee.delete()
<Warning>
`cognee.delete()` is deprecated since v0.3.9. Use `cognee.datasets.delete_data()` instead.
</Warning>
```python theme={null}
async def delete(
data_id: UUID,
dataset_id: UUID,
mode: str = "soft",
user: Optional[User] = None,
)
```
## Migration
```python theme={null}
# Old (deprecated)
await cognee.delete(data_id=item_id, dataset_id=ds_id)
# New (recommended)
await cognee.datasets.delete_data(dataset_id=ds_id, data_id=item_id)
```
See [datasets](/python-api/datasets) for the full data management API.
# forget()
Source: https://docs.cognee.ai/python-api/forget
Remove data with the unified v1.0 deletion API
# cognee.forget()
```python theme={null}
async def forget(
*,
data_id: Optional[UUID] = None,
dataset: Optional[str] = None,
dataset_id: Optional[UUID] = None,
everything: bool = False,
memory_only: bool = False,
user=None,
) -> dict
```
## Description
`forget()` is the unified deletion API in Cognee v1.0.
* Use it to remove a single data item from a dataset.
* Use it to delete an entire dataset.
* Use `everything=True` to remove all memory owned by the current user.
* Use `memory_only=True` to clear graph and vector memory while keeping raw dataset records so the content can be reprocessed.
For the full behavior walkthrough, see [Forget](/core-concepts/main-operations/forget).
## Parameters
<ParamField type="Optional[UUID]">
Specific data item to remove. Requires `dataset` or `dataset_id` to also be set.
</ParamField>
<ParamField type="Optional[str]">
Dataset name. When used alone, deletes the whole dataset. When paired with `data_id`, deletes only that item from the dataset.
</ParamField>
<ParamField type="Optional[UUID]">
Dataset UUID — the alternative to `dataset`, and the only way to address a dataset whose name is ambiguous. Passing both raises `ValueError: Provide either dataset or dataset_id, not both.`
</ParamField>
<ParamField type="bool">
Deletes all datasets and memory owned by the current user.
</ParamField>
<ParamField type="bool">
Deletes only graph and vector memory for the target dataset or data item while keeping underlying dataset records.
Deletion is driven by provenance, so it only reaches nodes recorded as belonging to the target dataset or data item. Nodes written by a custom pipeline whose payload is not a tracked data item — the code-graph pipeline, for instance, which takes a repository path — carry no provenance and are left in place. To clear those, use [`prune`](/python-api/prune) instead: code-graph ingestion skips a repository whose snapshot identity still matches the marker on its `CodeRepository` node, and only removing that node makes the next run reload every fact.
</ParamField>
<ParamField type="Any">
Runs the operation under a specific user context instead of the default user.
</ParamField>
## Return value
`forget()` returns a summary dictionary. Depending on the mode, it includes fields like `status`, `dataset_id`, `data_id`, or `datasets_removed`.
## Troubleshooting
<Accordion title="Fixing DatasetNotFoundError (404) on forget()">
Every mode except `everything=True` resolves the dataset to an id before deleting anything: `dataset` is matched by name against the datasets you hold the **`delete`** permission on, in your own tenant. When nothing in that set matches, the call fails with:
```text theme={null}
DatasetNotFoundError: Dataset 'product_docs' not found or not accessible. (Status code: 404)
```
The message is deliberately the same whether the dataset does not exist or you simply cannot delete it — distinguishing the two would leak which dataset names exist. Over HTTP the same failure returns **404** with `{"detail": "Dataset 'product_docs' not found or not accessible. [DatasetNotFoundError]"}`.
<Note>
Before this fix — which means every release up to and including the current one — this case crashed with `AttributeError: 'NoneType' object has no attribute 'id'`, and the HTTP endpoint reported it as a generic `500 {"error": "An error occurred during deletion."}`. If you match on that traceback or on the 500, switch to catching `DatasetNotFoundError` / the 404.
</Note>
The error points at dataset resolution, not at your data — and it is the same for `forget(dataset="product_docs")` and `forget(dataset="product_docs", memory_only=True)`, since both resolve the dataset the same way. Common causes:
* **No dataset with that name.** A typo, a dataset that was already forgotten, or content that actually landed in the default `main_dataset` because no `dataset_name` was passed to [`remember()`](/python-api/remember).
* **No `delete` grant.** `read` and `write` access are not enough: resolution only considers datasets you can delete, so a dataset you can read is indistinguishable from one that does not exist. The [ACL troubleshooting entry](/core-concepts/multi-user-mode/permissions-system/acl#troubleshooting) lists the datasets you can delete and shows how to grant the permission.
* **A dataset in another tenant.** Resolution keeps only datasets whose tenant matches yours, so a name reachable across a tenant boundary never resolves.
Because names are matched rather than derived, a dataset someone else shared with you *does* resolve by name, provided the share carries `delete` and you are in the same tenant. The reverse case is the one to watch: when two datasets you can delete share a name, resolution picks whichever comes first, so pass `dataset_id=UUID(...)` whenever the name is ambiguous.
`dataset_id` fails differently — an id you cannot delete raises `PermissionDeniedError: Request owner does not have necessary permission: [delete] for all datasets requested.` (or `... for any dataset.` when you have no deletable datasets at all) instead of `DatasetNotFoundError`, and returns **403** over HTTP. Both are `CogneeApiError` subclasses, so a single `except` covers the pair:
```python theme={null}
from cognee.exceptions import CogneeApiError
try:
await cognee.forget(dataset="product_docs")
except CogneeApiError as error:
print(error.name, error.status_code) # DatasetNotFoundError 404
```
Catch `DatasetNotFoundError` (from `cognee.modules.data.exceptions`) or `PermissionDeniedError` (from `cognee.modules.users.exceptions`) individually when the two cases need different handling. Invalid parameter combinations are unchanged: they still raise `ValueError` and return **422**.
To find the right name or id, list your datasets first, as in [Inspect what you've stored before forgetting](/core-concepts/main-operations/forget). Bear in mind that `list_datasets()` filters by `read`, so a name that appears there can still fail to resolve for `forget()`.
</Accordion>
## Examples
```python theme={null}
import cognee
# Delete a dataset
await cognee.forget(dataset="product_docs")
# Delete only graph and vector memory so the dataset can be re-cognified
await cognee.forget(dataset="product_docs", memory_only=True)
```
## Related
For lower-level data-management APIs, see [datasets](/python-api/datasets), [delete()](/python-api/delete), and [prune](/python-api/prune).
# improve()
Source: https://docs.cognee.ai/python-api/improve
Enrich an existing graph and bridge session memory with the v1.0 API
# cognee.improve()
```python theme={null}
async def improve(
dataset: Union[str, UUID] = "main_dataset",
*,
run_in_background: bool = False,
node_name: Optional[List[str]] = None,
session_ids: Optional[List[str]] = None,
build_global_context_index: bool = False,
build_truth_subspace: bool = False,
**kwargs,
)
```
## Description
`improve()` enriches an existing graph after ingestion.
* Without `session_ids`, it runs the normal enrichment pass over the dataset.
* With `session_ids`, it can also apply feedback weights, persist session Q\&A, persist agent traces, distill accepted session guidance into `session_learnings`, update the calling user's preference subgraph, and sync enriched graph context back into sessions.
* With `build_global_context_index=True`, it builds dataset-level summary buckets for graph completion retrieval.
* With `build_truth_subspace=True`, it builds truth-subspace anchors from distilled `session_learnings`.
The default enrichment pass does **not** project the dataset's graph into memory: its tasks stream triplets straight from the graph database, so `improve()` no longer pays a full-graph scan and `O(graph size)` memory cost on every run. The graph is still projected when you pass your own `extraction_tasks` or `enrichment_tasks` without `data`, because a caller-supplied task may need that projected fragment. Enrichment output is unchanged either way.
For the full behavior walkthrough, see [Improve](/core-concepts/main-operations/improve).
## Parameters
<ParamField type="Union[str, UUID]">
Dataset name or UUID to improve. Requires `write` permission — a dataset owned by another user must be given as a UUID.
</ParamField>
<ParamField type="bool">
Starts the improvement pipeline asynchronously.
</ParamField>
<ParamField type="Optional[List[str]]">
Restricts the projected graph fragment to specific named entities or node sets. It only applies when the enrichment pass actually projects the graph — that is, when you supply custom `extraction_tasks`/`enrichment_tasks` and no `data`. The default enrichment pass skips projection, so `node_name` has no effect there.
</ParamField>
<ParamField type="Optional[List[str]]">
Session IDs whose feedback, Q\&A content, trace activity, and accepted distilled guidance should be bridged into the permanent graph.
Passing `session_ids` also runs the user-preference stage, which folds rated turns and stated preferences from those sessions into the calling user's per-dataset `prefers` weights and preference text. That stage is a no-op that writes nothing unless [`PERSONALIZATION_ENABLED`](/setup-configuration/overview) is on, and it is best-effort — a failure is logged and never blocks the rest of `improve()`. See [User Preferences](/core-concepts/further-concepts/user-preferences).
</ParamField>
<ParamField type="bool">
Builds the [global context index](/core-concepts/further-concepts/global-context-index) after enrichment. This is skipped when `run_in_background=True`.
</ParamField>
<ParamField type="bool">
Opt-in flag that builds the [truth subspace](/guides/truth-subspace-reranking) from distilled `session_learnings` — after distillation and before enrichment. Only runs when `session_ids` is provided, and is best-effort (a build failure is logged and never blocks the rest of `improve()`). Off by default means no behavior change.
</ParamField>
## Additional keyword options
| Option | Type | What it does |
| ------------------ | -------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `extraction_tasks` | `list` | Overrides the extraction task set used during enrichment. |
| `enrichment_tasks` | `list` | Overrides the enrichment task set used during enrichment. |
| `data` | `Any` | Supplies explicit data to advanced improvement pipelines. Skips the graph projection, so `node_name`/`node_type` are not applied. |
| `node_type` | `Type` | Changes which node type the projected graph fragment targets. Like `node_name`, it only applies when the graph is projected. |
| `user` | `object` | Runs the operation under a specific user context. |
| `vector_db_config` | `dict` | Overrides vector database configuration for this call. |
| `graph_db_config` | `dict` | Overrides graph database configuration for this call. |
| `feedback_alpha` | `float` | Controls how strongly session feedback changes graph weights. |
## Return value
`improve()` returns the pipeline result from the enrichment pass, the same underlying shape used by legacy `memify()`.
## Troubleshooting
<Accordion title="Fixing 'PermissionDeniedError' on improve()">
The target dataset is resolved and authorized for `write` **once, before any stage runs**, so a bad `dataset` fails the whole call instead of silently enriching your default dataset:
```text theme={null}
PermissionDeniedError: Request owner does not have necessary permission: [write] for all datasets requested.
```
Common causes:
* **No `write` grant on the UUID.** Read access is not enough — every improvement stage writes. Ask the dataset owner for a `write` grant; see [Access Control Lists](/core-concepts/multi-user-mode/permissions-system/acl).
* **The UUID does not exist.** Unknown and unauthorized UUIDs are reported identically, so dataset existence is never confirmed to an unauthorized caller. Double-check the UUID against your datasets.
* **You passed another user's dataset *name*.** Names are owner-scoped: improving a name that another user happens to own does not raise — it resolves (or creates) *your own* dataset with that name and leaves theirs untouched. To improve a dataset owned by someone else, pass its **UUID**.
</Accordion>
## Examples
```python theme={null}
import cognee
await cognee.improve(
dataset="product_docs",
session_ids=["support_chat_7"],
)
```
```python theme={null}
await cognee.improve(
dataset="product_docs",
session_ids=["support_chat_7"],
build_truth_subspace=True,
)
```
```python theme={null}
await cognee.improve(
dataset="product_docs",
build_global_context_index=True,
)
```
## Related
See also [Global Context Index](/core-concepts/further-concepts/global-context-index) for how to use the generated summaries during search, and [memify()](/python-api/memify) if you need lower-level control over the legacy enrichment pipeline.
# Python API Reference
Source: https://docs.cognee.ai/python-api/index
Complete Python API reference for the cognee package
# Python API Reference
Complete reference for the `cognee` Python package. All functions are accessible
directly from the top-level `cognee` module.
```python theme={null}
import cognee
# v1.0 workflow
await cognee.remember("Your data here")
results = await cognee.recall("Your query")
```
## Core Operations
These are the default v1.0 entry points for storing, querying, enriching, and deleting memory.
<CardGroup>
<Card title="remember()" href="/python-api/remember" icon="brain">
Store data as permanent graph memory or fast session memory in one call.
</Card>
<Card title="recall()" href="/python-api/recall" icon="search">
Query memory with auto-routing and session-aware retrieval.
</Card>
<Card title="improve()" href="/python-api/improve" icon="sparkles">
Enrich an existing graph and bridge session memory into permanent memory.
</Card>
<Card title="forget()" href="/python-api/forget" icon="trash">
Remove a data item, an entire dataset, or all memory for the current user.
</Card>
<Card title="serve()" href="/python-api/serve" icon="plug">
Connect the SDK to Cognee Cloud or another remote Cognee instance.
</Card>
<Card title="push()" href="/python-api/push" icon="upload">
Upload a local dataset's already-built graph to a remote Cognee instance.
</Card>
</CardGroup>
## Legacy Operations
The lower-level pipeline operations are still available when you need direct control over each step.
<CardGroup>
<Card title="add()" href="/python-api/add" icon="plus">
Ingest text, files, or structured data into the knowledge base.
</Card>
<Card title="cognify()" href="/python-api/cognify" icon="brain">
Transform raw data into a structured knowledge graph.
</Card>
<Card title="search()" href="/python-api/search" icon="search">
Query the knowledge graph with a chosen SearchType.
</Card>
<Card title="memify()" href="/python-api/memify" icon="sparkles">
Enrich an existing knowledge graph with custom extraction and enrichment tasks.
</Card>
</CardGroup>
## Data Management
<CardGroup>
<Card title="datasets" href="/python-api/datasets" icon="database">
List, create, and delete datasets.
</Card>
<Card title="users" href="/python-api/users" icon="user">
Get the default user, look up users, check existence, and create users.
</Card>
<Card title="agents" href="/python-api/agents" icon="bot">
Create, list, inspect, and delete agents, and manage their connections.
</Card>
<Card title="update()" href="/python-api/update" icon="pen">
Update existing data items.
</Card>
<Card title="prune" href="/python-api/prune" icon="trash">
Delete stored data and reset Cognee system state.
</Card>
<Card title="delete()" href="/python-api/delete" icon="trash">
Delete data items (deprecated — use datasets).
</Card>
<Card title="run_migrations()" href="/python-api/run-migrations" icon="circle-fading-arrow-up">
Apply pending relational and vector database schema migrations.
</Card>
</CardGroup>
## Session Management
<CardGroup>
<Card title="Sessions Guide" href="/guides/sessions" icon="message-square">
Use `session_id` and inspect stored conversation history with `get_session()`.
</Card>
<Card title="Feedback System" href="/guides/feedback-system" icon="brain-circuit">
Add and clear feedback on stored session Q\&A entries.
</Card>
</CardGroup>
## Configuration & Utilities
<CardGroup>
<Card title="config" href="/python-api/config" icon="gear">
Configure LLM providers, databases, chunking, and more.
</Card>
<Card title="SearchType" href="/python-api/search-type" icon="list">
The search modes you can pass to cognee.search().
</Card>
<Card title="run_custom_pipeline()" href="/python-api/custom-pipeline" icon="route">
Execute custom task pipelines.
</Card>
<Card title="Data Models" href="/python-api/data-models" icon="shapes">
Key types: DataPoint, Task, PipelineRunInfo, SearchResult.
</Card>
</CardGroup>
## Visualization
<CardGroup>
<Card title="Visualization Payloads" href="/python-api/visualize" icon="braces">
`visualize_graph_json()` and friends: the graph, semantic layout, per-dataset previews, and live events as plain dicts.
</Card>
<Card title="get_schema_inventory()" href="/guides/schema-inventory" icon="table">
Summarize the knowledge graph by semantic type: per-type counts, sample names, and relationships.
</Card>
<Card title="visualize_memory_provenance()" href="/guides/memory-provenance" icon="folder-tree">
Render the tenant/user/agent/dataset ownership and data-flow graph (relational-only) to HTML. Pair with `get_memory_provenance_graph()` for raw `(nodes, edges)`.
</Card>
<Card title="report()" href="/python-api/report" icon="file-chart-column">
Generate a Graph Insight Report in Markdown: hub nodes, cross-node-set connections, edge provenance, and suggested questions.
</Card>
<Card title="validate()" href="/python-api/validate" icon="shield-check">
Cross-check a dataset's graph and vector stores: orphaned edges, identity-id mismatches, and nodes missing from the vector index.
</Card>
</CardGroup>
# memify()
Source: https://docs.cognee.ai/python-api/memify
Enrich an existing knowledge graph with custom tasks
# cognee.memify()
```python theme={null}
async def memify(
extraction_tasks: Optional[Sequence[Union[Task, str]]] = None,
enrichment_tasks: Optional[Sequence[Union[Task, str]]] = None,
data: Optional[Any] = None,
dataset: Union[str, UUID] = 'main_dataset',
user: User = None,
node_type: Optional[Type] = NodeSet,
node_name: Optional[List[str]] = None,
vector_db_config: Optional[dict] = None,
graph_db_config: Optional[dict] = None,
run_in_background: bool = False,
)
```
## Description
Enrichment pipeline in Cognee, can work with already built graphs. If no data is provided existing knowledge graph will be used as data,
custom data can also be provided instead which can be processed with provided extraction and enrichment tasks.
Provided tasks and data will be arranged to run the Cognee pipeline and execute graph enrichment/creation.
This is the core processing step in Cognee that converts raw text and documents
into an intelligent knowledge graph. It analyzes content, extracts entities and
relationships, and creates semantic connections for enhanced search and reasoning.
Args:
extraction\_tasks: List of Cognee Tasks to execute for graph/data extraction.
Entries may be Task instances or names of built-in memify tasks
(see cognee.memify\_pipelines.memify\_task\_registry).
enrichment\_tasks: List of Cognee Tasks to handle enrichment of provided graph/data from extraction tasks.
Entries may be Task instances or names of built-in memify tasks.
data: The data to ingest. Can be anything when custom extraction and enrichment tasks are used.
Data provided here will be forwarded to the first extraction task in the pipeline as input.
If no data is provided the whole graph (or subgraph if node\_name/node\_type is specified) will be forwarded
dataset: Dataset name or dataset uuid to process.
user: User context for authentication and data access. Uses default if None.
node\_type: Filter graph to specific entity types (for advanced filtering). Used when no data is provided.
node\_name: Filter graph to specific named entities (for targeted search). Used when no data is provided.
vector\_db\_config: Custom vector database configuration for embeddings storage.
graph\_db\_config: Custom graph database configuration for relationship storage.
run\_in\_background: If True, starts processing asynchronously and returns immediately.
If False, waits for completion before returning.
Background mode recommended for large datasets (>100MB).
Use pipeline\_run\_id from return value to monitor progress.
## Parameters
<ParamField type="Optional[Sequence[Union[Task, str]]]">Task objects and/or [supported task names](#supported-task-names) for graph/data extraction. The two forms can be mixed in one list.</ParamField>
<ParamField type="Optional[Sequence[Union[Task, str]]]">Task objects and/or [supported task names](#supported-task-names) for graph enrichment. The two forms can be mixed in one list.</ParamField>
<ParamField type="Optional[Any]">Data to ingest. If not provided, operates on existing knowledge graph.</ParamField>
<ParamField type="Union[str, UUID]">Dataset name or UUID to operate on.</ParamField>
<ParamField type="User">User performing the operation.</ParamField>
<ParamField type="Optional[Type]">Filter to specific entity types in the graph.</ParamField>
<ParamField type="Optional[List[str]]">Filter to specific named entities.</ParamField>
<ParamField type="Optional[dict]">Override vector database configuration.</ParamField>
<ParamField type="Optional[dict]">Override graph database configuration.</ParamField>
<ParamField type="bool">If true, return immediately and process in background.</ParamField>
## Supported task names
`extraction_tasks` and `enrichment_tasks` accept `Task` instances, task names as plain strings, or a mix of both in the same list. Names are resolved against a curated registry (`cognee.memify_pipelines.memify_task_registry`) that only exposes tasks runnable without required keyword arguments. Every call builds a fresh `Task` instance per name, so resolved tasks are never shared between runs.
The registry is shared by both parameters. These names are typically used for extraction:
| Name | Task |
| ------------------------------- | ---------------------------------------------------------------------------- |
| `extract_subgraph` | Yield the edges of the incoming subgraphs |
| `extract_subgraph_chunks` | Pull document chunks from the incoming subgraphs |
| `get_triplet_datapoints` | Convert graph triplets into indexable datapoints (`triplets_batch_size=100`) |
| `extract_user_sessions` | Extract not-yet-persisted Q\&A entries from the session cache |
| `extract_agent_trace_feedbacks` | Extract step-level agent trace content for the current user |
| `detect_entity_duplicates` | Find semantically near-duplicate `Entity` nodes |
And these for enrichment:
| Name | Task |
| ------------------------------ | -------------------------------------------------------------------- |
| `cognify_session` | Cognify session windows into the knowledge graph |
| `cognify_agent_trace_feedback` | Cognify agent trace session text into the knowledge graph |
| `apply_feedback_weights` | Update graph-element weights from feedback scores (`batch_size=100`) |
| `apply_frequency_weights` | Increment graph-element weights on each use (`batch_size=100`) |
| `merge_entity_duplicates` | Merge the duplicates found by `detect_entity_duplicates` |
| `index_data_points` | Index datapoints in the vector DB (`batch_size=100`) |
Each name is constructed with the same defaults the dedicated memify pipelines use, shown in parentheses above. To use different arguments, pass a `Task` instance instead of a name.
<Note>
Tasks that require call-specific parameters are not in the registry and stay SDK-only. For example, `extract_feedback_qas` requires `session_ids`, so it must be passed as `Task(extract_feedback_qas, session_ids=[...])` rather than by name.
</Note>
### Invalid task names
Names are resolved before any setup or database work happens, so a bad name fails fast without starting a pipeline run.
* **Python SDK** — an unknown name, or an entry that is neither a `Task` nor a `str`, raises `CogneeValidationError`. The message for an unknown name lists every supported name.
* **REST (`POST /api/v1/memify`)** — the same condition returns **422 Unprocessable Content** with the supported names in the error body. Previously any non-empty task-name list failed with a 500.
## How memify() differs from cognify()
| | `cognify()` | `memify()` |
| ------------ | ----------------------------------- | ------------------------------------------ |
| **Purpose** | Build knowledge graph from raw data | Enrich an existing graph |
| **Input** | Raw text/files | Existing graph or new data |
| **Pipeline** | Fixed (chunk → extract → build) | Customizable extraction + enrichment tasks |
| **Use case** | Initial processing | Iterative refinement, entity consolidation |
## Examples
```python theme={null}
import cognee
# Enrich existing graph with default tasks
await cognee.memify()
# Enrich a specific dataset
await cognee.memify(dataset="my_dataset")
# Custom extraction and enrichment
from cognee.modules.pipelines import Task
await cognee.memify(
extraction_tasks=[my_extractor_task],
enrichment_tasks=[my_enrichment_task],
dataset="my_dataset",
)
# Select built-in tasks by name
await cognee.memify(
extraction_tasks=["detect_entity_duplicates"],
enrichment_tasks=["merge_entity_duplicates"],
dataset="my_dataset",
)
# Mix task names with Task instances
from cognee.tasks.storage.index_data_points import index_data_points
await cognee.memify(
extraction_tasks=["get_triplet_datapoints"],
enrichment_tasks=[Task(index_data_points, task_config={"batch_size": 500})],
dataset="my_dataset",
)
# Filter to specific entity types
await cognee.memify(node_name=["Person", "Organization"])
```
See the [Memify pipeline guides](/guides/memify-session-persistence) for lower-level enrichment walkthroughs, or the [Self-Improvement Quickstart](/guides/self-improvement-quickstart) for the v1.0 user-facing flow.
# prune
Source: https://docs.cognee.ai/python-api/prune
Methods for deleting stored data and resetting Cognee system state.
# cognee.prune
Static class with methods for cleaning up Cognee data and system state.
## Methods
### prune.prune\_data()
```python theme={null}
await cognee.prune.prune_data()
```
Removes all raw data files from the file storage backend (local disk or S3,
configured via `DATA_ROOT_DIRECTORY`). Does **not** remove database records —
use `prune_system(metadata=True)` for that.
### prune.prune\_system()
```python theme={null}
await cognee.prune.prune_system(
graph: bool = True,
vector: bool = True,
metadata: bool = False,
cache: bool = True,
)
```
Cleans up system database resources selectively. Does **not** delete files
from storage — call `prune_data()` first if you also need to wipe raw files.
<ParamField type="bool">Delete all data from the graph database (e.g., Kuzu, Neo4j).</ParamField>
<ParamField type="bool">Delete all data from the vector store (e.g., LanceDB, Qdrant).</ParamField>
<ParamField type="bool">Delete the relational metadata database (datasets, data records, pipeline state). Defaults to `False` to prevent accidental loss of metadata.</ParamField>
<ParamField type="bool">Clear the cache. This wipes the cache directory (`CACHE_ROOT_DIRECTORY` — e.g. downloaded ontologies and tutorial data) and, when session caching or usage logging is enabled, also prunes the [session cache backend](/core-concepts/sessions-and-caching) (conversation history and usage logs). It does **not** touch graph, vector, or relational data.</ParamField>
<Warning>
`prune_system` has no permission checks and will wipe **all** graph and vector
data regardless of which user or dataset it belongs to. Only use it in local
development or test environments.
</Warning>
<Note>
**Prune cannot target a single dataset.** Neither `prune_data()` nor
`prune_system()` accepts a `dataset_id` — both always operate globally across
every dataset. To remove a single dataset, use
[`cognee.datasets.empty_dataset(dataset_id)`](/python-api/datasets#datasets-empty_dataset),
which deletes that dataset's graph content, data records, and the dataset entity
itself while leaving other datasets untouched. To remove a single data item, use
[`cognee.datasets.delete_data(dataset_id, data_id)`](/python-api/datasets#datasets-delete_data).
</Note>
## Examples
```python theme={null}
import cognee
# Full reset: remove raw files, databases, and metadata
await cognee.prune.prune_data() # wipes raw files from disk / S3
await cognee.prune.prune_system(metadata=True) # wipes graph, vector, relational DB, and cache
# Partial reset: clear graph and vector stores only (keep metadata and files)
await cognee.prune.prune_system(graph=True, vector=True, metadata=False, cache=False)
# Clear only the cache (session history, usage logs, cached files)
await cognee.prune.prune_system(graph=False, vector=False, metadata=False, cache=True)
```
## FAQ
<AccordionGroup>
<Accordion title="How do I re-cognify data without deleting my PostgreSQL metadata?">
Keep `metadata=False` (the default) if you want to preserve the relational
database. But for a true re-cognify of already-processed data, `prune_system()`
is **not** enough by itself: it clears graph/vector/cache storage, but it does
not reset the per-data-item `cognify_pipeline` status that incremental
`cognify()` uses to decide what to skip.
For the safe rebuild flow, use [`forget(..., memory_only=True)`](/core-concepts/main-operations/forget#forget-only-memory-for-a-dataset-keep-raw-files)
on the dataset you want to re-process. That preserves raw files, datasets, and
data records while clearing graph/vector memory and resetting cognify status:
```python theme={null}
# Remove only derived memory for one dataset, keep files + metadata
await cognee.forget(dataset="my_dataset", memory_only=True)
await cognee.cognify(datasets="my_dataset")
```
Setting `metadata=True` would delete the datasets, data records, and pipeline
state from the relational database (PostgreSQL or SQLite), forcing you to
`add()` your data again before re-cognifying.
</Accordion>
<Accordion title="Do I need to prune when switching vector databases?">
If you switch `VECTOR_DB_PROVIDER`, the new vector store starts empty, so you do
need to rebuild memory for the datasets you want to query. The safest documented
flow is dataset-scoped: update `VECTOR_DB_PROVIDER` (see [Vector Stores](/setup-configuration/vector-stores)),
then clear only the dataset's derived memory and re-cognify it:
```python theme={null}
await cognee.forget(dataset="my_dataset", memory_only=True)
await cognee.cognify(datasets="my_dataset")
```
This preserves the dataset's files and relational metadata while rebuilding the
graph and embeddings against the new vector store.
If you use `prune_system(vector=True, metadata=False)`, remember that it clears
the vector storage globally but does **not** reset cognify status for existing
data items, so a plain follow-up `cognify()` may skip them.
</Accordion>
<Accordion title="What exactly does cache=True clear?">
It removes the cache directory (`CACHE_ROOT_DIRECTORY`, holding cached files such
as downloaded ontologies and tutorial data) and, when session caching or usage
logging is enabled, prunes the [session cache backend](/core-concepts/sessions-and-caching)
(conversation history and usage logs). It never deletes graph, vector, or
relational data, so clearing the cache is safe to combine with any re-cognify
workflow.
</Accordion>
</AccordionGroup>
# push()
Source: https://docs.cognee.ai/python-api/push
Upload a local dataset's knowledge graph to Cognee Cloud
# cognee.push()
```python theme={null}
async def push(
dataset: Union[str, UUID] = "main_dataset",
*,
target_dataset: Optional[str] = None,
mode: str = "preserve",
run_in_background: bool = False,
url: Optional[str] = None,
api_key: Optional[str] = None,
user: User = None,
) -> PushResult
```
## Description
Upload a local dataset's **already-built knowledge graph** to a Cognee Cloud (or
any remote Cognee) instance.
The dataset's graph is exported to a [COGX archive](/core-concepts/further-concepts/cogx),
packed as a tarball, uploaded, and imported on the remote instance — preserving
the entities and relationships you extracted locally instead of re-deriving them
from the raw files.
This is what distinguishes `push()` from [`sync()`](/cognee-cloud/connections/syncing-local-instance):
`push()` ships the **graph** so the remote side does little or no LLM work, while
`sync()` ships the **raw data** and the remote instance rebuilds the graph itself.
<Note>
The dataset must already have a knowledge graph. If the export finds 0 nodes,
`push()` raises an error — run [`cognify()`](/python-api/cognify) (or
[`remember()`](/python-api/remember)) on the dataset first.
</Note>
## Authentication
`push()` reuses the `serve()` credential stack. Run [`cognee.serve()`](/python-api/serve)
(or `cognee-cli serve`) once to log in, then push any time. The remote target is
resolved in this order:
1. Explicit `url` / `api_key` arguments
2. An active `cognee.serve()` connection
3. `COGNEE_SERVICE_URL` / `COGNEE_API_KEY` environment variables
4. Saved credentials from a previous `serve()` login (`~/.cognee/cloud_credentials.json`)
If none resolve, `push()` raises a `RuntimeError` telling you to authenticate.
## Parameters
<ParamField type="Union[str, UUID]">Local dataset name or UUID to push. Requires read permission.</ParamField>
<ParamField type="Optional[str]">Dataset name on the remote instance. Defaults to the local dataset's name.</ParamField>
<ParamField type="str">Remote import fidelity. One of `preserve`, `hybrid`, or `re-derive` (see below).</ParamField>
<ParamField type="bool">If true, the remote import is scheduled and the call returns once the upload completes; poll the returned `pipeline_run_id` for progress. Recommended for large graphs.</ParamField>
<ParamField type="Optional[str]">Remote instance URL. Falls back to the active `serve()` connection, `COGNEE_SERVICE_URL`, or saved credentials.</ParamField>
<ParamField type="Optional[str]">API key for the remote instance. Falls back like `url`.</ParamField>
<ParamField type="User">Local user context for the export. Uses the default user when omitted.</ParamField>
## Import modes
| Mode | Remote behavior | LLM calls |
| ---------------------- | ------------------------------------------------------------------- | --------- |
| `preserve` *(default)* | Map the exported entities and facts directly into the remote graph | None |
| `hybrid` | Preserve the exported graph **and** cognify the raw content | Yes |
| `re-derive` | Ignore the exported graph; rebuild it from the raw content remotely | Yes |
## Returns
A `PushResult` dataclass:
<ParamField type="str">Status reported by the remote remember call (for example `"started"`).</ParamField>
<ParamField type="str">Name of the local dataset that was exported.</ParamField>
<ParamField type="str">Dataset name the graph was imported into on the remote instance.</ParamField>
<ParamField type="int">Number of nodes uploaded.</ParamField>
<ParamField type="int">Number of edges uploaded.</ParamField>
<ParamField type="Optional[str]">Remote migration pipeline run id; poll it when `run_in_background=True`.</ParamField>
<ParamField type="dict">Raw response returned by the remote remember endpoint.</ParamField>
```python theme={null}
from cognee import PushResult
```
## Examples
```python theme={null}
import cognee
# Build a graph locally, then push it to the cloud
await cognee.remember("docs/handbook.pdf", dataset="onboarding")
# Connect once; the credentials are saved and reused afterwards
await cognee.serve(url="https://your-tenant.aws.cognee.ai", api_key="your-api-key")
# Push the default dataset (main_dataset)
result = await cognee.push()
# Push a named dataset
result = await cognee.push("onboarding")
print(result.num_nodes, result.num_edges)
# Push to a different dataset name on the remote instance
await cognee.push("onboarding", target_dataset="prod_onboarding")
# Preserve the graph AND re-cognify the raw content on the remote side
await cognee.push("onboarding", mode="hybrid")
# Large graph: schedule the remote import and return after the upload
result = await cognee.push("onboarding", run_in_background=True)
print(result.pipeline_run_id) # poll this for progress
# Push to an explicit instance without a prior serve() login
await cognee.push(
"onboarding",
url="https://your-tenant.aws.cognee.ai",
api_key="your-api-key",
)
```
<Note>
The remote instance must run a Cognee version with COGX archive import support.
Against an older server the upload is accepted but ingested as a plain file; `push()`
detects this and raises an error rather than silently degrading. Upgrade the remote
instance, or use [`sync()`](/cognee-cloud/connections/syncing-local-instance) /
[`remember()`](/python-api/remember) with raw data instead.
</Note>
## See also
* [Syncing a Local Instance](/cognee-cloud/connections/syncing-local-instance) — connect the SDK to a remote instance and ship raw data with `sync()`
* [Cognee CLI → Push to Cloud](/cognee-cli/overview#push-to-cloud) — the same operation from the terminal (`cognee push`)
# recall()
Source: https://docs.cognee.ai/python-api/recall
Query memory with the v1.0 retrieval API
# cognee.recall()
```python theme={null}
async def recall(
query_text: str,
query_type: SearchType | None = None,
*,
datasets: list[str] | None = None,
dataset_ids: list[UUID] | None = None,
top_k: int = 15,
auto_route: bool = True,
scope: str | list[str] | None = None,
# plus the keyword-only options listed under "Additional keyword options"
) -> list[RecallResponse]
```
## Description
`recall()` is the main retrieval entry point in Cognee v1.0.
* It auto-routes queries by default when you do not specify `query_type`. Routing is rule-based (no LLM call) and falls back to `HYBRID_COMPLETION` when no cue matches — see [Auto-routing behavior](/core-concepts/main-operations/recall#examples-and-details) for the full cue-to-search-type mapping and when to override.
* It can search the permanent graph, session memory, or both.
* It returns `RecallResponse` items sourced from graph retrieval, session retrieval, or both depending on the request.
For the full behavior walkthrough, see [Recall](/core-concepts/main-operations/recall) and [Search Basics](/guides/search-basics).
## Prerequisites
`recall()` only reads from memory that already exists — it does not initialize anything on its own. Populate memory first with [`remember()`](/python-api/remember) (or the legacy [`add()`](/python-api/add) + [`cognify()`](/python-api/cognify) sequence). The first ingestion run creates the relational, vector, and graph databases and the default user.
```python theme={null}
import cognee
await cognee.remember("Einstein was born in Ulm.") # creates databases + ingests
results = await cognee.recall("Where was Einstein born?")
```
<Warning>
Calling `recall()` before any data has been ingested raises `RecallPreconditionError` (a `CogneeValidationError`, HTTP 422) with the message *"Recall prerequisites not met: no database/default user found."* It is triggered by the underlying `DatabaseNotCreatedError` (*"The database has not been created yet. Please call `await setup()` first."*) or `UserNotFoundError`. The fix is to run `remember()` (or `add()` + `cognify()`) first.
</Warning>
## Parameters
<ParamField type="str">
Natural-language query to run against memory.
</ParamField>
<ParamField type="SearchType | None">
Forces a specific retrieval strategy instead of using auto-routing. Leaving it unset when **no usable LLM is configured** resolves to `SearchType.CHUNKS` (plain vector search over chunks) instead of a completion, because nothing could write a completion answer. This is decided by LLM availability alone — the same key rule the [provider preflight](/setup-configuration/overview#provider-consistency-preflight) applies — and never by which extractor built the graph: a [GLiNER-built graph](/python-api/cognify#llm-free-extraction-with-gliner) answers completions normally as long as a key is present. An explicit `query_type` still selects any search type.
</ParamField>
<ParamField type="list[str] | None">
Restricts graph retrieval to the named datasets. Dataset names are resolved only against datasets owned by the current user. **When both `datasets` and `dataset_ids` are omitted, retrieval spans every dataset the current user has `read` access to** — not just a single default dataset. Pass this to narrow the search to specific datasets.
</ParamField>
<ParamField type="list[UUID] | None">
Restricts graph retrieval by dataset UUIDs instead of names. Use this for shared datasets that the current user can access but did not create. When provided, this takes precedence over `datasets` and the name-to-UUID lookup is skipped. Leaving both `datasets` and `dataset_ids` unset searches all of the user's readable datasets.
</ParamField>
<ParamField type="int">
Maximum number of results to return.
</ParamField>
<ParamField type="bool">
When `True`, Cognee chooses a retrieval strategy automatically if `query_type` is not set, using the rule-based query router. Set it to `False` to always use `HYBRID_COMPLETION`. An explicit `query_type` always takes precedence over routing. With no usable LLM configured, the no-`query_type` default is `CHUNKS` under either setting — see `query_type` above.
</ParamField>
<ParamField type="str | list[str] | None">
Which sources retrieval reads from. Accepts a single value or a list:
* `"graph"` — the permanent knowledge graph.
* `"session"` — session-cache Q\&A history, matched by keyword.
* `"trace"` — recorded agent traces, matched by keyword across function name, parameters, return value, and error message. See [Agent Session Traces](/guides/agent-session-traces).
* `"session_context"` — the session's active guidance, rendered read-only for the profile named by `context_profile`. Serving it never stamps or ages an entry.
* `"all"` — expands to `graph`, `session`, `trace`, and `session_context`.
* `"auto"` — the default when `scope` is omitted. Resolves to `graph` alone when there is no `session_id`, or when `session_id` is combined with an explicit `query_type`; otherwise to `session` plus `graph`. It never selects `trace` or `session_context` — ask for those by name.
`"tools"` and `"code"` are explicit opt-in only: neither `"auto"` nor `"all"` includes them, so name them yourself (`scope=["all", "tools"]` works). An unrecognized name raises `ValueError`. `"graph_context"` is a deprecated alias for `"graph"`.
</ParamField>
## Additional keyword options
| Option | Type | What it does |
| --------------------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `system_prompt` | `str` | Overrides the system prompt used for completion-style answers. |
| `system_prompt_path` | `str` | Loads the system prompt from a file path. |
| `node_name` | `list[str]` | Restricts retrieval to matching node names or node sets. |
| `node_name_filter_operator` | `str` | Controls how `node_name` filters are combined. |
| `only_context` | `bool` | Returns retrieved context without generating the final LLM answer. |
| `context_format` | `ContextFormat \| str` | Default `"context"`. Shape of the `only_context=True` result; ignored otherwise. `"context"` keeps the previous bare-context output. `"prompt"` returns the prompt envelope — `question`, `context`, `session_context`, `user_prompt`, `system_prompt` — reconstructed read-only: no LLM call and no session write, though it does cost one conversation-history read, which embeds the query. `user_prompt` and `system_prompt` are `None` for the non-generative search types (`CHUNKS`, `SUMMARIES`, `CODE`, …) and for `CYPHER` and `AGENTIC_COMPLETION`, which opt out of the preview. Under `"prompt"`, recall returns a **single** item for the envelope rather than one per context entry: the envelope is on `raw`, `text` is the rendered `user_prompt` (or the rendered context when there is no prompt), and a query that retrieved nothing returns no items at all. Because the preview cannot make the LLM call that rewrites your question, the envelope reports the prompt for the context actually retrieved rather than a replay of a full turn — see [The prompt envelope](/python-api/search#the-prompt-envelope). An invalid value raises `InvalidContextFormatError` (a `CogneeValidationError`, HTTP 422). |
| `session_id` | `str` | Enables session-aware retrieval and session-cache lookup. |
| `context_profile` | `str` | Default `"qa"`. Which body of session guidance the `"session_context"` scope renders: `"qa"` for guidance learned from conversation turns (sections `goals`, `rules`, `preferences`, `lessons_learned`), or `"agent"` for lessons learned from agent tool traces (sections `failure_lessons`, `tool_rules`, `environment_facts`, `workflow_state`, `success_patterns`). The two are separate — a query under one profile never returns the other's entries. Ignored unless `scope` includes `"session_context"`. |
| `wide_search_top_k` | `int` | Expands the candidate set used before final ranking in graph retrieval. Graph-only: any explicit value raises `InvalidHybridSearchConfig` (a `CogneeValidationError`, HTTP 422) when the recall runs `HYBRID_COMPLETION` — the fallback whenever auto-routing matches no cue — so pair it with an explicit graph-family `query_type`. See [Search Basics — Advanced Parameters](/guides/search-basics#advanced-parameters). |
| `triplet_distance_penalty` | `float` | Adjusts ranking for triplet-based retrieval paths. Graph-only, rejected on `HYBRID_COMPLETION` exactly like `wide_search_top_k`. |
| `feedback_influence` | `float` | Applies stored feedback weights during ranking where supported. |
| `verbose` | `bool` | Returns additional retrieval details from lower-level search flows. |
| `retriever_specific_config` | `dict` | Passes advanced configuration directly to the selected retriever. |
| `code_query` | `dict \| None` | Default `None`. Structured operation and arguments for the `"code"` scope, in the same shape as [`search(code_query=...)`](/python-api/search-type#per-search-type-parameters) — for example `{"operation": "impact_analysis", "name": "UserService"}`. `None` runs the default `explore` operation seeded with the query text. Only valid when `scope` includes `"code"`, which neither `"auto"` nor `"all"` implies; passing it otherwise raises `InvalidCodeQueryError`. Results carry `source="code"`, and a seed the code graph cannot resolve contributes nothing rather than failing the recall. |
| `response_model` | `type \| None` | Default `None`. Pydantic model class for structured completion output — see [Structured output](#structured-output-with-response_model) below. |
| `include_references` | `bool` | Default `False`. When set to `True`, appends a deterministic `Evidence:` block to completion-style answers, assembled in-process (no extra LLM call), and fills `metadata.evidence` on the result with structured references to the chunks, graph nodes, and graph edges placed in the LLM context. For graph completion the cited chunks come from the [edge-evidence sidecar](/setup-configuration/overview#edge-evidence). The block is omitted silently when no usable references exist. |
| `user` | `object` | Runs retrieval under a specific user context. |
| `llm_config` | `LLMConfig` | LLM settings to install into the current async context for this retrieval operation. Uses the active context config or global LLM config when omitted. Import from `cognee.infrastructure.llm.config`. |
| `embedding_config` | `EmbeddingConfig` | Embedding settings to install into the current async context for this retrieval operation. Uses the active context config or global embedding config when omitted. Import from `cognee.infrastructure.databases.vector.embeddings.config`. |
<Warning>
`recall()` accepts **only** the parameters documented above — it has no catch-all `**kwargs`. Passing an unsupported keyword such as `node_type` raises `TypeError: recall() got an unexpected keyword argument 'node_type'`. To restrict retrieval to specific nodes or node sets, use `node_name` (a `list[str]`). `node_type` is a [legacy `search()`](/python-api/search) parameter and is not exposed on `recall()`.
</Warning>
### Structured output with `response_model`
Pass a Pydantic model class to get a validated, parsed answer instead of free text — each result carries the validated payload as a dict in its `structured` field. All completion-style search types support it (`GRAPH_COMPLETION` and its variants except `GRAPH_SUMMARY_COMPLETION`, `RAG_COMPLETION`, `TRIPLET_COMPLETION`, `HYBRID_COMPLETION`, `TEMPORAL`, `AGENTIC_COMPLETION`):
```python theme={null}
from pydantic import BaseModel
import cognee
from cognee import SearchType
class NLPFacts(BaseModel):
field_name: str
parent_disciplines: list[str]
results = await cognee.recall(
query_text="What is NLP and which disciplines does it belong to?",
query_type=SearchType.GRAPH_COMPLETION,
response_model=NLPFacts,
)
results[0].structured
# {'field_name': 'Natural Language Processing', 'parent_disciplines': [...]}
```
`response_model` is shorthand for `retriever_specific_config={"response_model": ...}` — Cognee folds the parameter into the config before dispatching, so the dict form still works. Pass it in one place: supplying the **same** model class through both is allowed, but **different** classes raise `CogneeValidationError` (HTTP 422).
<Note>
**Remote mode.** A Python class cannot cross the HTTP boundary, so against a remote server (see [`serve()`](/core-concepts/main-operations/serve)) the SDK forwards `response_model.model_json_schema()` as the `response_schema` field of `POST /api/v1/recall`, and the server rebuilds a validation model from it. Only the schema's **structure** travels — custom validators and value constraints are not enforced server-side; rehydrate on the client (`NLPFacts.model_validate(results[0].structured)`) when you need them. See [Search & Recall — `response_schema`](/cognee-cloud/functionality/search-and-recall#structured-output-with-response_schema) for the supported schema subset and rejection rules.
</Note>
## Return value
`recall()` returns a list of `RecallResponse` items. Depending on the request, results may come from session memory, permanent graph retrieval, or both.
These items are **Pydantic objects, not plain dictionaries** — read fields with attribute access (`result.text`), not `result.get("text")` or `result["text"]`. Calling `.get()` on a result raises `AttributeError: 'ResponseGraphEntry' object has no attribute 'get'`.
The concrete type of each item is set by its `source` field (import from `cognee.modules.recall.types.RecallResponse`):
| `source` | Type | Key attributes |
| ------------------- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `"graph"` | `ResponseGraphEntry` | `text` (renderable answer, context, chunk text, or structured output), `kind`, `search_type`, `score`, `dataset_id`, `dataset_name`, `metadata`, `raw` (normalized payload for this item), `structured` |
| `"session"` | `ResponseQAEntry` | `time`, `qa_id`, `question`, `context`, `answer`, `feedback_text`, `feedback_score` |
| `"trace"` | `ResponseAgentTraceEntry` | session agent-trace fields |
| `"session_context"` | `ResponseSessionContextEntry` | `content`, `context_profile` |
| `"code"` | `ResponseCodeEntry` | same fields as `"graph"` entries — a deterministic code-graph fact from the `"code"` scope; only the `source` discriminator differs |
| `"tools"` | `ResponseToolEntry` | `tool_name`, `question`, `text`, `success`, `error`, `structured` |
| `"skills"` | `ResponseSkillEntry` | `text` (a `name: description` line), `skill` (metadata-only dict, never the procedure body), `score` (raw vector distance, lower is better) — appended by the [skill gate](#skill-gate) |
| `"system"` | `ResponseMarkerEntry` | `status`, `text`, `datapoint_count`, `threshold`, plus `error_class` and `error_message` when `status` is `"build_failed"` — a system-generated marker rather than retrieved data, see [Warming-up marker](#warming-up-marker) |
### Warming-up marker
Before running graph retrieval, `recall()` checks how far the target datasets got through a Cognee pipeline. This check is a single indexed relational query — it never spins up a graph or vector engine. It classifies the datasets into three readiness states:
| State | Meaning | Marker `status` |
| ------------ | ------------------------------------------------------------------------------ | ------------------------------------- |
| warm | A graph-writing pipeline run completed. | *(no marker — normal retrieval runs)* |
| never built | No pipeline run has ever been recorded for those datasets. | `memory_warming_up` |
| build failed | No graph-writing run has completed and the last graph-writing attempt errored. | `build_failed` |
For the two cold states the graph lane returns immediately, with a marker instead of a graph search plus an LLM call that could only come back empty:
```python theme={null}
# "analytics" exists, but nothing has ever been ingested into it
results = await cognee.recall("What does Cognee do?", datasets=["analytics"])
results[0].source # "system"
results[0].status # "memory_warming_up"
results[0].text # "Memory is still warming up: no knowledge graph data exists yet for the requested datasets."
results[0].datapoint_count # 0
results[0].threshold # 1 (the configured RECALL_WARMUP_THRESHOLD)
```
When the last ingestion for those datasets errored instead, the marker explains the failure rather than claiming memory is still warming up:
```python theme={null}
# the first cognify() for "analytics" errored
results = await cognee.recall("What does Cognee do?", datasets=["analytics"])
results[0].source # "system"
results[0].status # "build_failed"
results[0].text # "Memory build failed: the last ingestion for the requested datasets
# ended in an error (AuthenticationError: …). Fix the cause and
# re-run remember() or cognify()."
results[0].error_class # "AuthenticationError"
results[0].error_message # PII-scrubbed message of the underlying error
results[0].datapoint_count # 0
```
If you branch on `source`, add a `"system"` case — code that previously saw an empty list for a cold dataset now sees a one-item list carrying this marker. `text` is populated so consumers that just render text still display something sensible. If you branch on `status`, treat `build_failed` as a failure to surface to the user, not as a transient "still warming up" state that will resolve on its own: it only clears once a graph-writing run succeeds.
Details and exceptions:
* **The marker only appears when graph is the sole source.** In a multi-source recall (for example a session-scoped call that reads both session memory and the graph), a cold graph contributes `[]` instead, so the other sources — and the `tools` `on_empty` fallback — behave exactly as if graph retrieval had returned nothing.
* **`only_context=True` bypasses the check** and always runs normal retrieval, since those callers expect context rather than a marker.
* **Populated datasets are unaffected.** A dataset with a completed graph-writing run reads as warm. So does one that has only been `add()`-ed but not yet cognified — staged-only datasets fail safe to warm, and a failed `add()` never produces a `build_failed` marker, since staging says nothing about the graph.
* **A build still in flight reads as warm.** The never-built verdict requires that no pipeline run exists at all, so once a first `cognify()` has started — a background run, for example — recall runs a normal search that comes back empty rather than returning a `memory_warming_up` marker.
* **Only warm verdicts are cached.** Both cold states are re-probed on every recall, so the first recall after a successful rebuild sees the truth immediately.
* **The check fails open.** A probe or configuration error falls through to a normal search, so it can never block a real answer.
* **It can be turned off** with `RECALL_WARMUP_SHORTCIRCUIT=false` (or `cognee.config.set("recall_warmup_shortcircuit", False)`), which restores the previous behavior exactly. See [Recall warm-up](/setup-configuration/overview#recall-warm-up) for that variable and its two companions.
### Skill gate
When a query reads like a request for a procedure, `recall()` also looks for matching [skill](/examples/self-improving-skills) playbooks and appends them to the results as `"skills"` entries. The gate is a fixed set of weighted regexes — no LLM call, no I/O — that fires on phrasings such as `how do I …`, `steps to …`, `walk me through`, `playbook`, `runbook`, `checklist`, `workflow`, or the word `skill`. An operational verb alone (`set up`, `install`, `configure`, `deploy`, `migrate`, `troubleshoot`, and similar) is too weak to fire the gate by itself and only counts alongside one of those phrasings. A negated phrase does not count, using the same suppression as the query router.
When it fires, `recall()` starts a [`SKILLS`](/python-api/search-type) search for up to three skills concurrently with the main retrieval and appends the hits after the other sources. Each hit is a `ResponseSkillEntry` whose `skill` dict holds the skill's metadata only — the `procedure` body stays behind the `load_skill` tool or `GET /api/v1/skills/{skill_id}`.
* **Additive and fail-safe.** The main lanes never wait on the gate, it never replaces or blocks the main answer, and any error in the lookup contributes nothing rather than failing the recall. A dataset with no skills ingested yet simply adds no entries.
* **Single-dataset only.** Skills are scoped per dataset, so the gate runs only when exactly one dataset is targeted through `datasets` or `dataset_ids`. Otherwise it is skipped silently.
* **Skipped when it would be redundant.** It does not run when `"graph"` is not in scope, when `only_context=True`, or when `query_type` is already `SKILLS` or `AGENTIC_COMPLETION`.
* **It can be turned off** with `SKILL_GATE_ENABLED=false`.
### Source provenance in `metadata`
For chunk and summary results (`CHUNKS`, `CHUNKS_LEXICAL`, `SUMMARIES`), the `metadata` dict carries stable source identifiers so you can map a result back to the data you ingested and inspect the exact cited chunk. Only the keys present in the underlying payload are included:
| `metadata` key | Type | Meaning |
| --------------- | ----- | ------------------------------------------------------------------------------------------------------------------- |
| `data_id` | `str` | Id of the ingested `Data` item (cognify sets `Document.id = data.id`, so a chunk's `document_id` is the `data_id`). |
| `chunk_id` | `str` | The chunk's own node id — use it to look up the exact cited chunk. |
| `chunk_index` | `int` | 0-based position of the chunk within its document. |
| `document_name` | `str` | Name of the source document. |
Completion-style results (e.g. `GRAPH_COMPLETION`) carry an empty `metadata` dict unless `include_references=True` (see [`include_references`](#additional-keyword-options)); then `metadata.evidence` lists structured references to what was placed in the LLM context, and the same ids are also surfaced inline in the `Evidence:` block. `RAG_COMPLETION` bullets are rendered as `- chunk N of document NAME (data_id: …, chunk_id: …): "snippet"`, built from the retrieved chunk payloads with no DB migration required. `GRAPH_COMPLETION` bullets are rendered as `- chunk N of document NAME (data_id: …, chunk_id: …)` with no snippet, resolved through the [edge-evidence sidecar](/setup-configuration/overview#edge-evidence), which needs Alembic revision `f3a7b9c1d2e4` on an existing deployment.
```python theme={null}
results = await cognee.recall("What does Cognee do?")
for result in results:
if result.source == "graph":
print(result.text) # answer or chunk text
print(result.raw) # normalized payload for this item
elif result.source == "session":
print(result.answer)
```
<Note>
The `text_result`, `context_result`, and `objects_result` keys — plus `session_context_result`, `user_prompt_result`, and `system_prompt_result` when `context_format="prompt"` was requested — come from the legacy [`search(verbose=True)`](/python-api/search) API, which returns plain dicts. `recall()` does **not** produce those keys. For a graph-backed recall item, `result.text` is the display-ready value. `result.raw` preserves the normalized payload for that item; for completion-style searches, it is not the same thing as `objects_result`.
</Note>
### Relevance in `score`
`CHUNKS`, `SUMMARIES`, and `SKILLS` results carry the retriever's own relevance number in `score` (and, for chunks and summaries, under `raw["score"]` alongside the rest of the payload), so you can rank or fuse results across retrievers instead of trusting return order alone.
The value is the **raw backend distance, not a normalized similarity** — cosine distance for the built-in vector adapters — so **a lower number is a better match**, and the range depends on the vector store you run. Do not compare a score across two different backends, and do not read it as a 0–1 confidence.
`score` is `None` for search types whose payload carries no numeric score: completion-style search types, and `CHUNKS_LEXICAL` — whose BM25 ranking orders results but does not populate this field. Branch on `score is not None` before doing arithmetic with it.
```python theme={null}
results = await cognee.recall(
"Where is memory stored?",
query_type=SearchType.CHUNKS,
)
# Closest match first — lower distance is better.
ranked = sorted(
(r for r in results if getattr(r, "score", None) is not None),
key=lambda r: r.score,
)
```
For the full breakdown of session-hit shapes, graph-backed wrappers, and per-search-type payloads, see [Recall — What recall returns](/core-concepts/main-operations/recall#what-recall-returns).
## Examples
```python theme={null}
import cognee
results = await cognee.recall(
"What does Cognee do?",
datasets=["docs"],
top_k=5,
)
for result in results:
print(result)
```
<Warning>
With backend access control enabled, `datasets=["name"]` only resolves dataset names owned by the current user. If a dataset was created by Alice and shared with Bob, Bob should query it with `dataset_ids=[shared_id]`, not `datasets=["name"]`.
</Warning>
## Related
See also [SearchType](/python-api/search-type) and [search()](/python-api/search) when you need lower-level retrieval control.
# remember()
Source: https://docs.cognee.ai/python-api/remember
Store permanent or session memory with the v1.0 ingestion API
# cognee.remember()
```python theme={null}
async def remember(
data: Union[
BinaryIO,
list[BinaryIO],
str,
list[str],
DataItem,
list[DataItem],
MemoryEntry,
PresortReport,
dict,
],
dataset_name: str = "main_dataset",
*,
session_id: Optional[str] = None,
chunk_size: Optional[int] = None,
chunker: Optional[Any] = None,
custom_prompt: Optional[str] = None,
run_in_background: bool = False,
self_improvement: bool = True,
session_ids: Optional[List[str]] = None,
dry_run: Union[bool, Literal["presort"]] = False,
raise_on_error: bool = True,
**kwargs,
) -> Union[RememberResult, DryRunEstimate, PresortReport, dict]
```
## Description
`remember()` is the main ingestion entry point in Cognee v1.0.
* Without `session_id`, it stores permanent memory by running the ingestion pipeline for you.
* With `session_id`, it stores session memory in the cache for fast short-term retrieval.
* When `self_improvement=True`, Cognee also runs `improve()` to enrich the graph or bridge session content into permanent memory.
Because permanent memory is built through `cognify()`, the cognify feature flags apply to `remember()` too. Notably, setting `CONTRADICTION_DETECTION=true` makes every `remember()` call check the facts it just stored against the ones already in the graph and record each conflict as a `contradicts` edge — see [Contradiction detection](/python-api/cognify#contradiction-detection) — and `PROVENANCE_TRACKING=true` makes it append an audit-ledger entry for every document, chunk, entity, and relationship it produced, see [Provenance ledger](/python-api/cognify#provenance-ledger). Both off by default.
For the full behavior walkthrough, see [Remember](/core-concepts/main-operations/remember).
## Parameters
<ParamField type="Union[BinaryIO, list[BinaryIO], str, list[str], DataItem, list[DataItem], MemoryEntry, PresortReport, dict]">
Content to store. Supports text, file paths, URLs, file-like objects, `DataItem` values, lists of supported inputs, and typed session-memory entries. A GitHub/GitLab repository URL is recognised by shape, shallow-cloned, and indexed as a code graph rather than fetched as a web page — see [`add()`](/python-api/add#code-repository-urls) and the [Code Graph guide](/guides/code-graph). A `PresortReport` (or a path ending in `.presort.json`) is recognised as a presort report and applied instead of ingested — see [Folder presort](#folder-presort).
</ParamField>
<ParamField type="str">
Target dataset for permanent memory or for session-to-graph bridging.
</ParamField>
<ParamField type="Optional[str]">
Enables session-memory mode. When set, content is written to the session cache instead of going straight into the permanent graph.
</ParamField>
<ParamField type="Optional[int]">
Maximum chunk size for permanent ingestion. When omitted, Cognee uses its default chunking behavior.
</ParamField>
<ParamField type="Optional[Any]">
Custom chunking strategy for permanent ingestion.
</ParamField>
<ParamField type="Optional[str]">
Overrides the prompt used during graph extraction.
</ParamField>
<ParamField type="bool">
Starts the work asynchronously and returns a `RememberResult` you can await later.
</ParamField>
<ParamField type="bool">
When enabled, runs `improve()` automatically after storage to enrich the graph or bridge session content.
</ParamField>
<ParamField type="Optional[List[str]]">
Session IDs to sync newly enriched graph knowledge back into during the improvement pass.
</ParamField>
<ParamField type="Union[bool, Literal['presort']]">
When `True`, return a `DryRunEstimate` of LLM token usage and rough cost instead of ingesting data. No LLM calls are made, no data is ingested, and no graph is written. See [Dry-run cost estimation](#dry-run-cost-estimation).
When `"presort"`, treat `data` as a folder path and return a `PresortReport` instead of ingesting. This is a separate feature that happens to share the parameter — see [Folder presort](#folder-presort).
</ParamField>
<ParamField type="bool">
Forwarded to the internal `cognify()` step. When true, a blocking `remember()` whose graph build errored raises `CognifyFailedError` instead of returning a `RememberResult` with `status="errored"`. Set `False` to keep the previous behavior. See [Failed runs](/python-api/cognify#failed-runs).
</ParamField>
## Dry-run cost estimation
Pass `dry_run=True` to preview the LLM token usage and rough USD cost of a permanent `remember()` run **without ingesting data, making LLM calls, or writing the graph**:
```python theme={null}
import cognee
estimate = await cognee.remember("Einstein was born in Ulm.", dry_run=True)
print(estimate) # human-readable summary table
print(estimate.estimated_cost_usd)
print(estimate.to_dict()) # JSON-serializable dict
```
The call returns a `DryRunEstimate` with a stage-level breakdown (`structured_graph_extraction` and `chunk_summarization`). Its `operation` field is `"remember"`; the full field reference is documented under [`cognify()` → Dry-run cost estimation](/python-api/cognify#dry-run-cost-estimation).
**Supported inputs:** raw text, local text files, and `file://` URIs. Dataset resolution for the estimate is read-only.
The estimator mirrors real ingestion routing, so a bare path string is only read from disk when that file exists. An absolute-looking string that does not exist (for example `"/remember to call the dentist"`) is priced as raw text instead of failing, matching what a real run would ingest and bill.
Local path inputs are also subject to [`ACCEPT_LOCAL_FILE_PATH`](/setup-configuration/security#local-file-system-access), and the estimator detects absolute paths the same way real ingestion does — including Windows drive-letter paths such as `C:\notes.txt`. When the flag is disabled, a `file://` URI or a string pointing at an existing absolute local file raises rather than being estimated; an existing *relative* path, and any string that is not an existing file, are still priced as raw text.
**Rejected inputs** (raise a `ValueError` rather than being silently mis-estimated):
* `session_id` (session memory) — dry run only supports the permanent add+cognify path
* Typed `MemoryEntry` values and `MemorySource` imports
* Any `content_type` override, including `content_type="skills"` and `content_type="code"`
* Remote (`serve()`) mode — call `cognee.disconnect()` to estimate locally
* Remote URLs (`http`/`https`/`s3`), directories, and binary formats (PDF, images, audio, Office documents) that a real run would fetch, walk, or transcribe
<Note>
The estimate excludes the extra LLM calls that `improve()` makes when `self_improvement=True` (the default), as well as embedding costs.
</Note>
## Folder presort
Presort is a two-phase pre-ingestion pass for messy folders. It shares the `dry_run` parameter with the cost estimator above, but is an unrelated feature: nothing is priced, and the folder is inspected rather than ingested. For a runnable walkthrough of both phases, see the [Folder Presort guide](/guides/presort-downloads).
```python theme={null}
import os
import cognee
# Presort only reads inside its permitted roots, and ~/Downloads is not one of
# them by default — name it explicitly before scanning.
os.environ["COGNEE_ALLOWED_LOCAL_FILE_ROOTS"] = os.path.expanduser("~/Downloads")
# Phase 1 — analyze. Reads the folder, never writes to it.
report = await cognee.remember("~/Downloads", dry_run="presort")
print(report.summary())
# Review and adjust the apply decisions on the report itself
report.exclude_pii = True
report.apply_groups = [group.name for group in report.groups if group.kind != "code_project"]
# Phase 2 — apply. One dataset per proposed group.
results = await cognee.remember(report) # {dataset_name: RememberResult}
```
Pass `auto_apply=True` to do both in one call. The report is still produced and persisted first, and the ingest outcomes ride back on `report.apply_results` (`{dataset_name: RememberResult}`):
```python theme={null}
report = await cognee.remember("~/Downloads", dry_run="presort", auto_apply=True)
print(report.apply_results)
```
The analyze phase is **deterministic by default** — no LLM or embedding configuration is required. Pass `use_llm=True` to add LLM content classification, deeper PII detection, and semantic grouping.
### The report
`remember(folder, dry_run="presort")` returns a `PresortReport` (`cognee.tasks.presort.models`). It is a Pydantic model, so `to_dict()`, `to_json()`, `save(path)`, and `PresortReport.from_json(source)` are all available, and `summary()` returns the counts the CLI prints.
| Field | Contents |
| ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `files` | One `FileRecord` per scanned file: path, extension, MIME type, size, `family` bucket, `content_hash`, `cognee_status` (`new` / `staged` / `cognified` / `unknown`), and `known_in_datasets` |
| `junk` | Files skipped by the scan, each with a `reason` |
| `duplicates` | `DuplicateCluster` entries grouping identical content by hash. Paths are ordered — the first is the copy apply keeps |
| `versions` | `VersionCandidate` entries (`report_v2.pdf` and friends), ordered oldest → newest by modification time |
| `pii` | `PiiFinding` entries with `category`, `severity`, and `source` (`filename` / `content` / `llm`). Samples are always redacted; raw matches are never stored in the report |
| `groups` | `ProposedGroup` entries — the proposed `dataset_name`, `kind` (`code_project` / `folder` / `extension_family` / `semantic`), and member file paths |
| `relationships` | The generic relation view written to the graph by `apply_graph=True`, keyed by the relation names on the relationship spec |
| `skip_duplicates`, `exclude_pii`, `apply_groups` | The apply decisions, settable on the report before handing it back |
| `report_path` | Where the report was persisted, if it was |
| `warnings` | Degradations and skipped checks raised during the scan |
Unless `SYSTEM_ROOT_DIRECTORY` is unset or on S3, the report is also saved automatically to `<SYSTEM_ROOT_DIRECTORY>/presort/<scan_id>.presort.json`, so the analyze result survives even when apply fails.
### Applying a report
`remember()` recognises a report passed as `data` in three shapes: a `PresortReport` object, a dict carrying the `presort_report: true` marker, or a path ending in `.presort.json`. Ordinary `.json` ingestion is unaffected — only that exact suffix is treated as a report.
Apply ingests each selected group into its proposed dataset through the normal add → cognify (→ improve) chain with `incremental_loading=True`, so re-applying is idempotent: content the report marked `cognified` is skipped by the pipeline. Each group's items are tagged with the node set `["presort", "<group name>"]`. It returns `{dataset_name: RememberResult}`, not a single `RememberResult`; with `apply_graph=True` the relationship graph's outcome rides along under the extra key `presort_graph`.
### Presort options
Analyze-phase keywords (`dry_run="presort"`):
| Option | Type | Default | What it does |
| ------------------------ | --------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `include_subdirectories` | `bool` | `True` | Walk subdirectories, not just the top level |
| `use_llm` | `bool` | `False` | Add LLM content classification, deeper PII detection, and semantic grouping |
| `detect_pii` | `bool` | `True` | Run personal-data detection |
| `check_existing` | `bool` | `True` | Check which files Cognee already knows and whether they were cognified. When `False`, every `cognee_status` is `unknown` |
| `relationship_spec` | `dict` or `GraphSchemaSpec` | built-in spec | Custom relationship spec, written in the [JSON graph-model DSL](/guides/graph-model-from-json) |
| `dataset_prefix` | `str` | `""` | Prefix for proposed dataset names |
| `max_sample_bytes` | `int` | `65536` | Bytes of each file sampled for content-based detection |
Apply-phase keywords (`remember(report)`, or alongside `auto_apply=True`). When omitted, the value stored on the report is used:
| Option | Type | Default | What it does |
| ----------------- | ----------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `auto_apply` | `bool` | `False` | Apply the report immediately in the same `dry_run="presort"` call |
| `apply_groups` | `list[str]` | all groups | Ingest only these groups, matched by group name or dataset name. Raises `ValueError` when none of the names match a proposed group; names that miss alongside a name that matches are ignored |
| `skip_duplicates` | `bool` | `True` | Ingest one copy per duplicate cluster |
| `exclude_pii` | `bool` | `False` | Skip files with potential personal data |
| `node_set_extra` | `list[str]` | — | Extra node-set tags on top of `["presort", "<group name>"]` |
| `apply_graph` | `bool` | `False` | Also write the report's relationship graph (files, groups, duplicates, PII tags) into its own dataset |
| `graph_dataset` | `str` | `<folder>_presort_graph` | Dataset for `apply_graph` |
### Without a configured LLM
Presort degrades instead of failing. The deterministic scan always runs; `use_llm` is downgraded to the deterministic pass; apply stages files with `add()` only, leaving cognify/improve for later; and `apply_graph` is skipped because writing the relationship graph needs embeddings. Each degradation is recorded as a warning on the report and logged.
### Automatic presort
Set `PRESORT_FOLDERS_ENABLED=true` to presort plain folders automatically, with `auto_apply` on by default. The environment flag defaults to `false`, so **ordinary `remember(folder)` calls are unchanged unless you opt in.** Even when enabled, automatic presort applies only to a local directory path targeting `main_dataset` with no `session_id`, no `dataset_id`, and no `content_type`. Code-project directories keep the code-graph route, and remote (`serve()`) mode is excluded.
### Restrictions
Presort scans the local filesystem, and is bounded to the [permitted presort roots](/setup-configuration/security#presort-scan-roots). Scanning a folder outside them raises `ValueError: Path is outside the allowed local file roots. Add the folder to COGNEE_ALLOWED_LOCAL_FILE_ROOTS (os.pathsep-separated list) to allow it, or use the CLI's --allow-root flag.`; reading a saved report from outside them raises the shorter `ValueError: Local file path is outside allowed roots.`
`ValueError` is also raised for:
* `dry_run="presort"` with `session_id` or any `content_type`
* `dry_run="presort"` while connected to a remote instance via `serve()` — call `cognee.disconnect()` first
* more than one folder path in a single presort call, or a file path instead of a folder
* `dry_run` or `session_id` combined with applying a report
## Additional keyword options
These power-user options are forwarded to the underlying ingestion and graph-building steps.
| Option | Type | What it does |
| --------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `graph_model` | `Any` | Overrides the graph schema/model used during graph building. Defaults to `KnowledgeGraph`; pass a `DataPoint` subclass to constrain extraction to your own fields and relationships. See [Custom Graph Model](/guides/custom-graph-model). |
| `extractor` | `Literal["llm", "gliner"]` | Which implementation fills the extract-and-summarize step of the internal `cognify()` call. `"gliner"` builds the graph and the chunk summaries with a local GLiNER2 model instead of the LLM; omitting it falls back to the `GRAPH_EXTRACTOR` setting, which defaults to `"llm"`. Rejected with `ValueError` when combined with `session_id` or while connected to a remote instance via `serve()` — both check only whether you passed the argument, so a `GRAPH_EXTRACTOR` setting is ignored rather than rejected on those paths. `dry_run=True` rejects the resolved extractor, so the setting raises there too. The combinations `cognify()` itself refuses — a custom `graph_model`, `temporal_cognify=True` — apply to `remember()` as well. See [LLM-free extraction with GLiNER](/python-api/cognify#llm-free-extraction-with-gliner). |
| `node_set` | `List[str]` | Tags ingested content with one or more node sets. |
| `dataset_id` | `UUID` | Targets a specific existing dataset by UUID instead of resolving only by name. |
| `preferred_loaders` | `list` | Chooses preferred loaders for source files. |
| `importance_weight` | `float` | Stores a retrieval-ranking weight on ingested data records. |
| `incremental_loading` | `bool` | Reuses existing dataset state and processes only new or changed content when supported. |
| `data_per_batch` | `int` | Controls ingestion batching. |
| `chunks_per_batch` | `int` | Controls chunk-processing batching during graph building. |
| `config` | `Config` | Overrides the full Cognee config for the graph-building step. |
| `temporal_cognify` | `bool` | Enables temporal-aware graph building during the internal `cognify()` step. |
| `user` | `object` | Runs the operation under a specific user context. |
| `vector_db_config` | `dict` | Overrides vector database configuration for this call. |
| `graph_db_config` | `dict` | Overrides graph database configuration for this call. |
| `llm_config` | `LLMConfig` | LLM settings to install into the current async context and forward to both the ingestion and graph-building steps. Uses the active context config or global LLM config when omitted. Import from `cognee.infrastructure.llm.config`. |
| `embedding_config` | `EmbeddingConfig` | Embedding settings to install into the current async context and forward to both the ingestion and graph-building steps. Uses the active context config or global embedding config when omitted. Import from `cognee.infrastructure.databases.vector.embeddings.config`. |
## Return value
`remember()` returns:
* `RememberResult` for normal ingestion runs. You can inspect fields like `status`, `dataset_name`, `session_ids`, `elapsed_seconds`, and `raw_result`, or `await` the result when background mode is enabled.
* `DryRunEstimate` when `dry_run=True`, with aggregate token/cost totals plus the per-stage breakdown described above.
* `PresortReport` when `dry_run="presort"`, described under [Folder presort](#folder-presort).
* `dict` of `{dataset_name: RememberResult}` when `data` is a presort report.
A blocking permanent-memory run whose graph build fails does **not** return a `RememberResult` with `status="errored"` — it raises `CognifyFailedError`, carrying the dataset name and the root cause (`error_class`, `error_message`). Pass `raise_on_error=False` to get the errored result object back instead:
```python theme={null}
import cognee
from cognee.modules.pipelines.exceptions import CognifyFailedError
try:
result = await cognee.remember("Einstein was born in Ulm.")
except CognifyFailedError as error:
print(error.error_class, error.error_message)
```
Background runs (`run_in_background=True`) still surface the failure on the result object rather than raising, since there is no caller left to catch the exception: `await` the result and read `status == "errored"` along with `pipeline_run_id` and `raw_result`. See [Failed runs](/python-api/cognify#failed-runs) for the full behavior of the underlying `cognify()` step.
## Examples
```python theme={null}
import cognee
# Permanent memory
result = await cognee.remember(
"Cognee turns documents into AI memory.",
dataset_name="docs",
)
print(result.status)
# Session memory
await cognee.remember(
"The customer prefers weekly updates.",
session_id="sales_chat_1",
)
```
## Related
See also [add()](/python-api/add) and [cognify()](/python-api/cognify) if you need direct control over the legacy pipeline steps.
# report()
Source: https://docs.cognee.ai/python-api/report
Generate a Graph Insight Report from the knowledge graph
# cognee.report()
```python theme={null}
async def report(
datasets: Optional[Union[str, List[str]]] = "main_dataset",
output_path: Optional[str] = "graph_report.md",
top_n: int = 10,
user: Optional[User] = None,
) -> str
```
## Description
Generate a **Graph Insight Report** — a Markdown overview of what a knowledge graph
actually contains: its most connected nodes, links that cross node-set boundaries,
a breakdown of how its edges got there, and a handful of suggested questions for
exploring the graph further.
The report is **read-only**. It is computed from the graph engine's `get_graph_data()`
and makes no schema or storage changes, so it is safe to run against a production
graph. Run it after [`remember()`](/python-api/remember) / [`cognify()`](/python-api/cognify)
to sanity-check a freshly built graph.
The full Markdown is returned as a string **and** written to `output_path`, unless
you pass `output_path=None` to skip the file write.
## Parameters
<ParamField type="Optional[Union[str, List[str]]]">Dataset name(s) to analyse. A single string is treated as a one-element list.</ParamField>
<ParamField type="Optional[str]">File path for the generated Markdown report. Pass `None` to skip writing the file and only return the string.</ParamField>
<ParamField type="int">How many hub nodes and cross-set connections to surface.</ParamField>
<ParamField type="Optional[User]">User context for dataset access. Falls back to the default user.</ParamField>
<Note>
Only **one** graph is analysed per call: `datasets` is resolved to the datasets you
have read access to, and the **first** of those determines which graph is read.
Passing several dataset names does not merge them into a combined report — call
`report()` once per dataset instead.
</Note>
## Returns
`str` — the full Markdown report.
## What the report contains
<AccordionGroup>
<Accordion title="🏆 Hub Nodes">
The top `top_n` nodes ranked by a combined score of min–max-normalised **degree**
plus **PageRank**. Ranking prefers the knowledge layer (`Entity` and `EntityType`
nodes) and falls back to all content nodes when the graph has neither.
The analysis graph excludes `NodeSet` container nodes and `belongs_to_set`
membership edges, so organisational scaffolding never wins the hub ranking.
Each entry lists the node's name, its node sets, its degree, and its PageRank score.
A node with no node-set membership shows `—` in the set column — normal for
graphs built without [node sets](/core-concepts/further-concepts/node-sets).
</Accordion>
<Accordion title="🔗 Surprising Cross-Set Connections">
Entity-to-entity edges whose two endpoints have **different node-set membership** —
the places where your sources actually connect to each other. Fully disjoint pairs
(no shared set at all) rank first, then pairs are ordered by the combined PageRank
of their endpoints.
Membership is read from the `belongs_to_set` edges pointing at `NodeSet` nodes,
plus the comma-separated `source_node_set` property that chunks and documents carry.
If nothing qualifies, the section says so — add data under multiple
[node sets](/core-concepts/further-concepts/node-sets) to populate it.
</Accordion>
<Accordion title="🏷️ Edge Provenance">
A count and percentage split of how edges entered the graph:
| Tag | Meaning |
| ----------- | ---------------------------------------------------------------------------------------------------- |
| `EXTRACTED` | Both endpoints are `Entity` nodes — a relationship the LLM extracted from your content. |
| `DERIVED` | Everything else — structural scaffolding such as chunk→entity, entity→type, or chunk→document edges. |
<Warning>
These tags are **derived at report time from the endpoint node types**, not read
from the graph. Cognee stores no confidence score or provenance tag on edges, so
treat the split as a structural summary rather than a per-edge quality signal.
</Warning>
</Accordion>
<Accordion title="💡 Suggested Questions">
Four to five short questions for exploring the graph, generated by a single LLM
call over the first 2000 characters of the report body. This is the only LLM call
the report makes — every other section is computed deterministically.
Content questions ("How are X and Y related?") work directly with
[`recall()`](/python-api/recall) / `search()`. The LLM sometimes suggests
graph-**structure** questions instead — shortest paths, degree thresholds,
per-node edge counts — which the completion-style search types cannot compute;
use `SearchType.CYPHER` or `SearchType.NATURAL_LANGUAGE` for those. Questions
about `EXTRACTED`/`DERIVED` edges cannot be answered by any search type: those
tags exist only inside the report (see Edge Provenance above), not in the graph.
</Accordion>
</AccordionGroup>
## Cost and fallbacks
* **One LLM call**, for the Suggested Questions section only. If it fails (no API key,
rate limit, provider error), the failure is logged as a warning and the section
degrades to a single generic question — `report()` does not raise.
* **PageRank** is computed with `networkx` (sparse). If it is unavailable or fails
numerically, a warning is logged and hubs are ranked by degree alone, with PageRank
reported as `0.0000`.
* **Empty graph**: if the graph has no nodes, the report body is
`_Graph is empty — run cognify first._` instead of the four sections.
## Examples
```python theme={null}
import cognee
await cognee.remember("docs/handbook.pdf", dataset_name="onboarding")
# Write graph_report.md in the working directory and get the Markdown back
report_md = await cognee.report(datasets="onboarding")
print(report_md)
# Surface more hubs and connections, and write somewhere specific
await cognee.report(
datasets="onboarding",
output_path="reports/onboarding-graph.md",
top_n=25,
)
# Return the Markdown only, without touching the filesystem
report_md = await cognee.report(datasets="onboarding", output_path=None)
```
From the terminal:
```bash theme={null}
cognee-cli report --datasets onboarding --top-n 25 --output reports/onboarding-graph.md
```
### Through `search()`
The same retriever is registered as [`SearchType.GRAPH_REPORT`](/python-api/search-type),
which is useful when you want the report to flow through the normal search plumbing:
```python theme={null}
from cognee import SearchType
results = await cognee.search(
"", # ignored — the report always covers the whole graph
query_type=SearchType.GRAPH_REPORT,
top_k=25, # becomes the retriever's top_n
)
```
The query text is ignored and the result is a single-element list holding the same
Markdown string. Prefer `cognee.report()` when you want the file written for you.
## See also
* [Cognee CLI → Generate a Graph Insight Report](/cognee-cli/overview#generate-a-graph-insight-report) — the same operation from the terminal
* [SearchType](/python-api/search-type) — invoking the report through `search()`
* [Node Sets](/core-concepts/further-concepts/node-sets) — what the cross-set section groups by
# run_migrations
Source: https://docs.cognee.ai/python-api/run-migrations
Apply pending relational and vector database schema migrations
# cognee.run\_migrations
Applies all pending database schema migrations before the rest of your application starts.
```python theme={null}
await cognee.run_migrations()
```
<Note>
In releases before Cognee 1.5.0 this function was named `run_startup_migrations`. That name is gone — update calls to `cognee.run_migrations()` when you upgrade.
</Note>
It runs two steps in sequence:
1. **Relational schema** — executes `alembic upgrade head` against your configured relational database (SQLite by default, or Postgres).
* Cognee first checks whether the database already holds its schema. A database counts as **fresh** only in two cases: it is a local SQLite database whose file does not exist yet (a filesystem check made before connecting, so the probe never creates the file), or it is a database Cognee could inspect that has neither a `users` table nor an `alembic_version` table. A fresh database gets its schema built by running the entire migration chain — the initial revision carries the base schema, and `alembic_version` only ever records revisions that actually executed, so nothing is stamped. Anything else is migrated normally, so a pre-Alembic legacy database is upgraded rather than mistaken for a fresh one.
* A database Cognee **cannot inspect** — unreachable, still starting up, or rejecting the connection — is not treated as fresh. The inspection error propagates and the call fails; see [Errors](#errors).
* **Applying pending revisions to an existing database is this call's job alone.** The write paths also make sure the relational database exists: `setup()` before `add()`, and every pipeline run, call the same table-creation step. That step only builds a *fresh* database (the same check as above); against a database that already holds the schema it does a single table inspection and returns without touching Alembic, so ordinary requests no longer replay (or log) the migration chain on every call. Pending revisions on an existing database are applied only by `run_migrations()` — which the first `add()`, `cognify()`, or `remember()` in a process triggers once, unless [automatic migrations are disabled](#disabling-automatic-migrations) — or by `cognee-cli upgrade`.
* Revisions `c5d7e9f1a3b5` and `d6e8f0a2b4c6` complete the dataset-scoping of the `data` table — see [Dataset-scoping upgrade](#dataset-scoping-upgrade) below.
* This step also heals the SQL session-cache table on existing deployments. Revision `c3d5e7f9a1b2` deletes all but the newest row per `(user_id, session_id, entry_id)` in `cache_session_context` and creates the `uq_cache_session_context_entry` unique index that the cache adapter's upsert targets. It covers both the database Alembic is connected to and the standalone SQLite `cache.db` that `CACHE_BACKEND=sqlite` (the default) keeps next to the relational database. Cache tables are created on init rather than managed by Alembic, so fresh databases already get the index from the table definition and only pre-existing tables are touched. Because the migration mutates data and builds a unique index, it can take time and hold locks on a large `cache_session_context` table — back up the database and run it in a maintenance window.
2. **Vector schema** — runs the vector adapter's `run_migrations` method for every database that needs it:
* **Single-user mode** (`ENABLE_BACKEND_ACCESS_CONTROL=False`): migrates the single default vector engine.
* **Multi-user mode** (`ENABLE_BACKEND_ACCESS_CONTROL=True`, the default): iterates over every dataset database and migrates each one individually. A failure for one dataset is logged and skipped; the remaining datasets continue to migrate.
* If the active vector engine has no `run_migrations` method, Cognee logs a warning and skips that engine.
* If the `dataset_database` table does not exist yet (a fresh database), the vector migration step is skipped with a warning instead of raising. This is handled on both SQLite (`OperationalError`, "no such table") and PostgreSQL/pgvector (`ProgrammingError` / `UndefinedTableError`).
<Note>
**Migrations that re-embed batch their embedding requests.** Some data migrations re-embed rows through `cognify`'s indexing path; those embeddings are sent in batches, so a large migration no longer fails on provider request-size or rate limits. Batching follows the same [`EMBEDDING_BATCH_SIZE`](/setup-configuration/embedding-providers#batch-size) and `EMBEDDING_MAX_CONCURRENT_DATA_POINTS` settings as normal indexing; there is no migration-specific setting to tune.
</Note>
## Dataset-scoping upgrade
Cognee 1.5.0 makes `Data` rows dataset-scoped: a row belongs to exactly one dataset, and the same content added to two datasets is two independent rows (see [deduplication](/core-concepts/main-operations/legacy-operations/add)). Databases created by an earlier release stored one row shared across datasets through a `dataset_data` membership table, so upgrading needs a data migration, not just a schema change.
Two Alembic revisions do the relational half:
| Revision | What it does |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `c5d7e9f1a3b5` | Adds the `data.dataset_id` column and the `(dataset_id, owner_id, content_hash)` lookup index that backs add-time deduplication. Existing rows keep `dataset_id = NULL` until the next revision. |
| `d6e8f0a2b4c6` | Adds `data.legacy_id`, backfills every legacy row from its memberships, then drops `dataset_data`. |
The backfill splits rows per dataset:
* **One membership** (the overwhelming majority) — the row keeps its **original id** and is stamped with that dataset, so data ids you stored externally stay valid.
* **Several memberships** — the **oldest** membership keeps the original id; every other dataset gets a fresh row with a new id, all columns copied, and `legacy_id` recording the pre-split id. Relational ledger rows for those datasets are repointed to the new ids.
* **No memberships** — unreachable orphans, left untouched.
Because `legacy_id` is preserved, every id ever issued keeps resolving within its dataset: Cognee matches the exact id first, then falls back to `legacy_id`.
The graph and vector stores are then brought in line by the `rekey_fork_document_ids` migration in Cognee's data-migration chain, which runs after the relational step. For each split ("fork") document it re-keys the graph document node to the canonical relational id, updates the ledger's node/edge references, refreshes the `document_id` property on the document's chunk nodes, and re-upserts the chunks' vector index rows — in batched embedding requests, as described above — so their `document_id` payload cites the canonical id. Chunk point ids are unchanged. Fork rows are rare — same user, identical content, several datasets, all before the upgrade — so on most deployments this is one indexed relational query and no per-document work.
When a fork *does* exist on a large graph, the graph half of that re-key is real work. Its edge-identity and node-id lookups run as chunked index seeks and its provenance restore is attached in batches grouped by pipeline run, so the re-key completes within the worker subprocess deadline instead of timing out — but on a graph of \~100k nodes and \~300k edges it can still run for tens of minutes. On the default embedded Ladybug backend (this does not apply to Neo4j or Postgres graph stores, which have no such cap), give the store headroom under its size cap — the `kuzu_max_db_size` setting (`KUZU_MAX_DB_SIZE`) — before starting, and let the run finish: killing a worker mid-checkpoint can leave `.lbug.shadow` / `.wal.checkpoint` recovery files behind that block the next open, and repeated interrupted runs can exhaust the cap even while the database is small on disk.
Much of that graph work is set-based rather than row-by-row. On Ladybug/Kuzu, the provenance move is one `replace()` statement over the nodes and one over the edges still carrying the pre-fork key, so the batched restore described above handles only the residue: artifacts that already carry both keys are deliberately left to that generic attach-then-remove sweep, which dedupes them, so the migration still converges on re-run. Every other graph backend — and any engine that rejects those statements — takes the unchanged generic path for the whole move, and the migrated result is identical either way. The survivor-edge repair in the sibling `namespace_entity_type_node_ids` migration is trimmed the same way on any backend that can list its edges in one scan: it asks which at-risk edges were actually dropped and re-asserts only those, and blind-upserts every at-risk edge only where that scan isn't answerable. On a large fork subgraph these shortcuts make the migration substantially faster.
Both migrations are reversible, but the rollback order is the inverse of the upgrade order: run the `rekey_fork_document_ids` downgrade **before** the `d6e8f0a2b4c6` downgrade. The reverse map is built from `legacy_id`, so dropping that column first strands fork graphs on canonical ids with no way back. Downgrading also loses fork lineage permanently — the old schema has nowhere to keep it — so a later re-upgrade re-splits shared rows with fresh ids.
<Warning>
The backfill is **not an online migration**: an old-version replica still writes shared rows and reads `dataset_data`, both of which the migration retires. Stop all replicas, upgrade, then start them again. A concurrent second run self-cancels — both transactions contain the `dataset_data` drop, and the loser rolls back whole — and an interrupted run rolls back cleanly and simply reruns.
</Warning>
## Entrypoints
All migration functions live in the `cognee.run_migrations` module. `run_migrations` is also re-exported at the top level as `cognee.run_migrations`, so it is the recommended entrypoint for most applications.
| Function | Import | Migrates |
| --------------------------- | ------------------------------------------------------------------------------- | --------------------------------------------------------------- |
| `run_migrations` | `cognee.run_migrations` (or `from cognee.run_migrations import run_migrations`) | Relational schema **and** the graph/vector stores (recommended) |
| `run_relational_migrations` | `from cognee.run_migrations import run_relational_migrations` | Relational schema only (`alembic upgrade head`) |
## When to call it
| Scenario | Why |
| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| After upgrading the `cognee` package | New versions may add tables or columns to the relational schema. |
| Upgrading to Cognee 1.5.0 from an earlier release | Required: the [dataset-scoping backfill](#dataset-scoping-upgrade) rewrites existing `Data` rows and drops the `dataset_data` table. Stop all replicas first. |
| First run against an external database | The automatic startup run migrates Postgres and other external databases just like SQLite, but an explicit run lets you build the schema before the app rolls out. |
| Kubernetes / Docker init containers | Run migrations once before starting the main application pods. |
| Switching to a new relational DB provider | The new database starts empty and needs all migrations applied. |
| Running migrations on your own schedule | Turn off the automatic runs with [`ENABLE_AUTO_MIGRATIONS=false`](#disabling-automatic-migrations) and migrate explicitly instead. |
<Note>
For the default local setup (SQLite + LanceDB), Cognee handles migrations automatically when the API server starts. You only need to call `run_migrations()` explicitly in server deployments or CI pipelines where you manage database lifecycle yourself.
</Note>
## Disabling automatic migrations
`run_migrations()` is called for you in a few places: the FastAPI server's startup lifespan, the first write call in a process (`add()`, `cognify()`, `remember()`, `improve()`, or `memify()`), and — in the Docker image — the container entrypoint before the server binds its port.
Set `ENABLE_AUTO_MIGRATIONS=false` to disable **all** of those automatic runs:
```bash theme={null}
ENABLE_AUTO_MIGRATIONS=false
```
The variable defaults to `true`. It is disabled by `false`, `0`, or `no` (case-insensitive); any other value leaves automatic migrations on. When disabled, `run_migrations()` logs that it was skipped and returns an empty list without touching any database.
<Warning>
Disabling automatic migrations does not migrate anything on its own — Cognee will run against whatever schema already exists, and nothing else will move it forward: `add()`, `cognify()`, and pipeline runs build the schema only on a fresh database and never apply pending revisions to an existing one. Pair it with an explicit `cognee-cli upgrade` step before rolling out a new version: the CLI deliberately **ignores** this flag and runs the same locked relational + graph/vector sequence regardless. Note that the [init container](#kubernetes-init-container) below calls `run_migrations()`, so it is gated by this flag too — use `cognee-cli upgrade` as its command, or leave the flag unset in the init container's own environment.
</Warning>
For the container-startup side of this flag, see [Docker → Database Migrations on Startup](/how-to-guides/cognee-sdk/deployment/docker#database-migrations-on-startup).
## Example
```python theme={null}
import asyncio
import cognee
async def main():
# Apply all pending schema migrations before starting
await cognee.run_migrations()
# Normal usage
await cognee.add("Hello, world!", dataset_name="demo")
await cognee.cognify()
asyncio.run(main())
```
### Kubernetes init container
Run migrations as a one-shot init container so the main pod only starts after the schema is ready:
```yaml theme={null}
initContainers:
- name: migrate
image: your-cognee-image:latest
command: ["python", "-c", "import asyncio, cognee; asyncio.run(cognee.run_migrations())"]
envFrom:
- secretRef:
name: cognee-env
```
## Concurrency
Every migration flow runs under a single cross-process lock, so a host performs **at most one migration of any kind at a time**. If several processes start at once — multiple workers of the same server, parallel SDK runs, or several init containers — only one acquires the lock and migrates; the others block until it finishes, then re-read the stored revision and skip work that is already done. Nothing runs migrations in parallel.
Because of this, startup can block (and time-to-ready can increase) while another process holds the lock and migrates. This is expected under the [Kubernetes init container](#kubernetes-init-container) and multi-worker scenarios above — the wait is the coordination working as intended, not a hang.
The lock backend depends on your relational database:
* **Postgres** — a session-scoped advisory lock, which also serializes migrations **across hosts**. Use Postgres metadata when multiple hosts may start and migrate at the same time.
* **SQLite** — an OS advisory file lock placed next to the database file. It serializes multiple **processes on a single host** (multi-worker servers, parallel SDK runs) but **not across hosts or over NFS**.
## Errors
| Error | Cause | Fix |
| --------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `FileNotFoundError` | `alembic.ini` or the migrations directory is missing from the installed package. | Reinstall `cognee` — the package may be corrupted or partially installed. |
| `MigrationError` | Alembic exited with a non-zero return code. | Check the error message logged at `ERROR` level; usually a DB connection problem or a SQL conflict. |
| A connection or inspection error from the relational database (`OperationalError`, `ConnectionRefusedError`, …) | Cognee could not read the table list to decide whether the schema already exists — the database is unreachable, still starting up, or rejecting the credentials. | Make the database reachable and call `run_migrations()` again; the failed run is not marked done, so the next call retries. Cognee deliberately neither retries internally nor assumes the database is fresh — a wrong "fresh" verdict on an unreachable database would try to build the schema from scratch instead of surfacing the connection problem as the error it is. Order startup so the database is ready first (an [init container](#kubernetes-init-container), or `depends_on: condition: service_healthy` in Compose). |
| `RuntimeError: The session cache lives in a separate database this migration cannot reach` | You use a SQL session-cache backend and `CACHE_DB_URL` points at a non-SQLite database other than the one Alembic connects to, so revision `c3d5e7f9a1b2` cannot dedupe and index `cache_session_context` there. | Run the `DELETE` and `CREATE UNIQUE INDEX` statements printed in the error message against that cache database, then re-run migrations. There is deliberately no fallback: the migration blocks the deploy rather than leaving a cache database the upsert would fail on. |
<Tip>
Set `LOG_LEVEL=DEBUG` to see the full Alembic output when diagnosing migration failures.
</Tip>
## Related
* [Relational Databases](/setup-configuration/relational-databases) — configure SQLite or Postgres
* [Deployment Overview](/how-to-guides/cognee-sdk/deployment/index) — how to structure Cognee in production
* [Kubernetes (Helm)](/how-to-guides/cognee-sdk/deployment/helm) — full Kubernetes deployment guide
# search()
Source: https://docs.cognee.ai/python-api/search
Reference for search(), which queries the knowledge graph with a chosen SearchType.
# cognee.search()
```python theme={null}
async def search(
query_text: str,
query_type: SearchType = SearchType.HYBRID_COMPLETION,
user: Optional[User] = None,
datasets: Optional[Union[list[str], str]] = None,
dataset_ids: Optional[Union[list[UUID], UUID]] = None,
system_prompt_path: str = 'answer_simple_question.txt',
system_prompt: Optional[str] = None,
top_k: int = 15,
node_type: Optional[Type] = NodeSet,
node_name: Optional[List[str]] = None,
node_name_filter_operator: str = "OR",
only_context: bool = False,
context_format: Union[ContextFormat, str] = ContextFormat.CONTEXT,
session_id: Optional[str] = None,
wide_search_top_k: Optional[int] = 100,
triplet_distance_penalty: Optional[float] = 6.5,
verbose: bool = False,
retriever_specific_config: Optional[dict] = None,
llm_config: Optional[LLMConfig] = None,
embedding_config: Optional[EmbeddingConfig] = None,
include_references: bool = False,
) -> List[SearchResult]
```
## Description
Search and query the knowledge graph for insights, information, and connections.
This is the final step in the Cognee workflow that retrieves information from the
processed knowledge graph. It supports multiple search modes optimized for different
use cases - from simple fact retrieval to complex reasoning and code analysis.
Search Prerequisites:
* **LLM\_API\_KEY**: Required for GRAPH\_COMPLETION and RAG\_COMPLETION search types
* **Data Added**: Must have data previously added via `cognee.add()`
* **Knowledge Graph Built**: Must have processed data via `cognee.cognify()`
* **Dataset Permissions**: User must have 'read' permission on target datasets
* **Vector Database**: Must be accessible for semantic search functionality
Search Types & Use Cases:
**HYBRID\_COMPLETION** (Default - Recommended):
Natural language Q\&A over document passages plus the entity
neighbourhoods around them, answered by an LLM.
Best for: Most questions, where both source text and graph structure help.
Returns: Conversational AI responses backed by passages and graph context.
**GRAPH\_COMPLETION**:
Natural language Q\&A using full graph context and LLM reasoning.
Best for: Complex questions, analysis, summaries, insights.
Returns: Conversational AI responses with graph-backed context.
**RAG\_COMPLETION**:
Traditional RAG using document chunks without graph structure.
Best for: Direct document retrieval, specific fact-finding.
Returns: LLM responses based on relevant text chunks.
**CHUNKS**:
Raw text segments that match the query semantically.
Best for: Finding specific passages, citations, exact content.
Returns: Ranked list of relevant text chunks with metadata.
**SUMMARIES**:
Pre-generated hierarchical summaries of content.
Best for: Quick overviews, document abstracts, topic summaries.
Returns: Multi-level summaries from detailed to high-level.
**CODING\_RULES**:
Code-specific search with syntax and semantic understanding.
Best for: Finding functions, classes, implementation patterns.
Returns: Structured code information with context and relationships.
**CYPHER**:
Direct graph database queries using Cypher syntax.
Best for: Advanced users, specific graph traversals, debugging.
Returns: Raw graph query results.
**FEELING\_LUCKY**:
Intelligently selects and runs the most appropriate search type.
Best for: General-purpose queries or when you're unsure which search type is best.
Returns: The results from the automatically selected search type.
**CHUNKS\_LEXICAL**:
Token-based lexical chunk search (BM25-style lexical ranking). Best for: exact-term matching, stopword-aware lookups.
Returns: Ranked text chunks (optionally with scores).
Args:
query\_text: Your question or search query in natural language.
Examples:
* "What are the main themes in this research?"
* "How do these concepts relate to each other?"
* "Find information about machine learning algorithms"
* "What functions handle user authentication?"
query\_type: SearchType enum specifying the search mode.
Defaults to HYBRID\_COMPLETION for conversational AI responses.
user: User context for data access permissions. Uses default if None.
datasets: Dataset name(s) to search within. Searches all accessible if None.
Name lookup is limited to datasets owned by the searching user.
For shared datasets the user can access but does not own, use dataset\_ids instead.
* Single dataset: "research\_papers"
* Multiple datasets: \["docs", "reports", "analysis"]
* None: Search across all user datasets
dataset\_ids: Alternative to datasets - use specific UUID identifiers.
Required when searching a dataset the user can access but does not own.
system\_prompt\_path: Custom system prompt file for LLM-based search types.
Defaults to "answer\_simple\_question.txt".
top\_k: Maximum number of results to return (1-N)
Higher values provide more comprehensive but potentially noisy results.
node\_type: Filter results to specific entity types (for advanced filtering).
node\_name: Filter results to specific named entities (for targeted search).
session\_id: Optional session identifier for caching Q\&A interactions. When None,
resolves to the dataset-scoped default session: `default_session_<dataset_id>`
when the dataset is known, falling back to the global 'default\_session' when it
is not. Searches against different datasets therefore never share one session.
verbose: If True, returns detailed result information including graph representation (when possible).
retriever\_specific\_config: Optional dictionary of additional configuration parameters specific to the retriever being used.
include\_references: Defaults to False. When set to True, completion-style answers
(e.g. GRAPH\_COMPLETION, RAG\_COMPLETION) get a deterministic "Evidence:" block appended
to the answer text. The block is assembled in-process from the retrieved chunk payloads
or graph context — no extra LLM call is made — and is omitted silently when no usable
references are found. The return type and response schema are unchanged.
Returns:
list: Search results in format determined by query\_type:
**GRAPH\_COMPLETION/RAG\_COMPLETION**:
\[List of conversational AI response strings]
**CHUNKS**:
\[List of relevant text passages with source metadata]
**SUMMARIES**:
\[List of hierarchical summaries from general to specific]
**CODING\_RULES**:
\[List of structured code information with context]
**FEELING\_LUCKY**:
\[List of results in the format of the search type that is automatically selected]
Performance & Optimization:
* **GRAPH\_COMPLETION**: Slower but most intelligent, uses LLM + graph context
* **RAG\_COMPLETION**: Medium speed, uses LLM + document chunks (no graph traversal)
* **CHUNKS**: Fastest, pure vector similarity search without LLM
* **SUMMARIES**: Fast, returns pre-computed summaries
* **CODING\_RULES**: Medium speed, specialized for code understanding
* **FEELING\_LUCKY**: Variable speed, uses LLM + search type selection intelligently
* **top\_k**: Start with 15, increase for comprehensive analysis (max 100)
* **datasets**: Specify datasets to improve speed and relevance
Next Steps After Search:
* Use results for further analysis or application integration
* Combine different search types for comprehensive understanding
* Export insights for reporting or downstream processing
* Iterate with refined queries based on initial results
Environment Variables:
Required for LLM-based search types (GRAPH\_COMPLETION, RAG\_COMPLETION):
* LLM\_API\_KEY: API key for your LLM provider
Optional:
* LLM\_PROVIDER, LLM\_MODEL: Configure LLM for search responses
* VECTOR\_DB\_PROVIDER: Must match what was used during cognify
* GRAPH\_DATABASE\_PROVIDER: Must match what was used during cognify
## Parameters
<ParamField type="str">Natural language search query.</ParamField>
<ParamField type="SearchType">Type of search to perform.</ParamField>
<ParamField type="Optional[User]">User performing the search.</ParamField>
<ParamField type="Optional[Union[list[str], str]]">Dataset name(s) to search within. Dataset names are resolved only against datasets owned by the searching user.</ParamField>
<ParamField type="Optional[Union[list[UUID], UUID]]">Dataset UUID(s) to search within. Use these for shared datasets that the user can access but did not create.</ParamField>
<ParamField type="str">Path to a custom system prompt file.</ParamField>
<ParamField type="Optional[str]">Inline system prompt string (overrides system\_prompt\_path).</ParamField>
<ParamField type="int">Maximum number of results to return.</ParamField>
<ParamField type="Optional[Type]">Filter results to a specific [DataPoint](/core-concepts/building-blocks/datapoints) subclass type. Pass any class that inherits from `DataPoint`, including [custom classes](/guides/custom-data-models) you define in your own code. Defaults to `NodeSet` (the built-in group container). Internal Cognee types such as `ParagraphNode` are not part of the public API and cannot be used here.</ParamField>
<ParamField type="Optional[List[str]]">Names of the node sets to filter results to. Pass the same names used in `cognee.add(..., node_set=[...])`. Works with graph-completion search types (e.g. `GRAPH_COMPLETION`, `GRAPH_COMPLETION_COT`, `TEMPORAL`). See [NodeSets](/core-concepts/further-concepts/node-sets).</ParamField>
<ParamField type="str">Controls how multiple `node_name` values are combined. `"OR"` returns results connected to any of the specified node sets; `"AND"` returns results connected to all of them.</ParamField>
<ParamField type="bool">If true, return only the retrieved context without LLM completion.</ParamField>
<ParamField type="Union[ContextFormat, str]">Shape of the value returned when `only_context=True`; ignored otherwise. `"context"` (the default) returns the bare retrieval context, byte-identical to earlier releases. `"prompt"` returns the whole envelope a completion would be sent — see [The prompt envelope](#the-prompt-envelope) below. Any other value raises `InvalidContextFormatError` (a `CogneeValidationError`, HTTP 422).</ParamField>
<ParamField type="Optional[str]">Session ID for conversational context tracking.</ParamField>
<ParamField type="Optional[int]">Number of candidates for the wide search phase. When omitted, retrievers that use it fall back to their own default of `100`. Honored by the graph-completion family, `TEMPORAL`, `AGENTIC_COMPLETION`, and `RAG_COMPLETION`; any explicit value with the default `HYBRID_COMPLETION` raises `InvalidHybridSearchConfig` (a `CogneeValidationError`, HTTP 422) before retrieval starts — see [Search Basics — Advanced Parameters](/guides/search-basics#advanced-parameters).</ParamField>
<ParamField type="Optional[float]">Penalty factor for triplet distance in scoring. When omitted, graph retrievers fall back to their own default of `6.5`. Graph-only, and rejected on `HYBRID_COMPLETION` exactly like `wide_search_top_k`.</ParamField>
<ParamField type="bool">Include detailed retrieval metadata in results: `text_result`, `context_result`, and `objects_result`, plus three more keys when combined with `context_format="prompt"` (see [The prompt envelope](#the-prompt-envelope)).</ParamField>
<ParamField type="Optional[dict]">Additional configuration for the selected retriever.</ParamField>
<ParamField type="Optional[dict]">Structured operation and arguments for `SearchType.CODE` — for example `{"operation": "impact_analysis", "name": "UserService", "max_depth": 3}`. The `operation` key selects one of `query_facts`, `explore`, `traverse`, `find_path`, `impact_analysis`, `insights`, `architecture`, or `delta`; omitting it runs `explore` seeded with `query_text`. Add `"diagram": "mermaid"` (or `"dot"`) to any operation to get diagram source under `result["diagram"]`. Passing it with any other `query_type` raises `InvalidCodeSearchConfig`. Every operation's arguments and the fact kinds they filter on are listed under [SearchType — CODE](/python-api/search-type#per-search-type-parameters); the [Code Graph guide](/guides/code-graph) shows how to build the graph this searches.</ParamField>
<ParamField type="Optional[LLMConfig]">LLM settings to install into the current async context for completion-based search types. When omitted, Cognee uses the active context config or global LLM config. Import `LLMConfig` from `cognee.infrastructure.llm.config`.</ParamField>
<ParamField type="Optional[EmbeddingConfig]">Embedding settings to install into the current async context for semantic retrieval. When omitted, Cognee uses the active context config or global embedding config. Import `EmbeddingConfig` from `cognee.infrastructure.databases.vector.embeddings.config`.</ParamField>
<ParamField type="bool">When set to `True`, appends a deterministic `Evidence:` block to completion-style answers (such as `GRAPH_COMPLETION` and `RAG_COMPLETION`), listing the source chunks behind the answer, and fills the result's `evidence` list with structured references to the chunks, graph nodes, and graph edges that were placed in the LLM context. For graph completion the cited chunks come from the [edge-evidence sidecar](/setup-configuration/overview#edge-evidence), which requires Alembic revision `f3a7b9c1d2e4`. The block is built in-process with no extra LLM call and is omitted silently when no usable references exist.</ParamField>
## Returns
`List[SearchResult]`
For the full breakdown of per-mode output shapes, see [Search](/core-concepts/main-operations/legacy-operations/search). The search-type accordions there now document each mode’s output directly.
### The prompt envelope
`only_context=True` returns the bare retrieval context — no session guidance, no conversation history, no rendered prompt — while a real completion is sent all of it. Pass `context_format="prompt"` to get the whole envelope instead, as a dict with `question`, `context`, `session_context`, `user_prompt`, and `system_prompt`. Import `ContextFormat` from `cognee.modules.search.types`, or pass the plain string.
```python theme={null}
envelope = await cognee.search(
"What are the main entities?",
only_context=True,
context_format="prompt",
)
```
You get one dict per dataset, so a single-dataset search returns a one-element list — unlike `"context"`, which unwraps to the context itself.
* `user_prompt` and `system_prompt` are `None` for the non-generative search types (`CHUNKS`, `SUMMARIES`, `CODE`, …), which have no prompt template, and for `CYPHER` and `AGENTIC_COMPLETION`, which opt out of the preview. The session layer is still reported for all of them.
* `session_context` is the session layer for `session_id`, or for the dataset's default session when you omit it. With caching off it falls back to the durable preference block, and it is `""` only when no session layer resolves at all.
* With `verbose=True`, the same three values also appear on the verbose dict as `session_context_result`, `user_prompt_result`, and `system_prompt_result`. They are keyed off the requested format rather than off whether the values are set, so they are always present (possibly `None`) when you ask for the prompt shape, and never present otherwise.
Building the envelope reconstructs the session layer **read-only**: no LLM call is made and nothing is written back to the session. It does cost one conversation-history read, which embeds the query.
<Note>
The envelope reports the prompt for the context actually retrieved, not a replay of a full turn. A real turn first rewrites your question with an LLM call, and that rewrite fills `question`, selects the conversation history, and ranks the guidance block. The preview cannot make that call, so it uses your raw query for all three.
</Note>
## Examples
```python theme={null}
import cognee
from cognee import SearchType
# Default graph completion search
results = await cognee.search("What is Cognee?")
# RAG-style search
results = await cognee.search(
"How does chunking work?",
query_type=SearchType.RAG_COMPLETION,
)
# Get raw chunks without LLM completion
results = await cognee.search(
"knowledge graph",
query_type=SearchType.CHUNKS,
top_k=5,
)
# Search within specific datasets
results = await cognee.search(
"deployment options",
datasets=["infrastructure_docs"],
)
# Context-only mode (no LLM answer)
context = await cognee.search(
"What are the main entities?",
only_context=True,
)
# Session-aware conversational search
results = await cognee.search(
"Tell me more about that",
session_id="conversation_123",
)
# Recommended v1.0 entry point for global context summaries
results = await cognee.recall(
"What is the current state of the rollout plan?",
query_type=SearchType.GRAPH_COMPLETION,
datasets=["product_docs"],
retriever_specific_config={
"include_global_context_index": True,
"global_context_index_top_k": 3,
},
)
# Filter to a specific node set (data tagged with node_set=["projectA"] during add())
from cognee.modules.engine.models.node_set import NodeSet
results = await cognee.search(
"What are the key decisions?",
node_type=NodeSet,
node_name=["projectA"],
)
# AND operator — only return results that belong to both node sets
results = await cognee.search(
"Cross-cutting concerns",
node_type=NodeSet,
node_name=["projectA", "finance"],
node_name_filter_operator="AND",
)
```
## Troubleshooting
<Accordion title="Fixing 'Nodeset does not exist' (EntityNotFoundError)">
NodeSets are **created at ingestion time** and **referenced at search time** — the names must match. During `remember()` or `add()`, the `node_set` tags you pass are attached to your data and materialized as `NodeSet` nodes when the graph is built. At search time, passing `node_type=NodeSet` with `node_name=[...]` projects only the subgraph connected to those names.
If none of the requested names resolve to a populated subgraph, lower-level graph projection can raise:
```text theme={null}
EntityNotFoundError: Nodeset does not exist, or empty nodeset projected from the database.
```
In regular `search()` or `recall()` flows, this can also appear as empty context or no results, depending on the search path.
Common causes:
* **The name was never created during ingestion.** You filtered on a name (e.g. `"finance"`) that was not passed in `node_set` during any `remember()` or `add()` call. A NodeSet only exists if data was tagged with it.
* **A typo or case mismatch.** Names are matched exactly, so `"Finance"` and `"finance"` are different NodeSets.
* **Search ran before ingestion finished.** With `run_in_background=True`, await the `RememberResult` before searching so the `NodeSet` nodes exist.
* **Wrong scope.** NodeSets live in the graph they were ingested into. If you query a different dataset or a different user/tenant under [multi-user mode](/core-concepts/multi-user-mode/multi-user-mode-overview), the names won't be found.
* **`AND` filtering with no overlap.** Using `node_name_filter_operator="AND"` requires nodes connected to *all* listed names simultaneously; if no node belongs to every name, the projection is empty.
To resolve it, confirm the exact NodeSet names you used during ingestion and reuse them verbatim in `search()` or `recall()`, scoped to the same dataset.
</Accordion>
<Warning>
With backend access control enabled, `datasets=["name"]` does not mean "any dataset with this name that I can read". It resolves names only within datasets owned by the current user. If Bob is querying Alice's shared dataset `shared_dataset`, `datasets=["shared_dataset"]` can fail with `DatasetNotFoundError` even when Bob has permission to read it. In that case, use `dataset_ids=[shared_id]`.
</Warning>
See [SearchType](/python-api/search-type) for all available search modes.
<Note>
`include_global_context_index` only has an effect after the dataset has been improved with `build_global_context_index=True`. See [Global Context Index](/core-concepts/further-concepts/global-context-index).
</Note>
# SearchType
Source: https://docs.cognee.ai/python-api/search-type
Enum of the search modes you can pass to cognee.search().
# SearchType
Enum defining the available search modes for `cognee.search()`.
```python theme={null}
from cognee import SearchType
results = await cognee.search("query", query_type=SearchType.GRAPH_COMPLETION)
```
## Values
The **Retrieval source** column shows what each type reads from: **Vector** (semantic similarity over embeddings), **Graph** (knowledge-graph traversal / Cypher), **Vector + Graph** (semantic seeds *plus* graph context — the "semantics + graph" combination), or **Lexical** (keyword matching, no embeddings). Types marked *Varies* pick or combine sources at runtime.
| Value | Retrieval source | Description |
| ----------------------------------------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `SearchType.SUMMARIES` | Vector | Pre-generated hierarchical summaries of content. |
| `SearchType.CHUNKS` | Vector | Raw text segments matching semantically. |
| `SearchType.CHUNKS_LEXICAL` | Lexical | Token-based lexical chunk search (BM25-style keyword ranking, no embeddings). |
| `SearchType.RAG_COMPLETION` | Vector | Traditional RAG: retrieve chunks by semantic similarity, then LLM answer. |
| `SearchType.HYBRID_COMPLETION` | Vector + Graph | **Default.** Hybrid answer combining lexical chunks, semantic chunks, and graph/entity context. |
| `SearchType.TRIPLET_COMPLETION` | Vector | Semantic search over triplet embeddings, then LLM answer (needs `TRIPLET_EMBEDDING=true`). |
| `SearchType.GRAPH_COMPLETION` | Vector + Graph | Semantic seeds expanded into graph context, then LLM completion. |
| `SearchType.GRAPH_COMPLETION_DECOMPOSITION` | Vector + Graph | Decomposes a complex query into focused subqueries, retrieves graph context per subquery, then synthesizes a final answer. |
| `SearchType.GRAPH_SUMMARY_COMPLETION` | Vector + Graph | Graph context condensed via summaries before completion. |
| `SearchType.GRAPH_COMPLETION_COT` | Vector + Graph | Graph completion with iterative chain-of-thought reasoning. |
| `SearchType.GRAPH_COMPLETION_CONTEXT_EXTENSION` | Vector + Graph | Graph completion that iteratively feeds each answer back as a new query to widen subgraph coverage. |
| `SearchType.TEMPORAL` | Vector + Graph | Time-aware search: extracts time constraints, then combines graph events with semantic retrieval. |
| `SearchType.CYPHER` | Graph | Direct Cypher query against the graph database. |
| `SearchType.NATURAL_LANGUAGE` | Graph | Natural language translated to Cypher, then executed against the graph. |
| `SearchType.CODING_RULES` | Graph | Retrieves coding rules stored in a graph node set. |
| `SearchType.CODE` | Graph | Deterministic operations over a code graph built by the [code-graph pipeline](/guides/code-graph): `code_query` selects one operation (`query_facts`, `explore`, `traverse`, `find_path`, `impact_analysis`, `insights`, `architecture`, `delta`). No LLM or embedding call; see [CODE](#per-search-type-parameters) below for every operation and its arguments. |
| `SearchType.GRAPH_REPORT` | Graph | Ignores the query and reports on the **whole graph**: hub nodes, cross-node-set connections, edge provenance, and suggested questions. See [`report()`](/python-api/report). |
| `SearchType.SKILLS` | Vector | Semantic discovery of the [skill](/examples/self-improving-skills) playbooks scoped to one dataset. Returns skill **metadata only** — never the procedure body. Requires exactly one explicit dataset. |
| `SearchType.AGENTIC_COMPLETION` | Varies | Agentic completion that chooses sources via skills and tools. |
| `SearchType.FEELING_LUCKY` | Varies | Auto-selects the best search type (and its source) for the query. |
## Speed, cost, and recall depth
Two things drive the cost of a search type: **how many LLM calls it makes** and **how much retrieved context it puts into each prompt**. The tabs below give measured reference numbers from one benchmark run: **Search speed & recall depth** compares every type on latency, call counts, and recall depth, and **Session speed and overhead** shows what the default session settings add on top, including the prompt-token cost of each type.
<Note>
**Methodology.** Measured on cognee 1.4 (dev) with `openai/gpt-5-mini`, LanceDB and Kuzu, against a small knowledge base (10 short documents → 99 nodes / 191 edges), 3 questions × 2 runs per type (1 run for the iterative modes), default `top_k=15`. Token counts are provider-reported and summed over all LLM calls in the query. Absolute values scale with your corpus, model, and hardware — read them as ratios and orders of magnitude, not guarantees. On larger corpora token costs grow with chunk size: a single chunk can hold up to \~8,191 tokens, and no layer truncates the assembled context by default.
</Note>
<Tabs>
<Tab title="Search speed & recall depth">
The single-call types (upper block) were measured with session memory disabled (`AUTO_FEEDBACK=false`): a query costs exactly its retrieval plus at most one completion. The iterative multi-call modes (lower block) were measured with default session settings — subtract one LLM call for their sessions-off cost, small relative to these totals. Prompt-token costs are in the **Session speed and overhead** tab.
| Search type | Median latency | LLM<br />calls | Embedding<br />calls | What the LLM sees (recall depth) |
| ------------------------------------ | -------------------------------------------------------------------------------- | ---------------------------------- | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `CHUNKS_LEXICAL` | \~10 ms | 0 | 0 | — (returns top 15 chunks, BM25-ranked) |
| `CYPHER` *(estimated)* | \~1 ms | 0 | 0 | — (returns raw rows from your Cypher query) |
| `CODING_RULES` *(estimated)* | \~1 ms | 0 | 0 | — (returns rules stored in a graph node set) |
| `CODE` *(estimated)* | scales with the code graph on the first call, then \~ms from the cached snapshot | 0 | 0 | — (returns the operation's structured result: facts, nodes and edges, a path, an impact report, or a diagram) |
| `SUMMARIES` | \~0.3 s | 0 | 1 | — (returns top 15 pre-generated summaries) |
| `CHUNKS` | \~0.3 s | 0 | 1 | — (returns top 15 chunks, semantic) |
| `RAG_COMPLETION` | \~3.9 s | 1 | 1 | Top 15 chunks (\~1.7k chars here) |
| `TRIPLET_COMPLETION` *(estimated)* | \~4 s | 1 | 1 | Top triplet matches; requires `TRIPLET_EMBEDDING=true` at cognify time |
| `GRAPH_COMPLETION` | \~3.9 s | 1 | 1 | Top 15 triplets plus their nodes' text (\~7.5k chars here) |
| `HYBRID_COMPLETION` *(default)* | \~3.8 s | 1 | 1 | 15 fused lexical+semantic chunks, 15 entities × ≤10 edges, ≤15 facts (\~14k chars here) |
| `GRAPH_REPORT` *(estimated)* | scales with graph size | 1 | 0 | First 2,000 chars of the report; the one LLM call only generates the suggested questions, everything else is computed deterministically over the whole graph |
| `TEMPORAL` | \~40 s | 3 | 1 | Top time-filtered events (time-extraction call first; falls back to triplet context when no time constraint is found) |
| `FEELING_LUCKY` | \~40 s | 2–5 | 0–5 | Whatever the LLM-chosen type retrieves, after one type-selection call |
| `NATURAL_LANGUAGE` | \~50 s | 4 | 0 | Raw rows from the generated Cypher (up to `max_attempts=3` generation attempts; no embeddings involved) |
| `GRAPH_SUMMARY_COMPLETION` | \~50 s | 3 | 1 | Same triplet context as `GRAPH_COMPLETION`, condensed by a summarization call before answering |
| `GRAPH_COMPLETION_DECOMPOSITION` | \~60 s | 5–8 | 1–5 | Triplet context per subquery (query split into 1–5 subqueries, one retrieval sweep each), merged in a final synthesis |
| `GRAPH_COMPLETION_CONTEXT_EXTENSION` | \~60 s | 6 | ≤5 | Triplet context accumulated over up to `context_extension_rounds=4` retrieve-and-generate rounds |
| `GRAPH_COMPLETION_COT` | \~200 s | 14 | 5 | Accumulated triplet context re-sent every round — `max_iter=4` rounds × (answer + validation + follow-up) calls |
| `AGENTIC_COMPLETION` *(estimated)* | 60–200 s | varies, ≤ `max_iter=6` tool rounds | varies | Whatever the loaded skills and tools return; budget for the `GRAPH_COMPLETION_DECOMPOSITION`-to-`COT` range |
Rows marked *(estimated)* were not measured — they are derived from how the retriever is built (`CYPHER`, `CODING_RULES`, and `CODE` are direct graph reads with no LLM or embedding call — `CODE` loads the dataset's code subgraph once and caches the parsed snapshot for 60 seconds, so only the first call scales with graph size; `TRIPLET_COMPLETION` has the same shape as `RAG_COMPLETION`; `GRAPH_REPORT` reads every node and edge via `get_graph_data()`, so its latency grows with the graph rather than with `top_k`; `AGENTIC_COMPLETION` is bounded by the measured iterative modes). Embedding-call counts for the multi-call modes are likewise derived rather than measured: each retrieval sweep embeds the query once, and `FEELING_LUCKY` inherits whatever the chosen type embeds — anywhere from 0 for the direct graph readers up to 5 if the selector picks `GRAPH_COMPLETION_COT`.
The ordering to remember: **latency** is nearly identical for the three single-call completion types (the single LLM call dominates; retrieval differs by tens of milliseconds), while **token cost and recall depth grow together** — on this corpus roughly 1 : 4.7 : 7.5 for RAG : GRAPH : HYBRID (measured sessions-off). `GRAPH_COMPLETION` does the most retrieval work per query (it fans out one vector search per index collection — 5 collections by default, repeated for each searched dataset and each capped at `wide_search_top_k=100` results — plus a graph projection), but that stays cheap next to the completion call.
</Tab>
<Tab title="Session speed and overhead">
With the default `CACHING=true` and `AUTO_FEEDBACK=true`, **every search type — including retrieval-only ones — pays one extra session-turn LLM call** (\~1,800 prompt tokens here; `GRAPH_REPORT`, `CODE`, and `SKILLS` are the exceptions — their retrievers opt out of the session-turn step), and completion types additionally embed and store the Q\&A turn and carry conversation history in the prompt. These latencies were measured with the session-turn call serialized ahead of retrieval — the behavior that is now `SESSION_SEARCH_MODE=sequential`. Under the current `concurrent` default, `RAG_COMPLETION`, `GRAPH_COMPLETION`, `HYBRID_COMPLETION`, and `TRIPLET_COMPLETION` overlap that call with answer generation, so their defaults-on latency sits closer to their sessions-off figures (follow-up turns instead add a second retrieval query and its embedding call); the other rows still run sequentially and stand as measured. Token and call counts are unchanged either way. See [Sessions and Caching](/core-concepts/sessions-and-caching#session-context-guidance-auto-feedback). Same corpus, defaults on:
| Search type | Median latency | LLM<br />calls | Embedding<br />calls | Prompt<br />tokens |
| ------------------------------------ | ---------------------- | ---------------------------------- | -------------------- | ------------------------------ |
| `CHUNKS_LEXICAL` | \~10 s | 1 | 0 | \~1,800 |
| `CYPHER` *(estimated)* | \~10 s | 1 | 0 | \~1,800 |
| `CODING_RULES` *(estimated)* | \~10 s | 1 | 0 | \~1,800 |
| `CODE` *(estimated)* | same as sessions-off | 0 | 0 | 0 (skips the session turn) |
| `SUMMARIES` | \~10 s | 1 | 1 | \~1,800 |
| `CHUNKS` | \~10 s | 1 | 1 | \~2,300 |
| `RAG_COMPLETION` | \~20 s | 2 | 3 | \~3,300 |
| `TRIPLET_COMPLETION` *(estimated)* | \~20 s | 2 | 3 | \~RAG-level + session turn |
| `GRAPH_COMPLETION` | \~20 s | 2 | 3 | \~5,400 |
| `HYBRID_COMPLETION` *(default)* | \~20 s | 2 | 3 | \~6,600 |
| `GRAPH_REPORT` *(estimated)* | scales with graph size | 1 | 0 | \~600 (skips the session turn) |
| `TEMPORAL` | \~40 s | 3 | 3 | \~6,300 |
| `FEELING_LUCKY` | \~40 s | 2–5 | 0–7 | \~6,100 |
| `NATURAL_LANGUAGE` | \~50 s | 4 | 0 | \~5,100 |
| `GRAPH_SUMMARY_COMPLETION` | \~50 s | 3 | 3 | \~5,900 |
| `GRAPH_COMPLETION_DECOMPOSITION` | \~60 s | 5–8 | 3–7 | \~13,700 |
| `GRAPH_COMPLETION_CONTEXT_EXTENSION` | \~60 s | 6 | ≤7 | \~23,100 |
| `GRAPH_COMPLETION_COT` | \~200 s | 14 | 7 | \~45,700 |
| `AGENTIC_COMPLETION` *(estimated)* | 60–200 s | varies, ≤ `max_iter=6` tool rounds | varies | varies |
For the sessions-off cost, subtract one LLM call and \~1,800 prompt tokens — negligible for the multi-call modes, but the whole bill for the retrieval-only types. Embedding calls are derived rather than measured: completion types add two embedding calls on top of their retrieval embeddings from the first tab — one to vector-recall session history for the prompt, one to embed and store the finished Q\&A turn — while retrieval-only types add none. Rows marked *(estimated)* were not measured; `CYPHER`, `CODING_RULES`, and `TRIPLET_COMPLETION` pay the session-turn overhead like every other type, while `GRAPH_REPORT`, `CODE`, and `SKILLS` opt out of the session-turn step entirely, so their cost is the same either way.
<Note>
If you don't need conversational memory on a query, pass `only_context=True` to skip both the completion and the session-turn call; set `AUTO_FEEDBACK=false` to drop the session-turn call globally while keeping session storage, or `CACHING=false` to disable sessions entirely. See [Sessions and Caching](/core-concepts/sessions-and-caching).
</Note>
</Tab>
</Tabs>
<Note>
**To go faster:** prefer `CHUNKS`, `SUMMARIES`, or `CHUNKS_LEXICAL` (no completion call), pass `only_context=True`, or disable auto-feedback. **To go deeper:** start with `GRAPH_COMPLETION`, escalate to `GRAPH_COMPLETION_DECOMPOSITION` for multi-part questions or `GRAPH_COMPLETION_COT` for multi-hop reasoning — the tables above show what that escalation costs (roughly 3× and 10× the latency, 2.5× and 8× the prompt tokens of `GRAPH_COMPLETION` on this corpus). Lowering `max_iter` / `context_extension_rounds` via `retriever_specific_config` reduces cost proportionally. See [Search Basics — Advanced Parameters](/guides/search-basics#advanced-parameters).
</Note>
## Choosing a Search Type
<AccordionGroup>
<Accordion title="I want an LLM-generated answer grounded in my data">
Use `HYBRID_COMPLETION` (default) for the best balance of accuracy and context.
Use `GRAPH_COMPLETION` for graph-only context, or `RAG_COMPLETION` for a simpler chunk-based approach.
</Accordion>
<Accordion title="I want raw data, not an LLM answer">
Use `CHUNKS` for semantic chunk retrieval or `CHUNKS_LEXICAL` for keyword-based.
Use `SUMMARIES` for pre-generated summaries.
</Accordion>
<Accordion title="I want to query the graph directly">
Use `CYPHER` for raw Cypher queries or `NATURAL_LANGUAGE` to have cognee
translate your question to Cypher.
</Accordion>
<Accordion title="I'm not sure which to use">
Just call [`recall()`](/python-api/recall) without `query_type`: its rule-based router picks a
strategy from cues in your query and defaults to `HYBRID_COMPLETION`, so plain questions get a
generated answer. Queries with exact-phrase, coding-vocabulary, or Cypher-shaped cues route to
payload-returning types (`CHUNKS_LEXICAL`, `CODING_RULES`, `CYPHER`) — see
[Auto-routing behavior](/core-concepts/main-operations/recall#examples-and-details) for the full mapping.
`FEELING_LUCKY` is a different, LLM-based selector. It can choose retrieval-only types such as
`SUMMARIES` or `CHUNKS`, which return payloads instead of a generated answer, so avoid it when
you need an answer on every call.
</Accordion>
<Accordion title="I need more accurate or comprehensive answers from the graph">
All four graph-completion modes retrieve graph triplets and generate an LLM answer, but they differ in depth and latency:
| Mode | Strategy | Best for | Trade-off |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------- |
| `GRAPH_COMPLETION` | Single-pass retrieval + completion | Most queries — good accuracy, low latency | Baseline |
| `GRAPH_COMPLETION_DECOMPOSITION` | Decomposes query into 1–5 subqueries, retrieves graph context per subquery, then synthesizes a final answer | Multi-entity or multi-aspect questions (e.g. "Tell me about A and B") | One extra LLM call for decomposition; slightly higher latency |
| `GRAPH_SUMMARY_COMPLETION` | Graph context condensed via summaries before completion | Noisy or large graphs where a tighter context improves coherence | Slightly slower; summary quality matters |
| `GRAPH_COMPLETION_COT` | Iterative: retrieve → answer → validate → follow-up (up to `max_iter` rounds, default 4) | Complex multi-hop questions where stepwise reasoning helps | Higher latency; more LLM calls |
| `GRAPH_COMPLETION_CONTEXT_EXTENSION` | Iterative: retrieve → generate → use output as new query (up to `context_extension_rounds`, default 4) | Open-ended or exploratory queries needing a broader subgraph | Higher latency; early convergence stops extra rounds |
See [Search Basics — Advanced Parameters](/guides/search-basics#advanced-parameters) and [Retrievers](/core-concepts/main-operations/legacy-operations/search#retrievers) for full details.
</Accordion>
</AccordionGroup>
## Per-search-type parameters
Every type accepts the **common parameters** (`query_text`, `top_k`, `system_prompt`/`system_prompt_path`, `only_context`, `verbose`, `include_references`, `datasets`/`dataset_ids`, `user`, `session_id`) documented in [Search Basics — Parameters Reference](/guides/search-basics#parameters-reference). The graph-completion family additionally honors the graph-ranking knobs (`wide_search_top_k`, `triplet_distance_penalty`, `feedback_influence`, `neighborhood_depth`, `neighborhood_seed_top_k`) and node-set filters. `wide_search_top_k` and `triplet_distance_penalty` are the two knobs `HYBRID_COMPLETION` *rejects* rather than ignores — passing either with the default search type raises `InvalidHybridSearchConfig`; see [Search Basics — Advanced Parameters](/guides/search-basics#advanced-parameters). With `recall()`, use `node_name` and `node_name_filter_operator`; `node_type` is only exposed on lower-level `search()`.
The accordions below list each search type's type-specific parameters. Most entries are passed through `retriever_specific_config`; when a search type uses a common parameter in a special way, that is noted.
<AccordionGroup>
<Accordion title="SUMMARIES">
| Parameter | Type | Default | Explanation |
| --------- | ---- | ------- | ----------------------------------------------------------------- |
| — | — | — | No `retriever_specific_config` keys. Uses common parameters only. |
</Accordion>
<Accordion title="CHUNKS">
| Parameter | Type | Default | Explanation |
| --------------------------- | ------------------- | ------- | --------------------------------------------------------------------------- |
| `node_name` | `list[str] \| None` | `None` | Common parameter used to filter chunks by node set. |
| `node_name_filter_operator` | `"OR" \| "AND"` | `"OR"` | Common parameter controlling how multiple `node_name` filters are combined. |
</Accordion>
<Accordion title="CHUNKS_LEXICAL">
| Parameter | Type | Default | Explanation |
| --------- | ---- | ------- | ----------------------------------------------------------------- |
| — | — | — | No `retriever_specific_config` keys. Uses common parameters only. |
</Accordion>
<Accordion title="RAG_COMPLETION">
| Parameter | Type | Default | Explanation |
| --------------------------- | ------------------- | ------- | --------------------------------------------------------------------------- |
| `response_model` | `type` | `str` | Output type or Pydantic model for the completion response. |
| `node_name` | `list[str] \| None` | `None` | Common parameter used to filter chunks by node set. |
| `node_name_filter_operator` | `"OR" \| "AND"` | `"OR"` | Common parameter controlling how multiple `node_name` filters are combined. |
</Accordion>
<Accordion title="HYBRID_COMPLETION">
| Parameter | Type | Default | Explanation |
| ------------------------------ | ------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `response_model` | `type` | `str` | Output type or Pydantic model for the completion response. |
| `chunks_top_k` | `int` | `min(top_k, 10)` | Max merged lexical and semantic chunks included in context. |
| `entities_top_k` | `int` | `min(top_k, 10)` | Number of matched entities used for graph context. |
| `max_edges_per_entity` | `int` | `10` | Max connected edges listed per entity. |
| `facts_top_k` | `int` | `min(top_k, 10)` | Max edge-derived facts included in the related facts context; requires the entity lane on (`entities_top_k` above `0`). |
| `include_global_context_index` | `bool` | `False` | Adds global-context summaries to the retrieved context. |
| `global_context_index_top_k` | `int` | `3` | Number of global-context summaries to include when enabled. |
| `text_summaries_top_k` | `int \| None` | `None` | Optional limit for text summaries included by the hybrid retriever. |
| `use_importance_weight` | `bool` | `True` | Applies importance weighting when ranking hybrid context. |
</Accordion>
<Accordion title="TRIPLET_COMPLETION">
| Parameter | Type | Default | Explanation |
| --------------------------- | ------------------- | ------- | --------------------------------------------------------------------------- |
| `response_model` | `type` | `str` | Output type or Pydantic model for the completion response. |
| `node_name` | `list[str] \| None` | `None` | Common parameter used to filter triplets by node set. |
| `node_name_filter_operator` | `"OR" \| "AND"` | `"OR"` | Common parameter controlling how multiple `node_name` filters are combined. |
</Accordion>
<Accordion title="GRAPH_COMPLETION">
Retrieval runs as vector seeds → 1-hop graph traversal → triplet ranking (`top_k` triplets) → context assembly → LLM completion. See [the concrete lookup process](/core-concepts/main-operations/legacy-operations/search#retrievers) for each stage and how `top_k` and `wide_search_top_k` shape it.
| Parameter | Type | Default | Explanation |
| ------------------------------ | ------ | ------- | ------------------------------------------------------------- |
| `response_model` | `type` | `str` | Output type or Pydantic model for the completion response. |
| `include_global_context_index` | `bool` | `False` | Adds global-context summaries to the retrieved graph context. |
| `global_context_index_top_k` | `int` | `3` | Number of global-context summaries to include when enabled. |
</Accordion>
<Accordion title="GRAPH_COMPLETION_DECOMPOSITION">
| Parameter | Type | Default | Explanation |
| -------------------- | ------------------------------------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `response_model` | `type` | `str` | Output type or Pydantic model for the completion response. |
| `decomposition_mode` | `"answer_per_subquery" \| "combined_triplets_context"` | `"answer_per_subquery"` | Controls whether each subquery is answered separately or all retrieved triplets are merged before the final answer. |
</Accordion>
<Accordion title="GRAPH_SUMMARY_COMPLETION">
| Parameter | Type | Default | Explanation |
| ----------------------- | ----- | -------------------------------- | -------------------------------------------------------------- |
| `summarize_prompt_path` | `str` | `"summarize_search_results.txt"` | Prompt file used to summarize graph context before completion. |
</Accordion>
<Accordion title="GRAPH_COMPLETION_COT">
| Parameter | Type | Default | Explanation |
| ------------------------------- | ------ | ------------------------------------ | ----------------------------------------------------------------- |
| `max_iter` | `int` | `4` | Number of follow-up reasoning rounds after the initial retrieval. |
| `response_model` | `type` | `str` | Output type or Pydantic model for the completion response. |
| `validation_system_prompt_path` | `str` | `"cot_validation_system_prompt.txt"` | System prompt used to validate the current answer. |
| `validation_user_prompt_path` | `str` | `"cot_validation_user_prompt.txt"` | User prompt used to validate the current answer. |
| `followup_system_prompt_path` | `str` | `"cot_followup_system_prompt.txt"` | System prompt used to generate follow-up questions. |
| `followup_user_prompt_path` | `str` | `"cot_followup_user_prompt.txt"` | User prompt used to generate follow-up questions. |
</Accordion>
<Accordion title="GRAPH_COMPLETION_CONTEXT_EXTENSION">
| Parameter | Type | Default | Explanation |
| -------------------------- | ------ | ------- | -------------------------------------------------------------------------------- |
| `context_extension_rounds` | `int` | `4` | Max rounds for extending context by using the previous output as the next query. |
| `response_model` | `type` | `str` | Output type or Pydantic model for the completion response. |
</Accordion>
<Accordion title="TEMPORAL">
| Parameter | Type | Default | Explanation |
| ----------------------------- | ------ | ---------------------------------- | ------------------------------------------------------------------- |
| `response_model` | `type` | `str` | Output type or Pydantic model for the completion response. |
| `user_prompt_path` | `str` | `"graph_context_for_question.txt"` | Prompt file used to format temporal graph context for the question. |
| `system_prompt_path` | `str` | `"answer_simple_question.txt"` | System prompt file used for the final answer. |
| `time_extraction_prompt_path` | `str` | `"extract_query_time.txt"` | Prompt file used to extract temporal constraints from the query. |
</Accordion>
<Accordion title="CYPHER">
Requires a Cypher-capable graph backend (Kuzu or Neo4j); on a backend that cannot run Cypher, such as the Postgres graph demo backend, the search raises `SearchTypeNotSupported`.
| Parameter | Type | Default | Explanation |
| -------------------- | ----- | ------------------------------ | ---------------------------------------------- |
| `user_prompt_path` | `str` | `"context_for_question.txt"` | Prompt file used to format Cypher results. |
| `system_prompt_path` | `str` | `"answer_simple_question.txt"` | System prompt file used for result formatting. |
</Accordion>
<Accordion title="NATURAL_LANGUAGE">
Generates and executes Cypher, so it requires a Cypher-capable graph backend (Kuzu or Neo4j); on a backend that cannot run Cypher, such as the Postgres graph demo backend, the search raises `SearchTypeNotSupported`.
| Parameter | Type | Default | Explanation |
| -------------------- | ----- | ----------------------------------------- | ---------------------------------------------------------------- |
| `system_prompt_path` | `str` | `"natural_language_retriever_system.txt"` | System prompt file used to translate natural language to Cypher. |
| `max_attempts` | `int` | `3` | Max attempts to generate and run a useful graph query. |
</Accordion>
<Accordion title="CODING_RULES">
| Parameter | Type | Default | Explanation |
| ----------- | ------------------- | ------- | --------------------------------------------------------------- |
| `node_name` | `list[str] \| None` | `None` | Common parameter interpreted as the coding-rules node set name. |
</Accordion>
<Accordion title="CODE">
Runs one deterministic operation over the code graph the [code-graph pipeline](/guides/code-graph) built — no LLM, embedding, or vector call. Its arguments arrive through the dedicated `code_query` dict on [`search()`](/python-api/search) and [`recall()`](/python-api/recall) rather than `retriever_specific_config`; passing `code_query` with any other search type raises `InvalidCodeSearchConfig`. `top_k` and the prompt parameters have no effect, `query_text` is read only as a seed or name filter when the dict carries no other filter, and batched queries are not supported. Searching several datasets at once treats a seed that one dataset cannot resolve as an empty result for that dataset rather than a failed call.
`operation` selects exactly one graph operation; anything else in the dict is that operation's arguments. Omitting `operation` runs `explore`. An unknown name raises `CodeSearchValidationError`. `max_depth`, `max_nodes` and `limit` are clamped to the maximum shown, and a non-positive value falls back to the default.
| Operation | What it returns | Arguments |
| ----------------- | ------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `query_facts` | Facts matching the filters, plus `total`, `offset`, `limit`, `has_more` | `kind` / `kinds`, `name` (substring) or `names` (exact), `file` / `files` / `file_prefix`, `repo`, `relation_types`, `property` (alias `prop`) with `property_value` (alias `prop_value`), `offset`, `limit` (default 100, max 500) |
| `explore` | One fact's `focus` plus the nodes, edges and `stats` around it | `name` (or `focus`, or the query text) or `id`, `repo`, `direction` (default `both`), `node_types`, `relation_types`, `max_depth` (default 1, max 2), `max_nodes` (default 100, max 500) |
| `traverse` | Nodes, edges and `stats` reached from one or more seeds in one direction | `node_ids` / `id` / `start_id(s)`, `names` / `name` / `start(s)`, `repo`, `direction` (default `forward`), `node_types`, `relation_types`, `max_depth` (default 5, max 20), `max_nodes` (default 100, max 500), `type_rollup` |
| `find_path` | `found`, the shortest forward `path`, and the `edges` along it | `source` / `from` or `source_id` / `from_id`, `target` / `to` or `target_id` / `to_id`, `source_repo`, `target_repo`, `relation_types`, `max_depth` (default 10, max 20), `target_rollup` |
| `impact_analysis` | Dependents grouped `by_depth`, with `total_dependents`, `cross_repo_impact` and a `summary` | `target(s)` / `name(s)` / `id` / `node_ids`, `repo`, `node_types`, `relation_types`, `max_depth` (default 3, max 10), `max_nodes` (default 200, max 500), `include_dependencies` |
| `insights` | Explainer findings, each with the `evidence` facts it cites, plus `by_source` counts | `source` / `sources`, `min_confidence` (0–1), `informational`, `repo`, `name` (title substring), `offset`, `limit` (default 50, max 500) |
| `architecture` | Module-level `nodes` and counted, rolled-up `edges`, with `stats` | `node_types` (default `module`, `route`, `storage`, `service`), `repo`, `relation_types`, `max_nodes` (default 80, max 300) |
| `delta` | Per repository, the last ingestion's `delta` record and the snapshot `receipt` | `repo` |
`diagram` works on every operation (`"mermaid"`, `"dot"`, or `true` for Mermaid); `architecture` sets it to Mermaid by itself. Seeds resolve by exact name first, then by short name and substring — an ambiguous name raises, so pass an id or narrow with `repo`. The rollup flags (`type_rollup` on a reverse `traverse`, `target_rollup` on `find_path`, both on by default) widen a type seed to its methods and constructor, so a dependent that only calls a method still counts; set them to `false` for the literal node. `impact_analysis` always rolls up its targets and has no flag.
**Fact kinds.** Every extracted fact carries a `kind`. `query_facts` filters on it with `kind` (one) or `kinds` (several); `explore`, `traverse`, `impact_analysis` and `architecture` restrict the nodes they return with `node_types`. Both fields accept either the kind below or the node type it maps to (`"symbol"` and `"CodeSymbol"` are the same filter). `find_path` and `delta` take neither, and `insights` always returns `insight` facts.
| Kind | Node type | What it is |
| ------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `module` | `CodeModule` | A module or package. |
| `symbol` | `CodeSymbol` | A declared symbol; which one is on `symbol_kind` (`function`, `method`, `getter`, `struct`, `interface`, `type`, `class`, `variable`, `constant`, `enum`, and descriptive values newer extractors add). |
| `route` | `ApiEndpoint` | An API route the code exposes. |
| `storage` | `StorageResource` | A storage resource such as a table or a bucket. |
| `service` | `CodeService` | A deployable service. |
| `dependency` | `ExternalDependency` | An external dependency; the packages declared in the repository's manifests carry `type: package`. |
| `insight` | `CodeInsight` | A finding from one of enola's explainers (cycles, layers, hotspots, god-class, dependency-depth, exported-surface, complexity-outliers, …), linked to the facts it cites by `evidences` edges. |
| `intent` | `CodeIntent` | Architecture declared in `enola-intent.yaml` rather than measured from the source. |
| `association` | `CodeAssociation` | A framework model relationship such as Rails `has_many` / `belongs_to`. |
| `lint` | `CodeLintFinding` | A finding an external linter reported through enola's provider seam. |
| `extraction` | `CodeExtractionAccount` | One extractor's coverage account for a repository (`<extractor>:<account>`, e.g. `ruby:calls`), so a thin graph can be told apart from a thin extraction. |
| `test_ref` | `CodeTestReference` | A test-to-symbol reference. |
| `file_ref` | `CodeFileReference` | A file-level reference. |
Which kinds a repository actually produces depends on its languages and on what enola's extractors found, so a filter on a kind the snapshot has none of returns an empty page rather than an error. Repositories themselves (`CodeRepository`) are deliberately not facts — they appear only in `delta`.
</Accordion>
<Accordion title="AGENTIC_COMPLETION">
Requires the resolved scope to contain exactly one dataset.
| Parameter | Type | Default | Explanation |
| ---------------------------- | ---------------------------- | ---------------------- | ------------------------------------------------------------------ |
| `skills` | `list[str \| Skill] \| None` | `None` | Skill names or `Skill` objects to load into the agentic retriever. |
| `tools` | `list[str] \| None` | `None` | Optional whitelist of tool names available to the agent. |
| `max_iter` | `int` | `6` | Max tool-call iterations before forcing a final answer. |
| `agentic_system_prompt_path` | `str` | `"agentic_system.txt"` | System prompt file used inside the agent loop. |
| `agentic_user_prompt_path` | `str` | `"agentic_user.txt"` | User prompt file used inside the agent loop. |
| `response_model` | `type` | `str` | Output type or Pydantic model for the final response. |
</Accordion>
<Accordion title="GRAPH_REPORT">
Ignores `query_text` — the report always covers the whole graph — and opts out of
the session-turn preparation step, since there is no query to analyse.
| Parameter | Type | Default | Explanation |
| --------- | ----- | ------- | -------------------------------------------------------------------------------------------------------------------- |
| `top_k` | `int` | `15` | Common parameter passed through as the retriever's `top_n`: how many hub nodes and cross-set connections to surface. |
For the file-writing equivalent with its own `top_n` argument, see [`cognee.report()`](/python-api/report).
</Accordion>
<Accordion title="SKILLS">
Deterministic and non-generative — it makes no LLM call and opts out of the
session-turn preparation step. Scoped to **exactly one** dataset: pass a single
entry in `datasets`/`dataset_ids`, or the call raises `CogneeValidationError`
(`InvalidSkillsDatasetScope`).
| Parameter | Type | Default | Explanation |
| --------- | ----- | ------- | ----------------------------------------------------- |
| `top_k` | `int` | `15` | Common parameter: how many in-scope skills to return. |
Only skills that are active and whose `dataset_scope` contains the requested dataset
are returned; a skill with an empty `dataset_scope` never matches. Each result carries
the skill's metadata — `id`, `name`, `description`, `maintainer`, `maintainer_url`,
`version`, `tags`, `license`, `declared_tools`, `dataset_scope`, `is_active`,
`source_repo_url`, `source_dir` — and the raw vector `score`, a backend distance
where lower is better. The `procedure` body is deliberately withheld
to preserve progressive disclosure; load it with the `load_skill` tool or
`GET /api/v1/skills/{skill_id}`.
When no skills have been ingested yet the collection does not exist, and the search
returns an empty result rather than raising `NoDataError` — unlike `SUMMARIES`.
</Accordion>
<Accordion title="FEELING_LUCKY">
| Parameter | Type | Default | Explanation |
| --------- | ---- | ------- | --------------------------------------------------------------------------------------------------------------- |
| — | — | — | No direct `retriever_specific_config` keys. It selects another search type and inherits that type's parameters. |
</Accordion>
</AccordionGroup>
```python theme={null}
# Example: tune the chain-of-thought depth and request typed output
await cognee.recall(
query_text="How did the incident propagate across services?",
query_type=SearchType.GRAPH_COMPLETION_COT,
retriever_specific_config={"max_iter": 2},
)
```
See [Search Basics — Advanced Parameters](/guides/search-basics#advanced-parameters) for `retriever_specific_config` usage and [Retrievers](/core-concepts/main-operations/legacy-operations/search#retrievers) for per-retriever behavior.
# serve()
Source: https://docs.cognee.ai/python-api/serve
Connect the Cognee SDK to Cognee Cloud or a remote Cognee instance
# cognee.serve()
```python theme={null}
async def serve(
url: Optional[str] = None,
api_key: Optional[str] = None,
*,
management_url: Optional[str] = None,
auth0_domain: Optional[str] = None,
auth0_client_id: Optional[str] = None,
auth0_audience: Optional[str] = None,
) -> CloudClient
```
## Description
Connect the local Cognee Python SDK to Cognee Cloud or another remote Cognee API server.
After `serve()` connects, high-level SDK operations route to the remote instance instead of local storage. You can call methods on the returned `CloudClient`, or continue using top-level operations such as `cognee.remember()`, `cognee.recall()`, `cognee.improve()`, and `cognee.forget()`.
Use `cognee.disconnect()` to clear the active remote client and return the SDK to local mode.
<Note>
`serve()` changes where SDK operations execute. It does not copy existing local datasets to the remote instance. Use [`push()`](/python-api/push) to upload an already-built local graph, or call `remember()` after connecting to ingest directly into the remote instance.
</Note>
## Connection resolution
`serve()` resolves the remote target in this order:
1. Explicit `url` and `api_key` arguments
2. `COGNEE_SERVICE_URL` and `COGNEE_API_KEY` environment variables
3. Saved credentials from a previous Cognee Cloud login
4. Device login flow when connecting to Cognee Cloud interactively — available only when a device client ID is configured (the `COGNEE_AUTH0_DEVICE_CLIENT_ID` environment variable, or the `auth0_client_id` argument). There is no built-in default.
A resolved `url` — from either of the first two steps — takes the **direct** path: no device login, no tenant discovery. Only a bare `serve()` reaches steps 3 and 4.
### Credentials are verified at connect time
`/health` is unauthenticated, so it cannot tell a working API key from a rejected one. `serve()` therefore also probes an authenticated endpoint (`GET /api/v1/datasets`) before it reports a connection, so a bad key fails here instead of on your first real operation. When a `url` is resolved (the direct path):
| What the probe sees | What `serve()` does |
| -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `401` or `403` | Raises `CogneeConfigurationError` and saves nothing. With an `api_key` passed, the message says the key was rejected; without one, it explains how to supply one (including minting a key on a self-hosted server with `POST /api/v1/auth/api-keys`). |
| Instance unreachable | Logs a warning and continues — an instance that is merely down is not a configuration error. |
| Anything else | Connects, and saves the credentials so a later `serve()` reconnects without arguments. |
Saved credentials go through the same probe, with one difference: on that path both a rejected key and an unreachable instance fall through to re-authentication (a token refresh when possible, otherwise the device login) rather than to a connection that 401s on every call. When no device client ID is configured, that fall-through becomes the third error below.
### Errors
`serve()` raises `CogneeConfigurationError` (importable from `cognee.exceptions`) in these cases:
* **The instance rejected the key** — `name="ServeAuthenticationError"`, the 401/403 case above. The same error, with a message calling it a provisioning problem, is raised when a fresh Cloud login succeeds but the tenant instance rejects the API key that was just provisioned for it; nothing is saved in either case.
* **Nothing to connect with** — a bare `serve()` with no saved credentials, no environment variables, and no device client ID. `name="ServeConfigurationError"`; the message lists the ways to connect.
* **Saved credentials no longer work and cannot be renewed** — a bare `serve()` whose saved instance is unreachable or whose credentials were rejected, with no device client ID to re-authenticate through. Also `name="ServeConfigurationError"`; the message names the credentials file and the account on it, and suggests deleting the file if the credentials are stale.
<Note>
Passing `url` to a server that requires authentication without also passing `api_key` is a configuration error, not a silent downgrade: the probe returns `401` and `serve()` raises. A server with its default posture requires authentication — `REQUIRE_AUTHENTICATION` inherits from `ENABLE_BACKEND_ACCESS_CONTROL`, which is on by default, and setting it to `false` alone is ignored while access control is on — so `serve(url="http://localhost:8000")` needs an API key there. Only a server running with `ENABLE_BACKEND_ACCESS_CONTROL=false`, which also turns the auth requirement off, falls back to its default user and answers the probe without a key.
</Note>
## Parameters
<ParamField type="Optional[str]">Remote Cognee instance URL. When omitted, Cognee reads `COGNEE_SERVICE_URL`, saved Cloud credentials, or starts the Cloud login flow.</ParamField>
<ParamField type="Optional[str]">API key for the remote instance. When omitted, Cognee reads `COGNEE_API_KEY` or saved Cloud credentials. A server running with `ENABLE_BACKEND_ACCESS_CONTROL=false` does not require it.</ParamField>
<ParamField type="Optional[str]">Keyword-only. Override the Cognee Cloud Management API URL used for tenant discovery. Cloud path only; defaults to `COGNEE_CLOUD_URL`.</ParamField>
<ParamField type="Optional[str]">Keyword-only. Override the Auth0 domain used by the device login flow. Cloud path only.</ParamField>
<ParamField type="Optional[str]">Keyword-only. Auth0 device-login client ID. Without this or `COGNEE_AUTH0_DEVICE_CLIENT_ID`, a bare `serve()` cannot start the device login flow and raises `CogneeConfigurationError` when no saved credentials work.</ParamField>
<ParamField type="Optional[str]">Keyword-only. Override the Auth0 API audience requested by the device login flow. Cloud path only.</ParamField>
## Returns
A `CloudClient` configured for the connected instance.
The returned client exposes the main remote operations:
<ParamField type="method">Ingest data and build memory on the remote instance.</ParamField>
<ParamField type="method">Write a single session-memory entry on the remote instance.</ParamField>
<ParamField type="method">Query memory from the remote instance.</ParamField>
<ParamField type="method">Run enrichment or session-bridging on remote memory.</ParamField>
<ParamField type="method">Delete remote data, datasets, or memory state.</ParamField>
<ParamField type="method">Ingest data on the remote instance without building the graph.</ParamField>
<ParamField type="method">Build the knowledge graph on the remote instance.</ParamField>
<ParamField type="method">Query the remote knowledge graph.</ParamField>
<ParamField type="method">Replace one document in place on the remote instance via `PATCH /api/v1/update`.</ParamField>
<ParamField type="method">List the documents in a remote dataset via `GET /api/v1/datasets/{dataset_id}/data`.</ParamField>
## Operations that route to the remote instance
While a connection is active, these top-level SDK calls are proxied to the remote instance instead of touching local storage — you do not have to call the `CloudClient` method yourself:
| SDK call | Remote route |
| ----------------------------- | ---------------------------------------- |
| `cognee.remember()` | `POST /api/v1/remember` |
| `cognee.recall()` | `POST /api/v1/recall` |
| `cognee.improve()` | `POST /api/v1/improve` |
| `cognee.forget()` | `POST /api/v1/forget` |
| `cognee.add()` | `POST /api/v1/add` |
| `cognee.cognify()` | `POST /api/v1/cognify` |
| `cognee.search()` | `POST /api/v1/search` |
| `cognee.update()` | `PATCH /api/v1/update` |
| `cognee.datasets.list_data()` | `GET /api/v1/datasets/{dataset_id}/data` |
Routing happens before any local resolution, so these calls work against datasets that exist only on the remote instance, and they write nothing to the local store.
<Note>
The remote routes accept a narrower parameter surface than their local counterparts. `update()` proxies only `data_id`, `data`, `dataset_id`, `node_set`, and `chunk_level_diff`; anything else you pass is dropped and the server applies its own configuration. Most of the dropped parameters are named in a logged warning, but `user`, `incremental_loading`, and `data_cache` go silently. See [`update()`](/python-api/update#remote-mode-serve) for the full list.
</Note>
## Examples
<Tabs>
<Tab title="Cognee Cloud">
The interactive Cloud login needs a device client ID; without it a bare `serve()` raises instead of opening the login flow.
```bash theme={null}
export COGNEE_AUTH0_DEVICE_CLIENT_ID="your-device-client-id"
```
```python theme={null}
import cognee
# Opens the device login flow and saves reusable credentials.
client = await cognee.serve()
await client.remember("Cognee Cloud stores memory remotely.", dataset_name="docs")
results = await client.recall("Where is memory stored?")
await cognee.disconnect()
```
</Tab>
<Tab title="Explicit credentials">
```python theme={null}
import cognee
client = await cognee.serve(
url="https://your-tenant.aws.cognee.ai",
api_key="your-api-key",
)
await client.remember("Runbook: deploys happen on Tuesdays.", dataset_name="ops")
results = await client.recall("When do deploys happen?")
await cognee.disconnect()
```
</Tab>
<Tab title="Environment variables">
```bash theme={null}
export COGNEE_SERVICE_URL="https://your-tenant.aws.cognee.ai"
export COGNEE_API_KEY="your-api-key"
```
```python theme={null}
import cognee
client = await cognee.serve()
await client.recall("What is stored in this tenant?")
```
</Tab>
<Tab title="Local server">
Start a local Cognee backend first:
```bash theme={null}
cognee serve
```
Then connect the SDK to it. A server with its default posture requires authentication, so pass an API key minted with `POST /api/v1/auth/api-keys`; omit it only when the server runs with `ENABLE_BACKEND_ACCESS_CONTROL=false`:
```python theme={null}
import cognee
client = await cognee.serve(url="http://localhost:8000", api_key="your-api-key")
await client.remember("Cognee turns documents into AI memory.", dataset_name="docs")
await cognee.disconnect()
```
</Tab>
</Tabs>
## Disconnecting
```python theme={null}
await cognee.disconnect()
```
`disconnect()` closes the active remote connection for the current SDK process. Saved credentials are not deleted, so a later `serve()` call can reconnect without requiring a new login when the credentials are still valid.
## See also
* [Serve main operation](/core-concepts/main-operations/serve)
* [Cloud SDK](/cognee-cloud/connections/cloud-sdk)
* [Syncing a local instance](/cognee-cloud/connections/syncing-local-instance)
* [push()](/python-api/push)
# update()
Source: https://docs.cognee.ai/python-api/update
Update existing data in the knowledge base
# cognee.update()
```python theme={null}
async def update(
data_id: UUID,
data: Union[BinaryIO, list[BinaryIO], str, list[str]],
dataset_id: UUID,
user: User = None,
node_set: Optional[List[str]] = None,
vector_db_config: dict = None,
graph_db_config: dict = None,
preferred_loaders: dict[str, dict[str, Any]] = None,
incremental_loading: bool = True,
data_cache: bool = True,
chunk_level_diff: bool = True,
graph_model: type[BaseModel] = KnowledgeGraph,
custom_prompt: Optional[str] = None,
chunker: type = TextChunker,
policy: ChunkPolicy = DEFAULT_CHUNK_POLICY,
) -> Union[Dict[str, PipelineRunInfo], List[PipelineRunInfo], dict]
```
## Description
Update existing data in Cognee.
The document **keeps its `data_id`** across updates: the replacement is re-ingested pinned to the resolved id — the exact id, or the recorded pre-split `legacy_id` of a document forked by the [dataset-scoping upgrade](/python-api/run-migrations#dataset-scoping-upgrade) — so id mappings you hold externally never break. `update()` replaces exactly one document per call and never creates one: lists of more than one item are rejected, and a `data_id` that resolves to no document in the dataset raises `UpdateTargetNotFoundError` (HTTP 404). Use [`add()`](/python-api/add) for new documents.
Supported Input Types:
* **Text strings**: Direct text content (str) - any string that is not a `file://`, `s3://`,
`http://` or `https://` URL and does not point to an existing local file. An absolute-looking
string that is not an existing file (for example `"/remember to call the dentist"`) is ingested
as text content on every platform, including Windows.
* **File paths**: Local file paths as strings in these formats:
* Absolute paths: "/path/to/document.pdf" (treated as a file reference only when the file exists)
* File URLs: "file:///path/to/document.pdf" or "file://relative/path.txt"
* S3 paths: "s3://bucket-name/path/to/file.pdf"
* **Binary file objects**: File handles/streams (BinaryIO)
* **Lists**: Accepted only with exactly one item — `update()` replaces one document per call, and a longer list is rejected
Supported File Formats:
* Text files (.txt, .md, .csv)
* PDFs (.pdf)
* Images (.png, .jpg, .jpeg) - extracted via OCR/vision models
* Audio files (.mp3, .wav) - transcribed to text
* Code files (.py, .js, .ts, etc.) - parsed for structure and content
* Office documents (.docx, .pptx)
Workflow:
1. **Data Resolution**: Resolves file paths and validates accessibility
2. **Content Extraction**: Extracts text content from various file formats
3. **Dataset Storage**: Stores processed content in the specified dataset
4. **Metadata Tracking**: Records file metadata, timestamps, and user permissions
5. **Permission Assignment**: Grants user read/write/delete/share permissions on dataset
Args:
data\_id: UUID of existing data to update
data: The latest version of the data. Can be:
* Single text string: "Your text content here"
* Absolute file path: "/path/to/document.pdf" (must exist; otherwise the string is ingested as text)
* File URL: "file:///absolute/path/to/document.pdf" or "file://relative/path.txt"
* S3 path: "s3://my-bucket/documents/file.pdf"
* Single-item list: \["Updated content"] (lists of more than one item are rejected)
* Binary file object: open("file.txt", "rb")
dataset\_name: Name of the dataset to store data in. Defaults to "main\_dataset".
Create separate datasets to organize different knowledge domains.
user: User object for authentication and permissions. Uses default user if None.
Default user: "[default\_user@example.com](mailto:default_user@example.com)" (created automatically on first use).
Users can only access datasets they have permissions for.
node\_set: Optional list of node identifiers for graph organization and access control.
Used for grouping related data points in the knowledge graph.
vector\_db\_config: Optional configuration for vector database (for custom setups).
graph\_db\_config: Optional configuration for graph database (for custom setups).
dataset\_id: Optional specific dataset UUID to use instead of dataset\_name.
Returns:
With chunk\_level\_diff, a summary dict with the same keys for either status:
`{"status": "incremental" | "unchanged", "regions": n, "deleted_chunks": n, "added_chunks": n, "reused_chunks": n, "kept_chunks": n, "reindexed_chunks": n}`
Otherwise PipelineRunInfo: Information about the ingestion pipeline execution including:
* Pipeline run ID for tracking
* Dataset ID where data was stored
* Processing status and any errors
* Execution timestamps and metadata
## Parameters
<ParamField type="UUID">UUID of the data item to update.</ParamField>
<ParamField type="Union[BinaryIO, list[BinaryIO], str, list[str]]">New data to replace the existing data.</ParamField>
<ParamField type="UUID">UUID of the dataset containing the data.</ParamField>
<ParamField type="User">User performing the operation.</ParamField>
<ParamField type="Optional[List[str]]">List of node set names to associate.</ParamField>
<ParamField type="dict">Override vector database configuration.</ParamField>
<ParamField type="dict">Override graph database configuration.</ParamField>
<ParamField type="dict[str, dict[str, Any]]">Custom loader configuration.</ParamField>
<ParamField type="bool">If true, skip unchanged data. The skip runs whenever this **or** `data_cache` is true, so a full re-cognify requires both to be `False`.</ParamField>
<ParamField type="bool">Companion flag to `incremental_loading` — either one being true enables the already-processed skip for a data item.</ParamField>
<ParamField type="bool">Diff the new content against the stored text and re-ingest only the chunks the edit touched, leaving unaffected chunks and their entities in place. Falls back to the full delete-and-re-add flow when its preconditions are not met — see [How it works](#how-it-works) for the full set.</ParamField>
<ParamField type="type[BaseModel]">Pydantic model the extraction step fills in. A non-default model disables the chunk-level path.</ParamField>
<ParamField type="Optional[str]">Custom extraction prompt. Supplying one disables the chunk-level path.</ParamField>
<ParamField type="type">Chunking strategy. Must match the one that built the document's stored chunks; a mismatch falls back to the full flow rather than failing. Chunk-level path only.</ParamField>
<ParamField type="ChunkPolicy">Decides which chunks exist after the edit and what happens to the old ones. Chunk-level path only, and not exposed on the HTTP route.</ParamField>
## Returns
`Union[Dict[str, PipelineRunInfo], List[PipelineRunInfo], dict]`
With `chunk_level_diff` (the default), a summary dict with the same keys for either status: `{"status": "incremental" | "unchanged", "regions": n, "deleted_chunks": n, "added_chunks": n, "reused_chunks": n, "kept_chunks": n, "reindexed_chunks": n}`. When the full flow runs instead, `PipelineRunInfo` as described above.
## How it works
With `chunk_level_diff` left at its default, `update()` diffs the new content against the stored text and touches only the chunks the edit changed. When that path's preconditions are not met — a first ingestion, non-text content, a document whose stored chunks predate chunk-scoped ownership (anything ingested before v1.5.4), stored chunks that no longer tile the stored text, a graph or vector store without the narrow operations the path needs, a `chunker` that does not match the stored chunks, a `node_set` or changed `DataItem` metadata, a non-default `graph_model` or `custom_prompt`, or a per-call `vector_db_config`/`graph_db_config` — it falls back to a full **delete-then-re-add** cycle for the specified data item, logging a warning as it does:
1. **Delete** — removes the old data item's graph nodes, edges, and vector embeddings. Entities that are also referenced by *other* documents in the same dataset are preserved (shared nodes are not deleted).
2. **Add** — re-ingests the new version of the data into the dataset, pinned to the resolved `data_id`, so the document's id is the same after the update.
3. **Cognify** — re-runs the knowledge graph construction pipeline on the dataset, extracting entities and relationships from the updated content.
After `update()` completes, all graph nodes and relationships derived from the old content are removed and replaced with ones extracted from the new content. Relationships that no longer appear in the updated document are gone; new relationships found in the updated content are added. The document's `data_id` is unchanged throughout — only the derived content moves.
## Remote mode (`serve()`)
While a [`serve()`](/python-api/serve) connection is active, `update()` is proxied to the remote instance over `PATCH /api/v1/update`. The routing happens **before any local work**, so the call operates on the remote document and writes nothing to the local store — updating a dataset that exists only on the remote instance succeeds instead of failing with a "Dataset not found" error from the local store.
Only five parameters travel to the route:
| Parameter | How it is sent |
| ------------------ | ------------------------------ |
| `data_id` | query parameter |
| `dataset_id` | query parameter |
| `chunk_level_diff` | query parameter |
| `data` | multipart file field |
| `node_set` | repeated multipart form fields |
Every other parameter is dropped, and only some of them say so:
* `vector_db_config`, `graph_db_config`, `preferred_loaders`, `graph_model`, `custom_prompt`, `chunker`, and `policy` have no slot on the route. Passing any of them logs a warning naming the dropped parameters, and the server applies its own configuration for those settings.
* `user`, `incremental_loading`, and `data_cache` are dropped **silently**. `user` is redundant — the API key `serve()` connected with identifies the caller, and the server runs its own permission check — but `incremental_loading=False` and `data_cache=False` are no-ops over a connection, with nothing in the logs to say so.
The call still proceeds in both cases.
Three other differences from local mode:
* **A file path string is uploaded as text, not as the file it points at.** Locally, `data="/path/to/document.pdf"` (or an `s3://` or `file://` URL) resolves to that file and ingests its contents. Over `serve()` every string is sent as literal content, so the path itself becomes the document. To upload a file remotely, pass an open file object: `data=open("document.pdf", "rb")`.
* **`data` must be text or a file object.** Anything else raises a `TypeError`. As locally, a list is accepted only with exactly one item.
* **The return value is the server's raw JSON.** On the chunk-level path that is the same incremental summary you get locally, key for key. When the server falls back to the full delete-and-re-add flow, you get plain JSON keyed by dataset id rather than `PipelineRunInfo` objects.
The document keeps its `data_id` across a remote update, exactly as it does locally.
```python theme={null}
import cognee
client = await cognee.serve(url="https://your-tenant.aws.cognee.ai", api_key="your-api-key")
# Proxied to PATCH /api/v1/update on the connected instance.
response = await cognee.update(
data_id=uuid_of_remote_item,
data="Updated content for this document.",
dataset_id=uuid_of_remote_dataset,
)
await cognee.disconnect()
```
## Examples
```python theme={null}
import cognee
# Update a data item with new content
await cognee.update(
data_id=uuid_of_item,
data="Updated content for this document.",
dataset_id=uuid_of_dataset,
)
```
## Further details
<AccordionGroup>
<Accordion title="What incremental_loading skips">
`incremental_loading=True` (the default) tells the cognify step to skip data items that have already been processed successfully. Because `update()` deletes the old data item and adds a new one, the new item has no prior processing record and is always re-cognified. Unchanged documents already in the dataset retain their completed status and are skipped, so only the updated document is re-processed.
Set `incremental_loading=False` to force a full re-cognify of every document in the dataset — useful when you have changed your graph model or extraction prompt and want all content reprocessed.
```python theme={null}
# Force re-processing of every document in the dataset
await cognee.update(
data_id=uuid_of_item,
data="Updated content.",
dataset_id=uuid_of_dataset,
incremental_loading=False,
)
```
This applies to local mode only. Over a [`serve()`](#remote-mode-serve) connection the parameter never reaches the server, so the call runs with the instance's own settings and nothing is logged about it.
</Accordion>
<Accordion title="Shared nodes and relationship cleanup">
When the old data item is deleted, Cognee checks every node and edge it owned. Nodes that are *also* referenced by other documents in the same dataset are **preserved** — deleting one document does not break the rest of the graph. Only nodes and edges unique to the deleted data item are removed from both the graph database and the vector store.
After cognify re-runs on the updated content, new entities and relationships are extracted. Any relationship that existed in the old version but is absent from the new version will not be re-created, so the graph always reflects the current state of your data.
</Accordion>
<Accordion title="Observing graph changes">
Use [graph visualization](/guides/graph-visualization) or a [search query](/python-api/search) to inspect the knowledge graph before and after an update:
```python theme={null}
import cognee
from cognee import SearchType
# Query the graph after updating
results = await cognee.search(
"What entities are related to the updated topic?",
query_type=SearchType.GRAPH_COMPLETION,
)
print(results)
```
For a visual diff, launch the built-in graph explorer via `cognee-cli -ui` and compare the graph snapshots around the update.
</Accordion>
</AccordionGroup>
# cognee.users
Source: https://docs.cognee.ai/python-api/users
User management: get the default user, look up users, check existence, and create users
# User management
Helpers for resolving, looking up, and creating [users](/core-concepts/multi-user-mode/permissions-system/users). These live under `cognee.modules.users.methods` and are async, so `await` them.
```python theme={null}
from cognee.modules.users.methods import (
get_default_user,
get_user,
get_user_by_email,
get_user_id_by_email,
create_user,
)
```
## Methods
### get\_default\_user()
```python theme={null}
user = await get_default_user()
```
Returns the default user, creating it on first call if it does not exist yet. This is the same user Cognee resolves when you call `remember()`, `recall()`, etc. without passing `user=`.
The default email comes from `DEFAULT_USER_EMAIL` (falls back to `default_user@example.com`); the password used at creation comes from `DEFAULT_USER_PASSWORD` (falls back to `default_password`). The auto-created default user is a superuser. See the [Users concept page](/core-concepts/multi-user-mode/permissions-system/users) for the environment variables.
### get\_user()
```python theme={null}
user = await get_user(user_id)
```
Look up a user by UUID. Raises `EntityNotFoundError` if no user matches.
| Parameter | Type | Default | Notes |
| --------- | ------ | -------- | ---------------- |
| `user_id` | `UUID` | required | The user's UUID. |
### get\_user\_by\_email()
```python theme={null}
user = await get_user_by_email("alice@example.com")
```
Look up a user by email. Returns the `User` object, or `None` if no user has that email — use this for existence checks (see below).
| Parameter | Type | Default | Notes |
| ------------ | ----- | -------- | ----------------- |
| `user_email` | `str` | required | Email to look up. |
### get\_user\_id\_by\_email()
```python theme={null}
user_id = await get_user_id_by_email("alice@example.com")
```
Returns just the user's UUID for a given email, or `None` if no such user exists. Lighter than `get_user_by_email()` when you only need the id.
| Parameter | Type | Default | Notes |
| ------------ | ----- | -------- | ----------------- |
| `user_email` | `str` | required | Email to look up. |
### create\_user()
```python theme={null}
user = await create_user(
email="alice@example.com",
password="s3cret",
)
```
Creates a new user. Raises `fastapi_users.exceptions.UserAlreadyExists` if a user with that email already exists.
| Parameter | Type | Default | Notes |
| ---------------- | ---------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `email` | `str` | required | The new user's email (unique). |
| `password` | `str` | required | Plaintext password; stored hashed. |
| `is_superuser` | `bool` | `False` | Grants administrative privileges: manage other users/tenants/roles and access all datasets. |
| `is_active` | `bool` | `True` | Whether the account is active. Inactive users cannot authenticate. |
| `is_verified` | `bool` | `False` | Whether the email is treated as verified. |
| `auto_login` | `bool` | `False` | When `True`, refreshes the user record after creation so it is ready for an immediate login flow. |
| `parent_user_id` | `Optional[UUID]` | `None` | UUID of a parent user. Pass this when creating agent or service users so the parent automatically inherits permissions on any datasets those users create. Leave `None` for regular human users. |
<Note>
There is no global "list all users" helper. To enumerate the users in a [tenant](/core-concepts/multi-user-mode/permissions-system/tenants), use `get_users_in_tenant(tenant_id, user)` from `cognee.modules.users.tenants.methods`; the requesting `user` must have user-management permission on that tenant. It returns a list of dicts with `id`, `email`, and `roles`.
</Note>
## Examples
<AccordionGroup>
<Accordion title="Check if a user exists before creating">
`get_user_by_email()` (and `get_user_id_by_email()`) return `None` when no user matches, so you can check existence without a `try`/`except`:
```python theme={null}
from cognee.modules.users.methods import get_user_by_email, create_user
email = "alice@example.com"
user = await get_user_by_email(email)
if user is None:
user = await create_user(email=email, password="s3cret")
print("created", user.id)
else:
print("already exists", user.id)
```
If you prefer to attempt creation directly, catch `UserAlreadyExists`:
```python theme={null}
from fastapi_users.exceptions import UserAlreadyExists
from cognee.modules.users.methods import create_user, get_user_by_email
try:
user = await create_user(email=email, password="s3cret")
except UserAlreadyExists:
user = await get_user_by_email(email)
```
</Accordion>
<Accordion title="Get the default user and pass it to operations">
```python theme={null}
import cognee
from cognee.modules.users.methods import get_default_user
user = await get_default_user()
await cognee.add("Cognee turns data into memory.", user=user)
await cognee.cognify(user=user)
results = await cognee.search("What does Cognee do?", user=user)
```
Most top-level operations resolve the default user automatically when `user=` is omitted, so calling `get_default_user()` explicitly is only needed when you want the `User` object itself.
</Accordion>
<Accordion title="Create an agent user owned by a parent">
Pass `parent_user_id` so the parent user inherits permissions on datasets the agent creates:
```python theme={null}
from cognee.modules.users.methods import get_default_user, create_user
owner = await get_default_user()
agent = await create_user(
email="agent@example.com",
password="s3cret",
parent_user_id=owner.id,
)
```
See [Users](/core-concepts/multi-user-mode/permissions-system/users) for how parent/child permission inheritance works.
</Accordion>
</AccordionGroup>
# validate()
Source: https://docs.cognee.ai/python-api/validate
Cross-check a dataset's graph and vector stores for integrity problems
# cognee.validate()
```python theme={null}
async def validate(
dataset: Optional[Union[str, List[str]]] = "main_dataset",
user: Optional[User] = None,
) -> ValidationReport
```
## Description
Cross-check the graph and vector stores of a dataset and report where they disagree.
`validate()` answers three questions no other call answers: does every edge still point
at nodes that exist, does every deduplicated node still carry the id its own dedup
contract derives, and is every embeddable node actually present in its vector collection.
The check is **read-only** — it never writes to any store, so it is safe to run against a
production dataset. It is also **backend-agnostic**: it is driven entirely through
`GraphDBInterface.get_graph_data()` and `VectorDBInterface.retrieve()`, so it works
unmodified against every supported graph and vector backend without adapter-specific
code.
Run it after [`cognify()`](/python-api/cognify), after a large import, or after a
migration — the moments when the graph and the vector index can drift apart.
Only **one** graph is checked per call: every name in `dataset` must resolve to a dataset
you have `read` permission on, and the **first** of those determines which graph and vector
store are read. Passing several dataset names does not merge them into a combined report —
call `validate()` once per dataset instead.
A partially authorized list is rejected as a whole — `dataset=["mine", "not-mine"]` raises
rather than silently validating `mine` alone — and the default `dataset="main_dataset"`
raises too when that dataset does not exist for you yet. This does not depend on
`ENABLE_BACKEND_ACCESS_CONTROL`: the rejection is the same with access control on or off.
Note that names resolve only within datasets you **own**, so a dataset merely shared with
you is rejected when requested by name.
Passing `dataset=None` (or an empty list) still skips dataset resolution entirely — the only
path that enters no dataset context. With backend access control disabled that reads the
unscoped shared stores; with it enabled the call fails, since a dataset is required to
resolve the per-dataset databases.
<Warning>
**`validate()` fails closed on a dataset it cannot read.** If *any* of the given names does
not resolve to a dataset you have `read` permission on, the call raises
`DatasetNotFoundError` — message `"Dataset not found or not readable."`, importable from
`cognee.modules.data.exceptions` — *before* any graph or vector adapter is opened. There is
no fallback to the default or shared stores, so a report that comes back is always a report
about the dataset you asked for.
</Warning>
## Parameters
<ParamField type="Optional[Union[str, List[str]]]">Dataset name(s) to validate. A single string is treated as a one-element list.</ParamField>
<ParamField type="Optional[User]">User context for dataset access. Falls back to the default user.</ParamField>
## Returns
`ValidationReport` — a Pydantic model with three fields:
| Field | Type | Contents |
| --------- | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `status` | `ValidationStatus` | `"healthy"`, `"degraded"`, or `"unhealthy"`. |
| `summary` | `Dict[str, Any]` | `graph_nodes` (int), `graph_edges` (int), and `node_type_distribution` — a `{node type: count}` map over the whole graph, where typeless nodes are counted under `"unknown"`. |
| `issues` | `List[ValidationIssue]` | Every issue found, each with `severity` (`"error"` / `"warning"`), `type`, and a human-readable `detail` naming the node or edge. |
`status` is derived from the severities present, not from issue counts:
* any `error`-severity issue → **`unhealthy`**
* otherwise, any `warning`-severity issue → **`degraded`**
* no issues at all → **`healthy`**
`validate`, `ValidationReport`, `ValidationIssue`, and `ValidationStatus` are importable
from the `cognee` top level; the `IssueSeverity` and `IssueType` enums are available from
`cognee.api.v1.validate`. All three enums (`ValidationStatus`, `IssueSeverity`,
`IssueType`) subclass `str`, so comparing against plain strings
(`report.status == "healthy"`) works without importing them — but printing a member
shows the enum repr (`ValidationStatus.HEALTHY`), so use `.value` when you want the
plain string.
## What is checked
<AccordionGroup>
<Accordion title="orphaned_edge — error">
An edge whose `source` or `target` id is not in the graph's node set. Traversal that
reaches such an edge hits a dead end, so graph-based search silently loses the
connection.
The `detail` names the edge as `source -[relationship]-> target` along with the id(s)
that are missing. Typically the result of nodes being removed without their edges;
re-running `cognify()` for the affected dataset, or deleting and re-ingesting the
source data, rebuilds a consistent edge set.
</Accordion>
<Accordion title="identity_id_mismatch — warning">
A node of a type that declares `identity_fields` — `Entity` and `EntityType`, both
keyed on `name` — whose id does not equal the id its own class derives from its
properties via [`Type.id_for(...)`](/core-concepts/building-blocks/datapoints).
Cognee's dedup contract is that two nodes with the same identity value *are* the same
node because they hash to the same id. A node that reached the graph without going
through that contract (a raw write from an importer or a migration, or a node written
by an older Cognee version) can carry a stale or arbitrary id — which means a second,
correctly-derived node for the same entity can coexist as an undetected duplicate.
Hence a warning rather than an error: nothing is broken yet, but deduplication is no
longer guaranteed for that node.
The check is skipped for a node whose identity field is absent from its graph
properties — the id cannot be recomputed, and the node may simply predate the field.
To bring mismatched nodes onto the current scheme, re-cognify the affected datasets
from scratch so their ids are derived by the models.
</Accordion>
<Accordion title="missing_vector_entry — error">
A node of a type that declares `index_fields` — `Entity` (collection `Entity_name`)
and `DocumentChunk` (collection `DocumentChunk_text`) — with no matching point in its
`{type}_{index_field}` vector collection.
The node exists in the graph but is unreachable by every embedding-based search type,
silently: semantic search cannot surface it and cannot use it to seed graph traversal.
Re-running `cognify()` for the dataset re-indexes the missing nodes. If an entire node
type reports one issue per node, the collection itself is likely missing or empty
rather than individual points having been lost.
</Accordion>
</AccordionGroup>
The checked types are read from the real model classes' `identity_fields` / `index_fields`
metadata rather than being hardcoded, so a change to either contract is picked up here
automatically.
## Cost and when to run
* **No LLM calls** and no writes. The cost is entirely database reads.
* The graph is read in full through `get_graph_data()`, so time and memory scale with the
total number of nodes and edges — not with a sample. There is no sampling or limit
parameter.
* Vector lookups are **one batched `retrieve()` per collection** (at most two: `Entity_name`
and `DocumentChunk_text`), not one call per node.
* Because it is read-only, it is safe against production stores — but on a very large
dataset prefer a low-load window, and note that the number of `issues` is unbounded: a
badly drifted graph can return one issue per affected node or edge.
Good moments to run it: after `cognify()` or a bulk import, after applying migrations, as
a pre-flight check in CI, or on a schedule as integrity monitoring.
## Examples
```python theme={null}
import cognee
await cognee.add("docs/handbook.pdf", dataset_name="onboarding")
await cognee.cognify(datasets=["onboarding"])
report = await cognee.validate(dataset="onboarding")
print(report.status.value) # "healthy" | "degraded" | "unhealthy"
print(report.summary["graph_nodes"], report.summary["graph_edges"])
print(report.summary["node_type_distribution"]) # {"Entity": 128, "DocumentChunk": 14, ...}
for issue in report.issues:
print(f"[{issue.severity.value}] {issue.type.value}: {issue.detail}")
```
Acting on the status — for example, failing a CI pre-flight check on errors while letting
warnings through:
```python theme={null}
import cognee
from cognee import ValidationStatus
report = await cognee.validate(dataset="onboarding")
if report.status == ValidationStatus.UNHEALTHY:
errors = [i for i in report.issues if i.severity == "error"]
raise SystemExit(f"dataset integrity check failed with {len(errors)} error(s)")
if report.status == ValidationStatus.DEGRADED:
print(f"{len(report.issues)} warning(s) — deduplication may be incomplete")
```
Grouping issues by type to decide what to remediate:
```python theme={null}
from collections import Counter
report = await cognee.validate(dataset="onboarding")
print(Counter(issue.type.value for issue in report.issues))
# Counter({'missing_vector_entry': 12, 'identity_id_mismatch': 3})
```
## Over HTTP
The same check is exposed as a read-only endpoint on the API server:
```
GET /api/v1/validate
```
| Aspect | Behavior |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `dataset` query parameter | Repeatable; defaults to `main_dataset`. Pass `?dataset=a&dataset=b` to hand several names to the same resolution rule — every one must be readable by the caller, and the first is the one read. |
| Authentication | Required — the endpoint resolves the calling user and applies the same `read` permission check as the SDK function. |
| `200 OK` | Report body, when `status` is `healthy` or `degraded`. |
| `503 Service Unavailable` | Report body, when `status` is `unhealthy` — that is, whenever at least one `error`-severity issue was found. |
| `500 Internal Server Error` | `{"status": "error", "reason": "validation failed: ..."}` if the check itself raised — **including a dataset the caller cannot read**, which arrives as `"validation failed: DatasetNotFoundError: Dataset not found or not readable. (Status code: 404)"`. The `404` inside that string is the exception's own string form; the HTTP status is still `500`. |
<Warning>
The `503` is the single detail to plan for when wiring this into a health probe or a CI
check: an **`unhealthy` report is returned with a `503` status code**, not a `200`. Clients
that raise on non-2xx responses will treat a successful-but-failing validation as a
transport error, and a load balancer pointed at this path will pull the instance out of
rotation on a data-integrity finding. Read the response body to tell the two apart — a
`503` from this endpoint still carries the full report JSON, while a `500` carries
`{"status": "error", "reason": ...}`. The endpoint wraps *every* exception into that `500` —
an unreadable or unknown `dataset` name is a server error here, never a `404` — so if you
alert on rejected datasets, match the `reason` for `Dataset not found or not readable.`
rather than watching for a `4xx` status code.
</Warning>
```bash theme={null}
curl -i "$COGNEE_URL/api/v1/validate?dataset=onboarding" \
-H "Authorization: Bearer $COGNEE_TOKEN"
```
```json theme={null}
{
"status": "unhealthy",
"summary": {
"graph_nodes": 142,
"graph_edges": 310,
"node_type_distribution": { "Entity": 128, "DocumentChunk": 14 }
},
"issues": [
{
"severity": "error",
"type": "missing_vector_entry",
"detail": "Entity node '...' has no matching point in vector collection 'Entity_name' — it exists in the graph but is unreachable by semantic search."
}
]
}
```
## See also
* [`report()`](/python-api/report) — the other read-only diagnostic: what the graph *contains*, rather than whether it is consistent
* [`cognify()`](/python-api/cognify) — the pipeline that writes the nodes, edges, and vector points this call cross-checks
* [DataPoints](/core-concepts/building-blocks/datapoints) — `identity_fields`, `index_fields`, and the `id_for()` contract the identity check verifies
* [`run_migrations()`](/python-api/run-migrations) — apply pending schema migrations; validating afterwards confirms the stores still agree
# Visualization Payloads
Source: https://docs.cognee.ai/python-api/visualize
JSON payload builders behind the HTML visualization, and the HTTP endpoints that expose them
# Visualization payloads
[`visualize_graph()`](/guides/graph-visualization) renders a self-contained HTML page. The
functions on this page return the same data as plain dictionaries instead, so an external UI
or dashboard can render it itself. Both paths run through the same authorized, bounded graph
read and the same `preprocess()` step, so they cannot drift on the data — only on how each
packages it.
<Note>
These builders are **not** re-exported on the top-level `cognee` module. Import them from
`cognee.api.v1.visualize`.
</Note>
```python theme={null}
from cognee.api.v1.visualize import (
visualize_graph_json,
visualize_semantic_json,
build_brains_payload,
build_brains_summary_payload,
get_live_events,
get_memory_provenance_payload,
stream_dataset_updates,
)
```
## visualize\_graph\_json()
```python theme={null}
async def visualize_graph_json(
include_session_events: bool = True,
session_ids: list = None,
user: Optional[User] = None,
dataset: Optional[Union[str, UUID]] = "main_dataset",
*,
full: bool = False,
query: Optional[str] = None,
seed_node_ids: Optional[List[str]] = None,
recall_result: Optional[Any] = None,
neighborhood_depth: int = 2,
neighborhood_seed_top_k: int = 10,
max_nodes: int = 500,
) -> dict
```
The graph as a JSON-safe dict, **without** the semantic layout. Authorization, dataset
resolution and the bounded fetch are shared with `visualize_graph()`, so the same arguments
give you exactly the subgraph the HTML page would have shown — including the
[bounded-subgraph defaults](/guides/graph-visualization#advanced-usage).
### Parameters
<ParamField type="bool">Embed the caller's search and improve events in the payload as `search_events`, scoped to `dataset`. Sessions attributed to no dataset are not included.</ParamField>
<ParamField type="list">Restrict embedded session events to these sessions. An explicit list is an intentional override, used as given and not narrowed to `dataset`.</ParamField>
<ParamField type="Optional[User]">User context for dataset access. Falls back to the default user.</ParamField>
<ParamField type="Optional[Union[str, UUID]]">Dataset name or id to read.</ParamField>
<ParamField type="bool">Return the entire graph instead of a bounded subgraph.</ParamField>
<ParamField type="Optional[str]">Query string whose nearest vector hits seed the subgraph.</ParamField>
<ParamField type="Optional[List[str]]">Explicit seed node ids for neighborhood expansion.</ParamField>
<ParamField type="Optional[Any]">A `recall()` or search result whose graph provenance seeds the subgraph. Python-only — there is no HTTP equivalent.</ParamField>
<ParamField type="int">*k*-hop expansion depth around the seeds.</ParamField>
<ParamField type="int">Maximum number of seed nodes.</ParamField>
<ParamField type="int">Hard cap on nodes after expansion.</ParamField>
### Returns
`dict` — every field of the renderer's `PreprocessedGraph` snapshot, plus `search_events`:
| Key | Contents |
| --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| `nodes`, `links` | The graph itself, with per-node render metadata. |
| `color_maps` | Color assignments, keyed by `type` and `node_set`. |
| `schema_graph`, `schema_data` | The type-level view behind the Schema tab. |
| `memory_map` | The pipeline-structure view behind the Memory tab. |
| `pipeline_stages`, `edge_classes`, `bundles`, `provenance_index`, `has_meaningful_topological_rank` | Derived values the bundled renderer reads directly; a client replacing those modules needs them. |
| `search_events` | The caller's search and improve events for `dataset`, or `[]`. |
Semantic positions are deliberately absent — see below.
```python theme={null}
payload = await visualize_graph_json(query="natural language processing")
print(len(payload["nodes"]), len(payload["links"]))
```
## visualize\_semantic\_json()
```python theme={null}
async def visualize_semantic_json(
user: Optional[User] = None,
dataset: Optional[Union[str, UUID]] = "main_dataset",
*,
full: bool = False,
query: Optional[str] = None,
seed_node_ids: Optional[List[str]] = None,
recall_result: Optional[Any] = None,
neighborhood_depth: int = 2,
neighborhood_seed_top_k: int = 10,
max_nodes: int = 500,
) -> dict
```
Semantic positions and clusters for the same subgraph, computed **on demand**. Pass the same
dataset/seed/depth/cap arguments you passed to `visualize_graph_json()` to lay out the same
subgraph.
This is the one call that fetches embeddings and runs the PCA (or UMAP) projection over them,
bounded to `SEMANTIC_NODE_CAP` = 2000 nodes. The HTML render computes the layout on every
render; splitting it out here means a client that never opens a semantic view never pays for
it.
### Returns
`dict` with `semantic_positions` and `semantic_clusters`. Both are `null` together — the
layout is best-effort, so no embeddings resolving *or* the projection failing yields
`{"semantic_positions": null, "semantic_clusters": null}` rather than an error.
## build\_brains\_payload()
```python theme={null}
async def build_brains_payload(
user: Optional[User] = None,
max_nodes: int = 500,
) -> dict
```
A small graph preview for **every** dataset the caller may read — their own, their tenant's,
and anything granted to a role they belong to. There is no `dataset` argument: this is the
overview across brains, not one brain.
### Returns
`dict` keyed by dataset id, each value `{"name", "nodes", "links", "node_set_colors"}` — not
the full `visualize_graph_json()` shape, since an overview does not need one dataset's worth
of schema/memory/pipeline detail multiplied by every dataset. `max_nodes` is applied
independently per dataset; there is no larger combined cap.
## build\_brains\_summary\_payload()
```python theme={null}
async def build_brains_summary_payload(
user: Optional[User] = None,
) -> dict
```
The cheap counterpart of `build_brains_payload()`: the same datasets — the authorization
union is shared, so both list exactly the same brains — but only what an overview shows
(name, sources, size, colors), built from relational metadata and the per-cognify-run count
cache instead of one bounded graph read per dataset. A cold cache pays one count query per
cognify run whose count is not cached yet; every call after that pays no graph reads at all.
Reach for `build_brains_payload()` only when the node and link arrays themselves are needed.
### Returns
`dict` keyed by dataset id (as a string), each value
`{"name", "source_names", "node_count", "node_set_colors"}`:
| Key | Contents |
| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `name` | The dataset's name. |
| `source_names` | The dataset's distinct node set names, sorted; empty when the data was ingested without node sets. |
| `node_count` | Nodes in the dataset's graph as of its latest cognify run — the whole graph, not only entity nodes, and `0` for a dataset never cognified. |
| `node_set_colors` | Node set colors from the same rule `build_brains_payload()` uses. Same rule and same node sets give the same colors, but the two calls can be looking at different node sets — `/brains` reads them off its bounded graph fetch (and so sees sets that exist only in the graph, such as the memory sets `improve()` writes), this one reads them from the dataset's relational rows — and where the sets differ the colors do too. |
## get\_live\_events()
```python theme={null}
async def get_live_events(
dataset_id: UUID,
since: Optional[datetime] = None,
user: Optional[User] = None,
) -> Dict[str, Any]
```
The delta of search and improve events since a cursor, for refreshing a timeline without
rebuilding the whole graph payload.
`dataset_id` both gates and scopes. It gates with the same read-permission check every other
visualization entry point runs, and it scopes the events: only the caller's own sessions
attributed to that dataset contribute, matching the `search_events` already embedded in
`visualize_graph_json()` for the same dataset. A session attributed to no dataset — the plain
global `default_session` an unscoped or cross-dataset search runs in — contributes to no
dataset's timeline, since it could belong to any dataset the caller has queried. Attribution
is per session, not per answered turn: a session id reused across datasets stays with the
first dataset it touched, so its later turns appear on that dataset's timeline. Raises
`PermissionDeniedError` when the dataset does not exist or the caller cannot read it.
### Returns
`{"events": [...], "cursor": <ISO datetime string or None>}`. Omit `since` on the first call
to get everything available, then pass the previous response's `cursor` straight back as the
next `since` — the filter is strict (`>`, not `>=`), so no event is delivered twice. When
nothing new has happened the response echoes back the `since` you sent, or `null` if you sent
none.
## stream\_dataset\_updates()
```python theme={null}
async def stream_dataset_updates(
websocket: WebSocket,
dataset_id: UUID,
user: User,
since: Optional[datetime] = None,
) -> None
```
The push loop behind `WS /api/v1/visualize/subscribe/{dataset_id}` — sends a `ready` frame,
then pushes `live_events`, `graph_grew` and `heartbeat` frames until the client disconnects,
so a client can stop polling `get_live_events()` and refetching the graph payload entirely.
See [Live dataset updates](/cognee-cloud/functionality/dataset-management#live-dataset-updates)
for the frame shapes, cadences, and close codes.
Exposed for deployments that mount their own WebSocket route. It expects an **already
accepted, already authorized** connection — the caller owns accept and close, so a rejection
can carry a close code the client will actually see — and raises `PermissionDeniedError` when
read access to the dataset is lost while the stream is running (permission is re-checked on
every poll).
## get\_memory\_provenance\_payload()
```python theme={null}
async def get_memory_provenance_payload(
include_memory: bool = False,
scope_tenant_ids: Optional[List[Any]] = None,
scope_user_ids: Optional[List[Any]] = None,
) -> dict
```
The [memory-provenance graph](/guides/memory-provenance) — the ownership and data-flow story
read purely from the relational database — packaged as a dict. Same scoping rules as
[`get_memory_provenance_graph()`](/guides/memory-provenance#scoping-in-multi-tenant-deployments):
in multi-tenant deployments you **must** pass a scope, or the read spans every tenant.
### Returns
`dict` in the same shape `visualize_graph_json()` returns (`nodes`, `links`, `color_maps`,
`schema_graph`, `memory_map`, and the rest) — not the raw `(nodes, edges)` tuple
`get_memory_provenance_graph()` returns.
## Over HTTP
Each builder above has an HTTP endpoint in front of it. `GET /api/v1/visualize/json`,
`GET /api/v1/visualize/semantic` and `GET /api/v1/schema/provenance/json` are JSON siblings of
existing HTML endpoints; `/brains`, `/brains-summary` and `/live-events` have no HTML
equivalent. `POST /api/v1/visualize/multi` remains HTML-only, and `stream_dataset_updates()`
fronts a WebSocket route rather than a GET.
| Endpoint | Returns |
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `GET /api/v1/visualize/json` | `visualize_graph_json()`'s payload. No semantic layout. |
| `GET /api/v1/visualize/semantic` | `semantic_positions` and `semantic_clusters` for the same subgraph. |
| `GET /api/v1/visualize/brains` | `build_brains_payload()`'s per-dataset previews. Takes no `dataset_id` — only `max_nodes`. |
| `GET /api/v1/visualize/brains-summary` | `build_brains_summary_payload()`'s per-dataset overviews. Takes no parameters. |
| `GET /api/v1/visualize/live-events` | `{"events": [...], "cursor": ...}` for the caller's session events attributed to `dataset_id`. |
| `WS /api/v1/visualize/subscribe/{dataset_id}` | `stream_dataset_updates()`'s frames, pushed instead of polled — see [Live dataset updates](/cognee-cloud/functionality/dataset-management#live-dataset-updates). Takes an optional `since` reconnect cursor. |
| `GET /api/v1/schema/provenance/json` | `get_memory_provenance_payload()`'s payload, always scoped to the authenticated caller's tenant (or user). Takes `include_memory`. |
All of them require authentication. The routes that take a `dataset_id` enforce the same read
permission as `GET /api/v1/visualize`; `/brains` and `/brains-summary` take none and simply
return the datasets the caller can read. The WebSocket route also accepts the API key or
bearer token as a `?token=` query parameter, since a browser cannot set headers on a
WebSocket handshake — mind that the handshake URL, query string included, shows up in the
default access logs of common reverse proxies.
`/json` and `/semantic` accept the same query params as `GET /api/v1/visualize` —
`dataset_id` (required), `full`, `query`, `seed_node_ids`, `neighborhood_depth`,
`neighborhood_seed_top_k` and `max_nodes` — with the defaults listed above. Pass identical
arguments to both to get a graph and its semantic layout for the same subgraph.
```bash theme={null}
curl "$COGNEE_URL/api/v1/visualize/json?dataset_id=$DATASET_ID&query=natural+language+processing"
```
```json theme={null}
{
"nodes": [ ... ],
"links": [ ... ],
"color_maps": { "type": { ... }, "node_set": { ... } },
"schema_graph": { "nodes": [ ... ], "links": [ ... ] },
"memory_map": { ... },
"search_events": []
}
```
### Errors
JSON error bodies carry a fixed message and never the exception text — full detail is
server-logged instead.
* **403** — `/live-events` only, when the caller lacks read permission on the dataset (or it
does not exist).
* **409** — the payload could not be built. Note that a semantic layout that cannot be
computed is **not** an error: `/semantic` answers **200** with both fields `null`, and only
returns 409 if the underlying graph fetch itself fails.
The WebSocket route signals failure with close codes instead: **1008** when the caller is not
authenticated or lacks read permission on the dataset (a retry replays the same rejection),
**1011** when the stream failed server-side (reconnecting is reasonable).
## See also
* [Graph Visualization](/guides/graph-visualization) — rendering the same data to an interactive HTML file
* [Reading the Visualization](/guides/reading-the-visualization) — what each tab of that file shows, and which one to reach for
* [Memory Provenance](/guides/memory-provenance) — the ownership and data-flow projection behind `/schema/provenance`
* [Schema Inventory](/guides/schema-inventory) — a per-type summary of the graph, also available over HTTP
# Rust SDK Architecture
Source: https://docs.cognee.ai/rust/architecture
The single source of truth for the cognee-rust workspace layout, crate breakdown, cross-cutting design patterns, and key dependencies.
cognee-rust is a Rust port of the Python [cognee](https://github.com/topoteretes/cognee)
library — an AI memory pipeline that turns raw data into persistent, queryable
knowledge graphs. It targets both edge devices (Android, embedded) with local
models and drop-in parity with the Python `cognee` SDK (90%+ correctness).
**Core pipeline:** `add (ingest)` → `cognify (knowledge-graph extraction)` →
`search (context retrieval)`. See [operations](/rust/operations) for what each
operation does and [configuration](/rust/configuration) for how to configure them.
This page is the single source of truth for the workspace layout, the crate
breakdown, and the cross-cutting design patterns. `.claude/CLAUDE.md` and the
root `README.md` link here rather than duplicating it.
## Workspace structure
```
cognee-rust-oss/
├── Cargo.toml # Workspace root (edition 2024, resolver 3)
├── crates/
│ ├── models/ # Core data types: Data, Dataset, DataInput, Document, DocumentChunk
│ ├── storage/ # File storage abstraction (StorageTrait, LocalStorage)
│ ├── database/ # Metadata DB abstraction (IngestDb/SearchHistoryDb/DeleteDb)
│ ├── ingestion/ # Ingest pipeline + content hashing + URL crawler
│ ├── chunking/ # Text chunking (word→sentence→paragraph→TextChunker)
│ ├── cognify/ # Full cognify pipeline + memify enrichment pipeline
│ ├── search/ # Search pipeline with multiple retrieval strategies
│ ├── session/ # Session management and session store
│ ├── embedding/ # Multi-provider embedding engine (ONNX, OpenAI, Ollama, Mock)
│ ├── llm/ # LLM provider abstraction (OpenAI-compatible API adapter)
│ ├── graph/ # Graph DB abstraction (Ladybug embedded graph)
│ ├── vector/ # Vector DB abstraction (LanceDB default; brute-force on Android; pgvector feature-gated)
│ ├── ontology/ # Ontology resolution (RDF/JSON-LD loader, NoOp resolver)
│ ├── delete/ # Dataset/data deletion across all backends
│ ├── core/ # Task pipeline orchestration framework
│ ├── http-server/ # axum HTTP server (library + cognee-http-server binary)
│ ├── visualization/ # Self-contained HTML knowledge-graph visualization (d3.js)
│ ├── observability/ # OpenTelemetry tracing pipeline (OTLP exporter, telemetry feature)
│ ├── telemetry/ # Product-analytics client (send_telemetry → prometh.ai, opt-out)
│ ├── logging/ # Shared file logging (rotation, Python-compatible plain formatter)
│ ├── lib/ # Top-level library aggregating all crates (public api/ module)
│ ├── bindings-common/ # Shared SDK facade for the JS (Neon) + C-API bindings
│ ├── cli/ # CLI binary (cognee-cli)
│ ├── bench/ # Criterion benchmarks (add + cognify + search pipeline)
│ ├── utils/ # Shared utilities
│ └── test-utils/ # Mock implementations (MockStorage, MockGraphDB, MockVectorDB)
├── capi/ # C API bindings (FFI)
├── ts/ # JavaScript/TypeScript/Node bindings (Neon)
├── python/ # Python bindings (PyO3)
├── examples/ # Usage examples using the cognee crates
├── demo/ # Demo scripts (host and Android)
├── scripts/ # Build, check, and deployment scripts
├── docs/ # Documentation (this folder)
├── e2e-cross-sdk/ # Cross-SDK E2E tests (Rust ↔ Python interop)
└── .github/workflows/ # CI (ci.yml, oss-isolation.yml, publish-dry-run.yml,
# release-plz.yml, capi-release.yml, ts-prebuild.yml,
# http-parity.yml)
```
## Crate breakdown
**cognee-models** — Core data types shared across crates: `Data`, `Dataset`, `DataInput`, `Document`, `DocumentChunk`, `Entity`, `KnowledgeGraph`, etc. Pure data structures, no traits.
**cognee-storage** — Abstract file storage layer. Trait: `StorageTrait` (+ `StorageExt`, `StorageWriter`). Impls: `LocalStorage`, `MockStorage`.
**cognee-database** — Database abstraction for metadata persistence. Traits: `IngestDb`, `SearchHistoryDb`, `DeleteDb`. Impl: `DatabaseConnection` (SQLite/Postgres via SeaORM, implements all three traits).
**cognee-ingestion** — Pipeline for ingesting data: streams content, computes hashes, deduplicates, and stores. Main type: `AddPipeline`. No trait abstraction — uses `StorageTrait` and `IngestDb` from sibling crates.
**cognee-chunking** — Text chunking strategies ported from the Python chunking hierarchy (word → sentence → paragraph). Main entry point: `ExtractTextChunksPipeline`. Trait: `TokenCounter`. Impls: `WordCounter` (whitespace fallback), `HuggingFaceTokenCounter` (BPE/WordPiece, behind `hf-tokenizer`), `TikTokenCounter` (cl100k\_base BPE, behind `tiktoken`). `TokenCounterKind::from_env()` auto-selects the counter based on `EMBEDDING_PROVIDER` and `COGNEE_TOKEN_COUNTER`.
**cognee-cognify** — Knowledge-graph extraction pipeline: classify documents, chunk text, extract entities/relationships via LLM, summarize, store to graph and vector DBs. Entry points: `cognify()` / `cognify_datasets()`; main types: `CognifyConfig`, `CognifyInput`, `CognifyResult`, `FactExtractor`, `SummaryExtractor`. Also houses the **memify** sub-module (`MemifyConfig`, `MemifyResult`, `memify()`): reads the existing graph, creates triplet embeddings, indexes them for `SearchType::TripletCompletion`.
**cognee-search** — Unified search orchestration across multiple retrieval strategies. Main types: `SearchBuilder`, `SearchOrchestrator`. `SearchType` enum defines 15 search modes with corresponding retriever implementations.
**cognee-session** — Session management and QA-history storage. Trait: `SessionStore`. Impls: `FsSessionStore`, `RedisSessionStore`, `SeaOrmSessionStore`.
**cognee-embedding** — Text vectorization engine. Trait: `EmbeddingEngine`. Impls: `OnnxEmbeddingEngine` (local ONNX Runtime, BGE-Small-v1.5), `OpenAICompatibleEmbeddingEngine` (OpenAI/Azure/vLLM/llama.cpp/TEI via HTTP), `OllamaEmbeddingEngine`, `MockEmbeddingEngine`. `EmbeddingConfig::from_env()` + `create_engine()` factory select the provider. See [configuration](/rust/configuration#embedding).
**cognee-llm** — Async LLM abstraction with structured JSON output. Trait: `Llm` (+ auto-implemented `LlmExt`). Impls: `OpenAIAdapter` (OpenAI-compatible APIs, works with Ollama/vLLM), `MockLlm` (cassette-backed, `testing` feature). The on-device LiteRT adapter lives in the closed `cognee-llm-litert` crate shipped as part of `cognee-cloud-rs` and is not part of OSS.
**cognee-graph** — Graph database abstraction for knowledge-graph storage and traversal. Trait: `GraphDBTrait` (+ `GraphDBTraitExt`). Impls: `LadybugAdapter` (embedded Ladybug), `PgGraphAdapter` (feature `postgres`), `MockGraphDB`. Concurrency: Rust matches Python's default single-owning-process model for file-backed Ladybug; cross-process locking is intentionally out of scope (see [roadmap/](https://github.com/topoteretes/cognee-rs/blob/main/docs/roadmap/README.md)).
**cognee-vector** — Vector database abstraction for similarity search. Trait: `VectorDB`. Impls: `LanceDbAdapter` (embedded Apache-Arrow / Lance, on-disk; default on non-Android targets), `BruteForceVectorDB` (pure-Rust in-memory; default on Android and via `vector_db_url = ":memory:"`), `PgVectorAdapter` (Postgres + pgvector extension, feature `pgvector`), `MockVectorDB`. The embedded Qdrant adapter lives in the closed `cognee-vector-qdrant` crate shipped as part of `cognee-cloud-rs` and is not part of OSS.
**cognee-ontology** — RDF/OWL ontology integration for entity validation. Trait: `OntologyResolver`. Impls: `RdfLibOntologyResolver`, `NoOpOntologyResolver` (pass-through).
**cognee-delete** — Cascading deletion of data/datasets across all backends (relational → graph → vector → file storage). Main types: `DeleteService`, `AuthorizedDeleteService`.
**cognee-core** — Async runtime, task scheduling, and pipeline-execution primitives. Traits: `PipelineWatcher`, `ExecStatusManager`. Impls: `NoopWatcher`, `RayonThreadPool`, `NoopExecStatusManager`.
**cognee-http-server** — `axum`-based HTTP server. Library exposes `build_router`, `run`, and `AppState`; also builds the `cognee-http-server` binary. Routers mirror the Python FastAPI surface under `/api/v1/*`. See [http-server/](/rust/http-server).
**cognee-visualization** — Self-contained HTML knowledge-graph visualization (d3.js v7, force-directed, Canvas). Entry points: `visualize`/`render`/`render_multi_user`. Surfaces via the CLI `visualize` subcommand.
**cognee-observability** — OpenTelemetry tracing pipeline. Bridges `#[tracing::instrument]` sites into an OTLP exporter. Entry point: `init_telemetry` (tracing layer + RAII `TelemetryGuard`). Activated by `COGNEE_TRACING_ENABLED=true` or a non-empty `OTEL_EXPORTER_OTLP_ENDPOINT`; real exporter behind the `telemetry` feature. See [observability/opentelemetry.md](https://github.com/topoteretes/cognee-rs/blob/main/docs/observability/opentelemetry.md).
**cognee-telemetry** — Product-analytics client (`send_telemetry`). Fire-and-forget POST to `https://test.prometh.ai` per public API call; opt out with `TELEMETRY_DISABLED`, `ENV=test|dev`, or `--no-default-features`. See [observability/send\_telemetry.md](https://github.com/topoteretes/cognee-rs/blob/main/docs/observability/send_telemetry.md).
**cognee-logging** — Shared file-based logging: rotation, the Python-compatible plain-text formatter, and a noise-suppressing `EnvFilter`. Entry point: `init_logging`, called by the CLI and HTTP server. Env-var surface documented in [Configuration: logging](/rust/configuration#logging).
**cognee-bench** — Criterion benchmark crate (`batch_add_cognify`) exercising the add + cognify + search pipeline.
**cognee-bindings-common** — Shared SDK facade for the Neon JS and C-API bindings: `SdkError` (+ `code()`), `HandleState`, `CogneeServices`, and neon-free JSON wire helpers. Not a new user-facing Rust API — that remains `cognee_lib::api`.
**cognee-lib** — Unified public API facade. Re-exports all crates and adds an `api/` module mirroring the Python SDK: `forget`, `update`, `prune`, `recall`, `remember`, `improve`, plus `DatasetManager`. Houses the shared `Settings`/`ConfigManager` and runtime setters.
**cognee-cli** — Command-line binary (`cognee-cli`). See [tools/cli](/rust/tools/cli).
**cognee-utils** — Shared utilities: retry logic, deterministic ID generation (`generate_node_id`, `NAMESPACE_OID`, …), secret redaction (`redact`), and tracing attribute keys.
**cognee-test-utils** — Test helpers and mock implementations for integration tests.
## Architecture patterns
* **Feature strategy** — Individual crates define optional features with no defaults (`default = []`). The umbrella library (`cognee-lib`) and the CLI (`cognee-cli`) enable all non-platform-specific features by default, so a plain `cargo build` gives a fully-featured binary. Platform- and deployment-specific extras (e.g. on-device LiteRT inference, embedded Qdrant) ship in the closed `cognee-cloud-rs` companion repo; the `testing` feature stays opt-in. New feature-gated capabilities should be propagated up through `cognee-lib`/`cognee-cli` defaults unless platform- or test-only.
* **Trait-based abstractions** — `StorageTrait`, `IngestDb`, `GraphDBTrait`, `VectorDB`, `EmbeddingEngine`, `Llm`, `SessionStore`, etc. enable backend swapping and mock testing.
* **Prefer `dyn Trait`** — object-safe traits via `&dyn Trait` / `Arc<dyn Trait>` at call sites; monomorphized generics only when performance-critical.
* **Zero-copy where possible** — `WordChunk<'a>`, `SentenceChunk<'a>`, `ParagraphChunk<'a>` borrow `&str` slices via byte-offset tracking.
* **`Arc` for shared ownership** — `Arc<dyn Trait>` in pipelines; `Arc<Mutex<T>>` in mocks.
* **Async-first** — all I/O via tokio; trait methods are `async` since components may be local or remote.
* **Streaming-first** — `DataInput::process_by_chunks()`, `StorageTrait::store_stream()`, `ContentHasher::hash_content_stream()` avoid loading full files into memory.
* **Deterministic IDs** — same content + owner ⇒ same UUID via `uuid5(NAMESPACE_OID, …)` (content-addressed dedup). Chunk IDs: `uuid5(NAMESPACE_OID, "{document_id}-{chunk_index}")`.
* **Error types per crate** — each crate defines its own `thiserror` enum (`StorageError`, `ChunkingError`, `IngestionError`, …).
## Key dependencies
| Crate | Purpose |
| ---------------------------------------------------------------- | -------------------------------------------------- |
| `tokio` | Async runtime |
| `sea-orm` (SQLite, Postgres) | Relational DB ORM (metadata, sessions, provenance) |
| `ort` (ONNX Runtime) | Local model inference (embeddings) |
| `lbug` | Embedded graph database (Ladybug) |
| `reqwest` (rustls-tls) | HTTP client (URL crawling, LLM/embedding APIs) |
| `scraper` | HTML parsing for URL ingestion |
| `sophia` / `sophia_turtle` / `sophia_jsonld` | RDF/OWL ontology parsing |
| `uuid` (v4, v5) / `sha2` | ID generation / content hashing |
| `serde` / `serde_json` / `schemars` | Serialization + JSON schema |
| `tokenizers` / `tiktoken-rs` | Tokenization (embedding + chunking token counters) |
| `tracing` / `tracing-subscriber` | Structured logging + instrumentation |
| `opentelemetry` / `opentelemetry-otlp` / `tracing-opentelemetry` | OTLP trace export (`telemetry` feature) |
| `axum` / `tower` / `tower-http` | HTTP server |
| `async-trait` / `thiserror` / `clap` / `criterion` | Trait async / errors / CLI / benchmarks |
| `pyo3` / `neon` | Python / JavaScript bindings |
## Browsing the API docs (rustdoc)
API and type detail are documented inline in the code and rendered by rustdoc —
the rest of these docs link to it rather than restating signatures.
```bash theme={null}
# Build & open the whole workspace's API docs (no external deps):
cargo doc --no-deps --open
# Or a single crate:
cargo doc -p cognee-cognify --no-deps --open
```
CI already runs `cargo doc --no-deps` on every push (no hosted docs.rs site —
build locally). Each crate's `lib.rs` carries a top-level `//!` summary; start
from `cognee-lib` (the facade) and follow the re-exports.
| Area | Crate (package) | Start at |
| ----------------- | -------------------- | --------------------------------------------------- |
| Public SDK facade | `cognee-lib` | `api` module, `ConfigManager` |
| Ingest | `cognee-ingestion` | `AddPipeline` |
| Chunking | `cognee-chunking` | `TokenCounter`, `text_chunker` |
| Cognify / memify | `cognee-cognify` | `cognify`, `memify`, `CognifyConfig` |
| Search | `cognee-search` | `SearchBuilder`, `SearchType` |
| Embedding | `cognee-embedding` | `EmbeddingEngine`, `EmbeddingConfig` |
| LLM | `cognee-llm` | `Llm`, `OpenAIAdapter` |
| Graph | `cognee-graph` | `GraphDBTrait`, `LadybugAdapter` |
| Vector | `cognee-vector` | `VectorDB`, `BruteForceVectorDB`, `PgVectorAdapter` |
| Delete | `cognee-delete` | `DeleteService` |
| HTTP server | `cognee-http-server` | `build_router`, `run`, `AppState` |
# Core Concepts
Source: https://docs.cognee.ai/rust/concepts
The vocabulary behind cognee-rust: the stores that hold memory, the building blocks that produce it, and the terms shared across the API, CLI, and config.
The vocabulary behind cognee-rust: the stores that hold memory, the building
blocks that produce it, and the terms that show up across the API, CLI, and
config. This page is the conceptual map — API/type detail lives in rustdoc
(`cargo doc --no-deps --open`). For *what the system does* see
[operations](/rust/operations); for *how it fits together* see
[architecture](/rust/architecture).
## Architecture: three stores
Cognee keeps memory in three complementary backends. Every cognify run writes to
all three; search reads across them.
| Store | Role | Crate | Default backend |
| -------------- | --------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| **Relational** | Document tracking, deduplication, provenance/lineage, sessions | [`cognee-database`](https://github.com/topoteretes/cognee-rs/blob/main/crates/database/) | SQLite via SeaORM (Postgres supported) |
| **Vector** | Semantic similarity over embeddings (chunks, entities, summaries, triplets) | [`cognee-vector`](https://github.com/topoteretes/cognee-rs/blob/main/crates/vector/) | LanceDB embedded (brute-force on Android / `:memory:`; pgvector via feature) |
| **Graph** | Entity relationships — the knowledge graph itself | [`cognee-graph`](https://github.com/topoteretes/cognee-rs/blob/main/crates/graph/) | Embedded Ladybug |
Backend selection and connection settings are covered in
[tools/backends](/rust/tools/backends) and
[configuration](/rust/configuration); the layering is in
[architecture](/rust/architecture).
## Building blocks
### DataPoints
A **DataPoint** is the base storage-layer unit: a structured record that carries
a stable UUID, timestamps, a `type` discriminator, free-form `metadata`, and
provenance fields (`source_pipeline`, `source_task`, `source_node_set`,
`source_content_hash`). Typed graph nodes — `Entity`, `EntityType`, `EdgeType`,
`DocumentChunk`, etc. — embed a DataPoint as their `base`, exposed through the
`HasDataPoint` trait so provenance stamping can walk any node uniformly. When a
DataPoint is indexed, its serialized form becomes the vector-store payload
(`vector_metadata()`), keeping the on-disk shape comparable to Python's.
Rust: `DataPoint` / `HasDataPoint` in [`cognee-models`](https://github.com/topoteretes/cognee-rs/blob/main/crates/models/).
### Tasks
A **Task** is one reusable unit of work that transforms data — classify, chunk,
extract, summarize, embed. Tasks come in eight execution flavours (sync/async ×
single/iterator/stream × single-value/batch) so a step can stream, fan out, or
process whole batches. They are composed with optional per-task config
(`TaskInfo`: name, batch size, weight, rate limiter) and the pipeline executor
routes values between them.
Rust: `Task` / `TypedTask` / `TaskInfo` in [`cognee-core`](https://github.com/topoteretes/cognee-rs/blob/main/crates/core/).
### Pipelines
A **Pipeline** is an orchestrated sequence of Tasks with shared context
(database, graph, vector, cancellation, progress) and a watcher for status
events. The concrete pipelines are:
| Pipeline | What it composes | Crate / entry point |
| ----------- | --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| **add** | ingest → hash → dedup → persist | [`cognee-ingestion`](https://github.com/topoteretes/cognee-rs/blob/main/crates/ingestion/) (`AddPipeline`) |
| **cognify** | classify → chunk → extract → summarize → index → FK edges | [`cognee-cognify`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/) (`cognify()`) |
| **memify** | read graph → build triplets → embed → index | [`cognee-cognify`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/) (`memify()`) |
| **search** | route query → retrieve → (optionally) complete | [`cognee-search`](https://github.com/topoteretes/cognee-rs/blob/main/crates/search/) (`SearchOrchestrator`) |
Pipeline orchestration primitives (`PipelineWatcher`, `ExecStatusManager`,
thread pool) live in [`cognee-core`](https://github.com/topoteretes/cognee-rs/blob/main/crates/core/). The end-to-end flow is
described in [operations](/rust/operations).
## Key concepts
### Datasets
A **Dataset** is the organizational scope for memory operations: named, owned by
a user, optionally tenant-scoped. Data and DataPoints belong to one or more
datasets (`DataPoint.belongs_to_set`), and add / cognify / search / delete all
operate within a dataset scope. Dataset IDs are deterministic (UUID5 of name +
owner) for cross-SDK reproducibility.
Rust: `Dataset` in [`cognee-models`](https://github.com/topoteretes/cognee-rs/blob/main/crates/models/); lifecycle helpers in
`DatasetManager` ([`cognee-lib`](https://github.com/topoteretes/cognee-rs/blob/main/crates/lib/) `api::datasets`).
### Sessions
A **Session** is a temporary memory context — search/answer history and feedback
for a single conversational thread — distinct from permanent, graph-backed
storage. Passing a `--session-id` to `remember` / `recall` scopes a turn to that
session and lets retrieval reuse prior context; omitting it persists input as
permanent graph memory (see the memory API in [operations](/rust/operations)).
The store backend is pluggable.
Rust: `SessionStore` trait in [`cognee-session`](https://github.com/topoteretes/cognee-rs/blob/main/crates/session/), with
`FsSessionStore` (feature `fs`), `RedisSessionStore`, and `SeaOrmSessionStore`
backends.
### Node Sets
A **node set** is a tag attached to ingested data and the DataPoints derived from
it, used to categorize and later scope the knowledge base. It is **partially
realized** in Rust today:
* **Tagging at ingest** — `add` accepts a `node_set` (stored on `Data.node_set`
and propagated to derived DataPoints as `source_node_set`); the pipeline
executor can attach `node_set` provenance to task outputs (`Tagged` /
`TaggedMeta` in [`cognee-core`](https://github.com/topoteretes/cognee-rs/blob/main/crates/core/)).
* **Scoping memify** — memify enrichment can be restricted to a subset of the
graph by node *type* and node *name* via `--node-type` / `--node-name`
(`MemifyConfig::with_node_type_filter` / `with_node_name_filter`, backed by the
graph trait's `get_nodeset_subgraph`). The internal `persist_sessions` step
tags cached session data with a fixed node set.
Note: this is type/name-based subgraph filtering rather than a fully general
named-node-set query surface; treat node sets as a tagging-and-scoping primitive,
not a finished feature.
### Ontologies
An **ontology** grounds extracted entities in external, structured knowledge.
cognee-rust loads RDF/OWL ontologies (Turtle, RDF/XML, N-Triples, JSON-LD) and
uses them for fuzzy entity matching and subgraph enrichment during cognify. The
default is a no-op resolver (no grounding), matching Python's
`ontology_file=None`.
Rust: `OntologyResolver` trait in [`cognee-ontology`](https://github.com/topoteretes/cognee-rs/blob/main/crates/ontology/), with
`NoOpOntologyResolver` (default) and `RdfLibOntologyResolver`. Enabled per run
via cognify's `--ontology-file`; see [configuration](/rust/configuration).
### Loaders & Chunkers
**Loaders** handle file-format reading at ingest: a loader registry dispatches by
MIME type / extension to per-format loaders (text, PDF, CSV, HTML, image, audio,
and the `unstructured` office formats), most behind feature flags. **Chunkers**
then segment a document into token-bounded pieces through a word → sentence →
paragraph hierarchy, sizing chunks with a pluggable `TokenCounter`
(`WordCounter`, or the feature-gated HuggingFace / tiktoken counters).
Rust: loaders (`LoaderRegistry`, `DocumentLoader`) in
[`cognee-ingestion`](https://github.com/topoteretes/cognee-rs/blob/main/crates/ingestion/); chunking (`text_chunker`,
`TokenCounter`) in [`cognee-chunking`](https://github.com/topoteretes/cognee-rs/blob/main/crates/chunking/). Token-counter
selection is configured in [configuration](/rust/configuration).
## See also
* [operations](/rust/operations) — what the pipelines and memory API actually do
* [configuration](/rust/configuration) — env vars and runtime config for every concept above
* [architecture](/rust/architecture) — crate layering and design patterns
* [tools/backends](/rust/tools/backends) — choosing relational / vector / graph backends
* [roadmap/README.md](https://github.com/topoteretes/cognee-rs/blob/main/docs/roadmap/README.md) — what is partial or not yet implemented
# Configuration
Source: https://docs.cognee.ai/rust/configuration
Reference for configuring cognee-rust: how settings resolve and the env vars, defaults, and runtime API for every subsystem.
Canonical reference for configuring cognee-rust. The complete, field-level source
of truth is the [`Settings`](https://github.com/topoteretes/cognee-rs/blob/main/crates/lib/src/config.rs) struct and the
[`ConfigManager`](https://github.com/topoteretes/cognee-rs/blob/main/crates/lib/src/config.rs) runtime API — build the rustdoc with
`cargo doc -p cognee-lib --no-deps --open` to browse every field and setter with
its type. This page groups those fields by subsystem and gives the env-var name
and default for each.
## How configuration resolves
Three layers, lowest precedence first:
1. **Defaults** — `Settings::default()` in [`crates/lib/src/config.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/lib/src/config.rs).
2. **Persisted config file** *(CLI only)* — JSON at `~/.config/cognee-rust/config.json`
(`$XDG_CONFIG_HOME/cognee-rust/config.json`), managed by `cognee-cli config`.
See [`crates/cli/src/config_store.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cli/src/config_store.rs).
3. **Environment variables** — bound by `Settings::overlay_from_env()`. A `.env`
file in the working directory (or any ancestor) is loaded automatically via `dotenv`.
So: `defaults < config.json < env`. At runtime, code can also mutate settings
through `ConfigManager`'s `set_*` methods (below) or the binding config APIs.
Parsing notes: booleans accept `true|1|yes` / `false|0|no` (`cognee_utils::parse_env_bool`);
empty env values are treated as unset; numeric vars that fail to parse are ignored.
## LLM
Read by the LLM adapter. The deep reference is [tools/cli](/rust/tools/cli#llm-retries)
for retries and the `cognee-llm` rustdoc for the adapter.
| Env var (aliases) | `Settings` field | Default |
| ---------------------------------------------- | --------------------------- | ------------------- |
| `LLM_PROVIDER` | `llm_provider` | `openai` |
| `LLM_MODEL` / `OPENAI_MODEL` | `llm_model` | `openai/gpt-5-mini` |
| `LLM_API_KEY` / `OPENAI_TOKEN` | `llm_api_key` | *(empty)* |
| `LLM_ENDPOINT` / `OPENAI_URL` | `llm_endpoint` | *(empty)* |
| `LLM_API_VERSION` | `llm_api_version` | *(empty)* |
| `LLM_TEMPERATURE` | `llm_temperature` | `0.0` |
| `LLM_STREAMING` | `llm_streaming` | `false` |
| `LLM_MAX_COMPLETION_TOKENS` / `LLM_MAX_TOKENS` | `llm_max_completion_tokens` | `16384` |
| `LLM_MAX_RETRIES` | `llm_max_retries` | `2` |
| `LLM_MAX_PARALLEL_REQUESTS` | `llm_max_parallel_requests` | `20` |
| `MOCK_LLM` | `llm_mock` | `false` |
| `MOCK_LLM_CASSETTE` | `llm_cassette` | *(empty)* |
| `COGNEE_RECORD_LLM` | `llm_record_path` | *(empty)* |
A fallback LLM (`llm_fallback_provider/_model/_endpoint/_api_key`) is configurable
programmatically (no env binding). `MOCK_LLM` + cassettes power the offline
benchmark — see [performance/mock-benchmark.md](https://github.com/topoteretes/cognee-rs/blob/main/docs/performance/mock-benchmark.md).
## Embedding
Read by `EmbeddingConfig::from_env()` ([`crates/embedding/src/config.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/embedding/src/config.rs)).
| Env var (aliases) | `Settings` field | Default |
| -------------------------------------------------------- | ------------------------------- | ------------------------------------------------------ |
| `EMBEDDING_PROVIDER` | `embedding_provider` | `openai` (`onnx` on Android) |
| `EMBEDDING_MODEL` | `embedding_model_name` | `text-embedding-3-small` (`BGE-Small-v1.5` on Android) |
| `EMBEDDING_DIMENSIONS` | `embedding_dimensions` | `1536` (`384` on Android) |
| `EMBEDDING_ENDPOINT` | `embedding_endpoint` | *(empty)* |
| `EMBEDDING_API_KEY` (falls back to `LLM_API_KEY`) | `embedding_api_key` | *(empty)* |
| `EMBEDDING_API_VERSION` | `embedding_api_version` | *(empty)* |
| `EMBEDDING_MODEL_PATH` / `COGNEE_E2E_EMBED_MODEL_PATH` | `embedding_model_path` | `./target/models/BGE-Small-v1.5-model_quantized.onnx` |
| `EMBEDDING_TOKENIZER_PATH` / `COGNEE_E2E_TOKENIZER_PATH` | `embedding_tokenizer_path` | `./target/models/bge-small-tokenizer.json` |
| `EMBEDDING_MAX_SEQUENCE_LENGTH` | `embedding_max_sequence_length` | `512` |
| `EMBEDDING_BATCH_SIZE` | `embedding_batch_size` | `32` |
| `MOCK_EMBEDDING` | *(provider override)* | `false` (also accepts `deterministic`) |
Provider values: `onnx`, `fastembed`, `openai`, `openai_compatible`, `ollama`, `mock`.
## Vector database
| Env var | `Settings` field | Default |
| ------------------------------------------- | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `VECTOR_DB_PROVIDER` | `vector_db_provider` | `lancedb` (embedded, persistent) on non-Android; falls back to `brute-force` (in-memory) on Android |
| `VECTOR_DB_URL` | `vector_db_url` | *(empty — defaults to `{system_root_directory}/databases/cognee.lancedb`; set to `:memory:` to force the in-memory brute-force store)* |
| `VECTOR_DB_HOST` / `VECTOR_DB_PORT` | `vector_db_host` / `vector_db_port` | *(empty)* / `1234` |
| `VECTOR_DB_NAME` / `VECTOR_DB_KEY` | `vector_db_name` / `vector_db_key` | *(empty)* |
| `VECTOR_DB_USERNAME` / `VECTOR_DB_PASSWORD` | … | *(empty)* |
Supported providers:
* `lancedb` — embedded Apache-Arrow / Lance vector store, on disk. Default on
every target except Android. The on-disk layout matches the Python SDK's
default LanceDB store, so a Rust deployment can be opened from Python and
vice versa.
* `brute-force` — pure-Rust in-memory linear scan. Default on Android (where
LanceDB's native stack does not cross-compile). Selected on any target by
setting `vector_db_url = ":memory:"`.
* `pgvector` — Postgres + the `pgvector` extension; requires the `pgvector`
Cargo feature on the binary build.
Qdrant lives in closed `cognee-cloud-rs` as the `cognee-vector-qdrant` crate
and is not part of OSS. See [tools/backends](/rust/tools/backends).
Setting `vector_db_provider` to `qdrant` is rejected at component
initialization in OSS (it returns a config error rather than falling back).
## Graph database
| Env var | `Settings` field | Default |
| ----------------------------------------------------- | --------------------------------------------- | ----------------------------------------- |
| `GRAPH_DATABASE_PROVIDER` | `graph_database_provider` | `ladybug` |
| `GRAPH_FILE_PATH` | `graph_file_path` | *(empty; defaults under the system root)* |
| `GRAPH_DATABASE_URL` | `graph_database_url` | *(empty)* |
| `GRAPH_DATABASE_HOST` / `GRAPH_DATABASE_PORT` | `graph_database_host` / `graph_database_port` | *(empty)* / `123` |
| `GRAPH_DATABASE_NAME` / `GRAPH_DATABASE_KEY` | … | *(empty)* |
| `GRAPH_DATABASE_USERNAME` / `GRAPH_DATABASE_PASSWORD` | … | *(empty)* |
Supported providers: `ladybug`/`kuzu` (embedded), `postgres` (feature `pggraph`).
When Postgres graph credentials are unset they fall back to the relational `DB_*`
config (see [roadmap/cognify-compatibility-plan.md](https://github.com/topoteretes/cognee-rs/blob/main/docs/roadmap/cognify-compatibility-plan.md)).
## Relational database
| Env var | `Settings` field | Default |
| ----------------------------- | --------------------- | ----------------------------- |
| `DATABASE_URL` | `relational_db_url` | `sqlite:./cognee.db?mode=rwc` |
| `DB_PROVIDER` | `db_provider` | `sqlite` |
| `DB_HOST` / `DB_PORT` | `db_host` / `db_port` | `localhost` / `5432` |
| `DB_NAME` | `db_name` | `cognee_db` |
| `DB_USERNAME` / `DB_PASSWORD` | … | *(empty)* |
## Chunking & tokenizer
Read by [`crates/chunking/src/config.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/chunking/src/config.rs). Most
chunking knobs (`chunk_strategy` default `PARAGRAPH`, `chunk_size` `1500`,
`chunk_overlap` `10`, `chunk_engine`) are `Settings`/`CognifyConfig` fields without
env bindings. The token counter is env-selected:
| Env var | Purpose | Default |
| ----------------------- | ----------------------------------------- | ---------------------------- |
| `COGNEE_TOKEN_COUNTER` | `tiktoken` / `word` / `huggingface`(`hf`) | auto from embedding provider |
| `HUGGINGFACE_TOKENIZER` | model id when counter = `huggingface` | *(empty)* |
## Ontology
| Env var | `Settings` field | Default |
| ---------------------------- | ---------------------------- | --------- |
| `ONTOLOGY_FILE_PATH` | `ontology_file_path` | *(empty)* |
| `ONTOLOGY_RESOLVER` | `ontology_resolver` | `rdflib` |
| `ONTOLOGY_MATCHING_STRATEGY` | `ontology_matching_strategy` | `fuzzy` |
## System paths, users & datasets
| Env var | `Settings` field | Default |
| ---------------------------------------------- | ----------------------- | -------------------------------------- |
| `COGNEE_SYSTEM_ROOT_DIRECTORY` | `system_root_directory` | `./.cognee_system` |
| `COGNEE_DATA_ROOT_DIRECTORY` | `data_root_directory` | `./.data_storage` |
| `CACHE_ROOT_DIRECTORY` | `cache_root_directory` | `./.cognee_cache` |
| `COGNEE_DEFAULT_USER_ID` | `default_user_id` | nil UUID |
| `COGNEE_DEFAULT_DATASET_NAME` | `default_dataset_name` | `main_dataset` |
| `DEFAULT_USER_EMAIL` / `DEFAULT_USER_PASSWORD` | … | `default_user@example.com` / *(empty)* |
| `ENABLE_BACKEND_ACCESS_CONTROL` | `enable_access_control` | `false` |
Setting `system_root_directory` cascades to the default `graph_file_path` and
`vector_db_url` unless those are set explicitly.
## Session / cache & rate limiting
| Env var | `Settings` field | Default |
| ---------------------------------------------------------- | --------------------- | --------------------- |
| `CACHE_BACKEND` | `cache_backend` | `fs` |
| `CACHE_HOST` / `CACHE_PORT` | … | `localhost` / `6379` |
| `SESSION_TTL_SECONDS` | `session_ttl_seconds` | `604800` (7d) |
| `CACHING` | `enable_caching` | `true` |
| `LLM_RATE_LIMIT_ENABLED` / `_REQUESTS` / `_INTERVAL` | … | `false` / `60` / `60` |
| `EMBEDDING_RATE_LIMIT_ENABLED` / `_REQUESTS` / `_INTERVAL` | … | `false` / `60` / `60` |
## Logging
> **Canonical table.** Binding READMEs and `.env.example` link here. cognee writes
> structured logs to **stdout** and (when writable) to a rotating file. File logging
> is owned by [`cognee-logging`](https://github.com/topoteretes/cognee-rs/blob/main/crates/logging/), initialised by the CLI and HTTP
> server via `cognee_logging::init_logging`.
| Env var | Default | Purpose |
| ------------------------- | ---------------- | ---------------------------------------------------------------- |
| `COGNEE_LOG_FILE` | `true` | Master file-logging toggle (`false`/`0`/`no` disables). |
| `COGNEE_LOGS_DIR` | `~/.cognee/logs` | Log directory (falls back to `/tmp/cognee_logs` if unwritable). |
| `COGNEE_LOG_FORMAT` | `plain` | `plain` (Python-compatible) or `json`. Applies to stdout + file. |
| `COGNEE_LOG_ROTATION` | `daily` | `daily` / `hourly` / `minutely` / `never`. |
| `COGNEE_LOG_BACKUP_COUNT` | `5` | Rotated files retained by age. |
| `COGNEE_LOG_MAX_FILES` | `10` | Hard cap on retained log files. |
| `LOG_FILE_NAME` | *(timestamped)* | Override the log file name. |
| `RUST_LOG` / `LOG_LEVEL` | `info` | Level filter (`RUST_LOG` preferred). |
> **Multi-process warning** — when several cognee processes share one log file via
> `LOG_FILE_NAME`, rotation is not coordinated; concurrent rotation can corrupt the
> log. For sharded workers, give each shard its own `COGNEE_LOGS_DIR` (or unset
> `LOG_FILE_NAME` per shard).
## Observability & telemetry
cognee emits OpenTelemetry traces (behind the `telemetry` feature) and opt-out
product analytics. The **deep references** are
[observability/opentelemetry.md](https://github.com/topoteretes/cognee-rs/blob/main/docs/observability/opentelemetry.md) and
[observability/send\_telemetry.md](https://github.com/topoteretes/cognee-rs/blob/main/docs/observability/send_telemetry.md); the env surface:
| Env var | Default | Purpose |
| ------------------------------------------ | ------------------ | ----------------------------------------------------------- |
| `COGNEE_TRACING_ENABLED` | `false` | Activate OTLP trace export. |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | *(empty)* | OTLP collector endpoint (non-empty also activates tracing). |
| `OTEL_SERVICE_NAME` | `cognee` | Service name attribute. |
| `OTEL_EXPORTER_OTLP_HEADERS` / `_PROTOCOL` | *(empty)* / `grpc` | Exporter headers / protocol. |
| `OTEL_SPAN_PROCESSOR` | `batch` | `batch` or `simple`. |
| `OTEL_TRACES_SAMPLER` / `_ARG` | *(empty)* | Sampler selection. |
| `TELEMETRY_DISABLED`, `ENV=test\|dev` | *(unset)* | Opt out of product analytics. |
## HTTP server
The server binary reads its own env surface ([`crates/http-server/src/config.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/http-server/src/config.rs)) —
host/port, auth, body limits, pipeline registry, notebooks, health probes. See
[tools/http-server](/rust/tools/http-server) and
[HTTP server architecture: config](https://github.com/topoteretes/cognee-rs/blob/main/docs/http-server/architecture.md).
## Cloud
Cloud/Auth0 configuration (`COGNEE_CLOUD_URL`, `COGNEE_AUTH0_*`) and the
`serve()`/`disconnect()` flow live in the closed `cognee-cloud-rs` product
(the `cognee-cloud` crate) and are not part of OSS.
## Runtime configuration API
`ConfigManager` (`Arc<RwLock<Settings>>`) exposes typed setters used by the
bindings and CLI. Families: `set_llm_*`, `set_embedding_*`, `set_vector_db_*`,
`set_graph_*`, `set_chunk_*`, `set_relational_db_*`, `set_*_root_directory`,
`set_ontology_*`, `set_classification_model` / `set_summarization_model` /
`set_summarization_schema`, plus four bulk setters (`set_llm_config`,
`set_embedding_config`, `set_vector_db_config`, `set_graph_db_config`) and a
generic `set(key, value)`. Introspection: `read()`, `version()`, `get_settings()`
(secrets masked). Full signatures are in the
[`ConfigManager` rustdoc](https://github.com/topoteretes/cognee-rs/blob/main/crates/lib/src/config.rs). The binding ergonomics
(granular JS setters vs generic `set` in Python/C) are documented in
[Language bindings: configuration](/rust/tools/bindings#configuration).
## CLI `config` subcommand
`cognee-cli config get|set|unset <key>` reads/writes the persisted JSON file.
The settable keys are the snake\_case `Settings` field names — see
[`known_keys()`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cli/src/config_store.rs). Example:
```bash theme={null}
cognee-cli config set llm_max_retries 4
cognee-cli config get llm_model
cognee-cli config unset embedding_endpoint
```
# Rust (Cognee-RS)
Source: https://docs.cognee.ai/rust/getting-started
Build on-device AI memory pipelines in Rust with the cognee-cli binary.
<Note>
**Experimental.** Cognee-RS is a Rust port of the Python `cognee` SDK, built
for on-device AI memory (phone, smartwatch, embedded) and aiming for behavioral
parity with Python cognee. The source lives at
[github.com/topoteretes/cognee-rs](https://github.com/topoteretes/cognee-rs).
There is no `pip install` / hosted step — you build the CLI from source with Cargo.
</Note>
Cognee-RS exposes the same four-verb memory API as Python cognee —
**`remember`**, **`recall`**, **`improve`**, **`forget`** — composing the
`add → cognify → search` pipeline. The fastest way in is the `cognee-cli` binary.
## Prerequisites
* A **Rust toolchain** (edition 2024, MSRV 1.91) — install via [rustup](https://rustup.rs).
* An **OpenAI-compatible LLM API key**. The CLI hard-fails at startup if no LLM
key is configured. A local endpoint (e.g. Ollama) works too — you still pass a
dummy key.
## Build the CLI
```bash theme={null}
git clone https://github.com/topoteretes/cognee-rs
cd cognee-rs
cargo build --release -p cognee-cli # -> target/release/cognee-cli
# put it on your PATH for the snippets below
export PATH="$PWD/target/release:$PATH"
```
The default feature set wires a fully embedded, no-external-service stack:
**SQLite** (relational), **Ladybug** (graph), and **LanceDB** (embedded,
persistent vector index). Nothing else to install.
## Configure the LLM
A `.env` file in the working directory is auto-loaded. The only required setting
is the LLM API key:
```bash theme={null}
export LLM_API_KEY="sk-..." # canonical name (OPENAI_TOKEN is an accepted alias)
# optional overrides:
export LLM_MODEL="gpt-4o-mini" # the compiled default is openai/gpt-5-mini
export LLM_ENDPOINT="https://..." # alias: OPENAI_URL; empty -> OpenAI's API
```
<Note>
**Embeddings need a key by default too.** On desktop/server the default
embedding provider is OpenAI (`text-embedding-3-small`), reusing
`LLM_API_KEY` / `LLM_ENDPOINT` — so setting `LLM_API_KEY` alone is enough for
the full pipeline. To run embeddings fully local, set `EMBEDDING_PROVIDER=onnx`
(or `ollama`).
</Note>
<Accordion title="Fully local with Ollama">
```bash theme={null}
ollama serve &
ollama pull llama3.2:3b
export OPENAI_URL=http://localhost:11434/v1
export OPENAI_TOKEN=not-needed # dummy value still required — startup checks for a non-empty key
export OPENAI_MODEL=llama3.2:3b
export EMBEDDING_PROVIDER=ollama # or onnx — otherwise embeddings still call OpenAI
```
</Accordion>
## Your first memory
```bash theme={null}
# store, then ask — this is the whole loop
cognee-cli remember "Cognee turns raw data into a queryable knowledge graph."
cognee-cli recall "what does cognee do?"
```
* **`remember`** ingests the data, builds the knowledge graph, and runs a
self-improvement pass (disable with `--no-improve`).
* **`recall`** auto-routes the search type for you when `--query-type` is omitted.
<Note>
On desktop/server, the default vector index is **LanceDB and persistent**.
The pure-Rust brute-force vector index is in-memory and selected on Android,
or when you set `VECTOR_DB_URL=:memory:`. For Postgres-backed vectors, build
with the `pgvector` feature and point `VECTOR_DB_PROVIDER=pgvector` at a
Postgres instance.
</Note>
## Lower-level pipeline
`remember` / `recall` wrap the explicit stages, which exist as separate
subcommands for fine-grained control:
```bash theme={null}
# 1. Ingest data into a dataset (defaults to "main_dataset")
cognee-cli add ./notes.txt "some inline text" -d my_dataset
# 2. Build the knowledge graph
cognee-cli cognify -d my_dataset
# 3. Query it (defaults --query-type to GRAPH_COMPLETION)
cognee-cli search "Alan Turing" -t GRAPH_COMPLETION -k 10 -d my_dataset
```
Run `cognee-cli <command> --help` for the full flag list.
## Language bindings
The ergonomic `Cognee` class — `new(settings)` → `warm()` → `add()` /
`cognify()` / `search()` / `remember()` — is exposed by the bindings, which keep
the component graph alive across calls in one process:
* **Python** (PyO3): `from cognee_py import Cognee`
* **JavaScript/TypeScript** (Neon): `import { Cognee } from '@cognee/cognee-ts'`
* **C** (FFI): `#include "cognee_sdk.h"`
## Next Steps
<CardGroup>
<Card title="Cognee-RS on GitHub" href="https://github.com/topoteretes/cognee-rs" icon="github">
Full README, architecture docs, and the crate-by-crate workspace breakdown.
</Card>
<Card title="Python Quickstart" href="/getting-started/quickstart" icon="play">
The same remember / recall loop in the Python SDK.
</Card>
</CardGroup>
# Custom Graph Schema
Source: https://docs.cognee.ai/rust/guides/custom-graph-schema
Swap the LLM's structured output shape in cognify via summarization_model (wired) and graph_model (set but not consumed in the standalone pipeline).
cognee-rust mirrors two Python cognify knobs that swap the LLM's structured
output shape: `graph_model` (graph extraction) and `summarization_model`
(summaries). **Their wiring status differs — read carefully.**
## Summarization schema — wired
### What it does
Replaces the default `SummarizedContent` shape requested from the LLM during the
summarization stage with your own JSON Schema. Mirrors Python's
`CognifyConfig.summarization_model`.
### Requirement
The schema **must** contain a string `summary` property — the pipeline reads
`summary` to build each `TextSummary`. This is validated up front by
[`validate_summary_schema`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/src/config.rs), so a bad schema
fails at config time rather than mid-pipeline.
### Example (programmatic)
```rust theme={null}
use cognee_cognify::CognifyConfig;
use serde_json::json;
let schema = json!({
"type": "object",
"properties": {
"summary": { "type": "string" },
"keywords": { "type": "array", "items": { "type": "string" } }
},
"required": ["summary"]
});
let config = CognifyConfig::default()
.with_summary_schema(schema)?; // returns Err if `summary` is missing/non-string
```
This path **is consumed**: the summarization task constructs
`SummaryExtractor::new_with_schema(llm, config.summary_schema)`
([`crates/cognify/src/tasks.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/src/tasks.rs)), and the
extractor requests the custom schema when it is `Some`
([`crates/cognify/src/summarization/extractor.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/src/summarization/extractor.rs)).
### Via top-level config
`cognee-lib` exposes a runtime setter mirroring Python's
`cognee.config.set_summarization_model(...)`:
```rust theme={null}
settings.set_summarization_schema(schema)?; // crates/lib/src/config.rs
```
## Graph extraction schema — set but NOT consumed (standalone pipeline)
### What it does (in Python)
Python's `graph_model` lets you replace the default `KnowledgeGraph` extraction
shape with a custom Pydantic model.
### Status in Rust
`CognifyConfig.graph_schema` exists and has a builder
([`CognifyConfig::with_graph_schema`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/src/config.rs)), **but
the standalone cognify graph-extraction task does not read it.** A `graph_schema`
you set on `CognifyConfig` is effectively a no-op for the in-process pipeline.
Its only live consumers today are:
* **Dataset-config persistence** — stored/retrieved via
[`crates/database/src/ops/dataset_configurations.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/database/src/ops/dataset_configurations.rs).
* **HTTP server** — accepted on dataset-config payloads in
[`crates/http-server/src/routers/datasets.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/http-server/src/routers/datasets.rs)
(and validated by `graph_schema_to_graph_model` in
[`crates/llm/src/dynamic_model.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/llm/src/dynamic_model.rs)).
See [docs/roadmap/cognify-compatibility-plan.md](https://github.com/topoteretes/cognee-rs/blob/main/docs/roadmap/cognify-compatibility-plan.md)
— wiring `graph_schema` into the extraction task is tracked as follow-up work.
To customize extraction today, use a [custom prompt](/rust/guides/custom-prompts) instead.
## Pointers
* [`CognifyConfig`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/src/config.rs) — `summary_schema`, `graph_schema`, builders, `validate_summary_schema`.
* [`crates/cognify/src/summarization/extractor.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/src/summarization/extractor.rs) — summary schema consumption.
* [roadmap/cognify-compatibility-plan.md](https://github.com/topoteretes/cognee-rs/blob/main/docs/roadmap/cognify-compatibility-plan.md) — graph\_schema gap.
# Rust SDK Custom Prompts
Source: https://docs.cognee.ai/rust/guides/custom-prompts
Override the LLM prompt the cognify pipeline uses for entity/relationship (graph) extraction via the CognifyConfig builder.
## What it does
Overrides the LLM prompt that the cognify pipeline uses for the
entity/relationship (graph) extraction stage. Mirrors Python's cognify
`custom_prompt` parameter. When set, the [`FactExtractor`] uses your prompt
instead of the built-in default.
## When to use it
* Steer extraction toward a domain vocabulary ("treat each section header as a
Topic node…").
* Tighten or loosen what counts as an entity/relationship for your corpus.
## How it is wired
This knob **is consumed** by the standalone pipeline. `CognifyConfig` carries
`custom_extraction_prompt: Option<String>`, and the graph-extraction task passes
it straight into `FactExtractor::extract_facts(text, prompt)`
([`crates/cognify/src/tasks.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/src/tasks.rs), the
`extract_graph_from_data` task). `None` falls back to the default prompt.
## Example (programmatic)
```rust theme={null}
use cognee_cognify::CognifyConfig;
let config = CognifyConfig::default()
.with_custom_prompt(
"Extract people, organizations, and the roles connecting them.".to_string(),
);
// pass `config` to the cognify pipeline
```
The builder is [`CognifyConfig::with_custom_prompt`]; the field is
`custom_extraction_prompt`.
## CLI
There is currently **no CLI flag** for the extraction prompt — `cognee-cli
cognify` does not expose `custom_extraction_prompt`. Use it via the
`CognifyConfig` builder in Rust (or the HTTP/binding surfaces that accept a
cognify config). Note this is distinct from `cognee-cli search --system-prompt[-path]`, which sets the *search* answer-generation prompt, not
the cognify extraction prompt.
## Pointers
* [`CognifyConfig`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/src/config.rs) — `custom_extraction_prompt`, `with_custom_prompt`.
* [`crates/cognify/src/tasks.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/src/tasks.rs) — where the prompt reaches `FactExtractor`.
* [Operations](/rust/operations) — the cognify stage in context.
[`FactExtractor`]: https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/src/
[`CognifyConfig::with_custom_prompt`]: https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/src/config.rs
# Memify Node Filtering
Source: https://docs.cognee.ai/rust/guides/memify-node-filtering
Scope memify enrichment to a specific node set — a node type plus one or more node names — from the CLI or programmatically.
## What it does
`memify` enriches an existing knowledge graph by building and indexing triplet
embeddings from its edges. By default it runs over the whole graph; node
filtering scopes it to a specific **node set** — a node *type* plus one or more
node *names*. This is the "Node Sets" concept (see
[Concepts](/rust/concepts)): a named subgraph you can enrich or query
independently. Mirrors Python's `memify(node_type=…, node_name=…)`.
## When to use it
* Re-enrich only part of a large graph (e.g. just `Entity` nodes named "Acme").
* Build focused triplet indexes for a subset of your memory.
## CLI
```bash theme={null}
# Enrich the whole graph
cognee-cli memify -d my_dataset
# Scope to a node type + names (OR logic across names)
cognee-cli memify -d my_dataset --node-type Entity --node-name Acme --node-name Globex
```
Flags ([`crates/cli/src/cli.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cli/src/cli.rs), `MemifyArgs`):
* `--node-type <T>` — filter to one node type (e.g. `Entity`).
* `--node-name <N>` — repeatable; matches any of the given names (OR).
* `--batch-size <N>` — triplet extraction/embedding batch size (default 100).
Filtering takes effect only when **both** a node type and at least one node name
are supplied — the pipeline then pulls the matching node-set subgraph
([`crates/cognify/src/memify/extract_triplets.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/src/memify/extract_triplets.rs)).
## Programmatic
```rust theme={null}
use cognee_cognify::memify::MemifyConfig;
let config = MemifyConfig::default()
.with_node_type_filter("Entity".to_string())
.with_node_name_filter(vec!["Acme".to_string(), "Globex".to_string()])
.with_node_name_filter_operator("OR".to_string()); // "OR" (default) or "AND"
```
See [`MemifyConfig`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/src/memify/config.rs) —
`node_type_filter`, `node_name_filter`, `node_name_filter_operator` (validated to
be `"OR"` or `"AND"`).
## Pointers
* [`MemifyConfig`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/src/memify/config.rs) — filter fields and builders.
* [`crates/cognify/src/memify/extract_triplets.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/src/memify/extract_triplets.rs) — node-set subgraph selection.
* [Concepts](/rust/concepts) — Node Sets.
* [Operations](/rust/operations) — the memify stage.
# Ontology
Source: https://docs.cognee.ai/rust/guides/ontology
Ground cognify extraction against an RDF/OWL, Turtle, or JSON-LD ontology so entities are matched to a known vocabulary.
## What it does
Supplies an ontology (RDF/OWL, Turtle, or JSON-LD) to cognify so extracted
entities are matched and grounded against a known vocabulary. The
[`OntologyResolver`](https://github.com/topoteretes/cognee-rs/blob/main/crates/ontology/src/) loads the file and matches
extracted entity names/types to ontology terms during knowledge-graph
extraction.
## When to use it
* You have a controlled vocabulary / taxonomy and want extraction to align with
it (consistent entity types, canonical names).
* You want to constrain or normalize the messy free-form output of an LLM.
## CLI
`cognee-cli cognify` exposes the ontology file directly:
```bash theme={null}
cognee-cli cognify -d my_dataset --ontology-file ./ontologies/domain.owl
```
The flag is `--ontology-file <path>`
([`crates/cli/src/cli.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cli/src/cli.rs), `CognifyArgs`).
## Configuration (env / config keys)
You can also set the ontology globally instead of per-invocation:
| Env var | Config key | Default |
| ---------------------------- | ---------------------------- | --------- |
| `ONTOLOGY_FILE_PATH` | `ontology_file_path` | *(empty)* |
| `ONTOLOGY_RESOLVER` | `ontology_resolver` | `rdflib` |
| `ONTOLOGY_MATCHING_STRATEGY` | `ontology_matching_strategy` | `fuzzy` |
```bash theme={null}
export ONTOLOGY_FILE_PATH=./ontologies/domain.owl
cognee-cli cognify -d my_dataset
```
Note: `ontology_resolver` and `ontology_matching_strategy` are stored for
parity but the active path currently uses the RDF/JSON-LD/Turtle resolver with a
fuzzy matching strategy whenever an ontology file is set. See
[Configuration](/rust/configuration) for the full surface and
[`crates/lib/src/config.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/lib/src/config.rs) for the field
definitions and `set_ontology_*` setters.
## Supported formats
RDF/OWL, Turtle, and JSON-LD, parsed via the `sophia` family of crates. See
[`crates/ontology/`](https://github.com/topoteretes/cognee-rs/blob/main/crates/ontology/) for the resolver implementations and
the test fixtures under `crates/ontology/tests/fixtures/`.
## Pointers
* [`crates/ontology/`](https://github.com/topoteretes/cognee-rs/blob/main/crates/ontology/) — `OntologyResolver`, `RdfLibOntologyResolver`, `NoOpOntologyResolver`.
* [`crates/cli/src/cli.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cli/src/cli.rs) — `--ontology-file`.
* [Configuration](/rust/configuration) — ontology env vars.
* [Operations](/rust/operations) — cognify stage.
# Guides
Source: https://docs.cognee.ai/rust/guides/overview
Task-oriented how-tos for cognee-rust, each with what the feature does, when to use it, a runnable example, and a pointer into the code.
Focused, task-oriented how-tos for cognee-rust. Each guide states what the
feature does, when to use it, a runnable example, and a pointer into the code.
For the end-to-end story start with [Getting Started](/rust/getting-started)
and [Operations](/rust/operations).
| Guide | What it covers |
| ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| [Custom Prompts](/rust/guides/custom-prompts) | Override the LLM prompt used for entity/relationship extraction in cognify. |
| [Custom Graph Schema](/rust/guides/custom-graph-schema) | Supply a custom summarization schema (wired) and a custom graph-extraction schema (set-but-not-consumed in the standalone pipeline). |
| [Ontology](/rust/guides/ontology) | Ground extraction with an RDF/OWL/JSON-LD ontology. |
| [Temporal Cognify](/rust/guides/temporal-cognify) | Extract events/timestamps and query them with temporal recall. |
| [Memify Node Filtering](/rust/guides/memify-node-filtering) | Scope memify enrichment to specific node types / names (node sets). |
# Temporal Cognify
Source: https://docs.cognee.ai/rust/guides/temporal-cognify
Run the temporal cognify variant to extract events and timestamps, then query them with the TEMPORAL search type or recall.
## What it does
Runs the temporal variant of the cognify pipeline: instead of (or in addition
to) standard entity/relationship extraction, it extracts **events and
timestamps** so the knowledge graph supports temporal reasoning. Mirrors
Python's `temporal_cognify=True`.
## When to use it
* Your corpus is event-driven (logs, meeting notes, news, changelogs) and you
want to ask "what happened when?" / "what came before X?".
* You plan to query with temporal recall (see below).
## CLI
```bash theme={null}
cognee-cli cognify -d my_dataset --temporal-cognify
```
The flag is `--temporal-cognify`
([`crates/cli/src/cli.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cli/src/cli.rs), `CognifyArgs`). It maps
to `CognifyConfig.temporal_cognify`.
## Programmatic
```rust theme={null}
use cognee_cognify::CognifyConfig;
let config = CognifyConfig::default()
.with_temporal_cognify(true);
// also: .with_data_per_batch(n) tunes the temporal batch size (default 20)
```
See [`CognifyConfig`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/src/config.rs) (`temporal_cognify`,
`with_temporal_cognify`, `data_per_batch`).
## Querying temporal memory
Retrieve with the `TEMPORAL` search type:
```bash theme={null}
cognee-cli search "what happened before the launch?" -t TEMPORAL -d my_dataset
# or let recall auto-route to a temporal strategy:
cognee-cli recall "timeline of the project"
```
This uses `SearchType::Temporal` and the temporal retriever
([`crates/search/src/`](https://github.com/topoteretes/cognee-rs/blob/main/crates/search/src/)). The recall router can
auto-select the temporal strategy from the query
([`crates/search/src/query_router.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/search/src/query_router.rs)).
## Pointers
* [`CognifyConfig`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/src/config.rs) — `temporal_cognify`.
* [`crates/cli/src/cli.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cli/src/cli.rs) — `--temporal-cognify`, `-t TEMPORAL`.
* [Operations](/rust/operations) — cognify and search stages.
# HTTP Server
Source: https://docs.cognee.ai/rust/http-server
Design and reference for cognee-http-server, the axum server mirroring the Python FastAPI surface under /api/v1/*.
Design and reference for `cognee-http-server`, the `axum` server that mirrors the
Python FastAPI surface under `/api/v1/*`. To **run or embed** it, start at
[HTTP Server Tools](/rust/tools/http-server). This page links to the detailed
upstream reference.
## Cross-cutting design
* **[Architecture](https://github.com/topoteretes/cognee-rs/blob/main/docs/http-server/architecture.md)** — crate topology, dual-surface design (library + binary), middleware stack, config lifecycle.
* **[auth.md](https://github.com/topoteretes/cognee-rs/blob/main/docs/http-server/auth.md)** — stub; the auth subsystem (JWT, fastapi-users parity, password-hash migration, bearer/cookie/api-key) lives in the closed `cognee-http-cloud` crate.
* **[pipelines.md](https://github.com/topoteretes/cognee-rs/blob/main/docs/http-server/pipelines.md)** — background job lifecycle, `PipelineRunRegistry`, status mapping, durable vs live events.
* **[websocket.md](https://github.com/topoteretes/cognee-rs/blob/main/docs/http-server/websocket.md)** — subscription model, status semantics, terminal close behavior.
* **[tenants.md](https://github.com/topoteretes/cognee-rs/blob/main/docs/http-server/tenants.md)** — stub; multi-tenant schema, permission model, and ACL resolution lives in the closed `cognee-http-cloud` crate.
* **[observability.md](https://github.com/topoteretes/cognee-rs/blob/main/docs/http-server/observability.md)** — span instrumentation and telemetry attributes for the server.
## Endpoints
* **[routers/](https://github.com/topoteretes/cognee-rs/blob/main/docs/http-server/routers/README.md)** — one reference doc per router. **20 routers live in OSS** (`crates/http-server/src/routers/`); **11 routers live in the closed `cognee-http-cloud` crate** (auth, auth-register, auth-reset-password, auth-verify, api-keys, users, users-by-email, permissions, configuration, sync, checks). The closed-router docs in `routers/` are stubs that point at the `cognee-cloud-rs` repo (private).
Open design questions for these areas are tracked in
[Roadmap open questions](https://github.com/topoteretes/cognee-rs/blob/main/docs/roadmap/open-questions.md).
# Operations
Source: https://docs.cognee.ai/rust/operations
What cognee-rust does: the remember / recall / improve / forget memory API and the underlying add / cognify / memify / search pipeline.
What cognee-rust *does*. The primary surface is the **memory API** —
**`remember`**, **`recall`**, **`improve`**, **`forget`** — four high-level
operations that compose the lower-level pipeline (`add → cognify → memify →
search`). Every operation is reachable from each interface (CLI, language
bindings, HTTP server) — see [Tools](/rust/tools/overview). API/type detail lives in
rustdoc (`cargo doc --no-deps --open`); this page is the conceptual map.
## The memory API
Cognee's primary surface is four operations that turn raw input into queryable,
self-improving memory. They live in the [`cognee-lib`](https://github.com/topoteretes/cognee-rs/blob/main/crates/lib/) `api`
module (`cognee_lib::api::{remember, recall, improve, forget}`) and surface as
the always-built `cognee-cli` verbs `remember` / `recall` / `improve` / `forget`.
```
input ──remember──▶ memory (graph + vectors, optionally session)
query ──recall────▶ auto-routed answers
improve───▶ enriched / bridged memory
forget────▶ removed memory
```
### remember
Stores input as memory: it runs **add + cognify** and then, by default, the
**improve** enrichment pass. Accepts inline text and/or file paths.
* **Session memory** — pass a `--session-id` to scope the turn to a session
(session-backed QA history).
* **Permanent graph memory** — omit `--session-id` and the input is persisted as
permanent, graph-backed memory.
`remember ≈ add + cognify + improve`. rustdoc: `api::remember`.
### recall
Queries memory with **auto-routing**: when no query type is given, `recall`
picks an appropriate retrieval strategy automatically. It is session-aware (reads
session history when given a `--session-id`) and graph-backed. Results are
returned to the caller (printed to stdout by the CLI).
`recall ≈ auto-routed search`. rustdoc: `api::recall`.
### improve
Enriches memory and bridges sessions: runs the feedback/enrichment improvement
stages over the graph (memify-style triplet enrichment plus feedback weighting).
Can target specific sessions or graph nodes and tune the feedback weight.
rustdoc: `api::improve`.
### forget
Removes memory: a whole dataset, a specific data item, or everything. Cascades
across the relational, graph, and vector backends and file storage. rustdoc:
`api::forget`.
## Lower-level pipeline
The memory API composes these building blocks. They remain available directly
when you need fine-grained control over each stage. The classic flow is:
```
raw data ──add──▶ stored + deduplicated ──cognify──▶ knowledge graph + vectors ──search──▶ answers
```
### add (ingest)
Streams input, computes a content hash, deduplicates, and persists the data plus
metadata. Accepts text, file paths, and HTTP(S) URLs (fetched and routed by MIME
type). Deterministic UUID5 IDs make the same content + owner reproducible across
SDKs. Pipeline: [`cognee-ingestion`](https://github.com/topoteretes/cognee-rs/blob/main/crates/ingestion/) (`AddPipeline`).
### cognify (knowledge-graph extraction)
Turns stored data into a knowledge graph in six stages: **classify** documents →
**chunk** text → **extract** entities/relationships (LLM, batched) → **summarize**
(conditional) → **add data points** (six vector collections + provenance to the
relational DB) → **extract DLT FK edges**. Configurable via `CognifyConfig`
(chunk strategy, custom prompts/schemas, temporal mode). Pipeline:
[`cognee-cognify`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/) (`cognify()` / `cognify_datasets()`).
### memify (graph enrichment)
Standalone, idempotent enrichment: reads the existing graph, builds `Triplet`
objects from every edge (`"source → relationship → target"`), embeds them, and
indexes them into the `Triplet`/`text` vector collection for
`SearchType::TripletCompletion`. Pipeline:
[`cognee-cognify`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/) (`memify()`).
### search (retrieval)
Unified orchestration across 15 retrieval strategies selected by `SearchType`
([`crates/search/src/types/search_type.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/search/src/types/search_type.rs)):
`GraphCompletion` (default), `GraphCompletionCot`, `GraphCompletionContextExtension`,
`GraphSummaryCompletion`, `TripletCompletion`, `RagCompletion`, `Chunks`,
`Summaries`, `Temporal`, `Cypher`, `NaturalLanguage`, `FeelingLucky`, `Feedback`,
`CodingRules`, `ChunksLexical`. Entry: [`cognee-search`](https://github.com/topoteretes/cognee-rs/blob/main/crates/search/)
(`SearchBuilder` / `SearchOrchestrator`).
## Additional operations
These live in the [`cognee-lib`](https://github.com/topoteretes/cognee-rs/blob/main/crates/lib/) `api` module (and `DatasetManager`):
| Operation | What it does | rustdoc |
| ------------- | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------- |
| **delete** | Cascading removal of data/datasets across relational → graph → vector → file storage (with dry-run preview). | [`cognee-delete`](https://github.com/topoteretes/cognee-rs/blob/main/crates/delete/) `DeleteService` |
| **update** | Re-ingest changed data and re-cognify the affected subset. | `api::update` |
| **prune** | Reset system or all state (`prune_system` / `prune_data`). | `api::prune` |
| **visualize** | Render the graph to a self-contained d3.js HTML file. | [`cognee-visualization`](https://github.com/topoteretes/cognee-rs/blob/main/crates/visualization/) |
## Operation → interface map
| Operation | CLI | HTTP route | Binding method |
| ------------- | ---------------------------- | ------------------------ | -------------- |
| remember | `cognee-cli remember` | `POST /api/v1/remember` | `remember()` |
| recall | `cognee-cli recall` | `POST /api/v1/recall` | `recall()` |
| improve | `cognee-cli improve` | `POST /api/v1/improve` | `improve()` |
| forget | `cognee-cli forget` | `POST /api/v1/forget` | `forget()` |
| add | `cognee-cli add` | `POST /api/v1/add` | `add()` |
| cognify | `cognee-cli cognify` | `POST /api/v1/cognify` | `cognify()` |
| add + cognify | `cognee-cli add-and-cognify` | *(two calls)* | — |
| memify | `cognee-cli memify` | `POST /api/v1/memify` | `memify()` |
| search | `cognee-cli search` | `POST /api/v1/search` | `search()` |
| delete | `cognee-cli delete` | `POST /api/v1/delete` | `delete*()` |
| update | *(via run-sequence)* | `POST /api/v1/update` | `update()` |
| visualize | `cognee-cli visualize` | `POST /api/v1/visualize` | `visualize()` |
CLI flags and feature gates: [tools/cli](/rust/tools/cli). HTTP request/response
shapes: [http-server/routers/](https://github.com/topoteretes/cognee-rs/blob/main/docs/http-server/routers/README.md). Binding method
names per language: [tools/bindings](/rust/tools/bindings).
# Pluggable Backends
Source: https://docs.cognee.ai/rust/tools/backends
Swap cognee-rust storage and compute backends (LLM, embeddings, vector, graph, relational, storage, session, ontology, tokenizer) via configuration.
cognee-rust is built on trait abstractions so each storage/compute backend can be
swapped via configuration. Pick providers with the env vars / config keys in
[Configuration](/rust/configuration); the trait + adapter detail is in rustdoc
(`cargo doc -p <crate> --no-deps --open`).
| Concern | Trait (crate) | Providers | Selected by |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- |
| **LLM** | `Llm` ([`cognee-llm`](https://github.com/topoteretes/cognee-rs/blob/main/crates/llm/)) | `OpenAIAdapter` (OpenAI/Ollama/vLLM/llama.cpp), `MockLlm` (`testing`) | `LLM_PROVIDER`, `LLM_MODEL`, `LLM_ENDPOINT` |
| **Embeddings** | `EmbeddingEngine` ([`cognee-embedding`](https://github.com/topoteretes/cognee-rs/blob/main/crates/embedding/)) | `OnnxEmbeddingEngine` (local BGE-Small), `OpenAICompatibleEmbeddingEngine`, `OllamaEmbeddingEngine`, `MockEmbeddingEngine` | `EMBEDDING_PROVIDER` (+ `MOCK_EMBEDDING`) |
| **Vector DB** | `VectorDB` ([`cognee-vector`](https://github.com/topoteretes/cognee-rs/blob/main/crates/vector/)) | `LanceDbAdapter` (embedded, persistent, default on non-Android), `BruteForceVectorDB` (in-memory; Android or `:memory:`), `PgVectorAdapter` (feature `pgvector`), `MockVectorDB` (`testing`) | `VECTOR_DB_PROVIDER` (`lancedb`/`brute-force`/`pgvector`) |
| **Graph DB** | `GraphDBTrait` ([`cognee-graph`](https://github.com/topoteretes/cognee-rs/blob/main/crates/graph/)) | `LadybugAdapter` (embedded), `PgGraphAdapter` (feature `postgres`) | `GRAPH_DATABASE_PROVIDER` (`ladybug`/`kuzu`/`postgres`) |
| **Relational DB** | `IngestDb`/`SearchHistoryDb`/`DeleteDb` ([`cognee-database`](https://github.com/topoteretes/cognee-rs/blob/main/crates/database/)) | `DatabaseConnection` — SQLite / Postgres via SeaORM | `DB_PROVIDER`, `DATABASE_URL` |
| **File storage** | `StorageTrait` ([`cognee-storage`](https://github.com/topoteretes/cognee-rs/blob/main/crates/storage/)) | `LocalStorage` (`file://`), `MockStorage` | `STORAGE_BACKEND` |
| **Session store** | `SessionStore` ([`cognee-session`](https://github.com/topoteretes/cognee-rs/blob/main/crates/session/)) | `FsSessionStore`, `RedisSessionStore`, `SeaOrmSessionStore` | `COGNEE_SESSION_STORE` (server) |
| **Ontology** | `OntologyResolver` ([`cognee-ontology`](https://github.com/topoteretes/cognee-rs/blob/main/crates/ontology/)) | `RdfLibOntologyResolver`, `NoOpOntologyResolver` | `ONTOLOGY_RESOLVER` |
| **Tokenizer** (chunking) | `TokenCounter` ([`cognee-chunking`](https://github.com/topoteretes/cognee-rs/blob/main/crates/chunking/)) | `WordCounter`, `HuggingFaceTokenCounter` (feature), `TikTokenCounter` (feature) | `COGNEE_TOKEN_COUNTER` |
Notes:
* **Embedded by default.** A plain build runs entirely locally: LanceDB vector
index, embedded Ladybug (graph), SQLite (relational), local file storage.
The brute-force vector index is used on Android or with `VECTOR_DB_URL=:memory:`.
* **Feature gates.** `pgvector`, `pggraph`/`postgres`, and the
`hf-tokenizer`/`tiktoken` counters are cargo features, on by default in
`cognee-lib`/`cognee-cli` — see
[Architecture: feature strategy](/rust/architecture#architecture-patterns).
* **Closed-source companions.** Embedded Qdrant (`cognee-vector-qdrant`) and
on-device LiteRT inference (`cognee-llm-litert`, Android) live in the closed
`cognee-cloud-rs` repository and are not part of OSS.
* **Full Postgres stack** (relational + graph + vector on one Postgres) is the
one remaining adapter milestone — see [roadmap/](https://github.com/topoteretes/cognee-rs/blob/main/docs/roadmap/README.md).
`MockEmbeddingEngine`, `MockGraphDB`, `MockVectorDB`, and `MockStorage` (the
`testing` feature) back the test suite — see
[test patterns](https://github.com/topoteretes/cognee-rs/blob/main/.claude/CLAUDE.md#test-patterns).
# Language Bindings
Source: https://docs.cognee.ai/rust/tools/bindings
Python, C, and JavaScript SDKs for cognee-rust, built on a shared bindings-common core.
cognee-rust ships three language bindings on top of the Rust core. All share the
same SDK-tier implementation via [`cognee-bindings-common`](https://github.com/topoteretes/cognee-rs/blob/main/crates/bindings-common/)
(portable op bodies + stable error codes), so their surfaces line up 1:1. Each
exposes the same flow: `warm()` → `add()` → `cognify()` → `search()`.
| Binding | README | Entry type | Async model |
| ------------------------ | --------------------------------------------------------------------------------------- | ------------------------------------------------------- | ----------------------------------------------------- |
| **Python** (PyO3) | [python/README.md](https://github.com/topoteretes/cognee-rs/blob/main/python/README.md) | `Cognee` (`from cognee_py import Cognee`) | native `async` |
| **C** (FFI) | [capi/README.md](https://github.com/topoteretes/cognee-rs/blob/main/capi/README.md) | `cg_sdk_*` over an opaque handle | callback-based (+ optional `CgSdkWaiter` sync bridge) |
| **JavaScript/TS** (Neon) | [ts/README.md](https://github.com/topoteretes/cognee-rs/blob/main/ts/README.md) | `Cognee` (`import { Cognee } from '@cognee/cognee-ts'`) | Promise-based |
Module-level helpers exist in each binding for logging/telemetry setup. The
`serve()` / `disconnect()` cloud helpers live in the closed companion packages,
not in the OSS bindings. The full per-language method list lives in each
binding's README and its generated docs.
## Configuration
All three delegate to the same `ConfigManager` in `cognee-bindings-common`; the
difference is purely ergonomic (design decision A3.1, intentional for 0.1.0):
| Binding | Setter surface |
| ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **JS** | \~40 granular typed setters (`setLlmModel`, `setEmbeddingProvider`, …) **+** 4 bulk setters (`setLlmConfig`, `setEmbeddingConfig`, `setVectorDbConfig`, `setGraphDbConfig`) **+** generic `set(key, value)` |
| **C** | `cg_sdk_config_set` / `cg_sdk_config_set_str`, the 4 bulk setters, `cg_sdk_config_get` |
| **Python** | `set`, `set_str`, the 4 bulk setters, `get` |
The generic `set("llm_model", value)` already reaches every key, so C and Python
deliberately ship only the generic + bulk setters; JS adds the granular setters
as thin (infallible) sugar over the same underlying fields. Type errors surface
with the stable `CONFIG_TYPE_MISMATCH` code (C: `CG_ERR_CONFIG_TYPE_MISMATCH`) at
the call site. The full set of settable keys is the canonical `Settings` field
names — see [Configuration](/rust/configuration) and
[`crates/lib/src/config.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/lib/src/config.rs). Field names are
camelCase in JS, snake\_case in Python/C (matching the struct).
### Unification path (post-0.1.0)
If full parity is required, mirror the JS macro-driven approach: add
`set_llm_model(value)` / `cg_sdk_config_set_llm_model(handle, value)` etc. as thin
`set_str("llm_model", value)` wrappers in each binding's idiomatic case
(\~0.5 d/binding, mechanical).
# CLI Reference
Source: https://docs.cognee.ai/rust/tools/cli
The cognee-cli command-line binary: subcommands, flags, config, retries, and logging.
The command-line binary, built from the [`cognee-cli`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cli/) crate.
It drives the full pipeline and is also the on-device (Android) runner. Run
`cognee-cli <command> --help` for the authoritative flag list; the clap
definitions are in [`crates/cli/src/cli.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cli/src/cli.rs).
```bash theme={null}
cargo build --release # produces target/release/cognee-cli
```
## Subcommands
The **memory API** verbs (`remember` / `recall` / `improve` / `forget`) are the
primary surface; the `add` / `cognify` / `memify` / `search` commands below them
are the lower-level pipeline they compose. All of these are always built (not
feature-gated).
| Command | Purpose | Notable flags | Feature gate |
| ------------------------------ | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------- |
| `remember <data…>` | Store memory: `add` + `cognify` + (by default) `improve`. `<data…>` is inline text and/or file paths | `-d/--dataset-name` (`main_dataset`), `--session-id`, `--no-improve` (default OFF — improve runs by default), `--tenant-id` | — |
| `recall <query>` | Query memory with auto-routing (session-aware + graph-backed) | `-t/--query-type` (optional — omit to auto-route), `-d/--datasets` (repeatable), `-k/--top-k` (10), `--session-id`, `-f/--output-format` (`pretty`/`json`/`simple`, default `pretty`) | — |
| `improve` | Enrich memory / bridge sessions (feedback + enrichment stages) | `-d/--dataset-name` (`main_dataset`), `--session-id` (repeatable), `--node-name` (repeatable), `--feedback-alpha` (`0.1`), `--tenant-id` | — |
| `forget` | Remove memory (a dataset, a single data item, or everything) | `-d/--dataset-name` \| `--data-id` (UUID; requires `--dataset-name`; conflicts with `--all`) \| `--all`, `--tenant-id` | — |
| `add <inputs…>` | Ingest text / file paths / HTTP(S) URLs into a dataset | `-d/--dataset-name` (`main_dataset`), `--tenant-id` | — |
| `cognify` | Build the knowledge graph from one or more datasets | `-d/--datasets`, `--chunk-size`, `--chunker` (`TextChunker`/`LangchainChunker`/`CsvChunker`), `--ontology-file`, `-b/--background`, `--llm-max-retries`, `--llm-max-parallel-requests`, `--temporal-cognify` | — |
| `add-and-cognify <inputs…>` | `add` then `cognify` in one step | union of the above | — |
| `memify` | Enrich an existing graph with triplet embeddings | `-d/--datasets`, `--node-type`, `--node-name`, `--batch-size` (100) | — |
| `search <query>` | Query the graph/vectors | `-t/--query-type` (`GRAPH_COMPLETION`), `-d/--datasets`, `-k/--top-k` (10), `--system-prompt` / `--system-prompt-path`, `--session-id`, `-f/--output-format` (`pretty`/`json`), `--llm-max-retries` | — |
| `delete` | Remove data/datasets across all backends | `-d/--dataset-name` \| `--dataset-id`, `--data-id`, `--all`, `--mode` (`soft`/`hard`), `--dry-run`, `-f/--force` | — |
| `config get\|set\|unset <key>` | Read/write the persisted JSON config | — | — |
| `run-sequence` | Run a scripted add/cognify/search sequence | — | — |
| `visualize` | Render the graph to a self-contained HTML file | `-o/--output` (`~/graph_visualization.html`) | `visualization` |
| `bench` | Phase-timed benchmark driver | `--memories`, `--mock-llm`, `--output` | `bench` |
The feature-gated commands are enabled in the default build of `cognee-cli`
(except platform-specific ones). See [Architecture: feature strategy](/rust/architecture#architecture-patterns).
Cloud `serve` / `disconnect` are not part of OSS — they ship in the
closed-source `cognee-cli-cloud` binary (`cognee-cli-cloud serve --url …` /
`cognee-cli-cloud disconnect`).
## Memory API
The four primary verbs cover the common workflow end-to-end:
```bash theme={null}
# Store memory (add + cognify + improve). Inline text and/or file paths.
cognee-cli remember "Cognee turns data into a knowledge graph" ./notes.txt -d my_dataset
# Scope a turn to a session (session-backed memory) instead of permanent graph memory
cognee-cli remember "follow-up note" --session-id chat-42
# Query memory — omit -t to let recall auto-route the retrieval strategy
cognee-cli recall "what did we learn about X?" -d my_dataset -k 10
# Enrich memory / bridge sessions
cognee-cli improve -d my_dataset --session-id chat-42
# Remove memory
cognee-cli forget --all
cognee-cli forget -d my_dataset
cognee-cli forget --data-id 00000000-0000-0000-0000-000000000000 -d my_dataset
```
`remember` with a `--session-id` records session memory; without one it persists
permanent, graph-backed memory. `recall` is session-aware and graph-backed; with
no `-t/--query-type` it auto-routes to a suitable retrieval strategy.
## Lower-level pipeline
The memory verbs compose `add → cognify → search`, which you can also drive
directly for fine-grained control (`remember ≈ add + cognify + improve`;
`recall ≈ auto-routed search`):
```bash theme={null}
cognee-cli add ./notes.txt "some inline text" -d my_dataset
cognee-cli cognify -d my_dataset
cognee-cli search "what did we learn about X?" -t GRAPH_COMPLETION -d my_dataset -k 10
```
## `config` subcommand
Reads/writes `~/.config/cognee-rust/config.json`. Keys are the snake\_case
`Settings` field names. See [Configuration: CLI config](/rust/configuration#cli-config-subcommand).
```bash theme={null}
cognee-cli config set llm_max_retries 4
cognee-cli config get llm_model
cognee-cli config unset embedding_endpoint
```
## LLM retries
`--llm-max-retries N` (accepted by `cognify`, `add-and-cognify`, `search`)
overrides the retry count for structured-output LLM calls for that run; the
persistent default is the `llm_max_retries` config key (default `2`, minimum `1`).
The CLI flag wins over the config value. It is passed to `OpenAIAdapter` and
governs the strict-schema, function-call, and JSON-fallback parsing paths.
```bash theme={null}
cognee-cli cognify --llm-max-retries 4
cognee-cli search "What is TechCorp?" --llm-max-retries 4
```
## Logging
The CLI calls `cognee_logging::init_logging` at startup. The env-var surface
(`COGNEE_LOG_*`, `RUST_LOG`/`LOG_LEVEL`, `LOG_FILE_NAME`) is shared with the HTTP
server and bindings and documented canonically in
[Configuration: logging](/rust/configuration#logging). Example — JSON logs to
a custom directory:
```bash theme={null}
COGNEE_LOG_FORMAT=json COGNEE_LOGS_DIR=/var/log/cognee cognee-cli cognify -d main_dataset
```
# HTTP Server (Tools)
Source: https://docs.cognee.ai/rust/tools/http-server
Launch the cognee-http-server binary or embed it as a library; an axum server exposing the FastAPI surface under /api/v1/*.
An `axum`-based server that exposes the Python FastAPI surface under
`/api/v1/*`. Built from [`cognee-http-server`](https://github.com/topoteretes/cognee-rs/blob/main/crates/http-server/), which
is both a **library** (embed it in any Rust program) and a **standalone binary**
(distinct from `cognee-cli`).
This page is the launch/embed surface. The endpoint specs, auth, pipelines,
websockets, tenancy, and observability live in
[HTTP Server Reference](/rust/http-server).
## Run the binary
```bash theme={null}
cargo build --release -p cognee-http-server --features bin --bin cognee-http-server
cognee-http-server --host 0.0.0.0 --port 8000
```
Launch flags (each has an env fallback): `--host` (`HTTP_API_HOST`, `0.0.0.0`),
`--port` (`HTTP_API_PORT`, `8000`), `--env` (`ENV`), `--cors-allowed-origins`
(`CORS_ALLOWED_ORIGINS`). The full server env surface (auth, body limits,
pipeline registry, notebooks, health probes) is in
[`crates/http-server/src/config.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/http-server/src/config.rs) and
summarized in [Configuration: HTTP server](/rust/configuration#http-server).
## Embed the library
```rust theme={null}
let state = AppState::build(config).await?;
let router = cognee_http_server::build_router(state).await?;
// either drive it yourself…
axum::serve(listener, router).await?;
// …or use the helper that binds + serves:
cognee_http_server::run(addr, state).await?;
```
Public entry points: `build_router(AppState)` (assembles the router + middleware +
all sub-routers), `run(addr, AppState)` (binds a listener and serves), and
`AppState`. Routers delegate into `cognee-lib` facades — no business logic is
re-implemented in the server crate. See the
[architecture decisions](https://github.com/topoteretes/cognee-rs/blob/main/docs/http-server/architecture.md) for the dual-surface
design and middleware stack, and [Pluggable Backends](/rust/tools/backends) for how the
server's databases/providers are wired.
# Tools Overview
Source: https://docs.cognee.ai/rust/tools/overview
The ways to drive cognee-rust, the backends it runs on, and the supporting dev/ops tooling.
The ways to drive cognee-rust, the backends it runs on, and the supporting
dev/ops tooling.
## Interfaces
How you invoke the pipeline. All cover the same operations
([Operations](/rust/operations)).
* **[CLI Reference](/rust/tools/cli)** — `cognee-cli`: subcommands, flags, `config`, retries, logging.
* **[Language Bindings](/rust/tools/bindings)** — Python / C / JavaScript SDKs (shared `bindings-common`) + config-setter ergonomics.
* **[HTTP Server Tools](/rust/tools/http-server)** — `cognee-http-server`: launch the binary or embed the library. Endpoint specs under [HTTP Server Reference](/rust/http-server).
## Backends
* **[Pluggable Backends](/rust/tools/backends)** — pluggable providers (LLM, embeddings, vector, graph, relational, storage, session, ontology, tokenizer) with their selecting config keys and rustdoc links.
## Dev & ops tooling
* **Observability** — [OpenTelemetry](https://github.com/topoteretes/cognee-rs/blob/main/docs/observability/opentelemetry.md) (OTLP tracing) and [product analytics](https://github.com/topoteretes/cognee-rs/blob/main/docs/observability/send_telemetry.md) (opt-out product analytics).
* **Logging** — [Configuration: logging](/rust/configuration#logging).
* **Visualization** — `cognee-cli visualize` (see [CLI Reference](/rust/tools/cli)); [`cognee-visualization`](https://github.com/topoteretes/cognee-rs/blob/main/crates/visualization/).
* **Benchmarking** — [Mock benchmark](https://github.com/topoteretes/cognee-rs/blob/main/docs/performance/mock-benchmark.md) (offline mock-LLM benchmark) and its [design rationale](https://github.com/topoteretes/cognee-rs/blob/main/docs/performance/python-approach.md).
* **Build troubleshooting** — [Ladybug rebuilds](https://github.com/topoteretes/cognee-rs/blob/main/docs/build/lbug-rebuilds.md).
* **Releasing** — [Release process](https://github.com/topoteretes/cognee-rs/blob/main/docs/RELEASE.md).
# FalkorDB
Source: https://docs.cognee.ai/setup-configuration/community-maintained/falkordb
Use FalkorDB as a hybrid graph and vector store in Cognee.
FalkorDB is an open-source graph database optimized for GraphRAG.
It supports both cloud-hosted and self-hosted deployments.
<Note>
Cognee can use FalkorDB as both a [vector store](/setup-configuration/vector-stores) and a
[graph store](/setup-configuration/graph-stores) backend through this
[community-maintained](/setup-configuration/community-maintained/overview) [adapter](https://github.com/topoteretes/cognee-community/tree/main/packages/hybrid/falkordb).
</Note>
## Installation
This adapter is a separate package from core Cognee.
Before installing, complete the [Cognee installation](/getting-started/installation) and ensure
your environment is configured with [LLM and embedding providers](/setup-configuration/overview).
After that, install the adapter package:
```bash theme={null}
uv pip install cognee-community-hybrid-adapter-falkor
```
## Configuration
Run a local FalkorDB instance:
```bash theme={null}
docker run -p 6379:6379 -p 3000:3000 -it --rm falkordb/falkordb:edge
```
Configure in Python:
```python theme={null}
from cognee_community_hybrid_adapter_falkor import register
from cognee import config
config.set_vector_db_config(
{
"vector_db_provider": "falkor",
"vector_db_url": "localhost",
"vector_db_port": 6379,
}
)
config.set_graph_db_config(
{
"graph_database_provider": "falkor",
"graph_database_url": "localhost",
"graph_database_port": 6379,
}
)
```
Or via environment variables:
```dotenv theme={null}
VECTOR_DB_PROVIDER="falkor"
VECTOR_DB_URL="http://localhost:6379"
VECTOR_DB_KEY=""
GRAPH_DATABASE_PROVIDER="falkor"
GRAPH_DATABASE_URL="localhost"
GRAPH_DATABASE_PORT="6379"
```
## Important Notes
<Accordion title="Adapter Registration">
Import `register` from the adapter package before using FalkorDB with Cognee. This registers the adapter with Cognee's provider system.
</Accordion>
<Accordion title="Embedding Dimensions">
Ensure `EMBEDDING_DIMENSIONS` matches your embedding model. See [Embedding Providers](/setup-configuration/embedding-providers) for configuration.
Changing dimensions requires recreating collections or running `prune.prune_system()`.
</Accordion>
## Resources
<CardGroup>
<Card title="FalkorDB Docs" icon="book" href="https://docs.falkordb.com/">
Official documentation
</Card>
<Card title="Adapter Source" icon="github" href="https://github.com/topoteretes/cognee-community/tree/main/packages/hybrid/falkordb">
GitHub repository
</Card>
<Card title="Extended Example" icon="lightbulb" href="https://github.com/topoteretes/cognee-community/tree/main/packages/hybrid/falkordb/examples/example.py">
FAQ docs assistant example.
</Card>
</CardGroup>
<Columns>
<Card title="Graph Stores" icon="database" href="/setup-configuration/graph-stores">
Official vector providers
</Card>
<Card title="Community Overview" icon="users" href="/setup-configuration/community-maintained/overview">
All community integrations
</Card>
<Card title="Setup Overview" icon="settings" href="/setup-configuration/overview">
Configuration guide
</Card>
</Columns>
# Memgraph
Source: https://docs.cognee.ai/setup-configuration/community-maintained/memgraph
Use Memgraph as a graph store through a community-maintained adapter
Memgraph is an in-memory graph database that uses the Bolt protocol. It provides fast graph operations and is suitable for knowledge graph storage and relationship reasoning.
<Note>
Cognee can use Memgraph as a [graph store](/setup-configuration/graph-stores) backend through this [community-maintained](/setup-configuration/community-maintained/overview) [adapter](https://github.com/topoteretes/cognee-community/tree/main/packages/graph/memgraph).
</Note>
## Installation
This adapter is a separate package from core Cognee. Before installing, complete the [Cognee installation](/getting-started/installation) and ensure your environment is configured with [LLM and embedding providers](/setup-configuration/overview). After that, install the adapter package:
```bash theme={null}
uv pip install cognee-community-graph-adapter-memgraph
```
## Requirements
* Python 3.10 to 3.13
* Memgraph database instance
* neo4j driver (for Bolt protocol support)
## Configuration
Run a local Memgraph instance:
```bash theme={null}
docker run -d --name memgraph -p 7687:7687 memgraph/memgraph-platform
```
Configure in Python:
```python theme={null}
from cognee_community_graph_adapter_memgraph import register
from cognee import config
register()
config.set_graph_database_provider("memgraph")
config.set_graph_db_config({
"graph_database_url": "bolt://localhost:7687",
"graph_database_username": "memgraph",
"graph_database_password": "memgraph",
})
```
Or via environment variables:
```dotenv theme={null}
GRAPH_DATABASE_PROVIDER="memgraph"
GRAPH_DATABASE_URL="bolt://localhost:7687"
GRAPH_DATABASE_USERNAME="memgraph"
GRAPH_DATABASE_PASSWORD="memgraph"
```
## Important Notes
<Accordion title="Adapter Registration">
Import and call `register()` from the adapter package before using Memgraph with Cognee. This registers the adapter with Cognee's graph provider system.
</Accordion>
<Accordion title="Troubleshooting">
1. **Connection Errors**: Ensure Memgraph is running and accessible at the specified Bolt URL (default port 7687)
2. **Driver**: The adapter uses the neo4j driver for Bolt protocol support; ensure it is installed with Cognee or the community package
3. **Authentication**: Use the same credentials you configured for your Memgraph instance
</Accordion>
## Features
* Full support for Memgraph's property graph model
* Optimized queries for graph operations
* Async/await support and transaction support
* Graph completion search and Chain of Thought (COT) reasoning
* Direct graph data access via `get_graph_engine()` and HTML visualization with `cognee.visualize_graph()`
## Resources
<CardGroup>
<Card title="Adapter Source" icon="github" href="https://github.com/topoteretes/cognee-community/tree/main/packages/graph/memgraph">
GitHub repository
</Card>
<Card title="README & Example" icon="book" href="https://github.com/topoteretes/cognee-community/blob/main/packages/graph/memgraph/README.md">
Full usage example and configuration
</Card>
<Card title="Memgraph Docs" icon="external-link" href="https://memgraph.com/docs">
Memgraph database documentation
</Card>
</CardGroup>
<Columns>
<Card title="Graph Stores" icon="network" href="/setup-configuration/graph-stores">
Official graph providers
</Card>
<Card title="Community Overview" icon="users" href="/setup-configuration/community-maintained/overview">
All community integrations
</Card>
<Card title="Setup Overview" icon="settings" href="/setup-configuration/overview">
Configuration guide
</Card>
</Columns>
# Adapters Overview
Source: https://docs.cognee.ai/setup-configuration/community-maintained/overview
Adapters and extensions built by the Cognee community
Community-maintained integrations are adapters built and maintained by the Cognee community. These extend Cognee's functionality with additional providers and services.
<Note>
Community integrations are maintained separately from the core Cognee package. For issues or contributions, visit the [cognee-community repository](https://github.com/topoteretes/cognee-community).
</Note>
Everything installable lives under `packages/` in that repository. The repository's `experimental/` directory (n8n nodes, dlt demos, bauplan, tower) holds demos, not published packages.
## Available Integrations
### Vector Stores
* **[Qdrant](/setup-configuration/community-maintained/qdrant)** — High-performance vector search engine
* **[Redis](/setup-configuration/community-maintained/redis)** — Fast vector similarity search via Redis Search module
* **[Milvus](https://github.com/topoteretes/cognee-community/tree/main/packages/vector/milvus)** — Cloud-native vector database (docs coming soon)
* **[Pinecone](/setup-configuration/community-maintained/pinecone)** — Managed vector database
* **[Weaviate](https://github.com/topoteretes/cognee-community/tree/main/packages/vector/weaviate)** — Open-source vector search engine (docs coming soon)
* **[Azure AI Search](https://github.com/topoteretes/cognee-community/tree/main/packages/vector/azureaisearch)** — Azure cognitive search service (docs coming soon)
* **[OpenSearch](https://github.com/topoteretes/cognee-community/tree/main/packages/vector/opensearch)** — OpenSearch vector engine (docs coming soon)
* **[Turbopuffer](/setup-configuration/community-maintained/turbopuffer)** — High-performance vector database
* **[MOSS](https://github.com/topoteretes/cognee-community/tree/main/packages/vector/moss)** (docs coming soon)
* **[openGauss](https://github.com/topoteretes/cognee-community/tree/main/packages/vector/opengauss)** (docs coming soon)
* **[SingleStore](https://github.com/topoteretes/cognee-community/tree/main/packages/vector/singlestore)** (docs coming soon)
* **[Valkey](https://github.com/topoteretes/cognee-community/tree/main/packages/vector/valkey)** (docs coming soon)
### Hybrid Stores
Hybrid adapters back both the graph and the vector store with a single database.
* **[DuckDB](https://github.com/topoteretes/cognee-community/tree/main/packages/hybrid/duckdb)** — In-process analytical database (docs coming soon)
* **[FalkorDB](/setup-configuration/community-maintained/falkordb)** — Graph database with vector support (docs coming soon)
* **[ArcadeDB](https://github.com/topoteretes/cognee-community/tree/main/packages/hybrid/arcadedb)** (docs coming soon)
* **[HelixDB](https://github.com/topoteretes/cognee-community/tree/main/packages/hybrid/helixdb)** (docs coming soon)
### Graph Stores
* **[Memgraph](/setup-configuration/community-maintained/memgraph)** — In-memory graph database
* **[NetworkX](https://github.com/topoteretes/cognee-community/tree/main/packages/graph/networkx)** — Python graph library adapter (docs coming soon)
* **[ArcadeDB](https://github.com/topoteretes/cognee-community/tree/main/packages/graph/arcadedb)** (docs coming soon)
* **[pggraph](https://github.com/topoteretes/cognee-community/tree/main/packages/graph/pggraph)** — Graph store on Postgres (docs coming soon)
* **[Spanner](https://github.com/topoteretes/cognee-community/tree/main/packages/graph/spanner)** (docs coming soon)
* **[Turbopuffer](https://github.com/topoteretes/cognee-community/tree/main/packages/graph/turbopuffer)** — Turbopuffer also ships a graph adapter (docs coming soon)
* **[TuringDB](https://github.com/topoteretes/cognee-community/tree/main/packages/graph/turingdb)** (docs coming soon)
### Connectors (data sources)
Connectors expose a `dlt` source you pass straight to `remember()`, reusing Cognee's DLT ingestion path — so snapshot sync and forget-on-delete work without any core changes. Give each connector its own dataset. See the [dlt integration guide](/integrations/dlt-integration) for setup, credentials, and ingestion patterns.
* Slack
* Gmail
* Notion
* Confluence
* Google Drive
### Tasks, Pipelines and Retrievers
* `codify_tasks`, `codify_pipeline`, `code_retriever` — code graph extraction and retrieval
* `exa_tasks` — Exa search tasks
* `scrapegraph_tasks` — [ScrapeGraphAI](/integrations/scrapegraphai-integration) scraping tasks
### Observability
* **[KeywordsAI](https://github.com/topoteretes/cognee-community/tree/main/packages/observability/keywordsai)** — LLM monitoring and analytics; enable with `MONITORING_TOOL=keywordsai` and `KEYWORDSAI_API_KEY` (docs coming soon)
## Installing a community adapter
Community packages generally publish to PyPI as `cognee-community-<family>-<kind>-<name>` and import under the same name with underscores — for example `cognee-community-vector-adapter-qdrant` installs the module `cognee_community_vector_adapter_qdrant`. A few packages deviate slightly (connectors drop the kind, e.g. `cognee-community-connector-slack`), so check the package's `pyproject.toml` or README for the exact name.
Installing is only half the job: the provider name is not valid until the package registers itself, which happens through the package's `register` module. Registration lives in process memory, so it must run in every process, before Cognee touches any engine.
```python theme={null}
# Importing the register module performs the registration — it must happen
# before any Cognee call in this process. A few packages instead expose a
# register() function to call; check the adapter's page or README.
from cognee_community_vector_adapter_qdrant import register # noqa: F401
from cognee import config
config.set_vector_db_config({
"vector_db_provider": "qdrant",
"vector_db_url": "http://localhost:6333",
"vector_db_key": "...",
})
```
Under the hood, `register` calls `use_vector_adapter(name, AdapterClass)` or `use_graph_adapter(...)` to add the provider to Cognee's registry. Setting `VECTOR_DB_PROVIDER` or `GRAPH_DATABASE_PROVIDER` to a community name **without** registering raises:
```
EnvironmentError: Unsupported vector database provider: qdrant. Supported providers are: ...
```
Hybrid adapters register as **both** a graph and a vector adapter, so set both configurations to the same provider name.
<Warning>
**Multi-tenancy caveat.** With `ENABLE_BACKEND_ACCESS_CONTROL=true` (the default), both the graph and the vector backend must have a dataset-database handler, or Cognee raises an `EnvironmentError`. Core ships handlers only for its in-tree backends, so a community adapter works in this mode only if it registers its own handler via `use_dataset_database_handler` in its `register.py`.
Community adapters that ship a handler today: **Qdrant**, **MOSS**, **SingleStore**, **Turbopuffer** (both its vector and its graph adapter), **FalkorDB**, **ArcadeDB** (the hybrid package — the graph-only adapter ships none), and **HelixDB**. Every other community adapter requires `ENABLE_BACKEND_ACCESS_CONTROL=false`. The list moves as packages are updated — the definitive check is grepping the adapter's `register.py` for `use_dataset_database_handler`.
When an adapter does ship a handler, select it by its registered name — `vector_dataset_database_handler` in `set_vector_db_config()` (or `VECTOR_DATASET_DATABASE_HANDLER`), and the graph equivalent. See [dataset database handlers](/core-concepts/multi-user-mode/dataset-database-handlers/dataset-database-handlers-how-to-use-them) for how handlers are selected and registered.
</Warning>
### Connecting to a self-hosted store
`vector_db_url` and `vector_db_key` are not the only connection details Cognee hands a community vector adapter. `set_vector_db_config()` also accepts `vector_db_host`, `vector_db_port`, `vector_db_username`, and `vector_db_password` (or the matching `VECTOR_DB_HOST` / `VECTOR_DB_PORT` / `VECTOR_DB_USERNAME` / `VECTOR_DB_PASSWORD` environment variables), and Cognee forwards all four to the registered adapter's constructor.
Whether these values are actually *used* is up to the individual adapter — Cognee only passes them along. Most existing adapters build their connection from `vector_db_url` (and, for hybrid adapters, the graph configuration) and ignore the rest, so check the adapter's page or README before reaching for these keys. If you maintain an adapter, see [what Cognee passes your constructor](/contributing/adding-providers/adding-new-vector-database#what-cognee-passes-your-constructor) to start accepting them.
<Note>
`vector_db_port` reaches the adapter as a string and defaults to `"1234"` when you don't set one.
</Note>
## Verifying an install
Many packages under `packages/` ship an `examples/example.py` plus a `tests/` directory that goes beyond the example. Where the package provides one, run the example from the package directory to confirm your install and configuration end to end:
```bash theme={null}
uv run python examples/example.py
```
An LLM API key is still required for these runs — `LLM_API_KEY`, OpenAI by default — because the example exercises the full ingestion and retrieval flow, not just the adapter.
## Contributing
To contribute a new community integration, work in the [cognee-community repository](https://github.com/topoteretes/cognee-community):
1. **Branch from `main`.** Unlike the core Cognee repo, cognee-community has no `dev` branch.
2. **Follow the existing package layout:** a directory under `packages/<family>/<name>/` with `pyproject.toml`, a `README.md` covering install and usage, `examples/example.py`, and `tests/`.
3. **For a new database adapter**, implement `VectorDBInterface` or `GraphDBInterface` from core, expose a `register.py`, and run the shared conformance suite in `packages/shared/contract_suite/` (`vector_contract.py` / `graph_contract.py`).
4. **Register a dataset-database handler** with `use_dataset_database_handler(...)` if your backend can isolate per user + dataset — that is what makes it usable with access control enabled.
5. **Name the package** `cognee-community-<family>-<kind>-<name>` and add it to the tables in the repository README. Lint with the repo-root `ruff.toml`.
6. **Open a pull request** with your integration and its documentation.
The [vector](/contributing/adding-providers/adding-new-vector-database) and [graph](/contributing/adding-providers/adding-new-graph-database) adapter guides walk through these steps in detail.
## Support
For community integration support:
* Check the integration's README in the repository
* Open issues in the cognee-community repository
* Join the [Discord community](https://discord.gg/cqF6RhDYWz) for help
<Columns>
<Card title="Vector Stores" icon="database" href="/setup-configuration/vector-stores">
Official vector store providers
</Card>
<Card title="Setup Overview" icon="settings" href="/setup-configuration/overview">
Configuration overview
</Card>
</Columns>
# Pinecone
Source: https://docs.cognee.ai/setup-configuration/community-maintained/pinecone
Use Pinecone as a vector store through a community-maintained adapter
Pinecone is a managed vector database service that provides high-performance similarity search. It is available as a community-maintained adapter for Cognee.
<Note>
Cognee can use Pinecone as a [vector store](/setup-configuration/vector-stores) backend through this [community-maintained](/setup-configuration/community-maintained/overview) [adapter](https://github.com/topoteretes/cognee-community/tree/main/packages/vector/pinecone).
</Note>
## Installation
This adapter is a separate package from core Cognee. Before installing, complete the [Cognee installation](/getting-started/installation) and ensure your environment is configured with [LLM and embedding providers](/setup-configuration/overview). After that, install the adapter package:
```bash theme={null}
pip install cognee-community-vector-adapter-pinecone
```
## Registration
Unlike built-in providers, community adapters must be explicitly registered at the start of your script **before** calling any Cognee functions. This registers Pinecone in Cognee's list of known vector database providers:
```python theme={null}
from cognee_community_vector_adapter_pinecone import register
register()
```
If you skip this step, Cognee will not recognize `pinecone` as a valid provider and will raise a runtime error even if the package is installed.
## Configuration
Get your API key from the [Pinecone console](https://app.pinecone.io/) and configure Cognee to use it.
<Tabs>
<Tab title="Python (Programmatic)">
```python theme={null}
from cognee_community_vector_adapter_pinecone import register
from cognee import config
# Must register before configuring or using Cognee
register()
config.set_vector_db_config({
"vector_db_provider": "pinecone",
"vector_db_url": "https://your-index-host.pinecone.io",
"vector_db_key": "your_pinecone_api_key",
})
```
</Tab>
<Tab title="Environment Variables">
Set these in your `.env` file. **Remember**: you must still call `register()` in your code even when using environment variables.
```dotenv theme={null}
VECTOR_DB_PROVIDER="pinecone"
VECTOR_DB_URL="https://your-index-host.pinecone.io"
VECTOR_DB_KEY="your_pinecone_api_key"
```
Then in your script:
```python theme={null}
from cognee_community_vector_adapter_pinecone import register
register()
import cognee
# ... proceed with cognee operations
```
</Tab>
</Tabs>
## Important Notes
<Accordion title="Community adapter required">
Pinecone is not bundled with the default Cognee package. You must install and register the community adapter before configuring `VECTOR_DB_PROVIDER=pinecone`.
</Accordion>
<Accordion title="Registration is required every run">
The `register()` call must happen **before** any Cognee vector operations in each process. It is not persisted — you cannot register once and skip it on subsequent runs. Add it at the top of your entry-point script or application startup code.
</Accordion>
<Accordion title="Unsupported provider error">
If you set `VECTOR_DB_PROVIDER=pinecone` before installing and registering the adapter, Cognee will raise this runtime error:
```text theme={null}
EnvironmentError: Unsupported vector database provider: pinecone
```
Complete the installation and registration steps above before configuring Pinecone.
</Accordion>
<Accordion title="Embedding Dimensions">
Ensure `EMBEDDING_DIMENSIONS` matches the dimensions configured in your Pinecone index. A mismatch will cause errors when upserting vectors.
See [Embedding Providers](/setup-configuration/embedding-providers) for configuration details.
</Accordion>
## Resources
<CardGroup>
<Card title="Adapter Source" icon="github" href="https://github.com/topoteretes/cognee-community/tree/main/packages/vector/pinecone">
GitHub repository
</Card>
<Card title="Pinecone Docs" icon="book" href="https://docs.pinecone.io/">
Official Pinecone documentation
</Card>
</CardGroup>
<Columns>
<Card title="Vector Stores" icon="database" href="/setup-configuration/vector-stores">
Official vector providers
</Card>
<Card title="Community Overview" icon="users" href="/setup-configuration/community-maintained/overview">
All community integrations
</Card>
<Card title="Setup Overview" icon="settings" href="/setup-configuration/overview">
Configuration guide
</Card>
</Columns>
# Qdrant
Source: https://docs.cognee.ai/setup-configuration/community-maintained/qdrant
Use Qdrant as a vector store through a community-maintained adapter
Qdrant is a vector search engine that stores embeddings and performs similarity searches. It supports both cloud-hosted and self-hosted deployments.
<Note>
Cognee can use Qdrant as a [vector store](/setup-configuration/vector-stores) backend through this [community-maintained](/setup-configuration/community-maintained/overview) [adapter](https://github.com/topoteretes/cognee-community/tree/main/packages/vector/qdrant).
</Note>
## Installation
This adapter is a separate package from core Cognee. Before installing, complete the [Cognee installation](/getting-started/installation) and ensure your environment is configured with [LLM and embedding providers](/setup-configuration/overview). After that, install the adapter package:
```bash theme={null}
uv pip install cognee-community-vector-adapter-qdrant
```
## Configuration
<Tabs>
<Tab title="Docker (Local)">
Run a local Qdrant instance:
```bash theme={null}
docker run -p 6333:6333 -p 6334:6334 \
-v "$(pwd)/qdrant_storage:/qdrant/storage:z" \
qdrant/qdrant
```
Configure in Python:
```python theme={null}
from cognee_community_vector_adapter_qdrant import register
from cognee import config
register() # registers "qdrant" as a vector provider — must run before any cognee call
config.set_vector_db_config({
"vector_db_provider": "qdrant",
"vector_db_url": "http://localhost:6333",
"vector_db_key": "",
})
```
Or via environment variables:
```dotenv theme={null}
VECTOR_DB_PROVIDER="qdrant"
VECTOR_DB_URL="http://localhost:6333"
VECTOR_DB_KEY=""
```
</Tab>
<Tab title="Qdrant Cloud">
Get your API key and URL from the [Qdrant Cloud](https://qdrant.tech/documentation/cloud/) dashboard.
```python theme={null}
from cognee_community_vector_adapter_qdrant import register
from cognee import config
register() # registers "qdrant" as a vector provider — must run before any cognee call
config.set_vector_db_config({
"vector_db_provider": "qdrant",
"vector_db_url": "https://your-cluster.qdrant.io",
"vector_db_key": "your_api_key",
})
```
Or via environment variables:
```dotenv theme={null}
VECTOR_DB_PROVIDER="qdrant"
VECTOR_DB_URL="https://your-cluster.qdrant.io"
VECTOR_DB_KEY="your_api_key"
```
</Tab>
</Tabs>
## Important Notes
<Accordion title="Calling register() (fixes 'Unsupported vector database provider: qdrant')">
Qdrant is not built into core Cognee — it is a community adapter. Simply setting `VECTOR_DB_PROVIDER="qdrant"` is not enough; core Cognee only knows about `qdrant` once the adapter registers itself. If you skip this step you will hit:
```
OSError: Unsupported vector database provider: qdrant. Supported providers are: LanceDB, PGVector, neptune_analytics
```
Registration happens by **calling** `register()` (importing it is not enough) from the installed adapter package. Add the call in your application's entrypoint — the same Python process that later calls `add`, `cognify`, or `search` — and run it once, before any Cognee operation:
```python theme={null}
from cognee_community_vector_adapter_qdrant import register
import cognee
register() # must run before add/cognify/search
async def main():
await cognee.add("Cognee turns your data into an AI memory.")
await cognee.cognify()
results = await cognee.search("What does cognee do?")
print(results)
```
In Docker or any long-running server deployment the same rule applies: `register()` must run inside the container at startup (for example at the top of the module that boots your app), because registration lives in process memory and is not persisted. Setting the environment variable alone will not register the adapter.
</Accordion>
<Accordion title="Embedding Dimensions">
Ensure `EMBEDDING_DIMENSIONS` matches your embedding model. See [Embedding Providers](/setup-configuration/embedding-providers) for configuration.
Changing dimensions requires recreating collections or running `prune.prune_system()`.
</Accordion>
## Resources
<CardGroup>
<Card title="Qdrant Docs" icon="book" href="https://qdrant.tech/documentation/">
Official documentation
</Card>
<Card title="Adapter Source" icon="github" href="https://github.com/topoteretes/cognee-community/tree/main/packages/vector/qdrant">
GitHub repository
</Card>
<Card title="Extended Example" icon="lightbulb" href="https://github.com/topoteretes/cognee-community/tree/main/packages/vector/qdrant/example.py">
FAQ docs assistant example.
</Card>
</CardGroup>
<Columns>
<Card title="Vector Stores" icon="database" href="/setup-configuration/vector-stores">
Official vector providers
</Card>
<Card title="Community Overview" icon="users" href="/setup-configuration/community-maintained/overview">
All community integrations
</Card>
<Card title="Setup Overview" icon="settings" href="/setup-configuration/overview">
Configuration guide
</Card>
</Columns>
# Redis
Source: https://docs.cognee.ai/setup-configuration/community-maintained/redis
Use Redis as a vector store through a community-maintained adapter
Redis is a fast in-memory data store that supports vector similarity search through the Redis Search module. It supports both cloud-hosted (Redis Cloud) and self-hosted deployments.
<Note>
Cognee can use Redis as a [vector store](/setup-configuration/vector-stores) backend through this [community-maintained](/setup-configuration/community-maintained/overview) [adapter](https://github.com/topoteretes/cognee-community/tree/main/packages/vector/redis).
</Note>
## Installation
This adapter is a separate package from core Cognee. Before installing, complete the [Cognee installation](/getting-started/installation) and ensure your environment is configured with [LLM and embedding providers](/setup-configuration/overview). After that, install the adapter package:
```bash theme={null}
uv pip install cognee-community-vector-adapter-redis
```
## Configuration
<Tabs>
<Tab title="Docker (Local)">
Run a local Redis instance with the Search module enabled:
```bash theme={null}
docker run -d --name redis -p 6379:6379 redis:8.0.2
```
Configure in Python:
```python theme={null}
from cognee_community_vector_adapter_redis import register
from cognee import config
config.set_vector_db_config({
"vector_db_provider": "redis",
"vector_db_url": "redis://localhost:6379",
})
```
Or via environment variables:
```dotenv theme={null}
VECTOR_DB_PROVIDER="redis"
VECTOR_DB_URL="redis://localhost:6379"
```
</Tab>
<Tab title="Redis Cloud">
Get your connection URL from the [Redis Cloud](https://redis.io/try-free) dashboard. Make sure to enable the Search module.
```python theme={null}
from cognee_community_vector_adapter_redis import register
from cognee import config
config.set_vector_db_config({
"vector_db_provider": "redis",
"vector_db_url": "redis://user:password@your-redis-cloud-url:port",
})
```
Or via environment variables:
```dotenv theme={null}
VECTOR_DB_PROVIDER="redis"
VECTOR_DB_URL="redis://user:password@your-redis-cloud-url:port"
```
</Tab>
<Tab title="Redis with SSL">
For secure connections, use the `rediss://` protocol:
```python theme={null}
from cognee_community_vector_adapter_redis import register
from cognee import config
config.set_vector_db_config({
"vector_db_provider": "redis",
"vector_db_url": "rediss://localhost:6380",
})
```
Or via environment variables:
```dotenv theme={null}
VECTOR_DB_PROVIDER="redis"
VECTOR_DB_URL="rediss://localhost:6380"
```
</Tab>
</Tabs>
## Important Notes
<Accordion title="Adapter Registration">
Import `register` from the adapter package before using Redis with Cognee. This registers the adapter with Cognee's provider system.
</Accordion>
<Accordion title="Troubleshooting">
1. **Connection Errors**: Ensure Redis is running and accessible at the specified URL
2. **Search Module Missing**: Make sure Redis has the Search module enabled
3. **Embedding Dimension Mismatch**: Verify embedding engine dimensions match index configuration
4. **Collection Not Found**: Always create collections before adding data points
</Accordion>
## Resources
<CardGroup>
<Card title="RedisVL Docs" icon="book" href="https://docs.redisvl.com">
RedisVL library documentation (powers this adapter)
</Card>
<Card title="Adapter Source" icon="github" href="https://github.com/topoteretes/cognee-community/tree/main/packages/vector/redis">
GitHub repository
</Card>
<Card title="Extended Example" icon="lightbulb" href="https://github.com/topoteretes/cognee-community/blob/main/packages/vector/redis/examples/example.py">
Full usage example script
</Card>
</CardGroup>
<Columns>
<Card title="Vector Stores" icon="database" href="/setup-configuration/vector-stores">
Official vector providers
</Card>
<Card title="Community Overview" icon="users" href="/setup-configuration/community-maintained/overview">
All community integrations
</Card>
<Card title="Setup Overview" icon="settings" href="/setup-configuration/overview">
Configuration guide
</Card>
</Columns>
# Turbopuffer
Source: https://docs.cognee.ai/setup-configuration/community-maintained/turbopuffer
Use Turbopuffer as a vector store through a community-maintained adapter
Turbopuffer is a high-performance vector database designed for fast similarity search at scale.
<Note>
Cognee can use Turbopuffer as a [vector store](/setup-configuration/vector-stores) backend through this [community-maintained](/setup-configuration/community-maintained/overview) adapter.
</Note>
## Installation
This adapter is a separate package from core Cognee. Before installing, complete the [Cognee installation](/getting-started/installation) and ensure your environment is configured with [LLM and embedding providers](/setup-configuration/overview). After that, install the adapter package:
```bash theme={null}
pip install cognee-community-vector-adapter-turbopuffer
```
## Configuration
1. Create an account at [turbopuffer.com](https://turbopuffer.com)
2. Get your API key from the [dashboard](https://turbopuffer.com/dashboard)
Set the following environment variables in your `.env` file:
```dotenv theme={null}
TURBOPUFFER_API_KEY="your_api_key"
VECTOR_DATASET_DATABASE_HANDLER="turbopuffer"
# Optional: defaults to gcp-us-central1
# TURBOPUFFER_REGION="gcp-us-central1"
```
Then register the adapter in your code:
```python theme={null}
from cognee_community_vector_adapter_turbopuffer import register
```
See [available regions](https://turbopuffer.com/docs/regions) for supported `TURBOPUFFER_REGION` values.
## Important Notes
<Accordion title="Embedding Dimensions">
Ensure `EMBEDDING_DIMENSIONS` matches your embedding model. See [Embedding Providers](/setup-configuration/embedding-providers) for configuration.
Changing dimensions requires recreating collections or running `prune.prune_system()`.
</Accordion>
## Resources
<CardGroup>
<Card title="Turbopuffer Docs" icon="book" href="https://turbopuffer.com/docs">
Official documentation
</Card>
<Card title="Adapter Source" icon="github" href="https://github.com/topoteretes/cognee-community/tree/main/packages/vector/turbopuffer">
GitHub repository
</Card>
</CardGroup>
<Columns>
<Card title="Vector Stores" icon="database" href="/setup-configuration/vector-stores">
Official vector providers
</Card>
<Card title="Community Overview" icon="users" href="/setup-configuration/community-maintained/overview">
All community integrations
</Card>
<Card title="Setup Overview" icon="settings" href="/setup-configuration/overview">
Configuration guide
</Card>
</Columns>
# Embedding Providers
Source: https://docs.cognee.ai/setup-configuration/embedding-providers
Configure embedding providers for semantic search in Cognee
Embedding providers convert text into vector representations that enable semantic search. These vectors capture the meaning of text, allowing Cognee to find conceptually related content even when the wording is different.
<Info>
**New to configuration?**
See the [Setup Configuration Overview](./overview) for the complete workflow:
install extras → create `.env` → choose providers → handle pruning.
</Info>
## Supported Providers
Cognee supports multiple embedding providers:
* **OpenAI** — Text embedding models via OpenAI API (default)
* **Azure OpenAI** — Text embedding models via Azure OpenAI Service
* **Google Gemini** — Embedding models via Google AI
* **Mistral** — Embedding models via Mistral AI
* **AWS Bedrock** — Embedding models via AWS Bedrock
* **Ollama** — Local embedding models via Ollama
* **LM Studio** — Local embedding models via LM Studio
* **Fastembed** — CPU-friendly local embeddings
* **HuggingFace** — Embedding models via HuggingFace Inference API or Inference Endpoints
* **vLLM** — Self-hosted embedding models via vLLM
* **OpenAI-Compatible** — Direct OpenAI SDK for llama.cpp, vLLM, TEI, and any `/v1/embeddings` server (bypasses LiteLLM)
* **Custom** — OpenAI-compatible embedding endpoints routed through LiteLLM (DeepInfra, company-internal)
<Warning>
**LLM/Embedding Configuration**: If you configure only LLM or only embeddings, the other defaults to OpenAI. Cognee rejects this mismatch up front — `add()` and `remember()` fail with `ProviderConfigMismatchError` before any ingestion work happens. Configure both LLM and embeddings, or keep a working OpenAI API key for the side you leave at its defaults — see [LLM/Embedding Configuration](/setup-configuration/overview#configuration-workflow).
</Warning>
## Configuration
<Accordion title="Environment Variables">
Set these environment variables in your `.env` file:
* `EMBEDDING_PROVIDER` — The provider to use: `openai`, `gemini`, `mistral`, `bedrock`, `ollama`, `fastembed`, `openai_compatible`, `custom`. These are **not** the same values `LLM_PROVIDER` accepts — see [Valid EMBEDDING\_PROVIDER values and endpoint URL forms](#valid-embedding_provider-values-and-endpoint-url-forms)
* `EMBEDDING_MODEL` — The specific embedding model to use
* `EMBEDDING_DIMENSIONS` — The vector dimension size (must match your vector store)
* `EMBEDDING_API_KEY` — Your API key (falls back to `LLM_API_KEY` if not set — with [one exception](#valid-embedding_provider-values-and-endpoint-url-forms) when `LLM_PROVIDER="custom"`)
* `EMBEDDING_ENDPOINT` — Custom endpoint URL (for Azure, Ollama, or custom providers). **`EMBEDDING_API_BASE` is accepted as an alias**, since that is the name the LiteLLM and OpenAI ecosystem uses; `EMBEDDING_ENDPOINT` wins when both are set. Before this alias existed, a custom base set only as `EMBEDDING_API_BASE` was silently ignored and requests went to `api.openai.com` instead
* `EMBEDDING_API_VERSION` — API version (for Azure OpenAI)
* `EMBEDDING_MAX_COMPLETION_TOKENS` — Maximum **input** tokens per embedded text; used for tokenizer-based chunk sizing (optional, default `8191`). Set it to your embedding model's real input limit — see [Max Completion Tokens](#max-completion-tokens)
* `HUGGINGFACE_TOKENIZER` — HuggingFace Hub model ID that overrides the tokenizer Cognee uses for token counting when the embedding model is not itself a HuggingFace repo. Commonly used with Ollama embeddings (for example, `nomic-ai/nomic-embed-text-v1.5`).
For every embedding variable Cognee reads — including batching, concurrency, and rate limiting — with its default in one table, see the [embedding environment variable reference](/setup-configuration/overview#environment-variable-quick-reference).
</Accordion>
## Provider Setup Guides
<AccordionGroup>
<Accordion title="OpenAI (Default)">
OpenAI provides high-quality embeddings with good performance.
```dotenv theme={null}
EMBEDDING_PROVIDER="openai"
EMBEDDING_MODEL="openai/text-embedding-3-large"
EMBEDDING_DIMENSIONS="3072"
# Optional
# EMBEDDING_API_KEY=sk-... # falls back to LLM_API_KEY if omitted
# EMBEDDING_ENDPOINT=https://api.openai.com/v1
# EMBEDDING_API_VERSION=
# EMBEDDING_MAX_COMPLETION_TOKENS=8191
```
</Accordion>
<Accordion title="Azure OpenAI Embeddings">
Use Azure OpenAI Service for embeddings with your own deployment.
```dotenv theme={null}
EMBEDDING_PROVIDER="openai"
EMBEDDING_MODEL="azure/text-embedding-3-large"
EMBEDDING_ENDPOINT="https://<your-az>.cognitiveservices.azure.com/openai/deployments/text-embedding-3-large"
EMBEDDING_API_KEY="az-..."
EMBEDDING_API_VERSION="2023-05-15"
EMBEDDING_DIMENSIONS="3072"
```
If startup fails with `KeyError: 'Could not automatically map text-embedding-3-large to a tokeniser.'`, the installed `tiktoken` is too old to recognise the model. Cognee strips the `azure/` prefix and asks TikToken for the encoding of `text-embedding-3-large`, which requires `tiktoken>=0.5.2` (Cognee pins `>=0.8.0`). Upgrade in your environment:
```bash theme={null}
pip install --upgrade tiktoken
```
</Accordion>
<Accordion title="Google Gemini">
Use Google's embedding models for semantic search.
```dotenv theme={null}
EMBEDDING_PROVIDER="gemini"
EMBEDDING_MODEL="gemini/gemini-embedding-001"
EMBEDDING_API_KEY="AIza..."
EMBEDDING_DIMENSIONS="768"
```
</Accordion>
<Accordion title="Mistral">
Use Mistral's embedding models for high-quality vector representations.
```dotenv theme={null}
EMBEDDING_PROVIDER="mistral"
EMBEDDING_MODEL="mistral/mistral-embed"
EMBEDDING_API_KEY="sk-mis-..."
EMBEDDING_DIMENSIONS="1024"
```
**Installation**: Install the required dependency:
```bash theme={null}
pip install mistral-common[sentencepiece]
```
</Accordion>
<Accordion title="AWS Bedrock">
Use embedding models provided by the AWS Bedrock service.
```dotenv theme={null}
EMBEDDING_PROVIDER="bedrock"
EMBEDDING_MODEL="<your_model_name>"
EMBEDDING_DIMENSIONS="<dimensions_of_the_model>"
EMBEDDING_API_KEY="<your_api_key>"
EMBEDDING_MAX_COMPLETION_TOKENS="<max_tokens_of_your_model>"
```
</Accordion>
<Accordion title="Ollama (Local)">
Run embedding models locally with Ollama for privacy and cost control.
```dotenv theme={null}
EMBEDDING_PROVIDER="ollama"
EMBEDDING_MODEL="nomic-embed-text:latest"
EMBEDDING_ENDPOINT="http://localhost:11434/api/embed"
EMBEDDING_DIMENSIONS="768"
HUGGINGFACE_TOKENIZER="nomic-ai/nomic-embed-text-v1.5"
```
`HUGGINGFACE_TOKENIZER` is the HuggingFace repo ID of the tokenizer used for token-length counting when sending requests to the Ollama embedding endpoint.
**Installation**: Install Ollama from [ollama.ai](https://ollama.ai) and pull your desired embedding model:
```bash theme={null}
ollama pull nomic-embed-text:latest
```
`HUGGINGFACE_TOKENIZER` is **optional**. It is no longer part of Cognee's required startup validation, so setting `EMBEDDING_PROVIDER`, `EMBEDDING_MODEL`, and `EMBEDDING_DIMENSIONS` without it no longer raises a `ValidationError` on import. It is still recommended for Ollama: set it to the HuggingFace repo ID for the tokenizer that matches your embedding model so token-length counts stay accurate. See the `HUGGINGFACE_TOKENIZER environment variable` section below for how to find the correct value for your model.
If `HUGGINGFACE_TOKENIZER` is unset or points at a repo that cannot be loaded, tokenizer resolution no longer raises — Cognee logs an advisory warning and falls back to the TikToken tokenizer, so ingestion continues with approximate token counts. Cognee also logs an advisory warning whenever the `HUGGINGFACE_TOKENIZER` value differs from the `EMBEDDING_MODEL` id, so a genuine mismatch is not silent. Because an Ollama tag such as `nomic-embed-text:latest` is never identical to its HuggingFace repo id (`nomic-ai/nomic-embed-text-v1.5`), you will see this advisory even for a correct setup — it is a reminder to confirm the two share a tokenizer, not an error.
If a text input exceeds the model's context window, the Ollama embedding engine automatically falls back by splitting the batch in half and retrying both halves. For a single overlong text, it splits the string into two overlapping segments and averages the resulting embeddings. Cognee no longer pre-truncates text before sending it to Ollama, so this fallback only activates when the server returns a context-length error.
<Info>
**Zero-API-key setup**: To run fully offline with no OpenAI key, you must configure both the LLM provider **and** the embedding provider to use local backends. See the [Local Setup guide](/guides/local-setup) for a complete combined `.env` example.
</Info>
<Tip>
**Ollama falling behind?** Ollama processes requests sequentially. If it becomes unresponsive or returns errors under load, reduce `EMBEDDING_BATCH_SIZE` (default `36`) to send fewer chunks per call — values between `1` and `10` work well for most local hardware:
```dotenv theme={null}
EMBEDDING_BATCH_SIZE="5"
```
</Tip>
</Accordion>
<Accordion title="LM Studio (Local)">
Run embedding models locally with LM Studio for privacy and cost control.
```dotenv theme={null}
EMBEDDING_PROVIDER="custom"
EMBEDDING_MODEL="lm_studio/text-embedding-nomic-embed-text-1.5"
EMBEDDING_ENDPOINT="http://127.0.0.1:1234/v1"
EMBEDDING_API_KEY="."
EMBEDDING_DIMENSIONS="768"
```
**Installation**: Install LM Studio from [lmstudio.ai](https://lmstudio.ai/) and download your desired model from
LM Studio's interface.
Load your model, start the LM Studio server, and Cognee will be able to connect to it.
</Accordion>
<Accordion title="Fastembed (Local)">
Use Fastembed for CPU-friendly local embeddings without GPU requirements. Fastembed runs in-process via ONNX Runtime — no separate server, no API key, and no GPU needed.
```dotenv theme={null}
EMBEDDING_PROVIDER="fastembed"
EMBEDDING_MODEL="sentence-transformers/all-MiniLM-L6-v2"
EMBEDDING_DIMENSIONS="384"
```
**Installation**: Fastembed ships as an optional extra — it is **not** included in the base `cognee` install. Add it with:
```bash theme={null}
pip install 'cognee[fastembed]'
```
This pulls in `fastembed` and a compatible `onnxruntime` build. The first call downloads and caches the model weights from Hugging Face, so the initial run needs network access and a few hundred MB of disk for the model cache.
**Supported models**: Any model listed by [`fastembed`'s `TextEmbedding.list_supported_models()`](https://github.com/qdrant/fastembed). Common choices:
| `EMBEDDING_MODEL` | `EMBEDDING_DIMENSIONS` |
| ---------------------------------------- | ---------------------- |
| `sentence-transformers/all-MiniLM-L6-v2` | `384` |
| `BAAI/bge-small-en-v1.5` | `384` |
| `BAAI/bge-base-en-v1.5` | `768` |
| `BAAI/bge-large-en-v1.5` | `1024` |
| `nomic-ai/nomic-embed-text-v1.5` | `768` |
| `intfloat/multilingual-e5-large` | `1024` |
If `EMBEDDING_DIMENSIONS` is omitted, Cognee tries to auto-derive it from the fastembed model registry; set it explicitly to avoid a fallback to `3072` if lookup fails.
<Info>
**Context window handling**: When a text input exceeds the model's context window, Fastembed automatically splits the batch and retries. For a single overlong text, it splits the string into two overlapping segments and averages the resulting embeddings. If a single string is already too short to split further yet still exceeds the context window, Fastembed raises the terminal `EmbeddingContextWindowTooSmallError` instead of retrying (see [Timeout and Retry Behavior](#timeout-and-retry-behavior)). This mirrors the behavior of the OpenAI-compatible engine.
</Info>
<Info>
**Token counting**: Fastembed now counts tokens with the embedding model's own HuggingFace tokenizer (BGE, MiniLM, and E5 are wordpiece models), resolved from the fastembed model list. Earlier releases counted every Fastembed model with the OpenAI `gpt-4o` BPE tokenizer, which mis-sized chunks and skewed the `--dry-run` token estimate. You do not need to set `HUGGINGFACE_TOKENIZER` for Fastembed. If a model is not in the known list, Cognee logs an advisory warning and falls back to the TikToken tokenizer.
</Info>
</Accordion>
<Accordion title="HuggingFace">
Use embedding models from HuggingFace via the [HuggingFace Inference API](https://huggingface.co/docs/api-inference/index) (serverless) or dedicated [Inference Endpoints](https://huggingface.co/docs/inference-endpoints/index).
<Tabs>
<Tab title="Serverless">
```dotenv theme={null}
EMBEDDING_PROVIDER="custom"
EMBEDDING_MODEL="huggingface/BAAI/bge-large-en-v1.5"
EMBEDDING_API_KEY="hf_..."
EMBEDDING_DIMENSIONS="1024"
```
</Tab>
<Tab title="Dedicated Endpoint">
```dotenv theme={null}
EMBEDDING_PROVIDER="custom"
EMBEDDING_MODEL="huggingface/BAAI/bge-large-en-v1.5"
EMBEDDING_ENDPOINT="https://<your-endpoint-id>.<region>.aws.endpoints.huggingface.cloud"
EMBEDDING_API_KEY="hf_..."
EMBEDDING_DIMENSIONS="1024"
```
</Tab>
</Tabs>
**Installation**: Install the HuggingFace extra for tokenizer support:
```bash theme={null}
pip install cognee[huggingface]
```
<Info>
**HUGGINGFACE\_TOKENIZER with HuggingFace embeddings**: When using `EMBEDDING_PROVIDER="custom"` with a `huggingface/` model, Cognee automatically attempts to load a HuggingFace tokenizer from the model repo for token counting. If that fails, it falls back to the TikToken tokenizer. You do not need to set `HUGGINGFACE_TOKENIZER` manually for this provider — it is only required when using `EMBEDDING_PROVIDER="ollama"` (see the Ollama section above).
</Info>
</Accordion>
<Accordion title="vLLM">
Use vLLM to serve local or self-hosted embedding models with an OpenAI-compatible API.
**Example with Qwen3-Embedding-4B on port 8001:**
```dotenv theme={null}
EMBEDDING_PROVIDER="custom"
EMBEDDING_MODEL="hosted_vllm/Qwen/Qwen3-Embedding-4B"
EMBEDDING_ENDPOINT="http://localhost:8001/v1"
EMBEDDING_API_KEY="."
EMBEDDING_DIMENSIONS="2560"
```
<Warning>
**`hosted_vllm/` prefix required**: Include `hosted_vllm/` at the start of the model name so LiteLLM routes requests to your vLLM server. The model name after the prefix should match the model ID returned by your vLLM server's `/v1/models` endpoint.
</Warning>
**Tokenization**: Cognee automatically strips the `hosted_vllm/` prefix when loading the HuggingFace tokenizer, so no separate `HUGGINGFACE_TOKENIZER` setting is needed as long as the model name after the prefix is a valid HuggingFace model ID.
To verify the model name your vLLM server exposes, run:
```bash theme={null}
curl http://localhost:8001/v1/models
```
See the [LiteLLM vLLM documentation](https://docs.litellm.ai/docs/providers/vllm) for more details.
</Accordion>
<Accordion title="OpenAI-Compatible Local Servers (llama.cpp, TEI, vLLM)">
Use `EMBEDDING_PROVIDER="openai_compatible"` for any local inference server that exposes the standard `/v1/embeddings` endpoint. This provider talks directly to OpenAI-compatible embedding servers via the OpenAI Python SDK, bypassing LiteLLM.
Use this provider for: llama.cpp (`llama-server --embedding`), vLLM, Hugging Face TEI, LocalAI, Infinity, and similar servers.
<Tabs>
<Tab title="llama.cpp">
Start llama.cpp with embedding support:
```bash theme={null}
llama-server --model your-model.gguf --embedding --port 8080
```
```dotenv theme={null}
EMBEDDING_PROVIDER="openai_compatible"
EMBEDDING_MODEL="default"
EMBEDDING_ENDPOINT="http://localhost:8080/v1"
EMBEDDING_API_KEY="no-key-required"
EMBEDDING_DIMENSIONS="768"
```
</Tab>
<Tab title="vLLM">
```dotenv theme={null}
EMBEDDING_PROVIDER="openai_compatible"
EMBEDDING_MODEL="BAAI/bge-large-en-v1.5"
EMBEDDING_ENDPOINT="http://localhost:8001/v1"
EMBEDDING_API_KEY="."
EMBEDDING_DIMENSIONS="1024"
```
Unlike `EMBEDDING_PROVIDER="custom"` (LiteLLM), you do **not** need a `hosted_vllm/` prefix in the model name — use the model ID directly as reported by your vLLM server's `/v1/models` endpoint.
</Tab>
<Tab title="Hugging Face TEI">
```dotenv theme={null}
EMBEDDING_PROVIDER="openai_compatible"
EMBEDDING_MODEL="BAAI/bge-large-en-v1.5"
EMBEDDING_ENDPOINT="http://localhost:8080/v1"
EMBEDDING_API_KEY="."
EMBEDDING_DIMENSIONS="1024"
```
</Tab>
</Tabs>
<Info>
**Endpoint normalisation**: The engine automatically appends `/v1` to `EMBEDDING_ENDPOINT` if it is missing, and strips a trailing `/embeddings` suffix. You can pass either `http://localhost:8080` or `http://localhost:8080/v1` — both work.
</Info>
<Info>
**Tokenizer and token limits**: The `openai_compatible` engine automatically loads a tokenizer for chunk sizing. It first tries to load a HuggingFace tokenizer matching `EMBEDDING_MODEL`; if that fails (for example, because the model name is a local alias not on the HuggingFace Hub), it falls back to the TikToken tokenizer. The token limit passed to the tokenizer is controlled by `EMBEDDING_MAX_COMPLETION_TOKENS` (default `8191`). Set this to match your server's **input** length limit if it differs from the default:
```dotenv theme={null}
EMBEDDING_MAX_COMPLETION_TOKENS="4096"
```
This provider also works with hosted third-party `/v1/embeddings` endpoints, which usually enforce a smaller input length than the model's advertised context. See [Max Completion Tokens](#max-completion-tokens) for how to pick the value and how it interacts with `EMBEDDING_BATCH_SIZE`.
</Info>
<Info>
**HUGGINGFACE\_TOKENIZER is not needed for this provider.** The engine automatically tries to load a HuggingFace tokenizer using the model name (e.g. `BAAI/bge-large-en-v1.5`) for token counting. If that fails — for example when using `EMBEDDING_MODEL="default"` with llama.cpp — Cognee logs an advisory warning and falls back to TikToken, so token counts become approximate but ingestion continues. You do not need to set `HUGGINGFACE_TOKENIZER` when using `EMBEDDING_PROVIDER="openai_compatible"`.
</Info>
</Accordion>
<Accordion title="Custom Providers">
Use OpenAI-compatible embedding endpoints from other providers such as DeepInfra, OpenRouter, or a company-internal server. These are routed through LiteLLM and require a provider prefix in the model name.
**Required variables**: `EMBEDDING_PROVIDER="custom"`, `EMBEDDING_MODEL` (with a LiteLLM provider prefix such as `openrouter/`, `deepinfra/`, or `openai/`), `EMBEDDING_API_KEY`, and `EMBEDDING_DIMENSIONS` (set it explicitly — `custom` models are not in the auto-derive registry, so it otherwise falls back to `3072` and causes a vector-store shape mismatch). `EMBEDDING_ENDPOINT` is **optional**: omit it for a named LiteLLM prefix like `openrouter/` (LiteLLM supplies the base URL), and set it only when pointing at a specific `api_base` such as DeepInfra or a self-hosted server.
<Tabs>
<Tab title="DeepInfra">
```dotenv theme={null}
EMBEDDING_PROVIDER="custom"
EMBEDDING_MODEL="deepinfra/BAAI/bge-base-en-v1.5"
EMBEDDING_ENDPOINT="https://api.deepinfra.com/v1/openai"
EMBEDDING_API_KEY="<your-deepinfra-api-key>"
EMBEDDING_DIMENSIONS="768"
```
</Tab>
<Tab title="OpenRouter">
Use the `openrouter/` model prefix. Do not set `EMBEDDING_ENDPOINT`.
```dotenv theme={null}
EMBEDDING_PROVIDER="custom"
EMBEDDING_MODEL="openrouter/openai/text-embedding-3-small"
EMBEDDING_API_KEY="sk-or-..."
EMBEDDING_DIMENSIONS="1536"
```
**OpenRouter's embedding models are a separate catalogue.** The `/api/v1/models` endpoint you would use to find a chat model returns no embedding models at all, so checking it can leave the impression that OpenRouter has none. They live on their own endpoint:
```bash theme={null}
curl -s https://openrouter.ai/api/v1/embeddings/models | jq -r '.data[].id'
```
Prefix whichever slug you pick with `openrouter/`, and set `EMBEDDING_DIMENSIONS` to match that model — `custom` models are not in the auto-derive registry.
<Info>
**Automatic `encoding_format="float"` for OpenRouter**: Cognee detects OpenRouter routes — a model id beginning with `openrouter/`, an explicit `openrouter` provider, or an `openrouter.ai` endpoint host (all matched case-insensitively) — and sets `encoding_format="float"` on the embedding request. Older LiteLLM releases serialize an omitted `encoding_format` as JSON `null`, which OpenRouter rejects with a `400 invalid_value` error (it accepts only `"float"`/`"base64"`); forcing `"float"` avoids that. This guard is scoped to OpenRouter only, so it does not affect other `custom` providers. On current LiteLLM versions it is a no-op for `openrouter/`-prefixed models, but endpoint-based configs (an unprefixed model pointed at `openrouter.ai`) still rely on it — litellm's OpenAI handler injects the `null` even on current versions.
</Info>
</Tab>
<Tab title="Self-Hosted">
```dotenv theme={null}
EMBEDDING_PROVIDER="custom"
EMBEDDING_MODEL="openai/<your-internal-model-name>"
EMBEDDING_ENDPOINT="https://embeddings.internal.example.com/v1"
EMBEDDING_API_KEY="<internal-api-key>"
EMBEDDING_DIMENSIONS="<match-your-model>"
```
</Tab>
</Tabs>
<Info>
**No endpoint normalisation for `custom`**: Unlike [`openai_compatible`](#openai-compatible-local-servers-llama-cpp-tei-vllm), the `custom` provider passes `EMBEDDING_ENDPOINT` directly to LiteLLM as `api_base` with no automatic `/v1` appending or `/embeddings` stripping — and LiteLLM appends `/embeddings` itself, so `http://localhost:1234/v1/embeddings` is requested as `.../v1/embeddings/embeddings` and returns `404`. Set the endpoint to exactly the base URL your provider expects (e.g., `https://api.deepinfra.com/v1/openai`), or omit it entirely when using a named LiteLLM prefix such as `openrouter/`.
</Info>
</Accordion>
</AccordionGroup>
## Additional Information
<Accordion title="Valid EMBEDDING_PROVIDER values and endpoint URL forms">
`EMBEDDING_PROVIDER` and [`LLM_PROVIDER`](/setup-configuration/llm-providers#configuration) are configured independently and **do not accept the same values**. `LLM_PROVIDER` is checked against a fixed set of providers, while `EMBEDDING_PROVIDER` simply selects an embedding engine:
| `EMBEDDING_PROVIDER` | Engine | Also valid for `LLM_PROVIDER`? |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------- | ------------------------------ |
| `openai` (default) | LiteLLM — also the value used for [Azure embeddings](#azure-openai-embeddings) | Yes |
| `gemini`, `mistral`, `bedrock` | LiteLLM | Yes |
| `custom` | LiteLLM; `EMBEDDING_MODEL` needs a LiteLLM prefix (`lm_studio/`, `hosted_vllm/`, `openrouter/`, `openai/`, …) | Yes |
| `ollama` | Ollama's native `/api/embed` engine | Yes |
| `fastembed` | In-process ONNX, no endpoint or API key | No — embeddings only |
| `openai_compatible` | OpenAI SDK straight to `/v1/embeddings`, bypassing LiteLLM | No — embeddings only |
Only `fastembed`, `ollama`, and `openai_compatible` select a dedicated engine. **Any other value — including a typo — falls through to the LiteLLM engine without an error.**
* `EMBEDDING_PROVIDER="openai_compatible"` has no LLM counterpart. Setting `LLM_PROVIDER="openai_compatible"` fails with `ValueError: 'openai_compatible' is not a valid LLMProvider` — use `LLM_PROVIDER="custom"` instead ([LM Studio](/setup-configuration/llm-providers#lm-studio-local), [vLLM](/setup-configuration/llm-providers#vllm)).
* `anthropic`, `llama_cpp`, and `mcp-sampling` are LLM-only: they provide no embeddings, so pair them with one of the values above.
**Why a local `EMBEDDING_MODEL` can look like it is ignored**: the `EMBEDDING_PROVIDER` string is never sent to LiteLLM — it picks the engine (and, for `openrouter`, enables the automatic `encoding_format` guard described in [Custom Providers](#custom-providers)), nothing more. With `EMBEDDING_PROVIDER="custom"`, routing is decided entirely by the `EMBEDDING_MODEL` prefix, so an unprefixed model id is treated as an OpenAI model and the request goes to OpenAI (or fails on the missing key) instead of your server. Either add the matching prefix, or use `openai_compatible`, which sends the model id to your endpoint verbatim.
**Endpoint URL form** — the two OpenAI-compatible providers expect different things:
| Provider | `EMBEDDING_ENDPOINT` |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `openai_compatible` | Base URL ending in `/v1` — [normalised for you](#openai-compatible-local-servers-llama-cpp-tei-vllm), so `http://localhost:1234`, `.../v1`, and `.../v1/embeddings` all work. |
| `custom` | Base URL only — [passed verbatim to LiteLLM with no normalisation](#custom-providers), so a `/v1/embeddings` endpoint returns `404`. |
| `ollama` | Full native path: `http://localhost:11434/api/embed`. |
| `fastembed` | Not used. |
<Note>
With `LLM_PROVIDER="custom"`, `EMBEDDING_API_KEY` does **not** fall back to `LLM_API_KEY` for LiteLLM-routed embeddings — set it explicitly (use `"."` when your server needs no auth). The `openai_compatible` engine does fall back to `LLM_API_KEY`.
</Note>
For a full local `.env` covering both halves, see [LM Studio](/setup-configuration/llm-providers#lm-studio-local) or the [Local Setup guide](/guides/local-setup).
</Accordion>
<Accordion title="Which embedding models are supported?">
Cognee does not enforce a fixed allow-list of embedding models. Supported models depend on the provider configured in `EMBEDDING_PROVIDER`; Cognee forwards the embedding request to that provider and stores the returned vectors.
* **`fastembed`**: any model returned by [`TextEmbedding.list_supported_models()`](https://github.com/qdrant/fastembed), such as `sentence-transformers/all-MiniLM-L6-v2`. See the [Fastembed section](#fastembed-local) for the common list.
* **`ollama` / LM Studio**: any embedding model loaded locally, such as `bge-m3:latest` or `all-minilm:latest`.
* **`openai_compatible`**: any model exposed by a local `/v1/embeddings` server, including llama.cpp, TEI, vLLM, LocalAI, and Infinity.
* **`openai` / `gemini` / `mistral` / `bedrock` / `custom`**: any model available through the configured provider API, using the corresponding LiteLLM prefix such as `openai/` or `gemini/`.
Common model examples and dimensions are shown below. Set `EMBEDDING_DIMENSIONS` to match the model output size:
| Model | `EMBEDDING_DIMENSIONS` | How to run it |
| ------------------- | ---------------------- | ------------------------------------------------------------------------------------- |
| `all-MiniLM-L6-v2` | `384` | `fastembed` (`sentence-transformers/all-MiniLM-L6-v2`) or Ollama (`all-minilm`) |
| `all-mpnet-base-v2` | `768` | HuggingFace, `openai_compatible`, or vLLM (`sentence-transformers/all-mpnet-base-v2`) |
| `bge-m3` | `1024` | Ollama (`bge-m3`), HuggingFace, or `custom` (`BAAI/bge-m3`) |
**Dimensions must match the model's output size.** Cognee can auto-derive `EMBEDDING_DIMENSIONS` for models the `fastembed` or LiteLLM registries know; for anything else — local aliases, `custom` or `openai_compatible` endpoints — set it explicitly. See [How do I determine EMBEDDING\_DIMENSIONS?](#how-do-i-determine-embedding_dimensions) for how to find the value and what a mismatch does, and [Important Notes](#important-notes).
</Accordion>
<Accordion title="How do I determine EMBEDDING_DIMENSIONS?">
`EMBEDDING_DIMENSIONS` is the length of the vector your embedding model returns. It is a property of the model, not a free choice. Determine it in this order:
1. **Read the model card or provider docs** — for example `text-embedding-3-large` → `3072`, `nomic-embed-text-v1.5` → `768`, `BAAI/bge-m3` → `1024`. See the table in [Which embedding models are supported?](#which-embedding-models-are-supported) for more.
2. **Measure it** — embed one string and count the floats. This works for local aliases and self-hosted models that no registry knows:
```python theme={null}
import asyncio
from cognee.infrastructure.databases.vector.embeddings import get_embedding_engine
async def main():
vectors = await get_embedding_engine().embed_text(["hello"])
print(len(vectors[0])) # -> use this as EMBEDDING_DIMENSIONS
asyncio.run(main())
```
One caveat for LiteLLM-routed (`custom`) models: the engine sends the currently configured dimensions value as a request parameter, so an endpoint that rejects that parameter raises [UnsupportedParamsError](#unsupportedparamserror-passing-the-dimensions-parameter-to-litellm) instead of returning a vector. `openai_compatible` and `ollama` never send it and always measure cleanly.
**If you leave it unset**, Cognee derives it from the `fastembed` model registry (for `EMBEDDING_PROVIDER="fastembed"`), otherwise from LiteLLM's model metadata (`output_vector_size`), trying the model id, the id without its provider prefix, and `provider/model`. When neither knows the model it logs a warning and falls back to `3072`:
```
Could not auto-derive embedding_dimensions for provider='custom' model='...'. Falling back to 3072.
If your embedder produces vectors of a different size, set EMBEDDING_DIMENSIONS explicitly —
otherwise the first write into the vector store will fail with a shape mismatch.
```
Local aliases and `custom` / `openai_compatible` models are usually unknown, so set the value explicitly for them.
**What a mismatch does**: Cognee creates every vector collection using the configured size (`Vector(N)` in LanceDB, `vector(N)` in PGVector), so the first write of a differently sized vector fails with a shape/dimension error — the vectors are not silently truncated or padded. Because the size is baked into the collection schema, correcting the value also requires recreating the collections with `await cognee.prune.prune_system()` and re-ingesting. See [Dimension Consistency](/setup-configuration/vector-stores#dimension-consistency).
<Note>
OpenAI's `text-embedding-3-*` models are the exception where a smaller value is legitimate: Cognee forwards `EMBEDDING_DIMENSIONS` as the `dimensions` request parameter, and OpenAI returns shortened vectors. Most other models reject that parameter — see [UnsupportedParamsError](#unsupportedparamserror-passing-the-dimensions-parameter-to-litellm).
</Note>
</Accordion>
<Accordion title="Batch Size">
`EMBEDDING_BATCH_SIZE` controls how many text chunks are grouped into a single embedding API call. Cognee splits all chunks into batches of this size and sends them concurrently to the embedding engine.
| Variable | Default | Description |
| ---------------------- | ------- | ----------------------------- |
| `EMBEDDING_BATCH_SIZE` | `36` | Chunks per embedding API call |
**Local inference (Ollama, llama.cpp, LM Studio)**: Local servers handle one request at a time with limited concurrency. The default `36` can overwhelm them. Reduce the batch size if you see errors or slowdowns:
```dotenv theme={null}
EMBEDDING_BATCH_SIZE="5"
```
**Cloud providers**: Larger batches reduce the number of API calls and are efficient with cloud APIs. The default `36` suits most cloud providers. Where a provider caps the number of texts per request — Vertex AI is the case Cognee recognizes, and it enforces two such caps, a per-prediction instance cap and a smaller per-model batch cap, each rejected with its own wording (both listed under [Max Completion Tokens](#max-completion-tokens)) — exceeding either is no longer fatal: the LiteLLM engine halves the batch and retries (see [Over-length input recovery](#timeout-and-retry-behavior)). Setting `EMBEDDING_BATCH_SIZE` under the lower of the two is still the cheaper path, since each recovery adds round trips.
**Relationship to rate limiting**: Each batch counts as one request toward `EMBEDDING_RATE_LIMIT_REQUESTS`. A single file may produce many chunks — with `EMBEDDING_BATCH_SIZE=36`, a document split into 360 chunks generates 10 requests.
</Accordion>
<Accordion title="Max Completion Tokens">
Despite the name, embeddings have no "completion" — `EMBEDDING_MAX_COMPLETION_TOKENS` is the per-text **input** token budget Cognee's tokenizer uses to size chunks for the embedding model. It should reflect your embedding model's maximum input length.
| Variable | Default | Description |
| --------------------------------- | ------- | -------------------------------------------------------------------------- |
| `EMBEDDING_MAX_COMPLETION_TOKENS` | `8191` | Maximum input tokens per embedded text; an input to automatic chunk sizing |
**Observable impact:**
* **Chunk size, cost and latency.** Chunks are sized as `min(EMBEDDING_MAX_COMPLETION_TOKENS, LLM_MAX_COMPLETION_TOKENS // 2)`. A lower value forces smaller chunks, so a document produces more chunks → more embedding requests (and more LLM extraction calls) → higher latency and cost. A higher value (within the model's real limit) produces fewer, larger chunks. See [Chunkers](/core-concepts/further-concepts/chunkers) for how chunk size shapes the graph.
* **Over-length requests.** Setting this above the embedding model's real input limit can make individual texts exceed the limit. Cognee no longer fails outright — it splits the batch (or mean-pools an over-length string, see [Timeout and Retry Behavior](#timeout-and-retry-behavior)) — but that recovery adds latency, so it is not free. "Higher" is only better up to the model's actual limit.
**Tuning guidance:** match this to your embedding model's input limit. The default `8191` fits OpenAI's `text-embedding-3-*` models. For models with a smaller limit, lower it so chunks fit without triggering the split-and-pool fallback; the related `LLM_MAX_COMPLETION_TOKENS` ([LLM Providers](/setup-configuration/llm-providers#max-completion-tokens)) caps the other half of the formula. With hosted third-party `/v1/embeddings` endpoints, use the limit the endpoint *enforces* per text — often smaller than the model's advertised context — and leave headroom: token counts for these endpoints are usually approximate, because the model id is rarely a loadable HuggingFace repo and Cognee falls back to the TikToken tokenizer (see [How Cognee selects the tokenizer](#how-cognee-selects-the-tokenizer)).
### It is a chunk-sizing hint, not an enforced cap
Cognee never sends `EMBEDDING_MAX_COMPLETION_TOKENS` to the provider and never truncates text at request time — it only hands the value to the tokenizer that sizes chunks during `cognify()`. Two consequences:
* **Setting it too high silently produces over-length requests.** The provider rejects them; whether Cognee recovers depends on the error message (see below).
* **Setting it above `LLM_MAX_COMPLETION_TOKENS // 2` has no effect.** With the default `LLM_MAX_COMPLETION_TOKENS=16384`, chunks are capped at `8192` regardless — `EMBEDDING_MAX_COMPLETION_TOKENS=131072` and `=8192` behave identically until you also raise the LLM value.
### Chunk size vs. embedding request size
One embedding request carries `EMBEDDING_BATCH_SIZE` chunks (default `36`, see [Batch Size](#batch-size)), so the payload is roughly `chunk tokens × batch size`. Providers that cap the **whole request** rather than each text will reject a request built from chunks that each fit comfortably. If you hit a length error even though your chunks are within the model's per-text limit, lower `EMBEDDING_BATCH_SIZE` rather than `EMBEDDING_MAX_COMPLETION_TOKENS`.
Two such whole-request caps are now recovered automatically, both from Vertex AI, which enforces them separately: a per-prediction instance cap on how many instances one prediction may carry, and a much smaller per-model batch cap — `250` for `gemini-embedding-001`, against a 2048-instance prediction limit, so a batch can clear the first cap and still be refused by the second. Each is worded differently and Cognee matches a fragment of each (see below), so the LiteLLM engine halves the batch and retries instead of failing the run — see [Batch Size](#batch-size) for why staying under the lower cap is still the cheaper configuration.
### Recovery only fires on recognized error messages
The split-and-pool recovery is triggered by string matching on the provider's error, so a provider that phrases the limit differently gets no recovery — the run fails with `EmbeddingException` on `openai_compatible` and `ollama`, or with the provider's original error (re-raised after the retry window) on the LiteLLM engines:
| Engine | Recovers when the error… |
| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| LiteLLM (`openai`, `gemini`, `mistral`, `bedrock`, `custom`) | is a `ContextWindowExceededError`, or the message matches `maximum input length`, `instance(s) is allowed per prediction`, `too many instances`, or `batchSize value of` |
| `openai_compatible` | contains `context length`, `context window`, `too long`, `maximum context`, `maximum tokens`, or `max tokens` |
| `ollama` | contains any phrasing `openai_compatible` recognizes, plus `input length` |
The three LiteLLM phrasings after `maximum input length` are batch-size rejections, which Vertex AI words differently depending on which cap is hit: `instance(s) is allowed per prediction` for the per-prediction instance cap (for example `2048 instance(s) is allowed per prediction`), `too many instances` and `batchSize value of` for the per-model batch cap. These used to fall through the guard and be re-raised even though the engine already knew how to recover from them. Matching is case-insensitive and tolerant of the whitespace between words, but otherwise literal: only these phrasings recover.
`ollama` matches in two stages, and only the second is case-insensitive. The engine first tests the raw `error` field of the response body for `context length` or `input length` **case-sensitively**: a match short-circuits the retry ladder so the recovery runs on the first attempt, while anything else is retried for the full 128-second window before the recovery is attempted at all. Because the shape of that field varies by server — Ollama sends a string, an OpenAI-compatible proxy sends an object, and either can send `null` — it is coerced with `str()` before the test. Without that coercion an object-shaped error tested its own dict keys (never a match, so a terminal over-length error burned the whole window first) and a `null` raised a `TypeError` that no pattern could match, failing the run outright.
For example, `<400> InternalError.Algo.InvalidParameter: Range of input length should be [1, 33000]` matches none of these phrases, so Cognee raises instead of splitting. Configure the limits correctly rather than relying on the fallback.
</Accordion>
<Accordion title="Timeout and Retry Behavior">
The `LiteLLMEmbeddingEngine` applies two layers of protection against slow or unreachable endpoints:
| Limit | Value | Configurable |
| ------------------- | ----------- | -------------- |
| Per-attempt timeout | 300 seconds | No (hardcoded) |
| Total retry window | 128 seconds | No (hardcoded) |
**How retries work**: Failed attempts are retried with exponential back-off starting at 2 seconds, with random jitter, until the 128-second window is exhausted.
**Why the per-attempt timeout is larger than the retry window**: The 300-second deadline is measured per attempt and starts *before* any network I/O, so waiting for a free connection in the local HTTP pool and event-loop scheduling delays both count against it — under high cognify concurrency a perfectly healthy request can spend most of its budget queued client-side. A shorter deadline cancelled those queued requests and turned them into retries, which added more load. Because the retry window is evaluated *between* attempts, the two limits interact:
* A request that fails fast (connection refused, rate limit, `5xx`) is retried repeatedly until the 128-second window is exhausted, exactly as before.
* A request that genuinely hangs consumes the full 300 seconds on its first attempt. By the time it is cancelled the 128-second window has already elapsed, so it is **not** retried — the `EmbeddingException` is raised directly. A single hung request therefore blocks its task for up to 5 minutes.
The per-attempt deadline matches the one used by `OpenAICompatibleEmbeddingEngine`, so both engines now behave the same way.
### Errors that fail immediately
Most failures are retried until the 128-second window runs out. These are raised on the **first attempt** instead, because retrying cannot change the outcome — each one needs a configuration or budget change from you:
<table>
<colgroup>
<col />
<col />
<col />
<col />
</colgroup>
<thead>
<tr>
<th>Failure</th>
<th>Raised as</th>
<th>HTTP</th>
<th>What to change</th>
</tr>
</thead>
<tbody>
<tr>
<td>A spend cap is reached — typically a per-key or per-user cap on a LiteLLM proxy</td>
<td>`LLMPaymentRequiredError`</td>
<td>`402`</td>
<td>Raise the cap or top up the budget, then re-run. **Catch this instead of `EmbeddingException`**: it is outside that family, so existing handlers and `422` alerts no longer see it. The body carries the provider's own budget sentence with identifiers masked — see [402 Payment Required](/api-reference/introduction).</td>
</tr>
<tr>
<td>The endpoint rejects the credentials (`401`) or denies the model (`403`)</td>
<td>`EmbeddingCredentialsError` on `openai_compatible`; litellm's `AuthenticationError` / `PermissionDeniedError` on the LiteLLM engines</td>
<td>`422` / `500`</td>
<td>Fix `EMBEDDING_API_KEY`, or the key's entitlement for that model. `EmbeddingCredentialsError` subclasses `EmbeddingException` and keeps the API's `422`, so existing handlers still catch it; catch it specifically to tell a credential problem apart from a transient endpoint failure. The litellm classes are **not** `EmbeddingException` subclasses and no handler maps them, so `except EmbeddingException` does not catch them and through the API they escape the routers' generic handler as a `500`.</td>
</tr>
<tr>
<td>A single text exceeds the context window but is too short to split further</td>
<td>`EmbeddingContextWindowTooSmallError`</td>
<td>`422`</td>
<td>Check `EMBEDDING_ENDPOINT` and the model name. This fires only when a string under three characters is still rejected as over-length, which means the endpoint is rejecting even tiny inputs — no chunk-sizing setting can help at that point. Also subclasses `EmbeddingException`; default message `Text is too short to split further but exceeds context window.`</td>
</tr>
<tr>
<td>The task is cancelled</td>
<td>`asyncio.CancelledError`</td>
<td>—</td>
<td>Nothing. Cancelled tasks unwind promptly rather than holding the retry window open.</td>
</tr>
</tbody>
</table>
**Engine differences worth knowing.** Coverage is not identical across the four engines:
* **A `404` — wrong model name (including a missing `hosted_vllm/` prefix for vLLM), wrong base path — is not in the table: it consumes the full retry window on every engine.** On the LiteLLM engines the `404` is wrapped into a generic `EmbeddingException` before the retry decision is made, so it is retried like a transient failure and surfaces as `EmbeddingException: Failed to index data points using model <model>` (`422`) once the window runs out. `OpenAICompatibleEmbeddingEngine` keeps `404` retryable on purpose: an ingress or reverse proxy in front of a self-hosted server can `404` transiently during a rolling deploy, which is realistic for exactly this engine. Either way, on a persistent `404` check the model name and `EMBEDDING_ENDPOINT` rather than re-running.
* **`OllamaEmbeddingEngine` is not at parity on spend caps.** It rebuilds the failure as a bare `RuntimeError`, dropping the status code and the response object, so only the message-text signal survives. A rejection whose body carries `budget_exceeded` but no recognizable budget sentence — or a reworded or truncated one — is not classified there and still consumes the full window. It is covered at all only because `EMBEDDING_ENDPOINT` is never validated for provider shape, so this engine can be pointed at a proxy and reach its cap.
* **`FastembedEmbeddingEngine` embeds in process** and issues no requests, so spend caps and credential failures cannot reach it.
Spend caps are detected by the same four signals the LLM path uses — see [Retry Behavior](/setup-configuration/llm-providers#retry-behavior) — checked against the raised exception **and every error in its `__cause__` chain**, so a rejection wrapped in an `EmbeddingException` is still recognized.
**Over-length input recovery**: When the provider rejects an embedding request because the input exceeds the model's context window, `LiteLLMEmbeddingEngine` no longer fails the request. This covers both LiteLLM's `ContextWindowExceededError` and a plain `400 BadRequestError` whose message matches `maximum input length` (for example OpenAI's `maximum input length is 8192 tokens`, which is returned as a plain 400 by the embeddings API). On either of these:
* If the batch contains more than one text, it is split in half and each half is embedded in parallel, then the results are concatenated.
* If a single over-length string is left, it is split into two overlapping segments (the first two-thirds and the last two-thirds of the string), each segment is embedded, and the two vectors are averaged (mean-pooled) into one.
The same recovery now also runs for **Vertex AI batch-size rejections** — a `400 BadRequestError` whose message matches `instance(s) is allowed per prediction` (such as `2048 instance(s) is allowed per prediction`), `too many instances`, or `batchSize value of`. These are a limit on how many texts one request may carry, not on how long any single text is, so it is the first branch above — halving the batch and retrying each half — that resolves them; the recursion keeps halving until each request is under the provider's per-request cap. Previously this phrasing did not match the guard and the error was re-raised, failing the run.
This split-and-pool step recurses until each piece fits, so long documents that previously failed are now embedded automatically. If a single string is already too short to split further (fewer than three characters) yet still exceeds the context window, the engine raises the terminal `EmbeddingContextWindowTooSmallError` instead of retrying — this deterministic failure returns at once rather than consuming the full retry window (see [Errors that fail immediately](#errors-that-fail-immediately)). Any other `400 BadRequestError` (one whose message does **not** indicate an over-length input) is re-raised unchanged so genuinely malformed requests still fail fast — with one exception: a budget rejection that the proxy mapped to a `400` is converted to `LLMPaymentRequiredError` (`402`) before the over-length check runs, so it fails fast rather than reaching the caller as a raw provider error. The exact error phrasings that trigger this recovery — for this engine and for `openai_compatible` — are listed under [Max Completion Tokens](#max-completion-tokens).
**Common error messages and causes**:
* `EmbeddingException: Embedding request timed out. Check EMBEDDING_ENDPOINT connectivity.` — The attempt did not complete within 300 seconds. Verify that `EMBEDDING_ENDPOINT` is reachable from your network. Since the deadline also covers time spent queued client-side, check that your embedding concurrency is bounded before assuming the endpoint itself is at fault.
* `EmbeddingException: Cannot connect to embedding endpoint. Check EMBEDDING_ENDPOINT.` — TCP connection was refused or the server closed the connection before responding. Confirm the server is running and the URL is correct.
* `EmbeddingException: Failed to index data points using model <model>` — The provider returned a `404 Not Found`. Common causes: wrong model name, missing `hosted_vllm/` prefix for vLLM, or an unsupported model at that endpoint. Non-over-length `400 BadRequestError` responses are re-raised unchanged, except a budget rejection mapped to a `400`, which becomes `LLMPaymentRequiredError` (`402`).
* `400 invalid_value` on `encoding_format` from OpenRouter — Older LiteLLM releases serialize an omitted `encoding_format` as JSON `null`, which OpenRouter rejects. Cognee now forces `encoding_format="float"` on detected OpenRouter routes (see the Custom Providers accordion under [Provider Setup Guides](#provider-setup-guides)), so this should no longer occur; if you still see it, upgrade Cognee to a version that includes this guard.
**Diagnosing slow local servers**: If you see timeouts with Ollama, LM Studio, or vLLM, a large batch can still exceed the 300-second per-attempt limit — and because a stalled request now holds its task for up to 5 minutes before failing, the cost of each one is higher than a fast timeout would be. Reduce `EMBEDDING_BATCH_SIZE` to send fewer texts per request:
```dotenv theme={null}
EMBEDDING_BATCH_SIZE="5"
```
<Note>
The timeout and retry values are hardcoded in `LiteLLMEmbeddingEngine` and cannot be changed via environment variables. To use different limits, subclass `LiteLLMEmbeddingEngine` and override `embed_text` with a custom `@retry` decorator.
</Note>
</Accordion>
<Accordion title="UnsupportedParamsError: passing the dimensions parameter to LiteLLM">
For the LiteLLM-routed providers (`openai`, `gemini`, `mistral`, `bedrock`, and `custom`), Cognee sends the OpenAI-style `dimensions` parameter to `litellm.aembedding()` whenever an embedding dimension is configured. In the `.env` flow documented on this page, `EMBEDDING_DIMENSIONS` is required, so this parameter is normally present. Only some models accept `dimensions` (notably OpenAI's `text-embedding-3-*`). Other models — and some LiteLLM proxies — reject it with:
```
litellm.exceptions.UnsupportedParamsError: ... does not support parameters: ['dimensions']
```
Cognee does **not** expose an environment variable to suppress `dimensions` or to forward extra LiteLLM params. The intended fix is LiteLLM's own `drop_params` switch, which makes LiteLLM silently drop any parameter the target model does not support (including `dimensions`) instead of raising — so you can keep routing embeddings through your LiteLLM proxy:
<Tabs>
<Tab title="Python (LiteLLM SDK)">
Set the flag before you call any Cognee operation:
```python theme={null}
import litellm
litellm.drop_params = True
import cognee # embeddings now drop unsupported params instead of raising
```
</Tab>
<Tab title="LiteLLM proxy (config.yaml)">
If you run your own LiteLLM proxy, enable `drop_params` in the proxy config so unsupported params are stripped before forwarding upstream. You can set it proxy-wide:
```yaml theme={null}
litellm_settings:
drop_params: true
```
Or per model:
```yaml theme={null}
model_list:
- model_name: my-embedder
litellm_params:
model: openai/text-embedding-3-large
drop_params: true
```
</Tab>
</Tabs>
<Warning>
`drop_params` silences the error but does **not** change the vector size your model returns. Still set `EMBEDDING_DIMENSIONS` to the model's real output size so it matches your vector store — see [Which embedding models are supported?](#which-embedding-models-are-supported) and [Important Notes](#important-notes).
</Warning>
<Info>
If you don't need the LiteLLM proxy, the [`openai_compatible`](#openai-compatible-local-servers-llama-cpp-tei-vllm) provider talks to any `/v1/embeddings` server directly through the OpenAI SDK and never sends the `dimensions` parameter, so it avoids this error entirely — at the cost of bypassing LiteLLM.
</Info>
</Accordion>
<Accordion title="Rate Limiting">
Control client-side throttling for embedding calls to manage API usage and costs.
<Warning>
**Rate limiting is disabled by default.** You must explicitly set `EMBEDDING_RATE_LIMIT_ENABLED="true"` to activate it.
</Warning>
**Defaults (when rate limiting is enabled):**
| Variable | Default | Meaning |
| ------------------------------- | ------- | ------------------------- |
| `EMBEDDING_RATE_LIMIT_ENABLED` | `false` | Off by default — opt-in |
| `EMBEDDING_RATE_LIMIT_REQUESTS` | `60` | Max requests per interval |
| `EMBEDDING_RATE_LIMIT_INTERVAL` | `60` | Interval in seconds |
**What counts as one request?**
One rate-limit request = one `embed_text()` API call = one batch of chunks (not one chunk). With the default `EMBEDDING_BATCH_SIZE=36`, processing 360 chunks produces 10 requests. See the [Batch Size](#batch-size) section for how to tune batch size.
**Sizing guidance:**
Set `EMBEDDING_RATE_LIMIT_REQUESTS` to your provider's RPM limit and `EMBEDDING_RATE_LIMIT_INTERVAL` to `60`. Use \~80–90% of your provider's advertised limit to leave headroom.
**Example configurations for common provider tiers**
These examples target embedding endpoints, such as OpenAI embedding models like `text-embedding-3-large`.
<AccordionGroup>
<Accordion title="OpenAI - Tier 1">
```dotenv theme={null}
EMBEDDING_RATE_LIMIT_ENABLED="true"
EMBEDDING_RATE_LIMIT_REQUESTS="2700"
EMBEDDING_RATE_LIMIT_INTERVAL="60"
```
</Accordion>
<Accordion title="OpenAI - Free / Very Low Tier">
```dotenv theme={null}
EMBEDDING_RATE_LIMIT_ENABLED="true"
EMBEDDING_RATE_LIMIT_REQUESTS="180"
EMBEDDING_RATE_LIMIT_INTERVAL="60"
```
</Accordion>
<Accordion title="Google Gemini - Free Tier">
```dotenv theme={null}
EMBEDDING_RATE_LIMIT_ENABLED="true"
EMBEDDING_RATE_LIMIT_REQUESTS="1350"
EMBEDDING_RATE_LIMIT_INTERVAL="60"
```
</Accordion>
<Accordion title="Conservative Default">
```dotenv theme={null}
EMBEDDING_RATE_LIMIT_ENABLED="true"
EMBEDDING_RATE_LIMIT_REQUESTS="60"
EMBEDDING_RATE_LIMIT_INTERVAL="60"
```
</Accordion>
</AccordionGroup>
<Info>
Always verify your exact tier limits in your provider's dashboard — limits vary by model, tier, and region. The examples above are approximations for common tiers and may change.
</Info>
</Accordion>
<Accordion title="Testing and Development">
```dotenv theme={null}
# Mock embeddings for testing (returns zero vectors)
MOCK_EMBEDDING="true"
```
</Accordion>
<Accordion title="HUGGINGFACE_TOKENIZER environment variable">
The `HUGGINGFACE_TOKENIZER` environment variable specifies which Hugging Face tokenizer to use for counting tokens before sending text to the embedding model. It is **optional** — Cognee no longer requires it at startup, so omitting it does not raise a `ValidationError` on import — but it is recommended when using the **Ollama** provider for accurate token counting.
**Value format**: The value is the Hugging Face model repository ID — the `{organization}/{model-name}` path that appears in the URL on [huggingface.co/models](https://huggingface.co/models). This should match the underlying model used by your Ollama embedding.
For example, if the Ollama model `nomic-embed-text:latest` is built from `nomic-ai/nomic-embed-text-v1.5` on Hugging Face, set:
```dotenv theme={null}
HUGGINGFACE_TOKENIZER="nomic-ai/nomic-embed-text-v1.5"
```
### Common model-to-tokenizer mappings
| Ollama model | `HUGGINGFACE_TOKENIZER` value | Dimensions |
| ---------------------------------- | ---------------------------------------- | ---------- |
| `nomic-embed-text:latest` | `nomic-ai/nomic-embed-text-v1.5` | 768 |
| `bge-m3:latest` | `BAAI/bge-m3` | 1024 |
| `mxbai-embed-large:latest` | `mixedbread-ai/mxbai-embed-large-v1` | 1024 |
| `avr/sfr-embedding-mistral:latest` | `Salesforce/SFR-Embedding-Mistral` | 4096 |
| `all-minilm:latest` | `sentence-transformers/all-MiniLM-L6-v2` | 384 |
### Finding the tokenizer for any model
1. Look up the model on [huggingface.co/models](https://huggingface.co/models).
2. The repository ID is the `{organization}/{model-name}` part of the URL (e.g., `huggingface.co/BAAI/bge-m3` → `BAAI/bge-m3`).
3. Use the repository ID that corresponds to the model your Ollama tag is built from. The Ollama model page typically links to the original Hugging Face repository.
<Note>
`HUGGINGFACE_TOKENIZER` is only used by the Ollama embedding engine. It is not needed for OpenAI, Fastembed, `openai_compatible`, or other providers. For `openai_compatible`, the engine automatically tries the model name as a HuggingFace tokenizer ID and falls back to TikToken if unavailable — no separate `HUGGINGFACE_TOKENIZER` value is required.
</Note>
### Troubleshooting dependency errors
The HuggingFace tokenizer (`HuggingFaceTokenizer`, which wraps `transformers.AutoTokenizer`) backs token counting for the **Ollama** embedding engine. Cognee also attempts to use it for `custom` and `openai_compatible` chunk sizing; if it cannot load, Cognee falls back to TikToken. `transformers` is **not** part of the base `cognee` install — it is an optional dependency shipped by the `huggingface` and `ollama` extras (both pin `transformers>=4.46.3,<5`).
**`ModuleNotFoundError: No module named 'transformers'`**
The tokenizer was triggered (most often by `EMBEDDING_PROVIDER="ollama"`) but `transformers` is missing. This typically surfaces as `Connection to Embedding handler could not be established`. Install the extra:
```bash theme={null}
pip install 'cognee[huggingface]'
# or, if you are using Ollama embeddings:
pip install 'cognee[ollama]'
```
**`cannot import name 'is_offline_mode' from 'huggingface_hub'`**
This is a version mismatch: an old `transformers` (older than 4.46) is paired with a newer `huggingface_hub` that no longer exports `is_offline_mode`. Cognee's `transformers>=4.46.3,<5` pin is compatible with current `huggingface_hub` (Cognee resolves `huggingface_hub` 0.36.x). If you previously installed `transformers` manually, upgrade it into Cognee's supported range:
```bash theme={null}
pip install --upgrade "transformers>=4.46.3,<5"
```
Installing or reinstalling the `huggingface` / `ollama` extra pulls in a compatible `huggingface_hub` automatically, so prefer the extra over pinning `huggingface_hub` by hand.
## Important Notes
* **Dimension Consistency**: `EMBEDDING_DIMENSIONS` must match your vector store collection schema
* **API Key Fallback**: If `EMBEDDING_API_KEY` is not set, Cognee uses `LLM_API_KEY` (except for custom providers)
* **Tokenization**: `HUGGINGFACE_TOKENIZER` is optional and no longer enforced by Cognee's startup validation — but it is recommended for the Ollama provider; set it to the HuggingFace model repo ID that matches your embedding model for accurate token counting
* **Performance**: Local providers (Ollama, Fastembed) are slower but offer privacy and cost benefits
</Accordion>
<Accordion title="How Cognee selects the tokenizer">
Token counts drive chunk sizing and the `--dry-run` token estimate, so Cognee auto-selects a tokenizer that matches your embedding model:
* **`openai`** — the model's TikToken (BPE) encoding.
* **`gemini`** — the default TikToken encoding (Gemini has no local tokenizer, so counts are approximate).
* **`mistral`** — the Mistral tokenizer.
* **`fastembed`** — the model's own HuggingFace (wordpiece) tokenizer, resolved from the fastembed model list.
* **`ollama`** — the tokenizer named by `HUGGINGFACE_TOKENIZER`.
* **`openai_compatible` / `custom`** — the embedding model id used as a HuggingFace repo (e.g. `BAAI/bge-large-en-v1.5`); when that is not a loadable repo — such as a local alias or a LiteLLM-prefixed id — Cognee falls back to TikToken.
If a matching tokenizer cannot be loaded, Cognee logs an advisory warning and falls back to the TikToken tokenizer. This resolution **never raises**: a mismatched or missing tokenizer only degrades token-count accuracy (mis-sized chunks and a skewed `--dry-run` estimate), it does not stop ingestion.
### The "could not load a matching tokenizer" warning is benign
A repeated log line such as:
```
Could not load a matching tokenizer for embedding model 'openai/qwen3-vl-embedding-2b'
(... is not a valid model identifier listed on 'https://huggingface.co/models' ...).
Falling back to TikToken, so token counts are approximate.
```
means the model id is not loadable as a HuggingFace repo. That is expected for LiteLLM-prefixed ids (`openai/...`, `lm_studio/...`, `openrouter/...`), Ollama tags, and local aliases such as `default`. Embedding and ingestion continue normally — only chunk sizing and the `--dry-run` token estimate become approximate.
To silence it:
* **`ollama`** — set `HUGGINGFACE_TOKENIZER` to the HuggingFace repo your Ollama model is built from (see the mappings earlier in this section).
* **`openai_compatible`** — set `EMBEDDING_MODEL` to the served model's real HuggingFace repo id (for example `Qwen/Qwen3-Embedding-4B` instead of `default`). This provider needs no LiteLLM prefix, so the id can be the repo id directly.
* **`custom` with `hosted_vllm/`** — the prefix is stripped automatically before tokenizer resolution, so the warning does not appear as long as the model name after the prefix is a valid HuggingFace repo id (see the [vLLM](#vllm) accordion).
* **`custom` with any other prefix** (`openai/`, `lm_studio/`, `openrouter/`, …) — the prefix is required for routing and is passed to tokenizer resolution whole, so the warning cannot be avoided; treat it as informational. If your endpoint is OpenAI-compatible, switching to `openai_compatible` with the plain repo id removes it.
<Warning>
The warning text suggests setting `HUGGINGFACE_TOKENIZER`, but that variable is only consulted by the **Ollama** engine — `custom`, `openai_compatible`, `fastembed`, and the LiteLLM providers do not pass it to tokenizer resolution, so setting it there has no effect.
</Warning>
</Accordion>
<Columns>
<Card title="LLM Providers" icon="brain" href="/setup-configuration/llm-providers">
Configure LLM providers for text generation
</Card>
<Card title="Vector Stores" icon="database" href="/setup-configuration/vector-stores">
Set up vector databases for embedding storage
</Card>
<Card title="Overview" icon="settings" href="/setup-configuration/overview">
Return to setup configuration overview
</Card>
</Columns>
# Graph Stores
Source: https://docs.cognee.ai/setup-configuration/graph-stores
Configure graph databases for knowledge graph storage and relationship reasoning in Cognee
Graph stores capture entities and relationships in knowledge graphs. They enable Cognee to understand structure and navigate connections between concepts, providing powerful reasoning capabilities.
<Info>
**New to configuration?**
See the [Setup Configuration Overview](./overview) for the complete workflow:
install extras → create `.env` → choose providers → handle pruning.
For a complete, copy-paste `.env` block that combines this layer with a relational and a vector store, see [Store Configurations](/guides/store-configurations).
</Info>
## Supported Providers
Cognee supports multiple graph store options:
* **Kuzu** — Local file-based graph database (default)
* **Kuzu-remote** — Kuzu with HTTP API access
* **Neo4j** — Production-ready graph database (self-hosted server or Docker)
* **Neo4j Desktop** — Local Neo4j development setup with the Desktop app
* **Neo4j Aura** — Neo4j's fully managed cloud service
* **Postgres** — Stores graph nodes and edges in Postgres tables (demo, not production-ready). Select it with `postgres_demo`; the older value `postgres` is still accepted and resolves to the same adapter
* **Neptune** — Amazon Neptune cloud graph database
* **Neptune Analytics** — Amazon Neptune Analytics hybrid solution
* **Memgraph** — In-memory graph database (community adapter)
<Info>
**Local vs. cloud storage**: By default Cognee stores its graph in a local Kuzu file. To inspect a graph locally in Neo4j, use Neo4j Desktop. To persist data in the cloud, switch to a remote provider such as Neo4j Aura, Neptune, or a self-hosted Neo4j instance on a remote server.
</Info>
## Configuration
<Accordion title="Environment Variables">
Set these environment variables in your `.env` file:
* `GRAPH_DATABASE_PROVIDER` — The graph store provider (kuzu, kuzu-remote, neo4j, postgres\_demo, neptune, neptune\_analytics; `postgres` is accepted as an alias of `postgres_demo`)
* `GRAPH_DATABASE_URL` — Database URL or connection string
* `GRAPH_DATABASE_USERNAME` — Database username (optional)
* `GRAPH_DATABASE_PASSWORD` — Database password (optional)
* `GRAPH_DATABASE_NAME` — Database name (optional)
* `KUZU_BUFFER_POOL_SIZE` — For the file-based Ladybug/Kuzu store only: the buffer-pool ceiling in bytes (default `34359738368`, 32 GB). Cognee does not derive this from the machine's or container's available memory, so lower it when the engine runs somewhere capped below that — see the Memory sizing section of the [Docker Compose reference](/how-to-guides/cognee-sdk/deployment/docker-compose-reference) for Docker
* `KUZU_MAX_DB_SIZE` — For the file-based Ladybug/Kuzu store only: the maximum **on-disk** database size in bytes, as a power of two (default `34359738368`, 32 GB). This is a disk ceiling, not a memory reservation; raise it for graphs that outgrow it
* `KUZU_NUM_THREADS` — For the file-based Ladybug/Kuzu store only: how many threads the engine uses to execute queries (default `0`, which keeps its internal default of one per CPU)
* `SUBPROCESS_OPEN_LOCK_RETRIES` — For the file-based Ladybug/Kuzu store only: how many times a worker retries opening the graph file when another worker is still releasing its on-disk lock (default `10`; set to `0` or a negative value to disable retries and surface the lock error immediately)
* `SUBPROCESS_OPEN_LOCK_BACKOFF` — For the file-based Ladybug/Kuzu store only: starting delay in seconds for the exponential backoff between those open retries (default `0.1`; per-attempt delay is capped internally)
* `SUBPROCESS_IDLE_TTL_SECONDS` — For subprocess-backed engines only: how many seconds an idle engine is kept alive before a background reaper closes its worker process (default `600`). While the engine is kept warm it still holds the graph file lock and its memory, and a request arriving inside the window reuses it instead of respawning a worker. Set to `0` to close the engine at every dataset-context exit instead (negative values are clamped to `0`). See [Subprocess engine teardown coordination](/setup-configuration/permissions#subprocess-engine-teardown-coordination) for the full lifecycle
</Accordion>
## Setup Guides
<AccordionGroup>
<Accordion title="Kuzu (Default)">
Kuzu is file-based and requires no network setup. It's perfect for local development and single-user scenarios.
```dotenv theme={null}
GRAPH_DATABASE_PROVIDER="kuzu"
# Optional: override location
# SYSTEM_ROOT_DIRECTORY=/absolute/path/.cognee_system
# The graph file will default to <SYSTEM_ROOT_DIRECTORY>/databases/cognee_graph_kuzu
```
**Installation**: Kuzu is included by default with Cognee. No additional installation required.
**Data Location**: The graph is stored on disk. Path defaults under the Cognee system directory and is created automatically.
<Warning>
**Concurrency Limitation**: Kuzu uses file-based locking and is not suitable for concurrent use from different agents or processes. For multi-agent scenarios, use Neo4j instead.
Cognee runs the embedded Ladybug/Kuzu engine in a dedicated worker process and now retries opening the graph file when it hits transient on-disk lock contention (the `Could not set lock on file` error that could briefly occur while a previously used engine was still shutting down). If you still see this error under heavy churn, tune the open retries and backoff with `SUBPROCESS_OPEN_LOCK_RETRIES` and `SUBPROCESS_OPEN_LOCK_BACKOFF` (see the Environment Variables section above).
An idle graph worker is also kept alive — and so keeps holding the graph file lock — for up to `SUBPROCESS_IDLE_TTL_SECONDS` (default `600`) after its last use, which is what lets a follow-up request reuse it instead of respawning. Set `SUBPROCESS_IDLE_TTL_SECONDS=0` if you need the lock released at each dataset-context exit, for example when another process or tool has to open the same graph file.
</Warning>
</Accordion>
<Accordion title="Kuzu (Remote API)">
Use Kuzu with an HTTP API when you need remote access or want to run Kuzu as a service.
```dotenv theme={null}
GRAPH_DATABASE_PROVIDER="kuzu-remote"
GRAPH_DATABASE_URL="http://localhost:8000"
GRAPH_DATABASE_USERNAME="<optional>"
GRAPH_DATABASE_PASSWORD="<optional>"
```
**Installation**: Requires a running Kuzu service exposing an HTTP API.
</Accordion>
<Accordion title="Postgres">
<Warning>
The Postgres graph store is a **demo feature**. In production, use a graph-native backend such as Kuzu or Neo4j for the graph layer — Postgres remains a good default for the relational, vector, and session layers. A production-ready Postgres graph adapter is available as a licensed product; book a call with our sales team at [cognee.ai](https://www.cognee.ai).
</Warning>
Use Postgres as a graph store when you want relational metadata, PGVector, and graph state to live in the same Postgres service.
```dotenv theme={null}
GRAPH_DATABASE_PROVIDER="postgres_demo"
```
`postgres_demo` is the canonical provider name. The older value `postgres` is still accepted and resolves to the same adapter, so existing deployments keep working without a configuration change.
If `GRAPH_DATABASE_HOST`, `GRAPH_DATABASE_PORT`, `GRAPH_DATABASE_NAME`, `GRAPH_DATABASE_USERNAME`, and `GRAPH_DATABASE_PASSWORD` are omitted, Cognee falls back to the relational `DB_*` settings. This is the usual setup when the same Postgres database backs Cognee metadata, vectors, and graph state:
```dotenv theme={null}
DB_PROVIDER="postgres"
DB_NAME="cognee_db"
DB_HOST="127.0.0.1"
DB_PORT="5432"
DB_USERNAME="cognee"
DB_PASSWORD="cognee"
GRAPH_DATABASE_PROVIDER="postgres_demo"
```
For managed Postgres providers such as Neon, configure SSL and pooler settings through the relational database configuration — the Postgres graph engine reuses `DATABASE_CONNECT_ARGS`. See [Relational Databases](/setup-configuration/relational-databases#managed-postgres-with-ssl-connect-args).
The Postgres graph backend stores graph data in `graph_node` and `graph_edge` tables. It does not support raw Cypher queries, so both `SearchType.CYPHER` and `SearchType.NATURAL_LANGUAGE` (which generates and executes Cypher) raise `SearchTypeNotSupported` on this backend.
<Note>
**Writes are serialized across processes.** Every write path takes a transaction-scoped Postgres advisory lock (`pg_advisory_xact_lock`), so concurrent writers queue instead of running in parallel and updates from separate workers are no longer lost. Reads never take the lock, the lock is released on both commit and rollback, and the database user must be able to acquire advisory locks — standard PostgreSQL installations allow this.
</Note>
</Accordion>
<Accordion title="Neo4j (Self-Hosted)">
Neo4j is recommended for production environments where you need a powerful, dedicated graph database. Data is stored on the Neo4j server (local or remote), not on the Cognee host machine.
```dotenv theme={null}
GRAPH_DATABASE_PROVIDER="neo4j"
GRAPH_DATABASE_URL="bolt://localhost:7687"
GRAPH_DATABASE_NAME="neo4j"
GRAPH_DATABASE_USERNAME="neo4j"
GRAPH_DATABASE_PASSWORD="pleaseletmein"
```
**Installation**: Install Neo4j extras:
```bash theme={null}
pip install "cognee[neo4j]"
```
**Docker Setup**: Start the bundled Neo4j service with APOC + GDS plugins:
```bash theme={null}
docker compose --profile neo4j up -d
```
**Transient error handling**: Every Cypher query Cognee sends to Neo4j is retried automatically when the server reports a deadlock (`DeadlockDetected`), any `Neo.TransientError`, or a `DatabaseUnavailable` error. Retries use exponential backoff with jitter and allow up to 10 retries after the initial attempt before the original error is re-raised. All three error classes share the same retry budget — previously `DatabaseUnavailable` gave up one attempt earlier than the others. This retry logic is built into Cognee's Neo4j adapter (not the Neo4j driver itself) and is not configurable through an environment variable.
</Accordion>
<Accordion title="Neo4j Desktop (Local Development)">
Use Neo4j Desktop when you want a local Neo4j database with a graphical interface for inspecting Cognee's generated graph, or when you want to test Cognee's [multi-user mode](/core-concepts/multi-user-mode/multi-user-mode-overview) locally. Full multi-user support in Neo4j, including role-based access control and database isolation per user, is otherwise only available in the Enterprise edition; Neo4j Desktop includes these capabilities for local development at no cost. (Alternatively, the [`neo4j_community` dataset database handler](/core-concepts/multi-user-mode/dataset-database-handlers/existing-dataset-database-handlers/neo4j-community) gets per-dataset isolation on the free Community edition by running one Docker container per dataset.) This is still a `neo4j` graph provider from Cognee's perspective; Neo4j Desktop is only how you run and manage the local database.
<Steps>
<Step title="Create and start a local database">
In Neo4j Desktop:
1. Create a new project
2. Add a local DBMS/database
3. Set and save the database password
4. Start the database
Keep the Bolt port available at `7687` unless you intentionally changed it.
</Step>
<Step title="Install APOC">
Select the database in Neo4j Desktop, open the plugins area, install **APOC**, then restart the database.
APOC is required for Cognee's type-specific Neo4j labels. Without it, Cognee can still write graph data, but nodes may only show the generic `__Node__` label in Neo4j Browser.
</Step>
<Step title="Configure Cognee">
Add the Neo4j connection details to your `.env` file:
```dotenv theme={null}
GRAPH_DATABASE_PROVIDER="neo4j"
GRAPH_DATABASE_URL="bolt://localhost:7687"
GRAPH_DATABASE_NAME="neo4j"
GRAPH_DATABASE_USERNAME="neo4j"
GRAPH_DATABASE_PASSWORD="<your-neo4j-desktop-password>"
```
Install the Neo4j extra in the environment where Cognee runs:
```bash theme={null}
pip install "cognee[neo4j]"
```
</Step>
<Step title="Verify the connection">
Open Neo4j Browser from Neo4j Desktop and run:
```cypher theme={null}
RETURN apoc.version() AS apocVersion;
```
If the query returns a version, APOC is available. Then run your Cognee ingestion/search flow and inspect the graph in Neo4j Browser.
</Step>
</Steps>
<Note>
Neo4j Desktop is best for local development and graph inspection. For team or production deployments, use the Docker/self-hosted setup or Neo4j Aura so the database is managed independently from a developer workstation.
</Note>
</Accordion>
<Accordion title="Neo4j Aura (Cloud)">
[Neo4j Aura](https://neo4j.com/docs/aura/) is Neo4j's fully managed cloud service. Graph data is stored in Neo4j's cloud infrastructure — nothing is stored locally on your machine.
There are two ways to use Neo4j Aura with Cognee:
**Option 1 — Connect to an existing Aura instance**
Create a free or paid Aura instance at [console.neo4j.io](https://console.neo4j.io), then point Cognee at it using the `neo4j+s://` connection URI provided in your Aura console:
```dotenv theme={null}
GRAPH_DATABASE_PROVIDER="neo4j"
GRAPH_DATABASE_URL="neo4j+s://<your-instance-id>.databases.neo4j.io"
GRAPH_DATABASE_NAME="neo4j"
GRAPH_DATABASE_USERNAME="neo4j"
GRAPH_DATABASE_PASSWORD="<your-aura-password>"
```
**Option 2 — Auto-provisioned Aura instances per dataset (multi-user mode)**
Cognee's `Neo4jAuraDevDatasetDatabaseHandler` can automatically create and delete a dedicated Neo4j Aura instance for each Cognee dataset. This requires Neo4j Aura API credentials (OAuth):
```dotenv theme={null}
GRAPH_DATABASE_PROVIDER="neo4j"
GRAPH_DATASET_DATABASE_HANDLER="neo4j_aura_dev"
NEO4J_CLIENT_ID=<your_oauth_client_id>
NEO4J_CLIENT_SECRET=<your_oauth_client_secret>
NEO4J_TENANT_ID=<your_aura_tenant_id>
NEO4J_ENCRYPTION_KEY=<key_for_encrypting_stored_credentials>
```
See the [Neo4j Aura Dataset Database Handler](/core-concepts/multi-user-mode/dataset-database-handlers/existing-dataset-database-handlers/neo4j-aura-dev) page for full details on Option 2.
**Installation**: Install Neo4j extras:
```bash theme={null}
pip install "cognee[neo4j]"
```
</Accordion>
<Accordion title="Neptune (Graph-only)">
Use Amazon Neptune for cloud-based graph storage.
```dotenv theme={null}
GRAPH_DATABASE_PROVIDER="neptune"
GRAPH_DATABASE_URL="neptune-graph://<GRAPH_ID>"
# AWS credentials via environment or default SDK chain
# Required — no per-dataset database handler exists for Neptune
ENABLE_BACKEND_ACCESS_CONTROL="false"
```
**Installation**: Install Neptune extras:
```bash theme={null}
pip install "cognee[neptune]"
```
**Note**: AWS credentials should be configured via environment variables or AWS SDK.
**Access control must be off.** Cognee registers no per-dataset database handler for Neptune, so with `ENABLE_BACKEND_ACCESS_CONTROL` left at its default the handler stays `ladybug` and the first dataset access raises `The selected graph dataset to database handler does not work with the configured graph database provider`. All datasets then share the one graph.
</Accordion>
<Accordion title="Neptune Analytics (Hybrid)">
Use Amazon Neptune Analytics as a hybrid vector + graph backend.
```dotenv theme={null}
GRAPH_DATABASE_PROVIDER="neptune_analytics"
GRAPH_DATABASE_URL="neptune-graph://<GRAPH_ID>"
# AWS credentials via environment or default SDK chain
# Required — no per-dataset database handler exists for Neptune Analytics
ENABLE_BACKEND_ACCESS_CONTROL="false"
```
**Installation**: Install Neptune extras:
```bash theme={null}
pip install "cognee[neptune]"
```
**Note**: This is the same as the vector store configuration. Neptune Analytics serves both purposes.
**Access control must be off.** Cognee registers no per-dataset database handler for Neptune Analytics, so with `ENABLE_BACKEND_ACCESS_CONTROL` left at its default the first dataset access raises on the handler mismatch. All datasets then share the one graph.
</Accordion>
</AccordionGroup>
## Advanced Options
<Accordion title="Backend Access Control">
Enable per-user dataset isolation for multi-tenant scenarios.
```dotenv theme={null}
ENABLE_BACKEND_ACCESS_CONTROL="true"
```
This feature is available for Kuzu and other supported graph stores.
</Accordion>
## Provider Comparison
<Accordion title="Graph Store Comparison">
| Provider | Data Location | Setup | Performance | Use Case |
| ------------------- | -------------------- | --------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Kuzu | Local disk | Zero setup | Good | Local development |
| Kuzu-remote | Remote server | Server required | Good | Remote access |
| Neo4j (self-hosted) | Neo4j server | Server required | Excellent | Production |
| Neo4j Desktop | Local Neo4j database | Desktop app + APOC | Excellent | Local development, graph inspection, and local multi-user testing (Enterprise features available at no cost locally) |
| Neo4j Aura | Neo4j cloud | Aura account required | Excellent | Managed cloud |
| Postgres | Postgres tables | Postgres required | Good | **Demo** — co-locates graph state with relational metadata and PGVector. In production, use a graph-native backend or the licensed adapter (via [cognee.ai](https://www.cognee.ai)) |
| Neptune | AWS cloud | AWS required | Excellent | Cloud solution |
| Neptune Analytics | AWS cloud | AWS required | Excellent | Hybrid cloud solution |
</Accordion>
## Important Considerations
<Accordion title="Data Location">
* **Local providers** (Kuzu): Graph files are created automatically under `SYSTEM_ROOT_DIRECTORY`
* **Local Neo4j Desktop**: Graph data is stored in the database managed by Neo4j Desktop, not under Cognee's `SYSTEM_ROOT_DIRECTORY`
* **Remote providers** (Neo4j, Neptune): Require running services or cloud setup
* **Path management**: Kuzu graph files are managed automatically; Neo4j Desktop data is managed inside Neo4j Desktop
</Accordion>
<Accordion title="Performance Notes">
* **Kuzu**: Single-file storage with good local performance
* **Neo4j**: Excellent for production workloads with proper indexing
* **Neptune**: Cloud-scale performance with managed infrastructure
* **Hybrid solutions**: Combine graph and vector capabilities in one system
</Accordion>
## Community-Maintained Providers
Additional graph stores are available through community-maintained adapters:
* **[Memgraph](/setup-configuration/community-maintained/memgraph)** — In-memory graph database (Bolt protocol)
## Notes
<AccordionGroup>
<Accordion title="Backend Access Control">
When backend access control is enabled, Cognee can isolate graph data per dataset for supported providers such as Kuzu. This is mainly relevant for multi-user deployments where different users or workloads should not share the same graph state by default.
</Accordion>
<Accordion title="Path Management">
For local Kuzu setups, Cognee creates and manages the graph database automatically under the system directory. In most cases you do not need to manually create graph files or point Cognee at a specific path unless you are customizing `SYSTEM_ROOT_DIRECTORY`.
</Accordion>
<Accordion title="Cloud Integration">
Neptune-based setups rely on AWS credentials, network access, and the appropriate IAM permissions to connect successfully. Before using Neptune or Neptune Analytics, make sure your environment can authenticate through the standard AWS SDK credential chain.
</Accordion>
<Accordion title="Neo4j Type Labels">
**APOC is required for Neo4j type labels.** Cognee uses `apoc.create.addLabels` to apply type-specific labels to every node. Without it, Cognee can still store nodes, but Neo4j will only show the generic `__Node__` label, making entity types harder to inspect directly.
The bundled Docker Compose Neo4j profile already includes APOC. For self-hosted Neo4j instances, install the [APOC plugin](https://neo4j.com/docs/apoc/current/installation/) before connecting Cognee.
</Accordion>
<Accordion title="Troubleshooting: 'No nodes found'">
The warning `No nodes found in the database` means the graph DB at the current path is empty. Common causes:
<AccordionGroup>
<Accordion title="Path mismatch between runs">
For Kuzu, the graph is stored on disk at:
```text theme={null}
{SYSTEM_ROOT_DIRECTORY}/databases/cognee_graph_kuzu
```
`SYSTEM_ROOT_DIRECTORY` defaults to a `.cognee_system` folder inside the installed Cognee package directory (typically inside your virtual environment). If this path resolves differently across sessions — for example, after reinstalling packages or using a different virtual environment — each session sees an empty graph.
Pin it to an explicit absolute path in your `.env`:
```dotenv theme={null}
SYSTEM_ROOT_DIRECTORY=/home/user/my-project/.cognee_system
```
Raw ingested files (added via `add()`) are stored separately under `DATA_ROOT_DIRECTORY`, which defaults to `.data_storage` in the same package directory. Set both if you want fully portable storage:
```dotenv theme={null}
SYSTEM_ROOT_DIRECTORY=/home/user/my-project/.cognee_system
DATA_ROOT_DIRECTORY=/home/user/my-project/.data_storage
```
</Accordion>
<Accordion title="Data not yet processed in this session">
The graph persists on disk between runs — you do not need to re-run `add()` + `cognify()` every time. Calling `search()` or [`visualize_graph()`](/guides/graph-visualization) in a new session will find existing data, as long as `SYSTEM_ROOT_DIRECTORY` points to the same location as the original ingestion run.
</Accordion>
<Accordion title="Graph was pruned">
`cognee.prune.prune_system()` deletes all graph and system data. Re-run `add()` and `cognify()` to rebuild it.
</Accordion>
</AccordionGroup>
</Accordion>
</AccordionGroup>
<Columns>
<Card title="Vector Stores" icon="database" href="/setup-configuration/vector-stores">
Configure vector databases for embedding storage
</Card>
<Card title="Relational Databases" icon="database" href="/setup-configuration/relational-databases">
Set up SQLite or Postgres for metadata storage
</Card>
<Card title="Overview" icon="settings" href="/setup-configuration/overview">
Return to setup configuration overview
</Card>
</Columns>
# LLM Providers
Source: https://docs.cognee.ai/setup-configuration/llm-providers
Configure LLM providers for text generation and reasoning in Cognee
LLM (Large Language Model) providers handle text generation, reasoning, and structured output tasks in Cognee. You can choose from cloud providers like OpenAI and Anthropic, or run models locally with Ollama.
<Info>
**New to configuration?**
See the [Setup Configuration Overview](./overview) for the complete workflow:
install extras → create `.env` → choose providers → handle pruning.
</Info>
## Supported Providers
Cognee supports multiple LLM providers:
* **OpenAI** — GPT models via OpenAI API (default)
* **Azure OpenAI** — GPT models via Azure OpenAI Service
* **Google Gemini** — Gemini models via Google AI
* **Anthropic** — Claude models via Anthropic API
* **AWS Bedrock** — Models available via AWS Bedrock
* **Groq** — Fast inference via Groq API (via LiteLLM)
* **Ollama** — Local models via Ollama
* **LM Studio** — Local models via LM Studio
* **HuggingFace** — Models via HuggingFace Inference API or Inference Endpoints
* **llama.cpp** — Local models via llama-cpp-python (in-process or server mode)
* **Custom** — OpenAI-compatible endpoints (like vLLM, OpenRouter, DeepInfra, company-internal)
* **MCP Sampling** — Reuse the host harness's LLM via MCP `sampling/createMessage` (no `LLM_API_KEY`; only when Cognee runs as an MCP server under a host that grants sampling)
These names are not an allowlist. Any model reachable through an OpenAI-compatible endpoint can be configured — see [Custom Providers](#custom-providers) for the generic path and its compatibility requirements.
<Warning>
**LLM/Embedding Configuration**: If you configure only LLM or only embeddings, the other defaults to OpenAI. Cognee rejects this mismatch up front — `add()` and `remember()` fail with `ProviderConfigMismatchError` before any ingestion work happens. Configure both LLM and embeddings, or keep a working OpenAI API key for the side you leave at its defaults — see [LLM/Embedding Configuration](/setup-configuration/overview#configuration-workflow).
</Warning>
## Choosing a Model
Cognee always uses **two** models together: an **LLM** for entity/relationship extraction and reasoning, and an **embedding model** for semantic search. An embedding model is mandatory — every `cognify` run writes vectors to a [vector store](/setup-configuration/vector-stores), and [recall](/core-concepts/main-operations/recall) depends on them. If you only set one, the other silently falls back to OpenAI (see the warning above).
<AccordionGroup>
<Accordion title="Light vs. powerful LLM">
A small, fast model is the right default. Cognee ships with one (`openai/gpt-5-mini`) and the examples on this page use comparable light models such as `gpt-4o-mini`. Knowledge-graph extraction is many short, schema-constrained calls per document rather than a few long ones, so a light model keeps cost and latency low while handling most workloads well.
Reach for a more powerful model when:
* Your sources are dense or domain-specific (legal, medical, scientific) and you need higher-fidelity entities and relationships.
* You use a [custom graph model](/guides/custom-graph-model) or [ontology](/guides/ontology-support) with a complex schema the model must populate accurately.
* A light model produces noisy or incomplete graphs on your data.
Extraction relies on [structured output](/setup-configuration/structured-output-backends), so very small or weak models may return malformed JSON or lower-quality graphs. If a small local model struggles, try a stronger one or adjust the [instructor mode](#llm-instructor-modes).
</Accordion>
<Accordion title="Resource expectations for local models">
The embedding model is lightweight — defaults like `nomic-embed-text` (Ollama) or `all-MiniLM-L6-v2` ([Fastembed](/setup-configuration/embedding-providers#fastembed-local), CPU-only) run comfortably on a CPU or a small GPU and rarely dominate resource use.
The **LLM** is the constraint for local setups. The [Local Setup guide](/guides/local-setup) defaults to an 8B model (`llama3.1:8b`); as a rough guide, an 8B model quantized to 4-bit needs roughly 6 GB of free VRAM, while larger or less-quantized models need proportionally more. If a model does not fit, it spills to system RAM and CPU, which still works but is much slower — for [llama.cpp](#llama-cpp-local) you can tune `LLAMA_CPP_N_GPU_LAYERS` to offload only as many layers as fit. Limited VRAM does not change graph quality; it mainly affects how fast `cognify` runs, since extraction issues many sequential LLM calls per document. With low VRAM, prefer a smaller LLM and lower `EMBEDDING_BATCH_SIZE` (see [Embedding Providers](/setup-configuration/embedding-providers#batch-size)) over a large model that does not fit.
</Accordion>
<Accordion title="Recommended local models">
`llama3.1:8b` is the recommended Ollama default because it hits a practical sweet spot for Cognee's workload: it follows instructions and produces valid [structured output](/setup-configuration/structured-output-backends) reliably (the Ollama provider defaults to [`json_schema_mode`](#llm-instructor-modes), so Ollama 0.5+ enforces the schema rather than just being asked for JSON), while staying small enough to run on modest hardware (\~6 GB VRAM at 4-bit). Other validated options: `llama3.2:3b` for lightweight or resource-constrained environments, and `qwen2.5:14b` as a mid-size alternative; larger tags (`llama3.1:70b`, `llama3.3`, `qwen2.5:32b`) improve extraction fidelity on dense or domain-specific sources at proportionally higher resource cost. For the full classification — recommended and problematic models, and what Cognee logs at startup for each — see [Model Support Warning](#model-support-warning) in the Ollama setup guide below; on problematic models the failures surface as `InstructorRetryException` errors or empty graphs.
</Accordion>
</AccordionGroup>
## Configuration
<Accordion title="Environment Variables">
Set these environment variables in your `.env` file:
* `LLM_PROVIDER` — The provider to use: `openai`, `azure`, `anthropic`, `gemini`, `mistral`, `bedrock`, `ollama`, `llama_cpp`, `custom`, `mcp-sampling`. Optional — when it is unset, Cognee infers it from the `LLM_MODEL` prefix (see the note below)
* `LLM_MODEL` — The specific model to use
* `LLM_API_KEY` — Your API key for the provider (not used by `mcp-sampling`)
* `LLM_ENDPOINT` — Custom endpoint URL (for Azure, Ollama, or custom providers)
* `LLM_API_VERSION` — API version (for Azure OpenAI)
* `LLM_TEMPERATURE` — Sampling temperature, sent with every LLM call when you set it explicitly — and, on local inference servers (`ollama`, `llama_cpp`, LM Studio), also when you don't: there an unset value sends `0.0`. On every other provider, leaving it unset sends no temperature at all and the provider's own default applies (see [Temperature and Seed](#temperature-and-seed))
* `LLM_SEED` — Sampling seed for reproducible outputs, sent when set (provider support varies)
* `LLM_ANSWER_STREAMING` — Stream answer tokens out of recall's final answer call as they are generated, for clients consuming the [REST API's SSE response](/guides/deploy-rest-api-server#http-api-examples) (default `false`). The returned value is identical either way, so it is inert unless a client is reading the stream. Token streaming is supported on the default `litellm_native` structured-output backend; on `instructor` only the OpenAI-compatible adapters stream (`openai`, `custom`, and Azure without managed identity), and `baml` never streams
* `LLM_MAX_COMPLETION_TOKENS` — Maximum tokens per request (optional)
* `LLM_INSTRUCTOR_MODE` — Structured-output mode override for Instructor-backed LLM calls (optional)
* `LLM_EXTRACTION_*`, `LLM_SUMMARIZATION_*`, `LLM_QUERY_*` — Optional per-stage overrides that route individual pipeline stages to different models/providers (see [Per-Stage Model Routing](#per-stage-model-routing))
For every LLM variable Cognee reads — including rate limiting, fallback, transcription, and the local-runtime knobs — with its default in one table, see the [LLM environment variable reference](/setup-configuration/overview#environment-variable-quick-reference).
</Accordion>
A preflight LLM connection test can time out at 30s, especially against smaller models. To skip it, add `COGNEE_SKIP_CONNECTION_TEST=true` to your `.env`.
<Info>
**Why do model names have a prefix like `gemini/` or `openrouter/`?**
Cognee routes all LLM requests through [LiteLLM](https://docs.litellm.ai/docs/providers), which uses provider prefixes to identify the correct API endpoint. For example, Google lists their model as `gemini-2.0-flash`, but in Cognee you must write `gemini/gemini-2.0-flash`. This prefix tells LiteLLM to use the Gemini API. The same applies to custom providers — `openrouter/`, `hosted_vllm/`, `lm_studio/`, etc. See each provider section below for the correct format.
</Info>
### Provider Inference
`LLM_PROVIDER` is optional. When you don't set it, Cognee infers the provider from the prefix of `LLM_MODEL` — `anthropic/claude-3-5-sonnet` resolves to `anthropic`, `gemini/gemini-2.0-flash` to `gemini`, and so on.
Precedence, highest first:
1. An explicit `llm_provider` passed in Python (for example to `cognee.config.set_llm_config()`).
2. The `LLM_PROVIDER` environment variable.
3. Inference from the `LLM_MODEL` prefix.
An explicitly set provider always wins, even when it disagrees with the model prefix — that is what makes the Azure recipe below (`LLM_PROVIDER="openai"` with `LLM_MODEL="azure/gpt-4o-mini"`) keep working. A model id with no `/` (for example `llama3.1:8b`) has no prefix to infer from, so it falls back to the default `openai` unless you set `LLM_PROVIDER` yourself.
Inference only recognises prefixes Cognee has an adapter for: `openai`, `azure`, `anthropic`, `gemini`, `mistral`, `bedrock`, `ollama`, `llama_cpp`, `custom`, `mcp-sampling`.
Inference resolves `LLM_PROVIDER` *from* the model name. The opposite direction also exists, but only on the `litellm_native` [structured output backend](/setup-configuration/structured-output-backends): once the provider is known, a model id LiteLLM cannot route on its own — a bare `llama3.1:8b`, or a namespaced `library/phi4` — is re-qualified with that provider's prefix before the request is sent. See [Model-name qualification](/setup-configuration/structured-output-backends#configuration) for which providers are prefixed.
Inference runs once, when the configuration is loaded from the environment. Setting `llm_model` later in Python via `cognee.config.set_llm_config()` does **not** re-trigger it — the provider stays whatever it already was (the default `openai` unless configured otherwise). When you configure the model in Python instead of through `LLM_MODEL`, set `llm_provider` explicitly in the same call.
<Warning>
Any other prefix raises `ProviderNotDeducibleError` — Cognee refuses to guess rather than silently falling back to OpenAI. This includes LiteLLM-routed prefixes Cognee has no adapter of its own for, such as `openrouter/`, `groq/`, or `deepseek/`. For those, set `LLM_PROVIDER="custom"`, as every recipe in [Custom Providers](#custom-providers) already does — with `custom`, the prefix is passed straight through to LiteLLM for routing.
</Warning>
## Provider Setup Guides
<AccordionGroup>
<Accordion title="OpenAI (Default)">
OpenAI is the default provider and works out of the box with minimal configuration.
```dotenv theme={null}
LLM_PROVIDER="openai"
LLM_MODEL="gpt-4o-mini"
LLM_API_KEY="sk-..."
# Optional overrides
# LLM_ENDPOINT=https://api.openai.com/v1
# LLM_API_VERSION=
# LLM_MAX_COMPLETION_TOKENS=16384
```
</Accordion>
<Accordion title="Azure OpenAI">
Use Azure OpenAI Service with your own deployment.
```dotenv theme={null}
LLM_PROVIDER="openai"
LLM_MODEL="azure/gpt-4o-mini"
LLM_ENDPOINT="https://<your-resource>.openai.azure.com/openai/deployments/gpt-4o-mini"
LLM_API_KEY="az-..."
LLM_API_VERSION="2024-12-01-preview"
```
</Accordion>
<Accordion title="Google Gemini / Vertex AI">
Cognee routes Gemini requests through [LiteLLM](https://docs.litellm.ai/docs/providers/gemini). There are two ways to reach Gemini models: the **Google AI Studio** API (a single API key) or **Vertex AI** (Google Cloud project + service-account credentials).
<Tabs>
<Tab title="Google AI Studio (API key)">
The simplest setup. Get an API key from [Google AI Studio](https://aistudio.google.com/apikey) and use the `gemini/` model prefix.
```dotenv theme={null}
LLM_PROVIDER="gemini"
LLM_MODEL="gemini/gemini-2.0-flash"
LLM_API_KEY="AIza..."
# Optional
# LLM_ENDPOINT=https://generativelanguage.googleapis.com/
# LLM_API_VERSION=v1beta
```
This path talks to the Gemini REST API directly and needs no extra Google packages.
</Tab>
<Tab title="Vertex AI (Google Cloud)">
Use Vertex AI when your models are served through a Google Cloud project. Vertex routes through LiteLLM's `vertex_ai/` prefix and authenticates with Google Cloud credentials instead of an API key.
```dotenv theme={null}
LLM_PROVIDER="gemini"
LLM_MODEL="vertex_ai/gemini-2.0-flash"
LLM_API_KEY="." # placeholder; Vertex auth uses credentials below
# Google Cloud project + region (read by LiteLLM)
VERTEXAI_PROJECT="your-gcp-project-id"
VERTEXAI_LOCATION="us-central1"
# Path to your service-account key file (Application Default Credentials)
GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"
```
**Installation**: Vertex AI requires the Google client libraries, which Cognee does not bundle by default. Install them with:
```bash theme={null}
uv pip install google-cloud-aiplatform
```
`GOOGLE_APPLICATION_CREDENTIALS` points at a service-account JSON key. If you run inside Google Cloud (or after `gcloud auth application-default login`), you can omit it and rely on [Application Default Credentials](https://cloud.google.com/docs/authentication/application-default-credentials).
</Tab>
</Tabs>
</Accordion>
<Accordion title="Anthropic">
Use Anthropic's Claude models for reasoning tasks.
```dotenv theme={null}
LLM_PROVIDER="anthropic"
LLM_MODEL="claude-sonnet-4-5-20250929"
LLM_API_KEY="sk-ant-..."
```
</Accordion>
<Accordion title="Groq">
Groq provides fast inference for open models. Cognee routes Groq requests through [LiteLLM](https://docs.litellm.ai/docs/providers/groq) using the `groq/` model prefix.
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="groq/llama-3.3-70b-versatile"
LLM_API_KEY="gsk_..."
```
**Installation**: Install the Groq dependency:
```bash theme={null}
pip install cognee[groq]
```
**Popular Groq models** (use with the `groq/` prefix):
* `groq/llama-3.3-70b-versatile`
* `groq/llama3-8b-8192`
* `groq/mixtral-8x7b-32768`
* `groq/gemma2-9b-it`
See the [Groq model list](https://console.groq.com/docs/models) for all available models. Your Groq API key can be created in the [Groq Console](https://console.groq.com/keys).
<Info>
**No endpoint needed**: The `LLM_ENDPOINT` variable is not required for Groq — LiteLLM resolves the Groq API endpoint automatically from the `groq/` prefix.
</Info>
</Accordion>
<Accordion title="AWS Bedrock">
Use models available on AWS Bedrock for various tasks. For Bedrock specifically, you will need to
also specify some information regarding AWS.
```dotenv theme={null}
LLM_API_KEY="<your_bedrock_api_key>"
LLM_MODEL="eu.amazon.nova-lite-v1:0"
LLM_PROVIDER="bedrock"
LLM_MAX_COMPLETION_TOKENS="16384"
AWS_REGION="<your_aws_region>"
AWS_ACCESS_KEY_ID="<your_aws_access_key_id>"
AWS_SECRET_ACCESS_KEY="<your_aws_secret_access_key>"
AWS_SESSION_TOKEN="<your_aws_session_token>"
# Optional parameters
#AWS_BEDROCK_RUNTIME_ENDPOINT="bedrock-runtime.eu-west-1.amazonaws.com"
#AWS_PROFILE_NAME="<your_aws_profile_name>"
```
There are **multiple ways of connecting** to Bedrock models. Cognee picks the first one it finds, in this order:
1. Using an API key and region. Simply generate your key on AWS, and put it in the `LLM_API_KEY` env variable. If `LLM_API_KEY` is set, it takes precedence over the credential and profile methods below, so leave it empty when you want to use those.
2. Using AWS Credentials. You can only specify `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`, no need for the `LLM_API_KEY`.
In this case, if you are using temporary credentials (e.g. `AWS_ACCESS_KEY_ID` starting with `ASIA...`, such as those issued by `aws sso login` or `aws sts assume-role`), then you also
must specify the `AWS_SESSION_TOKEN`. All three values expire and must be refreshed when AWS rotates them.
3. Using AWS profiles. `AWS_PROFILE_NAME` is the **name** of a profile (for example `default` or `my-sso-profile`), not a path to a file or to a folder. Cognee hands the name to boto3, which resolves the credentials through the standard AWS chain using the shared config and credentials files at `~/.aws/config` and `~/.aws/credentials` (override their locations with the `AWS_CONFIG_FILE` and `AWS_SHARED_CREDENTIALS_FILE` env variables). This is the recommended path for **AWS SSO**: run `aws sso login --profile <your_aws_profile_name>` first, then set `AWS_PROFILE_NAME` to that profile name and Cognee will use the temporary SSO credentials boto3 caches for it — no need to copy the `ASIA...` keys into your `.env`.
4. Using an **ambient IAM role**. If `LLM_API_KEY`, the `AWS_ACCESS_KEY_ID`/`AWS_SECRET_ACCESS_KEY` pair, and `AWS_PROFILE_NAME` are all unset, Cognee passes no credentials to Bedrock and boto3 resolves them from the default AWS credential chain — an EC2 instance profile, ECS task role, or EKS IRSA service-account role. Unlike most providers, `bedrock` does not require `LLM_API_KEY`, so no missing-key error is raised; `LLM_PROVIDER`, `LLM_MODEL`, and `AWS_REGION` are enough to authenticate to Bedrock (an [embedding provider](/setup-configuration/embedding-providers) still needs its own configuration). Since resolution is first-match, remove any stale key, credential, or profile values from your `.env` and shell — they take precedence over the role.
**Installation**: Install the required dependency:
```bash theme={null}
pip install cognee[aws]
```
<Info>
**Model Name**
The name of the model might differ based on the region (the name begins with **eu** for Europe, **us** of USA, etc.)
</Info>
See the [AWS Bedrock Integration](/integrations/aws-bedrock-integration) guide for the full setup walkthrough.
</Accordion>
<Accordion title="Ollama (Local)">
Run models locally with Ollama for privacy and cost control.
```dotenv theme={null}
LLM_PROVIDER="ollama"
LLM_MODEL="llama3.1:8b"
LLM_ENDPOINT="http://localhost:11434"
LLM_API_KEY="ollama"
```
`LLM_API_KEY="ollama"` is a placeholder required by the client library — Ollama itself does not validate it.
<Warning>
**`LLM_ENDPOINT` is the bare host — no `/v1` suffix.** On the default `litellm_native` [structured output backend](/setup-configuration/structured-output-backends), LiteLLM routes Ollama over its native API and appends `/api/generate` itself, so a trailing `/v1` makes every request 404. Add `/v1` only if you set `STRUCTURED_OUTPUT_FRAMEWORK="instructor"`, which uses Ollama's OpenAI-compatible API instead. `EMBEDDING_ENDPOINT` is unaffected either way and keeps its `/api/embed` path.
</Warning>
**Installation**: Install Ollama from [ollama.ai](https://ollama.ai) and pull your desired model:
```bash theme={null}
ollama pull llama3.1:8b
```
**Namespaced tags and Hugging Face GGUFs**: Ollama model names are not always bare tags — they can be namespaced (`library/phi4`), and a GGUF pulled from Hugging Face keeps its full path. Both work as `LLM_MODEL` on the `litellm_native` [structured output backend](/setup-configuration/structured-output-backends), which qualifies them to `ollama/…` for routing:
```bash theme={null}
ollama pull hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF
```
```dotenv theme={null}
LLM_PROVIDER="ollama"
LLM_MODEL="hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF"
LLM_ENDPOINT="http://localhost:11434"
LLM_API_KEY="ollama"
STRUCTURED_OUTPUT_FRAMEWORK="litellm_native"
```
Setting `LLM_PROVIDER="ollama"` is required here, not optional: `library` and `hf.co` are not prefixes [provider inference](#provider-inference) recognises, so leaving `LLM_PROVIDER` unset raises `ProviderNotDeducibleError` at configuration load.
<Info>
**Zero-API-key setup**: To avoid falling back to OpenAI for embeddings, you must also configure the embedding provider to use a local backend. See the [Local Setup guide](/guides/local-setup) for a complete `.env` example using Ollama or Fastembed for both LLM and embeddings.
</Info>
### Model Support Warning
When `LLM_PROVIDER="ollama"`, Cognee classifies the configured `LLM_MODEL` against a built-in support matrix as the LLM configuration is loaded, and logs a warning for models it has not validated for structured graph extraction. The check is **advisory only** — nothing is blocked and no exception is raised, so `cognify()` runs either way. The line appears the first time the LLM configuration is loaded in the process.
| Classification | Models | Logged |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Recommended | `llama3`, `llama3.1`, `llama3.2`, `llama3.3`, and any `qwen2.5` tag of 14B or larger (`qwen2.5:14b`, `qwen2.5:32b`, …) | Nothing |
| Problematic | `mistral`, `phi3`, `phi3.5`, and `qwen2.5` below 14B or with no parseable size in its tag (`qwen2.5:7b`, `qwen2.5:latest`, bare `qwen2.5`) | A warning that the model has known limitations for structured graph extraction (schema validation errors, silent drops) and a suggestion to switch to a validated model |
| Unknown | Every model not in the matrix, including `gemma2` | A warning that the model has not been validated and that extraction quality may vary |
Matching ignores an `ollama/` prefix and everything after the `:` in a tag, so `llama3.1`, `llama3.1:8b`, and `ollama/llama3.1:70b` are all classified as `llama3.1`. `qwen2.5` is the exception: its tag is parsed for a parameter count, and that count is what decides between recommended and problematic. A custom Modelfile tag (see below) inherits the classification of its base name — `llama3.1:8b-8k` is still `llama3.1`.
Each distinct `LLM_MODEL` value warns at most once per process, so you see the line once even though the configuration is read many times. The message points at [`docs/ollama_models.md`](https://github.com/topoteretes/cognee/blob/dev/docs/ollama_models.md) in the repo, which carries a similar matrix plus troubleshooting notes.
### Known Issues
* **`ValidationError` on import (`Missing: [...]`)**: Cognee validates the LLM variables as an all-or-nothing group when `LLM_PROVIDER="ollama"` — if you set any of `LLM_MODEL`, `LLM_ENDPOINT`, or `LLM_API_KEY`, you must set all three. Setting only some raises `Value error, You have set some but not all of the required environment variables for LLM usage`.
The check reads the **resolved** LLM configuration rather than the process environment, so values loaded from a `.env` file count exactly the same as exported environment variables. A blank or whitespace-only value counts as unset, and `LLM_MODEL` counts as set only if you actually supply it — leaving it at its default (`openai/gpt-5-mini`) does not satisfy the group, so setting just `LLM_ENDPOINT` and `LLM_API_KEY` fails with `Missing: ['LLM_MODEL']`.
The embedding variables are **no longer** part of this check: `EMBEDDING_PROVIDER`, `EMBEDDING_MODEL` and `EMBEDDING_DIMENSIONS` are owned by the embedding configuration and can be set independently of one another. Setting them together is still the recommended configuration for a local Ollama setup, since Ollama embeddings need `HUGGINGFACE_TOKENIZER` for token counting and an explicit `EMBEDDING_DIMENSIONS` avoids a wrong auto-derived vector size:
```dotenv theme={null}
EMBEDDING_PROVIDER="ollama"
EMBEDDING_MODEL="nomic-embed-text:latest"
EMBEDDING_ENDPOINT="http://localhost:11434/api/embed"
EMBEDDING_DIMENSIONS="768"
HUGGINGFACE_TOKENIZER="nomic-ai/nomic-embed-text-v1.5"
```
See [Embedding Providers → Ollama](/setup-configuration/embedding-providers#ollama-local) for model-to-tokenizer mappings and how to find the right `HUGGINGFACE_TOKENIZER` value.
* **`NoDataError` with mixed providers**: Using Ollama as LLM and OpenAI as embedding provider may fail with `NoDataError`. Workaround: configure both LLM and embeddings to the same local provider (see the local setup guide above).
* **Audio transcription is not supported**: `AudioLoader` relies on a Whisper-compatible transcription endpoint. Cognee's Ollama adapter does not provide one, so audio ingestion will fail when `LLM_PROVIDER="ollama"`.
### Context Window (`num_ctx`) and Custom Modelfiles
If `cognify()` returns HTTP **500 errors** while the same model answers fine when you run `ollama run <model>` in a terminal, the usual cause is **context-window truncation**, not a connection problem.
Ollama's default context window can be much smaller than Cognee's extraction window. Cognee sizes extraction chunks from `LLM_MAX_COMPLETION_TOKENS` (default `16384`) — up to roughly half that per chunk — so the entity-extraction prompts it sends can be far larger than a short terminal prompt. When a prompt exceeds `num_ctx`, Ollama may truncate it, the model can return malformed or empty structured output, and Instructor's parse failure can surface as a 500.
`LLM_MODEL` is just an Ollama model tag, so the fix is to point it at a model whose `num_ctx` is large enough. Create a custom [Modelfile](https://docs.ollama.com/modelfile):
```dockerfile theme={null}
FROM llama3.1:8b
PARAMETER num_ctx 8192
```
Build the tag and reference it in your `.env`:
```bash theme={null}
ollama create llama3.1:8b-8k -f Modelfile
```
```dotenv theme={null}
LLM_PROVIDER="ollama"
LLM_MODEL="llama3.1:8b-8k"
LLM_ENDPOINT="http://localhost:11434"
LLM_API_KEY="ollama"
```
<Info>
A larger `num_ctx` uses more memory. If you can't raise it, lower `LLM_MAX_COMPLETION_TOKENS` instead so Cognee builds smaller chunks that fit the model's existing context window.
</Info>
### Connection Troubleshooting
If you see `cannot connect to host` or `connection refused` errors, the most common causes are an unreachable endpoint, the wrong protocol, or a Docker networking mismatch.
**Default endpoint protocol**
Ollama's local server speaks plain **HTTP**, not HTTPS. Cognee does not add TLS by default — the protocol is determined entirely by the scheme in `LLM_ENDPOINT` and `EMBEDDING_ENDPOINT`. Use `https://` only if you have placed Ollama behind a TLS-terminating reverse proxy (Caddy, nginx, Traefik, etc.). For a local Ollama setup, use:
| Variable | Value |
| ----------------------------- | ---------------------------------- |
| `LLM_ENDPOINT` (Ollama) | `http://localhost:11434` |
| `EMBEDDING_ENDPOINT` (Ollama) | `http://localhost:11434/api/embed` |
The Ollama embedding engine builds a secure SSL context for outgoing requests, but it is only applied when the endpoint URL uses `https://` — plain HTTP requests are not upgraded.
**`localhost` vs `host.docker.internal`**
Inside a Docker container, `localhost` refers to the container itself, not your host machine where Ollama is running. If Cognee runs in Docker and Ollama runs on the host, use `host.docker.internal` instead:
```dotenv theme={null}
LLM_ENDPOINT="http://host.docker.internal:11434"
EMBEDDING_ENDPOINT="http://host.docker.internal:11434/api/embed"
```
`host.docker.internal` is available on Docker Desktop (macOS/Windows) and on Linux when the `host-gateway` mapping is configured in `docker-compose.yml`. On Linux without that mapping, use `--network host` or the Docker bridge IP.
**Other common causes**
* **Ollama not running**: verify with `curl http://localhost:11434/api/tags` from the same machine and network namespace Cognee is running in.
* **Wrong port**: the default Ollama port is `11434`. If you started Ollama with `OLLAMA_HOST=0.0.0.0:<port>`, match that port in `LLM_ENDPOINT`.
* **Wrong path suffix on `LLM_ENDPOINT`**: on the default `litellm_native` backend the LLM endpoint must be the **bare host** (`http://localhost:11434`) — LiteLLM appends `/api/generate` to it for Ollama's native route. A trailing `/v1` produces:
```
MaskedHTTPStatusError: Client error '404 Not Found' for url 'http://localhost:11434/v1/api/generate'
litellm.APIConnectionError: OllamaException - 404 page not found
```
The fix is to drop the `/v1`. Add it back only if you have set `STRUCTURED_OUTPUT_FRAMEWORK="instructor"`, which uses Ollama's OpenAI-compatible API instead. The embedding endpoint is separate and must always end in `/api/embed`.
* **Bind address**: Ollama binds to `127.0.0.1` by default. To accept connections from other machines or Docker containers via a LAN IP, start it with `OLLAMA_HOST=0.0.0.0:11434`.
* **Inside Docker Compose**: if the LLM or embedding endpoint runs on your host machine, `localhost` inside the container points back to the container itself. Use `host.docker.internal` on Docker Desktop (macOS/Windows), or add a `host-gateway` mapping in `docker-compose.yml` on Linux.
```dotenv theme={null}
LLM_ENDPOINT="http://host.docker.internal:11434"
EMBEDDING_ENDPOINT="http://host.docker.internal:11434/api/embed"
```
If the service runs in the same Compose project, use the Compose **service name** instead of `localhost` for any `DB_HOST`, `VECTOR_DB_URL`, or `GRAPH_DATABASE_URL` setting.
</Accordion>
<Accordion title="HuggingFace">
Use models from HuggingFace via the [HuggingFace Inference API](https://huggingface.co/docs/api-inference/index) (serverless) or dedicated [Inference Endpoints](https://huggingface.co/docs/inference-endpoints/index).
<Tabs>
<Tab title="Serverless">
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="huggingface/mistralai/Mistral-7B-Instruct-v0.3"
LLM_API_KEY="hf_..."
```
</Tab>
<Tab title="Dedicated Endpoint">
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="huggingface/mistralai/Mistral-7B-Instruct-v0.3"
LLM_ENDPOINT="https://<your-endpoint-id>.<region>.aws.endpoints.huggingface.cloud/v1/"
LLM_API_KEY="hf_..."
```
</Tab>
</Tabs>
**Installation**: Install the HuggingFace extra to enable the HuggingFace tokenizer used for chunking:
```bash theme={null}
pip install cognee[huggingface]
```
<Info>
**Model names**: Use the full HuggingFace model repo ID after the `huggingface/` prefix (e.g., `huggingface/mistralai/Mixtral-8x7B-Instruct-v0.1`). Not all models on HuggingFace support the text generation inference API — check the model card for compatibility. The model is routed through [LiteLLM](https://docs.litellm.ai/docs/providers/huggingface).
</Info>
</Accordion>
<Accordion title="LM Studio (Local)">
Run models locally with LM Studio for privacy and cost control.
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="lm_studio/magistral-small-2509"
LLM_ENDPOINT="http://127.0.0.1:1234/v1"
LLM_API_KEY="."
LLM_INSTRUCTOR_MODE="json_schema_mode"
```
**Installation**: Install LM Studio from [lmstudio.ai](https://lmstudio.ai/) and download your desired model from
LM Studio's interface.
Load your model, start the LM Studio server, and Cognee will be able to connect to it.
<Info>
**Set up instructor mode**: `LLM_INSTRUCTOR_MODE` controls how Cognee asks the model for structured output. LM Studio models often work best with `json_schema_mode`. For more detail, see [LLM Instructor Modes](#llm-instructor-modes) below and [Structured Output Backends](/setup-configuration/structured-output-backends).
</Info>
### Complete `.env` (LLM + embeddings on one LM Studio server)
Configure the embedding side too, or embeddings still default to OpenAI — succeeding silently against api.openai.com if `OPENAI_API_KEY` is set in your environment, or failing on the missing key. Load both a chat model and an embedding model in LM Studio, then point Cognee at the same base URL:
```dotenv theme={null}
# LLM — routed through LiteLLM, so the model needs the lm_studio/ prefix
LLM_PROVIDER="custom"
LLM_MODEL="lm_studio/magistral-small-2509"
LLM_ENDPOINT="http://127.0.0.1:1234/v1"
LLM_API_KEY="."
LLM_INSTRUCTOR_MODE="json_schema_mode"
# Embeddings — talks to /v1/embeddings directly, model id used verbatim
EMBEDDING_PROVIDER="openai_compatible"
EMBEDDING_MODEL="text-embedding-nomic-embed-text-v1.5"
EMBEDDING_ENDPOINT="http://127.0.0.1:1234/v1"
EMBEDDING_API_KEY="."
EMBEDDING_DIMENSIONS="768"
```
Both endpoints are the **base** URL ending in `/v1` — not `/v1/chat/completions` or `/v1/embeddings`. `LLM_API_KEY`/`EMBEDDING_API_KEY` are placeholders LM Studio does not validate. Set `EMBEDDING_DIMENSIONS` to your embedding model's real output size (`768` for `nomic-embed-text-v1.5`), otherwise it falls back to `3072` and the first vector-store write fails with a shape mismatch.
The embedding half can also be routed through LiteLLM with `EMBEDDING_PROVIDER="custom"` and an `lm_studio/`-prefixed model — see [Embedding Providers → LM Studio](/setup-configuration/embedding-providers#lm-studio-local) and [Valid EMBEDDING\_PROVIDER values and endpoint URL forms](/setup-configuration/embedding-providers#valid-embedding_provider-values-and-endpoint-url-forms) for when to pick which.
</Accordion>
<Accordion title="llama.cpp (Local)">
Run models locally using [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) for full offline inference.
Cognee supports two setup modes:
* **Local mode** — Load a `.gguf` model directly in-process
* **Server mode** — Connect to a running `llama-cpp-python` server over HTTP
**Installation**: Install the required dependency:
```bash theme={null}
pip install cognee[llama-cpp]
```
<Info>
**Choosing a mode**: Use local mode for the simplest setup with no separate server process. Use server mode if you want to share one model across multiple processes or run the model on another machine.
</Info>
<AccordionGroup>
<Accordion title="Local Mode (In-Process)">
Load a GGUF model file directly. No server setup required.
```dotenv theme={null}
LLM_PROVIDER="llama_cpp"
LLAMA_CPP_MODEL_PATH="/path/to/your/model.gguf"
# Optional: context window size (default: 2048)
LLAMA_CPP_N_CTX=4096
# Optional: GPU layers to offload (default: 0 = CPU only, -1 = all layers on GPU)
LLAMA_CPP_N_GPU_LAYERS=35
# Optional: chat format (default: chatml)
LLAMA_CPP_CHAT_FORMAT="chatml"
```
<Info>
**GPU acceleration**: Set `LLAMA_CPP_N_GPU_LAYERS=-1` to offload all layers to GPU, or set a positive integer to offload a specific number of layers. Leave it at `0` for CPU-only inference.
**Concurrency**: In local in-process mode the model is loaded once and shared across calls. Because the underlying `llama_cpp.Llama` instance is not thread-safe, Cognee serializes concurrent structured-output calls (such as the per-chunk extraction that `cognify()` fans out) on that single instance. This means in-process requests are processed one at a time rather than in parallel; if you need parallel decoding, run a `llama-cpp-python` server and use **Server Mode** instead.
</Info>
</Accordion>
<Accordion title="Server Mode (OpenAI-Compatible)">
Connect to a running `llama-cpp-python` server. Start the server separately:
```bash theme={null}
python -m llama_cpp.server --model /path/to/your/model.gguf --port 8000
```
Then configure Cognee to connect to it:
```dotenv theme={null}
LLM_PROVIDER="llama_cpp"
LLM_ENDPOINT="http://localhost:8000/v1"
LLM_API_KEY="."
LLM_MODEL="your-model-name"
```
</Accordion>
</AccordionGroup>
</Accordion>
<Accordion title="Custom Providers">
Use any OpenAI-compatible endpoint — OpenRouter, vLLM, a company-internal gateway, or other services.
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="openai/<model-id-your-endpoint-reports>"
LLM_ENDPOINT="https://<your-provider-host>/v1"
LLM_API_KEY="<your-api-key>"
# Optional: fallback provider for content policy violations
# FALLBACK_MODEL=openrouter/openai/gpt-4o-mini
# FALLBACK_ENDPOINT=https://openrouter.ai/api/v1
# FALLBACK_API_KEY=or-...
```
See [Fallback Provider](#fallback-provider) in Advanced Options for full details.
**Custom Provider Prefixes**: When using `LLM_PROVIDER="custom"`, you must include the correct provider prefix in your model name. Cognee forwards requests to [LiteLLM](https://docs.litellm.ai/docs/providers), which uses these prefixes to route requests correctly.
`LLM_PROVIDER="custom"` is **required** for these recipes, not just conventional: Cognee cannot infer a provider from a prefix it has no adapter for, so omitting the line raises `ProviderNotDeducibleError`. See [Provider Inference](#provider-inference).
Common prefixes include:
* `hosted_vllm/` — vLLM servers
* `openrouter/` — OpenRouter
* `lm_studio/` — LM Studio
* `openai/` — OpenAI-compatible APIs
See the [LiteLLM providers documentation](https://docs.litellm.ai/docs/providers) for the full list of supported prefixes.
**Any OpenAI-compatible model works** — the providers listed on this page are examples, not an allowlist. Cognee does not validate `LLM_MODEL` against a set of known models: the value is passed through to LiteLLM, which forwards the request to whatever `LLM_ENDPOINT` you configure. A model that was released after your Cognee version, or one served only from a private gateway, is configured with the same template shown at the top of this section.
**Compatibility requirements**
* **OpenAI-compatible chat completions**: the endpoint must expose a `/v1/chat/completions` route that accepts `system` and `user` messages. `LLM_ENDPOINT` is the base URL (usually ending in `/v1`).
* **Exact model id**: whatever follows the LiteLLM prefix must be the id your server accepts — typically the id returned by the endpoint's `/v1/models`.
* **Structured output**: graph extraction requires the model to return JSON matching a schema. The `custom` provider defaults to the `json_mode` [instructor mode](#llm-instructor-modes); if extraction returns malformed JSON, try `tool_call` or `json_schema_mode`. Weaker models may produce lower-quality graphs even when the transport works — see [Structured Output Backends](/setup-configuration/structured-output-backends).
* **An API key value**: `custom` always sends one, so set `LLM_API_KEY="."` if your server does not authenticate.
* **Embeddings are separate**: the model does not need to serve embeddings — configure those independently under [Embedding Providers](/setup-configuration/embedding-providers).
<Info>
**Token limits for unknown models**: a model missing from LiteLLM's registry has no known output limit, so your `LLM_MAX_COMPLETION_TOKENS` is used as-is — lower it if the model's real output limit is smaller. See [Max Completion Tokens](#max-completion-tokens) under Advanced Options for how the ceiling is applied.
</Info>
Below are examples for common providers and patterns:
<Accordion title="DeepSeek">
Use DeepSeek's models for reasoning and chat via their OpenAI-compatible API.
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="deepseek/deepseek-chat"
LLM_ENDPOINT="https://api.deepseek.com/v1"
LLM_API_KEY="sk-..."
```
Get your API key from [platform.deepseek.com](https://platform.deepseek.com/api_keys). The `deepseek/` prefix tells LiteLLM to route to the DeepSeek API.
**Popular DeepSeek models** (use with the `deepseek/` prefix):
* `deepseek/deepseek-chat` — DeepSeek-V3 (general chat and instruction following)
* `deepseek/deepseek-reasoner` — DeepSeek-R1 (chain-of-thought reasoning)
<Info>
**Structured output**: DeepSeek's API is OpenAI-compatible, so the default `json_mode` for `custom` providers works well. If you encounter issues with structured output, try setting `LLM_INSTRUCTOR_MODE="tool_call"`.
</Info>
</Accordion>
<Accordion title="Kimi (Moonshot AI)">
Use Moonshot AI's Kimi models via their OpenAI-compatible API.
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="moonshot/moonshot-v1-32k"
LLM_ENDPOINT="https://api.moonshot.cn/v1"
LLM_API_KEY="sk-..."
```
Get your API key from [platform.moonshot.cn](https://platform.moonshot.cn/console/api-keys). The `moonshot/` prefix tells LiteLLM to route to the Moonshot AI API.
**Available Kimi models** (use with the `moonshot/` prefix):
* `moonshot/moonshot-v1-8k` — 8k context window
* `moonshot/moonshot-v1-32k` — 32k context window
* `moonshot/moonshot-v1-128k` — 128k context window (for long documents)
</Accordion>
<Accordion title="OpenRouter">
Use [OpenRouter](https://openrouter.ai) to access hundreds of models from a single API endpoint.
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="openrouter/deepseek/deepseek-r1"
LLM_ENDPOINT="https://openrouter.ai/api/v1"
LLM_API_KEY="sk-or-..."
```
Get your API key from [openrouter.ai/keys](https://openrouter.ai/keys). Browse all available models at [openrouter.ai/models](https://openrouter.ai/models) — prefix the model slug with `openrouter/`.
**Example models** (use with the `openrouter/` prefix):
* `openrouter/deepseek/deepseek-r1` — DeepSeek R1 via OpenRouter
* `openrouter/openai/gpt-4o-mini` — GPT-4o Mini via OpenRouter
<Warning>
**Model ids change — confirm before you copy.** OpenRouter adds and retires models continuously, and the free (`:free`) tier rotates fastest of all, so a slug that worked last month may return a model-not-found error today. Check the live catalogue rather than trusting an example:
```bash theme={null}
curl -s https://openrouter.ai/api/v1/models | jq -r '.data[].id'
```
</Warning>
**Embeddings need their own configuration.** These variables set only the LLM. If you leave `EMBEDDING_*` untouched it stays on the OpenAI defaults, so the LLM connects fine and ingestion fails later at embedding time on a missing or invalid OpenAI key. OpenRouter serves embedding models too — see [OpenRouter embeddings](/setup-configuration/embedding-providers#custom-providers) — or point `EMBEDDING_*` at OpenAI or a local provider.
</Accordion>
<Accordion title="DeepInfra">
Use DeepInfra to access open-source models via their OpenAI-compatible API.
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="deepinfra/meta-llama/Meta-Llama-3-8B-Instruct"
LLM_ENDPOINT="https://api.deepinfra.com/v1/openai"
LLM_API_KEY="<your-deepinfra-api-key>"
```
Find your model name in the [DeepInfra model catalog](https://deepinfra.com/models). The `deepinfra/` prefix tells LiteLLM to route to DeepInfra.
</Accordion>
<Accordion title="Company-Internal / Self-Hosted Endpoints">
Any internal LLM server that exposes an OpenAI-compatible REST API (e.g., a corporate vLLM deployment, internal TGI server, or private OpenRouter proxy) can be used with the `custom` provider.
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="openai/<your-internal-model-name>"
LLM_ENDPOINT="https://llm.internal.example.com/v1"
LLM_API_KEY="<internal-api-key-or-bearer-token>"
```
The model prefix you use (`openai/`, `hosted_vllm/`, etc.) determines which LiteLLM adapter handles the request. For most OpenAI-compatible servers, `openai/` works best. Set `LLM_API_KEY` to whatever bearer token your server requires (use `.` if no auth is needed).
</Accordion>
<Accordion title="vLLM">
Use vLLM for high-performance model serving with OpenAI-compatible API.
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="hosted_vllm/<your-model-name>"
LLM_ENDPOINT="https://your-vllm-endpoint/v1"
LLM_API_KEY="."
```
**Example with Gemma:**
```dotenv theme={null}
LLM_PROVIDER="custom"
LLM_MODEL="hosted_vllm/gemma-3-12b"
LLM_ENDPOINT="https://your-vllm-endpoint/v1"
LLM_API_KEY="."
```
<Warning>
**Important**: The `hosted_vllm/` prefix is required for LiteLLM to correctly route requests to your vLLM server. The model name after the prefix should match the model ID returned by your vLLM server's `/v1/models` endpoint.
</Warning>
To find the correct model name, see [their documentation](https://docs.litellm.ai/docs/providers/vllm).
</Accordion>
</Accordion>
<Accordion title="MCP Sampling (reuse the host's LLM, no API key)">
When Cognee runs **as an MCP server** (`cognee-mcp`) inside a host that grants the MCP `sampling` capability, `LLM_PROVIDER="mcp-sampling"` delegates completions to the host's own model through `sampling/createMessage`. No `LLM_API_KEY` is required.
```dotenv theme={null}
LLM_PROVIDER="mcp-sampling"
# LLM_MODEL is a preference hint only — the host chooses the actual model
LLM_MODEL="host-default"
```
<Warning>
**Preconditions**: This provider only works while Cognee is running as an MCP server inside a host process that granted the `sampling` capability to the client. If Cognee is not running under such a host — or the host did not grant sampling — the adapter fails closed with `MCPSamplingUnavailableError` before issuing any request. Treat that error as a configuration/capability issue: set `LLM_PROVIDER` to a provider with credentials, or run inside a sampling-granting host.
</Warning>
Host support varies, so check your host's MCP documentation before relying on this provider — as of early 2026 Claude Code does not yet grant sampling ([anthropics/claude-code#1785](https://github.com/anthropics/claude-code/issues/1785)).
**Completions only.** MCP sampling covers text completions — it does not provide embeddings, audio transcription, or image description. Because vector search needs embeddings, you must still configure an [embedding provider](/setup-configuration/embedding-providers) (audio transcription returns nothing and image description raises `NotImplementedError`).
**Structured output.** The MCP protocol returns free text only, so Cognee produces structured output by embedding the response model's JSON Schema in the prompt and running a bounded validate/repair loop (up to 5 attempts) before raising an error. Plain-string responses pass through unchanged.
Background tasks (such as the `cognify` tasks launched from within a request) inherit the host MCP session automatically via the SDK's per-request context, so no changes to `cognee-mcp` server code are needed.
</Accordion>
</AccordionGroup>
## Advanced Options
<Accordion title="Switching the LLM on an existing dataset">
LLM configuration is read at runtime from your environment/`.env` — it is **not** stored per dataset. Changing `LLM_PROVIDER` / `LLM_MODEL` therefore works fine on top of a dataset you have already processed; nothing about the existing data blocks the switch.
**What is not affected.** Already-processed data is left untouched: the entities and relationships in your [graph store](/setup-configuration/graph-stores) and the embeddings in your [vector store](/setup-configuration/vector-stores) are neither re-computed nor invalidated. Cognee does not re-run past extraction, and vectors depend on the [embedding model](/setup-configuration/embedding-providers), not the LLM, so [recall](/core-concepts/main-operations/recall) over existing data keeps working.
**What is affected.** The new LLM applies only to *future* work:
* Subsequent [`cognify`](/core-concepts/main-operations/legacy-operations/cognify) / [`memify`](/core-concepts/main-operations/legacy-operations/memify) runs — new data is extracted and summarized with the new model.
* Query-time reasoning during [`search`](/core-concepts/main-operations/legacy-operations/search) (e.g. `GRAPH_COMPLETION`) — answers are generated by the new model over the *existing* graph and vectors.
This means a graph can mix output from different LLMs: nodes written by the old model stay as-is, and only newly cognified data reflects the new one. If you want the whole dataset to reflect the new model's extraction quality, re-process it: run [`cognify`](/core-concepts/main-operations/legacy-operations/cognify) with `incremental_loading=False` to force a full reprocess, or empty the dataset, re-[`add`](/core-concepts/main-operations/legacy-operations/add) the source data, and run `cognify` again. Simply re-running `cognify` is not enough — it skips already-processed data by default.
<Warning>
Changing the **embedding** model is different: existing vectors were written with the old embeddings and become inconsistent with new ones. To change embeddings on an existing dataset you must re-process it — run `cognify` with `incremental_loading=False`, or delete and re-add the data — and if the new model has a different `EMBEDDING_DIMENSIONS`, remove the existing vector collections first (e.g. with [`prune`](/python-api/prune), which wipes **all** datasets). See [Embedding Providers](/setup-configuration/embedding-providers).
</Warning>
</Accordion>
<Accordion title="Per-Stage Model Routing">
By default Cognee uses a single model — the base `LLM_*` settings — for every stage of the pipeline. You can optionally route individual stages to different models or providers by setting stage-specific env var groups. Because **extraction runs once per chunk and typically dominates token spend**, it is often worth routing a cheaper or local model there while keeping a stronger model for summarization and query-time reasoning.
**Stages and their env groups**
| Env group | Stage it controls |
| --------------------- | ------------------------------------------------------------------ |
| `LLM_EXTRACTION_*` | Entity/relationship extraction during `cognify()` (runs per chunk) |
| `LLM_SUMMARIZATION_*` | Text summarization during `cognify()` |
| `LLM_QUERY_*` | Query-time completion during `search()` |
Each group accepts the same fields as the base `LLM_*` config, with the stage name in place of the leading `LLM`:
* `LLM_<STAGE>_MODEL`
* `LLM_<STAGE>_PROVIDER`
* `LLM_<STAGE>_ENDPOINT`
* `LLM_<STAGE>_API_KEY`
* `LLM_<STAGE>_API_VERSION`
**Fallback to base config**: any stage field you leave unset (empty or absent) falls back to the corresponding base `LLM_*` value, so you only set what you want to override. If you set no stage overrides at all, the effective config is identical to a single-model setup — **default single-model behavior is unchanged**.
**Only those five fields are overridden.** Everything else (temperature, `LLM_MAX_COMPLETION_TOKENS`, instructor mode, the rate-limit settings) is inherited from the base config — there are no `LLM_<STAGE>_*` equivalents for them. One default is re-derived rather than inherited: unless you set `LLM_RATE_LIMIT_REQUESTS` yourself, a stage routed to a local inference server resolves the [lower local RPM budget](#rate-limiting) of `10` instead of the `60` derived for a cloud base provider — in the example below, extraction resolves `10` while summarization and query keep `60`.
**Example** — route extraction to a local Ollama model while summarization and query keep using OpenAI:
```dotenv theme={null}
# Base config (used for any stage field left unset)
LLM_PROVIDER="openai"
LLM_MODEL="openai/gpt-5-mini"
LLM_API_KEY="sk-..."
# Extraction → local Ollama (cheap, high-volume)
LLM_EXTRACTION_MODEL="ollama_chat/llama3.1"
LLM_EXTRACTION_PROVIDER="ollama"
LLM_EXTRACTION_ENDPOINT="http://localhost:11434"
LLM_EXTRACTION_API_KEY=""
# Summarization and query keep the base model (set explicitly if you want a different one)
LLM_SUMMARIZATION_MODEL="openai/gpt-5-mini"
LLM_SUMMARIZATION_PROVIDER="openai"
LLM_QUERY_MODEL="openai/gpt-5-mini"
LLM_QUERY_PROVIDER="openai"
```
<Note>
The stage-level budget shapes that stage's *resolved config* only — the limiter that paces dispatch is built once, process-wide, from the base configuration, so if you route high-volume extraction to a local server, set `LLM_RATE_LIMIT_REQUESTS` explicitly to what that server can absorb. The re-derivation is also one-way: a *local* base provider resolves `10`, and a stage routed from there to a cloud provider keeps `10` rather than returning to `60`.
</Note>
No SDK or pipeline call signatures change when you enable per-stage routing. Each stage transparently gets its own cached client derived from its effective config, so concurrent stages can use different models safely.
</Accordion>
<Accordion title="LLM Instructor Modes">
When using the Instructor structured-output framework (opt-in via `STRUCTURED_OUTPUT_FRAMEWORK=instructor`; the default is `litellm_native`), Cognee instructs the model to return structured data in a specific way. The `LLM_INSTRUCTOR_MODE` environment variable controls which strategy is used.
Each provider has a built-in default that matches its API capabilities. Override it only when the default doesn't work for your specific model.
**Available modes:**
| Mode | Description | When to use |
| ------------------ | ------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `json_schema_mode` | Passes the full JSON Schema of the expected output in the request and enforces strict schema compliance. | OpenAI models that support the `response_format` / structured-output feature (e.g. GPT-4o). Also works well with Bedrock and with Ollama 0.5+, which can enforce a JSON Schema server-side. |
| `json_mode` | Instructs the model to return any valid JSON object. Instructor then validates and coerces it to the target schema. | Gemini, Generic/Custom endpoints, and any model that supports `response_format: json_object` but not strict schema enforcement. |
| `anthropic_tools` | Uses Anthropic's native tool-calling API to extract structured data. | Anthropic Claude models only. Leverages first-class tool-use support for reliable extraction. |
| `mistral_tools` | Uses Mistral's native tool-calling API to extract structured data. | Mistral models only. Mirrors the OpenAI function-calling interface provided by Mistral. |
| `tool_call` | Uses the generic OpenAI-style function/tool-calling API to define the schema as a callable tool. | OpenAI-compatible APIs that support function calling but not strict JSON schema output. |
| `md_json` | Asks the model to return JSON wrapped in a Markdown code block. Instructor extracts the block and validates it. | Models that reliably format code blocks but may not support `json_mode` (e.g. some self-hosted models). |
**Per-provider defaults (from source code):**
| Provider (`LLM_PROVIDER`) | Default mode |
| ------------------------------------ | ------------------ |
| `openai` (and Azure OpenAI) | `json_schema_mode` |
| `anthropic` | `anthropic_tools` |
| `gemini` | `json_mode` |
| `bedrock` | `json_schema_mode` |
| `mistral` | `mistral_tools` |
| `ollama` | `json_schema_mode` |
| `custom` (generic OpenAI-compatible) | `json_mode` |
**Example — override the mode:**
```dotenv theme={null}
LLM_INSTRUCTOR_MODE="json_schema_mode"
```
Override the default only when the model you are using requires a different mode. For example, LM Studio models typically need `json_schema_mode` even though the `custom` provider defaults to `json_mode`.
<Note>
**Ollama's default changed to `json_schema_mode`.** It was previously `json_mode`, which sent `response_format: {"type": "json_object"}` — that asks for *some* JSON and passes the Pydantic schema to the model as prompt text only, so validity is checked after the fact. With `json_schema_mode`, the schema is sent as a decoder constraint that Ollama enforces, which cuts first-attempt validation failures and the resulting `InstructorRetryException` on local models (measured 2/5 → 5/5 valid first attempts with `llama3.1:8b` and `max_retries=0`).
Ollama has supported JSON-schema structured outputs since **0.5**. If you run an older Ollama, set the previous behavior back explicitly:
```dotenv theme={null}
LLM_INSTRUCTOR_MODE="json_mode"
```
</Note>
</Accordion>
<Accordion title="Temperature and Seed">
Control the randomness of LLM responses with the `LLM_TEMPERATURE` and `LLM_SEED` environment variables.
| Variable | Default | Description |
| ----------------- | ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `LLM_TEMPERATURE` | unset — provider default applies (`0.0` on local inference servers) | Sampling temperature. `0.0` = deterministic / focused output. Higher values (e.g. `0.7`–`1.0`) produce more varied, creative responses. |
| `LLM_SEED` | unset — no seed sent | Sampling seed for reproducible outputs. Provider support varies. |
**Both are sent when you set them.** If you leave `LLM_TEMPERATURE` unset, Cognee sends no `temperature` at all and the provider's own default applies (OpenAI's is `1.0`) — it does not fall back to `0.0`. Same for `LLM_SEED`. Local inference servers are the one exception for temperature; see the note below.
```dotenv theme={null}
LLM_TEMPERATURE=0.0
LLM_SEED=42
```
**When to adjust**: setting `LLM_TEMPERATURE=0.0` is recommended for knowledge-graph extraction because it produces consistent, structured output; add `LLM_SEED` on top when you need runs to be reproducible. Raise the temperature only if you need more variety in generated text (e.g. conversational responses or creative summarisation).
Under the hood, both values are merged into `LLM_ARGS`, the provider kwargs Cognee sends with each call. A `temperature` or `seed` key given directly in `LLM_ARGS` wins over `LLM_TEMPERATURE` / `LLM_SEED`, so existing `LLM_ARGS='{"temperature": 0}'` setups keep working unchanged.
**Deterministic default on local inference servers.** When `LLM_TEMPERATURE` is not set explicitly, Ollama, llama.cpp, and LM Studio are sent `temperature: 0.0` rather than nothing — they accept the field, and leaving it out meant extraction ran at whatever the model itself defaults to (`1.0` for several Ollama models). vLLM is treated as a regular provider here and still gets no temperature when the variable is unset, as do all hosted providers. The precedence is unchanged, so both escape hatches still work: set `LLM_TEMPERATURE` to the value you want, or put a `temperature` key in `LLM_ARGS` to override the dedicated field entirely.
<Warning>
gpt-5 models — including the default `openai/gpt-5-mini` — reject any temperature other than their own default. Setting `LLM_TEMPERATURE` to something else (`0.0`, for instance) on a gpt-5 model makes every generation call fail. That restriction is why Cognee sends nothing on hosted providers unless you opt in: leave it unset, or set it only alongside a model that accepts custom temperatures. It does not apply to the local servers above, which is why they are exempt from that rule.
</Warning>
</Accordion>
<Accordion title="Max Completion Tokens">
`LLM_MAX_COMPLETION_TOKENS` sets the maximum number of tokens an LLM call may **generate** per request (passed to the provider as `max_tokens`/`max_completion_tokens`).
| Variable | Default | Description |
| --------------------------- | ------- | ------------------------------------------------------------------------ |
| `LLM_MAX_COMPLETION_TOKENS` | `16384` | Per-request output-token ceiling, and an input to automatic chunk sizing |
**Observable impact:**
* **Truncation.** If extraction or summarisation responses are larger than this ceiling, the provider stops generating mid-response. With [structured output](/setup-configuration/structured-output-backends) this surfaces as malformed/incomplete JSON or, with some local models, HTTP 500 errors. Raise the value if you see truncated output.
* **Effective value is clamped.** When the model is in [LiteLLM's](https://docs.litellm.ai/docs/providers) model registry, Cognee uses `min(model limit from LiteLLM's registry, LLM_MAX_COMPLETION_TOKENS)`. Setting it far above the registry limit has no effect — "higher" is not automatically "better".
* **Chunk size, cost and latency.** Extraction chunks are sized as `min(EMBEDDING_MAX_COMPLETION_TOKENS, LLM_MAX_COMPLETION_TOKENS // 2)` — so this value also caps how much text goes into each `cognify` chunk. A larger value means fewer, larger chunks (fewer LLM calls but more tokens per call); a smaller value means more, smaller chunks (more calls, finer-grained extraction). See [Chunkers](/core-concepts/further-concepts/chunkers) for how chunk size shapes the graph.
**Tuning guidance:** the default `16384` is a good starting point for cloud models. Lower it for **local models with a small context window** so chunks fit (see the Ollama [`num_ctx`](#ollama-local) note). Raise it only if your model supports a larger output window and you observe truncated extraction.
</Accordion>
<Accordion title="Rate Limiting">
Control client-side throttling for LLM calls to manage API usage and costs.
**The limiter starts off, but switches itself on when your provider shows signs of overload.** Cognee dispatches at full speed until a request comes back rate limited, times out, or returns HTTP 429/503/529 — then it logs one warning and paces every dispatch with the RPM budget below for a 15-minute cooldown. Set `LLM_RATE_LIMIT_ENABLED="true"` to pace from the first request instead, or `AUTO_RATE_LIMIT="false"` to stay unbounded no matter what the provider reports.
**Defaults:**
| Variable | Default | Meaning |
| ------------------------- | --------------------------------------- | --------------------------------------------------------------------------- |
| `AUTO_RATE_LIMIT` | `true` | Turn the limiter on automatically once the provider shows overload evidence |
| `LLM_RATE_LIMIT_ENABLED` | `false` | Pace every call from the start, without waiting for overload evidence |
| `LLM_RATE_LIMIT_REQUESTS` | `60` (`10` for local inference servers) | Max requests per interval |
| `LLM_RATE_LIMIT_INTERVAL` | `60` | Interval in seconds |
The cloud defaults (60 requests / 60 seconds) allow 1 request/second on average. Adjust both values to match your provider's tier limit.
**Lower default budget for local inference servers.** When `LLM_RATE_LIMIT_REQUESTS` is not set explicitly, Ollama, llama.cpp, and LM Studio get `10` requests per interval instead of `60` — they process requests near-serially, so the cloud default would still flood them. vLLM is treated as a regular provider: continuous batching absorbs concurrency like a cloud endpoint, so it keeps the `60` default. An explicitly configured `LLM_RATE_LIMIT_REQUESTS` always wins, whatever the provider.
**How auto rate limiting behaves:**
* **What counts as evidence**: a rate-limit error, a timeout (how overwhelmed local servers usually surface — they never send rate limits), or HTTP `429`, `503`, or `529`. Evidence is looked for through the whole exception cause chain, so a provider error wrapped by Instructor still counts. Slow responses on their own are not evidence.
* **Cooldown window**: the limiter stays engaged for 900 seconds (15 minutes). Further evidence inside the window extends it; once the window lapses quietly, behavior returns to whatever you configured.
* **One warning per episode**: the first piece of evidence in an episode logs a warning naming the cause and the budget being applied; evidence inside an active window extends it silently. A fresh episode after a quiet cooldown warns again.
* **Opting out**: `AUTO_RATE_LIMIT="false"` disables the automatic engagement entirely. `LLM_RATE_LIMIT_ENABLED="true"` keeps the limiter on from the start, independent of the auto behavior.
**How it works:**
* **Client-side limiter**: Cognee paces outbound LLM calls before they reach the provider
* **Moving window**: Spreads allowance across the time window for smoother throughput
* **Per-process scope**: In-memory limits don't share across multiple processes/containers
* **Retries are paced too**: adapters enter the limiter inside their retry loop, so retried attempts are throttled alongside first attempts
* **Auto-applied**: Works with all providers (OpenAI, Gemini, Anthropic, Ollama, Custom)
**Sizing guidance:**
Set `LLM_RATE_LIMIT_REQUESTS` to your provider's RPM (requests per minute) limit, and `LLM_RATE_LIMIT_INTERVAL` to `60`. To leave headroom, use \~80–90% of the advertised limit. Check your provider's dashboard for your current tier limits.
Each `cognify()` call issues multiple LLM requests (entity extraction, summarization, etc.) per document chunk — plan for several requests per chunk, not one.
**Example configurations for common provider tiers**
These examples target chat/completions-style LLM endpoints, such as OpenAI models like `gpt-4o-mini`.
<AccordionGroup>
<Accordion title="OpenAI - Tier 1">
```dotenv theme={null}
LLM_RATE_LIMIT_ENABLED="true"
LLM_RATE_LIMIT_REQUESTS="450"
LLM_RATE_LIMIT_INTERVAL="60"
```
</Accordion>
<Accordion title="OpenAI - Tier 2">
```dotenv theme={null}
LLM_RATE_LIMIT_ENABLED="true"
LLM_RATE_LIMIT_REQUESTS="4500"
LLM_RATE_LIMIT_INTERVAL="60"
```
</Accordion>
<Accordion title="Anthropic - Tier 1">
```dotenv theme={null}
LLM_RATE_LIMIT_ENABLED="true"
LLM_RATE_LIMIT_REQUESTS="45"
LLM_RATE_LIMIT_INTERVAL="60"
```
</Accordion>
<Accordion title="Google Gemini - Free Tier">
```dotenv theme={null}
LLM_RATE_LIMIT_ENABLED="true"
LLM_RATE_LIMIT_REQUESTS="13"
LLM_RATE_LIMIT_INTERVAL="60"
```
</Accordion>
<Accordion title="Conservative Default">
```dotenv theme={null}
LLM_RATE_LIMIT_ENABLED="true"
LLM_RATE_LIMIT_REQUESTS="60"
LLM_RATE_LIMIT_INTERVAL="60"
```
</Accordion>
</AccordionGroup>
<Info>
Always verify your exact tier limits in your provider's dashboard — limits vary by model, tier, and region. The examples above are approximations for common tiers and may change.
</Info>
</Accordion>
<Accordion title="Fallback Provider">
Cognee supports a primary-plus-fallback model configuration that automatically retries a failed request against a secondary provider. This is useful when your primary provider may reject certain content, and you want a fallback to handle those cases gracefully.
**When the fallback triggers**
The fallback is invoked only on **content policy violations** from the primary provider:
* `ContentFilterFinishReasonError` — the provider's output filter blocked the response
* `ContentPolicyViolationError` — the request was rejected for policy reasons
* `InstructorRetryException` containing "content management policy"
The fallback does **not** activate for network errors, rate limits, or authentication failures.
**Supported providers**
Fallback is available when `LLM_PROVIDER` is set to `openai`, `azure`, or `custom`. Other providers (Anthropic, Gemini, Mistral, Bedrock, Ollama) do not currently support the fallback chain.
**Configuration**
Set these three variables alongside your primary LLM configuration:
```dotenv theme={null}
# Primary provider
LLM_PROVIDER="openai"
LLM_MODEL="openai/gpt-4o-mini"
LLM_API_KEY="sk-..."
# Fallback provider (used only on content policy violations)
FALLBACK_MODEL="openrouter/openai/gpt-4o-mini"
FALLBACK_ENDPOINT="https://openrouter.ai/api/v1"
FALLBACK_API_KEY="or-..."
```
For `LLM_PROVIDER="custom"`, all three fallback variables (`FALLBACK_MODEL`, `FALLBACK_ENDPOINT`, `FALLBACK_API_KEY`) must be set. If any is missing, Cognee raises a `ContentPolicyFilterError` instead of falling back.
For `LLM_PROVIDER="openai"`, only `FALLBACK_MODEL` and `FALLBACK_API_KEY` are required. If set, `FALLBACK_ENDPOINT` is now forwarded to the OpenAI adapter and routes the fallback request to that base URL; if omitted, the fallback request uses the default OpenAI endpoint.
**Variable reference**
| Variable | Description |
| ------------------- | ------------------------------------------------------------------------------------------------------------ |
| `FALLBACK_MODEL` | Model identifier for the fallback provider (use LiteLLM prefix format, e.g. `openrouter/openai/gpt-4o-mini`) |
| `FALLBACK_ENDPOINT` | Base URL for the fallback provider's API (required for `custom`, optional for `openai`) |
| `FALLBACK_API_KEY` | API key for the fallback provider |
</Accordion>
<Accordion title="Retry Behavior">
Structured-output LLM calls (`acreate_structured_output`, used internally for entity extraction, summarization, and other graph-building steps) are wrapped in a shared retry policy that retries transient failures with exponential backoff.
**How long a failing call persists**
A call is allowed to give up only once **both** of these floors are met:
| Floor | Value | Meaning |
| -------------------- | ------- | ------------------------------------------------------------------- |
| Minimum attempts | `2` | At least two attempts are made before failing. |
| Minimum elapsed time | `~240s` | At least \~240 seconds of wall-clock time must pass before failing. |
Because both conditions must hold, a call against an unstable or rate-limited provider can keep retrying for up to a few minutes before it finally errors out. Backoff between attempts is exponential with jitter (starting around 8 seconds, capped near 128 seconds).
<Info>
These floors are internal defaults shared across the OpenAI, Azure OpenAI, Anthropic, Gemini, Mistral, Ollama, Llama.cpp, Custom, and BAML structured-output paths. They are **not** environment-configurable.
</Info>
Bedrock uses a separate retry path: its structured-output adapter relies on the Bedrock rate-limit/sleep retry wrapper and Instructor's Bedrock retry setting instead of the shared `~240s` retry floor.
Some errors are treated as non-transient and are **not** retried — they fail immediately: authentication errors, model-not-found errors, cancellations, payment/budget exhaustion, and quota/billing exhaustion. That includes the shared retry paths for OpenAI, Azure OpenAI, Anthropic, Gemini, Mistral, Ollama, Llama.cpp, Custom, and BAML, so interrupted jobs and worker shutdowns stop promptly rather than waiting out the backoff window. Bedrock uses a separate retry path, but cancellations still unwind immediately there as well.
**Quota / billing exhaustion is terminal.** When a provider reports that its quota or billing limit is exhausted, retrying cannot help, so the call fails fast instead of spinning through the retry window. The raw provider error is converted at the single `acreate_structured_output` choke point into an actionable `LLMQuotaExceededError` (provider- and framework-agnostic). The following provider wordings are classified as terminal:
| Pattern | Provider |
| --------------------------- | ----------------------------------------------- |
| `insufficient_quota` | OpenAI / Azure OpenAI (billing quota exhausted) |
| `quota_exceeded` | Generic provider quota-exhaustion code |
| `billing hard limit` | OpenAI (monthly hard limit reached) |
| `credit balance is too low` | Anthropic (prepaid credits exhausted) |
| `out of credits` | Generic |
<Warning>
Transient **per-minute rate limits** stay retryable. The patterns above are deliberately narrow: the bare phrase "exceeded your current quota" is intentionally **not** matched, because Gemini free tier uses it for recoverable `RESOURCE_EXHAUSTED` limits (OpenAI's terminal case is still caught via `insufficient_quota`). Monitoring and alerting should treat `LLMQuotaExceededError` as a terminal condition and respond by checking the provider billing/quota dashboard, raising the limit, or switching credentials — not by retrying.
</Warning>
**Budget exhaustion is terminal — including when it arrives wrapped.** A configured spend cap cannot clear inside a retry window, so a budget rejection fails the call immediately with `LLMPaymentRequiredError` (HTTP `402`) instead of spinning through the `~240s` floor. Cognee classifies it from four signals, checked against the raised exception **and every error in its `__cause__` chain**:
| Signal | Where it comes from |
| -------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| HTTP `402` | Any provider returning Payment Required directly |
| `litellm.BudgetExceededError` | litellm's own budget manager, when litellm is used as a library with a configured budget |
| HTTP `429` whose JSON body has `error.type: "budget_exceeded"` | LiteLLM proxy, reachable only while the response body is still unread |
| LiteLLM's budget wording in the error message | The wrapper-proof fallback (see below) |
The message check is what catches the common LiteLLM-proxy case. A proxy spend-cap rejection reaches Cognee as `InstructorRetryException` → `RetryError` → a client-side `RateLimitError`, and by then every structured signal is gone: the outer exception carries no `status_code` and no `response`, the inner error is a different class from `litellm.BudgetExceededError`, and its response body has already been consumed. Such a rejection used to be classified as ordinary throttling and retried through all three nested retry layers (Cognee's tenacity policy, Instructor's own retries, and the litellm/OpenAI client's); it is now terminal instead. The three LiteLLM sentence shapes matched cover every budget scope — virtual key, team member, team, project, organization, tag, internal user, end user, and per-model caps:
```text theme={null}
Budget has been exceeded! [<scope>] Current cost: <x>, Max budget: <y>
ExceededBudget: [End ]User=<id> over budget. Spend=<x>, Budget=<y>
LiteLLM {Virtual Key|End User}: <id>, exceeded budget for model=<model>
```
**The `402` carries the provider's own budget sentence.** Only the matched sentence is extracted — not the whole wrapper message, which embeds the model's partial completion — and identifiers within it are replaced with `<redacted>`. Masking is by label: `Virtual Key:`, `End User:`, `User=`, `Team=`, `Project=`, `Organization=`, `Tag=`, and `key_alias:` are covered, so the third sentence shape above is fully masked. The `Key=<alias> (sk-…-hint)` segment LiteLLM writes into the first shape is **not** — that alias and LiteLLM's own truncated key hint pass through into the `402` body. Spend and budget figures and the model name are kept, since they are the caller's own and the actionable part. The resulting message is `LLM budget exhausted: <sentence>`; when no sentence can be extracted, the default `LLM provider requires payment or token budget is exhausted.` is used instead. See [402 Payment Required](/api-reference/introduction) for the API response body.
Like the quota check above, the classification runs at the `acreate_structured_output` choke point, so it is provider- and framework-agnostic and also covers the plain-text (`response_model=str`) path that bypasses the adapters' own error handling. **Every litellm-instructor adapter also classifies at its own exit points** — OpenAI, Azure OpenAI (including its managed-identity path), Anthropic, Gemini, Mistral, Ollama, Bedrock, Llama.cpp, and Custom — in whichever clause each one uses to re-raise a provider failure: the handler that catches `InstructorRetryException` on OpenAI, Azure OpenAI, Gemini, Bedrock and Custom, and the catch-all exit on Anthropic, Mistral, Ollama and Llama.cpp. That matters twice over: a budget rejection whose embedded partial completion happens to mention a content policy is not misrouted to the content-policy fallback, and a caller reaching an adapter directly rather than through `LLMGateway` gets the typed `402` instead of a raw `InstructorRetryException` whose message it would have to match itself. The adapter-level conversion extracts the same provider sentence described above, so the `402` detail does not depend on which path converted it. A budget error is never downgraded to the quota error above: it stays the more specific `402`.
**On OpenAI and Azure OpenAI, a configured fallback model is attempted before the conversion.** A budget rejection from the primary model is deliberately *not* converted while a fallback is still configured — the fallback carries a different key, so a per-key spend cap is exactly the case it exists for. Conversion happens once the fallback caps out too, or immediately when no fallback is configured. **Gemini and `custom` convert without trying the fallback.** Gemini's fallback exists only for the content-policy path, so an ordinary budget rejection — wrapped in `InstructorRetryException`, with no content-policy wording — is converted on the spot and never reaches it; only a rejection whose text *also* reads as a policy violation gets the failover. The `custom` adapter classifies at the top of its `InstructorRetryException` clause, ahead of the fallback branch, so a budget rejection there never reaches the fallback either. Bedrock classifies ahead of its content-policy branch in the same way but has no fallback to preserve, and the remaining adapters (Anthropic, Mistral, Ollama, Llama.cpp) have no fallback path either, so they convert at their single exit point.
**Transcription and image transcription stop retrying too, but do not convert.** `create_transcript` and `transcribe_image` on the OpenAI, Azure OpenAI, Anthropic, Gemini, Mistral, Ollama, and Custom adapters share the retry condition used by structured-output calls, so budget and quota exhaustion are terminal there as well and no longer burn retries on a call that cannot succeed. The classification stops there: these helpers surface the raw provider error (typically a `RateLimitError`) rather than the typed `402`, because they are not routed through the `LLMGateway` choke point and do not classify at their own exit points. Alert on `LLMPaymentRequiredError` for structured-output and embedding calls; a budget rejection on the transcription path still has to be recognized by its message. These helpers also keep their own cap of `3` attempts; the `~240s` floor above does not apply to them.
The **embedding** path is classified separately, by the same detector, so a spend cap surfaces as the same `402` from either path — see [Errors that fail immediately](/setup-configuration/embedding-providers#errors-that-fail-immediately) for which engines are covered and how the failure differs there.
<Warning>
Matching is whole-sentence and deliberately narrow, because `str(InstructorRetryException)` concatenates the model's partial completions — loose matching would turn any document that merely mentions an exceeded budget into a hard `402`. Two consequences are worth knowing:
* **After upgrading LiteLLM or the proxy**, watch for `429` retry storms. If an upstream wording or the proxy's response shape changes, a rejection falls through the classifier and is treated as an ordinary retryable `429` again.
* **Text that quotes a budget rejection verbatim** — a pasted gateway log, a runbook excerpt — inside content being processed can trip the check and surface as a `402`. This is an accepted trade-off: it costs one already-failing call a wrong status code, whereas a missed rejection restores the retry storm.
</Warning>
**Operational note**: when a provider is flaky, expect higher tail latency and additional API calls (and therefore cost) while retries play out. This persistent retry improves resilience for transient failures but does not mask genuine misconfiguration such as a bad API key.
</Accordion>
<Accordion title="Custom Endpoints & Corporate Proxies">
`LLM_ENDPOINT` overrides the base URL Cognee uses to reach the LLM. Use it to point at an Azure deployment, a local server, an OpenAI-compatible proxy, or a company-internal gateway. For routing all outbound traffic through a corporate HTTP proxy without rewriting the endpoint, use the standard `HTTPS_PROXY` / `HTTP_PROXY` environment variables.
**Per-provider `LLM_ENDPOINT` semantics**
| `LLM_PROVIDER` | How `LLM_ENDPOINT` is used | Required? |
| -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- |
| `openai` | Passed to LiteLLM as `api_base`. Omit to use OpenAI's default (`https://api.openai.com/v1`). Set to point at a compatible gateway or proxy. | Optional |
| `azure` | Azure resource endpoint (e.g., `https://<resource>.openai.azure.com`). The deployment is selected by `LLM_MODEL`. | Required |
| `gemini` | Passed to LiteLLM as `api_base`. Omit to use the provider's default. | Optional |
| `mistral` | `LLM_ENDPOINT` is currently not used for generation; Cognee uses the default Mistral provider endpoint. | Not applicable |
| `ollama` | Base URL of your Ollama server. On the default `litellm_native` backend this is the bare host (`http://localhost:11434`), which LiteLLM extends with Ollama's native paths; on `STRUCTURED_OUTPUT_FRAMEWORK="instructor"` it is the OpenAI-compatible base (`http://localhost:11434/v1`). | Required |
| `custom` | Base URL of your OpenAI-compatible server (vLLM, OpenRouter, LM Studio, internal gateway, etc.). | Required |
| `llama_cpp` | Required only in server mode (URL of the `llama-cpp-python` server). Ignored in local in-process mode. | Server mode only |
| `anthropic` | Not read. Anthropic's SDK has its own internal base URL. To route through a proxy, use `HTTPS_PROXY`. | Not applicable |
| `bedrock` | Not read. Use `AWS_BEDROCK_RUNTIME_ENDPOINT` to override the Bedrock endpoint. | Not applicable |
**Routing through a corporate HTTP/HTTPS proxy**
Cognee's LLM transport is built on the `openai`, `anthropic`, `httpx`, and `litellm` Python clients, all of which honor the standard proxy environment variables. Set them in your shell or `.env` before starting Cognee:
```dotenv theme={null}
HTTPS_PROXY="http://proxy.corp.example.com:8080"
HTTP_PROXY="http://proxy.corp.example.com:8080"
# Optional: hosts that should bypass the proxy
NO_PROXY="localhost,127.0.0.1,.internal.example.com"
```
This is the right approach when the LLM provider's public URL is correct but your network blocks direct egress. No Cognee config change is needed — outbound LLM, embedding, and HTTP loader calls all pick up these variables automatically.
**Troubleshooting "not connected / cannot reach LLM"**
* **`LLM_ENDPOINT` typos** — values are stripped of surrounding quotes, but a missing scheme (`http://` / `https://`) or trailing path segment will surface as a connection error. For OpenAI-compatible endpoints, the URL must end in `/v1` (or whatever the server exposes).
* **Preflight timeout** — Cognee runs a 30s connection test at startup. If your proxy adds latency or your local model is slow to warm up, set `COGNEE_SKIP_CONNECTION_TEST=true` to skip it.
* **Provider mismatch** — if `LLM_ENDPOINT` points at a non-OpenAI server but `LLM_PROVIDER="openai"`, Cognee will hit the wrong route. For OpenAI-compatible third parties, use `LLM_PROVIDER="custom"` with the correct LiteLLM model prefix (see [Custom Providers](#custom-providers) above).
* **TLS interception** — if your corporate proxy uses its own CA, set `SSL_CERT_FILE` or `REQUESTS_CA_BUNDLE` to the CA bundle path so Python's HTTP clients trust the proxy certificate.
</Accordion>
## Notes
* If `EMBEDDING_API_KEY` is not set, Cognee falls back to `LLM_API_KEY` for embeddings
* Rate limiting helps manage API usage and costs
* Structured output frameworks ensure consistent data extraction from LLM responses
<Columns>
<Card title="Embedding Providers" icon="layers" href="/setup-configuration/embedding-providers">
Configure embedding providers for semantic search
</Card>
<Card title="Overview" icon="settings" href="/setup-configuration/overview">
Return to setup configuration overview
</Card>
<Card title="Relational Databases" icon="database" href="/setup-configuration/relational-databases">
Set up SQLite or Postgres for metadata storage
</Card>
</Columns>
# Logging
Source: https://docs.cognee.ai/setup-configuration/logging
Control Cognee logging and troubleshoot issues with logs.
Cognee writes structured logs to two destinations simultaneously: the console and a log file. A new log file is created each time Cognee starts. The file captures everything from `DEBUG` upward. The console shows only the level you configure.
## How Logging Works in Cognee
When you import or start Cognee, `setup_logging()` is called automatically. The first call in a process configures two handlers:
* **Console handler** — writes colored output to `stderr`, filtered by `LOG_LEVEL` (default: `INFO`)
* **File handler** — writes plain text to a timestamped `.log` file, always at `DEBUG` level
One log file is created per startup and shared across all sub-processes. Cognee keeps the 10 most recent log files and deletes older ones automatically. The active file is also capped by size and rotated — see [Size-Based Rotation](#size-based-rotation) below.
### Logging Is Configured Once Per Process
`setup_logging()` is idempotent. Cognee calls it from four places during a normal server start — importing `cognee`, importing `cognee.api.client`, importing the vector embeddings utilities, and server startup — but only the first call does any configuration work. That call installs the console and file handlers, sets up the `sys.excepthook`, opens the log file, and emits the startup messages. Every later call, including one you write yourself, returns a structlog logger and changes nothing: the handlers, log file, and excepthook from the first call stay in place, and the startup banner is not repeated. Because the first call wins, arguments you pass to a later `setup_logging(log_level=...)` call are ignored — see [Setting the Log Level](#setting-the-log-level) for the supported way to control verbosity.
The guard is per interpreter. A sub-process starts fresh, configures its own handlers, and appends to the same log file, which is how one file ends up holding output from all of them.
<Accordion title="Default log file locations">
The default `logs/` directory is created next to the `cognee` package directory, one level above it. The exact path depends on how Cognee is installed.
**Installed via pip** (e.g. `pip install cognee`):
```
~/.venv/lib/python3.x/site-packages/logs/
```
**Running from source** (cloned repository):
```
<repo-root>/logs/
```
**Using the MCP server:**
The MCP server uses the same logging setup. The log directory is the same as above, depending on your environment. When a background task is launched, the MCP server returns the full path to the active log file in its response.
<Warning>
When Cognee is installed via pip or used as an MCP server, log files end up inside your virtual environment's `site-packages/` directory. Set `COGNEE_LOGS_DIR` to write logs to a predictable location instead.
</Warning>
</Accordion>
### Size-Based Rotation
The file handler also caps how large a single startup's log can grow. Before writing each record it checks the current file size, and once the file has reached `COGNEE_LOG_MAX_BYTES` (default 50 MB) it rolls over: the active file is renamed to `<name>.log.1`, any existing backups shift up by one, and logging continues in a fresh file under the original name. At most `COGNEE_LOG_BACKUP_COUNT` backups (default 5) are kept, so on the defaults one startup's logs occupy roughly 300 MB at the ceiling — one active 50 MB file plus five 50 MB backups.
Rotation and the keep-10 cleanup are independent mechanisms. Rotation bounds the size of *one* startup's log; the keep-10 cleanup prunes *old startups'* files. The cleanup only matches files ending in `.log`, so numbered rotation backups (`.log.1`, `.log.2`, …) are not counted by it and are not removed when their originating startup's `.log` file is pruned. If you run with a large `COGNEE_LOG_BACKUP_COUNT` and restart often, prune the log directory yourself.
<Warning>
Rotation is a recent fix. In earlier releases `COGNEE_LOG_MAX_BYTES` and `COGNEE_LOG_BACKUP_COUNT` were accepted and handed to the file handler, but the handler never acted on them — a startup's log file grew without bound. If you set those variables before and saw no rotation, upgrading is what makes them take effect, and disk usage will now behave as the values describe.
</Warning>
## Controlling Logging
### Environment Variables
* `COGNEE_LOGS_DIR` — Absolute path to a custom log directory. Cognee creates it if it does not exist. Must be an absolute path — relative paths cause a startup error. Default: `logs/` next to the `cognee` package.
* `LOG_LEVEL` — Console log verbosity. One of `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`. Default: `INFO`. Does not affect the log file, which always captures `DEBUG` and above.
* `COGNEE_LOG_MAX_BYTES` — Size in bytes at which the active log file is rotated. Default: `52428800` (50 MB). Set it to `0` to turn size-based rotation off and let a startup's log file grow without a cap.
* `COGNEE_LOG_BACKUP_COUNT` — How many rotated backups to keep alongside the active file. Default: `5`. Backups are named `<name>.log.1` through `<name>.log.<count>`, oldest last.
### Setting the Log Level
`LOG_LEVEL` is read when logging is configured, which happens on the first `setup_logging()` call — and `import cognee` makes that call for you. So the level must be in place *before* the import:
```python theme={null}
import os
os.environ["LOG_LEVEL"] = "ERROR"
import cognee # logging is configured here, at ERROR
```
Setting it in the `.env` file works too, and takes precedence: Cognee loads `.env` with `override=True` at import time, just before configuring logging. If `LOG_LEVEL` is in your `.env`, change it there rather than in the shell or in code.
<Warning>
Calling `setup_logging(log_level=...)` yourself after `import cognee` has no effect on verbosity. Logging is already configured at that point, so the call returns a logger and leaves the console level untouched. Older examples that adjusted the level this way need to set `LOG_LEVEL` before importing Cognee instead.
</Warning>
### Configuration Examples
```dotenv theme={null}
# Write logs to a predictable absolute path
COGNEE_LOGS_DIR="/home/user/my-project/logs"
# Reduce console noise in production
LOG_LEVEL="ERROR"
# Show all output during development
LOG_LEVEL="DEBUG"
# Roll the log file every 10 MB and keep 3 backups (~40 MB per startup)
COGNEE_LOG_MAX_BYTES="10485760"
COGNEE_LOG_BACKUP_COUNT="3"
```
### Fallback Behavior
If Cognee cannot write to `COGNEE_LOGS_DIR`, it falls back to `/tmp/cognee_logs`. If that also fails, file logging is skipped silently and only console output is produced.
## Log Files and Their Content
### File Format
Log files use plain text, one entry per line. Timestamps are in UTC:
```
2025-02-14T15:32:47.123456 [WARNING ] From version 0.5.0 onwards... [cognee.shared.logging_utils]
2025-02-14T15:32:47.234567 [INFO ] Log file created at: /path/to/logs/2025-02-14_15-32-47.log [cognee.shared.logging_utils]
2025-02-14T15:32:47.345678 [INFO ] Logging initialized python_version=3.11.0 cognee_version=0.x.x os_info=... database_path=... graph_database_name=... vector_config=... relational_config=... [cognee.shared.logging_utils]
2025-02-14T15:32:47.456789 [INFO ] Database storage: /path/to/.cognee_system/databases [cognee.shared.logging_utils]
```
### Console Format
The console uses structlog's colored renderer, written to `stderr`. Level colors: `DEBUG` blue, `INFO` green, `WARNING` yellow, `ERROR` and `CRITICAL` red.
Logged exception tracebacks are rendered without per-frame local variables (`show_locals=False`). The traceback itself is still shown, but the values of locals in each frame are omitted. This keeps memory bounded when exceptions are logged in the retrieval/search path, where frame locals can hold graph objects carrying embedding vectors and deep node/edge references — rendering those recursively could spike memory to multiple GB and OOM-kill the process.
Cognee also installs a safe `sys.excepthook`, so uncaught non-`KeyboardInterrupt` exceptions are logged through structlog before Python's default traceback is printed. If rich traceback rendering fails, Cognee falls back to plain traceback output.
### Exception Details
When an exception is logged, the structured log event includes the exception type in `exception_type` alongside the exception message. This makes error entries easier to interpret because they show both what failed and the exception class that raised it, for example `ValueError`.
### What to Expect at Startup
Cognee initializes logging once per process, and that initialization logs the following four messages in order:
1. **INFO** — The full path to the log file created for this session.
2. **WARNING** — The `Cognee 1.0 changes` banner: the new remember/recall/forget/improve API, session memory on by default, multi-user access control on by default, and agents auto-verified on registration. This is expected and not an error.
3. **INFO** — `Logging initialized`, with system metadata attached as structured fields: Python version, structlog version, Cognee version, OS, and the database storage path.
4. **INFO** — The database storage path again, as a plain-text line.
If the log file path message is missing from the console, file logging may have failed. Check whether the log directory is writable or set `COGNEE_LOGS_DIR` to a path you control.
The active graph, vector, and relational providers are not part of these messages. Cognee defers that configuration logging to the first actual pipeline call, because importing those configs pulls in heavy dependencies.
### File Naming
Each log file is named after the startup timestamp:
```
YYYY-MM-DD_HH-MM-SS.log
```
Example: `2025-02-14_15-32-47.log`
When that file hits the size cap it is rotated, so a long-running startup leaves numbered backups next to it — `2025-02-14_15-32-47.log.1` is the most recent rotated segment, `.log.2` the one before it, and so on.
## Troubleshooting
<AccordionGroup>
<Accordion title="Can't find the log file">
Run this snippet after importing cognee to get the exact path:
```python theme={null}
from cognee.shared.logging_utils import get_log_file_location
print(get_log_file_location())
```
</Accordion>
<Accordion title="No log file is created">
Cognee falls back to console-only logging if the log directory is not writable. This is common in managed environments where `site-packages/` is read-only.
Set `COGNEE_LOGS_DIR` to an absolute path you own:
```dotenv theme={null}
COGNEE_LOGS_DIR="/home/user/my-project/logs"
```
</Accordion>
<Accordion title="A repeated startup banner">
Cognee used to redo the whole configuration on every `setup_logging()` call: each call replaced the root logger's handlers, reopened the log file, and re-emitted the startup messages — a server start could log the "Cognee 1.0 changes" warning four times.
Logging is now configured exactly once per process, so the banner is logged once. If you see it more than once, more than one process is writing to the same file: sub-processes share the parent's log file by design, and each one configures its own handlers.
</Accordion>
<Accordion title="LOG_LEVEL is ignored">
Logging is configured on the first `setup_logging()` call, which `import cognee` triggers. If you set `LOG_LEVEL` after that import — or call `setup_logging(log_level=...)` yourself — the level is already fixed and your change does nothing. Set `LOG_LEVEL` in your `.env` file or in the environment before importing Cognee. See [Setting the Log Level](#setting-the-log-level).
</Accordion>
</AccordionGroup>
<Columns>
<Card title="Overview" icon="settings" href="/setup-configuration/overview">
Return to setup configuration overview
</Card>
<Card title="Relational Databases" icon="database" href="/setup-configuration/relational-databases">
Configure metadata and state storage
</Card>
<Card title="Installation" icon="rocket" href="/getting-started/installation">
Install Cognee and set up your environment
</Card>
</Columns>
# Setup Configuration
Source: https://docs.cognee.ai/setup-configuration/overview
Configure Cognee to use your preferred LLM, embedding engine, and storage backends
Configure Cognee to use your preferred LLM, embedding engine, relational database, vector store, and graph store via environment variables in a local `.env` file.
This section provides beginner-friendly guides for setting up different backends, with detailed technical information available in expandable sections.
## What You Can Configure
Cognee uses a flexible architecture that lets you choose the best tools for your needs. We recommend starting with the defaults to get familiar with Cognee, then customizing each component as needed:
* **[LLM Providers](./llm-providers)** — Choose from OpenAI, Azure OpenAI, Google Gemini, Anthropic, Ollama, or custom providers (like vLLM) for text generation and reasoning tasks
* **[Structured Output Backends](./structured-output-backends)** — Configure LiteLLM + Instructor or BAML for reliable data extraction from LLM responses
* **[Embedding Providers](./embedding-providers)** — Select from OpenAI, Azure OpenAI, Google Gemini, Mistral, Ollama, Fastembed, or custom embedding services to create vector representations for semantic search
* **[Relational Databases](./relational-databases)** — Use SQLite for local development or Postgres for production to store metadata, documents, and system state
* **[Vector Stores](./vector-stores)** — Store embeddings in built-in backends such as LanceDB, PGVector, ChromaDB, or Neptune Analytics, or use community adapters such as Qdrant, Redis, and FalkorDB
* **[Graph Stores](./graph-stores)** — Build knowledge graphs with Kuzu, Kuzu-remote, Neo4j, Neptune, Neptune Analytics, or Memgraph to manage relationships and reasoning
* **[Dataset Separation & Access Control](./permissions)** — Configure dataset-level permissions and isolation
* **[Sessions & Caching](../core-concepts/sessions-and-caching)** — Enable conversational memory with Redis or filesystem cache adapters
<Info>
Want to run Cognee without a cloud API key? See the [Local Setup guide](/guides/local-setup) for step-by-step instructions using Ollama and Fastembed.
</Info>
## Data Flow And Verification
For the default local setup, user content stays on the local machine:
* Raw source files live under `DATA_ROOT_DIRECTORY`
* Cognee-managed state lives under `<SYSTEM_ROOT_DIRECTORY>/databases`
* The default `SYSTEM_ROOT_DIRECTORY` is `.cognee_system` when the variable is unset
That means the local defaults do not require a Cognee-managed backend. If you switch to remote LLM, embedding, database, object-storage, or cloud-connection settings, Cognee will contact the endpoints you configured instead.
For deployment-specific connection checks, see the local setup guide and the cloud SDK connection guide.
## How `.env` Is Loaded
Cognee loads `.env` values when the Python package is imported. Keep the file in your project root, or in the directory from which you run Python, so it is available before Cognee creates its runtime configuration objects.
<Note>
Cognee loads `.env` with overwrite behavior enabled. If the same key is set in both your shell and `.env`, the value from `.env` is the one Cognee uses after import.
</Note>
<AccordionGroup>
<Accordion title="Configuration Precedence">
| Priority | Source | When to use |
| -------- | ------------------------------------------------------------------------------ | --------------------------------------------------------------- |
| 1 | Runtime configuration methods, such as `cognee.config.set("llm_model", "...")` | Temporary changes inside one Python process |
| 2 | Values in `.env` | Persistent local development configuration |
| 3 | Shell, deployment, or `os.environ` variables | CI, containers, hosted deployments, secrets managers, and tests |
| 4 | Cognee defaults | Local defaults when nothing is configured |
Runtime configuration methods update Cognee's in-memory config objects and stay active for the duration of the current Python process, or until you call another setter. They do not write changes back to `.env` unless you pass `persist=True` to `cognee.config.set(...)` — which is what `cognee-cli config set` does. See the [Python API config reference](/python-api/config) for details.
</Accordion>
<Accordion title="Using os.environ">
Setting `os.environ["KEY"] = "value"` changes the current Python process environment. Use it for Cognee only before importing Cognee, and mainly for process or deployment settings:
```python theme={null}
import os
os.environ["LOG_LEVEL"] = "ERROR"
os.environ["COGNEE_LOG_FILE"] = "false"
import cognee
```
After `import cognee`, do not rely on `os.environ` to change Cognee behavior. Some code paths read environment variables lazily, but others read them during import, application startup, or cached config creation. Post-import `os.environ` changes are therefore inconsistent.
If the same key also exists in `.env`, Cognee's import-time `.env` loading overwrites the earlier `os.environ` value:
```python theme={null}
import os
os.environ["LLM_MODEL"] = "openai/gpt-4o-mini"
import cognee
```
```dotenv theme={null}
# .env
LLM_MODEL="openai/gpt-5-mini"
```
In this case, Cognee uses `openai/gpt-5-mini` after import. Use `os.environ` before importing Cognee only for keys that are not also defined in `.env`.
Use `.env`, shell variables, deployment variables, or pre-import `os.environ` for settings such as:
| Area | Environment variables |
| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Auth and access control | `ENABLE_BACKEND_ACCESS_CONTROL`, `REQUIRE_AUTHENTICATION`, `FASTAPI_USERS_JWT_SECRET`, `JWT_LIFETIME_SECONDS`, `HASH_API_KEY`, `ALLOW_HTTP_REQUESTS`, `ALLOW_CYPHER_QUERY`, `ACCEPT_LOCAL_FILE_PATH`, `COGNEE_ALLOWED_LOCAL_FILE_ROOTS` |
| Logging | `LOG_LEVEL`, `COGNEE_LOG_FILE`, `COGNEE_LOGS_DIR`, `COGNEE_LOG_MAX_BYTES`, `COGNEE_LOG_BACKUP_COUNT`, `COGNEE_LOG_SEARCH_HISTORY` |
| Cache and sessions | `CACHING`, `AUTO_FEEDBACK`, `SESSION_SEARCH_MODE`, `CACHE_BACKEND`, `CACHE_HOST`, `CACHE_PORT`, `CACHE_USERNAME`, `CACHE_PASSWORD`, `CACHE_SSL`, `CACHE_SSL_CERT_REQS`, `SESSION_TTL_SECONDS`, `PERSONALIZATION_ENABLED`, `PERSONALIZATION_INFLUENCE`, `PREFERENCE_ALPHA`, `PREFERENCE_BETA` |
| Recall warm-up | `RECALL_WARMUP_SHORTCIRCUIT`, `RECALL_WARMUP_THRESHOLD`, `RECALL_WARMUP_CACHE_TTL` |
| Provenance and edge evidence | `COGNEE_PROVENANCE_MODE`, `PROVENANCE_TRACKING`, `EDGE_EVIDENCE_ENABLED`, `EDGE_EVIDENCE_FLUSH_THRESHOLD` |
| Storage and cloud credentials | `STORAGE_BACKEND`, `STORAGE_BUCKET_NAME`, `COGNEE_REPOS_DIR`, `AWS_REGION`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_ENDPOINT_URL`, `COGNEE_SERVICE_URL`, `COGNEE_API_KEY` (legacy fallbacks: `COGNEE_CLOUD_API_URL`, `COGNEE_CLOUD_AUTH_TOKEN`) |
| Web/API/telemetry | `HTTP_API_HOST`, `HTTP_API_PORT`, `CORS_ALLOWED_ORIGINS`, `TAVILY_API_KEY`, `KEENABLE_API_KEY`, `KEENABLE_BASE_URL`, `KEENABLE_LIVE_FETCH`, `WEB_SCRAPER_TIMEOUT`, `TELEMETRY_DISABLED`, `TELEMETRY_ORIGIN`, `ENV` |
If you need to change supported runtime settings after import, use `cognee.config.set(...)` because it updates Cognee's in-memory runtime config directly:
```python theme={null}
import cognee
cognee.config.set("llm_model", "openai/gpt-5-mini")
```
</Accordion>
<Accordion title="What Can Be Overwritten at Runtime">
Use `cognee.config.set(...)` for runtime-safe Cognee settings: values that can be changed inside the current Python process without reinitializing the whole application. This mainly covers LLMs, embeddings, graph databases, vector databases, chunking, model overrides, and data/system root directories. For the full method list and the exact internal key names accepted by bulk setters, see the [Python API config reference](/python-api/config).
```python theme={null}
import cognee
cognee.config.set("llm_model", "openai/gpt-5-mini")
cognee.config.set("embedding_provider", "fastembed")
cognee.config.set("vector_db_provider", "lancedb")
cognee.config.set("vector_db_url", "./.cognee_system/databases/cognee.lancedb")
```
`cognee.config.set(key, value)` supports these generic keys:
| Area | Supported keys |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| LLM | `llm_provider`, `llm_model`, `llm_api_key`, `llm_endpoint` |
| Embeddings | `embedding_provider`, `embedding_model`, `embedding_dimensions`, `embedding_endpoint`, `embedding_api_key`, `embedding_api_version`, `embedding_max_completion_tokens`, `embedding_batch_size`, `huggingface_tokenizer` |
| Graph database | `graph_database_provider`, `graph_database_subprocess_enabled`, `kuzu_num_threads`, `kuzu_buffer_pool_size`, `kuzu_max_db_size` |
| Vector database | `vector_db_provider`, `vector_db_subprocess_enabled`, `vector_db_url`, `vector_db_key` |
| Chunking | `chunk_size`, `chunk_overlap`, `chunk_strategy`, `chunk_engine` |
| Models | `classification_model`, `summarization_model`, `graph_model` |
| Storage paths | `system_root_directory`, `data_root_directory` |
`cognee.config.set(...)` can replace `.env` or `os.environ` only for the supported runtime config keys above. It does not replace process-level environment variables.
Keep these in `.env`, shell/deployment variables, or pre-import `os.environ`: `ENABLE_BACKEND_ACCESS_CONTROL`, `REQUIRE_AUTHENTICATION`, `CACHING`, `CACHE_BACKEND`, `LOG_LEVEL`, `COGNEE_LOG_FILE`, `STORAGE_BACKEND`, `TAVILY_API_KEY`, `KEENABLE_API_KEY`, `TELEMETRY_DISABLED`, `TELEMETRY_ORIGIN`, `HTTP_API_HOST`, `HTTP_API_PORT`, and cloud or AWS credentials.
<Warning>
`cognee.config.set(key, value)` is not a free-form setter. Unsupported keys raise an error instead of silently creating new settings.
</Warning>
</Accordion>
<Accordion title="When to Restart">
Restart your Python process, server, notebook kernel, or container after editing `.env` if Cognee has already been imported. Runtime setters are useful for short-lived overrides, but `.env` changes are safest when applied before import.
When changing storage backends, database providers, embedding dimensions, or other settings that affect persisted data, review the pruning warning in the Configuration Workflow section before running ingestion again.
</Accordion>
<Accordion title="system_root_directory vs data_root_directory">
Cognee uses two top-level storage roots. The short version: `SYSTEM_ROOT_DIRECTORY` is for Cognee-managed databases and internal state, while `DATA_ROOT_DIRECTORY` is for source files and filesystem-backed cache/session data.
| Directory | Env var | Default | What it stores |
| --------------- | ----------------------- | ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **System root** | `SYSTEM_ROOT_DIRECTORY` | `.cognee_system` | Cognee's **internal** files: the `<SYSTEM_ROOT_DIRECTORY>/databases/` subfolder holding the relational store, vector store, and graph store. This is data Cognee generates and manages for you. |
| **Data root** | `DATA_ROOT_DIRECTORY` | `.data_storage` | **Your** data: the raw source files that `add()` copies into Cognee (text, PDFs, CSVs, images, audio, etc.), plus filesystem session/cache data. |
A third root sits outside this pair: `COGNEE_REPOS_DIR` (default `~/.cognee/repos`) holds the shallow clones of git repositories Cognee indexes as code graphs — from a repository URL passed to [`add()`](/python-api/add#code-repository-urls) or from `remember(..., content_type="code")`. It is always a **local** directory, even when `STORAGE_BACKEND` is S3, because `git` writes a working tree.
These defaults are not automatically placed in your project root. They resolve relative to the installed `cognee` package, which often means a path inside your virtual environment. For portable local projects, pin both values in `.env`:
```bash theme={null}
SYSTEM_ROOT_DIRECTORY="/abs/path/to/project/.cognee_system"
DATA_ROOT_DIRECTORY="/abs/path/to/project/.data_storage"
```
<Note>
`.env` values must be **absolute paths** (or `s3://...` URLs). Relative `.env` values raise a configuration error during config loading. Runtime setters such as `cognee.config.system_root_directory(...)` and `cognee.config.data_root_directory(...)` should also receive absolute paths. Setting `system_root_directory` cascades to the default relational, vector, and graph database paths under `<SYSTEM_ROOT_DIRECTORY>/databases`. See the [Python API config reference](/python-api/config) for the setter signatures.
</Note>
For path-mismatch troubleshooting, see [Graph Stores](/setup-configuration/graph-stores#troubleshooting-no-nodes-found). For backup and migration details, see [Backing up local data](#backing-up-local-data) and [Migrating to Another Instance](/how-to-guides/cognee-sdk/deployment#migrating-to-another-instance).
</Accordion>
<Accordion title="Default backends and when connections are established">
With a plain `pip install cognee` (no extras), Cognee uses three bundled, file-based backends. None of them require a separate server, and no extra dependencies are needed:
| Role | Default provider | Where data lives |
| --------------------------------------- | ------------------------------------------------------------------------------- | -------------------------------------------------------- |
| Relational (metadata, documents, state) | [SQLite](./relational-databases) (`DB_PROVIDER=sqlite`, database `cognee_db`) | `<SYSTEM_ROOT_DIRECTORY>/databases/cognee_db` |
| Vector (embeddings, semantic search) | [LanceDB](./vector-stores) (`VECTOR_DB_PROVIDER=lancedb`) | `<SYSTEM_ROOT_DIRECTORY>/databases/cognee.lancedb` |
| Graph (entities, relationships) | [Ladybug (Kuzu-compatible)](./graph-stores) (`GRAPH_DATABASE_PROVIDER=ladybug`) | `<SYSTEM_ROOT_DIRECTORY>/databases/cognee_graph_ladybug` |
Extras such as `cognee[postgres]`, `cognee[neo4j]`, `cognee[chromadb]`, or `cognee[neptune]` are only required when you switch a backend to one of those providers. The defaults above work without any of them.
**Connections are not opened at import.** `import cognee` only loads `.env` and builds in-memory configuration objects — it does not connect to any database. Each backend engine is created lazily, the first time an operation actually needs it (for example during `add()`, `cognify()`, or `search()`), and is then cached and reused for the rest of the process. For the file-based defaults, the database files are created automatically under `SYSTEM_ROOT_DIRECTORY` on first use, so there is no startup connection step to configure.
</Accordion>
<Accordion title="Backing up local data">
With the default file-based backends, all of Cognee's persistent state lives in two directories on disk, so a backup is just a copy of those two trees:
| Directory | Default | Contents |
| ----------------------- | ---------------- | ------------------------------------------------------------------------------------------------- |
| `SYSTEM_ROOT_DIRECTORY` | `.cognee_system` | The relational, vector, and graph stores under `<SYSTEM_ROOT_DIRECTORY>/databases/` |
| `DATA_ROOT_DIRECTORY` | `.data_storage` | The raw ingested source files that `add()` copies into Cognee, plus filesystem session/cache data |
Back up **both** directories together so the graph, vectors, relational metadata, and the source files they reference stay consistent with each other.
**Stop writes before copying.** SQLite, LanceDB, and the Kuzu-compatible graph store are embedded databases that write directly to these files. Copying them while an `add()`, `cognify()`, `memify()`, or `delete()` operation is in progress can capture a half-written, corrupt snapshot. For a safe, consistent backup, make sure no Cognee process is actively ingesting or mutating data — stop your Cognee service (or wait for all pipelines to finish), then copy the directories:
```bash theme={null}
# With the Cognee process stopped / idle
cp -r .cognee_system .cognee_system.backup
cp -r .data_storage .data_storage.backup
```
Operations that only read from the stores are safe, but default graph-completion searches with session caching can write session/cache data. To get a fully consistent backup, keep Cognee idle while copying. To restore, stop Cognee and replace the two directories with your backed-up copies.
If you have moved a backend off the file-based defaults — for example to [Postgres](./relational-databases), [PGVector](./vector-stores), or [Neo4j](./graph-stores) — back up that external database using its own tooling instead; only the file-based stores live under these directories.
</Accordion>
</AccordionGroup>
## Environment Variable Quick Reference
The tables below list the most commonly used configuration variables. For full details on each group, follow the links to the dedicated guides.
<Note>
Most configuration keys (LLM, embedding, database, etc.) are used without a `COGNEE_` prefix, but several Cognee-specific controls do use one, including logging, tracing, and cloud connection variables. The cloud-sync credentials `COGNEE_SERVICE_URL` and `COGNEE_API_KEY` are canonical; the older `COGNEE_CLOUD_API_URL` and `COGNEE_CLOUD_AUTH_TOKEN` names are still accepted as legacy fallbacks.
</Note>
<AccordionGroup>
<Accordion title="LLM">
Every LLM setting Cognee reads, with its default. Follow the links for the full explanation of each group in [LLM Providers](/setup-configuration/llm-providers).
**Model and credentials**
| Variable | Default | Description |
| ----------------------------------------------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `LLM_PROVIDER` | `openai` | Provider: `openai`, `azure`, `gemini`, `anthropic`, `ollama`, `mistral`, `bedrock`, `llama_cpp`, `custom`, `mcp-sampling`. Optional — when unset it is [inferred](/setup-configuration/llm-providers#provider-inference) from the `LLM_MODEL` prefix |
| `LLM_MODEL` | `openai/gpt-5-mini` | Model in `provider/model-name` format |
| `LLM_API_KEY` | — | API key for the LLM provider |
| `LLM_ENDPOINT` | — | Custom endpoint URL, required for Azure, Ollama, and `custom`. [Semantics differ per provider](/setup-configuration/llm-providers) |
| `LLM_API_VERSION` | — | API version (required for Azure) |
| `LLM_AZURE_USE_MANAGED_IDENTITY` | `false` | Authenticate Azure OpenAI with a managed identity instead of `LLM_API_KEY` |
| `LLM_EXTRACTION_*`, `LLM_SUMMARIZATION_*`, `LLM_QUERY_*` | base `LLM_*` values | [Per-stage overrides](/setup-configuration/llm-providers#per-stage-model-routing) for `MODEL`, `PROVIDER`, `ENDPOINT`, `API_KEY`, and `API_VERSION` |
| `FALLBACK_MODEL` / `FALLBACK_ENDPOINT` / `FALLBACK_API_KEY` | — | Secondary provider retried on content-policy violations, for `LLM_PROVIDER` `openai`, `azure`, or `custom` only. See [Fallback Provider](/setup-configuration/llm-providers#fallback-provider) |
**Generation**
| Variable | Default | Description |
| ----------------------------------------------------------------------------------------------- | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `LLM_TEMPERATURE` | unset | Response temperature (0.0–2.0), sent with every LLM call when you set it — unset means the provider's own default applies, except on local inference servers (Ollama, llama.cpp, LM Studio), where an unset value sends `0.0`. gpt-5 models, including the default `openai/gpt-5-mini`, reject any other value |
| `LLM_SEED` | unset | Sampling seed for reproducible outputs, sent only when set (provider support varies) |
| `LLM_ANSWER_STREAMING` | `false` | Stream answer tokens from recall's final answer call to clients consuming the [REST API's SSE response](/guides/deploy-rest-api-server#http-api-examples). Inert unless a client is reading the stream — the returned value is identical either way |
| `LLM_MAX_COMPLETION_TOKENS` | `16384` | Output-token ceiling per request (sent as `max_tokens`/`max_completion_tokens`), clamped to the model's own limit when LiteLLM knows it, and an input to [chunk sizing](/setup-configuration/llm-providers#max-completion-tokens) |
| `LLM_ARGS` | — | JSON object of extra provider kwargs merged into every call. Keys given here win over `LLM_TEMPERATURE` / `LLM_SEED` |
| `OLLAMA_NUM_CTX` | `2048` | Context window requested from Ollama on the `instructor` [structured-output path](/setup-configuration/structured-output-backends), and the token ceiling used for chunk sizing when `LLM_PROVIDER=ollama` and the model is unknown to LiteLLM |
| `LLAMA_CPP_MODEL_PATH` / `LLAMA_CPP_N_CTX` / `LLAMA_CPP_N_GPU_LAYERS` / `LLAMA_CPP_CHAT_FORMAT` | — / `2048` / `0` / `chatml` | [llama.cpp local mode](/setup-configuration/llm-providers#llama-cpp-local): GGUF path, context size, GPU layers to offload, chat template |
| `TRANSCRIPTION_MODEL` | `whisper-1` | Model used to transcribe audio files during ingestion |
| `STRUCTURED_OUTPUT_FRAMEWORK` | `litellm_native` | How structured output is produced: `litellm_native`, `instructor`, or `baml`. See [Structured Output Backends](/setup-configuration/structured-output-backends) |
| `LLM_INSTRUCTOR_MODE` | provider-specific | Structured-output strategy on the `instructor` framework — see [LLM Instructor Modes](/setup-configuration/llm-providers#llm-instructor-modes) for the per-provider defaults |
**Throughput, timeouts and retries**
| Variable | Default | Description |
| ----------------------------- | --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
| `AUTO_RATE_LIMIT` | `true` | Engage the client-side limiter automatically once the provider shows overload evidence (429/503/529, rate-limit errors, timeouts) |
| `LLM_RATE_LIMIT_ENABLED` | `false` | Pace every call from the first request instead of waiting for overload evidence |
| `LLM_RATE_LIMIT_REQUESTS` | `60` (`10` for local inference servers) | Max requests per interval. See [Rate Limiting](/setup-configuration/llm-providers#rate-limiting) |
| `LLM_RATE_LIMIT_INTERVAL` | `60` | Interval in seconds |
| `COGNEE_SKIP_CONNECTION_TEST` | `false` | Skip the startup LLM/embedding preflight, which times out after 30 seconds |
<Note>
Per-request LLM retry behavior is fixed in code rather than set by environment variable: structured-output calls retry with exponential backoff and jitter until at least 2 attempts and \~240 seconds have been spent — see [Retry Behavior](/setup-configuration/llm-providers#retry-behavior). The 30-second figure above is the startup connection test only.
</Note>
</Accordion>
<Accordion title="Embeddings">
Every embedding setting Cognee reads, with its default. Follow the links for the full explanation of each group in [Embedding Providers](/setup-configuration/embedding-providers).
| Variable | Default | Description |
| -------------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `EMBEDDING_PROVIDER` | `openai` | Provider: `openai`, `ollama`, `fastembed`, `gemini`, `mistral`, `bedrock`, `openai_compatible`, `custom`. These are [not the same values](/setup-configuration/embedding-providers#valid-embedding_provider-values-and-endpoint-url-forms) `LLM_PROVIDER` accepts |
| `EMBEDDING_MODEL` | `openai/text-embedding-3-large` | Model in `provider/model-name` format |
| `EMBEDDING_DIMENSIONS` | auto-derived (`3072` fallback) | Vector dimension size (must match your vector store). Cognee looks the model up in the LiteLLM/fastembed registries and falls back to `3072` with a warning — [set it explicitly](/setup-configuration/embedding-providers#how-do-i-determine-embedding_dimensions) for any non-default model |
| `EMBEDDING_API_KEY` | — | API key (falls back to `LLM_API_KEY` if unset; leaving the whole embedding section at its defaults while `LLM_PROVIDER` is non-OpenAI is rejected up front — see [Provider consistency preflight](#provider-consistency-preflight)) |
| `EMBEDDING_ENDPOINT` | — | Custom endpoint URL (required for Ollama, etc.). `EMBEDDING_API_BASE` is accepted as an alias — the LiteLLM/OpenAI name — and `EMBEDDING_ENDPOINT` wins when both are set |
| `EMBEDDING_API_VERSION` | — | API version (for Azure OpenAI) |
| `EMBEDDING_MAX_COMPLETION_TOKENS` | `8191` | Maximum **input** tokens per embedded text; a [chunk-sizing hint](/setup-configuration/embedding-providers#max-completion-tokens), not an enforced cap |
| `EMBEDDING_BATCH_SIZE` | `36` | Chunks per embedding API call. Lower it for local servers — see [Batch Size](/setup-configuration/embedding-providers#batch-size) |
| `EMBEDDING_MAX_CONCURRENT_DATA_POINTS` | `150` | Data points allowed in flight during indexing; concurrent batches are `max(1, this // EMBEDDING_BATCH_SIZE)` |
| `EMBEDDING_INPUT_TYPE` | — | Value sent as the non-standard `input_type` field in the embedding request. Only needed by providers that require it (e.g. NVIDIA NIM's `nv-embed` family); ignored elsewhere |
| `EMBEDDING_RATE_LIMIT_ENABLED` | `false` | Opt-in client-side throttling of embedding calls. See [Rate Limiting](/setup-configuration/embedding-providers#rate-limiting) |
| `EMBEDDING_RATE_LIMIT_REQUESTS` | `60` | Max embedding requests per interval (one request = one batch) |
| `EMBEDDING_RATE_LIMIT_INTERVAL` | `60` | Interval in seconds |
| `HUGGINGFACE_TOKENIZER` | — | HuggingFace Hub model ID that overrides the tokenizer Cognee uses for token counting when the embedding model is not itself a HuggingFace repo. Commonly used with Ollama embeddings (for example, `nomic-ai/nomic-embed-text-v1.5`). |
| `TOKENIZERS_PARALLELISM` | — | Optional environment variable used by Hugging Face tokenizers. If Cognee loads a Hugging Face tokenizer, setting this to `false` can suppress the "tokenizers parallelism" warning that may appear in forked or multi-process environments. |
<Note>
Embedding timeouts and retries are fixed in code rather than set by environment variable: a 300-second per-attempt timeout and a 128-second total retry window with exponential backoff. See [Timeout and Retry Behavior](/setup-configuration/embedding-providers#timeout-and-retry-behavior).
</Note>
</Accordion>
<Accordion title="Databases">
| Variable | Default | Description |
| ----------------------------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DB_PROVIDER` | `sqlite` | Relational DB: `sqlite`, `postgres` |
| `DB_HOST` / `DB_PORT` / `DB_USERNAME` / `DB_PASSWORD` | — | Postgres connection details |
| `POOL_ARGS` | — | JSON SQLAlchemy connection-pool args for the relational engine (Postgres **and** SQLite) and the Postgres graph adapter **(demo)**, and — when `VECTOR_POOL_ARGS` is unset — for per-dataset PGVector engines. On SQLite only the pool keys (`pool_size`, `max_overflow`, `pool_recycle`, `pool_timeout`, `pool_pre_ping`, `poolclass`) are applied, and leaving it unset keeps the `NullPool` default — see [SQLite connection pooling](/setup-configuration/relational-databases#advanced-options). Must be a JSON object. |
| `VECTOR_DB_PROVIDER` | `lancedb` | Vector store provider. Built-in options include `lancedb`, `pgvector`, `chromadb`, and `neptune_analytics`; community adapters add providers such as `qdrant`, `redis`, and `falkordb`. |
| `VECTOR_DB_URL` | — | Vector store connection URL |
| `VECTOR_POOL_ARGS` | — | JSON connection-pool args for per-dataset PGVector engines in multi-user mode. Must be a JSON object; invalid JSON raises a configuration error. When unset, PGVector per-dataset engines inherit the relational `POOL_ARGS`, falling back to `pool_size=2` and `max_overflow=20` under backend access control when neither is set. |
| `GRAPH_DATABASE_PROVIDER` | `ladybug` | Graph store: `ladybug`, `ladybug-remote`, `kuzu`, `kuzu-remote`, `neo4j`, `neptune` |
| `GRAPH_DATABASE_URL` | — | Graph store connection URL |
| `GRAPH_DATABASE_USERNAME` / `GRAPH_DATABASE_PASSWORD` | — | Graph store credentials |
</Accordion>
<Accordion title="Cognify pipeline">
Which implementation fills the extract-and-summarize step of `cognify()`, plus the optional tasks it appends to its default pipeline. The optional tasks are off by default — with those flags unset the task list is exactly the standard pipeline. The cognify config is cached for the lifetime of the process, so set these before the first `cognify()` call.
| Variable | Default | Description |
| ------------------------------------ | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `GRAPH_EXTRACTOR` | `llm` | `llm` keeps the LLM extraction path unchanged. `gliner` builds the graph **and** the chunk summaries with the local GLiNER2 model instead — no LLM call for either; embeddings still run. Requires the `gliner` extra (`uv pip install "cognee[gliner]"`), and downloads \~800 MB of model weights on first use. The per-call `cognify(extractor=...)` / `remember(extractor=...)` argument wins over this setting. See [LLM-free extraction with GLiNER](/python-api/cognify#llm-free-extraction-with-gliner). |
| `CONTRADICTION_DETECTION` | `false` | Set `true` to compare the facts each run touched against the facts already stored around them and record every conflict as a `contradicts` edge. See [Contradiction detection](/python-api/cognify#contradiction-detection). |
| `CONTRADICTION_CONFIDENCE_THRESHOLD` | `0.5` | Minimum LLM confidence for a fact pair to be flagged. |
| `CONTRADICTION_MAX_FACTS` | `500` | Cap on the facts sent to the LLM in a single check. |
| `PROVENANCE_TRACKING` | `false` | Set `true` to append the opt-in provenance-ledger task, writing document → chunk → entity → relationship lineage into the append-only `provenance_entries` table in the relational database. Adds relational writes on every `cognify()` run, so plan for disk and backup sizing. The table ships as Alembic revision `b8c1d3e5f7a9` — run migrations before enabling it on an existing deployment. See [Provenance ledger](/python-api/cognify#provenance-ledger). |
</Accordion>
<Accordion title="Folder presort">
Controls whether plain folder inputs are scanned and split into datasets before they are ingested. Off by default — with the flag unset, `remember(folder)` ingests the folder exactly as it always has.
| Variable | Default | Description |
| ------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `PRESORT_FOLDERS_ENABLED` | `false` | Set `true` to automatically presort a local folder handed to `remember()` and apply the resulting report in the same call. Only plain local directories targeting `main_dataset` with no session, dataset id, or `content_type` qualify; code-project directories keep the code-graph route, and `s3://`, `http(s)://`, `file://` inputs and sessions connected to a remote instance through `serve()` are excluded. See [Folder presort](/python-api/remember#folder-presort). |
Presort can always be triggered per call without the flag, via `remember(folder, dry_run="presort")` or `cognee-cli remember <folder> --presort`. Which folders it may read is governed separately by `COGNEE_ALLOWED_LOCAL_FILE_ROOTS` — see [Presort scan roots](/setup-configuration/security#presort-scan-roots).
</Accordion>
<Accordion title="Edge evidence">
Three independently configured systems share the word *provenance*, and these settings control only the third:
| System | Flag | What it records |
| ----------------- | ---------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| Node stamping | `COGNEE_PROVENANCE_MODE` (`lightweight` by default, or `deep` / `disabled`) | Which pipeline run and task wrote each graph node, as `source_*` fields on the node itself |
| Audit ledger | `PROVENANCE_TRACKING` (`false` by default; see [Provenance ledger](/python-api/cognify#provenance-ledger)) | A hash-chained lineage history in `provenance_entries`, for compliance rather than retrieval |
| **Edge evidence** | **`EDGE_EVIDENCE_ENABLED`** | **Which document chunk supports each graph edge, in `provenance_edge_evidence`, read back as citations at search time** |
| Variable | Default | Description |
| ------------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `EDGE_EVIDENCE_ENABLED` | `true` | **On by default.** Records one row per (edge, source chunk) pair as `cognify()` writes the graph, so a graph edge can be traced back to the chunk it was extracted from. Only edges extracted from document chunks are captured — edges Cognee derives some other way carry no evidence. |
| `EDGE_EVIDENCE_FLUSH_THRESHOLD` | `10000` | Pending evidence rows for one data item before an early bulk flush. Below the threshold, a data item's rows are written once, when it finishes. Values under `100` are rejected. |
The table ships as Alembic revision `f3a7b9c1d2e4`, so a deployment upgrading from `1.5.3` or earlier must [run migrations](/python-api/run-migrations) before the capture path has anywhere to write.
Evidence is read back only when you ask for references: with [`include_references=True`](/python-api/search#parameters) on a dataset-scoped search, a completion whose context contained graph edges resolves those edges through the sidecar to their source chunks — capped at 5 chunks per edge and 50 in total. Up to 50 resolved chunks land in the result's structured evidence list (the `evidence` field of a `search()` result, `metadata.evidence` on a `recall()` result), and the first five are also rendered as an `Evidence:` block in the answer text. The lookup is a single indexed relational query, and a failure to resolve is logged and skipped rather than failing the search.
Ingestion only appends to the table, and graph deletion never consults it. An observation counts as active only when it carries no pipeline-run id or its run reached a completed terminal state, so a failed or rolled-back run leaves rows behind that are simply never read. Rows are removed when their document is deleted (`forget()`, `delete_data`, `delete_dataset`) or its memory is dropped with `forget(memory_only=True)`; the lookup also skips rows whose document no longer exists.
</Accordion>
<Accordion title="Recall warm-up">
Before `recall()` runs graph retrieval, it classifies the target datasets as warm, never built, or build failed — a single indexed relational query, with no graph or vector engine spin-up. Datasets whose graph has never been built return an instant `memory_warming_up` marker, and datasets whose last graph-writing run errored return a `build_failed` marker carrying the root cause (`error_class`, `error_message`), instead of a graph search plus an LLM call that could only come back empty. See [Warming-up marker](/python-api/recall#warming-up-marker) for the response shape and its exceptions.
| Variable | Default | Description |
| ---------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `RECALL_WARMUP_SHORTCIRCUIT` | `true` | Kill switch. Set `false` to skip the check entirely and have recall's graph lane behave exactly as it did before. |
| `RECALL_WARMUP_THRESHOLD` | `1` | Minimum datapoint count treated as warm. The default means only never-processed datasets short-circuit; the probe is binary, so raising it cannot make a populated dataset read as cold. |
| `RECALL_WARMUP_CACHE_TTL` | `60` | Seconds a *warm* verdict is cached in-process, so repeated recalls against a populated dataset skip even the probe. Cold verdicts are never cached, so the first recall after `cognify()` sees the new data immediately. |
All three are also settable at runtime with `cognee.config.set(...)` (`recall_warmup_shortcircuit`, `recall_warmup_threshold`, `recall_warmup_cache_ttl`). The check fails open: a probe or configuration error — including a malformed `RECALL_WARMUP_*` value — falls through to a normal search rather than failing the recall.
</Accordion>
<Accordion title="Storage & Logging">
| Variable | Default | Description |
| ----------------------- | ---------------- | ---------------------------------------------------------------------------------------------------------------- |
| `STORAGE_BACKEND` | `local` | Storage backend: `local`, `s3` |
| `DATA_ROOT_DIRECTORY` | `.data_storage` | Root directory for data files |
| `SYSTEM_ROOT_DIRECTORY` | `.cognee_system` | Root directory for system files |
| `COGNEE_LOGS_DIR` | `{package}/logs` | Override the logs directory path |
| `LOG_LEVEL` | `INFO` | Logging level: `DEBUG`, `INFO`, `WARNING`, `ERROR` |
| `TELEMETRY_DISABLED` | `false` | Set `true` to disable anonymous telemetry |
| `TELEMETRY_ORIGIN` | `sdk` | Value stamped on every telemetry event as `telemetry_origin`, so events can be segmented by where they come from |
</Accordion>
<Accordion title="Sessions & Caching">
Cognee uses a cache backend to store session history (Q\&A turns) so that searches with the same `session_id` can include prior interactions as conversational context. See [Sessions and Caching](/core-concepts/sessions-and-caching) for the full guide.
| Variable | Default | Description |
| --------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `CACHING` | `true` | Enable session caching. Set to `false` to run searches without conversational memory. |
| `AUTO_FEEDBACK` | `true` | Enable automatic session-context guidance for session-capable completion searches. Set to `false` to disable the extra turn-analysis LLM call and use plain conversation history. |
| `SESSION_SEARCH_MODE` | `concurrent` | How a session turn executes: `concurrent` runs the turn analysis alongside retrieval and answer generation (retrieving with both the raw question and a deterministic rewrite), `sequential` runs the analysis before a single retrieval so its outputs affect the same turn. See [Sessions and Caching](/core-concepts/sessions-and-caching#session-context-guidance-auto-feedback). |
| `CACHE_BACKEND` | `sqlite` | Cache backend: `sqlite` (local SQL `cache.db` file), `postgres` (external SQL database), `redis` (external in-memory store), `fs` (local disk via diskcache), or `tapes` (local cache plus Tapes mirroring). |
| `CACHE_DB_URL` | — | Optional SQLAlchemy async URL for the SQL cache backends. When unset, `sqlite` uses a `cache.db` file next to the relational SQLite database and `postgres` falls back to the relational `DB_*` settings. |
| `CACHE_HOST` | `localhost` | Redis hostname (used when `CACHE_BACKEND=redis`). |
| `CACHE_PORT` | `6379` | Redis port. |
| `CACHE_USERNAME` | — | Optional Redis username. |
| `CACHE_PASSWORD` | — | Optional Redis password. |
| `CACHE_SSL` | `false` | Connect to Redis over TLS (used when `CACHE_BACKEND=redis`). Enable for managed Redis with in-transit encryption (AWS ElastiCache, GCP Memorystore, Azure Cache for Redis). |
| `CACHE_SSL_CERT_REQS` | `required` | TLS certificate verification when `CACHE_SSL` is enabled: `required`, `optional`, or `none`. |
| `SESSION_TTL_SECONDS` | `604800` | Expiry for cached session entries (7 days). Set to `0` to disable expiry. |
The default `sqlite` needs no setup and suits local development. Use `postgres` or `redis` when you want the session cache in an external service — for example in production, so the cache outlives the Cognee host. Use `tapes` when you want filesystem-backed sessions plus mirroring of new Q\&A turns to a running Tapes ingest service.
</Accordion>
<Accordion title="Personalization">
Per-user preference personalization is **off by default**. When it is on, ratings collected in a session are folded by `improve(session_ids=...)` into weighted `prefers` edges for that user, and those weights nudge ranking in graph, hybrid, and RAG retrieval. See [User Preferences](/core-concepts/further-concepts/user-preferences) for the full behavior.
| Variable | Default | Description |
| --------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `PERSONALIZATION_ENABLED` | `false` | Master switch. Set `true` to let per-user preferences be written by `improve()` and applied at retrieval time. With it off, nothing is written and no ranking changes. |
| `PERSONALIZATION_INFLUENCE` | `0.3` | The most personalization may move a ranking score, as a fraction: `0.3` means at most 30%. Must be in `[0, 1]`. |
| `PREFERENCE_ALPHA` | `0.3` | How far one rating pulls a `prefers` edge weight toward its target. Must be in `(0, 1]`. |
| `PREFERENCE_BETA` | `0.02` | How much an untouched `prefers` edge weight fades toward neutral per conversation turn. Must be in `[0, 1)`. |
These values are validated when configuration is loaded: an out-of-range `PERSONALIZATION_INFLUENCE`, `PREFERENCE_ALPHA`, or `PREFERENCE_BETA` raises a `ValueError` at startup rather than failing silently later.
</Accordion>
<Accordion title="Debug Mode">
To enable verbose logging in a self-hosted Cognee instance, set `LOG_LEVEL` in your `.env`:
```dotenv theme={null}
LOG_LEVEL="DEBUG"
```
Verbose logging covers pipeline execution, LLM calls, database queries, and graph operations—useful when troubleshooting data processing or provider configuration.
</Accordion>
</AccordionGroup>
## Docker Environment Variables
Use the same variable names as in your `.env`; pass them with `docker run -e` or load them from a file with `--env-file`.
<AccordionGroup>
<Accordion title="Examples">
```bash theme={null}
docker run \
-e LLM_PROVIDER=ollama \
-e LLM_MODEL=ollama/llama3.2 \
-e LLM_ENDPOINT=http://host.docker.internal:11434 \
-e EMBEDDING_PROVIDER=ollama \
-e EMBEDDING_MODEL=nomic-embed-text:latest \
-e EMBEDDING_ENDPOINT=http://host.docker.internal:11434/api/embed \
-e EMBEDDING_DIMENSIONS=768 \
-e HUGGINGFACE_TOKENIZER=nomic-ai/nomic-embed-text-v1.5 \
cognee/cognee:main
```
Or using an env file:
```bash theme={null}
docker run --env-file .env cognee/cognee:main
```
</Accordion>
</AccordionGroup>
## Observability & Telemetry
Cognee includes built-in telemetry to help you monitor and debug your knowledge graph operations. You can control telemetry behavior with environment variables:
* **`TELEMETRY_DISABLED`** (boolean, optional): Set to `true` to disable all telemetry collection (default: `false`)
* **`TELEMETRY_ORIGIN`** (string, optional): Labels where the events come from. Every event carries its value as a `telemetry_origin` property, so telemetry can be segmented by origin — the Cognee-managed cloud sets `TELEMETRY_ORIGIN=cloud`, and anything else reports the default `sdk`. It is read from the environment on each event, so it cannot be set through `cognee.config.set(...)` (default: `sdk`)
When telemetry is enabled, Cognee automatically collects:
* Search query performance metrics
* Processing pipeline execution times
* Error rates and debugging information
* System resource usage
<Info>
Telemetry data helps improve Cognee's performance and reliability. It's collected anonymously and doesn't include your actual data content.
</Info>
## Configuration Workflow
1. Install Cognee with all optional dependencies:
* **Local setup**: `uv sync --all-extras`
* **Library**: `pip install "cognee[all]"`
2. Create a `.env` file in your project root (if you haven't already) — see [Installation](/getting-started/installation) for details
3. Choose your preferred providers and follow the configuration instructions from the guides below
If you would rather start from a working combination than assemble one, [Store Configurations](/guides/store-configurations) gives a complete `.env` block per supported relational + vector + graph stack, with the extras and Docker commands each one needs.
<Warning>
**Configuration Changes**: If you've already run Cognee with default settings and are now changing your configuration (e.g., switching from SQLite to Postgres, or changing vector stores), you should call pruning operations before the next cognification to ensure data consistency.
</Warning>
### Provider consistency preflight
`add()` and `remember()` run a zero-network provider-consistency check at the top of the call, **before any ingestion work happens**, and raise `ProviderConfigMismatchError` on a mismatch instead of letting it surface minutes later as an opaque authentication error mid-cognify. The check looks for two mismatches:
* **Only the LLM is configured** — `LLM_PROVIDER` is set to something other than `openai` while the embedding settings are still untouched (`EMBEDDING_PROVIDER=openai`, `EMBEDDING_MODEL=openai/text-embedding-3-large`, no `EMBEDDING_API_KEY`, no `EMBEDDING_ENDPOINT`). Your provider key — or, with `LLM_PROVIDER=custom`, no key at all — would otherwise be sent to the OpenAI embeddings endpoint. Fix it by setting `EMBEDDING_PROVIDER`, `EMBEDDING_MODEL`, and `EMBEDDING_API_KEY` (or `EMBEDDING_ENDPOINT` for a local embedder such as Ollama), or by providing an OpenAI `EMBEDDING_API_KEY` to keep the default embedder.
* **Only embeddings are configured** — embedding settings are customised but `LLM_API_KEY` is unset for a provider that requires one. `bedrock` and `llama_cpp` are exempt (they authenticate with AWS credentials or run locally), as is `azure` with managed identity. This half of the check is skipped when the work the call is about to do runs no LLM task at all — an `add()` of plain text, or a graph build on the [`gliner` extractor](/python-api/cognify#llm-free-extraction-with-gliner) — since a missing LLM key is then not a problem. The embedding half always runs.
The check compares settings only — it never verifies that the credentials work. Run [`cognee-cli doctor`](/cognee-cli/overview#diagnose-your-setup) to see the same report before you ingest anything, and `cognee-cli doctor --probe` to also test live LLM and embedding calls.
To skip the check, set `COGNEE_SKIP_PREFLIGHT`, `COGNEE_SKIP_CONNECTION_TEST`, or `MOCK_EMBEDDING` to `true`, `1`, or `yes` (case-insensitive), so CI, mocked, and offline runs with deliberately partial provider config are unaffected. A passing check is cached for the lifetime of the process; a failing one is not, so a config fix applied in-process is picked up on the next `add()` / `remember()` call without a restart.
<Warning>
**LLM/Embedding Configuration**: If you configure only the LLM or only embeddings, the other side silently defaults to OpenAI. Configure both, or keep a working OpenAI `EMBEDDING_API_KEY` / `LLM_API_KEY` for the side you leave at its defaults.
</Warning>
<Columns>
<Card title="LLM Providers" icon="brain" href="/setup-configuration/llm-providers">
Configure OpenAI, Azure, Gemini, Anthropic, Ollama, or custom LLM providers (like vLLM)
</Card>
<Card title="Structured Output Backends" icon="code" href="/setup-configuration/structured-output-backends">
Configure LiteLLM + Instructor or BAML for reliable data extraction
</Card>
<Card title="Embedding Providers" icon="layers" href="/setup-configuration/embedding-providers">
Set up OpenAI, Mistral, Ollama, Fastembed, or custom embedding services
</Card>
</Columns>
<Columns>
<Card title="Relational Databases" icon="database" href="/setup-configuration/relational-databases">
Choose between SQLite for local development or Postgres for production
</Card>
<Card title="Vector Stores" icon="database" href="/setup-configuration/vector-stores">
Configure LanceDB, PGVector, Qdrant, Redis, ChromaDB, FalkorDB, or Neptune Analytics
</Card>
<Card title="Graph Stores" icon="network" href="/setup-configuration/graph-stores">
Set up Kuzu, Neo4j, or Neptune for knowledge graph storage
</Card>
</Columns>
# Permissions Setup
Source: https://docs.cognee.ai/setup-configuration/permissions
Configure Cognee's permission system and access control
Enable Cognee's permission system for data isolation and access control. For detailed concepts, see [Cognee Permissions System](/core-concepts/multi-user-mode/permissions-system/overview).
## Enable Permission System
Set the environment variable to enable access control:
```dotenv theme={null}
ENABLE_BACKEND_ACCESS_CONTROL=true # this is set to true by default
REQUIRE_AUTHENTICATION=true
```
### Auto-enable behavior
When `ENABLE_BACKEND_ACCESS_CONTROL` is not explicitly set, Cognee automatically enables multi-user mode if the configured graph and vector setup passes the runtime compatibility checks.
At a high level, that means both of the following must be true:
* The configured graph dataset handler is supported and matches the selected graph provider.
* The configured vector dataset handler is supported and matches the selected vector provider.
Set `ENABLE_BACKEND_ACCESS_CONTROL=false` to keep single-user mode regardless of which databases are configured.
For the supported backend combinations and handler details, see [Security & Privacy](/setup-configuration/security) and [Dataset Database Handlers](/core-concepts/multi-user-mode/dataset-database-handlers/dataset-database-handlers-what-are-they).
## Dataset Queue
When backend access control is enabled, Cognee can limit the number of dataset-level operations that run concurrently. This caps overall concurrent dataset work and can reduce contention when many tasks access datasets at the same time.
The queue is **enabled by default** and is automatically disabled when `ENABLE_BACKEND_ACCESS_CONTROL=false`.
```dotenv theme={null}
# Maximum concurrent dataset slots (default: 128, matching DATABASE_MAX_LRU_CACHE_SIZE)
DATASET_QUEUE_MAX_CONCURRENT=10
```
When the limit is reached, additional dataset operations wait until a slot is freed. Enabling the queue trades some parallel throughput for consistency on concurrent workloads with many datasets.
### Engine cache pinning
Cognee caches per-dataset graph and vector engines in a fixed-size LRU cache sized by `DATABASE_MAX_LRU_CACHE_SIZE`. While a dataset holds a queue slot, its graph and vector engines are **pinned** and are not evicted by capacity pressure — even if they are the least-recently-used entries. This prevents a dataset that an admitted pipeline is still using (for example, one idling on an LLM call mid-`cognify`) from having its engine closed underneath it, which was a source of DB lock and cache-overflow errors.
As a result, when every cached entry is pinned, the engine cache can briefly exceed `DATABASE_MAX_LRU_CACHE_SIZE`. This overflow is bounded by `DATASET_QUEUE_MAX_CONCURRENT` (no more datasets than the queue admits can be pinned at once).
When the queue is disabled (`DATASET_QUEUE_ENABLED=false`, or `ENABLE_BACKEND_ACCESS_CONTROL=false`), nothing is pinned and eviction falls back to plain least-recently-used recency.
### Subprocess engine teardown coordination
When subprocess-mode databases are in use (`graph_database_subprocess_enabled=true` or `vector_db_subprocess_enabled=true`), the queue also coordinates the release of the cached per-dataset graph and vector engines on dataset-context exit. This only runs once the exiting task is the last holder of that dataset's queue slot — so an in-flight task that still has the dataset open will not observe a torn-down engine.
Releasing hands the engines back to the engine cache, which then either keeps them warm for the subprocess idle keep-alive window or evicts them. Eviction itself is synchronous, so a dying engine leaves the cache before the queue slot is freed and the next caller can never fetch it. The engine's `close()` is not synchronous: for subprocess-backed adapters the cache dispatches it onto its own dedicated close threads. A search or recall response therefore returns without waiting for the worker process to exit and drop its file lock.
Safety is preserved by a pending-close latch rather than by blocking the caller. From the moment a close starts until it has fully completed (worker exited, lock released), the cache records it as pending for that engine, and the next creation of the same engine — through the queue or through a direct `get_graph_engine()` / vector equivalent — waits for it: async callers suspend, and synchronous callers on a thread with no running event loop block. Every such wait is bounded at 300 seconds; on timeout Cognee logs a warning, stops waiting, and falls back to the worker's own open retries (`SUBPROCESS_OPEN_LOCK_RETRIES` and `SUBPROCESS_OPEN_LOCK_BACKOFF`, see [Graph Stores](/setup-configuration/graph-stores)). One case deliberately does not wait — a synchronous caller already running on an event loop, since the close may need that same loop to make progress — and there the open retries are the only backstop.
Teardown problems are logged, never surfaced to the caller: a `close()` that raises is logged as a warning with its traceback, an overrun wait logs the warning above, broken close bookkeeping is logged at `ERROR`, and no engine creation, eviction, or idle sweep fails over any of it. If you see repeated lock-contention errors when opening a database, check the logs for these teardown warnings first.
What that last-holder release does is controlled by `SUBPROCESS_IDLE_TTL_SECONDS`:
```dotenv theme={null}
# Keep an idle subprocess engine alive this many seconds before closing it.
# Default: 600. Set to 0 to close at every last-holder release.
SUBPROCESS_IDLE_TTL_SECONDS=600
```
* **TTL greater than `0` (default `600`)** — the engine is kept warm: instead of being closed, its idle timestamp is refreshed, and a background reaper closes it only after a full TTL with no use. A request for the same dataset arriving inside that window reuses the live worker and skips both the close and the respawn.
* **TTL `0`** — the engine is evicted and force-closed at the last-holder release. This is the behavior that applied before the keep-alive existed, and setting `0` is how you restore it.
Negative values are clamped to `0`, and fractional values are accepted.
The reaper is a daemon thread (`subprocess-idle-reaper`) that starts lazily on the first kept-alive release and then sweeps every engine cache every `max(5, min(60, TTL / 4))` seconds. The sweep skips two kinds of entries:
* Datasets that currently hold a queue slot — the same pin that protects them from capacity eviction (see [Engine cache pinning](#engine-cache-pinning) above), so an operation running longer than the TTL cannot have its engine closed underneath it.
* Engines that are not subprocess-backed. Remote stores such as Neo4j, Postgres, and PGVector hold no worker process and no file lock, so they are never reaped and keep plain LRU behavior.
The trade-off is operational: a kept-warm worker holds its database file lock, its memory, and its PID for up to the TTL after its last use. The number of idle workers retained this way is still bounded by the engine cache capacity (`DATABASE_MAX_LRU_CACHE_SIZE`). Lower the TTL, or set it to `0`, if you need locks and memory released promptly — for example when another tool needs to open the same data directory, or on memory-tight multi-tenant deployments.
If you set `DATASET_QUEUE_ENABLED=false` while leaving subprocess mode on, there is no teardown at all, and `SUBPROCESS_IDLE_TTL_SECONDS` has no effect: the release path — and with it both the keep-alive refresh and the reaper — never runs. Subprocess engines are not closed when a dataset context exits, and the database file's `flock()` will remain held until the cached engine is closed or evicted, or until the worker process shuts down. Keep the queue enabled when running with subprocess databases under concurrent multi-dataset workloads.
## Database Setup
Choose your relational database:
* **SQLite** — Local development (auto-creates files)
* **Postgres** — Production (requires manual setup)
See [Relational Databases](./relational-databases) for detailed configuration.
## Authentication
### API Server
Start the server with authentication:
```bash theme={null}
uvicorn cognee.api.client:app --host 0.0.0.0 --port 8000
```
**Default credentials (development only):**
* Username: `default_user@example.com`
* Password: `default_password`
### Programmatic Access
See [Permission Snippets](/guides/permission-snippets) for complete programmatic examples.
## Data Organization
Data is automatically organized by user and dataset. Each user gets isolated storage:
```
.cognee_system/databases/<user_uuid>/
├── <dataset_uuid>.pkl # Kùzu graph database
└── <dataset_uuid>.lance.db/ # LanceDB vector database
```
## Troubleshooting
<AccordionGroup>
<Accordion title="Permission denied">
If a request fails with a permission error:
* Confirm the request is authenticated as the expected user.
* Confirm the target dataset belongs to that user, or has been shared with them.
* If you are testing locally, verify `REQUIRE_AUTHENTICATION=true` and `ENABLE_BACKEND_ACCESS_CONTROL=true` match the mode you expect.
For complete authenticated request examples, see [Permission Snippets](/guides/permission-snippets).
</Accordion>
<Accordion title="Data isolation">
With access control enabled, Cognee stores graph and vector data per user and per dataset. If data appears to leak across users or is missing unexpectedly:
* Verify `ENABLE_BACKEND_ACCESS_CONTROL=true`.
* Verify you are reading and writing as the intended authenticated user.
* Check that separate user-specific database files exist on disk:
```bash theme={null}
ls -la .cognee_system/databases/<user_uuid>/
```
Different users should have different database paths and dataset files.
</Accordion>
<Accordion title="401/403 on add or search">
When access control is enabled, `VECTOR_DB_PROVIDER` and `VECTOR_DATASET_DATABASE_HANDLER` must resolve to a compatible pair. For the built-in providers below, leaving the handler at its default lets Cognee auto-select the matching handler. Requests fail when you explicitly choose an incompatible handler.
| Vector provider | Resolved handler in access-control mode | Do you need to set `VECTOR_DATASET_DATABASE_HANDLER`? |
| ------------------- | --------------------------------------- | ----------------------------------------------------- |
| `lancedb` (default) | `lancedb` | No |
| `pgvector` | `pgvector` | No, unless you want to set it explicitly |
| `turso` | `turso` | No, unless you want to set it explicitly |
Example — using PGVector with access control:
```dotenv theme={null}
VECTOR_DB_PROVIDER=pgvector
# Optional: Cognee resolves this to pgvector automatically when omitted.
VECTOR_DATASET_DATABASE_HANDLER=pgvector
VECTOR_DB_URL=postgresql://user:pass@localhost:5432/cognee_db
```
</Accordion>
<Accordion title="Local Neo4j + multi-user mode: provider/handler mismatch error">
**Symptom**: Cognee raises an `EnvironmentError` about a graph provider/handler mismatch when `GRAPH_DATABASE_PROVIDER=neo4j` and `ENABLE_BACKEND_ACCESS_CONTROL=true`.
**Root cause**: A plain Neo4j connection is not supported for multi-user mode. Cognee's runtime check validates that the configured graph provider matches a supported dataset database handler (`supported_dataset_database_handlers`); providers that pass with their default handler are `ladybug`/`kuzu`, `postgres_demo` (or its accepted alias `postgres`), and `turso`. Neo4j is only supported in multi-user mode through a dataset database handler — `neo4j_aura_dev` (provisions a Neo4j Aura cloud instance per dataset) or `neo4j_community` (runs a local Neo4j Community Docker container per dataset) — so enabling `ENABLE_BACKEND_ACCESS_CONTROL=true` with `GRAPH_DATABASE_PROVIDER=neo4j` and no handler leads to this error.
<Tabs>
<Tab title="Single-User Local Neo4j">
Recommended for self-hosted Neo4j deployments:
```dotenv theme={null}
GRAPH_DATABASE_PROVIDER=neo4j
GRAPH_DATABASE_URL=bolt://localhost:7687
GRAPH_DATABASE_USERNAME=neo4j
GRAPH_DATABASE_PASSWORD=yourpassword
ENABLE_BACKEND_ACCESS_CONTROL=false
```
</Tab>
<Tab title="Multi-User with Neo4j Aura">
Use [Neo4j Aura with the `neo4j_aura_dev` handler](/core-concepts/multi-user-mode/dataset-database-handlers/existing-dataset-database-handlers/neo4j-aura-dev), which provisions a dedicated Aura cloud instance per dataset:
```dotenv theme={null}
GRAPH_DATABASE_PROVIDER=neo4j
GRAPH_DATASET_DATABASE_HANDLER=neo4j_aura_dev
NEO4J_CLIENT_ID=<your_oauth_client_id>
NEO4J_CLIENT_SECRET=<your_oauth_client_secret>
NEO4J_TENANT_ID=<your_aura_tenant_id>
NEO4J_ENCRYPTION_KEY=<a_strong_random_key>
```
</Tab>
<Tab title="Multi-User with Local Docker">
Use the [`neo4j_community` handler](/core-concepts/multi-user-mode/dataset-database-handlers/existing-dataset-database-handlers/neo4j-community), which runs one Neo4j Community Docker container per dataset on the host you control. Requires a reachable Docker daemon:
```dotenv theme={null}
GRAPH_DATABASE_PROVIDER=neo4j
GRAPH_DATASET_DATABASE_HANDLER=neo4j_community
NEO4J_ENCRYPTION_KEY=<a_strong_random_key>
```
</Tab>
</Tabs>
</Accordion>
</AccordionGroup>
<Columns>
<Card title="Permission System" icon="brain" href="/core-concepts/multi-user-mode/permissions-system/overview">
Learn about users, tenants, roles, and ACL
</Card>
<Card title="Usage Guide" icon="book-open" href="/guides/permission-snippets">
How to use permission features
</Card>
</Columns>
# Relational Databases
Source: https://docs.cognee.ai/setup-configuration/relational-databases
Configure relational databases for metadata and state storage in Cognee
Relational databases store metadata, document information, and system state in Cognee. They track documents, chunks, and provenance (where data came from and how it's linked).
<Info>
**New to configuration?**
See the [Setup Configuration Overview](./overview) for the complete workflow:
install extras → create `.env` → choose providers → handle pruning.
For a complete, copy-paste `.env` block that combines this layer with a vector and a graph store, see [Store Configurations](/guides/store-configurations).
</Info>
## Supported Providers
Cognee supports these relational database options:
* **SQLite** — File-based database, works out of the box (default)
* **Postgres** — Production-ready database with external hosting options
* **Turso (libSQL)** — A SQLite-compatible drop-in with optional embedded-replica sync for a hosted Turso database
## Configuration
<Accordion title="Environment Variables">
Set these environment variables in your `.env` file:
* `DB_PROVIDER` — The database provider (sqlite, postgres, turso)
* `DB_NAME` — Database name
* `DB_HOST` — Database host (Postgres only)
* `DB_PORT` — Database port (Postgres only)
* `DB_USERNAME` — Database username (Postgres only)
* `DB_PASSWORD` — Database password (Postgres only)
* `DB_TURSO_URL` — Remote Turso database URL, e.g. `libsql://<your-db>.turso.io` (Turso remote mode only; leave unset for a local libSQL file)
* `DB_TURSO_AUTH_TOKEN` — Auth token for the remote Turso database (Turso remote mode only)
</Accordion>
## Setup Guides
<AccordionGroup>
<Accordion title="SQLite (Default)">
SQLite is file-based and requires no additional setup. It's perfect for local development and single-user scenarios.
```dotenv theme={null}
DB_PROVIDER="sqlite"
DB_NAME="cognee_db"
```
**Installation**: SQLite is included by default with Cognee. No additional installation required.
**Data Location**: Data is stored under the Cognee system directory. You can override the root with `SYSTEM_ROOT_DIRECTORY` in your `.env` file.
</Accordion>
<Accordion title="Postgres">
Postgres is recommended for production environments or when you need external hosting.
The example below assumes **Cognee runs on your machine** (a script, `cognee-cli`, or `uvicorn` — not a container) reaching Postgres on a port published to the host, whether that Postgres is installed natively or is the `docker compose --profile postgres` container. For other topologies, see the **Which `DB_HOST` to use** table below.
<Tabs>
<Tab title=".env">
Set the connection in your `.env` file:
```dotenv theme={null}
DB_PROVIDER="postgres"
DB_NAME="cognee_db"
DB_HOST="127.0.0.1" # Cognee on the host
DB_PORT="5432"
DB_USERNAME="cognee"
DB_PASSWORD="cognee"
```
</Tab>
<Tab title="Python">
Instead of (or in addition to) the `.env` file, set the same values at runtime with `cognee.config.set_relational_db_config()`. Call it before any `add()`, `cognify()`, or `remember()` so the connection is used from the first operation:
```python theme={null}
import asyncio
import cognee
async def main():
cognee.config.set_relational_db_config(
{
"db_provider": "postgres",
"db_name": "cognee_db",
"db_host": "127.0.0.1", # Cognee on the host
"db_port": "5432",
"db_username": "cognee",
"db_password": "cognee",
}
)
await cognee.remember(["Cognee stores its metadata in Postgres."])
print(await cognee.recall(query_text="Where is metadata stored?"))
asyncio.run(main())
```
The dictionary keys match the `DB_*` variables in the `.env` tab. To also route embeddings and the graph into the same Postgres instance, pair this with [`set_vector_db_config({"vector_db_provider": "pgvector"})`](/setup-configuration/vector-stores) and [`GRAPH_DATABASE_PROVIDER="postgres_demo"`](/setup-configuration/graph-stores#postgres) (the older value `postgres` is still accepted).
</Tab>
</Tabs>
**Installation**: Install the Postgres extras:
```bash theme={null}
pip install "cognee[postgres]"
# or for binary version
pip install "cognee[postgres-binary]"
```
**Docker Setup**: Use the built-in Postgres service:
```bash theme={null}
docker compose --profile postgres up -d
```
**Which `DB_HOST` to use**: the value is always resolved from inside the Cognee process, so it depends on where Cognee runs, not on where Postgres runs:
| Cognee runs… | Postgres runs… | `DB_HOST` |
| --------------------------------------------- | --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| On the host (script, `cognee-cli`, `uvicorn`) | On the host, or a container publishing `5432` | `127.0.0.1` |
| In a container | On the host (installed natively) | `host.docker.internal` — on plain Linux Docker, start the container with `--add-host=host.docker.internal:host-gateway` |
| In Docker Compose | In the same Compose project | `postgres` — the Compose **service name** |
| Anywhere | Managed/remote (Neon, RDS, Azure) | The provider's hostname |
Cognee's `docker-compose.yml` maps `host.docker.internal` to `host-gateway` on both the `cognee` and `cognee-mcp` services and defaults `DB_HOST` to it — a default that only takes effect once you set `DB_PROVIDER=postgres`, since the file ships with `DB_PROVIDER=sqlite`. To use the bundled `--profile postgres` service instead, set `DB_HOST=postgres`. See the [Docker Compose reference](/how-to-guides/cognee-sdk/deployment/docker-compose-reference) for the full service layout.
**Migrations**: The Cognee API server runs startup migrations during its lifespan startup. For standalone scripts, CI, or deployments where you manage database lifecycle explicitly, run [`run_migrations`](/python-api/run-migrations) before serving traffic — especially the first time you point Cognee at a fresh external Postgres database, or after upgrading the `cognee` package:
```python theme={null}
import cognee
await cognee.run_migrations()
```
</Accordion>
<Accordion title="Neon Postgres">
Neon works with Cognee through the normal `postgres` relational provider. Cognee can use Neon Postgres for relational metadata, document information, chunks, pgvector storage, and Postgres graph state. Neon requires SSL/TLS for connections.
Before configuring Cognee, create a Neon project, branch, database, and role, then copy the connection string from **Connection Details** in the Neon dashboard. A Neon connection string usually looks like this:
```text theme={null}
postgresql://user:password@ep-example.us-east-2.aws.neon.tech/neondb?sslmode=require
```
Cognee does not read that connection string as a single value — there is no `DATABASE_URL` setting, so a `DATABASE_URL` your platform injects is ignored. Map the connection string into Cognee's `DB_*` variables instead:
```dotenv theme={null}
DB_PROVIDER="postgres"
DB_NAME="neondb"
DB_HOST="ep-example.us-east-2.aws.neon.tech"
DB_PORT="5432"
DB_USERNAME="user"
DB_PASSWORD="password"
DATABASE_CONNECT_ARGS='{"ssl": "require", "timeout": 10}'
```
Install Postgres support in the environment where Cognee runs:
```bash theme={null}
pip install "cognee[postgres]"
# or
pip install "cognee[postgres-binary]"
```
Express Neon's `?sslmode=require` parameter as `DATABASE_CONNECT_ARGS='{"ssl": "require"}'`; query parameters in the connection string have nowhere to go once it is split up. `DATABASE_CONNECT_ARGS` must be valid JSON. Cognee forwards these arguments to the main relational engine, per-dataset PGVector engines, and the `GRAPH_DATABASE_PROVIDER="postgres_demo"` graph engine **(demo)**.
One Neon database can also back PGVector and the Postgres graph store:
```dotenv theme={null}
VECTOR_DB_PROVIDER="pgvector"
GRAPH_DATABASE_PROVIDER="postgres_demo"
```
<Warning>
The Postgres graph store is a **demo feature** — in production, use a graph-native backend such as Kuzu or Neo4j. A production-ready adapter is available as a licensed product; book a call with our sales team at [cognee.ai](https://www.cognee.ai). See [Graph Stores](/setup-configuration/graph-stores) for details.
</Warning>
For PGVector, enable the extension once in the Neon database:
```sql theme={null}
CREATE EXTENSION IF NOT EXISTS vector;
```
**Application database vs source database**: the `DB_*` variables configure Cognee's own application database. Cognee uses this database for its internal metadata and state. To ingest an external Postgres database as data, pass that source database connection to `cognee.add()` instead:
```python theme={null}
await cognee.add(
"postgresql://user:pass@host:5432/source_db",
dataset_name="postgres_data",
)
```
That source database is separate from Cognee's application database.
**Direct vs pooled Neon hosts**: use the direct Neon host for setup, schema migrations, and default Cognee connections. The direct hostname does not contain `-pooler`:
```dotenv theme={null}
DB_HOST="ep-example.us-east-2.aws.neon.tech"
```
Neon pooled hosts route through PgBouncer and contain `-pooler` in the hostname:
```dotenv theme={null}
DB_HOST="ep-example-pooler.us-east-2.aws.neon.tech"
```
Prefer the direct host unless you have a specific need for pooled, high-concurrency application traffic after setup. Run migrations against the direct endpoint only. Neon PgBouncer does not support every session-level operation migrations and maintenance may rely on, and Cognee maintenance operations such as `CREATE DATABASE` and `DROP DATABASE` cannot run through the pooler. If setup or migrations fail on a `-pooler` host, switch to the direct host and retry.
You can verify the same credentials with `psql`:
```bash theme={null}
psql "postgresql://user:password@ep-example.us-east-2.aws.neon.tech/neondb?sslmode=require"
```
If you use Cognee's relational database migration features with Neon, keep the application database and migration source database separate. Use direct hosts for both while running migrations:
```dotenv theme={null}
# Application DB: Cognee's internal metadata store
DB_PROVIDER="postgres"
DB_HOST="ep-example.us-east-2.aws.neon.tech"
DB_PORT="5432"
DB_USERNAME="user"
DB_PASSWORD="password"
DB_NAME="neondb"
DATABASE_CONNECT_ARGS='{"ssl": "require"}'
# Migration DB: source data to convert into Cognee's knowledge graph
MIGRATION_DB_PROVIDER="postgres"
MIGRATION_DB_HOST="ep-source.us-east-2.aws.neon.tech"
MIGRATION_DB_PORT="5432"
MIGRATION_DB_USERNAME="readonly_user"
MIGRATION_DB_PASSWORD="readonly_password"
MIGRATION_DB_NAME="source_app_db"
```
Use a different `MIGRATION_DB_NAME` unless you intentionally want to migrate Cognee's own internal tables into the knowledge graph.
</Accordion>
<Accordion title="Turso (libSQL)">
A libSQL database file *is* a SQLite file, so Turso is a drop-in for the SQLite backend: Cognee talks to it through the same `aiosqlite` driver, the same sqlite dialect, and the same sqlite-dialect Alembic migrations. No migration changes are needed when switching between SQLite and Turso.
**Installation**: Install the Turso extra:
```bash theme={null}
pip install "cognee[turso]"
```
**Local / embedded**: A libSQL file stored on disk under the Cognee data directory (named by `DB_NAME`). This is identical to the SQLite backend:
```dotenv theme={null}
DB_PROVIDER="turso"
DB_NAME="cognee_db"
```
**Remote (embedded replica)**: Set `DB_PROVIDER="turso"` and point at a hosted Turso database with `DB_TURSO_URL` and `DB_TURSO_AUTH_TOKEN`:
```dotenv theme={null}
DB_PROVIDER="turso"
DB_NAME="cognee_db"
DB_TURSO_URL="libsql://<your-db>.turso.io"
DB_TURSO_AUTH_TOKEN="<your-token>"
```
In remote mode Cognee reads and writes a fast local replica through `aiosqlite` exactly as in local mode, while `libsql-experimental` handles embedded-replica sync with the hosted primary. The replica is seeded from the primary before first use, and Cognee attempts a sync after each write within the operation. Seeding and syncing run off the event loop and are best-effort: a slow or unreachable primary is logged and never blocks or breaks a database operation, and the local replica stays usable.
<Note>
The remote write path applies through `aiosqlite`; whether libSQL's sync propagates those writes to the hosted primary depends on the driver's replica write-capture and should be confirmed against a live Turso database. The local drop-in path is fully exercised offline.
</Note>
<Info>
Turso is a SQLite-compatible drop-in for Cognee's core relational backend, but DLT-based ingestion connectors do not yet treat `DB_PROVIDER="turso"` the same as `sqlite`. The main add → cognify → search pipeline is covered; DLT connector support needs a follow-up.
</Info>
</Accordion>
</AccordionGroup>
## Advanced Options
<Accordion title="Migration Configuration">
The `MIGRATION_DB_*` variables point to a **source** database that you want to extract and migrate **into** Cognee's knowledge graph. This is entirely separate from the application database (`DB_*`) that Cognee uses for its own internal metadata and state.
| Variable | Application DB (`DB_*`) | Migration DB (`MIGRATION_DB_*`) |
| -------- | ---------------------------------------------- | ----------------------------------------- |
| Purpose | Cognee's internal metadata store | Source data you want converted to a graph |
| Contains | Cognee's own tables (documents, chunks, state) | Your application's tables and rows |
**Does the migration DB need to be a different database than the application DB?**
In practice, use a different database (different `DB_NAME` / `MIGRATION_DB_NAME`) unless you intentionally want to migrate Cognee's own internal tables into the knowledge graph. They can still live on the same Postgres server as long as they are different databases.
<Tabs>
<Tab title="SQLite Source">
Use this when your source data is in a SQLite file, regardless of what `DB_PROVIDER` is set to:
```dotenv theme={null}
# Application DB (Cognee's internal store)
DB_PROVIDER="postgres"
DB_NAME="cognee_db"
DB_HOST="127.0.0.1"
DB_PORT="5432"
DB_USERNAME="cognee"
DB_PASSWORD="cognee"
# Migration DB (your source data — a separate SQLite file)
MIGRATION_DB_PROVIDER="sqlite"
MIGRATION_DB_PATH="/path/to/migration/directory"
MIGRATION_DB_NAME="my_app_data.sqlite"
```
</Tab>
<Tab title="Same Postgres Server">
Use this when your source data is in a separate Postgres database on the same server as Cognee's application DB. Set `MIGRATION_DB_NAME` to a **different** database name for the usual case:
```dotenv theme={null}
# Application DB (Cognee's internal store)
DB_PROVIDER="postgres"
DB_NAME="cognee_db"
DB_HOST="127.0.0.1"
DB_PORT="5432"
DB_USERNAME="cognee"
DB_PASSWORD="cognee"
# Migration DB (your source data — different DB name on the same Postgres server)
MIGRATION_DB_PROVIDER="postgres"
MIGRATION_DB_HOST="127.0.0.1"
MIGRATION_DB_PORT="5432"
MIGRATION_DB_USERNAME="cognee"
MIGRATION_DB_PASSWORD="cognee"
MIGRATION_DB_NAME="my_app_db" # usually different from DB_NAME above
```
</Tab>
<Tab title="Different Postgres Server">
Use this when your source data lives on a separate Postgres instance:
```dotenv theme={null}
# Migration DB (separate Postgres instance)
MIGRATION_DB_PROVIDER="postgres"
MIGRATION_DB_HOST="db.example.com"
MIGRATION_DB_PORT="5432"
MIGRATION_DB_USERNAME="readonly_user"
MIGRATION_DB_PASSWORD="readonly_password"
MIGRATION_DB_NAME="production_db"
```
</Tab>
</Tabs>
See the [Relational Database Migration example](/examples/relational-db-migration) for a complete walkthrough of migrating schema and data into a knowledge graph.
</Accordion>
<Accordion title="Managed Postgres with SSL (connect args)">
Managed Postgres providers (Neon, RDS/Aurora, Azure Database for PostgreSQL) often require SSL. Pass asyncpg/SQLAlchemy connection arguments through the `DATABASE_CONNECT_ARGS` environment variable, which takes a JSON object:
```dotenv theme={null}
DATABASE_CONNECT_ARGS='{"ssl": "require", "timeout": 10}'
```
These connect args are forwarded to Cognee's main relational engine, per-dataset PGVector engines, and the `GRAPH_DATABASE_PROVIDER="postgres_demo"` graph engine **(demo)**. The maintenance engine that runs CREATE/DROP DATABASE also uses the SSL setting. Leaving `DATABASE_CONNECT_ARGS` unset is a no-op, so in-cluster Postgres needs no change.
The maintenance engine talks to Postgres over asyncpg, which expects an `ssl` key rather than libpq's `sslmode`; if you supply `sslmode`, its value is mapped to asyncpg's `ssl` for maintenance operations. For **Neon** specifically, the maintenance engine also rewrites a `-pooler.` host to its direct endpoint, because CREATE/DROP DATABASE cannot run through Neon's PgBouncer connection pooler.
The value must be a valid JSON object; invalid JSON raises a configuration error.
</Accordion>
<Accordion title="SQLite connection pooling (POOL_ARGS)">
`POOL_ARGS` is honored on the SQLite relational engine, not just on Postgres.
**Leaving it unset changes nothing.** The SQLite engine still uses SQLAlchemy's `NullPool` (no connection reuse), the same 120-second driver connect timeout, and the same WAL / `synchronous=NORMAL` / `busy_timeout=120000` pragmas on every connection. Nothing needs to be configured unless you deliberately want a pool.
**Opt into a bounded pool** by setting any of the pool-sizing keys, which switches the engine from `NullPool` to SQLAlchemy's default bounded pool:
```dotenv theme={null}
POOL_ARGS='{"pool_size": 5, "max_overflow": 10}'
```
The keys applied on the SQLite engine are `pool_size`, `max_overflow`, `pool_recycle`, `pool_timeout`, `pool_pre_ping`, and `poolclass`. Any other engine keyword in `POOL_ARGS` is ignored on SQLite so it cannot collide with the SQLite-specific connect args Cognee assembles.
**`poolclass` accepts the `"nullpool"` string**, normalized to the `NullPool` class exactly as on the Postgres engine — so the same `POOL_ARGS` value means the same thing on both backends:
```dotenv theme={null}
POOL_ARGS='{"poolclass": "nullpool"}'
```
**Contradictory options fail fast.** `NullPool` takes no sizing arguments, so combining it with sizing keys raises a `TypeError` when the engine is created rather than silently picking one:
```dotenv theme={null}
# invalid — raises TypeError at engine creation
POOL_ARGS='{"poolclass": "nullpool", "pool_size": 5}'
```
Pick one or the other: sizing keys for a bounded pool, or `"poolclass": "nullpool"` on its own.
Enabling a bounded pool changes how many SQLite connections Cognee keeps open and its resource usage — each pooled connection holds a file handle and, on the async driver, a worker thread. It is also not a fix for connections abandoned by callers: a session that is never closed still holds its connection regardless of the pool class.
</Accordion>
<Accordion title="Backend Access Control">
Enable per-user dataset isolation for multi-tenant scenarios.
```dotenv theme={null}
ENABLE_BACKEND_ACCESS_CONTROL="true"
```
This feature is available for both SQLite and Postgres.
</Accordion>
## Troubleshooting
<Accordion title="Common Issues">
**Postgres Connectivity**: Verify the database is listening on `DB_HOST:DB_PORT` and credentials are correct:
```bash theme={null}
psql -h 127.0.0.1 -U cognee -d cognee_db
```
**Docker Networking**: The right host depends on where the Cognee process runs — see [Which `DB_HOST` to use](#setup-guides) in the Postgres setup guide.
**SQLite Concurrency**: SQLite connections now open in WAL (Write-Ahead Logging) journal mode with `synchronous=NORMAL` and a 120-second busy timeout, and the driver connect timeout is also 120 seconds. This lets concurrent writers wait for the write lock (up to the busy timeout) instead of immediately failing, which greatly reduces the `sqlite3.OperationalError: database is locked` errors that could occur under Cognee's parallel `cognify()` writes. No configuration is required — these settings apply automatically to every SQLite connection, and they stay in place even if you change the pool class (see *SQLite connection pooling (POOL\_ARGS)* under [Advanced Options](#advanced-options)). WAL mode creates `-wal` and `-shm` sidecar files next to the database file; include them when backing up or copying the database. For heavier multi-user workloads, still prefer Postgres. Note that this only smooths out the parallel writes Cognee itself issues within a single process — it is not a mechanism for sharing the database between multiple Cognee processes.
**SQLite File Locks on Windows (pruning/deleting)**: When pruning or deleting a SQLite database, Cognee now disposes the cached SQLAlchemy engine (clearing the relational-engine cache and forcing garbage collection) before removing the file, so the underlying connection releases the file handle. If a stubborn Windows file lock still prevents removal after the retries, deletion no longer raises — it logs a warning and continues. In that case, the SQLite file may remain on disk and can be removed manually after the process releases the handle.
</Accordion>
<Accordion title="TypeError: int() argument ... not 'NoneType' (DB_PORT unset)">
With `DB_PROVIDER="postgres"`, Cognee assembles the connection URL from the split `DB_*` variables and casts the port with `int()`. `DB_PORT` has **no default** — it stays unset until you give it a value — so a Postgres provider without `DB_PORT` fails as the engine is built. In the API server and the Docker image that happens during [startup migrations](/how-to-guides/cognee-sdk/deployment/docker#migration-fails-first-boot), immediately after `Running database migrations...`:
```text theme={null}
File ".../cognee/infrastructure/databases/relational/create_relational_engine.py", line 65, in create_relational_engine
port=int(db_port),
TypeError: int() argument must be a string, a bytes-like object or a real number, not 'NoneType'
```
The usual cause is supplying the connection as one URL. The Python package has no `DATABASE_URL` setting for its application database, so a platform-injected `DATABASE_URL` from a managed Postgres add-on is read by nothing and `DB_HOST`, `DB_PORT`, `DB_USERNAME`, and `DB_PASSWORD` all stay unset. (`DATABASE_URL` *is* a setting in the separate [Rust SDK](/rust/configuration#relational-database).) Split the URL into the `DB_*` variables — `postgresql://user:password@db.example.com:5432/cognee_db` becomes:
```dotenv theme={null}
DB_PROVIDER="postgres"
DB_HOST="db.example.com"
DB_PORT="5432"
DB_USERNAME="user"
DB_PASSWORD="password"
DB_NAME="cognee_db"
```
The port is cast before the URL is assembled, so this `TypeError` is what you see even when the host and credentials are missing too — set all of them, not just `DB_PORT`. Query parameters such as `?sslmode=require` do not carry over either; express them as `DATABASE_CONNECT_ARGS` (see *Managed Postgres with SSL* under [Advanced Options](#advanced-options)). `DB_PROVIDER` itself defaults to `sqlite`, which ignores the `DB_*` host, port, and credential settings, so this failure only appears once you switch the provider to `postgres`.
</Accordion>
<Accordion title="Neon SSL and connect args">
Neon requires SSL/TLS. Cognee takes the connection as split `DB_*` settings rather than a single URL, so a `?sslmode=require` query parameter has nowhere to live — pass SSL through `DATABASE_CONNECT_ARGS` instead:
```dotenv theme={null}
DB_PROVIDER="postgres"
DB_NAME="neondb"
DB_HOST="ep-example.us-east-2.aws.neon.tech"
DB_PORT="5432"
DB_USERNAME="user"
DB_PASSWORD="password"
DATABASE_CONNECT_ARGS='{"ssl": "require", "timeout": 10}'
```
`DATABASE_CONNECT_ARGS` must be valid JSON. Invalid JSON raises a configuration error before Cognee connects.
</Accordion>
<Accordion title="Missing LLM API key">
Operations that extract or answer over memory need an LLM provider. If `remember()`, `cognify()`, `recall()`, or related workflows fail because no LLM credentials are configured, set the provider API key in your environment:
```dotenv theme={null}
LLM_API_KEY="sk-..."
```
See [LLM Providers](/setup-configuration/llm-providers) for provider-specific settings.
</Accordion>
<Accordion title="asyncpg prepared-statement / connection-pooler errors">
On Postgres and PGVector, Cognee connects through the asyncpg driver, which caches prepared statements **per connection**. When you place a **transaction-mode** connection pooler in front of Postgres — PgBouncer in `transaction` mode, or the Supabase / Neon connection poolers — a single client connection is multiplexed across many short-lived server backends. The cached statement names can then collide or vanish between checkouts, surfacing as:
```text theme={null}
asyncpg.exceptions.DuplicatePreparedStatementError: prepared statement "__asyncpg_stmt_1__" already exists
```
or as intermittent `connection is closed` / `InterfaceError` pool errors under concurrency.
**Preferred fix — use the direct (session-mode) endpoint.** Point `DB_HOST` (and `VECTOR_DB_HOST`) at the direct Postgres endpoint rather than the transaction pooler. Cognee already does this for its own `CREATE`/`DROP DATABASE` maintenance work, rewriting a Neon `-pooler.` host to the direct endpoint, because those statements cannot run through PgBouncer.
**If you must route through a transaction-mode pooler**, disable asyncpg's statement cache through [`DATABASE_CONNECT_ARGS`](/setup-configuration/relational-databases):
```dotenv theme={null}
DATABASE_CONNECT_ARGS='{"statement_cache_size": 0, "prepared_statement_cache_size": 0}'
```
These connect args are forwarded to the main relational engine, the per-dataset PGVector engines, and the `GRAPH_DATABASE_PROVIDER="postgres_demo"` graph engine **(demo)**, so all three asyncpg connections stop caching prepared statements. You can combine them with the SSL keys in the same JSON object (for example `{"ssl": "require", "statement_cache_size": 0}`).
</Accordion>
<Accordion title="Too many connections (Neon free tier / hosted Postgres limits)">
When the same Postgres backs relational metadata, PGVector, and the `GRAPH_DATABASE_PROVIDER="postgres_demo"` graph store, Cognee can open more connections than a low connection limit allows — most commonly on Neon's free tier — surfacing as:
```text theme={null}
asyncpg.exceptions.TooManyConnectionsError: sorry, too many clients already
# or: FATAL: remaining connection slots are reserved for non-replication superuser connections
```
Cognee opens a **separate SQLAlchemy connection pool per engine**, and the `DB_*` / `DATABASE_CONNECT_ARGS` settings are reused across all of them:
* **Relational engine** — QueuePool with `pool_size=5` and `max_overflow=35` (up to 40 connections), plus `pool_pre_ping=True` and `pool_recycle=280`.
* **PGVector** — when backend access control is off and the relational provider is Postgres, PGVector **reuses the relational engine** and adds no connections of its own. It creates its own pool only under `ENABLE_BACKEND_ACCESS_CONTROL="true"` (one engine per dataset), sized from `VECTOR_POOL_ARGS` if set, otherwise from `POOL_ARGS`, and falling back to `pool_size=2`, `max_overflow=20` when neither is set.
* **Postgres graph store (demo)** — always its own pool, with leaner defaults `pool_size=2` and `max_overflow=20` (up to 22 connections). Under access control it is also created per dataset.
* **SQL cache engine** — the session cache builds its own engine whenever caching (or usage logging) is on and `CACHE_BACKEND` is `sqlite` or `postgres`, and it reads the relational `POOL_ARGS` for it. It adds no defaults of its own, so with `POOL_ARGS` unset the engine takes SQLAlchemy's own pool defaults. Only a Postgres cache consumes slots on the server — either `CACHE_BACKEND="postgres"` or a `CACHE_DB_URL` pointing at Postgres.
So a single-user setup with `GRAPH_DATABASE_PROVIDER="postgres_demo"` can reach roughly 40 + 22 connections at peak, plus the cache engine's pool when the cache is also in Postgres, and backend access control multiplies the per-dataset pools by the number of datasets.
When sizing for concurrency, note that authenticated API traffic no longer doubles its draw on the relational pool: an API-key request used to check out a second connection and hold it for the request's full lifetime, so it needed two slots at once — see the *Connections stuck in idle in transaction, or the pool deadlocking under concurrency* accordion below.
**Shrink the pools** to fit the server's `max_connections`. `POOL_ARGS` applies to the relational engine, is reused by the Postgres graph engine and the SQL cache engine, and also sizes per-dataset PGVector engines whenever `VECTOR_POOL_ARGS` is unset; `VECTOR_POOL_ARGS` applies to per-dataset PGVector engines only, where it takes precedence over `POOL_ARGS`. Both take a JSON object:
```dotenv theme={null}
POOL_ARGS='{"pool_size": 2, "max_overflow": 4}'
VECTOR_POOL_ARGS='{"pool_size": 2, "max_overflow": 2}'
```
To minimize idle connections entirely, disable pooling so each operation opens and closes its own connection:
```dotenv theme={null}
POOL_ARGS='{"poolclass": "nullpool"}'
```
`poolclass` is given as the string `"nullpool"` (case-insensitive), which Cognee normalizes to SQLAlchemy's `NullPool` class. The relational engine and the SQL cache engine both accept that string form, so this single value covers both — there is no separate cache-pool setting. On releases before version 1.4.2, the cache engine passed the string to SQLAlchemy unchanged and failed at startup with `CacheConnectionError: Failed to initialize SQL cache engine for …: 'str' object has no attribute '__dict__'`; if you hit that, upgrade rather than dropping `poolclass` from `POOL_ARGS`.
The tradeoff is that under `NullPool` every SQLAlchemy session becomes a full connection setup — TCP, TLS, and SCRAM-SHA-256, about 14 ms of event-loop CPU each on asyncpg 0.30 before any network latency — so what you pay is set by how many sessions an operation opens rather than by a pool size. For sizing, an `add()` costs roughly **4 relational sessions and 7 statements per file**, measured over a 164-PDF add with real s3fs and asyncpg against Postgres with `poolclass: nullpool`. Those are peak-concurrency multipliers as much as CPU costs: the pipeline processes items concurrently, so a large add opens several of these connections at once against your server's `max_connections`. That figure lands as of the fix in [PR #4589](https://github.com/topoteretes/cognee/pull/4589), which cut it from \~11 sessions and \~15 statements per file; on releases before it, size against the higher number.
Alternatively, route application traffic through Neon's pooled (`-pooler`) endpoint, which supports far more concurrent clients — but disable asyncpg's prepared-statement cache when doing so (see the *asyncpg prepared-statement / connection-pooler errors* accordion above), and keep setup and migrations on the direct endpoint.
If the pool fills up even though your workload is small, check whether the connections are stuck in `idle in transaction` — see the accordion below.
A pool that exhausts itself well below the sizes above is usually not under-sized: it can be connection *overlap* inside a single request, where one call holds a pooled connection while acquiring a second one — the pool fills at roughly half the concurrency its size suggests, and deadlocks once concurrency reaches `pool_size + max_overflow`. Shrinking `POOL_ARGS` does not help there — it makes the deadlock arrive sooner. That class of overlap is fixed in version 1.4.2, so upgrade rather than resize if this is what you are seeing.
</Accordion>
<Accordion title="Connections stuck in idle in transaction, or the pool deadlocking under concurrency">
If Postgres accumulates backends sitting in `idle in transaction` and the pool eventually exhausts itself — with failures spreading to every request, authentication included, because the API-key lookup is itself a database query — several distinct Cognee-side causes produced that symptom, and all of them are fixed in version 1.4.2. Confirm the symptom from the server:
```sql theme={null}
SELECT state, count(*) FROM pg_stat_activity WHERE datname = 'cognee_db' GROUP BY state;
```
(Substitute your `DB_NAME` for `cognee_db` if you changed it.)
There is nothing to configure: upgrade to version 1.4.2 or later (see the [changelog](/changelog)) and redeploy. To verify, re-run the query above while authenticated traffic is flowing — authenticated requests should no longer hold a connection beyond their API-key lookup. Any `idle in transaction` backends that remain are not coming from the fixed causes. The remaining causes are:
* **Your own application code** holding a session open across a slow `await`.
* **Cognee's background pipeline runs** being abandoned at shutdown, a separate known issue.
</Accordion>
<Accordion title="DatabaseNotCreatedError (Postgres)">
For Postgres, the database named in `DB_NAME` must already exist before Cognee connects. Unlike SQLite, Cognee does **not** issue `CREATE DATABASE` for Postgres — it connects directly to `DB_NAME` and creates only the tables. If the database itself is missing, create it once with your Postgres tooling:
```bash theme={null}
createdb -h 127.0.0.1 -U cognee cognee_db
# or: psql -h 127.0.0.1 -U cognee -c "CREATE DATABASE cognee_db;"
```
(The built-in Docker Postgres service from `docker compose --profile postgres up -d` already creates this database for you.)
If you specifically see `DatabaseNotCreatedError` ("The database has not been created yet. Please call `await setup()` first."), Cognee reached Postgres but its tables (e.g. `principals`) don't exist yet. Run setup once to initialize the schema:
```python theme={null}
from cognee.modules.engine.operations.setup import setup
await setup()
```
`remember()` creates the tables automatically through its underlying `add()` and `cognify()` steps, so this typically only surfaces when calling `search()` or `recall()` first on a fresh database.
</Accordion>
## When to Use Each
* **SQLite**: Local development, single-user applications, simple deployments
* **Postgres**: Production environments, multi-user applications, external hosting, co-location with pgvector
* **Turso (libSQL)**: A SQLite drop-in when you want a hosted, replicated database — the same aiosqlite driver and Alembic migrations apply unchanged, with optional embedded-replica sync against a remote Turso primary
<Columns>
<Card title="Vector Stores" icon="database" href="/setup-configuration/vector-stores">
Configure vector databases for embedding storage
</Card>
<Card title="Graph Stores" icon="network" href="/setup-configuration/graph-stores">
Set up graph databases for knowledge graphs
</Card>
<Card title="Overview" icon="settings" href="/setup-configuration/overview">
Return to setup configuration overview
</Card>
</Columns>
# Security & Privacy
Source: https://docs.cognee.ai/setup-configuration/security
Protect sensitive data and access in self-hosted Cognee deployments.
Cognee exposes several environment variables that let you harden a self-hosted deployment for production use.
Some controls are enforced by default, while others remain permissive for local development, so you should review each setting before exposing Cognee to untrusted users or networks.
## Security Controls
<AccordionGroup>
<Accordion title="Authentication">
### Require authentication for all API requests
`ENABLE_BACKEND_ACCESS_CONTROL` is the canonical posture switch; `REQUIRE_AUTHENTICATION` is an optional override on the auth requirement alone.
| `ENABLE_BACKEND_ACCESS_CONTROL` | `REQUIRE_AUTHENTICATION` | Effective behavior |
| ------------------------------- | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `true` (default) | unset | Multi-tenant mode: per-user/dataset isolated DBs **and** API endpoints require an authenticated user. |
| `false` | unset | Single-user mode: shared DB **and** auth requirement off. |
| any | `true` | Auth is forced on (useful for a single-user deployment behind a shared token). |
| `false` | `false` | Auth off. |
| `true` | `false` | **Misconfiguration** — multi-tenant mode always requires auth. Cognee logs a warning at startup and forces `REQUIRE_AUTHENTICATION=true`. |
When neither variable is set, the default is multi-tenant mode with authentication required.
<Info>
At startup, Cognee logs an `auth posture: ...` line summarizing the resolved decision and the reason (default, inherited from `ENABLE_BACKEND_ACCESS_CONTROL`, explicit `REQUIRE_AUTHENTICATION`, or forced on by multi-tenant mode). Use this log line to verify what is actually in effect after deployment.
</Info>
<Warning>
If `ENABLE_BACKEND_ACCESS_CONTROL=true` (the default), authentication is **enforced automatically** regardless of the value of `REQUIRE_AUTHENTICATION`. Setting `REQUIRE_AUTHENTICATION=false` in this mode is ignored and a warning is logged at startup.
</Warning>
### JWT token settings
```dotenv theme={null}
FASTAPI_USERS_JWT_SECRET="super_secret" # default — CHANGE IN PRODUCTION
JWT_LIFETIME_SECONDS=3600 # default: 1 hour
```
`FASTAPI_USERS_JWT_SECRET` must be the same across all instances (e.g., all Kubernetes pods) so that a token issued by one pod is accepted by another. Use a long, randomly generated string in production and never commit the real value to version control.
`JWT_LIFETIME_SECONDS` controls how long a bearer token or cookie remains valid before the user must log in again.
### Default user credentials
When an operation runs without an explicit authenticated user, Cognee falls back to a built-in **default user** — created on first use as a superuser. This happens for SDK/library calls, and for HTTP requests when authentication is off (single-user mode). Override its credentials with:
```dotenv theme={null}
DEFAULT_USER_EMAIL="default_user@example.com" # default
DEFAULT_USER_PASSWORD="default_password" # default
```
When `ENABLE_BACKEND_ACCESS_CONTROL=true`, HTTP endpoints require an authenticated user, so the default user is not used to serve unauthenticated requests — but SDK calls still fall back to it. Set `DEFAULT_USER_EMAIL` and `DEFAULT_USER_PASSWORD` before the default user is first created to give this auto-created superuser known credentials you can then log in with. Changing these values later does not update an existing user; update or recreate that user separately, then restart the process so Cognee reads the new environment. See [Users](/core-concepts/multi-user-mode/permissions-system/users) for the full permission model.
</Accordion>
<Accordion title="Data Protection">
### API Key Storage
```dotenv theme={null}
HASH_API_KEY="False" # default
```
When `false`, API keys are stored as plaintext in the relational database.
When `true`, each key is hashed with **SHA-256** before storage. The raw key is shown to the user only once at creation time and cannot be recovered afterward.
<Warning>
**Migration note:** Enabling `HASH_API_KEY` on a running system that already has plaintext API keys stored will break those existing keys immediately — the lookup hashes the incoming value and finds no match. You must either delete and re-issue all existing keys, or run a one-off migration to SHA-256-hash the existing `api_key` column values.
</Warning>
### Local File System Access
```dotenv theme={null}
ACCEPT_LOCAL_FILE_PATH=True # default
```
When `true`, Cognee accepts local filesystem paths as data sources (e.g., `/etc/passwd`). This is convenient for local development but dangerous when Cognee is exposed as a multi-user backend — an authenticated user could read arbitrary files that the Cognee process has access to.
Set to `false` when running Cognee as a backend service:
```dotenv theme={null}
ACCEPT_LOCAL_FILE_PATH=False
```
The flag governs `file://` URIs and path strings that resolve to an existing local file; those inputs raise `IngestionError: Local files are not accepted.` when it is disabled. It is not a filter on submitted text: an absolute-looking string that does not point to an existing file is ingested as text content either way, since Cognee never treats it as a path.
#### Allowed local roots
Independently of `ACCEPT_LOCAL_FILE_PATH`, you can confine every local path Cognee dereferences to a set of **allowed roots**:
```dotenv theme={null}
COGNEE_ALLOWED_LOCAL_FILE_ROOTS=/srv/exports:/data/dumps # unset by default
```
The allowlist is **opt-in**. When the variable is unset — the default — there is no root restriction at all: any local path is readable, subject only to `ACCEPT_LOCAL_FILE_PATH`. That is what lets a local install ingest a repository or document tree from wherever it happens to live on the machine. **Set this variable on any deployment reachable by untrusted callers**; leaving it unset there means an authenticated caller can name any path the Cognee process can read.
When it is set, its entries are the allowed roots, separated by the platform path separator (`:` on Linux/macOS, `;` on Windows). Cognee's own `DATA_ROOT_DIRECTORY`, `SYSTEM_ROOT_DIRECTORY`, cache, log, and repository-clone (`COGNEE_REPOS_DIR`) directories are always appended to that list, so a restrictive value cannot lock Cognee out of its own storage. The clone root has to be on the list because a cloned repository's document files (README, docs) are ingested by path exactly like a local project's, and pass through this same check.
Paths are canonicalized with `realpath` before the containment check, so a symlink inside an allowed root that points outside it is rejected rather than followed. Once the variable is set, a path outside every allowed root raises:
```
ValueError: Local file path is outside allowed roots.
```
On the ordinary `add()` / `remember()` ingestion path, a path-looking string outside the allowed roots is never read from disk — it falls through and is ingested as plain text instead, though an explicit `file://` URI raises `IngestionError` with the same message rather than falling through. Callers that read a file explicitly, such as the [memory-migration sources](/examples/migrate-memory-systems), surface the error directly. A local repository path handed to the code-graph pipeline reports the same rejection as a `CodeRepositoryError` instead:
```
CodeRepositoryError: Repository path '<spec>' is outside the allowed local roots.
Add its root to COGNEE_ALLOWED_LOCAL_FILE_ROOTS to index it.
```
#### Presort scan roots
[Folder presort](/python-api/remember#folder-presort) is the one path that does **not** inherit the permissive default above. A scan opens every candidate file in a folder to hash it and sample it for personal data, so it always requires a bounded root set — even when `COGNEE_ALLOWED_LOCAL_FILE_ROOTS` is unset.
When the variable is set, its entries are the permitted roots, exactly as for ordinary ingestion. When it is unset, presort falls back to its own bounded default instead of "anywhere":
* the current working directory
* the system temporary directory
* Cognee's own `DATA_ROOT_DIRECTORY`, `SYSTEM_ROOT_DIRECTORY`, cache, log, and repository-clone roots
Paths are canonicalized with `realpath` before the containment check, so a symlink pointing out of a permitted root is rejected rather than followed. A folder outside every permitted root is refused rather than falling through to text ingestion — the scan raises `ValueError: Path is outside the allowed local file roots. Add the folder to COGNEE_ALLOWED_LOCAL_FILE_ROOTS (os.pathsep-separated list) to allow it, or use the CLI's --allow-root flag.` The same check guards reading a saved `*.presort.json` report back in, which surfaces the shorter `ValueError: Local file path is outside allowed roots.`
The CLI's `--allow-root` flag is the explicit opt-in for a folder outside those roots:
```bash theme={null}
cognee-cli remember ~/Downloads --presort --allow-root
```
<Warning>
`--allow-root` appends the folder you named to `COGNEE_ALLOWED_LOCAL_FILE_ROOTS` for the lifetime of that command, which widens what presort — and everything else running in that process — may read from disk. On a shared or untrusted machine, prefer setting `COGNEE_ALLOWED_LOCAL_FILE_ROOTS` deliberately to the exact folders you intend to scan.
</Warning>
Presort scanning is read-only: it never moves, renames, or deletes anything in the folder. Personal-data findings in the report are stored redacted (`j***@example.com`); the raw matched values are never written into the report.
### Cypher Query Access
```dotenv theme={null}
ALLOW_CYPHER_QUERY=True # default
```
When `true`, users can execute raw Cypher queries against the graph database (`SearchType.CYPHER`) and use natural language-to-Cypher translation (`SearchType.NATURAL_LANGUAGE`). Disable this to limit users to higher-level semantic search only:
```dotenv theme={null}
ALLOW_CYPHER_QUERY=False
```
### Outbound HTTP Requests (SSRF Protection)
```dotenv theme={null}
ALLOW_HTTP_REQUESTS=True # default
```
When Cognee ingests an `http(s)` URL (via `add()` → `save_data_item_to_storage`) or crawls a page, it fetches that URL **server-side**. To prevent Server-Side Request Forgery (SSRF, CWE-918), every user-supplied outbound URL is now validated before any request is made.
`ALLOW_HTTP_REQUESTS` (default `true`) gates outbound HTTP(S) fetching. Set it to `false` to disable all remote URL ingestion — any `http(s)` URL is then rejected:
```dotenv theme={null}
ALLOW_HTTP_REQUESTS=False
```
Falsey values are `false`, `0`, `no`, and `off` (case-insensitive); any other value leaves outbound fetching enabled.
When outbound requests are allowed, each URL still passes through these checks before it is fetched:
* **Scheme** — only `http` and `https` are permitted. Other schemes (e.g. `ftp://`, `gopher://`) are rejected.
* **Host** — the URL must contain a host, and that host must resolve. Hostless or unresolvable URLs are rejected.
* **Resolved address** — the host is resolved and *every* resolved IP is checked. The request is blocked if any address is loopback, private, link-local, reserved, multicast, unspecified, or IPv6 site-local. This blocks internal and cloud-metadata targets such as `127.0.0.1`, `::1`, `169.254.169.254`, `10.0.0.0/8`, `172.16.0.0/12`, and `192.168.0.0/16`. IP-literal URLs (e.g. `http://[::1]`) are validated directly, so DNS-rebinding and IP-literal bypasses are also caught.
To keep public URLs from redirecting the server to an internal address, the crawler's HTTP client runs with `follow_redirects=False` — redirects are not followed automatically.
A GitHub/GitLab repository URL is fetched server-side too, by `git` rather than by the HTTP client: `add()` and `remember()` shallow-clone it into `COGNEE_REPOS_DIR` (default `~/.cognee/repos`) and index it as a code graph. The clone URL clears the same outbound checks as any other `http(s)` ingestion, and `ALLOW_HTTP_REQUESTS=false` blocks it. On a deployment reachable by untrusted callers, size the disk behind `COGNEE_REPOS_DIR` for the clones it will accumulate — they are kept and reused across calls, not deleted after ingestion.
<Warning>
A blocked or disabled request raises `SSRFProtectionError` (an HTTP **403** `CogneeValidationError`) instead of silently fetching the internal target. If you see this during ingestion, check that the URL is a public `http(s)` address that resolves to a routable IP, and that `ALLOW_HTTP_REQUESTS` is not set to `false`.
</Warning>
### Encrypting Neo4j Credentials
When using the `neo4j_aura_dev` or `neo4j_community` dataset database handler for multi-user mode, Cognee stores per-dataset Neo4j connection info in the relational database — for `neo4j_aura_dev` the provisioned Aura instance, for `neo4j_community` the per-dataset Docker container. The stored **database password** is encrypted with **Fernet symmetric encryption**; both handlers derive the encryption key from the same `NEO4J_ENCRYPTION_KEY`:
```dotenv theme={null}
NEO4J_ENCRYPTION_KEY="test_key" # default — CHANGE IN PRODUCTION
```
The default value `"test_key"` is intentionally insecure. Replace it with a long random string in any environment that stores real Neo4j credentials.
<Info>
The Aura API credentials used to create or delete instances (`NEO4J_CLIENT_ID`, `NEO4J_CLIENT_SECRET`, and `NEO4J_TENANT_ID`) are read from environment variables when needed and are **not** stored in the relational database by this handler.
</Info>
### Encrypting Integration Credentials
Third-party OAuth integrations (Slack, GitHub, and Linear) store their tokens in the relational `integration_credentials` table. The token payload is encrypted with **AES-256-GCM** under a fresh random 96-bit nonce per write; only non-secret metadata — the workspace label, Slack team and enterprise ids, the bot and installing member's user ids, the granted OAuth scopes, and the allowed-channel list — is stored in the clear.
Keys are configured as a **keyring** — a set of 32-byte keys, each addressed by a short key id — so that keys can be rotated without re-encrypting existing rows:
```dotenv theme={null}
INTEGRATION_CREDENTIALS_KEYS='{"1": "<base64-encoded 32-byte key>"}'
INTEGRATION_CREDENTIALS_ACTIVE_KEY_ID="1" # default
```
| Variable | Meaning |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
| `INTEGRATION_CREDENTIALS_KEYS` | JSON object of every currently decryptable key, `{"<key_id>": "<base64 key>"}`. Preferred. |
| `INTEGRATION_CREDENTIALS_ACTIVE_KEY_ID` | The key id new rows are written under; it must exist in the ring. Defaults to `"1"`. |
| `INTEGRATION_CREDENTIALS_KEY` | Legacy single-key fallback, used only when `INTEGRATION_CREDENTIALS_KEYS` is unset. Loaded into the ring under id `"1"`. |
There is no default key and no derived fallback: if neither variable is set, or a key does not decode to exactly 32 bytes, credential writes fail with a `RuntimeError` rather than producing ciphertext nobody can decrypt later. Generate a key with:
```bash theme={null}
python -c "import base64, os; print(base64.b64encode(os.urandom(32)).decode())"
```
**Rotation.** Add the new key to `INTEGRATION_CREDENTIALS_KEYS` alongside the old one under a new id, then point `INTEGRATION_CREDENTIALS_ACTIVE_KEY_ID` at it. Every row records the key id it was written with, so existing rows keep decrypting under their original key; reconnecting an integration rewrites its row under the active key. Remove an old key from the ring only once no row still references it.
**Revocation.** Disconnecting an integration marks the stored credential revoked and makes a best-effort call to the provider's revoke endpoint. Provider-side removal has the same effect: for Slack, an `app_uninstalled` or `tokens_revoked` event deactivates the stored installation, so no usable token survives an uninstall.
</Accordion>
<Accordion title="Multi-User Isolation">
### Dataset & Multi-User Isolation
```dotenv theme={null}
ENABLE_BACKEND_ACCESS_CONTROL=True # default
```
When enabled, Cognee creates isolated storage per user + dataset combination and enforces permission checks on every read and write operation. This is the primary control for preventing cross-tenant data leakage in multi-user deployments.
Database support requirements:
| Layer | Supported backends |
| ---------- | ------------------------------------------- |
| Relational | SQLite, PostgreSQL |
| Vector | LanceDB, PGVector |
| Graph | Kuzu, Neo4j Aura (`neo4j_aura_dev` handler) |
If you configure an unsupported backend (e.g., Qdrant, Weaviate), disable access control to avoid runtime errors:
```dotenv theme={null}
ENABLE_BACKEND_ACCESS_CONTROL=False
```
Setting `ENABLE_BACKEND_ACCESS_CONTROL=false` alone also disables the auth requirement (single-user mode). You only need to add `REQUIRE_AUTHENTICATION` if you want to override that default — for example, `REQUIRE_AUTHENTICATION=true` to keep auth on for a single-user deployment behind a shared token.
See [Dataset Database Handlers](/core-concepts/multi-user-mode/dataset-database-handlers/dataset-database-handlers-what-are-they) for the full list of supported handlers.
</Accordion>
<Accordion title="Disabling Authentication for Local Development">
When running Cognee locally for development or testing, you can disable authentication so that API calls succeed without a bearer token. Setting the single posture switch is enough:
```dotenv theme={null}
ENABLE_BACKEND_ACCESS_CONTROL=false
```
With `ENABLE_BACKEND_ACCESS_CONTROL=false` and `REQUIRE_AUTHENTICATION` unset, `REQUIRE_AUTHENTICATION` inherits from the posture switch and the auth requirement is turned off automatically. (Previously, both variables had to be set to `false` independently; that is no longer required.)
These values are read once when the server process starts, so you must restart the server after changing them. Check the `auth posture: ...` line in the startup logs to confirm the resolved decision.
When authentication is disabled, unauthenticated requests are automatically served under a built-in default user (`default_user@example.com`). The relational database must be initialized before the first request so that this default user can be looked up or created.
**Troubleshooting 401 errors after setting the variables**
| Symptom | Cause | Fix |
| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| Still getting `401` on all endpoints | `ENABLE_BACKEND_ACCESS_CONTROL` is still `true`, or `REQUIRE_AUTHENTICATION=true` is set explicitly and overrides it | Set `ENABLE_BACKEND_ACCESS_CONTROL=false` and unset (or also set to `false`) `REQUIRE_AUTHENTICATION` |
| Startup log says auth was "forced on by multi-tenant mode" | `REQUIRE_AUTHENTICATION=false` is combined with `ENABLE_BACKEND_ACCESS_CONTROL=true` — an unsafe combination that Cognee coerces to auth-on | To disable auth, also set `ENABLE_BACKEND_ACCESS_CONTROL=false` |
| `401` persists after editing `.env` | Server not restarted | Restart the server — the variables are evaluated at import time |
| `500 Failed to create default user` | Relational DB not initialized | Call `cognee.prune.prune_system()` once, or let the server run its startup migrations before sending requests |
<Warning>
Only use `ENABLE_BACKEND_ACCESS_CONTROL=false` in local or trusted environments unless you have another protection layer in place. It disables multi-user isolation, and it also disables the HTTP auth requirement unless you explicitly set `REQUIRE_AUTHENTICATION=true`.
</Warning>
</Accordion>
</AccordionGroup>
## Recommended Production Settings
<AccordionGroup>
<Accordion title="Recommended Production Settings">
```dotenv theme={null}
# Authentication
REQUIRE_AUTHENTICATION=True
FASTAPI_USERS_JWT_SECRET="<random-64-char-string>"
JWT_LIFETIME_SECONDS=3600
# API key security
HASH_API_KEY=True
# Multi-user isolation
ENABLE_BACKEND_ACCESS_CONTROL=True
# Prevent arbitrary file reads (set False for backend deployments)
ACCEPT_LOCAL_FILE_PATH=False
# Confine local path reads to specific roots — unset means any path the process can read.
# Only relevant when local paths must be accepted (e.g. ingesting a mounted volume).
# COGNEE_ALLOWED_LOCAL_FILE_ROOTS=/srv/exports:/data/dumps
# Limit direct graph queries (optional — set False to restrict to semantic search)
ALLOW_CYPHER_QUERY=False
# Outbound URL ingestion is validated against SSRF; set False to disable remote fetches entirely
ALLOW_HTTP_REQUESTS=True
# Neo4j credential encryption (only required when using the neo4j_aura_dev or neo4j_community handler)
NEO4J_ENCRYPTION_KEY="<random-64-char-string>"
```
<Info>
For detailed instructions on the multi-user permission system (users, tenants, roles, and ACL), see [Cognee Permissions System](/core-concepts/multi-user-mode/permissions-system/overview).
</Info>
</Accordion>
</AccordionGroup>
<Columns>
<Card title="Permissions Setup" icon="lock" href="/setup-configuration/permissions">
Enable dataset isolation and access control
</Card>
<Card title="Multi-User Mode" icon="users" href="/core-concepts/multi-user-mode/multi-user-mode-overview">
Understand multi-tenant architecture
</Card>
</Columns>
# Structured Output Backends
Source: https://docs.cognee.ai/setup-configuration/structured-output-backends
Configure structured output frameworks for reliable data extraction in Cognee
Structured output backends ensure reliable data extraction from LLM responses. Cognee supports three frameworks that convert LLM text into structured Pydantic models for knowledge graph extraction and other tasks.
<Info>
**New to configuration?**
See the [Setup Configuration Overview](./overview) for the complete workflow:
install extras → create `.env` → choose providers → handle pruning.
</Info>
## Supported Frameworks
Cognee supports three structured output approaches:
* **LiteLLM Native** — Validates responses into Pydantic models using LiteLLM's own `response_format`, without the `instructor` dependency (default)
* **LiteLLM + Instructor** — Provider-agnostic client with Pydantic coercion (opt-in)
* **BAML** — DSL-based framework with type registry and guardrails (opt-in)
All three frameworks produce the same Pydantic-validated outputs, so your application code remains unchanged regardless of which backend you choose.
## How It Works
Cognee uses a unified interface that abstracts the underlying framework:
```python theme={null}
from cognee.infrastructure.llm.LLMGateway import LLMGateway
await LLMGateway.acreate_structured_output(text, system_prompt, response_model)
```
The `STRUCTURED_OUTPUT_FRAMEWORK` environment variable determines which backend processes your requests, but the API remains identical. It defaults to `litellm_native`, so `instructor` is no longer in the default call path — set the variable explicitly to opt back into `instructor` or into `baml`.
## Configuration
<Tabs>
<Tab title="LiteLLM + Instructor">
An opt-in framework — no extra install needed, but you must select it explicitly since the default is `litellm_native`. Uses LiteLLM and the `instructor` library to coerce LLM responses into Pydantic models.
```dotenv theme={null}
STRUCTURED_OUTPUT_FRAMEWORK=instructor
```
Optionally, control how the model is prompted for structured output:
```dotenv theme={null}
# Override instructor mode (e.g. json_mode, tool_call, markdown_json_mode)
# Leave unset to use the provider's default — see "Instructor Modes" below.
LLM_INSTRUCTOR_MODE=json_schema_mode
```
</Tab>
<Tab title="BAML">
BAML is an alternative structured output framework that uses a DSL-based type registry to extract data. It is particularly useful when small local models (such as Ollama models like `llama3.1:8b` or `qwen3.5:0.8b`) struggle to produce valid structured output with instructor, causing repeated `InstructorRetryException` errors.
**Installation**: BAML requires a separate install:
```bash theme={null}
pip install "cognee[baml]"
```
**Configuration**: BAML uses its own LLM settings, independent of the main `LLM_*` variables:
```dotenv theme={null}
STRUCTURED_OUTPUT_FRAMEWORK=baml
# BAML-specific LLM settings (required)
BAML_LLM_PROVIDER=openai
BAML_LLM_MODEL=gpt-4o-mini
BAML_LLM_API_KEY=sk-...
# Optional BAML overrides
# BAML_LLM_ENDPOINT=https://api.openai.com/v1
# BAML_LLM_API_VERSION=
# BAML_LLM_TEMPERATURE=0.0
```
`BAML_LLM_PROVIDER` and `BAML_LLM_MODEL` accept the same provider names and model identifiers as the main LLM configuration. You can point BAML at a different model than your main LLM — for example, use a small Ollama model for general text generation while routing structured extraction through a cloud model.
<Warning>
If `STRUCTURED_OUTPUT_FRAMEWORK=baml` is set but the `cognee[baml]` extra is not installed, Cognee will raise an `ImportError` on startup. Run `pip install "cognee[baml]"` to resolve it.
</Warning>
<Accordion title="Using BAML for Small Local Models">
Small Ollama models (e.g. `llama3.1:8b`, `qwen3.5:0.8b`) often fail to produce valid JSON-structured output when using the instructor backend, resulting in repeated `InstructorRetryException` errors during `cognify` for types like `KnowledgeGraph` or `SummarizedContent`.
Switching to BAML bypasses instructor entirely and uses BAML's own extraction pipeline, which is more forgiving with smaller models:
```dotenv theme={null}
# Main LLM — small local model via Ollama
LLM_PROVIDER=ollama
LLM_MODEL=llama3.1:8b
LLM_ENDPOINT=http://localhost:11434/v1
LLM_API_KEY=ollama
# Use BAML for structured extraction (can point to a different, more capable model)
STRUCTURED_OUTPUT_FRAMEWORK=baml
BAML_LLM_PROVIDER=openai
BAML_LLM_MODEL=gpt-4o-mini
BAML_LLM_API_KEY=sk-...
```
See the [Local Setup guide](/guides/local-setup) for a complete Ollama configuration including embeddings.
</Accordion>
</Tab>
<Tab title="LiteLLM Native (Default)">
LiteLLM Native is the default framework. It validates responses into Pydantic models using LiteLLM's own `response_format`, **without** the `instructor` library. No extra install is needed beyond the base package, and no configuration is needed to use it — set the variable only if you want to pin the value explicitly:
```dotenv theme={null}
STRUCTURED_OUTPUT_FRAMEWORK=litellm_native
```
It reuses the standard `LLM_*` settings (provider, model, API key, endpoint) — no separate configuration block. Behavior depends on the provider:
* **Schema-native path** — For providers LiteLLM reports as schema-capable (OpenAI, Azure, Gemini, Mistral, Bedrock, and others), your Pydantic model is passed directly as `response_format` and the returned JSON is validated against it. This path has two tiers, **strict** and **non-strict** (see below).
* **JSON-object fallback** — For providers without native schema support (Ollama, llama.cpp, custom OpenAI-compatible endpoints), Cognee requests a JSON object, injects the model's JSON Schema into the prompt, and validates the result. Before validating, Cognee unwraps a markdown code fence if one wraps the whole reply — models on this path routinely answer with ` ```json … ``` ` even when asked not to. The unwrap is anchored to the entire payload, so JSON that merely *contains* backticks inside a value is left untouched. On a validation failure it feeds the error back to the model and retries so it can self-correct, up to 3 attempts before raising the last validation error.
Routing between the two paths is automatic per model (via `litellm.supports_response_schema`); you don't select the path yourself.
**Strict and non-strict tiers on the schema-native path.** Cognee first asks for **strict** schema enforcement, where the provider constrains decoding to your schema. Providers only enforce a restricted subset of JSON Schema, so a model they cannot enforce is rejected with a schema-related `400` — typically a discriminated union (which emits `oneOf`/`discriminator`) or a free-form dict field (which OpenAI 400s with `'additionalProperties' is required to be supplied and to be false`). Rather than giving up on the native path there, Cognee retries the same request once with a **non-strict** `json_schema` payload: the raw `model_json_schema()` travels as guidance rather than as a decoding constraint, so those constructs are accepted, and conformance is still checked app-side by validating against your original Pydantic model. Only if the non-strict payload is *also* rejected does Cognee fall through to the JSON-object fallback.
The demotion is remembered for the process. The `(model, response model)` pair is recorded in a module-level set, so the rejected strict request is paid **once per process instead of on every call** — later calls for that pair skip the strict attempt and go straight to non-strict. Models whose schema strict mode accepts (the majority, including `KnowledgeGraph` extraction) keep their strict, grammar-constrained guarantee untouched. Both tiers log a warning naming the model and response model when they are rejected.
Output that comes back from either tier but fails Pydantic validation also routes to the JSON-object fallback, which has the error-feedback retry loop, rather than bubbling into the transport-level retry that would re-send the same prompt with no correction.
<Note>
Neither the tiering nor the demotion cache has a configuration knob or environment variable — there is no setting that forces strict mode or disables the non-strict retry. The cache lives in process memory only, so it starts empty on every restart.
</Note>
**Model-name qualification.** LiteLLM routes on a *provider-qualified* model name, while Cognee keeps provider and model as separate settings — so `LLM_PROVIDER="ollama"` with `LLM_MODEL="phi4"`, a configuration that is valid for `instructor`, would reach LiteLLM as a bare `phi4` and fail with `litellm.BadRequestError: LLM Provider NOT provided` before a request is even sent. On this path Cognee first asks LiteLLM to resolve `LLM_MODEL` on its own, and only when that lookup fails does it prefix the name with the configured provider:
| `LLM_MODEL` (with `LLM_PROVIDER="ollama"`) | Sent to LiteLLM |
| -------------------------------------------- | --------------------------------------------------- |
| `phi4` | `ollama/phi4` |
| `library/phi4` | `ollama/library/phi4` |
| `hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF` | `ollama/hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF` |
| `ollama/phi4` | `ollama/phi4` — already resolvable, untouched |
Prefixing applies to `ollama`, `anthropic`, `gemini`, `mistral`, and `bedrock`. `openai` and `azure` are deliberately excluded: LiteLLM already resolves bare OpenAI model names, and Azure needs a deployment-specific form Cognee must not guess at. A slash does **not** by itself mean the name carries a provider — Ollama accepts namespaced tags and a GGUF pulled from Hugging Face keeps its full path — so slash-containing names go through the same check rather than being assumed routable. Anything LiteLLM already resolves is returned unchanged (`gpt-4o`, `openai/gpt-5-mini`, `azure/gpt-4o-mini`), so qualification cannot re-route a configuration that works today; the only cost is one extra local `get_llm_provider` lookup, with no network call. Any other provider is left as-is — if LiteLLM rejects its name, qualify `LLM_MODEL` yourself.
Qualification reads `LLM_PROVIDER`, so a namespaced model id needs the provider set explicitly. `LLM_MODEL="library/phi4"` on its own raises `ProviderNotDeducibleError` at configuration load, before qualification ever runs — `library` and `hf.co` are not provider prefixes Cognee recognises. See [Provider Inference](/setup-configuration/llm-providers#provider-inference).
<Note>
This is the framework Cognee uses when `STRUCTURED_OUTPUT_FRAMEWORK` is unset. Transient errors (including rate limits) are retried with backoff, while budget/quota exhaustion is surfaced as `LLMPaymentRequiredError` and is **not** retried. Content-policy violations fall back only when the standard fallback settings are configured (at minimum `FALLBACK_MODEL` and `FALLBACK_API_KEY`); otherwise Cognee raises `ContentPolicyFilterError` immediately.
</Note>
</Tab>
</Tabs>
## Instructor Modes
When `STRUCTURED_OUTPUT_FRAMEWORK=instructor`, the **instructor mode** controls *how* Cognee asks the model for structured output — for example via the model's native JSON-schema response, a plain JSON object, or a tool/function call. The value of `LLM_INSTRUCTOR_MODE` is passed directly to the `instructor` library's `Mode`, so it must be one of instructor's supported mode strings.
`LLM_INSTRUCTOR_MODE` is **empty by default**. When it is unset, Cognee either applies a provider-specific mode or defers to the underlying Instructor/LiteLLM default, so in most cases you don't need to set it at all:
| `LLM_PROVIDER` | Behavior when `LLM_INSTRUCTOR_MODE` is unset |
| ------------------------------------------------- | -------------------------------------------- |
| `openai`, `azure` with `gpt-5` models | `json_schema_mode` |
| `openai`, `azure` with other models | use Instructor/LiteLLM default |
| AWS Bedrock | `json_schema_mode` |
| `ollama` | `json_schema_mode` |
| `gemini`, `custom` (OpenAI-compatible), llama.cpp | `json_mode` |
| `anthropic` | `anthropic_tools` |
| `mistral` | `mistral_tools` |
Common values you can set explicitly include `json_schema_mode`, `json_mode`, `tool_call`, and `markdown_json_mode`.
Ollama's default is `json_schema_mode` — it was `json_mode` in earlier Cognee versions. Ollama 0.5+ enforces the Pydantic schema as a decoder constraint instead of only being asked for "some JSON", so local models fail first-attempt validation (and trigger `InstructorRetryException`) far less often. On an Ollama older than 0.5, which has no JSON-schema enforcement, set `LLM_INSTRUCTOR_MODE=json_mode` to restore the previous behavior.
<Tip>
**Which mode for OpenAI models (e.g. `gpt-5-mini`)?** Leave `LLM_INSTRUCTOR_MODE` unset, or set `json_schema_mode` — Cognee applies `json_schema_mode` to `gpt-5` models, and it is the recommended mode for OpenAI models that support native JSON-schema responses. Only override it when you point Cognee at a custom or local OpenAI-compatible endpoint that rejects JSON-schema responses; in that case try `json_mode` first, then `markdown_json_mode` or `tool_call`.
</Tip>
## Setting Structured Output in a Script
You don't have to use `.env` — the same settings can be configured directly in Python. Both the framework and the instructor mode are attributes on the internal `LLMConfig`.
<Tabs>
<Tab title="set_llm_config">
Pass the exact attribute names (`structured_output_framework`, `llm_instructor_mode`) to `cognee.config.set_llm_config()`:
```python theme={null}
import cognee
cognee.config.set_llm_config({
"structured_output_framework": "instructor", # or "baml", "litellm_native"
"llm_instructor_mode": "json_mode", # any instructor Mode string
})
```
</Tab>
<Tab title="os.environ">
Set the environment variables **before** importing cognee:
```python theme={null}
import os
os.environ["STRUCTURED_OUTPUT_FRAMEWORK"] = "instructor"
os.environ["LLM_INSTRUCTOR_MODE"] = "json_mode"
import cognee # reads the variables on first config access
```
</Tab>
</Tabs>
<Warning>
Switching to **BAML** at runtime via `set_llm_config()` does not initialize BAML's client registry, which is built when the config is first constructed. To use BAML, set `STRUCTURED_OUTPUT_FRAMEWORK=baml` (and the `BAML_LLM_*` variables) in `.env` or via `os.environ` **before** importing cognee.
</Warning>
## Important Notes
* **Unified Interface**: Your application code uses the same `acreate_structured_output()` call regardless of framework
* **Provider Flexibility**: LiteLLM + Instructor and LiteLLM Native reuse the standard `LLM_*` provider settings; BAML uses its own `BAML_LLM_*` block
* **Output Consistency**: All three produce Pydantic-validated results
* **Performance**: Framework choice doesn't significantly impact performance
## Troubleshooting
<Accordion title="`1 validation error for Response: content Input should be a valid string ... input_type=dict`">
This error appears during `recall()` / `search()` with completion search types such as `GRAPH_COMPLETION`, `GRAPH_SUMMARY_COMPLETION`, `GRAPH_COMPLETION_COT`, and `RAG_COMPLETION`.
**Cause.** These search types ask the LLM for a plain-text answer (the retriever uses `response_model=str`). When the configured instructor mode doesn't match what your model/provider actually supports, the model wraps its answer in a JSON object instead of returning plain text. The `instructor` backend then can't coerce that `dict` into the expected string field, so Pydantic raises `Input should be a valid string ... input_type=dict`. This is common with OpenAI-compatible, custom, and local (Ollama / LM Studio) endpoints.
**Fixes:**
* **Align the instructor mode with your provider.** OpenAI/Azure `gpt-4o`/`gpt-5` models work with the default `json_schema_mode`, and so does Ollama 0.5+, which now defaults to it as well. Endpoints that don't support JSON-schema responses — including an Ollama older than 0.5 — usually need a different mode:
```dotenv theme={null}
# Try one of these for OpenAI-compatible / local endpoints
LLM_INSTRUCTOR_MODE=json_mode
# LLM_INSTRUCTOR_MODE=markdown_json_mode
# LLM_INSTRUCTOR_MODE=tool_call
```
* **Switch to BAML** if a small/local model keeps wrapping answers in JSON. BAML bypasses instructor's coercion and is more forgiving of loose model output:
```dotenv theme={null}
STRUCTURED_OUTPUT_FRAMEWORK=baml
BAML_LLM_PROVIDER=openai
BAML_LLM_MODEL=gpt-4o-mini
BAML_LLM_API_KEY=sk-...
# For local/OpenAI-compatible endpoints:
# BAML_LLM_ENDPOINT=http://localhost:11434/v1
# BAML_LLM_API_KEY=ollama
```
* **Skip the LLM completion step** to confirm retrieval works independently of model output formatting. Pass `only_context=True` to return the retrieved context directly — see [Search Basics](/guides/search-basics). If retrieval succeeds with `only_context=True`, the problem is the structured-output configuration above, not your graph.
</Accordion>
<Accordion title="`1 validation error for SummarizedContent: description Input should be a valid string ... input_type=list`">
This error appeared during `cognify()` (chunk summarization) on small local models — Ollama and llama.cpp setups were the usual reporters.
**Cause.** `SummarizedContent` has two fields: `summary`, which Cognee actually consumes, and `description`, which is unused and kept only for backwards compatibility. Both are still part of the JSON Schema handed to the model, so a model is free to fill `description` — and smaller models routinely answered it with a list of bullets instead of a string. Strict validation then failed the whole structured-output call, the retries exhausted, and an entire `cognify()` run died on a field nothing reads.
**Fix.** Upgrade Cognee — `description` now coerces whatever the model returns instead of rejecting it: `None` becomes `""`, a list or tuple is joined with newlines, and anything else is stringified. `summary`, the field that is actually consumed, keeps strict validation, so a model that fails to produce a usable summary still surfaces as an error. This is a model-level change, so it applies to every structured-output backend; no configuration change is required.
</Accordion>
<Accordion title="`AttributeError: type object 'str' has no attribute 'model_json_schema'` (Gemini)">
This error is raised from `GeminiAdapter.acreate_structured_output` during `recall()` / `search()` with completion search types (`GRAPH_COMPLETION`, `RAG_COMPLETION`, etc.).
**Cause.** These search types request a plain-text answer with `response_model=str`. Gemini's default instructor mode is `json_mode`, which handles `str` correctly. If you override `LLM_INSTRUCTOR_MODE` with a schema- or tool-based mode (`json_schema_mode`, `tool_call`, `mistral_tools`, …), Instructor tries to call `str.model_json_schema()` — a method that only exists on Pydantic models — and crashes.
**Fix — force `json_mode` for Gemini.** Either leave `LLM_INSTRUCTOR_MODE` unset so Gemini falls back to its `json_mode` default, or set it explicitly:
```dotenv theme={null}
LLM_PROVIDER="gemini"
LLM_MODEL="gemini/gemini-2.0-flash"
LLM_API_KEY="AIza..."
LLM_INSTRUCTOR_MODE="json_mode"
```
To set it in-script, pass it through `set_llm_config` — `llm_instructor_mode` is a valid key on the LLM config:
```python theme={null}
import cognee
cognee.config.set_llm_config({"llm_instructor_mode": "json_mode"})
```
<Warning>
`cognee.config.set("llm_instructor_mode", "json_mode")` raises `InvalidConfigAttributeError: 'llm_instructor_mode' is not a valid attribute of the configuration` — the generic `config.set()` only accepts a fixed set of keys. Use `cognee.config.set_llm_config({...})` (or the `LLM_INSTRUCTOR_MODE` env var) instead.
</Warning>
See [LLM Instructor Modes](/setup-configuration/llm-providers#llm-instructor-modes) for the full list of modes and per-provider defaults.
</Accordion>
<Accordion title="`ValueError: Unsupported type for BAML mapping: str | None`">
This error is raised with `STRUCTURED_OUTPUT_FRAMEWORK=baml` while BAML builds a dynamic type for a response model that has a PEP 604 optional field (`X | None`). In practice it surfaces during `recall()` / `search()` with completion search types such as `GRAPH_COMPLETION`, because the completion response model contains `str | None` fields — so recall fails on local/Ollama + BAML setups.
**Cause.** BAML's dynamic type builder previously recognized only `typing.Union` / `typing.Optional`. PEP 604 unions written as `X | None` have a different origin (`types.UnionType`), so they missed the Optional/Union branch and fell through to the unsupported-type error. `typing.Optional[str]` worked; the equivalent `str | None` did not.
**Fix.** Upgrade Cognee — BAML now maps PEP 604 `X | None` unions the same way it maps `typing.Optional`, so optional fields work with either syntax. No configuration change is required.
</Accordion>
<Accordion title="`Invalid JSON: expected value at line 1 column 1` (litellm_native)">
This error is raised with `STRUCTURED_OUTPUT_FRAMEWORK=litellm_native` on the JSON-object fallback path — the path every provider without native schema support lands on (Ollama, llama.cpp, custom OpenAI-compatible endpoints, and any model LiteLLM does not report as schema-capable). It shows up during `cognify()` for response models such as `SummarizedContent` or `KnowledgeGraph`:
````
ValidationError: 1 validation error for SummarizedContent
Invalid JSON: expected value at line 1 column 1
[input_value='```json\n{\n "summary":...\n}\n```', input_type=str]
````
**Cause.** The model wrapped otherwise-valid JSON in a markdown code fence. Pydantic's `model_validate_json()` sees a backtick at column 1 and rejects the whole reply. The self-correction retry could not rescue it either — a model that fences once fences again — so all three attempts were spent and the run failed.
**Fix.** Upgrade Cognee — the fallback path now strips a code fence that wraps the entire reply before validating, and leaves backticks inside JSON values alone. No configuration change is required.
</Accordion>
<Columns>
<Card title="LLM Providers" icon="brain" href="/setup-configuration/llm-providers">
Configure LLM providers for text generation
</Card>
<Card title="Overview" icon="settings" href="/setup-configuration/overview">
Return to setup configuration overview
</Card>
<Card title="Custom Prompts" icon="text-wrap" href="/guides/custom-prompts">
Learn about custom prompt configuration
</Card>
</Columns>
# Vector Stores
Source: https://docs.cognee.ai/setup-configuration/vector-stores
Configure vector databases for embedding storage and semantic search in Cognee
Vector stores hold embeddings for semantic similarity search. They enable Cognee to find conceptually related content based on meaning rather than exact text matches.
<Info>
**New to configuration?**
See the [Setup Configuration Overview](./overview) for the complete workflow:
install extras → create `.env` → choose providers → handle pruning.
For a complete, copy-paste `.env` block that combines this layer with a relational and a graph store, see [Store Configurations](/guides/store-configurations).
</Info>
## Supported Providers
Cognee supports multiple vector store options through built-in providers and community-maintained adapters:
| Provider | Status | Notes |
| --------------------- | ------------------------ | ----------------------------------------------------------------------- |
| **LanceDB** | Built-in, default | File-based vector store, works out of the box |
| **PGVector** | Built-in extra | Postgres-backed vector storage with pgvector extension |
| **Turso (libSQL)** | Built-in extra | libSQL vector store; embedded local file or remote Turso cloud |
| **Neptune Analytics** | Built-in extra | Amazon Neptune Analytics hybrid solution |
| **ChromaDB** | Optional extra / adapter | HTTP server-based vector database; may require installing extra support |
| **Qdrant** | Community adapter | High-performance vector database and similarity search engine |
| **Redis** | Community adapter | Fast vector similarity search via Redis Search module |
| **FalkorDB** | Community adapter | Hybrid graph + vector database |
## Configuration
<Accordion title="Environment Variables">
Community adapters must still be installed and registered in your application startup code before Cognee can use their provider value.
<Tabs>
<Tab title="Local Path">
Use this shape for LanceDB.
```dotenv theme={null}
VECTOR_DB_PROVIDER="lancedb"
# Optional path or URL. Defaults to <SYSTEM_ROOT_DIRECTORY>/databases/cognee.lancedb
VECTOR_DB_URL="/absolute/or/relative/path/to/cognee.lancedb"
# Optional. Defaults to true.
VECTOR_DB_SUBPROCESS_ENABLED="true"
# Optional. Max concurrent async RPCs in flight per subprocess worker.
# Defaults to 16. Must be > 0 (worker init raises ValueError otherwise).
SUBPROCESS_WORKER_MAX_INFLIGHT="16"
# Optional. Seconds an idle subprocess engine is kept alive before its worker
# is closed. Defaults to 600. Set to 0 to close at every dataset-context exit.
SUBPROCESS_IDLE_TTL_SECONDS="600"
```
</Tab>
<Tab title="Postgres">
Use this shape for PGVector.
```dotenv theme={null}
VECTOR_DB_PROVIDER="pgvector"
VECTOR_DB_HOST="localhost"
VECTOR_DB_PORT="5432"
VECTOR_DB_NAME="cognee_db"
VECTOR_DB_USERNAME="cognee"
VECTOR_DB_PASSWORD="cognee"
# Optional SQLAlchemy pool args JSON.
VECTOR_POOL_ARGS='{"pool_size": 2, "max_overflow": 2}'
```
If the explicit `VECTOR_DB_*` Postgres values are omitted, Cognee falls back to the relational `DB_HOST`, `DB_PORT`, `DB_NAME`, `DB_USERNAME`, and `DB_PASSWORD` settings. When backend access control is enabled, configure the explicit `VECTOR_DB_*` values.
</Tab>
<Tab title="Turso">
Use this shape for Turso / libSQL. One adapter handles both modes.
Embedded (local file):
```dotenv theme={null}
VECTOR_DB_PROVIDER="turso"
VECTOR_DB_URL="/absolute/path/to/cognee.turso.db"
```
Remote Turso cloud:
```dotenv theme={null}
VECTOR_DB_PROVIDER="turso"
VECTOR_DB_URL="libsql://your-db.turso.io"
VECTOR_DB_KEY="your_turso_auth_token"
```
A URL scheme of `libsql://`, `http(s)://`, or `ws(s)://` is treated as a remote server (auth token sent via `VECTOR_DB_KEY`); any other value is treated as a local embedded file path. Requires the `cognee[turso]` extra.
</Tab>
<Tab title="Neptune">
Use this shape for Neptune Analytics.
```dotenv theme={null}
VECTOR_DB_PROVIDER="neptune_analytics"
VECTOR_DB_URL="neptune-graph://<GRAPH_ID>"
# Required — no per-dataset database handler exists for Neptune Analytics
ENABLE_BACKEND_ACCESS_CONTROL="false"
```
AWS credentials are resolved through the environment or the default AWS SDK chain.
</Tab>
<Tab title="ChromaDB">
Use this shape for ChromaDB.
```dotenv theme={null}
VECTOR_DB_PROVIDER="chromadb"
VECTOR_DB_URL="http://localhost:8000"
VECTOR_DB_KEY=""
```
ChromaDB support may require installing the optional extra or adapter before setting `VECTOR_DB_PROVIDER="chromadb"`.
</Tab>
<Tab title="Community Adapters">
Use this shape for community adapters that connect through a URL.
Provider values:
* `qdrant`
* `redis`
* `pinecone`
* `falkor`
```dotenv theme={null}
VECTOR_DB_PROVIDER="qdrant"
VECTOR_DB_URL="http://localhost:6333"
VECTOR_DB_KEY=""
```
Use this shape for FalkorDB:
```dotenv theme={null}
VECTOR_DB_PROVIDER="falkor"
VECTOR_DB_URL="localhost"
VECTOR_DB_PORT="6379"
GRAPH_DATABASE_PROVIDER="falkor"
GRAPH_DATABASE_URL="localhost"
GRAPH_DATABASE_PORT="6379"
```
Use `VECTOR_DB_KEY` for Qdrant Cloud, Pinecone, or other authenticated deployments. Redis usually puts credentials in the URL. FalkorDB also needs `VECTOR_DB_PORT`, and hybrid graph + vector setups need the matching `GRAPH_DATABASE_*` variables.
</Tab>
<Tab title="Turbopuffer">
Use this shape for Turbopuffer.
```dotenv theme={null}
TURBOPUFFER_API_KEY="your_api_key"
VECTOR_DATASET_DATABASE_HANDLER="turbopuffer"
# Optional: defaults to gcp-us-central1
TURBOPUFFER_REGION="gcp-us-central1"
```
This adapter uses custom `TURBOPUFFER_*` environment variables instead of the normal `VECTOR_DB_URL` / `VECTOR_DB_KEY` shape.
</Tab>
</Tabs>
</Accordion>
## Setup Guides
<AccordionGroup>
<Accordion title="LanceDB (Default)">
LanceDB is file-based and requires no additional setup. It's perfect for local development and single-user scenarios.
```dotenv theme={null}
VECTOR_DB_PROVIDER="lancedb"
# Optional, can be a path or URL. Defaults to <SYSTEM_ROOT_DIRECTORY>/databases/cognee.lancedb
# VECTOR_DB_URL=/absolute/or/relative/path/to/cognee.lancedb
```
**Installation**: LanceDB is included by default with Cognee. No additional installation required.
**Data Location**: Vectors are stored in a local directory. Defaults under the Cognee system path if `VECTOR_DB_URL` is empty.
</Accordion>
<Accordion title="PGVector">
PGVector stores vectors inside your Postgres database using the pgvector extension.
```dotenv theme={null}
VECTOR_DB_PROVIDER="pgvector"
# If these are omitted, Cognee falls back to the relational DB settings
# (DB_HOST, DB_PORT, DB_NAME, DB_USERNAME, DB_PASSWORD).
VECTOR_DB_HOST="localhost"
VECTOR_DB_PORT="5432"
VECTOR_DB_NAME="cognee_db"
VECTOR_DB_USERNAME="cognee"
VECTOR_DB_PASSWORD="cognee"
```
**Installation**: Install the Postgres extras:
```bash theme={null}
pip install "cognee[postgres]"
# or for binary version
pip install "cognee[postgres-binary]"
```
**Docker Setup**: Use the built-in Postgres with pgvector:
```bash theme={null}
docker compose --profile postgres up -d
```
**Note**: If using your own Postgres, ensure `CREATE EXTENSION IF NOT EXISTS vector;` is available in the target database.
For Neon Postgres, run that extension statement once in the Neon database where Cognee stores vectors. Neon still uses the regular Cognee `pgvector` provider; configure the relational `DB_*` values and `DATABASE_CONNECT_ARGS` as described in [Relational Databases](/setup-configuration/relational-databases#neon-postgres), then set `VECTOR_DB_PROVIDER="pgvector"`.
When backend access control is enabled, configure the explicit `VECTOR_DB_HOST`, `VECTOR_DB_PORT`, `VECTOR_DB_NAME`, `VECTOR_DB_USERNAME`, and `VECTOR_DB_PASSWORD` values instead of relying on the relational DB fallback.
</Accordion>
<Accordion title="Turso (libSQL)">
Turso stores vectors in libSQL. The same adapter works either **embedded** (a local `.turso.db` file) or against a **remote** Turso cloud database.
Embedded (local file):
```dotenv theme={null}
VECTOR_DB_PROVIDER="turso"
VECTOR_DB_URL="/absolute/path/to/cognee.turso.db"
```
Remote Turso cloud:
```dotenv theme={null}
VECTOR_DB_PROVIDER="turso"
VECTOR_DB_URL="libsql://your-db.turso.io"
VECTOR_DB_KEY="your_turso_auth_token"
```
**Installation**: Install the Turso extra:
```bash theme={null}
pip install "cognee[turso]"
```
This pulls in `libsql-experimental`. If the extra is missing, selecting `VECTOR_DB_PROVIDER="turso"` raises an `ImportError` at engine creation with the install hint.
**URL detection**: A `libsql://`, `http(s)://`, or `ws(s)://` URL connects to a remote libSQL server (using `VECTOR_DB_KEY` as the auth token). Any other value is treated as a local embedded file path.
**Multi-user mode**: Setting `VECTOR_DB_PROVIDER="turso"` automatically selects the `turso` dataset database handler (you do not need to set `VECTOR_DATASET_DATABASE_HANDLER` yourself). With `ENABLE_BACKEND_ACCESS_CONTROL=True`, each dataset gets its own embedded libSQL file named `{dataset_id}.turso.db` under `<SYSTEM_ROOT_DIRECTORY>/databases/{user_id}/`. Deleting a dataset evicts its cached engine and removes that file.
</Accordion>
<Accordion title="Qdrant">
Qdrant requires a running instance of the Qdrant server.
```dotenv theme={null}
VECTOR_DB_PROVIDER="qdrant"
VECTOR_DB_URL="http://localhost:6333"
```
**Installation**: Since Qdrant is a community adapter, you have to install the community package:
```bash theme={null}
pip install cognee-community-vector-adapter-qdrant
```
**Configuration**: To make sure Cognee uses Qdrant, you have to register it beforehand with the following line:
```python theme={null}
from cognee_community_vector_adapter_qdrant import register
register()
```
For more details on setting up Qdrant, visit the [more detailed description](/setup-configuration/community-maintained/qdrant) of this adapter.
**Docker Setup**: Start the Qdrant service:
```bash theme={null}
docker run -p 6333:6333 -p 6334:6334 \
-v "$(pwd)/qdrant_storage:/qdrant/storage:z" \
qdrant/qdrant
```
**Access**: Default port is 6333 for the database, and you can access the Qdrant dashboard at "localhost:6333/dashboard".
</Accordion>
<Accordion title="Redis">
Redis can be used as a vector store through the Redis Search module, providing fast vector similarity search capabilities.
```dotenv theme={null}
VECTOR_DB_PROVIDER="redis"
VECTOR_DB_URL="redis://localhost:6379"
# VECTOR_DB_KEY is optional and not used by Redis
```
**Installation**: Since Redis is a community adapter, you have to install the community package:
```bash theme={null}
pip install cognee-community-vector-adapter-redis
```
**Configuration**: To make sure Cognee uses Redis, you have to register it beforehand with the following line:
```python theme={null}
from cognee_community_vector_adapter_redis import register
register()
```
You can also configure Redis programmatically:
```python theme={null}
from cognee import config
config.set_vector_db_config({
"vector_db_provider": "redis",
"vector_db_url": "redis://localhost:6379",
})
```
For more details on setting up Redis, visit the [more detailed description](/setup-configuration/community-maintained/redis) of this adapter.
**Docker Setup**: Start a Redis instance with Search module enabled:
```bash theme={null}
docker run -d --name redis -p 6379:6379 redis:8.0.2
```
Or use **Redis Cloud** with the Search module enabled: [Redis Cloud](https://redis.io/try-free)
**Connection URL Examples**:
* Local: `redis://localhost:6379`
* With authentication: `redis://user:password@localhost:6379`
* With SSL: `rediss://localhost:6380`
</Accordion>
<Accordion title="ChromaDB">
ChromaDB support is optional and may not be installed in your Cognee environment by default.
```dotenv theme={null}
VECTOR_DB_PROVIDER="chromadb"
VECTOR_DB_URL="http://localhost:8000"
VECTOR_DB_KEY=""
# ChromaDB does not currently support Cognee backend access control dataset routing.
ENABLE_BACKEND_ACCESS_CONTROL="False"
```
**Installation**: Install ChromaDB support before configuring `VECTOR_DB_PROVIDER=chromadb`:
```bash theme={null}
pip install "cognee[chromadb]"
```
If you are using ChromaDB through a community adapter package instead of a Cognee extra, install that adapter package and call its `register()` function before running Cognee vector operations.
**Docker Setup**: Start a ChromaDB server:
```bash theme={null}
docker run -p 8000:8000 chromadb/chroma
```
</Accordion>
<Accordion title="FalkorDB">
FalkorDB can serve as both graph and vector store, providing a hybrid solution.
```dotenv theme={null}
VECTOR_DB_PROVIDER="falkor"
VECTOR_DB_URL="localhost"
VECTOR_DB_PORT="6379"
```
**Installation**: Since FalkorDB is a community adapter, you have to install the community package:
```bash theme={null}
pip install cognee-community-hybrid-adapter-falkor
```
**Configuration**: To make sure Cognee uses FalkorDB, you have to register it beforehand with the following line:
```python theme={null}
from cognee_community_hybrid_adapter_falkor import register
register()
```
For more details on setting up FalkorDB, visit the [more detailed description](/setup-configuration/community-maintained/falkordb) of this adapter.
**Docker Setup**: Start the FalkorDB service:
```bash theme={null}
docker run -p 6379:6379 -p 3000:3000 -it --rm falkordb/falkordb:edge
```
**Access**: Default ports are 6379 (DB) and 3000 (UI).
</Accordion>
<Accordion title="Neptune Analytics">
Use Amazon Neptune Analytics as a hybrid vector + graph backend.
```dotenv theme={null}
VECTOR_DB_PROVIDER="neptune_analytics"
VECTOR_DB_URL="neptune-graph://<GRAPH_ID>"
# AWS credentials via environment or default SDK chain
# Required — no per-dataset database handler exists for Neptune Analytics
ENABLE_BACKEND_ACCESS_CONTROL="false"
```
**Installation**: Install Neptune extras:
```bash theme={null}
pip install "cognee[neptune]"
```
**Note**: URL must start with `neptune-graph://` and AWS credentials should be configured via environment variables or AWS SDK.
**Access control must be off.** Cognee registers no per-dataset database handler for Neptune Analytics, so with `ENABLE_BACKEND_ACCESS_CONTROL` left at its default the handler stays `lancedb` and the first dataset access raises `The selected vector dataset to database handler does not work with the configured vector database provider`.
</Accordion>
</AccordionGroup>
## Important Considerations
<Accordion title="Dimension Consistency">
Ensure `EMBEDDING_DIMENSIONS` matches your vector store collection/table schemas:
* PGVector column size
* LanceDB Vector size
* ChromaDB collection schema
Changing dimensions requires recreating collections.
</Accordion>
<Accordion title="PGVector table layout (where embeddings are stored)">
PGVector does **not** use a single `embeddings` table. Cognee creates one table per indexed field, named `{DataPointType}_{field}` — for example `DocumentChunk_text`, `Entity_name`, `EntityType_name`, and `TextSummary_text`. Each of these collection tables has exactly three columns:
| Column | Type | Description |
| --------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `id` | `uuid` (primary key) | Matches the corresponding node id in the [graph store](/setup-configuration/graph-stores) |
| `payload` | `json` | Serialized data point — includes the embedded `text` plus reference scalars such as `document_id`, `document_name`, `chunk_index`, `source_chunk_id`, and `belongs_to_set` |
| `vector` | `vector(N)` | The embedding itself; `N` is your `EMBEDDING_DIMENSIONS` / model vector size |
When PGVector shares the relational Postgres database, these collection tables live in the same schema as the snake\_case relational metadata and provenance tables (`data`, `datasets`, `dataset_data`, `nodes`, and `edges`). If you configure PGVector with separate `VECTOR_DB_*` settings, the collection tables live in that vector database instead. When using the [Postgres graph store](/setup-configuration/graph-stores), graph data is stored in `graph_node` and `graph_edge`.
Cognee distinguishes vector collections by their PascalCase first letter. Text content is stored in the JSON `payload` and in `TEXT` columns; there are no fixed-width `varchar(255)` columns (the relational provenance `nodes.label` / `nodes.type` columns were migrated from `varchar(255)` to `TEXT`).
</Accordion>
<Accordion title="Provider Comparison">
| Provider | Setup | Performance | Use Case |
| ----------------- | --------------------- | ----------- | ----------------------------------------- |
| LanceDB | Zero setup | Good | Local development |
| PGVector | Postgres required | Excellent | Production with Postgres |
| Turso (libSQL) | `cognee[turso]` extra | Good | Embedded local file or remote Turso cloud |
| Neptune Analytics | AWS required | Excellent | Cloud hybrid solution |
| ChromaDB | Server required | Good | Dedicated vector store |
| Qdrant | Server required | Excellent | High-performance vector search |
| Redis | Server required | Excellent | Low-latency in-memory search |
| FalkorDB | Server required | Good | Hybrid graph + vector |
</Accordion>
## Troubleshooting
<Accordion title="Too many open files on macOS">
This issue is most commonly reported with LanceDB-backed search workloads.
`VECTOR_DB_SUBPROCESS_ENABLED` defaults to `true`. In high-fanout searches — for example, many queries across many datasets — this can create a large number of OS subprocesses. On macOS, that can quickly hit the default file descriptor limit (`ulimit -n`, often `256`) and surface as:
```text theme={null}
OSError: [Errno 24] Too many open files
```
Because the exception often appears inside LanceDB internals, it may not be obvious that the underlying issue is subprocess and file-descriptor exhaustion rather than a LanceDB data problem.
If this happens:
```dotenv theme={null}
VECTOR_DB_SUBPROCESS_ENABLED=false
```
And raise the shell limit before starting Cognee:
```bash theme={null}
ulimit -n 4096
```
This is especially relevant for audit-style or multi-dataset search runs where Cognee fans queries out across several datasets in parallel.
</Accordion>
<Accordion title="Subprocess concurrency and per-call failures">
When `VECTOR_DB_SUBPROCESS_ENABLED` is `true`, concurrent async RPCs to the LanceDB subprocess run in parallel rather than serializing behind a single session lock. Each request carries its own id and is routed back to its own waiter, so concurrent add and search operations no longer queue behind one another.
`SUBPROCESS_WORKER_MAX_INFLIGHT` (default `16`) bounds how many async operations a worker runs at once, keeping the worker's memory footprint predictable. The value must be `> 0`; a zero or negative value fails worker initialization with a `ValueError` rather than silently degrading. Raise it for higher concurrency (for example, `SUBPROCESS_WORKER_MAX_INFLIGHT=64`), or set a large value to effectively remove the cap.
A per-call timeout or cancellation now resolves only that individual call and leaves the subprocess session running for other in-flight and future calls. The session is torn down only on genuine session-ending events (worker crash, shutdown, or respawn), which propagate a `SubprocessTransportError` to any calls still pending. Synchronous calls (such as the Kuzu graph backend) are unaffected and continue to run serially.
Subprocess engines are also no longer closed the moment a dataset context exits. `SUBPROCESS_IDLE_TTL_SECONDS` (default `600`) keeps an idle engine warm, and a background reaper thread closes it only after a full TTL with no use — so repeated or interactive queries against the same dataset reuse the live worker instead of paying for a close plus a respawn. Datasets with an active dataset-queue slot are skipped by the reaper, and non-subprocess engines (PGVector and other remote stores) are never reaped.
The trade-off: a kept-warm worker holds its memory and its PID (and, for the graph store's embedded Kuzu/Ladybug worker, its database file lock — LanceDB takes no exclusive file lock) for up to the TTL after its last use, with the total number of retained idle workers still bounded by `DATABASE_MAX_LRU_CACHE_SIZE`. Lower the TTL, or set `SUBPROCESS_IDLE_TTL_SECONDS=0` to close engines at each dataset-context exit as before, if you are memory-tight or need locks released promptly. See [Subprocess engine teardown coordination](/setup-configuration/permissions#subprocess-engine-teardown-coordination) for the full lifecycle.
</Accordion>
## Community-Maintained Providers
Additional vector stores are available through community-maintained adapters:
* **[Qdrant](/setup-configuration/community-maintained/qdrant)** — Vector search engine with cloud and self-hosted options
* **[Redis](/setup-configuration/community-maintained/redis)** — Fast vector similarity search
* **[FalkorDB](/setup-configuration/community-maintained/falkordb)** — Hybrid vector and graph store
* **[Pinecone](/setup-configuration/community-maintained/pinecone)** — Managed vector database (requires separate install + registration)
* **[Turbopuffer](/setup-configuration/community-maintained/turbopuffer)** — High-performance vector database
* **Milvus, Weaviate, and more** — See [all community adapters](/setup-configuration/community-maintained/overview)
## Notes
* **Embedding Integration**: Vector stores use your embedding engine from the Embeddings section
* **Dimension Matching**: Keep `EMBEDDING_DIMENSIONS` consistent between embedding provider and vector store
* **Performance**: Local providers (LanceDB) are simpler but cloud providers offer better scalability
<Columns>
<Card title="Embedding Providers" icon="layers" href="/setup-configuration/embedding-providers">
Configure embedding providers for vector generation
</Card>
<Card title="Graph Stores" icon="network" href="/setup-configuration/graph-stores">
Set up graph databases for knowledge graphs
</Card>
<Card title="Overview" icon="settings" href="/setup-configuration/overview">
Return to setup configuration overview
</Card>
</Columns>
# Cognee Cloud
Source: https://docs.cognee.ai/typescript/cloud
Connect a local @cognee/cognee-ts process to Cognee Cloud with the module-level serve and disconnect functions.
`serve` and `disconnect` are module-level functions (not instance methods)
because they operate on global cloud state.
```ts theme={null}
import { serve, disconnect } from '@cognee/cognee-ts';
// Direct mode (no Auth0 flow; headless-friendly)
const { serviceUrl } = await serve({ url: "http://localhost:8000", apiKey: "key" });
console.log("Connected to", serviceUrl);
// Cloud mode (Auth0 device-code flow — requires a TTY)
await serve();
// Tear down
await disconnect();
await disconnect({ wipeCredentials: true }); // also removes the local credential cache
```
For the hosted service itself, see the [Cognee Cloud](/cognee-cloud/overview)
documentation.
# Configuration
Source: https://docs.cognee.ai/typescript/configuration
Configure the @cognee/cognee-ts SDK through constructor settings, the c.config setters, and environment variables.
Settings resolve in three layers: compiled-in defaults, environment variables,
and anything you pass to the `Cognee` constructor or change later through
`c.config`. Constructor and `c.config` values win over the environment.
## Constructor
```ts theme={null}
const c = new Cognee(settings?)
```
`settings` is an optional object (or JSON string) that overrides env-derived
defaults. Keys are the canonical Settings field names (`llmModel`,
`embeddingProvider`, `vectorDbProvider`, etc.). Absent keys keep their
env-variable or compiled-in default.
## Config
Use `c.config` to change settings after construction. Granular setters are
synchronous and take effect immediately (the engines are lazily rebuilt on the
next pipeline call).
```ts theme={null}
c.config.setLlmModel("gpt-5");
c.config.setLlmApiKey(process.env.OPENAI_TOKEN!);
c.config.setEmbeddingProvider("openai");
c.config.setEmbeddingModel("text-embedding-3-small");
// Bulk setters (throw on unknown key or type mismatch) — one per subsystem:
c.config.setLlmConfig({ model: "gpt-5", temperature: 0.2 });
c.config.setEmbeddingConfig({ provider: "openai", model: "text-embedding-3-small" });
c.config.setVectorDbConfig({ provider: "brute-force" });
c.config.setGraphDbConfig({ provider: "kuzu" });
// Generic key-value setter:
c.config.set("llmModel", "gpt-5-mini");
// Read back the current config (secret fields are redacted):
const cfg = c.config.get();
console.log(cfg);
```
## Environment variables
| Variable | Purpose |
| ------------------------------------------------------------ | ------------------------------------------------------------------- |
| `OPENAI_URL` | LLM API base URL (OpenAI-compatible endpoint). |
| `OPENAI_TOKEN` | LLM API key. |
| `OPENAI_MODEL` | LLM model name (default: `gpt-4o-mini`). |
| `EMBEDDING_PROVIDER` | Embedding provider: `openai`, `ollama`, `onnx`, `mock`. |
| `EMBEDDING_MODEL` | Embedding model name. |
| `EMBEDDING_DIMENSIONS` | Embedding vector dimensions. |
| `EMBEDDING_ENDPOINT` | Embedding API base URL (falls back to `OPENAI_URL`). |
| `EMBEDDING_API_KEY` | Embedding API key (falls back to `OPENAI_TOKEN`). |
| `MOCK_EMBEDDING` | Set `true` to use zero-vector mock embeddings (no model download). |
| `COGNEE_BINDING_SUPPRESS_LOGS` | Suppress the auto-installed stderr fmt subscriber. |
| `COGNEE_HOST_SDK` | Suppress binding-armed analytics when the host is an embedding SDK. |
| `TELEMETRY_DISABLED`, `ENV` | Standard analytics opt-outs for `setupTelemetryAnalytics()`. |
| `RUST_LOG`, `LOG_LEVEL` | `tracing-subscriber` env-filter level overrides. |
| `COGNEE_LOG_*`, `LOG_FILE_NAME` | Consumed by `setupLogging()`. |
| `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_SERVICE_NAME`, `OTEL_*` | Consumed by `setupTelemetry()`. |
The logging and telemetry variables are read by the setup functions described in
[Runtime and Observability](/typescript/runtime).
# TypeScript (@cognee/cognee-ts)
Source: https://docs.cognee.ai/typescript/getting-started
Build AI-memory pipelines in Node.js with the @cognee/cognee-ts SDK.
<Note>
**`@cognee/cognee-ts`** provides Node.js bindings for the
[cognee-rs](https://github.com/topoteretes/cognee-rs) AI-memory SDK, built with
[Neon](https://neon-bindings.com/). It is published on npm as
[`@cognee/cognee-ts`](https://www.npmjs.com/package/@cognee/cognee-ts).
</Note>
Cognee transforms raw text, files, and URLs into a persistent, queryable knowledge
graph. The high-level API is **`remember`** (ingest + extract in one call) →
**`recall`** (source-aware retrieval). These wrap the lower-level
**`add`** → **`cognify`** → **`search`** stages, which remain available when you
need finer control.
## Installation
```bash theme={null}
npm install @cognee/cognee-ts
```
## Quick start
```ts theme={null}
import { init, Cognee } from '@cognee/cognee-ts';
// Initialise the SDK runtime (call once at process start).
init();
const c = new Cognee({
llmModel: "gpt-5-mini",
llmApiKey: process.env.OPENAI_TOKEN,
});
// Warm up engines (builds embedding model, resolves default user).
await c.warm();
// Ingest content and extract a knowledge graph in one call.
await c.remember({ type: "text", text: "The quick brown fox jumps over the lazy dog." }, "demo");
// Recall an answer with source-aware routing.
const recall = await c.recall("What does the fox do?");
console.log(recall.searchResponse?.result?.data);
```
The quick start touches three things you will use in every project: the
runtime `init()` call, the `Cognee` constructor, and the `remember` / `recall`
pair. The pages below cover each in depth.
## Next Steps
<CardGroup>
<Card title="Configuration" href="/typescript/configuration" icon="settings">
Constructor settings, the `c.config` setters, and every environment variable the SDK reads.
</Card>
<Card title="Memory API" href="/typescript/memory-api" icon="brain">
`remember`, `recall`, `improve`, `forget`, and `rememberEntry`.
</Card>
<Card title="Resource Managers" href="/typescript/resources" icon="database">
Datasets, sessions, notebooks, and user / pipeline-run admin.
</Card>
<Card title="Legacy Pipeline Operations" href="/typescript/legacy-operations" icon="code">
Staged `add` → `cognify` → `search` control, plus `memify`.
</Card>
<Card title="Runtime and Observability" href="/typescript/runtime" icon="activity">
Runtime lifecycle, logging, and telemetry setup.
</Card>
<Card title="Cognee Cloud" href="/typescript/cloud" icon="cloud">
Connect a local SDK to Cognee Cloud with `serve` and `disconnect`.
</Card>
<Card title="@cognee/cognee-ts on npm" href="https://www.npmjs.com/package/@cognee/cognee-ts" icon="npm">
Package page, versions, and the full README with runnable examples.
</Card>
<Card title="Rust SDK" href="/rust/getting-started" icon="rust">
The underlying cognee-rs engine and its CLI.
</Card>
</CardGroup>
# Legacy Pipeline Operations
Source: https://docs.cognee.ai/typescript/legacy-operations
Staged add, cognify, addAndCognify, search, and memify calls for explicit control over the @cognee/cognee-ts pipeline.
Use these lower-level operations when you need explicit staged control over the
pipeline. For most applications, prefer `remember()` and `recall()` from the
[Memory API](/typescript/memory-api).
## add
Ingest one or more data items into a named dataset.
```ts theme={null}
// Text
await c.add({ type: "text", text: "…" }, "my-dataset");
// File
await c.add({ type: "file", path: "/abs/path/to/doc.txt" }, "my-dataset");
// URL
await c.add({ type: "url", url: "https://example.com/article" }, "my-dataset");
// Binary (name is required for MIME detection)
await c.add({ type: "binary", bytes: buffer, name: "report.pdf" }, "my-dataset");
// Multiple items at once
await c.add([
{ type: "text", text: "First document" },
{ type: "file", path: "/abs/path/two.txt" },
], "my-dataset");
```
## cognify
Extract entities and relationships into the knowledge graph.
```ts theme={null}
await c.cognify("my-dataset");
// With options
await c.cognify("my-dataset", {
chunkSize: 512,
summarization: true,
triplet: true, // also index triplet embeddings (enables TRIPLET_COMPLETION search)
});
```
## addAndCognify
Ingest and extract in a single call.
```ts theme={null}
const { add, cognify } = await c.addAndCognify(
{ type: "text", text: "…" },
"my-dataset"
);
```
## search
Query the knowledge graph directly. Defaults to `GRAPH_COMPLETION`.
```ts theme={null}
const result = await c.search("What is the capital of France?");
// With options
const result = await c.search("summarise recent events", {
searchType: "SUMMARIES",
topK: 5,
datasets: ["news"],
});
```
All 15 search types are supported (SCREAMING\_SNAKE\_CASE):
`GRAPH_COMPLETION`, `SUMMARIES`, `CHUNKS`, `RAG_COMPLETION`, `TRIPLET_COMPLETION`,
`GRAPH_SUMMARY_COMPLETION`, `CYPHER`, `NATURAL_LANGUAGE`, `GRAPH_COMPLETION_COT`,
`GRAPH_COMPLETION_CONTEXT_EXTENSION`, `FEELING_LUCKY`, `FEEDBACK`, `TEMPORAL`,
`CODING_RULES`, `CHUNKS_LEXICAL`.
## memify
Index triplet embeddings from the existing knowledge graph. Enables
`TRIPLET_COMPLETION` search. Idempotent.
```ts theme={null}
await c.memify();
```
# Maintenance and Visualization
Source: https://docs.cognee.ai/typescript/maintenance
Replace or prune stored data and render the knowledge graph to HTML with @cognee/cognee-ts.
## Maintenance operations
```ts theme={null}
// Replace a data item (delete → re-add → re-cognify)
await c.update("old-data-uuid", { type: "text", text: "updated content" }, "my-dataset");
// Remove all files from storage (metadata DB untouched)
await c.pruneData();
// Wipe graph, vector, metadata, and/or cache backends
await c.pruneSystem({ pruneGraph: true, pruneVector: true });
```
To remove memory by item or dataset rather than wiping backends, use `forget`
from the [Memory API](/typescript/memory-api).
## Visualisation
```ts theme={null}
// Get the HTML string
const html = await c.visualize();
// Write to a file (returns the absolute path)
const path = await c.visualizeToFile({ destinationPath: "/tmp/graph.html" });
```
Requires the `visualization` feature compiled into the native addon.
# Memory API
Source: https://docs.cognee.ai/typescript/memory-api
The high-level memory operations in @cognee/cognee-ts: remember, recall, improve, forget, and rememberEntry.
Use these operations first. They are the high-level memory API in the TypeScript
SDK. For explicit staged control over the underlying pipeline, see
[Legacy Pipeline Operations](/typescript/legacy-operations).
## remember
Store content as permanent graph memory, or as session memory when you pass
`sessionId`.
```ts theme={null}
// Permanent graph memory: ingest, extract, and enrich in one call.
await c.remember({ type: "text", text: "…" }, "my-dataset", {
selfImprovement: true,
});
// Session memory: fast short-term memory for a conversation or agent run.
await c.remember({ type: "text", text: "…" }, "my-dataset", {
sessionId: "session-id",
selfImprovement: true,
});
```
## recall
Retrieve from session and graph memory. With `scope: "auto"`, session memory is
checked before graph memory.
```ts theme={null}
const result = await c.recall("What did we discuss?", {
sessionId: "session-uuid",
scope: "auto", // "graph" | "session" | "trace" | "graph_context" | "all"
});
```
## improve
Run the session-to-graph bridge and graph enrichment pipeline.
```ts theme={null}
await c.improve({
datasetName: "my-dataset",
sessionIds: ["session-uuid"],
});
```
## forget
Remove permanent memory by item, dataset, or everything.
```ts theme={null}
// Forget a single item
await c.forget({ kind: "item", dataId: "uuid", dataset: { name: "my-dataset" } });
// Forget an entire dataset
await c.forget({ kind: "dataset", dataset: { name: "my-dataset" } });
// Forget everything
await c.forget({ kind: "all" });
```
## rememberEntry
Store a typed memory entry (`"qa"`, `"trace"`, or `"feedback"`) in a session.
```ts theme={null}
const result = await c.rememberEntry(
{ type: "qa", question: "…", answer: "…" },
"my-dataset",
"session-uuid",
{ tenant: "tenant-id" }, // optional
);
```
# Low-level Pipeline API
Source: https://docs.cognee.ai/typescript/pipeline-api
Build custom task pipelines with the pipeline namespace exported by @cognee/cognee-ts.
The original pipeline engine API is available under the `pipeline` namespace:
```ts theme={null}
import { pipeline, init } from '@cognee/cognee-ts';
init();
const task = pipeline.createTask((input: pipeline.CogneeValue, ctx: pipeline.TaskContext) => {
// process input …
return input;
});
const p = new pipeline.Pipeline("my pipeline");
p.addTask(new pipeline.TaskInfo(task));
const [result] = await p.execute([pipeline.CogneeValue.fromString("hello")], ctx);
```
All pipeline symbols are also available as flat re-exports at the top level of
`@cognee/cognee-ts`:
```ts theme={null}
import {
Pipeline,
TaskInfo,
createTask,
CogneeValue,
TaskContext,
RunHandle,
CancellationHandle,
CancellationToken,
createCancellationPair,
ProgressToken,
Watcher,
createWatcher,
createNoopWatcher,
} from '@cognee/cognee-ts';
```
# Resource Managers
Source: https://docs.cognee.ai/typescript/resources
Manage datasets, sessions, notebooks, and users through the namespaced managers on a Cognee handle.
Each `Cognee` handle exposes namespaced managers for the resources behind the
memory API. They are plain async methods; none of them require a pipeline run.
## Datasets
```ts theme={null}
const datasets = await c.datasets.list();
const items = await c.datasets.listData(datasetId);
const hasContent = await c.datasets.has(datasetId);
const statuses = await c.datasets.status([id1, id2]);
await c.datasets.empty(datasetId);
await c.datasets.deleteData(datasetId, dataId);
await c.datasets.deleteAll();
```
## Sessions
```ts theme={null}
const entries = await c.sessions.get("session-uuid", { lastN: 10 });
await c.sessions.addFeedback("session-uuid", "qa-uuid", "Great answer!", 5);
await c.sessions.deleteFeedback("session-uuid", "qa-uuid");
const ctx = await c.sessions.getGraphContext("session-uuid");
await c.sessions.setGraphContext("session-uuid", "new context");
```
## Notebooks
```ts theme={null}
// List all notebooks for the current user.
const notebooks = await c.notebooks.list();
// Create a new notebook with optional cells and deletability flag.
const nb = await c.notebooks.create("My Notes", [], true);
// Partially update a notebook (name, cells, or both).
const updated = await c.notebooks.update(nb.id, { name: "Renamed Notes" });
// Delete a notebook — returns true if a row was removed.
const removed = await c.notebooks.delete(nb.id);
```
## Users and pipeline-run admin
```ts theme={null}
// Resolve (or lazily create) the default user for this handle.
const user = await c.users.getOrCreateDefault();
// Unblock a dataset stuck in "running" state so it can be re-cognified.
await c.users.resetPipelineRunStatus(datasetId, "cognify_pipeline");
// Reset all pipeline-run statuses for a dataset at once.
await c.users.resetDatasetPipelineRunStatus(datasetId);
```
# Runtime and Observability
Source: https://docs.cognee.ai/typescript/runtime
Boot and shut down the Rust runtime behind @cognee/cognee-ts, and wire up logging, tracing, and analytics.
The native addon runs on a Rust tokio runtime. Boot it once at process start
with `init()` before any async operation, and optionally attach logging and
telemetry.
## Initialisation and observability
```ts theme={null}
import {
init,
initWithThreads,
shutdown,
setupLogging,
setupTelemetry,
setupTelemetryAnalytics,
} from '@cognee/cognee-ts';
// Boot the Rust tokio runtime (required before any async op).
init();
// Alternatively boot with a fixed worker-thread count.
initWithThreads(4);
// Optional: add file logging (reads COGNEE_LOG_*, LOG_FILE_NAME, LOG_LEVEL).
setupLogging();
// Optional: enable OTLP trace export (reads OTEL_* env vars).
setupTelemetry();
// Optional: enable product-analytics emission (returns true if armed).
const armed = setupTelemetryAnalytics();
// Tear the runtime down (e.g. before process exit).
shutdown();
```
Each handle also exposes `await c.ownerId()`, returning the owner UUID used for
deterministic, per-tenant ID generation.
Set `COGNEE_BINDING_SUPPRESS_LOGS=1` before `require`ing the module to skip the
auto-installed stderr subscriber if your host manages the logging pipeline.
The environment variables these functions read are listed in
[Configuration](/typescript/configuration#environment-variables).
|