dlt (Data Load Tool) - Cognee Documentation

Ingest structured relational data — databases, CSV files, and dlt resources — directly into cognee’s knowledge graph. Foreign keys become graph edges, tables become schema nodes, and each row becomes a searchable document, all built deterministically from the schema without LLM extraction.

Why Use This Integration

Schema-Aware Graphs: Foreign key relationships are preserved as first-class edges in the knowledge graph
Deterministic Graph Construction: Structured data bypasses LLM entity extraction — no hallucination risk
Mixed Ingestion: Combine structured (dlt) and unstructured (text, PDF) data in the same dataset
Multiple Input Modes: Pass explicit dlt resources, CSV file paths, or database connection strings
Write Dispositions: Control how data is synced — merge (upsert), append, or replace

Installation

Nothing to install. dlt[sqlalchemy] and pandas are core cognee dependencies, so every route on this page works on a plain install. The one exception is the OneNote connector, which also needs beautifulsoup4 (see its Installation section). Install cognee:

pip install cognee

Or with uv:

uv pip install cognee

cognee[dlt] is kept as an empty extra, so existing install commands, Dockerfiles, and CI jobs that name it stay valid — it simply resolves to nothing. The same applies to the official cognee/cognee Docker image: CSVs added there follow the dlt_csv_loader path described under CSV Files, as they do everywhere else. See Optional Extras and Document Loaders.

Quick Start

1. Ingest a dlt Resource

Define a dlt resource and pass it to cognee.remember(). The dlt-specific structured-ingestion options primary_key, write_disposition, SQL query, and max_rows_per_table are accepted by cognee.remember() and forwarded to the underlying ingestion step. After ingestion, use cognee.recall(...) to query the graph.

import dlt
import cognee
import asyncio

@dlt.resource()
def users_and_pets():
    yield [
        {
            "id": 1,
            "name": "Alice",
            "pets": [
                {"id": 1, "name": "Fluffy", "type": "cat"},
                {"id": 2, "name": "Spot", "type": "dog"},
            ],
        },
        {
            "id": 2,
            "name": "Bob",
            "pets": [{"id": 3, "name": "Fido", "type": "dog"}],
        },
    ]

async def main():
    await cognee.remember(
        users_and_pets,
        dataset_name="users_and_pets",
        primary_key="id",
    )
    results = await cognee.recall(
        query_text="Which pet does Alice have?",
        datasets=["users_and_pets"],
    )
    print(results)

asyncio.run(main())

dlt automatically detects nested structures (like pets inside each user) and creates separate tables with foreign key relationships.

The lower-level cognee.add(...) + cognee.cognify(...) pair still accepts the same dlt kwargs and remains useful when you need to run ingestion and graph building as separate steps. For the runnable end-to-end version of this walkthrough, see examples/demos/ingestion_and_migration/dlt_ingestion_example.py.

2. Build and Query the Graph

Once remember() finishes ingesting and building the graph, use cognee.recall(...) to query it.

Other Input Modes

CSV Files

CSV files are handled by the dlt_csv_loader, which the loader engine registers above the plain-text csv_loader on every install. Selection happens inside the ingest pipeline rather than before it, so the same route applies to every way a CSV can arrive — a local path, a file:// or s3:// location, or an uploaded file:

await cognee.remember(
    "/path/to/employees.csv",
    dataset_name="employees",
)

Each CSV is staged through dlt and becomes one manifest record per original file. The manifest’s identity is derived from the original file name — not the temporary copy an s3:// download or an upload lands in — so it stays stable across runs and a re-add updates the existing record instead of creating a second one. As with any dlt source, the rows skip chunking and LLM entity extraction.

Delimiter detection

The delimiter is detected per file, so semicolon-, tab-, and pipe-separated exports load as proper tables instead of collapsing into a single column — or, when a text field contains commas of its own, failing outright on a ragged split. No configuration is required and there is no knob to set. Cognee reads up to the first 20 lines of the file, ignoring blank ones, and parses them with each of ,, ;, \t, | in that order, picking the first delimiter that splits every sampled line into the same number of columns, with more than one column. Parsing (rather than counting characters) means quoted fields are respected, so a comma-heavy text column — prose, an embedded JSON list — no longer forces a semicolon-delimited file onto commas. Candidates are tried in order, so a comma wins ties, and , is also the fallback when the file is empty or no candidate produces a consistent column count.

If a genuinely ambiguous file — one where a second candidate also yields a consistent column count — loads with the wrong delimiter, convert it to comma-separated before ingesting, or request the plain-text csv_loader for that call (below).

The DLT options given to remember() / add() in dlt_config reach CSV files too, as do bare keyword arguments (add() takes all six; remember() takes only primary_key, write_disposition, query and max_rows_per_table). Cognee folds them into the dlt_csv_loader entry of preferred_loaders:

await cognee.remember(
    "/path/to/employees.csv",
    dataset_name="employees",
    dlt_config={"primary_key": "id", "write_disposition": "merge"},
)

You can still set the options under the loader directly. dlt_csv_loader accepts primary_key, write_disposition, max_rows_per_table, column_value_columns, and temporal_columns:

await cognee.remember(
    "/path/to/employees.csv",
    dataset_name="employees",
    preferred_loaders=[
        {"dlt_csv_loader": {"primary_key": "id", "write_disposition": "merge"}}
    ],
)

If you set an option both on the call and under preferred_loaders["dlt_csv_loader"], cognee raises a ValueError instead of picking one. To flatten a CSV into plain text instead, request csv_loader explicitly for that call. Since dlt became a core dependency there is no install shape that gives you text flattening by default, so this per-call override is the only way to get it:

await cognee.remember(
    "/path/to/employees.csv",
    dataset_name="employees",
    preferred_loaders=[{"csv_loader": {}}],
)

A CSV that yields no rows now fails loudly with an IngestionError naming the source, rather than being ingested as an empty record. dlt_csv_loader likewise requires the dataset and user context that the ingest pipeline supplies, so it cannot be invoked outside of remember() / add().

Database Connection String

Ingest tables directly from an existing database:

await cognee.remember(
    "postgresql://user:pass@host/db",
    dataset_name="company_db",
    primary_key="id",
)

The connection string is a source cognee reads from, never a destination it writes to — ingested rows land in cognee’s own configured stores, including vector embeddings of each row, so they’re searchable by semantic similarity as well as graph traversal. To change where cognee stores memory, configure the graph, vector, and relational providers.

Supported databases via auto-detection: SQLite, PostgreSQL, MySQL, MSSQL, Oracle. Hosted Postgres providers such as Neon work with their standard postgresql:// connection strings; keep provider-required SSL parameters such as ?sslmode=require. Amazon Redshift is also compatible since it speaks the PostgreSQL wire protocol — use a standard postgresql:// connection string pointing to your Redshift endpoint.

Over the HTTP API (POST /api/v1/add, POST /api/v1/remember), a connection string is refused with 422 DatabaseIngestionDisabled unless the server sets ALLOW_DATABASE_INGESTION=true. The SDK, and the CLI and MCP server running in-process, accept connection strings while the variable is unset, and false refuses them everywhere. A network connection string may carry only the sslmode, connect_timeout, application_name and charset options, so ?sslmode=require still works. A sqlite:/// path must name an existing file. Connection strings that point at Cognee’s own databases are always refused. See Database Connection-String Ingestion.

For Snowflake and Google BigQuery, construct a dlt source directly and pass it to cognee.remember() (see the Cloud Data Warehouses accordion below). You can optionally filter with a SQL WHERE clause:

await cognee.remember(
    "postgresql://user:pass@host/db",
    dataset_name="engineering_team",
    primary_key="id",
    query="SELECT * FROM employees WHERE department = 'Engineering'",
)

The FROM target may also be schema-qualified. Cognee splits schema.table before building the source — the schema is handed to dlt and the bare table name is what the WHERE clause is matched against — so the filter applies to the qualified table just as it does to a bare one:

await cognee.remember(
    "postgresql://user:pass@host/db",
    dataset_name="engineering_team",
    primary_key="id",
    query="SELECT * FROM public.employees WHERE department = 'Engineering'",
)

The FROM target may also carry a table alias, with or without AS: query="SELECT * FROM employees e WHERE department = 'Engineering'" filters exactly like the unaliased form above; previously the alias broke the parse and the whole table was loaded unfiltered. The WHERE clause is applied as written as long as it does not reference the alias — leave columns unqualified, or qualify them with the table name. Two query shapes cannot be replayed against the single-table select cognee hands to dlt, so they raise a ValueError instead of loading the table unfiltered:

A WHERE that references the table alias — SELECT * FROM employees e WHERE e.department = 'Engineering'. The filter is replayed against the bare table, where the alias does not exist. Rewrite the columns bare (department = 'Engineering') or qualify them with the table name (employees.department = 'Engineering'). The check is a textual scan for <alias>.<column> across the whole clause, so a string literal that happens to match trips it too — under the alias f, WHERE path = 'f.txt' raises even though no column is qualified. A longer alias (emp rather than e) avoids the collision.
A JOIN in the FROM clause — SELECT o.id FROM orders o JOIN customers c ON o.customer_id = c.id WHERE c.active. The source targets a single table, so a join across tables has no representation. Create a database view over the joined result and ingest that view, or ingest the table without a filter. Only the JOIN keyword is recognized: a comma-style join (SELECT * FROM orders o, customers c WHERE c.active) still parses as a plain select on the first table with its WHERE dropped, loading that table unfiltered — write joins with JOIN so the error fires, or ingest a view.

query accepts a SELECT with an unquoted FROM target — either a bare table name (employees) or a dot-qualified one (public.employees) — an optional table alias (employees e / employees AS e), and an optional WHERE clause. Quoted identifiers such as "public"."employees" are not parsed and raise a ValueError. Omitting WHERE loads the whole table.

Mixed Structured + Unstructured

Combine dlt resources with unstructured text in a single dataset:

text = """Alice has two pets: a cat named Fluffy and a dog named Spot.
Bob has a dog named Fido, who is friendly with both Fluffy and Spot."""

await cognee.remember(
    [text, users_and_pets],
    dataset_name="users_and_pets_with_text",
    primary_key="id",
)

Structured data creates deterministic graph nodes from the schema, while unstructured text goes through LLM-based entity extraction. Both are combined in the same knowledge graph.

DLT Options

Options that apply only to dlt sources (connection strings, dlt resources, and CSV files) can be grouped in one dlt_config dict on add(). remember() passes it through to add():

await cognee.remember(
    "postgresql://user:pass@host/db",
    dataset_name="company_db",
    dlt_config={
        "primary_key": "id",
        "write_disposition": "merge",
        "max_rows_per_table": 10_000,
        "temporal_columns": {"orders": ["order_date"], "*": ["*"]},
    },
)

Key	Type	What it does
`primary_key`	`str`	Upsert key. See Write Dispositions.
`write_disposition`	`str`	`replace`, `merge`, or `append`.
`query`	`str`	SQL `SELECT … WHERE` filter for a connection string.
`max_rows_per_table`	`int`	Per-table row cap for this call. See Bounding Ingestion.
`column_value_columns`	`dict[str, list[str]]`	Columns whose cells become shared `ColumnValue` nodes. See Column Value Nodes.
`temporal_columns`	`dict[str, list[str]]`	Columns whose date cells become `Timestamp` edges. See Time Anchoring.

The read options primary_key, write_disposition and max_rows_per_table apply to every dlt source, connectors included, while query applies only to connection strings. The exception is that document-mode connector sources always read their whole table and ignore max_rows_per_table. Document-mode connector sources also ignore the two column selections (column_value_columns, temporal_columns).
A key you leave out falls back to its default: DLT_MAX_ROWS_PER_TABLE, DLT_COLUMN_VALUE_COLUMNS, or DLT_TEMPORAL_COLUMNS for the three keys that have an environment variable, and the built-in default for the others.
The dict is validated. An unknown key raises a pydantic ValidationError.
The bare keyword arguments (primary_key="id", …) are still accepted for compatibility. If you give the same option both in dlt_config and as a keyword argument, cognee raises a ValueError instead of merging them.

Time Anchoring

Cognee anchors dlt rows in time by default. When a cell in a selected column holds an ISO 8601 month (2024-03), date (2024-03-01), or date-time (2024-03-01 10:15:00), the row gets an edge to the shared Timestamp node for that period. The edge is named after the column. Prose, relative phrases, and bare years are never read from a cell. Foreign-key columns are skipped, because they already link rows. A date primary key is kept, because a time series is keyed by the date it is about. Fractional seconds are dropped. A date-time with a zone (Z, +02:00) is shifted to UTC. The Timestamp node is the same one that text extraction produces for that instant. A row dated 2024-03-01 and a document that says “1 March 2024” therefore share one node. The temporal_columns selection decides which columns count. It has the same shape as column_value_columns: a map from table name to column names, with "*" as a wildcard on either side. A table you name gets exactly its list. An empty list turns that table off. Only tables you don’t name fall back to the "*" entry.

Default {"*": ["*"]}: every date-shaped cell of every table.
Narrow it when a table has bookkeeping dates. For example, {"orders": ["order_date"], "*": ["*"]} stops an updated_at column from anchoring orders rows to the time they were revised.
{} turns time edges off.

Set it per call with dlt_config={"temporal_columns": {...}} (or the bare temporal_columns= keyword on add()). To change the process-wide default, set the DLT_TEMPORAL_COLUMNS environment variable as JSON:

DLT_TEMPORAL_COLUMNS='{"orders": ["order_date"], "audit_log": [], "*": ["*"]}'

This is on by default. dlt sources you ingest or re-ingest after upgrading get Timestamp edges for their date cells, which can change what TEMPORAL search returns. Set DLT_TEMPORAL_COLUMNS='{}' to keep building graphs without them.

Rows in Hybrid and TEMPORAL Search

When a dataset holds dlt rows, the hybrid chunk lane also searches the row vectors (the DltRow collection) next to document chunks. This applies to HYBRID_COMPLETION and TEMPORAL. In TEMPORAL search, rows anchored to Timestamp nodes inside the question’s time window join the candidates and rank ahead of undated ones; only rows dated more coarsely than the question rank lower. A question like “which orders were placed in March 2024?” can therefore reach the matching rows even when their text alone does not match the question well.

Column Value Nodes

By default, every non-key cell of every table becomes a shared ColumnValue node. Cognee makes one node per unique (table, column, value), and each row gets an edge to it named after the column. Rows that share a value, such as a country, a genre, or a status, therefore meet in one node. Graph traversal, degree counts, and the visualization can then see individual cells.

Primary-key and foreign-key columns are never value nodes. A primary key is the row’s identity, and a foreign key is already an edge to the row it references.
Null and empty cells are skipped.
A date cell that becomes a Timestamp edge (see Time Anchoring) is represented by that Timestamp node only, not by a ColumnValue node as well. Each cell gets one node.

The column_value_columns selection has the same shape as temporal_columns: a map from table name to column names, with "*" as a wildcard on either side. A table you name gets exactly its list, and only tables you don’t name fall back to the "*" entry.

Default {"*": ["*"]}: every non-key column of every table.
Narrow it to keep one-off values such as free text out. For example, {"orders": ["status"]} makes value nodes only for orders.status.
{} turns value nodes off.

Set it per call with dlt_config={"column_value_columns": {...}} (or the bare column_value_columns= keyword on add()). To change the process-wide default, set the DLT_COLUMN_VALUE_COLUMNS environment variable as JSON:

DLT_COLUMN_VALUE_COLUMNS='{"orders": ["status"]}'

This is on by default. Each unique value costs one embedding, and so does each column node. dlt sources you ingest or re-ingest after upgrading get value nodes for every non-key column, so the node count, the embedding count, and the visualization grow. Set DLT_COLUMN_VALUE_COLUMNS='{}' to keep building graphs without them, or set DLT_MAX_COLUMN_VALUE_LENGTH to skip long free-text cells (see Bounding Ingestion).

Column Nodes

Each column that produces value nodes also gets a SchemaColumn node. The node is named table:column, using dlt’s normalized identifiers, so Chinook’s Genre.Name becomes genre:name. An is_column_of edge links it to its SchemaTable, and a has_value edge links it to each distinct value of the column: its ColumnValue nodes (type DltColumn in the graph) and, for date cells, its Timestamp nodes. Like SchemaTable, a column node is scoped to its source document, so two sources with the same table:column get separate column nodes. A column gets a node only when it has value nodes. The column_value_columns and temporal_columns selections, the key-column exclusion, null and empty cell skipping, and DLT_MAX_COLUMN_VALUE_LENGTH therefore decide which columns appear. The column node’s name is embedded (the SchemaColumn_name collection). Each column node and each has_value edge is written once per run, even when its values recur across batches. To list every value of one column, filter graph search to that column node:

import cognee
from cognee.tasks.schema.models import SchemaColumn

results = await cognee.search(
    "List every genre in genre:name.",
    query_type=cognee.SearchType.GRAPH_COMPLETION,
    datasets=["chinook_columns"],
    node_type=SchemaColumn,
    node_name=["genre:name"],
    top_k=50,  # large enough to cover the column's values
)

top_k ranks edges, so the default does not return a whole column. Set it high enough to cover the column. A very large column may not fit in the answering model’s context. If two sources in one dataset share a table:column name, the same filter matches both column nodes. To search one source alone, put it in its own dataset. The runnable example is examples/guides/dlt_column_search.py.

dlt sources cognified before column nodes were added have none until their memory is rebuilt: run cognee.forget(dataset="<dataset>", data_id=<source data id>, memory_only=True) (or drop data_id to rebuild the whole dataset), then cognee.cognify(datasets=["<dataset>"]). Expect one extra node per table:column that has value nodes and one has_value edge per distinct value.

Write Dispositions

Control how data is synced on repeated runs using the write_disposition parameter:

replace (default): Drop and recreate tables on each run. Use for full snapshot refreshes.
merge: Upsert by primary key — updates existing rows, inserts new ones. Best for data that changes over time.
append: Always insert without deduplication. Use for time-series data and event logs.

# Append mode — every call adds new rows, no dedup
await cognee.remember(
    event_resource,
    dataset_name="events",
    primary_key="id",
    write_disposition="append",
)

write_disposition only controls dlt’s staging snapshot. It does not by itself make a re-run pick up rows a source has gained, because the ingestion skip in add() short-circuits an already-ingested source first — see Re-Ingesting a Source and Re-ingesting a source that keeps growing.

How It Works

Source Detection: cognee identifies dlt resources and connection strings in the input. .csv files are not detected here — they are routed by the loader engine’s dlt_csv_loader inside the ingest pipeline (see CSV Files) and join the flow from step 2 onward
Pipeline Execution: A dlt pipeline loads data into a per-dataset staging database
Schema Extraction: Table schemas, primary keys, and foreign keys are extracted. Key hints are read from the resources of the source being loaded, so a source never picks up foreign keys that an earlier source staged under the same table name
Graph Construction: Each row becomes a document node; foreign keys become edges between nodes (see Foreign Key Resolution), date-shaped cells become edges to shared Timestamp nodes (see Time Anchoring), and other non-key cells become edges to shared ColumnValue nodes (see Column Value Nodes); each column with value nodes gets a SchemaColumn node linked to them (see Column Nodes)
LLM Bypass: Structured rows skip chunking, entity extraction, and summarization — the graph is built entirely from schema metadata and cell values

Each table becomes a SchemaTable node scoped to its source document, linked to it by an is_table_of edge, so two sources with a table of the same name get separate table nodes. Its columns that have value nodes link to it through is_column_of edges from their SchemaColumn nodes. For a connection-string source, the table description names the database it was read from (Relational table '<table>' from <database> with N rows, loaded through dlt.). The database is the SQLite file name or the server database name, never the host or credentials. Each row is keyed on the primary_key you pass, if the table has that column. Otherwise cognee uses the table’s declared primary key (all columns of a composite key, with their values joined by :), then an id column, then the first column, and logs a warning for the first-column case. The key sets the row’s node id and is what foreign keys match against. With write_disposition="merge", dlt also upserts on the primary_key you pass. Before cognee 1.6.6, cognee ignored a source’s declared primary key and used id or the first column. A table whose real primary key is neither of those (a non-id key or a composite key) gets new row node ids the next time it is ingested. If you keep row node ids outside cognee, re-read them after that ingest.

Bounding Ingestion

Two opt-in caps guard against large sources; both default to 0, meaning no cap.

Rows per table: pass the max_rows_per_table kwarg to remember() / add() to bound the per-table row count for a single call, or set the DLT_MAX_ROWS_PER_TABLE environment variable to change the process-wide default.
Column-value length: DLT_MAX_COLUMN_VALUE_LENGTH bounds the length of the cell values that become shared ColumnValue nodes — cell-level graph nodes that link rows sharing the same value in a selected column. (Columns are selected with column_value_columns — in dlt_config on remember() / add(), or as a bare add() kwarg — or with DLT_COLUMN_VALUE_COLUMNS; every non-key column is selected by default, see Column Value Nodes.) A positive value skips selected cells longer than that many characters, dropping rather than truncating them — worthwhile for free-text-heavy columns or wildcard ("*") selection, since long one-off values make poor shared nodes and each unique value costs one embedding. Unlike max_rows_per_table, this cap has no per-call kwarg: set the environment variable or the dlt_max_column_value_length ingestion config field. Cognee previously applied a fixed 256-character cap unconditionally; set DLT_MAX_COLUMN_VALUE_LENGTH=256 to keep that behavior.

Re-Ingesting a Source

Each relational dlt source is stored as a single record whose identity is stable: it is derived from the dataset name and the source name (dlt_source:{dataset_name}:{source_name}) and does not depend on the data. Re-running remember() / add() on the same source therefore never creates a second copy — but what happens to the existing record depends on how you call it:

Plain re-add (the default): add() is idempotent. A source that has already been ingested keeps its record, content hash, and cognify status as-is — whether or not the upstream data changed — and nothing is reprocessed.
Explicit re-ingest: to pick up upstream changes, call add(..., incremental_loading=False, data_cache=False) — the completed-skip runs whenever either flag is on, so both must be off — or use update() with the record’s UUID. The record then updates in place under the same identity (a content hash over the source’s tables and rows tracks the change), so the source is never absent from the store, and the next cognify() purges the source’s previously derived artifacts from the graph and vector stores before re-emitting the current rows. Rows deleted upstream disappear, and changed rows do not keep their stale values alongside the new ones. Between the re-ingesting add() and cognify(), searches still return the source’s previous rows; they are replaced only once the re-cognify completes.

The purge is a real delete, re-authorized as one: re-ingesting a changed dlt source requires delete permission on the dataset. If that permission is missing, the run fails instead of continuing — silently skipping the purge would leave stale rows in the graph and present them as current.

The purge also reaches the session cache: cached session turns whose answers used the purged graph elements are removed, along with the feedback and session-context lessons distilled from them, so a re-ingest cannot replay the source’s old rows out of session memory. This part is best-effort — a session-cache error is logged as a warning and never fails the run. See Invalidation when the underlying data is deleted for how the targeted pass decides what is contaminated.

Renaming a Source

Because the identity is keyed on the dataset and source names, renaming either one is a remove + add, not an in-place rename. The new name ingests from scratch as a fresh source, and the records under the old name stay in the dataset — a re-ingest only reconciles the source names it just ingested, so it will not treat the old name’s records as orphans. Delete them explicitly (for example with cognee.forget(...)) if you do not want both. For the same reason, two dlt sources that share a name within a single add() call would resolve to the same identity. cognee raises an error rather than letting one silently overwrite the other, so give each source a distinct name. CSVs are keyed the same way, but through dlt_csv_loader rather than that check: a CSV’s source name is its file-name stem, with runs of characters other than letters, digits, and underscores replaced by _, leading and trailing separators stripped, and the result lowercased. Two CSVs whose stems normalize to the same string (for example Employees 2024.csv and employees_2024.csv) therefore share one manifest record within a dataset, with the later ingest updating the earlier one. Give CSV files names that stay distinct under that normalization.

Foreign Key Resolution

The staging tables dlt creates have no key constraints, so cognee reads a source’s foreign keys from the key hints in dlt’s schema. dlt records those hints only when the sql_database source is built with resolve_foreign_keys=True. cognee sets it for you when you pass a connection string, with or without query. If you build a sql_database(...) source yourself, pass it too (see Cloud Data Warehouses). A foreign key becomes a graph edge only when both the source row and the target row are loaded by the same add() (or remember()) call. How foreign keys are matched to rows:

Each foreign key is matched on its own. Two foreign keys to the same table, and a table that references itself, each produce their own edges.
Composite foreign keys are matched on all of their columns together, in the target’s key order, and produce one edge named after all of them (for example order_id_line_no_references_order_lines).
Foreign keys to a UNIQUE column (columns other than the target table’s key) link to the one loaded row that holds the value. If several loaded rows hold the value, no edge is created for it, and cognee logs Foreign key <table>.<column> -> <ref_table>.<ref_column> skipped values that several rows of '<ref_table>' hold.
Edge names are <foreign key columns joined by _>_references_<target table>, for example dept_id_references_departments. The target is the bare table name, so the name is the same on SQLite and Postgres.
An edge is written by the batch that writes the later of its two rows, so it is created even when the rows land in different batches.

Some foreign keys find no target row. cognee logs a warning for each of these cases, so missing edges can be diagnosed:

Target table not loaded by this add(): if the call loaded no rows of the referenced table, for example a single-table query or an empty table, its rows get no edge. cognee logs one warning for each foreign key column, saying this add() loaded no rows of '<ref_table>' and how many rows got no edge. The foreign key definition is still kept on the table’s SchemaTable node. Loading the tables in separate add() calls on one dataset does not help: every connection-string source is named sql_database, so the second call maps to the same source record and is skipped as already ingested. Load the related tables in one add().
Target row not loaded: if the referenced table was loaded but the row a foreign key points at was not, most commonly because the target table hit a max_rows_per_table cap you set, the reference is dropped and no edge is created. The warning identifies the dropped references as source_table.column -> ref_table:value. If you see missing edges, raise max_rows_per_table so the referenced rows are included.
Duplicate primary keys within a table: if multiple rows in a table share the same primary key, foreign key edges that target that key resolve to the last such row loaded; earlier rows with the same key are shadowed for FK targeting. The warning names the affected table and how many rows share a key with an earlier row.
Key hints dlt cannot provide: a resource whose table name depends on its items has no static key hints, so its rows get no foreign key edges. A foreign key hint whose column lists differ in length is skipped. cognee logs a warning in both cases.

SchemaTable and SchemaRelationship node ids include the source document they belong to. Ids you stored from a source cognified before node ids were scoped this way no longer match the nodes cognee emits; look them up again after you re-ingest the source.

Connectors

Connectors are dlt sources for a specific system. The list below keeps the current connector packages visible; the routing details are tucked away for reference.

Connector Modes

Cognee supports two DLT connector modes:

Relational connectors take the default dlt path described above: each row becomes a schema-context document and foreign keys become edges, all built deterministically without LLM extraction.
Document-mode connectors opt each row into normal cognify instead: the row is turned into a text document that goes through LLM entity extraction, just like unstructured text passed to remember().

A dlt source opts into document mode by setting the cognee_document_source attribute (via the document_source_tag() helper in cognee.tasks.ingestion.dlt_utils) to a short source tag. cognee then routes every row from that source through cognify rather than the relational schema-context path:

Each row is built from its title and content columns (rendered as # {title}\n\n{content}, or just the content when there is no title), with optional url and id columns preserved in metadata.
system_metadata["source"] is set to the connector’s own tag (for example "notion") instead of "dlt", alongside title and, when present, url and external_id (from the row’s id).
The rendered row text is stored literally. A row whose content happens to be nothing but a URL or an existing local file path is kept as that text — cognee never fetches or reads it the way it would a string you passed to add() yourself — and such a row is never recorded as having originated from that location. This applies to every document-mode row, titled or not, because the content comes from the provider rather than from the caller.

Because the tag is connector-provided, the shared ingestion engine stays connector-agnostic: a connector declares its own nature rather than being hard-coded by name.Per-row node sets. A document-mode row can name its own node sets in the reserved cognee_node_set column (NODE_SET_COLUMN in cognee.tasks.ingestion.dlt_utils). The column must hold a JSON list of names; cognee applies a dlt json type hint to it on every document source, so the list stays on the row instead of becoming a child table.

Each name is prefixed with the source tag (<source_tag>:) unless it already starts with it, so provider data can never target cognee’s own node sets such as skills or user_context. Names that resolve to the same node set (for example Project A and project_a) are deduplicated, and the first spelling is kept.
The row’s node sets are merged with the node_set you pass to add() / remember(), so the document, its chunks, and the entities extracted from it land in both, and node_name filtering in recall works per row. The Notion connector, for example, tags each page with ["notion:<workspace_id>:<root_id>"].
An unset value (None or [], which read back as NULL, "[]", or "null") is left out of the row’s content hash, so adding the column without any tags does not change existing document ids or re-ingest rows. Do not emit an empty string: dlt stores it in a variant column and the sync stops. Changing a row’s actual node sets re-ingests it like any other column change. A row states its full node sets, so a re-ingested row with no node set (unset column or []) clears the node sets stored for that document instead of keeping them.

Any other shape stops the whole sync: a bare string, number, or object, a value dlt moved into a cognee_node_set__v_* variant column, text that is not JSON, or an entry that is not a non-empty string. cognee raises a CogneeValidationError named DltNodeSetColumnNotAList that names the source, table, and row, and no document from that sync is stored. Emit a list even for a single name (["name"]) and re-sync the affected rows.

Per-row structure. A document-mode row can say where it sits in its source’s tree in the reserved cognee_structure column (STRUCTURE_COLUMN in cognee.tasks.ingestion.dlt_utils), which also gets a dlt json type hint. It holds {"ancestors": [{"kind", "id", "name"?, "document"?}, ...]}, nearest parent first. An entry with "document": true is another row of the same source and staging table, matched by its id. It can only be the last entry, and it cannot be the row itself. Other entries are containers that have no row of their own, such as a Notion database. A row at the top of its tree has an empty list. An unset value means the row has no position.

The column is never part of the row’s content hash. A row that only moved keeps its data_id and is not extracted again. cognee writes the new position onto the existing record.
After each cognify(), cognee reconciles the structure of the whole dataset, without an LLM. It adds child_of edges from each child’s Document node to its parent’s. The parent is found by external_id within the same source, staging table, and dataset. Entries without "document": true become StructureContainer nodes, scoped to the dataset. A parent that is not in the dataset gets no edge. Stale edges are removed, so a moved row keeps one parent edge. A dataset whose rows carry no structure is left untouched.
The pass runs when the cognify run completes, and also when some items in the run failed. On a completed run, a reconciliation error is raised to a foreground cognify() caller even though the run is already recorded as completed. It repeats on every cognify of that dataset until it is fixed. On a failed run, the run’s own error is the one you get.
Any other shape stops the whole sync with a CogneeValidationError named DltStructureColumnInvalid, and no document from that sync is stored. That includes a value without an ancestors list, an entry with unknown keys, an entry without a non-empty kind and id, or a structure on a row with no id.

Hosts and connectors can check DOCUMENT_SYNC_VERSION in cognee.tasks.ingestion.dlt_utils to gate on core capability. Version 2 adds the per-row cognee_node_set column, and version 3 adds the per-row cognee_structure column. The Notion, Obsidian, and OneNote connectors refuse to run on a core older than 3 and the Apollo and Asana connectors on one older than 2, while the Gmail, Google Drive, Google Calendar, and Linear connectors still need only version 1.Sync and orphan cleanup. Document sources read back their full current snapshot (max_rows_per_table=0) and honor the write_disposition you pass: use replace for snapshot sources with no delete feed and merge with a hard-delete tombstone column for incremental sources that emit real deletions. primary_key defaults to id. Orphan cleanup is scoped to the source tag and the staging table the source loaded, so reconciling one document source (or one Drive folder) only removes that table’s rows, and relational ("dlt") rows and other sources’ rows are never cross-deleted in a mixed dataset. A table that tombstones have emptied still counts as loaded, so deleting the last message or file upstream forgets its document too. An incremental sync that brings no changes still re-reads the source’s own staged rows, so a dataset whose processing failed, or whose documents were removed locally, recovers on the next sync without anything changing upstream. Documents ingested before staging tables were recorded on each row are kept rather than guessed at. Deleting an orphan also invalidates the cached session turns that used that row’s graph elements, so a row removed upstream stops surfacing from session memory too; that step is best-effort and logged as a warning if it fails.

Orphan cleanup runs in the foreground of add() / remember(), and only after the pipeline finishes successfully — a run that returns an error leaves the previous records in place. Background runs (run_in_background=True) skip it, logging Skipping DLT orphan cleanup for background ingestion, because the detached pipeline has not committed when the call returns. To propagate upstream deletions, sync the affected source in the foreground.Cleanup sweeps the dataset the rows were just written to, found by the id add() resolved for it rather than by name, whether you passed dataset_name or dataset_id. A sync into a dataset shared with you by id therefore cleans that dataset, never your own dataset of the same name. Cleanup deletes, so it needs delete permission on that dataset. With write permission only, add() still commits your rows but skips the sweep and logs Skipping dlt orphan cleanup for dataset '<name>': no delete permission, so rows deleted upstream stay in the dataset. Upstream deletions then do not propagate until someone with delete permission syncs the source.

Gmail connector

Gmail Connector

The Gmail connector is a first-class dlt source that turns your inbox into cognee memory. It reuses the same remember() + dlt path described above, so it gets incremental re-sync and forget-on-delete for free. gmail_source() returns a dlt resource that you hand directly to cognee.remember().

This connector reads the content of your email. It is strictly opt-in — nothing is fetched until you construct a source and call remember(). Scope what you ingest with label_ids, keep the OAuth token file (token.json) private, and prefer a dedicated dataset so you can wipe it with a single cognee.forget(...).

The Gmail connector is bundled with the Cognee SDK (from v1.6.1) and imported from cognee.tasks.ingestion.connectors. It supersedes the standalone cognee-community-connector-gmail package, so no community package is needed. To let users connect their mailboxes through Cognee’s API instead of a local token file, see the Gmail integration.

Installation

pip install "cognee[gmail]"

Or with uv:

uv pip install "cognee[gmail]"

The gmail extra pulls in google-api-python-client, google-auth, and google-auth-oauthlib. They are imported lazily: importing cognee.tasks.ingestion.connectors works without the extra, and the ImportError naming it is raised only when the Gmail client is built, during the remember() call.

One-Time OAuth Setup

The connector authenticates with Gmail via the OAuth2 installed-app (Desktop app) flow using the read-only scope https://www.googleapis.com/auth/gmail.readonly — it never modifies your mailbox.

In the Google Cloud Console, enable the Gmail API, configure an OAuth consent screen (add yourself as a test user), and create an OAuth 2.0 Client ID of type Desktop app.
Download the client-secret JSON and save it as credentials.json (or point credentials_path at it).
The first run opens a browser to consent and caches the resulting user token at token.json (token_path). Later runs reuse and silently refresh that token.

Usage

import cognee
from cognee.tasks.ingestion.connectors import gmail_source

source = gmail_source(
    credentials_path="credentials.json",
    token_path="token.json",
    label_ids=["INBOX"],
)

await cognee.remember(
    source,
    dataset_name="gmail_inbox",
    primary_key="id",
    write_disposition="merge",   # incremental upsert by message id
    max_rows_per_table=0,        # 0 = no row cap (see note below)
)

gmail_source() accepts these keyword-only parameters:

Parameter	Default	Description
`credentials_path`	`"credentials.json"`	Path to the OAuth client-secret JSON (Desktop app).
`token_path`	`"token.json"`	Where the cached user token is read/written.
`label_ids`	`None`	Restrict to messages carrying all of these Gmail label ids (e.g. `["INBOX"]`). `None` covers the whole mailbox except Spam and Trash.
`max_results`	`None`	Load only the newest N messages (handy for demos/tests). Each capped run starts again from the newest message and saves no incremental cursor, so later runs don’t sync only the changes. `None` = no cap.
`resource_name`	`"gmail_messages"`	dlt resource (and staging table) name. Give each mailbox its own name when several share one dataset or pipeline, so their cursors stay separate.
`service`	`None`	A pre-built Gmail API client, used instead of the local OAuth flow. `build_gmail_service_from_access_token(token)` builds one from an access token you already hold.
`check_active`	`None`	Optional callable run during extraction; raising from it aborts the sync (the integrations API uses it to stop when a connection is revoked mid-sync).
`quota_units_per_minute`	`5000`	Gmail API quota budget. Gmail allows 6,000 units per user per minute and fetching one message costs 20, so the default fetches about 250 messages a minute. Sources with the same `resource_name` in one process share one budget, and the first value set for that name wins.

The query and include_spam_trash parameters of the community package are gone.The returned resource (named resource_name, gmail_messages by default) is preconfigured with primary_key="id", write_disposition="merge", and a _deleted hard-delete column, so combined with primary_key="id" on remember() it performs idempotent upserts by Gmail message id.

Gmail is a document source, so Cognee always reads its whole staging table back, whatever max_rows_per_table or DLT_MAX_ROWS_PER_TABLE says. Forget-on-delete therefore always compares against the whole synced corpus; passing max_rows_per_table=0 just makes that explicit.

How It Works

Incremental sync: The first run lists the label from the newest message down. If it wasn’t capped with max_results and finished, it records the mailbox historyId in dlt’s per-resource state; later runs call users.history.list(startHistoryId=...) and emit only the delta (added / changed / deleted messages).
Forget-on-delete: Messages reported as deleted or trashed by the History API are emitted with the _deleted hard-delete marker. dlt removes those rows from its destination on merge, and cognee’s existing orphan_cleanup then purges them from the graph, vector, and relational stores.
Scope changes: Changing the set of label_ids makes the next run do a full backfill of the new scope, so older messages that now match are picked up; reordering the same labels keeps the cursor. That backfill also forgets every previously synced message outside the new scope, so narrowing label_ids removes the mail that no longer matches. On incremental runs, a message that leaves the scope (for example, trashed INBOX mail) is forgotten the same way. A backfill capped by max_results, or one that fails midway, forgets nothing.
History expiry: Gmail expires history after roughly a week. On the resulting 404 the connector loads every message in the label again rather than silently stalling.
Transient errors: every Gmail API call retries up to 6 times. A quota rejection (a 429, or a 403 with reason rateLimitExceeded or userRateLimitExceeded) that survives those retries then waits out a full 60-second window and tries again, up to five times. A call that still fails — a message fetch or the History API request — aborts the run before the cursor advances, so the next sync retries from the same point. Only a real 404 from the History API counts as the expired history above, and only a 404/410 on a message fetch counts as a deletion.

Deletions only propagate from a foreground run — see the orphan-cleanup note above. For a runnable walkthrough, and for when the cursor carries over, what capped syncs and prune do, and how to load a whole inbox, see the Gmail Ingestion guide.

Linear connector

The Linear connector turns one Linear team’s issues, comments, and projects into cognee memory. linear_source() returns a dlt resource you hand directly to cognee.remember(), so it takes the document-mode path described above (its source tag is linear) and reuses the same merge upserts by row id. It does not emit deletions yet — see What Is Ingested below.

This connector reads the content of your Linear workspace. It is strictly opt-in — nothing is fetched until you construct a source and call remember(). The token’s visibility is the scope, so private teams it can see are included, and anyone with read access to the target dataset can see what was ingested from them. Prefer a dedicated dataset so you can wipe it with a single cognee.forget(...).

This is the connector you drive from your own code with a token you already hold. It is not the Linear integration, which installs Cognee into a workspace as an OAuth agent.

Installation

Nothing to install. Unlike Gmail, Google Drive, and Google Calendar, the Linear connector needs no extra — dlt is a core cognee dependency — so cognee.tasks.ingestion.connectors carries it out of the box. All you need is an access token.

Usage

import cognee
from cognee.tasks.ingestion.connectors import linear_source

await cognee.remember(
    linear_source(
        team_id="<team uuid>",
        access_token="<oauth token or lin_api_... key>",
    ),
    dataset_name="linear",
    primary_key="id",
    write_disposition="merge",   # REQUIRED — the add pipeline defaults to "replace"
    max_rows_per_table=0,        # 0 = no row cap
)

Pass the team’s id (the uuid), not its key. One source is one team, so give each team its own linear_source() call. Run it again against the same dataset to sync only what changed.

write_disposition="merge" is mandatory. With the default "replace", the second sync drops and reloads the table instead of upserting into it.

linear_source() accepts these keyword-only parameters:

Parameter	Default	Description
`team_id`	— (required)	The Linear team’s id — the uuid, not the team key.
`access_token`	`None`	An OAuth access token or a personal `lin_api_...` API key. Required unless you pass `service`.
`service`	`None`	A pre-built Linear client used instead of building one from `access_token`. `build_linear_service(token)` builds one; mainly an injection point for tests and hosts.
`resource_name`	derived from `team_id`	dlt resource (and staging table) name. Defaults to `linear_` followed by the lowercased team id with runs of non-alphanumeric characters replaced by `_`. Keep the `linear_` prefix if a host tracks the table by it.
`check_active`	`None`	Optional callable run before each request; raising from it aborts the sync (hosts use it to stop when a connection is revoked mid-sync).
`include_archived`	`True`	Include archived issues, projects, and comments. Linear archives closed issues automatically, so leaving this on keeps most of a team’s history. Trashed issues are always skipped.
`page_size`	`50`	Rows per page, and also the maximum — larger values are clamped. A page of issues with their nested comments has to stay under Linear’s 10,000-point per-query complexity limit.
`max_requests`	`1500`	Request budget for a single run. A bigger team finishes over several runs.

The returned resource is preconfigured with primary_key="id", write_disposition="merge", and a _deleted hard-delete column.

What Is Ingested

Rows are flat {id, title, content, url, _deleted} documents whose ids are prefixed by kind:

Row	Id	Title	Content
Issue	`issue:<uuid>`	`ENG-12 Title`	state, assignee, priority, labels, project, parent, description, and the issue’s comments
Project	`project:<uuid>`	project name	status, lead, start and target dates, description, content

A comment is part of its issue’s document rather than a row of its own, so asking about a discussion finds the issue it belongs to.

Sub-teams are not included. Select the sub-team itself to ingest its issues.
An issue moved to another team stays in the old team’s table.
A project shared by several teams is ingested once per selected team.
Deletions are not propagated yet. Trashed issues are skipped, but anything already ingested stays in memory: the _deleted column is declared and always False. A comment deleted in Linear is hard-deleted there, so it is never seen again and stays in the issue’s document until the issue changes next.

How It Works

Incremental by updatedAt: Linear sorts newest first and has no change feed, so each of the three streams — issues, comments, projects — keeps its own floor, plus the ids seen exactly at that floor, in dlt’s per-resource state. A run with no changes yields nothing.
Comments are their own stream: an edited comment does not reliably bump its issue’s updatedAt, so changed comments are read separately and their parent issues are re-rendered.
A run is bounded: max_requests (1,500 by default) caps one run. Linear allows 5,000 requests and 2,000,000 complexity points per hour for an OAuth app user (2,500 and 3,000,000 for a personal API key), shared with everything else that user does.
It stops cleanly rather than raising or sleeping: on a Linear rate limit (a gateway HTTP 429 counts as one), remaining quota below a reserve, a spent request budget, or a token that expires after progress was made, the source stops where it is, keeps the state it reached, and records it in the cognee_sync_stats dict on the returned resource — failed, plus failed_rate_limit, failed_budget, or failed_auth. Raising would make dlt roll the run’s state back, so a team too big for one hour of quota would never finish; sleeping would hold the process-wide staging lock.
The next run resumes from the cut point, down to the page cursor it stopped at, so even a large group of rows sharing one timestamp is read across several runs.
Errors that do raise: a token rejected before any progress was made, a team that cannot be found, a pagination response that does not advance, and other API failures raise LinearAuthError, LinearTeamNotFoundError, or LinearAPIError. Their messages carry codes only — never the token, the query variables, or a response body.

Tokens

The source takes a plain access token and never refreshes it; a host that owns the OAuth flow passes a fresh token per run. OAuth tokens are sent as Bearer, personal API keys (lin_api_...) are sent as is. Surrounding whitespace is stripped, and a token that is empty or contains spaces, non-ASCII, or non-printable characters is refused when the source is built, without echoing the value.

Google Calendar connector

The Google Calendar connector syncs your calendars into cognee memory, incrementally, and forgets events when they are deleted. google_calendar_source() returns a dlt resource you hand directly to cognee.remember(). It takes the document-mode path described above (its source tag is google_calendar): each event becomes one document that goes through chunking and LLM entity extraction. The calendar name, organizer, and attendees with their responses are written into the document, so the same person is linked across meetings.

This connector reads personal data from the calendars you select. It is strictly opt-in — nothing is fetched until you construct a source and call remember(). Keep the token file private and prefer a dedicated dataset so you can wipe it with a single cognee.forget(...).

The connector is bundled with the Cognee SDK and imported from cognee.tasks.ingestion.connectors. It replaces the cognee-community Google Calendar package, so no community package is needed. To let users connect their calendars through Cognee’s API instead of a local token file, see the Google Calendar integration.

Installation

pip install "cognee[google-calendar]"

The google-calendar extra pulls in google-api-python-client, google-auth, and google-auth-oauthlib. They are imported lazily: importing cognee.tasks.ingestion.connectors works without the extra, and the ImportError naming it is raised only when the Calendar client is built.

One-Time OAuth Setup

The connector uses the OAuth2 installed-app (Desktop app) flow with the read-only scope https://www.googleapis.com/auth/calendar.readonly only. It never creates, changes, or deletes anything in Google Calendar.

In the Google Cloud Console, create a project and enable the Google Calendar API.
Under Google Auth Platform, set up the consent screen as External, keep it in Testing, add your own account as a test user, and add the calendar.readonly scope.
Under Clients, create an OAuth client of type Desktop app and download its JSON.

Point the connector at it and sign in once (a browser opens for consent):

export GOOGLE_CALENDAR_CREDENTIALS_PATH=~/client_secret.json
export GOOGLE_CALENDAR_TOKEN_PATH=~/.cognee_google_calendar_token.json

import os

from cognee.tasks.ingestion.connectors.google_calendar import (
    build_calendar_service,
    list_calendars,
)

# Caches the token in the second file, written with mode 0600.
service = build_calendar_service(
    os.path.expanduser(os.environ["GOOGLE_CALENDAR_CREDENTIALS_PATH"]),
    os.path.expanduser(os.environ["GOOGLE_CALENDAR_TOKEN_PATH"]),
)
print(list_calendars(service))  # the calendar ids you can sync

Apps left in Testing get refresh tokens that expire after 7 days. Publish the app, or sign in again, for longer use.

Headless or hosted: instead of the browser flow, set GOOGLE_CALENDAR_REFRESH_TOKEN with GOOGLE_CALENDAR_CLIENT_ID and GOOGLE_CALENDAR_CLIENT_SECRET, or pass a ready-made access token (access_token= or GOOGLE_CALENDAR_ACCESS_TOKEN), which is never refreshed. build_calendar_service_from_access_token(token) builds a client from a token you already hold. When several are set, an access token wins, then the refresh token, then the installed-app flow. A source built with no credentials at all raises a ValueError right away, before any pipeline work starts.

Usage

import cognee
from cognee.tasks.ingestion.connectors import google_calendar_source

await cognee.remember(
    google_calendar_source(calendar_ids=["primary"]),  # GOOGLE_CALENDAR_* settings for auth
    dataset_name="my_calendar",
    primary_key="id",
    write_disposition="merge",   # REQUIRED — incremental upserts and deletes
    max_rows_per_table=0,        # 0 = no row cap
)

answer = await cognee.recall("Who is in my Thursday meetings?", datasets=["my_calendar"])

Run the same remember() call again whenever you want to sync.

Parameter	Setting (env var)	Default	Description
`calendar_ids`	`GOOGLE_CALENDAR_IDS` (comma-separated)	`primary`	Calendars to sync. An empty list is refused, because it would forget every synced event. `primary` and the real id of the primary calendar are the same calendar and give one set of documents.
`settings`	`GOOGLE_CALENDAR_CLIENT_ID`, `_CLIENT_SECRET`, `_REFRESH_TOKEN`, `_CREDENTIALS_PATH`, `_TOKEN_PATH`	read from the environment	A `GoogleCalendarSettings` object. `token_path` defaults to `~/.cognee_google_calendar_token.json`.
`access_token`	`GOOGLE_CALENDAR_ACCESS_TOKEN`	`None`	A ready-made token for hosts and quick tries; never refreshed.
`resource_name`	—	`"google_calendar_events"`	dlt resource (and staging table) name. Give each account its own name when several sync into one dataset, so their state stays separate.
`check_active`	—	`None`	Optional callable run before every API request; raising from it aborts the sync.
`service`	—	`None`	A pre-built Calendar client, mainly for tests.

All parameters except calendar_ids are keyword-only. The returned resource is preconfigured with primary_key="id", write_disposition="merge", and a _deleted hard-delete column.

How Sync and Forget-on-Delete Work

Sync tokens: the first run lists every event of each calendar; later runs pass that calendar’s nextSyncToken and get only what changed. The token is saved in dlt state once the whole round has finished.
Recurring events: a series is one document with its RRULE, and a changed single occurrence is its own document.
All-day events show their last day: a one-day event reads all day, 2026-11-08, a three-day one all day, 2026-11-08 to 2026-11-10.
Deleted events come back as cancelled and are forgotten if they ever reached memory. A cancelled single occurrence forgets only that occurrence; deleting a whole series also forgets its changed occurrences. An event created and deleted between two syncs is neither stored nor counted.
Expired sync token: a 410 Gone lists that calendar again and reconciles it like a snapshot, so events deleted in the meantime are still forgotten.
Removed calendars: removing a calendar from calendar_ids forgets its events. A calendar is also forgotten when it was synced before and has left the account’s calendar list. A 404 for a calendar that is still listed, or that was never synced (most likely a typo), is a failure, not a deletion.
Failures never delete: a calendar that fails keeps its old state and is retried on the next run, while the other calendars finish and save theirs. The run raises only when no calendar finished. A 401, a 429, or a 403 with a rate-limit reason ends the run early, since the remaining calendars would fail the same way; any other 403 (lost access) fails only its own calendar. This holds for HTTP responses only: a transport error, or an auth error raised by the Google client library before any HTTP call (such as google-auth’s RefreshError), ends the whole run.

Deletions only propagate from a foreground run — see the orphan-cleanup note above. For a runnable walkthrough, see the Google Calendar Ingestion guide.After a run, the cognee_sync_stats dict on the returned resource holds counts only — never titles or content: scanned, skipped, deleted, calendars, and failed, plus failed_auth (HTTP 401), failed_rate_limit (HTTP 429, or 403 with a rate-limit reason), or failed_calendar_sync (anything else) for the reason a calendar failed.

What Is Read

From the selected calendars only: title, calendar name, start and end with time zone, location, organizer, attendees with name, email, and response, the recurrence rule, status, event type, the Google Meet link, the description, and the event link.
Not read: attachments, conference details beyond the Meet link, reminders, colors, and other calendars. workingLocation events are skipped.
dlt state stores each calendar’s id, its sync token, and its event ids — not titles or descriptions. Logs and errors carry counts and HTTP statuses only; calendar ids and request URLs are left out.
Event text becomes ordinary cognee documents, so during cognify it is sent to your configured LLM and embedding providers like any other data. Use local models if that matters to you.
The token file holds a refresh token and is written with mode 0600. Never commit it; revoke access any time at myaccount.google.com/permissions and delete the file.

Limitations

The full history of each calendar is read on the first run; the sync-token API allows no date window.
workingLocation events are skipped and attachments are not ingested.

Notion connector

The Notion connector syncs the Notion pages and database rows you choose into cognee memory, incrementally and with forget-on-delete. notion_source() returns a dlt resource you hand directly to cognee.remember(), so it takes the document-mode path described above (its source tag is notion): each page becomes a text document that goes through normal chunking and LLM graph extraction.

This connector reads the content of your Notion pages. It is strictly opt-in — nothing is fetched until you construct a source and call remember(), and only the roots you name (plus everything nested under them) are read. Prefer a dedicated dataset so you can wipe it with a single cognee.forget(...).

On a self-hosted Cognee backend, users can connect a workspace over OAuth and pick pages in a picker instead of calling notion_source() by hand. See the Notion integration, which runs this connector for each selected root.

Installation

Nothing to install. The Notion connector talks to the Notion REST API over httpx, a core cognee dependency, so like Linear it needs no extra and ships in cognee.tasks.ingestion.connectors with a plain cognee install.

Setup

Create a Notion internal integration and copy its token.
Share every page or database you want synced with that integration (... > Connections > Add connection).
Set NOTION_API_KEY, or pass token= directly, plus your LLM_API_KEY as for any other cognee run.

Usage

import cognee
from cognee.tasks.ingestion.connectors import notion_source

await cognee.remember(
    notion_source(
        root_page_ids=["<page id from the page URL>"],
        resource_name="notion_<connection id>",
    ),
    dataset_name="my_notion_space",
    primary_key="id",
    write_disposition="merge",   # incremental upsert + tombstones
    max_rows_per_table=0,        # 0 = no row cap
)

A database (or one of its data sources) works the same way:

notion_source(root_data_source_ids=["<database or data source id>"])

Run it again against the same dataset to pick up edits and forget pages that are gone.notion_source() accepts these parameters:

Parameter	Default	Description
`token`	`None`	Notion integration token. Falls back to the `NOTION_API_KEY` environment variable; with neither, a `ValueError` is raised.
`root_page_ids`	`None`	Page ids to walk, plus everything nested under them.
`root_data_source_ids`	`None`	Data source ids to sync, plus everything nested under their rows. A database id is also accepted and resolved into that database’s data sources. At least one of `root_page_ids` / `root_data_source_ids` is required, or a `ValueError` is raised.
`workspace_id`	`None`	Stable workspace id used in each row’s node set. When omitted it is resolved once from the integration’s bot user (`users/me`); the sync fails rather than emitting untagged rows if it cannot be determined.
`resource_name`	`"notion_pages"`	dlt resource name — also the staging table and the key of the incremental state. See the warning below.
`check_active`	`None`	Optional callable run during extraction; raising from it aborts the sync (hosts use it to stop when a connection is revoked mid-sync).
`http_client`	`None`	A pre-built `httpx` client, mainly a test-injection point; when omitted one is built for the sync.

The returned resource is preconfigured with primary_key="id", write_disposition="merge", and a _deleted hard-delete column. Counts-only diagnostics (never titles or content) are exposed in the cognee_sync_stats dict on the returned resource.

Every Notion source that syncs into the same dataset needs its own resource_name. Two sources sharing one share state, so each run tombstones and forgets the other’s pages. A run refuses to continue when the stored state belongs to a different workspace, but two root sets in one workspace under one name cannot be told apart from an edited selection. Renaming a source starts it over, and documents synced under the old name are not cleaned up.

What Is Ingested

One document per page. Rows are {id, title, content, url, _deleted}. A database row is a page too: its properties (rich_text, select, multi_select, status, date, people, number, checkbox, url, email, relation ids) are rendered as Name: value lines at the top of its content. Relation and people values stay raw ids/names in that text; they are not resolved into graph edges.
Nested content is followed: sub-pages, databases embedded in pages, and content inside columns, toggles, callouts, lists, and synced blocks, at any depth. A synced block that copies another page’s original is not searched; its sub-pages are only synced when the original’s page is under a selected root. Archived and trashed pages are skipped.
Node sets: each row carries cognee_node_set = ["notion:<workspace_id>:<root_id>"] — always ids, never titles — so the page’s document and chunks belong to the root you configured it under. When a page is reachable from two selected roots, the nearest root wins.
Structure: each row also carries cognee_structure (see Connector Modes above). A sub-page lists its parent page, even when it sits inside a column or toggle: the parent is the page that holds it, not the block. A database row lists its data source, its database, and the page the database sits on; under a root you passed as a database or data source id, the page is not listed. A root has an empty list. After each cognify, these become child_of edges: page to parent page, database row to data source, data source to database, and database to its page. Databases and data sources become StructureContainer nodes, and the graph visualization treats child_of as a structural edge. A parent outside the selected roots, or one that was forgotten, gets no edge. Deleting a page leaves its children where they are, and a database disappears with its last row (on a graph that does not store its own provenance, at the next cognify).
Attachments (file, image, PDF, video blocks) become a name/caption reference line only; the binary is not downloaded.

How It Works

Re-walk every run: Notion has no delete feed, so each run walks the selected trees again to rediscover what still exists. A page’s content is only re-rendered when its last_edited_time, its root, or its position in the tree changed since the previous run; unchanged pages are not re-ingested.
Forget-on-delete: pages that vanished (unshared, trashed, or moved out of the selected roots) are emitted as _deleted tombstones; dlt drops them on merge and cognee’s orphan_cleanup forgets them from the graph, vector, and relational stores (foreground runs only — see the orphan-cleanup note above).
Moving a page between two selected roots, or changing roots so that another one reaches it, changes its node set (part of the row’s identity), so it is re-ingested once under the new root. Moving a page inside its root keeps its document id and does not run extraction again; only its child_of edge changes. Renaming a database re-reads its rows once, because its title is part of their structure.
Upgrading: the first sync after upgrading to a Cognee with DOCUMENT_SYNC_VERSION 3 re-reads every page once to record its position. That costs Notion API requests, not extraction, except for pages whose rendered text changed in the meantime.
Cost: a run makes at least one request per page even when nothing changed — a 20,000-page tree is tens of thousands of requests, hours at Notion’s rate limit of about 3 requests per second. The connector does not throttle itself; HTTP 429 and 5xx responses are retried (up to 5 attempts) using Retry-After or exponential backoff.

Errors

A run aborts before any tombstone is written — so a partial walk can never look like a mass deletion — when:

a transient error (network failure, or a 429/5xx that exhausts the retries) occurs;
listing a readable page’s blocks or querying a data source fails, or Notion reports more results without a next_cursor;
one of your explicit roots returns 403/404 (a configuration problem, not a deletion);
a data source has more than 10,000 rows (see limitations).

A 403/404 on a page or database reached through the tree, by contrast, means that item is gone, and it is tombstoned.

Limitations

Data sources over 10,000 rows abort every run: Notion cuts a query off there, and syncing only the first 10,000 would forget the rest.
The walk holds cognee’s process-wide dlt staging lock, so while a large tree syncs every other dlt ingestion in the same process (Drive, Gmail, Google Calendar, Linear, CSV) waits.
Teamspaces are not exposed by Notion’s public REST API, so pages are grouped by selected root, never by teamspace.
Text shown from other objects goes stale: a page is only re-rendered when it changes itself, so sub-page and database titles, synced copies, and mentions stay as they were until the page is edited.
Pinned to Notion API version 2025-09-03, the first with data sources as first-class objects.

Apollo connector

The Apollo connector syncs one Apollo.io team’s contacts, accounts, and sequences into cognee memory, incrementally, and forgets records when they are deleted in Apollo. apollo_source() returns a dlt resource you hand directly to cognee.remember(), so it takes the document-mode path described above (its source tag is apollo): each record becomes a document that goes through chunking and LLM entity extraction. It reads your team’s own CRM records, never Apollo’s enrichment database, and never creates, changes, or deletes anything in Apollo.

This connector reads personal data about people outside your organisation. It is strictly opt-in — nothing is fetched until you construct a source and call remember(). During cognify the rendered text is sent to your configured LLM and embedding providers like any other document, so check that you may send third-party prospect data there, or use local models. Anyone with read access to the dataset can see the names it holds, so keep it in its own dataset and share it only with people allowed to see the CRM.

Installation

Nothing to install. The Apollo connector is bundled with the Cognee SDK and needs no extra: its REST client uses requests, which dlt (a core dependency) already requires.

Setup

Create an API key in Apollo under Settings > Integrations > API. A master key works. A scoped key needs the endpoints below, and only those of the kinds you sync:

Endpoint	Needed by
`api/v1/contact_stages/index`	every run: it names contact stages and carries the team id
`api/v1/contacts/search`, `api/v1/contacts/show`	contacts
`api/v1/account_stages/index`	accounts
`api/v1/accounts/search`, `api/v1/accounts/show`	accounts
`api/v1/emailer_campaigns/search`	sequences, and contacts (they name their sequences)
`api/v1/labels/index`	contacts and accounts
`api/v1/fields/index`	only with `custom_field_ids`

contacts/show and accounts/show are read only to check a record that stopped appearing. Apollo answers HTTP 403 for a team or plan without API access to an endpoint, and the run stops with an ApolloAccessError that names it. That access comes with a paid plan or a trial, not a free plan; the connector has not been tested against a live account yet. A 403 on contacts/show or accounts/show does not stop the run: the record is kept and counted in failed_access.

Usage

import cognee
from cognee.tasks.ingestion.connectors import apollo_source

await cognee.remember(
    apollo_source(api_key="<apollo api key>", resource_name="apollo_sales"),
    dataset_name="apollo_crm",
    primary_key="id",
    write_disposition="merge",   # REQUIRED — incremental upserts and deletes
    max_rows_per_table=0,        # 0 = no row cap; orphan cleanup must see the whole corpus
)

answer = await cognee.recall(
    "Which contacts at Acme are in an active sequence?", datasets=["apollo_crm"]
)

Run the same remember() call again whenever you want to sync.apollo_source() accepts these keyword-only parameters:

Parameter	Default	Description
`resource_name`	— (required)	Names this connection: it keys the sync state and the cleanup of deleted records. One per Apollo team.
`api_key`	`None`	Apollo API key. Required unless you pass `service`. A malformed key is refused when the source is built, without echoing it.
`service`	`None`	A pre-built client, from `build_apollo_service(api_key)`.
`include`	all three	Which kinds to sync: `"contacts"`, `"accounts"`, `"sequences"`. Must be a non-empty subset.
`contact_stage_ids`, `contact_label_ids`	no filter	Only contacts in these stages or lists.
`account_stage_ids`, `account_label_ids`	no filter	Only accounts in these stages or lists.
`custom_field_ids`	none	Custom fields to render, as `GET /fields?source=custom` lists their ids: `contact.<id>` or `account.<id>`. An id that list does not return stops the run with an `ApolloSourceError`. Not picklist or multi-select fields, see below.
`max_requests`	`500`	Budget of API calls in one run; a call the client retries counts once.
`check_active`	`None`	Optional callable run before every API request; raising from it aborts the sync.

The returned resource is preconfigured with primary_key="id", write_disposition="merge", and a _deleted hard-delete column.Labels are what Apollo’s UI calls lists. Every argument that takes ids takes a list or tuple of id strings; anything else, a bare string included, is refused. Changing a filter, include, or custom_field_ids changes what is selected or rendered: records that leave the selection are forgotten, and changing custom_field_ids re-ingests every record whose rendered text changes as a result.Stage and list filters are checked against Apollo’s own lists on every run. An id Apollo does not list (a typo, a deleted stage, or a list GET /labels does not return, such as another user’s private list) stops the run with an ApolloSourceError that names the ids, before anything is emitted or forgotten — a filter that matched nothing would otherwise make every synced record look deleted.

One connection per Apollo team. The first run stores the team id Apollo reports, and a run under another team’s key with the same resource_name is refused, because it would forget the first team’s records. Give every Apollo team its own resource_name, even in one dataset. A host that keeps one connection per team can read the id with one request:

from cognee.tasks.ingestion.connectors.apollo import apollo_team_id, build_apollo_service

team_id = apollo_team_id(build_apollo_service("<apollo api key>"))

What Is Ingested

Rows are flat {id, title, content, url, _deleted, cognee_node_set} documents whose ids are prefixed by kind:

Row	Id	Title	Content
Contact	`contact:<id>`	name	a sentence with its account and job title, then its account, contact stage, lists, listed custom fields, and sequences
Account	`account:<id>`	name, else domain, else “Unnamed account”	a sentence with its domain, then its account stage, lists, and listed custom fields
Sequence	`sequence:<id>`	name	a sentence with its status (active, inactive, archived), then its number of steps

Every text opens with one sentence that names its kind, and the rest is Label: value lines:

Apollo contact Dana Whitfield works at Acme as Chief Financial Officer.
Account: Acme (acme.com)
Contact stage: Interested
Sequences: Q4 Outreach (active); Renewals (paused)

A contact names its account by the account’s own title from the accounts listing, and falls back to its own organization_name when that account is not synced or names nothing. An account with no name is titled by its domain, else “Unnamed account”, so accounts with neither a name nor a domain share one entity in the graph. Renaming an account changes its text and that of every one of its contacts, so each is extracted again. The stage lines say Contact stage and Account stage so that a stage with the same name on both does not merge into one entity. Ids and stages are written as names. Never rendered: emails, phone numbers, and social profile links (Apollo’s search returns them; the connector drops them before rendering), anything from Apollo’s enrichment data, the dates and steps of a contact’s sequence activity, a sequence’s open and reply rates, and who owns a record. The activity feed, attachments, notes, tasks, and calls are not read.Each row’s cognee_node_set is ["apollo:<team id>:<kind>"], with <kind> one of contacts, accounts, or sequences, so the record’s document, chunks, and entities belong to that node set. Scope a recall to Apollo, or to one kind, with node_name:

from cognee.tasks.ingestion.connectors.apollo import apollo_team_id, build_apollo_service

team_id = apollo_team_id(build_apollo_service("<apollo api key>"))
await cognee.recall(
    "Which contacts at Acme are in an active sequence?",
    datasets=["apollo_crm"],
    node_name=[f"apollo:{team_id}:contacts"],
)

The node set and the wording of the text are part of each document’s identity, and are fixed: changing either would give every record a new document and one more extraction on your LLM key. A tombstone carries no node set.

Custom fields are not rendered unless you list them in custom_field_ids, because teams keep phone numbers and personal emails in them. Do not list picklist or multi-select fields: Apollo returns option ids for them, not labels.

How Sync and Forget-on-Delete Work

Fingerprints: Apollo has no change feed and no deleted flag, so each run lists every included kind in full and emits only the records whose rendered text changed, using a hash kept in dlt’s per-resource state.
Deleted records: a contact or account that stops appearing in a complete listing is read once more before it is forgotten. It is forgotten when Apollo answers 404 or its legacy “this contact has been deleted” text (HTTP 422), or when it still exists but left the selected stages or lists. Any other answer, a 403 or 5xx included, keeps it. A sequence gone from the sequence list is forgotten at once.
Deselected kinds: leaving a kind out of include forgets its records on the next run.
Capped listings: Apollo search returns at most 50,000 records per listing. A bigger selection must be narrowed with stage or list filters; until then the run reports failed_cap and never detects deletions, because a capped listing is incomplete.
Back-off: a record whose deletion check keeps failing (no access, or an error Apollo does not call “deleted”) is left alone for ten cycles after five failed checks in a row, then tried again. It stays in memory the whole time and is counted in held_back.
Rate limits: a run never waits out a rate limit. When Apollo answers 429, fewer than 10 calls are left in any minute, hour, or day window of the endpoint it is about to call, or max_requests is spent, the run stops cleanly, keeps its progress, and resumes at the same page next time. A key rejected after rows were emitted ends the run the same way (failed_auth); one rejected before any row raises ApolloAuthError, and dlt drops that run’s state. The windows are read from every answer that carries them, error answers included, and kept per endpoint (/contacts/search and /contacts/{id} are two), so what another endpoint has left does not matter. An answer without them leaves the windows last seen for its endpoint; it is not confirmed that Apollo sends them on a 404 or 422, and if it does not, record reads that only get those answers are stopped by max_requests or a 429 alone.
Refused listings raise: an HTTP 403 on a search or lookup raises ApolloAccessError naming the endpoint instead of cutting the run, so the next run lists again and nothing is forgotten.
Redirects are errors: a 3xx answer raises ApolloAPIError and is never followed, because the API key travels in a header requests would send to the new host.
Record ids are checked before a deletion check: an id with anything but letters and digits is never requested; the check counts as failed and the record is kept.

Deletions only propagate from a foreground run — see the orphan-cleanup note above. A run costs a few lookup requests (stages, lists, the sequence list), one request per 100 records of each included kind, and one per record that stopped appearing; a large workspace takes several runs.After a run, the cognee_sync_stats dict on the returned resource holds counts only: scanned, skipped, deleted, failed, requests (API calls made; a retried call counts once), and held_back (records in back-off when the run started — not a failure and not part of failed), plus the reason a run or a record failed:

failed_rate_limit, failed_budget: the run stopped early and resumes next time.
failed_auth: Apollo rejected the key. After rows were emitted the run stops early and resumes next time; before any row it raises, and the next run starts the cycle over.
failed_cap: a listing hit the 50,000-record limit, so deletions were not detected.
failed_deletion_check: the check of one record failed (a 5xx, or an answer cognee cannot read).
failed_access: the key or plan cannot call an endpoint (HTTP 403). On a record read, that one record is kept; on a listing or lookup, the run raised.

A run that raises, or is cut short, also adds one to failed.

Privacy and `forget()`

dlt state stores the Apollo team id, each record’s id with a fingerprint, and account_names — each synced account’s title and domain, which contacts use to name their account (about 72 bytes per account). It stores no other names or text. dlt’s staging database for the dataset (dlt_database_<dataset>) holds the full rendered text of every record, and dlt keeps every completed load package, with the rendered text of each row it loaded, in its data directory (~/.dlt of the server process unless DLT_DATA_DIR moves it). Logs and errors carry counts, statuses, and endpoints; record ids are replaced by {id} and the key is never logged or stored.

Neither cognee.forget() nor a deletion in Apollo erases a prospect for good. forget() removes the documents, graph, and vectors, but not the staged copy of the rendered text or the dlt state, so the next sync under the same resource_name rebuilds every document — also after forget(dataset=...). Deleting a prospect in Apollo (or moving it out of the selected stages or lists) and syncing also removes its staged row, but dlt’s completed load packages still hold its text. To erase it from the server, delete the load packages under dlt’s data directory and clear the staging database yourself.

Limitations

Needs API access to the search endpoints (a paid plan or trial).
At most 50,000 records per kind and filter selection.
Emails, phone numbers, and social profiles are never ingested; attachments, notes, tasks, and calls are not read.
Only a 404 or Apollo’s legacy 422 “deleted” text counts as a deletion.

Asana connector

The Asana connector syncs the tasks of the Asana projects you select into cognee memory, incrementally, and forgets a task when it leaves those projects. asana_source() returns a dlt resource you hand directly to cognee.remember(), so it takes the document-mode path described above (its source tag is asana): each task and each selected project becomes a document that goes through chunking and LLM entity extraction. One source is one access token and one selection of projects. It never changes anything in Asana.

Installation

Nothing to install. The Asana connector is bundled with the Cognee SDK and needs no extra: its REST client uses requests, which dlt (a core dependency) already requires.

Setup

Create a personal access token in the Asana developer console, or use an OAuth access token. The token needs read access to the projects and their tasks, comments, subtasks, and events; a personal access token has the access of the user who created it. The connector never refreshes a token.
Collect the gids of the projects to sync (the number in a project’s URL). There is no “whole workspace” mode: a workspace-wide sync is a list of its project gids, so an empty selection can never mean everything.

The connector has not been tested against a live Asana workspace yet.

Usage

import cognee
from cognee.tasks.ingestion.connectors import asana_source

await cognee.remember(
    asana_source(
        resource_name="asana_website",
        access_token="<personal access token>",
        project_gids=["1201234567890123"],
    ),
    dataset_name="asana",
    primary_key="id",
    write_disposition="merge",   # REQUIRED — incremental upserts and deletes
    max_rows_per_table=0,        # 0 = no row cap; orphan cleanup must see the whole corpus
    self_improvement=False,      # skip the improve() pass after each sync
)

answer = await cognee.recall("Who is working on the launch post?", datasets=["asana"])

Run the same remember() call again whenever you want to sync.asana_source() accepts these keyword-only parameters:

Parameter	Default	Description
`resource_name`	— (required)	Names this connection: it names the staging table and keys the sync state. A non-blank string that does not start with `_dlt_`. Two connections in one dataset need two names, or each sync forgets the other’s tasks.
`project_gids`	— (required)	The projects to sync: a non-empty list or tuple of gid strings. It is sorted and deduplicated. A bare string, an empty list, or a gid with anything but letters and digits raises `ValueError`.
`access_token`	`None`	Asana personal access token or OAuth access token. Required unless you pass `service`. Never read from the environment.
`service`	`None`	A pre-built client, from `build_asana_service(access_token)`. A client of your own must accept `request(method, path, *, params, check_active)` and call `check_active` before each attempt it makes.
`max_requests`	`1500`	Budget of all API calls in one run: the listing, the task reads (three per task), and the checks of missing tasks. An int of at least `3`. A call the client retries counts once.
`check_active`	`None`	Optional callable run before every attempt of every request, retries included; raising from it stops the run.

The returned resource is preconfigured with primary_key="id", write_disposition="merge", and a _deleted hard-delete column.

What Is Ingested

Rows are flat {id, title, content, url, _deleted, cognee_node_set} documents whose ids are prefixed by kind:

Row	Id	Title	Content
Task	`task:<gid>`	task name	completed or not (and the day it was completed), assignee, due date, the selected projects it is in with its section, custom fields, description, subtasks, comments
Project	`project:<gid>`	project name	project description

Completed: yes
Completed on: 2026-10-14
Assignee: Ada
Due: 2026-10-20
Project: Website launch (section: In progress)
Priority: High

Draft, then review with marketing.

Subtasks:
- [x] Outline
- [ ] Screenshots
  Assignee: Grace

Comments:
- Grace: Looks good

A task is one document holding its comments (oldest first) and one level of subtasks. A task in two selected projects is one document that names both; projects outside the selection are never named. Nothing volatile is rendered (no modified_at, no time of day), so an unchanged task keeps its content hash and is not extracted again. Completed on: is the UTC day of Asana’s completed_at; reopening the task removes the line. People are names, not entities, and no email address is requested.Every row carries a node set: a task gets asana:<gid> for every selected project it is in, sorted, and a project row gets asana:<its own gid>. Scope a recall to one project with node_name:

await cognee.recall(
    "Who is working on the launch post?",
    datasets=["asana"],
    node_name=["asana:1201234567890123"],
)

Node sets use project gids, never names, so renaming a project changes no node set. The row ids, node set names, and rendered text are fixed: changing them would give every task a new document and one more extraction on your LLM key. A tombstone carries no node set.

How Sync and Forget-on-Delete Work

Finding changes: every run lists each selected project’s tasks with their modified_at and reads a task again when it is new or its modified_at changed. What modified_at does not show (a new or edited comment, a renamed or removed subtask, a renamed project or section) comes through the Events API, with one sync token per project. An expired sync token (HTTP 412) re-reads the whole project.
Forgetting: a task is forgotten only when a complete listing of the selected projects lacks it and a check of its memberships (GET /tasks/{gid}, one request) confirms it is gone: a 403 or 404, or no selected project left. It was deleted in Asana, moved out of the selection, or its project was deselected. A run that fails or is cut inside the listing forgets nothing.
Unreadable tasks are kept: a task that is still listed but answers 403, 404, 5xx, nothing, or a body of the wrong shape keeps its old document, is counted in failed, and is asked again on the next run (state["retry"]). It is never given up on.
Unresolved projects: a project that answers 403 or 404 is skipped for the run, the others sync, and its row and tasks are kept. It is asked again on every run and counted in unresolved. A deleted project or a mistyped gid looks the same, so its documents stay until you remove it from the selection.
Changing the selection: a project you add is read in full. A project you remove forgets its project row and the tasks that were only in it; tasks also in a kept project are read again so the removed project’s line leaves their text.
Runs that stop: a rate limit (HTTP 429), a spent max_requests, three tasks in a row answering 5xx or nothing, or a token rejected after rows were emitted ends the run cleanly. It never sleeps and does not read Retry-After. Progress is kept: tasks not reached go to state["pending"], and a listing cut partway resumes at the page it stopped at (state["listing"]), so a large selection finishes over several runs.
Errors: a token rejected before any row raises AsanaAuthError. Any other failing answer outside a task read raises AsanaAPIError with its status; a missing-task check that answers 403 or 404 forgets the task instead, and one that answers 5xx or nothing keeps it, counted in failed like a read that failed. An answer of the wrong shape in a listing, the events feed, a project read, or a missing-task check raises AsanaResponseError. A run whose budget is too small to make any progress raises AsanaSourceError, giving the numbers. dlt drops the state of a run that raises, so the next run starts from the last good state and forgets nothing.

After a run, the cognee_sync_stats dict on the returned resource holds counts only:

Key	Meaning
`scanned`	tasks listed
`skipped`	listed tasks that did not need reading
`deleted`	tombstones emitted
`failed`	tasks that could not be read, plus one for the request that stopped a stopped run or for a run that raises
`kept`	the retry queue the run committed (not part of `failed`)
`pending`	the pending queue the run committed (not part of `failed`)
`requests`	API calls made (a retried call counts once)
`unresolved`	selected projects Asana answered 403 or 404 for this run

A run that stopped or raised for a reason also carries exactly one of failed_rate_limit, failed_budget, failed_unavailable, or failed_auth.

Limitations

Budget: a listing costs about projects × (1 + event pages + ceil(tasks / 100)) requests, and a run reads about (max_requests - listing) / 3 tasks. A first sync of 3,000 tasks in one project takes about seven runs at the default budget. Deselecting a big project costs one check per task.
Lock hold time: a run holds cognee’s process-wide dlt staging lock while it walks, for at most max_requests requests plus the client’s retries (three attempts, backing off 1 s then 2 s). There is no wall-clock deadline.
Renaming a project or section changes the text of every task in it, so each is read and extracted again.
Not read: subtasks of subtasks, attachments, task history other than comments, tags, followers, and dependencies.
If the token loses access to every task but the listing still works, a run ends with no reason key; only failed and kept show it.

Privacy and `forget()`

The connector reads task names, descriptions, comments with their authors’ names, assignee names, due dates, section and project names, subtasks, custom field values, and project descriptions of the selected projects. That text goes to your configured LLM and embedding providers during cognify, like any document — including whatever people typed into comments and descriptions. Anyone with read access to the dataset can read all of it.The token lives in the client object only. Leading and trailing whitespace is stripped, and a token with whitespace, control, or non-ASCII characters inside it is refused without being echoed. Errors carry the HTTP status and an endpoint with ids replaced (/tasks/{gid}); neither the token nor any task id is logged. dlt state holds ids, timestamps, and page offsets, never text.

cognee.forget() removes the documents, graph, and vectors, but not dlt’s staged copy of the rendered text (dlt_database_<dataset>) or the sync state, so the next sync under the same dataset and resource_name rebuilds the documents. To keep a task out, delete it in Asana or take its project out of the selection, and sync. Even then, dlt’s completed load packages (under ~/.dlt of the server process unless DLT_DATA_DIR moves it) still hold its text.

Obsidian connector

The Obsidian connector syncs a local Obsidian vault (a folder of Markdown notes) into cognee memory, incrementally and with forget-on-delete. obsidian_source() returns a dlt resource you hand directly to cognee.remember(), so it takes the document-mode path described above (its source tag is obsidian): each note becomes a text document that goes through normal chunking and LLM graph extraction.Unlike cognee.remember("<vault folder>"), which stores each note once, an edited note replaces its old document and a deleted note is forgotten on the next sync: a note’s id is its path in the vault.

This connector reads your personal notes. Nothing is read until you construct a source and call remember(). During cognify the note text, frontmatter included, is sent to your configured LLM and embedding providers like any other document, so check that you may send your notes there, or use local models. Anyone with read access to the dataset can read the notes, so keep a vault in its own dataset, and use folders to leave out what should not be read.

Installation

Nothing to install. The connector is bundled with the Cognee SDK and needs no extra, no account, no token, and no network access: it reads files from a folder on the machine that runs it. It is SDK only: there is no hosted integration for it, because nothing hosted can reach a vault on your disk.

Usage

import cognee
from cognee.tasks.ingestion.connectors import obsidian_source

source = obsidian_source(vault_path="~/Documents/Vault")
await cognee.remember(
    source,
    dataset_name="my_vault",
    primary_key="id",
    write_disposition="merge",  # REQUIRED, the add pipeline defaults to "replace"
    max_rows_per_table=0,
    self_improvement=False,
)

# A normal return can still have skipped notes: read the counts.
assert source.cognee_sync_stats["failed"] == 0

Run the same call again, with a fresh source, whenever you want to sync.obsidian_source() accepts these keyword-only parameters:

Parameter	Default	Description
`vault_path`	— (required)	The vault folder, `~` expanded and resolved. A `ValueError` unless it is an existing directory.
`resource_name`	`obsidian_<12 hex of the sha256 of the resolved vault path>`	Names this vault: it keys the staging table, the sync state, and the cleanup of deleted notes. A `ValueError` if it is not a non-empty string.
`folders`	`None`	Vault-relative folders to read, written with `/`, subfolders included. `None` is the whole vault.
`check_active`	`None`	Optional callable run before each note is read and before each row is published; raising from it aborts the sync.

The returned resource is preconfigured with primary_key="id", write_disposition="merge", and a _deleted hard-delete column.folders is a non-empty list or tuple such as ["Projects/Work", "Journal"], sorted and deduplicated. A bare string, an empty list, an absolute path, a folder name starting with a dot, a backslash, or surrounding spaces is a ValueError when the source is built, because each would select nothing and the next run would forget every note.

The default resource_name follows the vault path. Moving or renaming the vault folder changes it, so the next sync ingests every note again under the new name, while documents synced under the old name stay until you forget them. Pass a resource_name to pin it. Two sources over the same vault with different folders share the default name, one table, and one state, so each run forgets the notes the other selected: put them in two datasets, or give each its own resource_name.

What Is Ingested

Only regular *.md files under the vault, or under the selected folders. Not read: anything whose name starts with a dot (.obsidian, .trash, .git, hidden files such as ._note.md), other file types (attachments such as images and PDFs are ignored; an embed of one stays as the text you wrote), symbolic links (a linked folder is not entered), FIFOs, and device files. A note becomes one row:

Field	Value
`id`	`obsidian:<path of the note in the vault, with />`
`title`	the frontmatter `title` when it is a non-empty string, else the file name without `.md`
`content`	the file text exactly as read, frontmatter included
`url`	empty
`cognee_node_set`	`obsidian:tag:<tag>` for each frontmatter tag, only when there are any
`cognee_structure`	the note’s folders, nearest first, only for a note inside a folder

Tags come from the frontmatter tags key, or the older tag, as a list or a single string (one tag, not split on commas). A leading # is dropped; non-string entries are skipped. Inline #tags in the body, aliases, and other properties are not read, and [[wikilinks]] stay text rather than becoming edges. Scope a recall to tagged notes with node_name=["obsidian:tag:work"]. Adding, removing, or renaming a tag re-ingests the note.
Frontmatter that cannot be parsed, is not a YAML mapping, or is not closed costs the note only its title and tags; it still syncs with its text unchanged.
Structure: each folder is {"kind": "folder", "id": "<folder path>", "name": "<folder name>"} (see Connector Modes above). After each cognify, each note gets a child_of edge to its folder and each folder to its parent; folders become StructureContainer nodes.

How Sync and Forget-on-Delete Work

No cursor: every run walks the whole selection and reads every note. A note whose text did not change makes the same row, cognee recognises its content hash, and nothing is extracted again, so a quiet sync costs file reads and no LLM calls. File times are never used.
Forgotten notes: a note an earlier run published that a complete listing no longer contains is emitted as a _deleted tombstone. That covers deleted notes, renamed or moved notes (the new path is a new document; renaming a folder does this to every note in it), and notes outside a narrowed folders selection. Deletions only propagate from a foreground run — see the orphan-cleanup note above.
State: dlt resource state holds the ids of the notes the last complete run published. It is written after the last row, so a run that raises keeps the old state.

Errors

The listing is complete before the first row is published, so a run aborts before anything is synced or forgotten when:

vault_path is no longer a directory when the run starts: ObsidianVaultUnavailableError;
the vault root has no entries at all (not even .obsidian) although an earlier run published notes — a drive or mount that did not come up looks exactly like this: ObsidianVaultUnavailableError. To forget everything on purpose, leave one file or folder in the vault;
a folder inside the vault cannot be listed: ObsidianListingError, naming the folder relative to the vault;
a selected folder is a file, or exists on disk only under another spelling (letter case, Unicode form, a symbolic link): ObsidianListingError.

A note that cannot be read or is not valid UTF-8 is neither published nor forgotten: it keeps what memory already holds, is logged with its relative path, and is counted in failed and failed_read.After a run, the cognee_sync_stats dict on the returned resource holds counts only: synced (notes read and published), deleted (tombstones), and failed / failed_read (unreadable notes; equal today). An empty dict means dlt resumed a pending load without extracting this resource: build a fresh source and run again.

Limitations

A vault with entries but no notes forgets everything it published before, and so does a folders selection whose folders are all gone from disk. A selected folder that does not exist forgets its notes without an error, because a folder deleted on purpose must forget them; once it has no notes left, every run logs a warning naming it until you remove it from folders.
The Unicode form (NFC or NFD) of a file name is part of the note id, so a sync tool that changes it re-ingests every note and forgets the old ones.
Only one active sync per vault and dataset is supported. A large vault is read in full on every run, one note at a time.

cognee.forget() does not erase a note for good. forget(data_id=...) or forget(dataset=...) removes the documents, graph, and vectors, but not dlt’s staged copy of the text (dlt_database_<dataset>) or the dlt state, so the next sync under the same dataset and resource_name rebuilds them. To erase a note, delete it in the vault (or move its folder out of folders) and sync. dlt also keeps completed load packages, with the text of every row, under ~/.dlt of the process.

OneNote connector

The OneNote connector syncs one selected OneNote notebook (its sections, nested section groups, and pages) into cognee memory through Microsoft Graph, incrementally, and forgets a page when Graph gives proof that it was deleted. onenote_source() returns a dlt resource you hand directly to cognee.remember(), so it takes the document-mode path described above (its source tag is onenote): each page becomes a text document that goes through normal chunking and LLM graph extraction.

This connector reads your personal notes. Nothing is fetched until you construct a source and call remember(). During cognify the page text is sent to your configured LLM and embedding providers, like any other document, and anyone with read access to the target dataset can read it, so keep the notebook in its own dataset.

Installation

The OneNote connector is bundled with the Cognee SDK. It uses httpx, which a plain cognee install already brings, and beautifulsoup4, which it does not: install it with pip install beautifulsoup4 (the cognee[scraping] extra also includes it). Without it, the first page the connector fetches fails with an ImportError, which remember() surfaces wrapped in a DLTIngestionError. It needs a core with DOCUMENT_SYNC_VERSION 3 or later and raises a RuntimeError on an older one.

Setup

Bring a delegated Microsoft Graph access token with the Notes.Read permission. OneNote’s API does not support app-only access. The connector never signs in and never refreshes a token: pass a token string, or a zero-argument callable that returns a valid token for the same account on every call. Only the public-cloud graph.microsoft.com endpoint is supported.

Usage

import cognee
from cognee.tasks.ingestion.connectors import list_onenote_notebooks, onenote_source

notebooks = list_onenote_notebooks(token)  # [{"id": ..., "displayName": ...}, ...]

source = onenote_source(
    token,
    notebook_id="<notebook id>",
    account_id="<stable account and tenant id, not the token>",
)
await cognee.remember(
    source,
    dataset_name="my_onenote",
    primary_key="id",
    write_disposition="merge",  # required; the add pipeline defaults to "replace"
    max_rows_per_table=0,
)

# A normal return is not proof of a finished sync: read the counts.
stats = source.cognee_sync_stats
assert stats["failed"] == 0 and stats["complete"] == 1

Run the same call again whenever you want to sync. One source is one notebook: to sync several notebooks, build one source per notebook id. list_onenote_notebooks(token, *, http_client=None) reads only notebook metadata, never page content. Any write_disposition other than merge stops the sync before any request: the connector raises a ValueError, which remember() surfaces wrapped in a DLTIngestionError.onenote_source() accepts these parameters:

Parameter	Default	Description
`token`	— (required)	Access token string, or a zero-argument callable returning one.
`notebook_id`	— (required, keyword-only)	The notebook to sync.
`account_id`	— (required, keyword-only)	Stable, non-secret id of the account and tenant. Never the token or a display name.
`resource_name`	`onenote_<24 hex of account_id and notebook_id>`	Names this connection: it keys the staging table, the sync state, and the cleanup of deleted pages. Normalized to dlt’s snake-case table name; must not be empty.
`http_client`, `service`	a client built for the run	Two names for one pre-built `httpx.Client` (tests, proxies). Give at most one.
`check_active`	`None`	Host checkpoint, called before every request and before every row is published. Raising from it aborts the sync.
`max_requests`	`300`	Budget of HTTP attempts in one run, retries included.

Every OneNote source that syncs into one dataset needs its own resource_name (the default already differs per account and notebook). A run refuses to continue when the stored state belongs to another account or notebook. Renaming a source starts it over, and documents synced under the old name are not cleaned up.

What Is Ingested

One document per page. Rows are {id, title, content, url, cognee_structure}: the Graph page id (the primary key), the page title (Untitled when empty), the page’s oneNoteWebUrl, and the page’s HTML body as plain text.
Images and attachments are not downloaded. Images become Image: <alt text> (<reference>) and attachments Attachment: <name> (<reference>) lines; links keep their target in parentheses. Handwriting and text inside images are not read.
Structure: each row carries cognee_structure (see Connector Modes above), nearest parent first: the section, its section groups from the innermost out, then the notebook, each with its Graph id and name. After each cognify these become child_of edges, Notebook ← Section ← Page, with the containers as StructureContainer nodes. Renaming a notebook, section, or section group, or moving a page to another section, keeps every page’s document id and runs no extraction again; only the child_of edges change.
No node set: rows carry no cognee_node_set. People named in a note are text, not typed entities.

How It Works

Fingerprints: Graph has no delta feed for OneNote, so every run lists the whole notebook (metadata only) and compares each page with what the last complete run published. A page is fetched and published only when it is new or its fingerprint changed: its lastModifiedDateTime, title, link, or the notebook, section groups, and section above it. Unchanged pages are neither fetched nor published again, so a quiet sync extracts nothing. Renaming or moving a container costs one HTML request per page below it, but no re-extraction.
All or nothing per notebook: documents and deletions are published only after the complete notebook was read and validated. An expected failure (a Graph error, a rate limit, a spent max_requests budget, malformed metadata, or an unreadable page) returns normally with failed > 0 and complete == 0, publishes nothing, and checkpoints progress, so the next run resumes without repeating it. A failure of check_active or of a token callable raises unchanged.
A malformed page blocks its notebook: a page whose id or lastModifiedDateTime is missing or blank, or whose title is missing or not a string, fails the run before anything is published. Fix or remove the page and the next run recovers.

Deletions

A page that was synced and is missing from a complete listing is read once more. It is forgotten only when:

Graph answers 404 or 410 with code 20113 (“the resource specified in the request was deleted”), or
the page still exists but now belongs to another notebook (a positive move).

Code 20102 is not proof of deletion. It also fits a page this token can no longer see, for example after a scope change. A page that answers 20102 keeps its row and its document and is counted in the unresolved stat; the rest of the notebook syncs normally, and the run still reports complete == 1 and failed == 0. Such pages are asked about again in later runs with whatever request budget is left over. A page stops being unresolved once it is listed again or Graph answers 20113 for it. A host that must notice these pages should watch unresolved, because nothing else in the stats changes.Deletions are emitted as _deleted tombstones and only propagate from a foreground run — see the orphan-cleanup note above. A notebook you stop syncing keeps its memory.

Which of the two codes Graph returns for a page deleted inside a notebook that still exists has not been checked against a live account; the connector has not been run against a live Microsoft account. If Graph answers 20102 for such a page, it stays in memory (counted in unresolved) until Graph answers 20113 for it. cognee.forget() only removes it until the next sync; see Privacy and forget() below.

Sync Stats

After a run, the cognee_sync_stats dict on the returned resource holds counts only: scanned (pages seen), skipped (pages unchanged since the last sync), deleted, failed, failed_rate_limit, failed_budget, unresolved, requests, and complete (1 only when the whole notebook was read and published). An empty dict means dlt resumed a pending load without extracting this resource: build a fresh source and run again.

Limits and Retries

Request budget: Microsoft Graph throttles OneNote at 120 requests per minute and 400 per hour per app and user. The connector makes one request at a time, and the default max_requests=300 keeps one run under the hourly allowance. It counts every attempt, retries included, and is not an hourly limiter, so several notebooks or other apps of the same user share the 400. A very large notebook can need several runs; if listing alone spends the budget, raise max_requests.
Retries: a 429, 500, 502, 503, or 504 is retried up to five attempts in total with a 1, 2, 4, 8 second fallback; a Retry-After header is honoured when present. The total sleep of one sync is capped at 60 seconds (RETRY_WAIT_BUDGET_SECONDS), because the walk holds cognee’s process-wide dlt staging lock; once it is spent, the run stops cleanly (failed_rate_limit for a 429) and resumes next time.
Redirects are never followed, and every URL, paging links included, must be under https://graph.microsoft.com/v1.0/me/onenote/.
Tokens: a 401 stops the run with failed > 0; refresh the token and run again. Surrounding whitespace is stripped from the token, and a token with whitespace inside is refused before any request. The token is never logged, stored, or put in an error; errors carry a status and a Graph code, never a response body.

Privacy and `forget()`

dlt state stores a hash of the account and notebook, page ids, and a fingerprint per page, plus the parsed text of pages fetched but not yet published by a cut-short run. dlt’s staging database for the dataset (dlt_database_<dataset>) holds the rendered text of every page, and dlt keeps completed load packages, with the rendered text of each row, under ~/.dlt of the process.

cognee.forget(data_id=...) or forget(dataset=...) removes the documents, graph, and vectors, but not the staged copy of the text or the dlt state, so the next sync under the same dataset and resource_name rebuilds everything. Deleting a page in OneNote removes it from the graph but not from dlt’s load packages.

Limitations

One notebook per source, and one active sync per account and dataset.
Paging uses $skip, so a page created while another is deleted mid-walk can be missed for one run; the next run picks it up.
Images, attachments, and handwriting are not read.
Not yet verified against a live Microsoft account.

Use Cases

CRM and Relational Data

Load customer, order, and product tables from a database. Foreign keys between tables (e.g., order.customer_id → customer.id) become graph edges, enabling cross-table queries like “Which customers ordered product X?”

CSV Analytics Pipeline

Point cognee at CSV exports from analytics tools. Each row becomes a searchable node in the graph, and you can combine them with unstructured reports in the same dataset.

Event Log Ingestion

Use write_disposition="append" to stream event batches into cognee without deduplication. Query across the full event history with natural language.

Database Mirroring

Use write_disposition="merge" to keep cognee’s graph in sync with a live database. Rows that are removed upstream are cleaned up best-effort; any orphaned rows that fail to delete are logged and retried on the next ingest.

Cloud Data Warehouses (Snowflake, Redshift, BigQuery)

Amazon Redshift speaks the PostgreSQL wire protocol, so the standard connection string auto-detection works:

await cognee.remember(
    "postgresql://user:pass@my-cluster.us-east-1.redshift.amazonaws.com:5439/mydb",
    dataset_name="redshift_data",
    primary_key="id",
)

Snowflake requires constructing a dlt sql_database source manually (install snowflake-sqlalchemy first):

pip install cognee snowflake-sqlalchemy

from dlt.sources.sql_database import sql_database
import cognee

source = sql_database(
    credentials="snowflake://user:password@account_identifier/database/schema?warehouse=MY_WH",
    table_names=["orders", "customers"],
    resolve_foreign_keys=True,
)

await cognee.remember(source, dataset_name="snowflake_data", primary_key="id")

The account_identifier is the part before .snowflakecomputing.com in your Snowflake URL (e.g. myorg-myaccount). Omit table_names to ingest all tables in the schema. Keep resolve_foreign_keys=True on any sql_database source you build yourself. Without it, dlt records no foreign keys and the rows get no foreign key edges (see Foreign Key Resolution).Google BigQuery works the same way using dlt’s BigQuery connector — construct the source and pass it directly to cognee.remember(). See the dlt sql_database docs for connector-specific setup.

Remember Operation

Learn more about data ingestion in cognee

dlt Documentation

Official dlt documentation and guides

​Why Use This Integration

​Installation

​Quick Start

​1. Ingest a dlt Resource

​2. Build and Query the Graph

​Other Input Modes

​CSV Files

​Delimiter detection

​Database Connection String

​Mixed Structured + Unstructured

​DLT Options

​Time Anchoring

​Rows in Hybrid and TEMPORAL Search

​Column Value Nodes

​Column Nodes

​Write Dispositions

​How It Works

​Bounding Ingestion

​Re-Ingesting a Source

​Renaming a Source

​Foreign Key Resolution

​Connectors

​Gmail Connector

​Installation

​One-Time OAuth Setup

​Usage

​How It Works

​Installation

​Usage

​What Is Ingested

​How It Works

​Tokens

​Installation

​One-Time OAuth Setup

​Usage

​How Sync and Forget-on-Delete Work

​What Is Read

​Limitations

​Installation

​Setup

​Usage

​What Is Ingested

​How It Works

​Errors

​Limitations

​Installation

​Setup

​Usage

​What Is Ingested

​How Sync and Forget-on-Delete Work

​Privacy and forget()

​Limitations

​Installation

​Setup

​Usage

​What Is Ingested

​How Sync and Forget-on-Delete Work

​Limitations

​Privacy and forget()

​Installation

​Usage

​What Is Ingested

​How Sync and Forget-on-Delete Work

​Errors

​Limitations

​Installation

​Setup

​Usage

​What Is Ingested

​How It Works

​Deletions

​Sync Stats

​Limits and Retries

​Privacy and forget()

​Limitations

​Use Cases

Remember Operation

dlt Documentation

Why Use This Integration

Installation

Quick Start

1. Ingest a dlt Resource

2. Build and Query the Graph

Other Input Modes

CSV Files

Delimiter detection

Database Connection String

Mixed Structured + Unstructured

DLT Options

Time Anchoring

Rows in Hybrid and TEMPORAL Search

Column Value Nodes

Column Nodes

Write Dispositions

How It Works

Bounding Ingestion

Re-Ingesting a Source

Renaming a Source

Foreign Key Resolution

Connectors

Gmail Connector

Installation

One-Time OAuth Setup

Usage

How It Works

Installation

Usage

What Is Ingested

How It Works

Tokens

Installation

One-Time OAuth Setup

Usage

How Sync and Forget-on-Delete Work

What Is Read

Limitations

Installation

Setup

Usage

What Is Ingested

How It Works

Errors

Limitations

Installation

Setup

Usage

What Is Ingested

How Sync and Forget-on-Delete Work

Privacy and `forget()`

Limitations

Installation

Setup

Usage

What Is Ingested

How Sync and Forget-on-Delete Work

Limitations

Privacy and `forget()`

Installation

Usage

What Is Ingested

How Sync and Forget-on-Delete Work

Errors

Limitations

Installation

Setup

Usage

What Is Ingested

How It Works

Deletions

Sync Stats

Limits and Retries

Privacy and `forget()`

Limitations

Use Cases