What pipelines are
Pipelines coordinate ordered Tasks into a reproducible workflow. Default Cognee operations like Remember run on top of the same execution layer. You typically do not call low-level functions directly; you trigger pipelines through the higher-level operations unless you need staged control.Prerequisites
- Dataset: a container (name or UUID) where your data is stored and processed. Every document remembered by Cognee belongs to a dataset.
- User: the identity for ownership and access control. A default user is created and used if none is provided.
- More details are available below
How pipelines run
Somewhat unsurprisingly, the function used to run pipelines is calledrun_pipeline.
Cognee uses a layered execution model: a single call to run_pipeline orchestrates multi-dataset processing by running per-file pipelines through the sequence of tasks.
- Statuses are yielded as the pipeline runs and written to databases where appropriate
- User access to datasets and files is carefully verified at each layer
- Pipeline run information includes dataset IDs, completion status, and error handling
- Background execution uses queues to manage status updates and avoid database conflicts
run_pipeline is an async generator: iterate it to drive the runs. For each dataset it yields PipelineRunStarted, then PipelineRunCompleted or PipelineRunErrored. With incremental_loading or data_cache set, a data item already completed for pipeline_name in that dataset is skipped and recorded as PipelineRunAlreadyCompleted in the run’s data_ingestion_info.
The first-use LLM connection probe runs only when the run needs an LLM; embeddings are always probed. For a task list, that is derived from the tasks (any Task with needs_llm=True); the needs_llm argument applies only when tasks is a per-item task resolver.
Pipeline Names and Re-runs
Pipeline Names and Re-runs
Every
run_pipeline call takes a pipeline_name parameter (default: "custom_pipeline"). The name keys the run records a pipeline writes and the per-document status it stamps on each data item, so it decides which earlier work a new run can skip.Reserved pipeline names
Three pipeline names are used internally and carry special meaning:How re-runs avoid duplicate work
A pipeline never refuses to start because of a dataset’s earlier run history: every call starts a new dataset-level run, whatever the newestpipeline_runs row for that dataset says. Two mechanisms keep that from doing work twice:- Concurrent runs on the same dataset are serialized by the per-dataset lock (see “Per-dataset serialization” below), so a second run waits for the first instead of racing it.
- Already-processed documents are skipped one by one by incremental loading (
incremental_loading=True, the default foradd()andcognify();run_pipelineandrun_custom_pipeline()default toFalse;data_cache=Trueturns it on as well), which reads the per-document status stamped on each data record forpipeline_name— not the dataset’s run history. After youadd()new data and callcognify()again, documents already processed for that dataset are reported asPipelineRunAlreadyCompletedand only the new ones are processed.forget()clears that per-document status for the data it removes.
The
use_pipeline_cache parameter has been removed from run_pipeline and run_custom_pipeline(), along with the dataset-level check it enabled and the reset_dataset_pipeline_run_status helper that unblocked it. Passing use_pipeline_cache now raises a TypeError; drop the argument. It defaulted to False and the built-in operations never enabled it, so no working configuration changes behavior. add() and forget() also no longer write a DATASET_PROCESSING_INITIATED row: the value stays in PipelineRunStatus only because rows in existing databases still carry it.Per-dataset serialization
Per-dataset serialization
Pipeline runs are serialized per dataset. Before a run starts,
run_pipeline_per_dataset acquires a lock keyed on the dataset ID, so two runs that target the same dataset execute one after another — the second waits until the first finishes — while runs on different datasets still proceed in parallel. This protects each dataset from concurrent writers (for example, two cognify() calls on the same dataset) without globally serializing all pipeline activity.Delete operations share the same per-dataset lock. Deleting a dataset or a single data item (including the memory-clearing forget() paths) acquires the lock keyed on that dataset ID before it mutates anything, so a delete waits for an in-flight add(), cognify(), or memify() run on the same dataset to finish — and a pipeline run started while a delete is in progress waits for the delete. Two deletes targeting the same dataset serialize the same way. Deletes on different datasets still proceed in parallel.improve() holds the same per-dataset lock for its whole run, so cognify(), add() and forget() on that dataset wait for a running improve, including its LLM stages, and an improve waits for them. The pipelines its stages start re-enter the lock the improve holds (see “Nested (re-entrant) runs” below).Data selections
When you pass cogneeData rows as data to run_pipeline or run_custom_pipeline(), the run reads those rows again from the relational database once it holds the dataset’s lock, because a row may have been updated or deleted while the run was waiting:- Tasks receive the rows’ current values, not the instances you passed in.
- Rows deleted in the meantime are dropped. A selection that ends up empty runs over no data, never over the whole dataset.
- A row that belongs to another dataset raises
CogneeValidationError(after the lock is acquired). Pass only rows of the dataset the pipeline runs on. - Items that are not
Datarows pass through unchanged.
data (or with an empty list), the run still processes every row of the dataset.Nested (re-entrant) runs
A pipeline task may legitimately start another pipeline on the same dataset — for example, a session-driven run callingadd() or cognify() on the dataset it is already processing. Because the per-dataset lock is not re-entrant, re-acquiring it from within the same execution would self-deadlock. Cognee detects that the current execution already holds the dataset’s lock and lets the nested run proceed without re-locking; external runs (and deletes) on that dataset stay queued behind the lock the ancestor run holds. The same re-entrancy applies to deletes, since they acquire the lock from the same registry.Custom pipeline naming
Custom pipeline naming
For your own pipelines, choose a unique A custom name has one trade-off for pipelines that write graph data:
pipeline_name that does not conflict with cognify_pipeline, indexing_pipeline or add_pipeline. Using a unique name means:- State tracking is isolated to your pipeline — documents completed by the built-in
cognify()are not skipped by your pipeline, and vice versa.
improve() counts only the names in cognee.modules.improve.graph_changes.WRITE_PIPELINE_NAMES as graph changes. The default custom_pipeline is on that list; a name like my_enrichment_pipeline is not, so its graph writes are invisible to improve()’s enrichment gate and the dataset can be reported as already enriched. Keep the default name for pipelines that write to the graph.Crash recovery and stuck pipelines
Crash recovery and stuck pipelines
How an interrupted run ends depends on how it was interrupted:On the cancellation path the run is now finalized like any other failed run: the rollback handler runs first, then the terminal
- A hard kill —
SIGKILL, a container OOM, power loss — stops the process before any cleanup code can run, so the pipeline run record in the relational database is left with aDATASET_PROCESSING_STARTEDstatus. - A cancellation — a graceful server shutdown or restart, or any other
asyncio.CancelledErrordelivered to the task driving the run — is cooperative, so the run’s terminal handler gets to run. The run is finalized asDATASET_PROCESSING_ERROREDrather than being left in progress, and the cancellation is then re-raised so it still propagates to the caller exactly as before.
Marking cancelled runs as errored is new. Previously the terminal handler caught only
Exception, and asyncio.CancelledError is a BaseException, so a cancelled run never reached the error-logging step and its record stayed at DATASET_PROCESSING_STARTED indefinitely.pipeline_runs row is written with outcome FAILED, error_class CancelledError, and a scrubbed error_message, and a PipelineRunErrored event is yielded before the CancelledError is re-raised. Behavior for ordinary exceptions is unchanged.No pipeline consults the dataset-level status before it starts — the next call starts a new dataset-level run, serialized by the per-dataset lock. A stuck DATASET_PROCESSING_STARTED record therefore never blocks a re-run, although completed data items can still be skipped by their per-document pipeline status.Documents that were mid-processing when the run died have no completed mark, so the retry processes them again from the start — re-chunking, re-extracting and re-embedding them.
Automatic recovery
Two automatic layers clean up failed or abandoned runs, so a re-run can resume instead of redoing or half-skipping work:- Rollback on error — when a cognify run fails (including when it is cancelled), an error handler rolls back the run per item. Documents the failed run finished — stamped complete for the dataset and reported
PipelineRunCompletedby that run — keep their graph, vector, and relational artifacts, their completedcognify_pipelinestatus, and their edge evidence. Only the work of the items the run never finished is removed, and theircognify_pipelinestatus is cleared. Data items completed in earlier successful runs keep their status too. The next run therefore re-extracts only the failed or unfinished documents. A non-incremental run (incremental_loading=False, data_cache=False) never stamps items, so an item it completed is kept only when an earlier run had already stamped it; otherwise it is rolled back with the unfinished ones. Theindexing_pipelinerun of a cognify uses the same handler. Artifacts are selected by run id, so rolling back a failed cognify run leaves the chunks the indexing run stored, and the dataset stays searchable. One exception: withSTORAGE_BACKEND=s3and an embedded graph or relational database (Ladybug, SQLite), a run that fails before its writes are pushed to S3 is rolled back as a whole, stamps included, because the next process would otherwise skip documents S3 never received. - Recovery for abandoned runs — when the API server starts, and every 5 minutes after that, it closes every pipeline run that a process that has since died left at
DATASET_PROCESSING_STARTED— for every pipeline, not only cognify: a run ofadd,memify, the code graph, a migration, or a custom pipeline is closed the same way. Candidates are found perpipeline_run_id, so a run abandoned while a newer run of the same pipeline on the same dataset later completed is still found; a run whose newest row is alreadyDATASET_PROCESSING_ERROREDorDATASET_PROCESSING_COMPLETEDis left untouched. Runs whose newest row is younger thanCOGNEE_STALE_RUN_RECOVERY_MIN_AGE_SECONDS(env var, default3600seconds) are left alone because they may still be executing in another live worker or replica. Past that age, a run is closed only if the server can take the run’s ownership lock, the OS file lock at<system_root>/pipeline_run_locks/<token>/owner.lock, where<token>is therecovery_lockvalue on the run’sSTARTEDrow: a run whose writer still holds the lock, a run with no marker (for example one started before markers existed), and a run whose marker is not on this server’s filesystem are left atSTARTED, and a run whose dataset is locked by a run in this process is skipped until a later check. Each candidate is first rolled back with its pipeline’s own rollback handler —cognify_pipelineandindexing_pipelineare the only pipelines that register one; the others are closed without a rollback, exactly as when they fail inline — and then closed with aDATASET_PROCESSING_ERROREDrow whoseerror_classisAbandonedPipelineRunError. If the rollback raises, no row is written and the run staysSTARTED, so the next check retries it. The check runs when the API server starts and then every 5 minutes while it is up (a fixed interval with no setting); library-only usage does not trigger it. Because it keys onDATASET_PROCESSING_STARTED, it covers hard kills; a cancelled run already reaches the terminalDATASET_PROCESSING_ERROREDstatus on its own, so there is nothing for the reaper to pick up and no wait for the minimum-age threshold.
DATASET_PROCESSING_ERRORED row that closes an abandoned run copies the STARTED row’s user, tenant, start time, origin, and parent operation, so it describes the run that died rather than the process that closed it. The run therefore stays visible in the activity feed as abandoned instead of disappearing.A dataset’s status is the newest row of the pipeline on it, so the closing row must not overtake a later run. When another run of the same pipeline has since written rows on the dataset, the closing row is dated one microsecond after the abandoned run’s own newest row (its ended_at is still the time of closing). It closes the abandoned run, and the dataset keeps the later run’s status. Previously the row was dated at the time of closing, so after recovery a dataset whose later run had completed read as failed. When no later run exists, the row is dated now, as before. In the rare case that the two runs’ newest rows are at most a microsecond apart, there is no room beneath the later run: recovery logs Recovery failed for run …; will retry, the run stays DATASET_PROCESSING_STARTED (its rollback has already run), and each later recovery pass rolls it back and fails the same way until another run of the pipeline writes a newer row on the dataset.Recovery rolls back only the unfinished part of an abandoned cognify run. A document is marked complete for a dataset the moment its own tasks finish, mid-run, and a later run skips documents so marked — so by the time recovery finds the abandoned run, a newer run may already have trusted those documents and skipped them. Recovery therefore keeps every document the abandoned run had marked complete: its artifacts and its completed cognify_pipeline status stay, and only the run’s unfinished work is removed (an artifact is deleted only when no kept document still references it). Previously recovery undid the whole run, which could delete completed documents that nothing re-processed until the next cognify(). The inline rollback on error path now follows the same per-item rule. Recovery has no per-item results from the dead run, so there the completed stamp alone decides what is kept.By default a per-document processing error aborts the whole run; set the RAISE_INCREMENTAL_LOADING_ERRORS env var to false to log the error and continue with the remaining data items instead.What gets stored in a pipeline run record
What gets stored in a pipeline run record
Pipeline runs are persisted in the relational
pipeline_runs table. Alongside the status, IDs, and pipeline name, the record keeps a run_info JSON column. It does not store the run’s input: no text passed to add() and no Data IDs. On a pipeline run’s rows it holds only these keys:recovery_lockon theDATASET_PROCESSING_STARTEDrow, when the run holds an ownership marker for abandoned-run recovery.progresson theDATASET_PROCESSING_STARTEDrow, described below.erroron theDATASET_PROCESSING_ERROREDrow, holding the scrubbed error message.
improve()’s per-stage stamps.The DATASET_PROCESSING_COMPLETED row stores an empty run_info. To find which documents a cognify run processed, use the graph’s per-run source refs or the provenance_edge_evidence table (pipeline_run_id, data_id).Earlier releases stored a data key here: the first 512 characters of text passed to add(), which no forget() mode removed, or the ID of every document in the dataset for each cognify run. Alembic revision b7e4c2a9d6f1 removes data from rows already stored. It keeps every other key and deletes no rows. Running it twice is safe. Its downgrade is a no-op, so the removed values cannot be restored. The disk space comes back only after the database’s own VACUUM.The progress key is read back — by GET /api/v1/datasets/status/progress. It is an in-flight snapshot of {completed_items, total_items, current_stage} written as items finish, and it is an exception to the append-only rule below (integration sync rows, described below, are the other): a progress tick updates the run’s existing DATASET_PROCESSING_STARTED row in place rather than inserting. Inserting per tick would grow pipeline_runs without bound as batches accumulate, so progress is stored as metadata inside the started state — PipelineRunStatus gains no new member for it. Three consequences worth knowing:- Ticks are throttled to roughly 20 database writes per run (
max(1, total_items // 20), with the first and last item always persisted), so the snapshot is coarse by design — that keeps write pressure off the default SQLite backend, which serializes writers behind a file lock. - Concurrent ticks for the same run race on a read-modify-write with no locking: last write wins. For a display-only signal that only risks a slightly stale snapshot between ticks.
- A late tick can never make a finished run read as running again. Status readers pick the newest row, so a tick that lands after the terminal row merely updates the older
STARTEDrow in place, invisibly; and the defensive path for aSTARTEDrow that has gone missing drops the tick outright rather than inserting a newSTARTEDrow with a latercreated_atthan the terminal one.
- A pipeline run writes several rows that share one
pipeline_run_id(started → terminal). Only the terminal row carries the run’soutcome,tokens_in/tokens_outandembedding_tokens. - Non-pipeline operations (
search,recall,remember,forget,delete,prune, integration syncs) write exactly one row each, withstatus,pipeline_nameandpipeline_idleftNULL— which is what keeps them invisible to readers that look up a dataset’s latest pipeline status.
GET /api/v1/activity/pipeline-runs):- Deduplicate by
pipeline_run_idbefore summing, or a single run is counted once per row it wrote. parent_operation_idlinks a child operation to its parent’spipeline_run_id, forming a tree — but token counts already chain up into the parent. Sum one level only; summing across levels double-counts.
(created_at, id) index matching the newest-first, id-tiebroken order that the activity feed and the time-bucketed usage queries read it in. Existing databases pick it up from the usual Alembic upgrade, so there is no manual step unless you run with ENABLE_AUTO_MIGRATIONS=false — then apply it with cognee-cli upgrade as you would any other revision.pipeline_runs as the record of every operation
pipeline_runs as the record of every operation
pipeline_runs is no longer pipeline-only: it is Cognee’s local activity record — an audit table of every operation run on the deployment, kept in the deployment’s own relational database (SQLite or Postgres, whichever is configured). The rows never leave your instance, are readable by whoever can query that database, and are unrelated to Cognee’s anonymous product telemetry.Operations that never run a pipeline — search(), recall(), remember(), improve(), forget(), datasets.delete_data(), and both prune paths — each write exactly one self-contained row when they finish, so a single indexed SQL query answers “what did this user run, did it work, how long did it take, and what did it cost in tokens” without parsing JSON.Integration syncs are the exception to writing on finish. Each sync writes one integration_sync_progress row when it starts and updates it in place (run_info.sync_progress, ended_at, outcome) until it ends, so a running sync shows up with ended_at and outcome still NULL. Any ending other than succeeded, partial included, records outcome: "failed". The sync’s own state is in run_info.sync_progress.state, which GET /api/v1/integrations/{provider}/sync/progress reads.These operation rows are written with status, pipeline_name, and pipeline_id left NULL. That is deliberate: every status reader filters on pipeline_name/status, so operation rows are invisible to the caching, crash-recovery, and status-reporting logic described above. Each row still gets its own pipeline_run_id — the operation’s id, which children reference as parent_operation_id. The PipelineRunStatus enum is unchanged.Operation-record columns
All of these columns are nullable, and each writer fills only the subset it knows. Operation rows and the terminal (completed/errored) row of a pipeline run carry the full set; a run’s started row carriesuser_id, tenant_id, operation_name, started_at, origin and its sessions.Recording is fail-open
The recorder can never break the operation it records. The wrapped operation’s exceptions always propagate unchanged, and a failure to persist the row is logged and swallowed.How it is logged depends on why the write failed. Asqlalchemy.exc.OperationalError — the relational store not answering — is treated as an expected state: the row is skipped and one DEBUG line is logged with no traceback, record_operation: skipping <operation> record, relational store unavailable (...). Two ordinary situations produce it: nothing exists before setup(), and prune_system(metadata=True) drops the relational database including pipeline_runs, so its own record is self-erasing by design. Every other persistence failure still logs a WARNING with a traceback (record_operation: failed to persist <operation> record).The classification is by exception type, not by whether the failure was expected, so a genuine outage that surfaces as an OperationalError — a refused Postgres connection, a connection timeout, a SQLite lock timeout — is skipped just as quietly. A ledger that has stopped recording therefore shows up as missing rows rather than as warnings; set LOG_LEVEL=DEBUG to see the skipped writes. An operation that only launches background work writes its row when that work finishes, and that write is logged exactly the same way.Cost and accuracy
Recording is a singleINSERT off the hot path — roughly 2.5 ms per operation, with no additional LLM or network calls. The recorded LLM token counts (tokens_in / tokens_out) are the provider-billed response.usage figures when the provider reports usage (falling back to a character-based estimate when it does not), so they include hidden reasoning tokens; embedding_tokens is the embedding provider’s reported usage, falling back to the embedding tokenizer’s count; usage you read out of this table may therefore be higher than counts derived from visible output alone. Nothing about billing changes — only the accuracy of what Cognee reports.Upgrading
The columns are added by a single idempotent Alembic migration (a7f3c9e1b5d2); embedding_tokens was added later by d4a8e2f6b1c9, applied the same way. Old rows keep NULL in the new columns — there is no backfill, and existing readers are unaffected. The API server applies migrations at startup; for standalone scripts, CI, or self-managed database lifecycles, run run_migrations after upgrading the cognee package.PipelineContext and ctx injection
PipelineContext and ctx injection
PipelineContext is the runtime context object that Cognee automatically builds and injects into any task that declares a ctx parameter. It carries the user, dataset, and per-item context for the current pipeline run, and provides an extras dict for custom state.The framework inspects each task function’s signature. If it finds a parameter named
ctx, it passes the current PipelineContext when the task runs. Matching is by parameter name, not by type annotation.Tasks that do not declare ctx simply receive no context and are unaffected.Using extras for custom pipeline state
Using extras for custom pipeline state
Pass a dict as the
extras argument to run_pipeline or run_tasks. Every task in the pipeline can read those values from ctx.extras.ctx.extras is always a plain dict, never None.Accessing user and dataset in a task
Accessing user and dataset in a task
The The built-in
user and dataset fields are most useful when you need to write provenance records or apply per-tenant logic:add_data_points task already does this automatically, so you typically only need to read these fields when writing your own storage tasks.The built-in ingest_data task reads ctx.dataset too, but conditionally, and the condition matters if you compose it into your own pipeline. run_pipeline resolves and write-checks the run’s dataset once, so ingest_data reuses ctx.dataset instead of re-resolving it per item — but only when that dataset demonstrably is the one the call targets: matching the dataset_id argument, or matching the dataset_name argument for a dataset the calling user owns in the same tenant. Point ingest_data at any other dataset and it ignores ctx.dataset, resolving the dataset itself and running the usual write-permission check, so the reuse is a saved lookup and never a widened permission.Making ctx optional
Making ctx optional
Always default
ctx to None so the task can also be called directly in tests or scripts without a running pipeline:Error handling and exception propagation
Error handling and exception propagation
When a task raises while processing a data item, the pipeline logs the error, yields a What a propagated Cognee exception looks like. Cognee’s own exception types derive from After-run hook failures. The
PipelineRunErrored status, and then re-raises the original exception to the caller. A failing data item therefore both surfaces a PipelineRunErrored event and propagates the underlying exception out of the pipeline run, rather than being silently collapsed into an error status only.Because of this, wrap pipeline runs in try/except when you iterate them, so you can react to the propagated exception:CogneeApiError, which carries three attributes you can read in the handler: message, name, and status_code, plus an optional fourth, remediation, set on the errors that know their own fix. Its __str__ formats them as "{name}: {message} (Status code: {status_code})", followed by " Fix: {remediation}" when a remediation is set (see the remediation hint) — for example, passing a list containing something other than Task instances to a custom pipeline raises "WrongTaskTypeError: tasks argument must be a list of Task class instances, got str in the list. (Status code: 400)". The base constructor also populates Exception.args with (message, name), so repr() and raise ... from error chaining behave the way they do for any standard Python exception:after_run_completed hook is awaited after each dataset’s run completes, and after_items_ran also when items in the run failed. If either fails on a dataset whose run completed, run_pipeline still runs the remaining datasets, then raises one AfterRunHookError naming every dataset whose hook failed. On a run that failed, the run’s own error wins: when the failure is only reported as PipelineRunErrored, the hook’s error is logged, and when the run raises, its error is re-raised with the hook’s error as its cause.Not every propagated error is a CogneeApiError, though — plain built-in exceptions and errors from underlying database or LLM libraries surface too, so keep the broad except Exception above as the safety net.CogneeApiError.__init__ also logs the exception centrally, at ERROR by default. Some exceptions that represent expected control flow rather than failures — such as an adapter reporting an unsupported capability — opt out with log=False, so a missing ERROR log line for one of those does not mean the exception was swallowed. It still propagates to your except block exactly as above.Layered execution
Layered execution
- Innermost layer: individual task execution with telemetry and recursive task running in batches
- Middle layer: per-dataset pipeline management and task orchestration
- Outermost layer: multi-dataset orchestration and overall pipeline execution
- Execution modes: blocking (wait for completion) or background (return immediately with “started” status)
- With no
datasetspassed, in either mode, the run resolves to every dataset the run’s user has write access to — the user supplied, or the default user when none is given - Datasets run one after another, one run per dataset, each under its dataset lock
Customization approaches and tips
Customization approaches and tips
- Use Remember for the default ingestion path
- Modify transformation steps without touching low-level functions, avoid going below
run_pipeline - Custom tasks let you extend or replace default behavior
Users
Users
- Identity: represents who owns and acts on data. If omitted, a default user is used
- Ownership: every ingested item is tied to a user; content is deduplicated per owner
- Permissions: enforced per dataset (read/write/delete/share) during processing and API access
Datasets
Datasets
- Container: a named or UUID-scoped collection of related data and derived knowledge
- Scoping:
remember()writes into a specific dataset, and dataset-scoped pipelines process the dataset(s) you pass - Lifecycle: new names create datasets and grant the calling user permissions; UUIDs let you target existing datasets (given permission)
- Resolution in
run_pipeline: write permission is required on every dataset; a name the user has no dataset for creates that dataset; a UUID the user has no write permission on, including an unknown one, raisesPermissionDeniedError; no dataset to run on at all raisesDatasetNotFoundError
Tasks
Learn about the individual processing units that make up pipelines
DataPoints
Understand the structured outputs that pipelines produce
Main Operations
See how pipelines are used in Remember and lower-level ingestion workflows