What are Pipelines
A pipeline runs tasks in order over your data.remember and recall are built on three of them: add, cognify and search. improve runs its own stages, and forget wraps the delete service.
How Pipelines work in Rust
A Pipeline is an orchestrated sequence of Tasks with shared context (database, graph, vector, cancellation, progress) and a watcher for status events. The concrete pipelines are:
Pipeline orchestration primitives (
PipelineWatcher, ExecStatusManager,
thread pool) live in cognee-core. The end-to-end flow is
described in operations.
What a cognify run does when it fails, how a killed run is recovered, and how re-runs skip finished items are covered in Cognify and Serialization and recovery of pipeline runs.
How pipelines run
You usually trigger pipelines through the operations (remember, add, cognify) rather than calling the executor yourself. To run your own, build a Pipeline from tasks, a TaskContext with the stores and the run’s PipelineContext, and pass both to execute (crates/core/src/pipeline.rs):
execute returns the outputs of the last task, or an ExecutionError. One call is one run:
- Ids. The run gets a random run id, which is its
pipeline_run_id. Itspipeline_idis deterministic:uuid5over the user id, the pipeline name and the dataset id from thePipelineContext, orPipeline::idwhen the pipeline has no name. - Status events. The watcher receives
on_pipeline_run_startedbefore the first task,on_task_started/on_task_completedaround each single-value task, and thenon_pipeline_run_completedoron_pipeline_run_errored. The compacton_pipeline/on_taskmethods receive the same lifecycle asPipelineStatus/TaskStatusvalues, includingItemSkippedfor an item skipped as already done.NoopWatcherignores everything;DbPipelineWatcher(featurepipeline-run-registry) writes the run’spipeline_runsrows. - Dataset scope. When the
PipelineContextnames a dataset, the whole run executes in that dataset’s scope, so under per-dataset storage every task reaches that dataset’s own graph and vector databases. - Other entry points.
execute_in_backgroundtakes owned (Arc) arguments, spawns the run on the current Tokio runtime and returns aPipelineRunHandleyou canwait()on orabort().execute_blockingruns it on a new single-threaded runtime for non-async callers, and panics when called inside a Tokio runtime. Both returnPipelineRunResult, whoserun_idisPipeline::id, not the run’spipeline_run_id.
ExecutionError::NoTasks or InvalidConfig before any event is sent.
Pipeline Names and Re-runs
Pipeline Names and Re-runs
A pipeline’s name (
Pipeline::with_name) keys its run records and its per-item status, so it decides which earlier work a new run can skip.Reserved pipeline names
How re-runs avoid duplicate work
No pipeline is refused because of the dataset’s earlier runs: every call starts a new run. Work is skipped per item instead.- In the executor. Set
Pipeline::with_data_idto a function that returns a stable id for an input, and give theTaskContextanExecStatusManager. Before the first task the executor asksis_completed(data_id, pipeline_name, dataset_id); a completed item is skipped and reported asPipelineStatus::ItemSkipped. After the chain finishes it callsmark_completed, ormark_failedwhen a task failed. The defaultNoopExecStatusManagerreports nothing as completed, so every item runs. - In cognify.
cognify()checks its completion markers before it builds the pipeline, so finished items are never classified, chunked or sent to an LLM. See Re-cognifying is incremental. - In add.
AddPipeline’s data-id function returnsNone, so the executor never skips an add input.
Per-dataset serialization
Per-dataset serialization
execute itself takes no lock. The operations do: cognify() holds the dataset’s lock for its whole run, and add, dataset-scoped deletes and improve take the same in-process registry (cognee_core::DatasetLocks::global()), so writers to one dataset queue and different datasets run in parallel. The lock does not span processes. Details are in Serialization and recovery of pipeline runs and Improve: Locking.A custom pipeline that writes to a dataset should take the same lock:Nested (re-entrant) runs
hold marks the dataset as held for the current Tokio task, so a nested hold or lock of the same dataset inside it returns at once instead of waiting on itself. That is how an improve stage can run cognify() on the dataset the improve already holds. A guard taken with plain lock does not mark the dataset held, and work moved to another task with tokio::spawn does not inherit the mark.Custom pipeline naming
Custom pipeline naming
Give your pipelines a unique name. Status tracking through an When your watcher persists rows, every name except
ExecStatusManager is keyed by (data_id, pipeline_name, dataset_id), so a unique name keeps your skipped items separate from the built-in ones, and your run rows separate from the dataset’s add and cognify status.search, recall and improve_pipeline counts as a write in improve’s change check, so after a run of your pipeline the next improve runs its global context index stage again instead of reporting it already completed.Crash recovery and stuck pipelines
Crash recovery and stuck pipelines
How a run ends depends on how it was stopped:
- A task error or a cancellation (
CancellationHandle::cancel, checked before each task) ends the run in-process. The watcher receiveson_pipeline_run_errored, so a persisting watcher writes theERROREDrow. - A killed process, a dropped future or
PipelineRunHandle::abortstops the run without running any more of its code, so its newest row staysSTARTED.
STARTED row never blocks a re-run. Startup recovery closes the cognify and add runs a dead process left STARTED, rolling a cognify run back first; it recognises them by the ownership token in their STARTED row. A custom pipeline without Pipeline::with_recovery_lock writes {} there, and recovery leaves its run alone. See Serialization and recovery of pipeline runs and When a cognify run fails.What gets stored in a pipeline run record
What gets stored in a pipeline run record
A persisting watcher (
DbPipelineWatcher) writes the relational pipeline_runs table (crates/database/src/entities/pipeline_run.rs). Each lifecycle event inserts a new row, so one run leaves a DATASET_PROCESSING_STARTED row and then a DATASET_PROCESSING_COMPLETED or DATASET_PROCESSING_ERRORED row, sharing one pipeline_run_id. Deduplicate by pipeline_run_id when you count runs.The table does not store the run’s input. The status strings and
run_info keys are the ones Python cognee writes.Fields a task publishes with TaskContext::publish_payload_field go to a separate pipeline_run_payload_fields table, one row per run and key; see Custom pipeline state.The table also holds rows that record a change rather than a pipeline run: a delete writes a forget row on each dataset it touched, and an improve run that may have written to the graph writes an improve row that carries its stage stamps in run_info.Error handling and propagation
Error handling and propagation
execute returns Err(ExecutionError) when the run fails. Items run in order, and the first item whose chain fails stops the run; with Pipeline::with_concurrency above 1, the current group of items finishes first. Before returning, the executor sends on_pipeline_run_errored with the error text.TaskError is a boxed std::error::Error, so you can downcast source to your task’s own error type. Retries, batch tasks and per-task rate limits are described on the Tasks page. cognify() adds its own failure report and rollback on top; see When a cognify run fails.Layered execution
Layered execution
- Outermost layer: the operation.
cognify()andAddPipelineresolve the dataset, take its lock where they need it, build theTaskContextand callexecute.cognify()makes one run per dataset. - Middle layer:
execute. It runs each input through the whole task chain, one item at a time, or up toPipeline::with_concurrencyitems at once. - Innermost layer: the task chain, run depth-first. A task that returns an iterator or a stream has its items gathered into batches and passed on to the next task before the next batch is pulled.
- Execution modes: awaited (
execute), spawned (execute_in_background) or blocking (execute_blocking).
cognify() passes its data items to the executor as a single input, so one cognify run is one item for the executor.Users and datasets
Users and datasets
execute does not look up datasets or check permissions. It reads the user and dataset ids from the PipelineContext you give it and attributes the run, its rows and the DataPoints it stamps to them. Resolving a dataset name, creating a dataset and checking access are the operations’ job; see Datasets and Datasets and access.Related
Tasks
The steps of a pipeline
PipelineContext
Run identity tasks can read
Operations
What the pipelines are used for