Skip to main content

What are Pipelines

A pipeline runs tasks in order over your data. remember and recall are built on three of them: add, cognify and search. improve runs its own stages, and forget wraps the delete service.

How Pipelines work in Rust

A Pipeline is an orchestrated sequence of Tasks with shared context (database, graph, vector, cancellation, progress) and a watcher for status events. The concrete pipelines are: Pipeline orchestration primitives (PipelineWatcher, ExecStatusManager, thread pool) live in cognee-core. The end-to-end flow is described in operations. What a cognify run does when it fails, how a killed run is recovered, and how re-runs skip finished items are covered in Cognify and Serialization and recovery of pipeline runs.

How pipelines run

You usually trigger pipelines through the operations (remember, add, cognify) rather than calling the executor yourself. To run your own, build a Pipeline from tasks, a TaskContext with the stores and the run’s PipelineContext, and pass both to execute (crates/core/src/pipeline.rs):
execute returns the outputs of the last task, or an ExecutionError. One call is one run:
  • Ids. The run gets a random run id, which is its pipeline_run_id. Its pipeline_id is deterministic: uuid5 over the user id, the pipeline name and the dataset id from the PipelineContext, or Pipeline::id when the pipeline has no name.
  • Status events. The watcher receives on_pipeline_run_started before the first task, on_task_started / on_task_completed around each single-value task, and then on_pipeline_run_completed or on_pipeline_run_errored. The compact on_pipeline / on_task methods receive the same lifecycle as PipelineStatus / TaskStatus values, including ItemSkipped for an item skipped as already done. NoopWatcher ignores everything; DbPipelineWatcher (feature pipeline-run-registry) writes the run’s pipeline_runs rows.
  • Dataset scope. When the PipelineContext names a dataset, the whole run executes in that dataset’s scope, so under per-dataset storage every task reaches that dataset’s own graph and vector databases.
  • Other entry points. execute_in_background takes owned (Arc) arguments, spawns the run on the current Tokio runtime and returns a PipelineRunHandle you can wait() on or abort(). execute_blocking runs it on a new single-threaded runtime for non-async callers, and panics when called inside a Tokio runtime. Both return PipelineRunResult, whose run_id is Pipeline::id, not the run’s pipeline_run_id.
A malformed pipeline (no tasks, a batch size or concurrency of 0) fails with ExecutionError::NoTasks or InvalidConfig before any event is sent.
A pipeline’s name (Pipeline::with_name) keys its run records and its per-item status, so it decides which earlier work a new run can skip.

Reserved pipeline names

Do not name your own pipeline add_pipeline or cognify_pipeline. Its rows would get the same pipeline_id as the built-in operation’s for that user and dataset, and the dataset’s reported add or cognify status would come from your run.

How re-runs avoid duplicate work

No pipeline is refused because of the dataset’s earlier runs: every call starts a new run. Work is skipped per item instead.
  • In the executor. Set Pipeline::with_data_id to a function that returns a stable id for an input, and give the TaskContext an ExecStatusManager. Before the first task the executor asks is_completed(data_id, pipeline_name, dataset_id); a completed item is skipped and reported as PipelineStatus::ItemSkipped. After the chain finishes it calls mark_completed, or mark_failed when a task failed. The default NoopExecStatusManager reports nothing as completed, so every item runs.
  • In cognify. cognify() checks its completion markers before it builds the pipeline, so finished items are never classified, chunked or sent to an LLM. See Re-cognifying is incremental.
  • In add. AddPipeline’s data-id function returns None, so the executor never skips an add input.
execute itself takes no lock. The operations do: cognify() holds the dataset’s lock for its whole run, and add, dataset-scoped deletes and improve take the same in-process registry (cognee_core::DatasetLocks::global()), so writers to one dataset queue and different datasets run in parallel. The lock does not span processes. Details are in Serialization and recovery of pipeline runs and Improve: Locking.A custom pipeline that writes to a dataset should take the same lock:

Nested (re-entrant) runs

hold marks the dataset as held for the current Tokio task, so a nested hold or lock of the same dataset inside it returns at once instead of waiting on itself. That is how an improve stage can run cognify() on the dataset the improve already holds. A guard taken with plain lock does not mark the dataset held, and work moved to another task with tokio::spawn does not inherit the mark.
Give your pipelines a unique name. Status tracking through an ExecStatusManager is keyed by (data_id, pipeline_name, dataset_id), so a unique name keeps your skipped items separate from the built-in ones, and your run rows separate from the dataset’s add and cognify status.
When your watcher persists rows, every name except search, recall and improve_pipeline counts as a write in improve’s change check, so after a run of your pipeline the next improve runs its global context index stage again instead of reporting it already completed.
How a run ends depends on how it was stopped:
  • A task error or a cancellation (CancellationHandle::cancel, checked before each task) ends the run in-process. The watcher receives on_pipeline_run_errored, so a persisting watcher writes the ERRORED row.
  • A killed process, a dropped future or PipelineRunHandle::abort stops the run without running any more of its code, so its newest row stays STARTED.
A STARTED row never blocks a re-run. Startup recovery closes the cognify and add runs a dead process left STARTED, rolling a cognify run back first; it recognises them by the ownership token in their STARTED row. A custom pipeline without Pipeline::with_recovery_lock writes {} there, and recovery leaves its run alone. See Serialization and recovery of pipeline runs and When a cognify run fails.
A persisting watcher (DbPipelineWatcher) writes the relational pipeline_runs table (crates/database/src/entities/pipeline_run.rs). Each lifecycle event inserts a new row, so one run leaves a DATASET_PROCESSING_STARTED row and then a DATASET_PROCESSING_COMPLETED or DATASET_PROCESSING_ERRORED row, sharing one pipeline_run_id. Deduplicate by pipeline_run_id when you count runs.The table does not store the run’s input. The status strings and run_info keys are the ones Python cognee writes.Fields a task publishes with TaskContext::publish_payload_field go to a separate pipeline_run_payload_fields table, one row per run and key; see Custom pipeline state.The table also holds rows that record a change rather than a pipeline run: a delete writes a forget row on each dataset it touched, and an improve run that may have written to the graph writes an improve row that carries its stage stamps in run_info.
execute returns Err(ExecutionError) when the run fails. Items run in order, and the first item whose chain fails stops the run; with Pipeline::with_concurrency above 1, the current group of items finishes first. Before returning, the executor sends on_pipeline_run_errored with the error text.
TaskError is a boxed std::error::Error, so you can downcast source to your task’s own error type. Retries, batch tasks and per-task rate limits are described on the Tasks page. cognify() adds its own failure report and rollback on top; see When a cognify run fails.
  • Outermost layer: the operation. cognify() and AddPipeline resolve the dataset, take its lock where they need it, build the TaskContext and call execute. cognify() makes one run per dataset.
  • Middle layer: execute. It runs each input through the whole task chain, one item at a time, or up to Pipeline::with_concurrency items at once.
  • Innermost layer: the task chain, run depth-first. A task that returns an iterator or a stream has its items gathered into batches and passed on to the next task before the next batch is pulled.
  • Execution modes: awaited (execute), spawned (execute_in_background) or blocking (execute_blocking).
cognify() passes its data items to the executor as a single input, so one cognify run is one item for the executor.
execute does not look up datasets or check permissions. It reads the user and dataset ids from the PipelineContext you give it and attributes the run, its rows and the DataPoints it stamps to them. Resolving a dataset name, creating a dataset and checking access are the operations’ job; see Datasets and Datasets and access.

Tasks

The steps of a pipeline

PipelineContext

Run identity tasks can read

Operations

What the pipelines are used for