> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognee.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Rust SDK Pipelines

> Pipelines in cognee-rust: ordered task sequences with shared context, and the add, cognify and search pipelines built from them.

## What are Pipelines

A pipeline runs tasks in order over your data. `remember` and `recall` are built on three of them: add, cognify and search. `improve` runs its own stages, and `forget` wraps the delete service.

## How Pipelines work in Rust

A **Pipeline** is an orchestrated sequence of Tasks with shared context
(database, graph, vector, cancellation, progress) and a watcher for status
events. The concrete pipelines are:

| Pipeline | What it composes | Crate / entry point |
| - | - | - |
| **add** | ingest → hash → dedup → persist | [`cognee-ingestion`](https://github.com/topoteretes/cognee-rs/blob/main/crates/ingestion/) (`AddPipeline`) |
| **cognify** | classify → chunk → extract → summarize → index → FK edges | [`cognee-cognify`](https://github.com/topoteretes/cognee-rs/blob/main/crates/cognify/) (`cognify()`) |
| **search** | route query → retrieve → (optionally) complete | [`cognee-search`](https://github.com/topoteretes/cognee-rs/blob/main/crates/search/) (`SearchOrchestrator`) |

Pipeline orchestration primitives (`PipelineWatcher`, `ExecStatusManager`,
thread pool) live in [`cognee-core`](https://github.com/topoteretes/cognee-rs/blob/main/crates/core/). The end-to-end flow is
described in [operations](https://docs.cognee.ai/rust/core-concepts/main-operations/overview.md).

What a cognify run does when it fails, how a killed run is recovered, and how re-runs skip finished items are covered in [Cognify](https://docs.cognee.ai/rust/core-concepts/main-operations/legacy-operations/cognify.md#when-a-cognify-run-fails) and [Serialization and recovery of pipeline runs](https://docs.cognee.ai/rust/configuration/storage.md#serialization-and-recovery-of-pipeline-runs).

## How pipelines run

You usually trigger pipelines through the operations (`remember`, `add`, `cognify`) rather than calling the executor yourself. To run your own, build a `Pipeline` from [tasks](https://docs.cognee.ai/rust/core-concepts/building-blocks/tasks.md), a `TaskContext` with the stores and the run's [PipelineContext](https://docs.cognee.ai/rust/core-concepts/building-blocks/pipeline-context.md), and pass both to `execute` ([`crates/core/src/pipeline.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/core/src/pipeline.rs)):

```rust theme={null}
use std::sync::Arc;
use cognee_core::{execute, NoopWatcher, Pipeline, Task, TaskContextBuilder, TaskInfo, Value};

let pipeline = Pipeline::new("double numbers")
    .with_name("my_double_pipeline")
    .with_task(TaskInfo::new(Task::sync_typed(|n: &i64, _ctx| Ok(Box::new(n * 2)))).with_name("double"));

let (_cancel, ctx) = TaskContextBuilder::new()
    .thread_pool(thread_pool)
    .database(database)
    .graph_db(graph_db)
    .vector_db(vector_db)
    .pipeline_context(pipeline_ctx)
    .build()?;

let inputs: Vec<Arc<dyn Value>> = vec![Arc::new(2_i64), Arc::new(3_i64)];
let outputs = execute(&pipeline, inputs, Arc::new(ctx), &NoopWatcher).await?;
```

`execute` returns the outputs of the last task, or an `ExecutionError`. One call is one run:

* **Ids.** The run gets a random run id, which is its `pipeline_run_id`. Its `pipeline_id` is deterministic: `uuid5` over the user id, the pipeline name and the dataset id from the `PipelineContext`, or `Pipeline::id` when the pipeline has no name.
* **Status events.** The watcher receives `on_pipeline_run_started` before the first task, `on_task_started` / `on_task_completed` around each single-value task, and then `on_pipeline_run_completed` or `on_pipeline_run_errored`. The compact `on_pipeline` / `on_task` methods receive the same lifecycle as `PipelineStatus` / `TaskStatus` values, including `ItemSkipped` for an item skipped as already done. `NoopWatcher` ignores everything; `DbPipelineWatcher` (feature `pipeline-run-registry`) writes the run's `pipeline_runs` rows.
* **Dataset scope.** When the `PipelineContext` names a dataset, the whole run executes in that dataset's scope, so under per-dataset storage every task reaches that dataset's own graph and vector databases.
* **Other entry points.** `execute_in_background` takes owned (`Arc`) arguments, spawns the run on the current Tokio runtime and returns a `PipelineRunHandle` you can `wait()` on or `abort()`. `execute_blocking` runs it on a new single-threaded runtime for non-async callers, and panics when called inside a Tokio runtime. Both return `PipelineRunResult`, whose `run_id` is `Pipeline::id`, not the run's `pipeline_run_id`.

A malformed pipeline (no tasks, a batch size or concurrency of 0) fails with `ExecutionError::NoTasks` or `InvalidConfig` before any event is sent.

> [accordion] Pipeline Names and Re-runs
>
> A pipeline's name (`Pipeline::with_name`) keys its run records and its per-item status, so it decides which earlier work a new run can skip.
>
> ### Reserved pipeline names
>
> | Name | Used by | What reads it |
> | - | - | - |
> | `add_pipeline` | `AddPipeline` | `DatasetManager::get_status` reports the newest row under this name as the dataset's add status |
> | `cognify_pipeline` | `cognify()` | `DatasetManager::get_status` (cognify status); the cognify completion markers on each data item are stored under this key |
>
> > [!WARNING]
> >
> > Do not name your own pipeline `add_pipeline` or `cognify_pipeline`. Its rows would get the same `pipeline_id` as the built-in operation's for that user and dataset, and the dataset's reported add or cognify status would come from your run.
>
> ### How re-runs avoid duplicate work
>
> No pipeline is refused because of the dataset's earlier runs: every call starts a new run. Work is skipped per item instead.
>
> * **In the executor.** Set `Pipeline::with_data_id` to a function that returns a stable id for an input, and give the `TaskContext` an `ExecStatusManager`. Before the first task the executor asks `is_completed(data_id, pipeline_name, dataset_id)`; a completed item is skipped and reported as `PipelineStatus::ItemSkipped`. After the chain finishes it calls `mark_completed`, or `mark_failed` when a task failed. The default `NoopExecStatusManager` reports nothing as completed, so every item runs.
> * **In cognify.** `cognify()` checks its completion markers before it builds the pipeline, so finished items are never classified, chunked or sent to an LLM. See [Re-cognifying is incremental](https://docs.cognee.ai/rust/core-concepts/main-operations/legacy-operations/cognify.md#re-cognifying-is-incremental).
> * **In add.** `AddPipeline`'s data-id function returns `None`, so the executor never skips an add input.

> [accordion] Per-dataset serialization
>
> `execute` itself takes no lock. The operations do: `cognify()` holds the dataset's lock for its whole run, and `add`, dataset-scoped deletes and `improve` take the same in-process registry (`cognee_core::DatasetLocks::global()`), so writers to one dataset queue and different datasets run in parallel. The lock does not span processes. Details are in [Serialization and recovery of pipeline runs](https://docs.cognee.ai/rust/configuration/storage.md#serialization-and-recovery-of-pipeline-runs) and [Improve: Locking](https://docs.cognee.ai/rust/core-concepts/main-operations/improve.md#locking).
>
> A custom pipeline that writes to a dataset should take the same lock:
>
> ```rust theme={null}
> use cognee_core::DatasetLocks;
>
> let outputs = DatasetLocks::global()
>     .hold([dataset_id], execute(&pipeline, inputs, ctx, &watcher))
>     .await?;
> ```
>
> ### Nested (re-entrant) runs
>
> `hold` marks the dataset as held for the current Tokio task, so a nested `hold` or `lock` of the same dataset inside it returns at once instead of waiting on itself. That is how an improve stage can run `cognify()` on the dataset the improve already holds. A guard taken with plain `lock` does not mark the dataset held, and work moved to another task with `tokio::spawn` does not inherit the mark.

> [accordion] Custom pipeline naming
>
> Give your pipelines a unique name. Status tracking through an `ExecStatusManager` is keyed by `(data_id, pipeline_name, dataset_id)`, so a unique name keeps your skipped items separate from the built-in ones, and your run rows separate from the dataset's add and cognify status.
>
> ```rust theme={null}
> let pipeline = Pipeline::new("enrich documents")
>     .with_name("my_enrichment_pipeline")
>     .with_task(enrich_task);
> ```
>
> When your watcher persists rows, every name except `search`, `recall` and `improve_pipeline` counts as a write in `improve`'s change check, so after a run of your pipeline the next improve runs its [global context index](https://docs.cognee.ai/rust/core-concepts/main-operations/improve.md#global-context-index) stage again instead of reporting it already completed.

> [accordion] Crash recovery and stuck pipelines
>
> How a run ends depends on how it was stopped:
>
> * **A task error or a cancellation** (`CancellationHandle::cancel`, checked before each task) ends the run in-process. The watcher receives `on_pipeline_run_errored`, so a persisting watcher writes the `ERRORED` row.
> * **A killed process, a dropped future or `PipelineRunHandle::abort`** stops the run without running any more of its code, so its newest row stays `STARTED`.
>
> A `STARTED` row never blocks a re-run. Startup recovery closes the `cognify` and `add` runs a dead process left `STARTED`, rolling a cognify run back first; it recognises them by the ownership token in their `STARTED` row. A custom pipeline without `Pipeline::with_recovery_lock` writes `{}` there, and recovery leaves its run alone. See [Serialization and recovery of pipeline runs](https://docs.cognee.ai/rust/configuration/storage.md#serialization-and-recovery-of-pipeline-runs) and [When a cognify run fails](https://docs.cognee.ai/rust/core-concepts/main-operations/legacy-operations/cognify.md#when-a-cognify-run-fails).

> [accordion] What gets stored in a pipeline run record
>
> A persisting watcher (`DbPipelineWatcher`) writes the relational `pipeline_runs` table ([`crates/database/src/entities/pipeline_run.rs`](https://github.com/topoteretes/cognee-rs/blob/main/crates/database/src/entities/pipeline_run.rs)). Each lifecycle event inserts a new row, so one run leaves a `DATASET_PROCESSING_STARTED` row and then a `DATASET_PROCESSING_COMPLETED` or `DATASET_PROCESSING_ERRORED` row, sharing one `pipeline_run_id`. Deduplicate by `pipeline_run_id` when you count runs.
>
> | Column | What it holds |
> | - | - |
> | `id` | The row's own id |
> | `created_at` | When the row was written |
> | `status` | `DATASET_PROCESSING_STARTED`, `…_COMPLETED` or `…_ERRORED` (`…_INITIATED` only on rows from older versions) |
> | `pipeline_run_id` | The run id |
> | `pipeline_name`, `pipeline_id` | The pipeline's name and deterministic id |
> | `dataset_id` | The dataset, `NULL` for a run without one |
> | `run_info` | `{}`, or `{"recovery_lock": …}` on the `STARTED` row of a run with an ownership token; `{"error": …}` on the `ERRORED` row |
> | `user_id` | The user who triggered the run |
>
> The table does not store the run's input. The status strings and `run_info` keys are the ones Python cognee writes.
>
> Fields a task publishes with `TaskContext::publish_payload_field` go to a separate `pipeline_run_payload_fields` table, one row per run and key; see [Custom pipeline state](https://docs.cognee.ai/rust/core-concepts/building-blocks/pipeline-context.md#custom-pipeline-state).
>
> The table also holds rows that record a change rather than a pipeline run: a delete writes a `forget` row on each dataset it touched, and an improve run that may have written to the graph writes an `improve` row that carries its stage stamps in `run_info`.

> [accordion] Error handling and propagation
>
> `execute` returns `Err(ExecutionError)` when the run fails. Items run in order, and the first item whose chain fails stops the run; with `Pipeline::with_concurrency` above 1, the current group of items finishes first. Before returning, the executor sends `on_pipeline_run_errored` with the error text.
>
> | Variant | When |
> | - | - |
> | `TaskFailed { task_index, attempts, source }` | A task returned `Err` and its retries (the pipeline's `RetryPolicy`) ran out. `source` is the task's own `TaskError` |
> | `Cancelled` | The run's cancellation token was cancelled |
> | `NoTasks`, `InvalidConfig { reason }` | The pipeline was malformed |
> | `RunRecordFailed { reason }` | Every task succeeded, but the watcher could not write the `COMPLETED` row. The run counts as failed and is recorded `ERRORED` |
>
> ```rust theme={null}
> use cognee_core::ExecutionError;
>
> match execute(&pipeline, inputs, ctx, &watcher).await {
>     Ok(outputs) => { /* outputs of the last task */ }
>     Err(ExecutionError::TaskFailed { task_index, source, .. }) => {
>         tracing::warn!("task {task_index} failed: {source}");
>     }
>     Err(e) => return Err(e.into()),
> }
> ```
>
> `TaskError` is a boxed `std::error::Error`, so you can downcast `source` to your task's own error type. Retries, batch tasks and per-task rate limits are described on the [Tasks](https://docs.cognee.ai/rust/core-concepts/building-blocks/tasks.md) page. `cognify()` adds its own failure report and rollback on top; see [When a cognify run fails](https://docs.cognee.ai/rust/core-concepts/main-operations/legacy-operations/cognify.md#when-a-cognify-run-fails).

> [accordion] Layered execution
>
> * Outermost layer: the operation. `cognify()` and `AddPipeline` resolve the dataset, take its lock where they need it, build the `TaskContext` and call `execute`. `cognify()` makes one run per dataset.
> * Middle layer: `execute`. It runs each input through the whole task chain, one item at a time, or up to `Pipeline::with_concurrency` items at once.
> * Innermost layer: the task chain, run depth-first. A task that returns an iterator or a stream has its items gathered into batches and passed on to the next task before the next batch is pulled.
> * Execution modes: awaited (`execute`), spawned (`execute_in_background`) or blocking (`execute_blocking`).
>
> `cognify()` passes its data items to the executor as a single input, so one cognify run is one item for the executor.

> [accordion] Users and datasets
>
> `execute` does not look up datasets or check permissions. It reads the user and dataset ids from the `PipelineContext` you give it and attributes the run, its rows and the DataPoints it stamps to them. Resolving a dataset name, creating a dataset and checking access are the operations' job; see [Datasets](https://docs.cognee.ai/rust/core-concepts/further-concepts/datasets.md) and [Datasets and access](https://docs.cognee.ai/rust/configuration/datasets-and-access.md).

## Related

> [card] [Tasks](https://docs.cognee.ai/rust/core-concepts/building-blocks/tasks.md)
>
> The steps of a pipeline

> [card] [PipelineContext](https://docs.cognee.ai/rust/core-concepts/building-blocks/pipeline-context.md)
>
> Run identity tasks can read

> [card] [Operations](https://docs.cognee.ai/rust/core-concepts/main-operations/overview.md)
>
> What the pipelines are used for


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.