Skip to main content

What is PipelineContext

PipelineContext tells a running task which pipeline run it belongs to, who started it, which dataset it works on and which data item is being processed. Tasks use it to attribute what they write, for example to stamp provenance on DataPoints.

How tasks reach it

Every task runs with a TaskContext (cognee-core, task_context.rs). It carries the shared resources of the run: TaskContext::pipeline() returns the PipelineContext. It panics when none is set, so call it only from tasks that always run inside a pipeline executor; otherwise read the pipeline_ctx option. A Rust task always receives the TaskContext and reads what it needs from it.

Fields

user_label() gives the value used as a DataPoint’s source_user: the email when set, else the user id, else nothing.

current_data

current_data is the original value that entered the pipeline for this data item, not the previous task’s output. The executor sets it once per item, and every task in the chain sees the same value, batch tasks included. Before cognee-rs #120 (August 2026), current_data held the previous task’s output, and batch tasks saw None. A custom task written against that behaviour must use its input argument instead. Because the value stays reachable until the item’s whole chain finishes, a pipeline whose input holds a large buffer (such as DataInput::Binary) keeps it in memory longer. With Pipeline::with_concurrency above 1, more items overlap. Stream large payloads instead of putting them in the input value.

Provenance stamping

provenance_visited makes sure a DataPoint that passes through several tasks is stamped once, with the first task’s name.

Setting the context

The caller builds the PipelineContext and puts it on the TaskContext with TaskContextBuilder::pipeline_context. Its fields are public and it has no constructor, so you write all of them:
Leave current_data and run_id unset: execute fills them in. At the start of each run it also clears provenance_visited, so a reused context does not carry stamps from an earlier run. The user and dataset ids decide the run’s attribution: execute derives the run’s pipeline_id from the user id, the pipeline name and the dataset id, and runs in the dataset’s scope when a dataset is set. See How pipelines run.

Custom pipeline state

PipelineContext has no free-form map for your own values. A task is a Send + Sync + 'static closure, so it takes its settings by capturing them:
To pass values out of a run, a task calls TaskContext::publish_payload_field(key, value) with a JSON value:
The field goes to the watcher set on the TaskContext (TaskContextBuilder::pipeline_watcher), not to the one passed to execute, and the call does nothing when that watcher is unset or the task runs outside execute. DbPipelineWatcher stores each field in the pipeline_run_payload_fields table, one row per run and key, where a later value for the same key replaces the earlier one. PipelineRunRepository::get_payload(run_id) reads them back as a JSON map.

Examples and details

Read the identity fields when a task writes its own records or applies per-user logic:
The stores on the TaskContext already point at the run’s dataset: when dataset_id is set, execute runs every task in that dataset’s scope, so under per-dataset storage graph_db and vector_db reach the dataset’s own databases.
Every task of a run sees the same PipelineContext values: the same user, dataset, run id and provenance_visited set. Only current_data changes, once per data item. Tasks receive the context behind an Arc and cannot change it, so data flows from task to task through task outputs. For settings several tasks share, capture clones of one Arc:
TaskContext::pipeline() panics without a PipelineContext. A task that may also run outside a pipeline executor, in a test or called directly, reads the option instead:

Tasks

The units of work that read the context

Pipelines

The runs the context describes

DataPoints

Where provenance is stamped