What are Tasks
Tasks are the steps a pipeline is made of. Each one takes values in, does one job, and passes values on to the next task.- A task wraps a Rust closure that receives its input and the run’s
TaskContext. - Every task has the same shape to the executor, so batching, retries, rate limiting and progress work the same way for all of them.
- A task can return any type, but the built-in tasks mostly create or enrich DataPoints.
How Tasks work in Rust
A Task is one reusable unit of work that transforms data — classify, chunk, extract, summarize, embed. Tasks come in eight execution flavours (sync/async × single/iterator/stream × single-value/batch) so a step can stream, fan out, or process whole batches. They are composed with optional per-task config (TaskInfo: name, batch size, weight, rate limiter) and the pipeline executor
routes values between them. A stream task that fails after it has started
yielding fails the run: the failure is not retried, because the stream is
already partly consumed, and items already passed downstream are not rolled
back.
Rust: Task / TypedTask / TaskInfo in cognee-core.
Why Tasks Exist
- They give blocking functions, async functions, iterators and streams one calling convention, so the executor can chain any of them.
- They let output flow straight into the next step: items from an iterator or stream are passed on in batches while the producer is still running.
- They give batching control per step, which matters for LLM and I/O-heavy work.
- They are the building blocks of Pipelines.
Core Concepts
- Values: tasks exchange type-erased values (
Arc<dyn Value>). AnySend + Sync + 'statictype is aValue. The typed constructors downcast the input to your concrete type for you. - Execution: the executor calls each task the same way, whatever its flavour.
- Batching: set per pipeline with
Pipeline::with_batch_size(default 32) or per task withTaskInfo::with_batch_size. - Composition: one task’s output is the next task’s input.
PipelineBuilderchecks at compile time that the types line up. - Context: every call receives an
Arc<TaskContext>with the run’s stores, cancellation and progress tokens, and its PipelineContext.
Dependencies & Ordering
Each task expects a certain input type and produces a certain output type. The built-in cognify pipeline shows the chain:CognifyInput→ClassifiedDocumentsClassifiedDocuments→ExtractedChunksExtractedChunks→SummarizedData(entities, relationships and summaries)SummarizedData→CognifyResult(after writing to the graph and vector databases)
Task input type mismatch. Build pipelines with PipelineBuilder and TypedTask to catch this at compile time instead.
Built-in Tasks
- Ingestion (the add pipeline,
add_pipeline): a process-input task (DataInput→ProcessedInput) that loads and hashes each input, then a persist task (ProcessedInput→Data) that resolves the dataset, deduplicates and stores the record. - Classification:
classify_documents - Chunking:
extract_chunks_from_documents - Graph extraction and summarization:
extract_graph_and_summarize, one stage that runsextract_graph_from_dataand summarization concurrently over the same chunks - Persistence:
add_data_points
cognify_pipeline. See cognify for what each stage writes.
Examples and details
Task API & Constructor
Task API & Constructor
Task is an enum with one variant per flavour. Each variant has a raw constructor over type-erased values and a typed constructor over your own types:I/O: the input and output types. The closure borrows the input as&Iand returns an ownedBox<O>.Arc<TaskContext>: the run’s context, passed to every call.TaskError:Box<dyn std::error::Error + Send + Sync + 'static>. ReturningErrfails the task.
TaskInfo::new(task) wraps a task with optional per-task settings. A Task or TypedTask converts into a TaskInfo with defaults, so Pipeline::with_task accepts either.TypedTask<I, O> has the same eight constructors (sync, async_fn, sync_iter, async_stream, and their *_batch forms) and keeps I and O in its type, which PipelineBuilder uses to check the chain.Supported Task Types
Supported Task Types
The executor handles eight flavours. Single-value flavours are called once per item; batch flavours receive a slice of accumulated items.
The raw constructors (
Task::sync, Task::async_fn, Task::sync_iter, Task::async_stream, Task::sync_batch, Task::async_batch, Task::sync_iter_batch, Task::async_stream_batch) take Arc<dyn Value> or &[Box<dyn Value>] instead.The sync flavours are called inline by the executor. Async flavours must return a 'static future, so copy or clone what you need from the input before async move. Iterators and streams must be 'static too.The typed stream constructors wrap every item in Ok. A producer that can fail mid-stream builds a ValueStream (a stream of Result<Box<dyn Value>, TaskError>) and uses Task::async_stream.All built-in tasks are single-value: classify_documents is Sync, the others are Async.Writing a Custom Task
Writing a Custom Task
TypedTask and PipelineBuilder:- Predictable input and output types
- Easy to chain and to check with
PipelineBuilder - Clear data flow between steps
Execution Flow
Execution Flow
Each input item runs through the whole task chain before the next one starts (or up to
Pipeline::with_concurrency items at a time). Within an item, tasks run in order and each output becomes the next task’s input.When a task returns an iterator or stream, the executor gathers its items into batches. The size comes from the consuming task’s batch_size, or the pipeline default when it has none. A full batch is handed on before the next items are pulled:- a batch flavour receives the whole batch as one slice;
- any other flavour receives the batch’s items one at a time.
batch_size on a non-batch task is not a no-op: it decides how many upstream items are held in memory at once.Two sentinel values steer the flow. Returning DroppedSentinel discards the current item. Returning PassthroughSentinel forwards the task’s input unchanged, but only for a task marked with_enriches(); on any other task it is an error.Task::parallel runs several single-value tasks on the same input and keeps the last result. TypedTask::try_parallel runs two and merges both outputs; cognify’s extract_graph_and_summarize stage is built this way.TaskInfo
TaskInfo
TaskInfo bundles a task with per-task settings:Rate Limiting
Rate Limiting
A rate limiter implements the The task’s own limiter wins over
RateLimiter trait (async fn acquire(&self)). cognee-core ships TokenBucketLimiter and SemaphoreLimiter.Pipeline::with_rate_limiter. A single-value task acquires a token once per attempt, so every retry waits again. A batch task acquires once per batch.Errors, Retries and Streams
Errors, Retries and Streams
- A single-value task that returns
Erris retried according to the pipeline’sRetryPolicy(NoRetryby default). When the attempts run out, the run fails withExecutionError::TaskFailed, which carries the task index, the number of attempts and the error. - Batch flavours are not retried and emit no per-task watcher events. They must handle their own errors. Rate limiting, progress, cancellation and provenance stamping still apply to them.
- A stream that yields an
Errafter it has started fails the producing task. That failure is not retried, and items already passed downstream are not rolled back. Errors before the first item belong in the task’s outerResult.
Related
Pipelines
Chain tasks together
PipelineContext
What a task can read about its run
DataPoints
What tasks produce