Skip to main content

What are Tasks

Tasks are the steps a pipeline is made of. Each one takes values in, does one job, and passes values on to the next task.
  • A task wraps a Rust closure that receives its input and the run’s TaskContext.
  • Every task has the same shape to the executor, so batching, retries, rate limiting and progress work the same way for all of them.
  • A task can return any type, but the built-in tasks mostly create or enrich DataPoints.

How Tasks work in Rust

A Task is one reusable unit of work that transforms data — classify, chunk, extract, summarize, embed. Tasks come in eight execution flavours (sync/async × single/iterator/stream × single-value/batch) so a step can stream, fan out, or process whole batches. They are composed with optional per-task config (TaskInfo: name, batch size, weight, rate limiter) and the pipeline executor routes values between them. A stream task that fails after it has started yielding fails the run: the failure is not retried, because the stream is already partly consumed, and items already passed downstream are not rolled back. Rust: Task / TypedTask / TaskInfo in cognee-core.

Why Tasks Exist

  • They give blocking functions, async functions, iterators and streams one calling convention, so the executor can chain any of them.
  • They let output flow straight into the next step: items from an iterator or stream are passed on in batches while the producer is still running.
  • They give batching control per step, which matters for LLM and I/O-heavy work.
  • They are the building blocks of Pipelines.

Core Concepts

  • Values: tasks exchange type-erased values (Arc<dyn Value>). Any Send + Sync + 'static type is a Value. The typed constructors downcast the input to your concrete type for you.
  • Execution: the executor calls each task the same way, whatever its flavour.
  • Batching: set per pipeline with Pipeline::with_batch_size (default 32) or per task with TaskInfo::with_batch_size.
  • Composition: one task’s output is the next task’s input. PipelineBuilder checks at compile time that the types line up.
  • Context: every call receives an Arc<TaskContext> with the run’s stores, cancellation and progress tokens, and its PipelineContext.

Dependencies & Ordering

Each task expects a certain input type and produces a certain output type. The built-in cognify pipeline shows the chain:
  • CognifyInput → ClassifiedDocuments
  • ClassifiedDocuments → ExtractedChunks
  • ExtractedChunks → SummarizedData (entities, relationships and summaries)
  • SummarizedData → CognifyResult (after writing to the graph and vector databases)
If the value a task receives is not the type it was built for, the call panics with Task input type mismatch. Build pipelines with PipelineBuilder and TypedTask to catch this at compile time instead.

Built-in Tasks

  • Ingestion (the add pipeline, add_pipeline): a process-input task (DataInput → ProcessedInput) that loads and hashes each input, then a persist task (ProcessedInput → Data) that resolves the dataset, deduplicates and stores the record.
  • Classification: classify_documents
  • Chunking: extract_chunks_from_documents
  • Graph extraction and summarization: extract_graph_and_summarize, one stage that runs extract_graph_from_data and summarization concurrently over the same chunks
  • Persistence: add_data_points
The four cognify tasks run in that order in cognify_pipeline. See cognify for what each stage writes.

Examples and details

Task is an enum with one variant per flavour. Each variant has a raw constructor over type-erased values and a typed constructor over your own types:
Key parts:
  • I / O: the input and output types. The closure borrows the input as &I and returns an owned Box<O>.
  • Arc<TaskContext>: the run’s context, passed to every call.
  • TaskError: Box<dyn std::error::Error + Send + Sync + 'static>. Returning Err fails the task.
TaskInfo::new(task) wraps a task with optional per-task settings. A Task or TypedTask converts into a TaskInfo with defaults, so Pipeline::with_task accepts either.TypedTask<I, O> has the same eight constructors (sync, async_fn, sync_iter, async_stream, and their *_batch forms) and keeps I and O in its type, which PipelineBuilder uses to check the chain.
The executor handles eight flavours. Single-value flavours are called once per item; batch flavours receive a slice of accumulated items.The raw constructors (Task::sync, Task::async_fn, Task::sync_iter, Task::async_stream, Task::sync_batch, Task::async_batch, Task::sync_iter_batch, Task::async_stream_batch) take Arc<dyn Value> or &[Box<dyn Value>] instead.The sync flavours are called inline by the executor. Async flavours must return a 'static future, so copy or clone what you need from the input before async move. Iterators and streams must be 'static too.The typed stream constructors wrap every item in Ok. A producer that can fail mid-stream builds a ValueStream (a stream of Result<Box<dyn Value>, TaskError>) and uses Task::async_stream.All built-in tasks are single-value: classify_documents is Sync, the others are Async.
For a compile-time check of the chain, use TypedTask and PipelineBuilder:
To run a pipeline, see Pipelines.Why small, DataPoint-focused tasks are easiest to compose:
  • Predictable input and output types
  • Easy to chain and to check with PipelineBuilder
  • Clear data flow between steps
Each input item runs through the whole task chain before the next one starts (or up to Pipeline::with_concurrency items at a time). Within an item, tasks run in order and each output becomes the next task’s input.When a task returns an iterator or stream, the executor gathers its items into batches. The size comes from the consuming task’s batch_size, or the pipeline default when it has none. A full batch is handed on before the next items are pulled:
  • a batch flavour receives the whole batch as one slice;
  • any other flavour receives the batch’s items one at a time.
Setting batch_size on a non-batch task is not a no-op: it decides how many upstream items are held in memory at once.Two sentinel values steer the flow. Returning DroppedSentinel discards the current item. Returning PassthroughSentinel forwards the task’s input unchanged, but only for a task marked with_enriches(); on any other task it is an error.Task::parallel runs several single-value tasks on the same input and keeps the last result. TypedTask::try_parallel runs two and merges both outputs; cognify’s extract_graph_and_summarize stage is built this way.
TaskInfo bundles a task with per-task settings:
A rate limiter implements the RateLimiter trait (async fn acquire(&self)). cognee-core ships TokenBucketLimiter and SemaphoreLimiter.
The task’s own limiter wins over Pipeline::with_rate_limiter. A single-value task acquires a token once per attempt, so every retry waits again. A batch task acquires once per batch.
  • A single-value task that returns Err is retried according to the pipeline’s RetryPolicy (NoRetry by default). When the attempts run out, the run fails with ExecutionError::TaskFailed, which carries the task index, the number of attempts and the error.
  • Batch flavours are not retried and emit no per-task watcher events. They must handle their own errors. Rate limiting, progress, cancellation and provenance stamping still apply to them.
  • A stream that yields an Err after it has started fails the producing task. That failure is not retried, and items already passed downstream are not rolled back. Errors before the first item belong in the task’s outer Result.

Pipelines

Chain tasks together

PipelineContext

What a task can read about its run

DataPoints

What tasks produce