What is the improve operation
improve runs the self-improvement loop over a dataset. It turns session activity (ratings, Q&A, agent traces, preferences) into lasting graph memory and keeps dataset-wide structures such as the global context index up to date. remember runs it automatically unless you turn that off.
Stages
Runs the self-improvement loop over a dataset as nine gated stages, in this order:feedback_weights, persist_session_qa,
persist_agent_traces, extract_agent_context, distill_sessions,
update_user_preferences, build_truth_subspace, review_conflicts,
global_context_index. The first seven read the given sessions and are
skipped with no_session_ids without them; the last two work on the graph
alone and run by default. Each stage reports a StageResult (status, reason,
error, counts, duration_ms) on the returned ImproveResult (dataset_id,
dataset_name, session_ids, stages, background, finished, error,
memify_run), whose status summarises them (completed, errored,
skipped, running). The earlier memify (triplet enrichment) and
sync_graph_to_session stages are removed, and the old counters
(stages_run, feedback_entries_processed, sessions_persisted, …) are now
stage counts. Stages named in improve.stages_disabled
(IMPROVE_STAGES_DISABLED, a comma-separated list) are skipped first, with
disabled_by_config; an unknown name, or the fatal persist_session_qa, makes
the configuration invalid and the run fails before any stage. feedback_alpha
defaults to improve.feedback_alpha (0.1). The options that only scoped or
fed the removed memify stage are gone too
(node_name and the CLI --node-name flag, extraction_tasks,
enrichment_tasks, data, and the bindings’ nodeName). ImproveResult still serializes the deprecated memify_run key,
as {}. The bindings’ improve options accept datasetId (a UUID string): the
improve targets that dataset, possibly
one shared with the caller, instead of looking datasetName up among the
caller’s own datasets.
Background runs
Background runs.cognee::api::improve_in_background runs improve
in the background: it validates the configuration and
feedback_alpha and resolves the dataset first — a name resolves to the
caller’s dataset or is created for it, a dataset id that names no dataset
(InvalidArgument; PermissionDenied when an ACL store is wired, which checks
the grant first) or one the caller may not write (PermissionDenied; the
ACL’s write grant, or ownership without an ACL) is refused — then spawns the
run as a process-wide background task and returns the running result with
the resolved dataset id and name. Wait for it with
cognee::wait_for_background_tasks; every binding exposes the same wait
(waitForBackgroundTasks in TS and Java, wait_for_background_tasks in
Python, cg_sdk_wait_for_background_tasks in C). The wait takes an optional
timeoutSeconds (timeout_seconds in Python), which cancels nothing when it
elapses, and reports {finished, pending}: finished is false only when
the timeout elapsed with work still running. An improve op with
runInBackground: true uses this entry point. ImproveParams::run_in_background
on improve() itself is only a label, for a host that dispatches the run on
its own. The bindings’ improve options also accept buildGlobalContextIndex
and reviewConflicts, which follow their improve.* setting when omitted (an
explicit value wins either way), and buildTruthSubspace, which opts in to
that stage.
Feedback weights
Besides an explicitfeedback_score, a turn’s answer is rated implicitly by the latest
kind = "feedback" session-context row that references it, and an implicit
rating moves the graph elements at alpha * 0.5; an explicit score still wins.
Each turn records which node and edge ids it already moved, so a retry moves
only the rest; ids missing from the graph keep the turn pending, and a turn is
marked done after 3 attempts. A graph or session-store error on one turn counts
it as skipped instead of aborting the stage.
Breaking: SessionQAEntry::memify_metadata (and
SessionQAUpdate::memify_metadata) is now
Option<HashMap<String, serde_json::Value>> instead of
Option<HashMap<String, bool>> (TypeScript: Record<string, unknown> | null),
so it holds the lists and numbers Python writes there; read a flag with
Value::as_bool. The old bool-only decode dropped the whole map when it met one.
For callers of cognee_cognify::memify, FeedbackApplyResult gained public
fields (done, errored, elements_moved, elements_pruned,
elements_failed), so a struct literal no longer compiles, and FeedbackError
gained the MetadataNotUpdated and InvalidSessionIds variants.
Failures and skipped stages
The run fails open — a failed stage is recorded and the next one runs — except that a failure inpersist_session_qa (losing session Q&A would be data
loss) stops the run with ApiError::ImproveAborted carrying the partial
result, and an exhausted LLM budget skips every later stage with
budget_exhausted. In the bindings that abort is SdkError::ImproveAborted
(code IMPROVE_ABORTED): Python’s CogneeImproveAbortedError.improve_result,
JavaScript’s ImproveAbortedError.improveResult; C (CG_ERR_RUNTIME) and Java
append improve_result: <json> to the message. The stages that draft text with
an LLM are skipped with no_llm_configured when none is configured.
review_conflicts and global_context_index run by default
(improve.review_conflicts, improve.global_context_index; a per-call
review_conflicts / build_global_context_index wins either way) and report
turned_off when off. review_conflicts is not ported yet and always reports
a skip reason (turned_off, then no_llm_configured, then
edge_evidence_disabled while cognify.edge_evidence_enabled is off, then
not_ported).
Breaking: both stages used to be opt-in, and a switched-off one used to
report opt_in_disabled (build_truth_subspace still does).
ImproveParams::build_global_context_index is now Option<bool>, and
ImproveParams / OwnedImproveParams gained review_conflicts: Option<bool>,
so a struct literal needs both (None follows the setting). Set
IMPROVE_GLOBAL_CONTEXT_INDEX=false and IMPROVE_REVIEW_CONFLICTS=false to
keep the old behavior.
Agent context lessons
extract_agent_context runs while automatic session context is on
(auto_feedback with session caching; otherwise auto_feedback_disabled)
and an LLM is configured. Per session it reads the trace steps newer than the
session’s extraction watermark (plus 3 already-processed ones, at most 40),
shows the model the session’s existing agent lessons,
and stores at most 5 new lessons as agent-profile session-context entries
(sections tool_rules, workflow_state, success_patterns,
failure_lessons, environment_facts). A lesson is stored only with
confidence ≥ 0.75, cut to 280 characters; one whose normalized text an
entry of the same section already has is not stored again. The watermark
advances only after the session’s extraction succeeded. Counts: lessons,
sessions_failed; the stage is errored when any session failed, and the
other sessions still run. Error texts shown to the model are redacted
(tokens, secrets, UUIDs, long numbers).
User preferences
update_user_preferences runs while personalization_enabled is on. It folds
the sessions’ turns into one UserPreference node per (user, dataset), an
internal node (never embedded, never returned by a search) in the
user_preferences node set, and weighted prefers edges from it to the graph
nodes the turns’ answers used (used_graph_element_ids.node_ids). A turn’s
rating is its feedback_score, else the rating a feedback row inferred for
it; a rating r pulls each node’s weight toward (r - 1) / 4 by
preference_alpha, from its decayed value
0.5 + (w - 0.5) * (1 - preference_beta)^(idle turns). Every turn advances
the node’s turn counter once; an edge that decayed to within 0.01 of neutral
is retired. The node’s text collects the usable preferences-section
session-context entries newer than its watermark, newest first, capped at
2000 characters on line boundaries. Turns are marked in memify_metadata
(preference_turn_counted, preference_weights_applied), so a re-run
counts and applies each once. Counts: turns_applied, edges_written,
edges_pruned, text_lines_added; nothing to fold is completed with the
reason no_changes. In Rust, preferences-section entries come from a session
search’s turn analysis (while auto_feedback is on with session caching), which
also records the rating a feedback row infers for the previous answer.
With personalization on, searches read these back for the requesting user in
the search’s single dataset: the decayed prefers weights scale the matched
nodes’ distances in graph triplet ranking (every graph completion type) and
re-rank RAG completion’s chunks (fetching wide_search_top_k candidates
first, then trimming to top_k), by at most personalization_influence;
graph and RAG completion also append the preference text (under
## What this user prefers) to the question and context. The hybrid search
uses them too: the weights re-rank its chunk lane, and its completion and
only_context preview carry the same preference guidance. On a session search
the preference lines instead join the Preferences section of the one
## Active session guidance block (budgets of 400 characters per section and
1200 in total), served after the question and context together with the
session’s stored guidance entries; there is no second block. An answered turn
records the ids of the guidance entries it served (used_session_context_ids),
so the next turn’s ratings of that guidance apply. Session history now goes
into the user prompt, ahead of the rendered template, rather than into the
system prompt. A RAG completion turn now records its chunks’ ids, so
ratings can weight chunks.
Locking
A run holds its dataset’s lock for its whole length, then a claim on every session it targets (session:{user_id}:{session_id}), and releases both once
it ends. Both wait, with no timeout: improves that share a dataset or a session
run one after the other, as pipeline runs on one dataset do, instead of the
later one skipping; runs on different datasets and sessions do not wait. The
dataset lock is one shared registry: cognee_core::DatasetLocks::global()
is the default of add (AddPipeline::new_with_algorithm; AddPipeline::new
takes no lock unless one is set), dataset-scoped deletes
(DeleteService) and improve (ImproveParams::dataset_locks, else the add
pipeline’s registry, else the global one), and cognify always
holds it for its run, so in one process those writers queue per dataset. A
host with its own registry wires it into the add pipeline, the delete service
and improve. The stages’ own adds and
cognifies re-enter the improve’s hold. A dataset that does not exist yet is
held by the id an add of it would create.
Global context index
global_context_index (on by default: the
improve.global_context_index setting, IMPROVE_GLOBAL_CONTEXT_INDEX, also
for the automatic improve after remember(); per call,
build_global_context_index in the Rust improve() API,
--build-global-context-index[=BOOL] in the CLI or buildGlobalContextIndex
in the bindings overrides it either way) builds the
dataset’s global context index with the default from_entities method,
which makes no LLM call: one GlobalContextSummary bucket per document
naming the entities it is distinctively about (tf·idf over the dataset’s
chunks, up to 20) and up to 40 relation lines among them, plus a root of
three lines (Topics:, Entities:, Documents:). Ids are uuid5
ids, so the text and ids match the index Python cognee builds on a shared
store byte for byte. Each run computes the
whole index, writes only the nodes whose text changed (graph node, vector in
GlobalContextSummary_text, summarized_in edge to the root and
summarizes edge to the document) and deletes the index nodes it did not
compute; the nodes’ ownership rows name the dataset, so deleting the
dataset removes them. Its count nodes_written is the nodes it wrote. The
opt-in LLM method (vector-placed buckets of TextSummary nodes, levels, LLM
bucket and root summaries) is
cognee_cognify::build_global_context_index with
GlobalContextIndexMethod::Llm; improve() does not use it. A graph or hybrid completion search with include_global_context_index
prepends the root and the global_context_index_top_k (3) nearest bucket
summaries to its context (World summary: / Relevant areas:). A
dataset-scoped search reads only its datasets’ root and summaries (their
dataset_id), so datasets sharing one graph and vector store never see each
other’s index. With no triplet found, the prelude alone is the context, and an
only_context search previews the prompt built from it.
global_context_index reports already_completed (no_writes_since_last_improve)
when no pipeline_runs row other than a search, a recall or a host’s
improve_pipeline record was written since it last ran. A delete records a
forget row on every dataset it touched, and an improve run whose stage may
have written to the graph (anything but the index itself, unless its write
counts say it wrote nothing) records one improve row, so both count as
changes. A run that did not end errored stamps the stage’s start in the run_info of
its improve operation row ({"global_context_index": {"status", "started_at"}}),
where Python cognee also keeps it; the newest such row is the watermark, and the row itself
is not a change. Without a relational database, a stamp, or on any error
reading them, the stage runs. (The graph_sync_checkpoints table that held the
stamps before is gone, and with it CheckpointStore and
ImproveParams::checkpoint_store; a migration moves existing stamps into improve rows, so
no stage reruns because of the upgrade.)
What is not ported yet is tracked in
not-implemented.md.
rustdoc: api::improve.
Related
Remember
Runs improve after storing
Sessions
Where session lessons live
Improve configuration
improve.* settings