Skip to main content

What is the improve operation

improve runs the self-improvement loop over a dataset. It turns session activity (ratings, Q&A, agent traces, preferences) into lasting graph memory and keeps dataset-wide structures such as the global context index up to date. remember runs it automatically unless you turn that off.

Stages

Runs the self-improvement loop over a dataset as nine gated stages, in this order: feedback_weights, persist_session_qa, persist_agent_traces, extract_agent_context, distill_sessions, update_user_preferences, build_truth_subspace, review_conflicts, global_context_index. The first seven read the given sessions and are skipped with no_session_ids without them; the last two work on the graph alone and run by default. Each stage reports a StageResult (status, reason, error, counts, duration_ms) on the returned ImproveResult (dataset_id, dataset_name, session_ids, stages, background, finished, error, memify_run), whose status summarises them (completed, errored, skipped, running). The earlier memify (triplet enrichment) and sync_graph_to_session stages are removed, and the old counters (stages_run, feedback_entries_processed, sessions_persisted, …) are now stage counts. Stages named in improve.stages_disabled (IMPROVE_STAGES_DISABLED, a comma-separated list) are skipped first, with disabled_by_config; an unknown name, or the fatal persist_session_qa, makes the configuration invalid and the run fails before any stage. feedback_alpha defaults to improve.feedback_alpha (0.1). The options that only scoped or fed the removed memify stage are gone too (node_name and the CLI --node-name flag, extraction_tasks, enrichment_tasks, data, and the bindings’ nodeName). ImproveResult still serializes the deprecated memify_run key, as {}. The bindings’ improve options accept datasetId (a UUID string): the improve targets that dataset, possibly one shared with the caller, instead of looking datasetName up among the caller’s own datasets.

Background runs

Background runs. cognee::api::improve_in_background runs improve in the background: it validates the configuration and feedback_alpha and resolves the dataset first — a name resolves to the caller’s dataset or is created for it, a dataset id that names no dataset (InvalidArgument; PermissionDenied when an ACL store is wired, which checks the grant first) or one the caller may not write (PermissionDenied; the ACL’s write grant, or ownership without an ACL) is refused — then spawns the run as a process-wide background task and returns the running result with the resolved dataset id and name. Wait for it with cognee::wait_for_background_tasks; every binding exposes the same wait (waitForBackgroundTasks in TS and Java, wait_for_background_tasks in Python, cg_sdk_wait_for_background_tasks in C). The wait takes an optional timeoutSeconds (timeout_seconds in Python), which cancels nothing when it elapses, and reports {finished, pending}: finished is false only when the timeout elapsed with work still running. An improve op with runInBackground: true uses this entry point. ImproveParams::run_in_background on improve() itself is only a label, for a host that dispatches the run on its own. The bindings’ improve options also accept buildGlobalContextIndex and reviewConflicts, which follow their improve.* setting when omitted (an explicit value wins either way), and buildTruthSubspace, which opts in to that stage.

Feedback weights

Besides an explicit feedback_score, a turn’s answer is rated implicitly by the latest kind = "feedback" session-context row that references it, and an implicit rating moves the graph elements at alpha * 0.5; an explicit score still wins. Each turn records which node and edge ids it already moved, so a retry moves only the rest; ids missing from the graph keep the turn pending, and a turn is marked done after 3 attempts. A graph or session-store error on one turn counts it as skipped instead of aborting the stage. Breaking: SessionQAEntry::memify_metadata (and SessionQAUpdate::memify_metadata) is now Option<HashMap<String, serde_json::Value>> instead of Option<HashMap<String, bool>> (TypeScript: Record<string, unknown> | null), so it holds the lists and numbers Python writes there; read a flag with Value::as_bool. The old bool-only decode dropped the whole map when it met one. For callers of cognee_cognify::memify, FeedbackApplyResult gained public fields (done, errored, elements_moved, elements_pruned, elements_failed), so a struct literal no longer compiles, and FeedbackError gained the MetadataNotUpdated and InvalidSessionIds variants.

Failures and skipped stages

The run fails open — a failed stage is recorded and the next one runs — except that a failure in persist_session_qa (losing session Q&A would be data loss) stops the run with ApiError::ImproveAborted carrying the partial result, and an exhausted LLM budget skips every later stage with budget_exhausted. In the bindings that abort is SdkError::ImproveAborted (code IMPROVE_ABORTED): Python’s CogneeImproveAbortedError.improve_result, JavaScript’s ImproveAbortedError.improveResult; C (CG_ERR_RUNTIME) and Java append improve_result: <json> to the message. The stages that draft text with an LLM are skipped with no_llm_configured when none is configured. review_conflicts and global_context_index run by default (improve.review_conflicts, improve.global_context_index; a per-call review_conflicts / build_global_context_index wins either way) and report turned_off when off. review_conflicts is not ported yet and always reports a skip reason (turned_off, then no_llm_configured, then edge_evidence_disabled while cognify.edge_evidence_enabled is off, then not_ported). Breaking: both stages used to be opt-in, and a switched-off one used to report opt_in_disabled (build_truth_subspace still does). ImproveParams::build_global_context_index is now Option<bool>, and ImproveParams / OwnedImproveParams gained review_conflicts: Option<bool>, so a struct literal needs both (None follows the setting). Set IMPROVE_GLOBAL_CONTEXT_INDEX=false and IMPROVE_REVIEW_CONFLICTS=false to keep the old behavior.

Agent context lessons

extract_agent_context runs while automatic session context is on (auto_feedback with session caching; otherwise auto_feedback_disabled) and an LLM is configured. Per session it reads the trace steps newer than the session’s extraction watermark (plus 3 already-processed ones, at most 40), shows the model the session’s existing agent lessons, and stores at most 5 new lessons as agent-profile session-context entries (sections tool_rules, workflow_state, success_patterns, failure_lessons, environment_facts). A lesson is stored only with confidence ≥ 0.75, cut to 280 characters; one whose normalized text an entry of the same section already has is not stored again. The watermark advances only after the session’s extraction succeeded. Counts: lessons, sessions_failed; the stage is errored when any session failed, and the other sessions still run. Error texts shown to the model are redacted (tokens, secrets, UUIDs, long numbers).

User preferences

update_user_preferences runs while personalization_enabled is on. It folds the sessions’ turns into one UserPreference node per (user, dataset), an internal node (never embedded, never returned by a search) in the user_preferences node set, and weighted prefers edges from it to the graph nodes the turns’ answers used (used_graph_element_ids.node_ids). A turn’s rating is its feedback_score, else the rating a feedback row inferred for it; a rating r pulls each node’s weight toward (r - 1) / 4 by preference_alpha, from its decayed value 0.5 + (w - 0.5) * (1 - preference_beta)^(idle turns). Every turn advances the node’s turn counter once; an edge that decayed to within 0.01 of neutral is retired. The node’s text collects the usable preferences-section session-context entries newer than its watermark, newest first, capped at 2000 characters on line boundaries. Turns are marked in memify_metadata (preference_turn_counted, preference_weights_applied), so a re-run counts and applies each once. Counts: turns_applied, edges_written, edges_pruned, text_lines_added; nothing to fold is completed with the reason no_changes. In Rust, preferences-section entries come from a session search’s turn analysis (while auto_feedback is on with session caching), which also records the rating a feedback row infers for the previous answer. With personalization on, searches read these back for the requesting user in the search’s single dataset: the decayed prefers weights scale the matched nodes’ distances in graph triplet ranking (every graph completion type) and re-rank RAG completion’s chunks (fetching wide_search_top_k candidates first, then trimming to top_k), by at most personalization_influence; graph and RAG completion also append the preference text (under ## What this user prefers) to the question and context. The hybrid search uses them too: the weights re-rank its chunk lane, and its completion and only_context preview carry the same preference guidance. On a session search the preference lines instead join the Preferences section of the one ## Active session guidance block (budgets of 400 characters per section and 1200 in total), served after the question and context together with the session’s stored guidance entries; there is no second block. An answered turn records the ids of the guidance entries it served (used_session_context_ids), so the next turn’s ratings of that guidance apply. Session history now goes into the user prompt, ahead of the rendered template, rather than into the system prompt. A RAG completion turn now records its chunks’ ids, so ratings can weight chunks.

Locking

A run holds its dataset’s lock for its whole length, then a claim on every session it targets (session:{user_id}:{session_id}), and releases both once it ends. Both wait, with no timeout: improves that share a dataset or a session run one after the other, as pipeline runs on one dataset do, instead of the later one skipping; runs on different datasets and sessions do not wait. The dataset lock is one shared registry: cognee_core::DatasetLocks::global() is the default of add (AddPipeline::new_with_algorithm; AddPipeline::new takes no lock unless one is set), dataset-scoped deletes (DeleteService) and improve (ImproveParams::dataset_locks, else the add pipeline’s registry, else the global one), and cognify always holds it for its run, so in one process those writers queue per dataset. A host with its own registry wires it into the add pipeline, the delete service and improve. The stages’ own adds and cognifies re-enter the improve’s hold. A dataset that does not exist yet is held by the id an add of it would create.

Global context index

global_context_index (on by default: the improve.global_context_index setting, IMPROVE_GLOBAL_CONTEXT_INDEX, also for the automatic improve after remember(); per call, build_global_context_index in the Rust improve() API, --build-global-context-index[=BOOL] in the CLI or buildGlobalContextIndex in the bindings overrides it either way) builds the dataset’s global context index with the default from_entities method, which makes no LLM call: one GlobalContextSummary bucket per document naming the entities it is distinctively about (tf·idf over the dataset’s chunks, up to 20) and up to 40 relation lines among them, plus a root of three lines (Topics:, Entities:, Documents:). Ids are uuid5 ids, so the text and ids match the index Python cognee builds on a shared store byte for byte. Each run computes the whole index, writes only the nodes whose text changed (graph node, vector in GlobalContextSummary_text, summarized_in edge to the root and summarizes edge to the document) and deletes the index nodes it did not compute; the nodes’ ownership rows name the dataset, so deleting the dataset removes them. Its count nodes_written is the nodes it wrote. The opt-in LLM method (vector-placed buckets of TextSummary nodes, levels, LLM bucket and root summaries) is cognee_cognify::build_global_context_index with GlobalContextIndexMethod::Llm; improve() does not use it. A graph or hybrid completion search with include_global_context_index prepends the root and the global_context_index_top_k (3) nearest bucket summaries to its context (World summary: / Relevant areas:). A dataset-scoped search reads only its datasets’ root and summaries (their dataset_id), so datasets sharing one graph and vector store never see each other’s index. With no triplet found, the prelude alone is the context, and an only_context search previews the prompt built from it. global_context_index reports already_completed (no_writes_since_last_improve) when no pipeline_runs row other than a search, a recall or a host’s improve_pipeline record was written since it last ran. A delete records a forget row on every dataset it touched, and an improve run whose stage may have written to the graph (anything but the index itself, unless its write counts say it wrote nothing) records one improve row, so both count as changes. A run that did not end errored stamps the stage’s start in the run_info of its improve operation row ({"global_context_index": {"status", "started_at"}}), where Python cognee also keeps it; the newest such row is the watermark, and the row itself is not a change. Without a relational database, a stamp, or on any error reading them, the stage runs. (The graph_sync_checkpoints table that held the stamps before is gone, and with it CheckpointStore and ImproveParams::checkpoint_store; a migration moves existing stamps into improve rows, so no stage reruns because of the upgrade.) What is not ported yet is tracked in not-implemented.md. rustdoc: api::improve.

Remember

Runs improve after storing

Sessions

Where session lessons live

Improve configuration

improve.* settings