Let an Agent Rewrite Its Own Weak Skill

One of the skills your agent loads on every pull request quietly does the wrong job — it grades the reviewer’s tone and never looks at the code. Nothing errors, so nobody notices until someone reads a transcript, and then a person has to sit down and edit the playbook by hand.

What You’ll Build

Three SKILL.md playbooks in a bundled skills/ folder — a diff explainer, a deliberately flawed PR-comment evaluator that judges politeness only, and a critic that grades the other two — are remembered into one dataset as skills. A single agentic run then loads all three in order against a two-line diff that drops a None check and a reviewer comment that says nothing more than “This is bad”, and returns JSON naming which skill failed, the score it deserves, and the instruction it is missing. That verdict is recorded as a skill run, which drafts a skill-improvement proposal; applying the proposal rewrites the flawed skill’s procedure in the graph, and the script prints the skill’s text before and after so you can read the edit the loop made for you. The complete runnable script is examples/demos/feedback/skill_feedback_loop/skill_feedback_loop_demo.py — this page walks through its key moments rather than reproducing it.

Features in Play

Remember — content_type="skills" turns a folder of SKILL.md files into skill nodes in one dataset, which is what makes them loadable by name later
Recall — one AGENTIC_COMPLETION call is the agent run under test: it loads the three skills, does the work, and returns the JSON verdict the rest of the script acts on
Sessions — a single session_id spans the agentic run and the record written about it, so the evaluation and the improvement belong to the same episode
Skill-improvement proposals — a skill-run entry with a low success_score drafts a proposal; applying it by id is what actually rewrites the skill’s procedure

What to Expect

The excerpts below are from a real run, trimmed. The script prints five numbered lines to stdout; two LiteLLM notice blocks that appeared between the first and second line during the agentic loop are elided here. The score, the feedback wording, and the text of the rewritten skill are all live LLM output and vary from run to run. Three skills go in, and the flawed one comes out with a failing score. The first line confirms the folder walk found all three SKILL.md files. The second is the agentic run’s verdict: it singled out pr-comment-evaluator, as the task and the critic skill both steer it to, and scored it well under the 0.30 ceiling the critic sets for a tone-only evaluation.

1. remember -> stored 3 skills
...
2. evaluation -> pr-comment-evaluator scored 0.20

A proposal is drafted and applied in one pass. The third line names the proposal that the skill-run record drafted and that the script then applied by id — the same id you could inspect with GET /api/v1/proposals/{proposal_id} before applying it.

3. improve proposal -> applied proposal_id=5efe8e2e-ed96-4d2a-83d5-fbcfa03128cb

The skill’s procedure changes in place. The last two lines are the same skill read back by name on either side of the rewrite, flattened onto one line each. Before, it is the one-sentence playbook that refuses to look at code. After, it is a procedure that takes the diff and the comment as inputs, parses the diff into hunks, and lists “removed None-checks” among the patterns to detect — then scores the comment on a separate technical_score next to tone_score. The full rewritten procedure runs to several hundred words; only its opening is shown.

4. skill before -> # pr-comment-evaluator Only judge whether the PR comment sounds polite. Do not discuss code risk or technical correctness.
5. skill after -> # pr-comment-evaluator Inputs provided to the skill: `diff` (unified diff or patch text) and `comment` (the reviewer comment text). Behavioral constraints: - Do not mutate state or modify any files. Do not run commands that change the repository. - Operate only on the provided `diff` and `comment` inputs. Procedure (follow exactly): 1. Parse the `diff` into changed hunks. For each hunk, extract file path, line ranges, removed lines, and added lines. 2. For each hunk, identify concrete code changes that could introduce a bug or behavior change. ...

Before You Start

Complete Quickstart to understand basic operations
Ensure you have LLM Providers configured — the agentic pass is a live tool-calling loop and the proposal is LLM-drafted, so the score, the feedback text, and the rewritten skill differ from run to run
Set RECALL_WARMUP_SHORTCIRCUIT=false in your environment before running: skills ingestion writes skill nodes without logging a graph build, so recall’s warm-up guard reads the dataset as empty and returns a warming-up marker instead of an answer, which the script cannot parse. See Recall warm-up for the variable
Run it from a checkout of the cognee repo: the script reads its three playbooks from the skills/ folder and its diff and reviewer comment from the data/ folder next to it, so a copy-pasted script has nothing to ingest
The run opens with cognee.forget(everything=True) and does not redirect cognee’s storage roots, so it clears whatever memory the current configuration points at — run it against a scratch instance rather than storage you want to keep

How It Works

Stage 1: Ingest the Skills Folder

    remembered = await cognee.remember(
        str(SKILLS_ROOT),
        dataset_name=DATASET_NAME,
        content_type="skills",
    )
    print(f"1. remember -> stored {remembered.items_processed} skills")
    user, datasets = await resolve_authorized_user_datasets(UUID(remembered.dataset_id))
    dataset = datasets[0]

content_type="skills" walks the skills/ directory for SKILL.md files and stores each one as a skill scoped to toy-skill-feedback-loop — skills are always dataset-scoped, and the agentic run later needs exactly one dataset to work in. The user and dataset resolved from the returned dataset id are what the script uses to read and rewrite a skill’s body directly at the end.

Stage 2: Write the Task That Exposes the Flaw

TASK_TEMPLATE = """Use the skills in this exact order:
1. Load diff-risk-explainer and explain the concrete bug risk in the diff.
2. Load pr-comment-evaluator and evaluate the reviewer comment.
3. Load skill-feedback-writer and decide which skill needs a better instruction.

The skills are plain instructions. After you load each skill, do the work yourself.
The pr-comment-evaluator skill is intentionally flawed because it judges tone only. If its output
does not compare the reviewer comment against the concrete bug risk, target pr-comment-evaluator
and give a score of 0.30 or lower.

Return only JSON with keys:
diff_risk_summary, comment_evaluation, skill_to_improve, score, feedback, missing_instruction.

Diff:
{diff_text}

Reviewer comment:
{comment_text}
"""

The task names the order the three skills run in and pins the output to a fixed set of JSON keys, so the script can read skill_to_improve and score out of a free-text answer. It also tells the agent what a failure looks like: pr-comment-evaluator is written to judge tone only, and an evaluation that never mentions the diff’s dropped None check earns 0.30 or lower. The bundled skill-feedback-writer playbook carries the same rule, so the low score comes from the skills as much as from the prompt.

Stage 3: Run the Three Skills in One Agentic Pass

    answer = await cognee.recall(
        task,
        query_type=SearchType.AGENTIC_COMPLETION,
        datasets=DATASET_NAME,
        retriever_specific_config={
            "skills": SKILL_NAMES,
            "max_iter": 6,
        },
        session_id=SESSION_ID,
    )
    feedback = parse_json_answer(answer)
    score = score_from_feedback(feedback)
    skill_to_improve = str(feedback["skill_to_improve"])

AGENTIC_COMPLETION hands the agent the three skill names and lets it pull each procedure in with the load_skill tool, up to six tool rounds. The agent sees only names and descriptions up front — the bodies arrive when it asks for them — which is why the run is a fair test of the playbooks rather than of one long prompt. datasets=DATASET_NAME keeps the scope to a single dataset, which AGENTIC_COMPLETION requires; the script’s own helpers then pull the JSON out of the answer and clamp the score into 0.0–1.0.

Stage 4: Record the Weak Run and Draft a Proposal

    proposal_result = await cognee.remember(
        SkillRunEntry(
            selected_skill_id=skill_to_improve,
            task_text=task,
            result_summary=feedback_summary(feedback),
            success_score=score,
            feedback=-1.0 if score < 0.7 else 1.0,
        ),
        dataset_name=DATASET_NAME,
        session_id=SESSION_ID,
        skill_improvement={
            "skill_name": skill_to_improve,
            "apply": False,
            "score_threshold": 0.9,
        },
    )
    proposal_id = next(
        item["proposal_id"]
        for item in proposal_result.items
        if item.get("kind") == "skill_improvement_proposal"
    )

The verdict goes back into memory as a SkillRunEntry: which skill was used, the task it was used for, the critic’s feedback and missing instruction as the result summary, and the score as both a raw success_score and a -1.0/1.0 signal. skill_improvement is what turns that record into a rewrite — score_threshold: 0.9 means any run scoring below 0.9 is bad enough to draft against, and apply: False stops at the draft so the proposal can be inspected before it changes anything. The proposal’s id comes back among the entry’s result items.

Stage 5: Apply the Proposal and Read the Skill Back

    before = await skill_body(skill_to_improve, dataset, user)
    await improve_skill(
        skill_to_improve,
        dataset=dataset,
        user=user,
        proposal_id=proposal_id,
        apply=True,
    )
    after = await skill_body(skill_to_improve, dataset, user)

Reading the skill’s procedure on either side of improve_skill(..., apply=True) is what makes the loop visible: the same lookup by name, before and after the proposal is applied, over the same dataset. The before text is the flawed instruction that only judges politeness; the after text is the drafted replacement, which the critic’s missing_instruction asked to compare the reviewer comment against the concrete bug risk. Nothing here re-ingests the folder — the skill node is edited in place, so the next agentic run loads the new text.

Run It

uv run python examples/demos/feedback/skill_feedback_loop/skill_feedback_loop_demo.py

Recall

How AGENTIC_COMPLETION and the other search types are chosen and scoped.

Remember

The operation that stores the skills folder, and everything else, in memory.

Feedback System

Rating answers in a session, the other half of cognee’s feedback loop.

Tune How Strongly Ratings Steer an Answer

The sibling demo, where feedback moves retrieval ranking instead of a playbook.

Getting Started

Core Concepts

Setup Configuration

Guides

Examples

CLI

Rust SDK

TypeScript SDK

OSS

Let an Agent Rewrite Its Own Weak Skill

What You’ll Build

Features in Play

What to Expect

Before You Start

How It Works

Stage 1: Ingest the Skills Folder

Stage 2: Write the Task That Exposes the Flaw

Stage 3: Run the Three Skills in One Agentic Pass

Stage 4: Record the Weak Run and Draft a Proposal

Stage 5: Apply the Proposal and Read the Skill Back

Run It

Recall

Remember

Feedback System

Tune How Strongly Ratings Steer an Answer

​What You’ll Build

​Features in Play

​What to Expect

​Before You Start

​How It Works

​Stage 1: Ingest the Skills Folder

​Stage 2: Write the Task That Exposes the Flaw

​Stage 3: Run the Three Skills in One Agentic Pass

​Stage 4: Record the Weak Run and Draft a Proposal

​Stage 5: Apply the Proposal and Read the Skill Back

​Run It

Recall

Remember

Feedback System

Tune How Strongly Ratings Steer an Answer

What You’ll Build

Features in Play

What to Expect

Before You Start

How It Works

Stage 1: Ingest the Skills Folder

Stage 2: Write the Task That Exposes the Flaw

Stage 3: Run the Three Skills in One Agentic Pass

Stage 4: Record the Weak Run and Draft a Proposal

Stage 5: Apply the Proposal and Read the Skill Back

Run It