Skip to main content
Cognee publishes its memory accuracy results on two long-term memory benchmarks. It scores 0.79 on BEAM 100K and about 91% on LoCoMo. Each result has a technical report, and its answers, judge verdicts and configs are published in the Cognee repository, so you can check every number without rerunning anything, or rerun it yourself.

​
Results

​
BEAM

BEAM tests memory over long, multi-session user–assistant conversations, with relevant evidence scattered across many turns. Its questions cover ten abilities: information extraction, multi-hop reasoning, knowledge updates, temporal reasoning, summarization, preference following, abstention, contradiction resolution, event ordering and instruction following. An LLM judge scores each rubric criterion 0, 0.5 or 1, and a question’s score is the average over its criteria. Method. Each conversation is ingested with add and cognify, the two lower-level steps that remember() runs, so the benchmark can set its own chunking and session handling: one document per conversation batch and one chunk per user–assistant turn, plus the global context index and session memory (the 100K configuration builds the index but doesn’t query it). Questions are answered with hybrid_completion retrieval, which combines chunk, graph and summary search. Answer prompts are chosen by BEAM question type, following the convention of other published BEAM results.
  • 100K is the primary result. Retrieval settings (chunk top-k 20, entity top-k 20) and prompts were tuned on two conversations, checked on two more, and scored on a held-out conversation.
  • 10M is an exploratory scale check. It routes each question type to different retrieval settings, which were selected on the same 20 questions that were scored, so it is not a held-out estimate.

​
LoCoMo

LoCoMo tests memory over ten long conversations between two people, with single-hop, multi-hop, temporal, open-domain and adversarial questions. Method. Each conversation is ingested through remember() as dated windows of 6 turns, one chunk per window, with Cognee’s standard extraction and summary prompts. Every question uses hybrid_completion retrieval with 20 passages, 20 entities and 20 facts, and the same answer prompt; the question’s category is never shown to the answering model. Answers come from gpt-5-mini and are graded by a gpt-5.1 judge. The runs used Cognee 1.5.4 with Ladybug, LanceDB and SQLite. Two answer passes over the same memories scored 90.7% (1,598 of 1,762) and 91.3% (1,608 of 1,762). By question type, the 91.3% pass scored: The total leaves out the 96 open-domain questions, which need general knowledge, and the 143 questions whose gold answer an audit flags, 224 questions in all, since 15 are in both groups. The 143 flagged questions are listed with each audit’s reason in report_artifacts/gold_exclusions.json; open-domain questions are left out by category. At the hybrid retriever’s own default depth (5 passages, 5 entities and 5 facts, below recall()’s default of 10 each), with the global context index, resolved dates and adapted extraction and summary prompts, one run scored 89.7% (1,580 of 1,761 scored questions).

​
Limitations

  • BEAM scores come from one conversation per scale, 20 questions each. Per-ability scores rest on two questions each and are directional.
  • BEAM is fully public. Its questions and rubrics informed the prompts and retrieval settings, and there is no hidden test set. The 100K split reduces this risk; the 10M result has no such split.
  • The 100K and 10M scores are not a controlled comparison. The conversations, ingestion models and retrieval selection differ between them.
  • LoCoMo is not held out. Its ten conversations were used to develop the setup.
  • LLM judges vary between runs. Read small differences between scores with caution.
  • The 10M ingestion can’t be rerun exactly as published. It ran in Cognee’s distributed mode, and its BEAM-specific orchestration is not in the public code.

​
Reproduce the results

Run the benchmarks with Cognee’s built-in evaluation framework. Both runs are LLM-cost-intensive. For BEAM 100K, preprocess and ingest the conversations, run the evaluation with the packaged config, then aggregate the runs into the mean score and standard deviation:
Without --execute-compressions, preprocessing only reports what it would compress. For LoCoMo, one command ingests, answers and grades all ten conversations:
See Built-in Evaluation Framework for setup, options and how to benchmark retrieval on your own data.

​
Next steps

Evaluation framework

Run these benchmarks and your own

Token efficiency

What memory costs per call, measured