Skip to main content
Cognee ships a self-contained evaluation framework at cognee/eval_framework/ that lets you benchmark retrieval quality on standard multi-hop QA datasets, compare different search strategies, and inspect results in an interactive HTML dashboard — all without any third-party evaluation service.
This page covers the built-in evaluation framework. For the DeepEval integration (LLM-as-a-judge scoring), see Evaluation with DeepEval.

Overview

The pipeline has four sequential stages:

Quick Start

Before you start, complete Quickstart, make sure your corpus is already processed and indexed or let the corpus builder create it from scratch, and set LLM_API_KEY.

Basic Usage

This minimal example runs the built-in evaluation pipeline and writes an HTML dashboard:
This runs all four stages in sequence and writes my_dashboard.html to the current directory.

What just happened

  • Benchmark setup — EvalConfig defines which benchmark to use, how many samples to ingest, and which retrieval and evaluation engines to run.
  • Full evaluation run — main(params=config.to_dict()) executes corpus building, answer generation, evaluation, and dashboard generation in one flow.
  • Output artifacts — the run produces metrics files and an HTML dashboard you can open locally to inspect results.
Open my_dashboard.html in any browser. You will see:
  • Distribution histograms — score distributions per metric (10 bins).
  • Confidence-interval bar chart — mean score ± 95% CI for each metric.
  • Details table — per-question breakdown with generated answer, golden answer, retrieved context, score, and LLM rationale.
For containerized runs, a Dockerfile is included at cognee/eval_framework/Dockerfile.

LoCoMo Benchmark

The LoCoMo long-conversation memory benchmark has its own runner. It does not go through run_eval or BENCHMARK. Instead it drives cognee the way a user builds a memory, then grades the answers with a separate LLM judge:
How a run works:
  • Isolation. Each conversation runs in its own child processes. Their DATA_ROOT_DIRECTORY and SYSTEM_ROOT_DIRECTORY live under <run-dir>/conv_NN/, so conversations never see each other’s memory and every memory stays on disk for re-querying. The children run with ENABLE_BACKEND_ACCESS_CONTROL=false, CACHING=true and AUTO_FEEDBACK=false.
  • Ingest. Each conversation becomes one overview document (speakers and session timeline) plus one document per window of consecutive turns (--window-turns, default 6), each with a dated session header. All of them go through a single remember() call, which runs add + cognify + improve. The four ingestion flags in the table below change how the memory is built; all of them are off by default. They only apply when ingestion runs: a conversation whose ingestion already completed is skipped unless you pass --force-ingest.
  • Answer. By default every question uses hybrid_completion retrieval with chunks_top_k, entities_top_k and facts_top_k set to 20 and max_edges_per_entity set to 10 (configs/locomo_hybrid_20_20_20.json). Pass --config-json-path cognee/eval_framework/locomo/configs/locomo_hybrid_5_5_5_gci.json to use 5/5/5 instead (max_edges_per_entity stays 10), with the global context index added to the context (include_global_context_index, global_context_index_top_k 3). The checked-in top-k 5 reference run used the same prompts and settings that the four ingestion flags now apply, as in the example above. Every question gets the same answer prompt, and the dataset’s category label is never shown to the model.
  • Judge. The answers are graded by the llm_judge metric (see Available Metrics below).
  • Aggregate. The results are summarized in <run-dir>/locomo_summary.json and locomo_summary.md. The total leaves out open-domain questions, which appear only in the per-category numbers. With gold exclusions, locomo_summary.md reports only the numbers without the corrupted questions; the JSON keeps the numbers on all questions too.
LLM_API_KEY is required. It is read from the environment or .env, or you can pass it with --llm-api-key.
A full run is LLM-cost-intensive. Two reference runs are checked into the repo under cognee/eval_framework/locomo/report_artifacts/, one per folder, each with its per-question results, its summary and its exact configuration in run_config.json. See Accuracy Benchmarks for their setup and scores.

Token Usage Analysis

The framework ships a standalone CLI utility at cognee/eval_framework/token_usage_analysis/ that estimates the token cost of Cognee persistent memory versus full-context prompting. It chunks an input text, measures the real ingestion token usage of running a few representative chunks through Cognee, and reports the break-even query count — after how many repeated queries full-context prompting has spent more tokens than Cognee’s one-time ingestion plus per-query recall. It can optionally write cumulative-cost plots. Install the eval dependencies and run the tool from its own directory:
The script loads the repo-root .env, so it must contain a working LLM_PROVIDER, LLM_MODEL, and API key. If --llm-models is omitted, the configured LLM_MODEL is used. It always writes a JSON report; --plot additionally writes the cumulative-cost figure. You must supply exactly one input form (--file, --dir, or --text). Key options:
--plot requires matplotlib, which is included in the evals extra (installed by uv sync --dev --all-extras).
For the full cost model, sample corpora, and precomputed results, see the in-repo cognee/eval_framework/token_usage_analysis/README.md and the results/chunk_4095/ and results/chunk_8191/ folders.

Code

Each stage exposes a standalone async function you can call from your own scripts:
You can restrict which instances are evaluated using INSTANCE_FILTER:
A typical workflow for comparing two retrieval strategies:
Open both HTML files side-by-side to compare F1 and exact-match scores across retrieval strategies.

Further details

All options are read from an .env file (or environment variables) via a Pydantic BaseSettings class (EvalConfig).
All metrics return a {"score": float | None, "reason": str} dict. Aggregate statistics include mean and a 95% confidence interval computed with 10 000 bootstrap samples.
The TASK_GETTER_TYPE variable controls how each corpus document is processed during the corpus-building stage: