cognee/eval_framework/ that lets you benchmark retrieval quality on standard multi-hop QA datasets, compare different search strategies, and inspect results in an interactive HTML dashboard — all without any third-party evaluation service.
This page covers the built-in evaluation framework. For the DeepEval integration (LLM-as-a-judge scoring), see Evaluation with DeepEval.
Overview
The pipeline has four sequential stages:Quick Start
Before you start, complete Quickstart, make sure your corpus is already processed and indexed or let the corpus builder create it from scratch, and setLLM_API_KEY.
Basic Usage
This minimal example runs the built-in evaluation pipeline and writes an HTML dashboard:my_dashboard.html to the current directory.
What just happened
- Benchmark setup —
EvalConfigdefines which benchmark to use, how many samples to ingest, and which retrieval and evaluation engines to run. - Full evaluation run —
main(params=config.to_dict())executes corpus building, answer generation, evaluation, and dashboard generation in one flow. - Output artifacts — the run produces metrics files and an HTML dashboard you can open locally to inspect results.
my_dashboard.html in any browser. You will see:
- Distribution histograms — score distributions per metric (10 bins).
- Confidence-interval bar chart — mean score ± 95% CI for each metric.
- Details table — per-question breakdown with generated answer, golden answer, retrieved context, score, and LLM rationale.
For containerized runs, a
Dockerfile is included at cognee/eval_framework/Dockerfile.LoCoMo Benchmark
The LoCoMo long-conversation memory benchmark has its own runner. It does not go throughrun_eval or BENCHMARK. Instead it drives cognee the way a user builds a memory, then grades the answers with a separate LLM judge:
- Isolation. Each conversation runs in its own child processes. Their
DATA_ROOT_DIRECTORYandSYSTEM_ROOT_DIRECTORYlive under<run-dir>/conv_NN/, so conversations never see each other’s memory and every memory stays on disk for re-querying. The children run withENABLE_BACKEND_ACCESS_CONTROL=false,CACHING=trueandAUTO_FEEDBACK=false. - Ingest. Each conversation becomes one overview document (speakers and session timeline) plus one document per window of consecutive turns (
--window-turns, default 6), each with a dated session header. All of them go through a singleremember()call, which runs add + cognify + improve. The four ingestion flags in the table below change how the memory is built; all of them are off by default. They only apply when ingestion runs: a conversation whose ingestion already completed is skipped unless you pass--force-ingest. - Answer. By default every question uses
hybrid_completionretrieval withchunks_top_k,entities_top_kandfacts_top_kset to 20 andmax_edges_per_entityset to 10 (configs/locomo_hybrid_20_20_20.json). Pass--config-json-path cognee/eval_framework/locomo/configs/locomo_hybrid_5_5_5_gci.jsonto use 5/5/5 instead (max_edges_per_entitystays 10), with the global context index added to the context (include_global_context_index,global_context_index_top_k3). The checked-in top-k 5 reference run used the same prompts and settings that the four ingestion flags now apply, as in the example above. Every question gets the same answer prompt, and the dataset’s category label is never shown to the model. - Judge. The answers are graded by the
llm_judgemetric (see Available Metrics below). - Aggregate. The results are summarized in
<run-dir>/locomo_summary.jsonandlocomo_summary.md. The total leaves out open-domain questions, which appear only in the per-category numbers. With gold exclusions,locomo_summary.mdreports only the numbers without the corrupted questions; the JSON keeps the numbers on all questions too.
LLM_API_KEY is required. It is read from the environment or .env, or you can pass it with --llm-api-key.
A full run is LLM-cost-intensive. Two reference runs are checked into the repo under
cognee/eval_framework/locomo/report_artifacts/, one per folder, each with its per-question results, its summary and its exact configuration in run_config.json. See Accuracy Benchmarks for their setup and scores.Token Usage Analysis
The framework ships a standalone CLI utility atcognee/eval_framework/token_usage_analysis/ that estimates the token cost of Cognee persistent memory versus full-context prompting. It chunks an input text, measures the real ingestion token usage of running a few representative chunks through Cognee, and reports the break-even query count — after how many repeated queries full-context prompting has spent more tokens than Cognee’s one-time ingestion plus per-query recall. It can optionally write cumulative-cost plots.
Install the eval dependencies and run the tool from its own directory:
.env, so it must contain a working LLM_PROVIDER, LLM_MODEL, and API key. If --llm-models is omitted, the configured LLM_MODEL is used. It always writes a JSON report; --plot additionally writes the cumulative-cost figure.
You must supply exactly one input form (--file, --dir, or --text). Key options:
--plot requires matplotlib, which is included in the evals extra (installed by uv sync --dev --all-extras).cognee/eval_framework/token_usage_analysis/README.md and the results/chunk_4095/ and results/chunk_8191/ folders.
Code
Running Individual Stages
Running Individual Stages
Each stage exposes a standalone async function you can call from your own scripts:
Filtering Benchmark Instances
Filtering Benchmark Instances
You can restrict which instances are evaluated using
INSTANCE_FILTER:Comparing Search Strategies
Comparing Search Strategies
A typical workflow for comparing two retrieval strategies:Open both HTML files side-by-side to compare F1 and exact-match scores across retrieval strategies.
Further details
All options are read from an.env file (or environment variables) via a Pydantic BaseSettings class (EvalConfig).
Corpus Builder
Corpus Builder
Answer Generation
Answer Generation
Evaluation
Evaluation
Dashboard
Dashboard
Supported Benchmarks
Supported Benchmarks
Available Metrics
Available Metrics
All metrics return a
{"score": float | None, "reason": str} dict. Aggregate statistics include mean and a 95% confidence interval computed with 10 000 bootstrap samples.Pipeline Strategies
Pipeline Strategies
The
TASK_GETTER_TYPE variable controls how each corpus document is processed during the corpus-building stage:QA Engines
QA Engines