> ## Documentation Index > Fetch the complete documentation index at: https://docs.cognee.ai/llms.txt > Use this file to discover all available pages before exploring further. # Folder Presort > Scan a messy folder for junk, duplicates, and personal data before ingesting it as datasets A minimal guide to pre-organizing a folder before it reaches your graph. Presort scans a directory without touching the files on disk, reports what is junk, duplicated, versioned, personal, or already in cognee, and then ingests the groups you approve — one dataset per group. ## Before You Start * Complete [Quickstart](/getting-started/quickstart) to understand basic operations * Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — the analyze phase is deterministic, but the apply phase runs cognify * Read [Datasets](/core-concepts/further-concepts/datasets) — presort ingests each proposed group into its own dataset * Set `PRESORT_FOLDER` to the folder you want to scan, or let the script use `~/Downloads` * Name that folder as a permitted scan root before running the script — the defaults are the working directory, the temp directory, and cognee's own storage, so a folder under `$HOME` is always outside them: `export COGNEE_ALLOWED_LOCAL_FILE_ROOTS="$HOME/Downloads"`. See [Presort scan roots](/setup-configuration/security#presort-scan-roots) ## Code in Action ```python theme={null} import asyncio import os from pathlib import Path import cognee FOLDER = os.environ.get("PRESORT_FOLDER", str(Path.home() / "Downloads")) async def main(): # Phase 1: analyze. Returns a PresortReport (also auto-saved under # cognee's system directory as .presort.json). report = await cognee.remember(FOLDER, dry_run="presort") summary = report.summary() print(f"Scanned {summary['files']} files ({summary['junk']} junk skipped)") print(f"Already in cognee: {summary['cognee_status']}") print( f"Duplicate clusters: {summary['duplicate_clusters']} ({summary['wasted_bytes']} wasted bytes)" ) print(f"Potential personal data: {summary['pii_findings']} findings") for group in report.groups: print( f" group {group.name!r} -> dataset {group.dataset_name!r} ({len(group.file_paths)} files)" ) # Review/adjust the apply decisions on the report itself. report.exclude_pii = True # keep files with personal data out of the graph report.skip_duplicates = True # ingest one copy per duplicate cluster report.apply_groups = [group.name for group in report.groups if group.kind != "code_project"] # Phase 2: apply. One dataset per proposed group; re-running is idempotent # (already-cognified content is skipped by incremental loading). results = await cognee.remember(report) for dataset_name, result in results.items(): print(f"ingested dataset {dataset_name!r}: {result}") # The presorted data is now queryable per dataset. answers = await cognee.recall("What documents do I have?", datasets=list(results)) for answer in answers: print(answer) if __name__ == "__main__": asyncio.run(main()) ``` The complete runnable script is on GitHub: [`examples/guides/presort_downloads.py`](https://github.com/topoteretes/cognee/blob/dev/examples/guides/presort_downloads.py). ## What Just Happened ### Step 1: Scan the Folder ```python theme={null} FOLDER = os.environ.get("PRESORT_FOLDER", str(Path.home() / "Downloads")) # Phase 1: analyze. Returns a PresortReport (also auto-saved under # cognee's system directory as .presort.json). report = await cognee.remember(FOLDER, dry_run="presort") ``` `dry_run="presort"` turns `remember()` into an analyzer: it walks the folder, reads samples of the files, and returns a `PresortReport` instead of ingesting anything. Nothing on disk is moved, renamed, or deleted. The report is also saved to `/presort/.presort.json`, so the analyze result survives a failed apply — `report.report_path` holds the exact location, or `None` when `SYSTEM_ROOT_DIRECTORY` is unset or on S3 and the report was not persisted. ### Step 2: Review the Report ```python theme={null} summary = report.summary() print(f"Scanned {summary['files']} files ({summary['junk']} junk skipped)") print(f"Already in cognee: {summary['cognee_status']}") print( f"Duplicate clusters: {summary['duplicate_clusters']} ({summary['wasted_bytes']} wasted bytes)" ) print(f"Potential personal data: {summary['pii_findings']} findings") for group in report.groups: print( f" group {group.name!r} -> dataset {group.dataset_name!r} ({len(group.file_paths)} files)" ) ``` `report.summary()` is the at-a-glance view: file and junk counts, exact-duplicate clusters with the bytes they waste, version candidates, potential personal data, and a `cognee_status` breakdown of how many files are new, staged, or already cognified. `report.groups` holds the proposal itself — each group carries the `dataset_name` it would land in and the files it would take there. ### Step 3: Set the Apply Decisions ```python theme={null} report.exclude_pii = True # keep files with personal data out of the graph report.skip_duplicates = True # ingest one copy per duplicate cluster report.apply_groups = [group.name for group in report.groups if group.kind != "code_project"] ``` The report is editable, and the apply phase reads these three fields off it. `exclude_pii` defaults to `False`, so setting it is what actually changes the run — it drops files with potential personal data. `skip_duplicates` is already on by default; it is spelled out to keep the decision visible. `apply_groups` narrows the run to the groups you name — here every group except code projects, which belong in a [code graph](/guides/code-graph) rather than a document dataset. ### Step 4: Apply the Report and Recall ```python theme={null} results = await cognee.remember(report) for dataset_name, result in results.items(): print(f"ingested dataset {dataset_name!r}: {result}") answers = await cognee.recall("What documents do I have?", datasets=list(results)) for answer in answers: print(answer) ``` Passing the report back to `remember()` runs the second phase: each approved group goes through the normal add → cognify chain into its own dataset, and you get back a `{dataset_name: result}` mapping. Ingestion is incremental, so re-running the script after adding a few files only processes what is new. From there the folder is ordinary cognee memory — `recall()` takes the dataset names as its scope. ## Advanced Usage The analyze phase needs no LLM or embedding configuration: junk filtering, duplicate detection, version candidates, and folder-based grouping are all deterministic. Pass `use_llm=True` to `remember(folder, dry_run="presort", use_llm=True)` for LLM content classification, deeper PII detection, and semantic grouping. The apply phase runs cognify and does need a configured LLM. Without one, presort degrades rather than fails: the deterministic scan still runs, `use_llm` is downgraded, and apply stages files with `add()` only — each reported as a warning on the report. When you trust the defaults, `remember(FOLDER, dry_run="presort", auto_apply=True)` collapses both phases into one call instead of the two this guide makes — see [auto\_apply](/python-api/remember#folder-presort) for what comes back. `cognee-cli remember --presort` runs the same two phases from the shell, with `--allow-root` in place of the environment variable. The commands and their apply-time flags are in the [CLI reference](/cognee-cli/overview#remember-data). How the datasets presort proposes organize documents, permissions, and processing. Every option the two presort phases accept, including the apply overrides.