Organizing Your Data

Your assistant’s memory holds both your product documentation and your sales-call transcripts, and its answers have started blending the two — a question about API rate limits comes back quoting whatever a rep promised a customer last quarter. The memory is doing its job by linking related facts; the problem is that everything lives in one undifferentiated pile.

What You’ll Build

Four small files — two technical documents (an API reference and an architecture guide) and two sales-call transcripts (where a rep promises “unlimited requests” and “instant sync”) — are ingested three ways: everything in one dataset, one dataset with node-set tags, and two walled-off datasets with node sets inside each. The same questions run under every layout, and each recall prints as a comparable QUERY / SCOPE / ANSWER block, so you can see exactly what each layout changes: the unfiltered pile blends promise with spec, node sets scope a query to one group while still answering cross-domain questions, and datasets make leakage impossible. The run ends with the rule of thumb the demo exists to teach: node sets are tags, datasets are walls. The complete runnable demo — the script plus the four data files it reads — is examples/demos/organizing_your_data/ — this page walks through its key moments rather than reproducing it.

Features in Play

NodeSets — tags each source with soft, overlappable groups at write time; node_name scopes retrieval to a tag at read time
Datasets — the hard boundary with its own permissions and storage that keeps sales transcripts out of technical answers entirely
Remember — ingests the four bundled files under each layout’s labels
Recall — runs the same questions under every layout, scoped by datasets and node_name

What to Expect

The excerpts below are from a real run, trimmed, and follow the same seven stages as How It Works: that section explains what each stage does, this one shows the output each stage produces. Stages 2 and 5 are remember() calls that print nothing, so they appear together with the recall that first reveals their effect. Every recall prints as a QUERY / SCOPE / NOTE / ANSWER block, and each ANSWER line is prefixed with the dataset it came from, so the same question can be compared across scopes by eye. Every answer is a live LLM call, so the wording varies from run to run. Stage 1 — one pile, one retrieval pool. With no filter, the rate-limit question retrieves from the API reference and the sales calls alike. In this run the model resolved the conflict in favor of the documentation, but nothing in the query kept the rep’s promise out of the pool; the NOTE line names the competition to watch for.

==============================================================================
(1) ALL DATA IN ONE DATASET — the setup that confuses the LLM
==============================================================================

  QUERY : What rate limits does the Acme API enforce?
  SCOPE : everything (no filter)
  NOTE  : docs and sales calls share one retrieval pool — the documented limit (100/min) competes with the rep's 'unlimited requests' promise
  ANSWER: [main_dataset] - Default: 100 requests per minute per API key.
          - Enterprise: can be raised to 500 requests per minute per key.
...
          - No plan offers unlimited requests.

Stages 2 and 3 — node sets scope the same question two ways. The tagging in Stage 2 prints nothing; its effect shows the moment Stage 3 scopes a recall to one tag. Both recalls run against one dataset and one graph. Scoped to tech_docs, the answer is the documented limit; scoped to sales_calls with the same call shape, it is what the rep said, with no mention of the spec.

  QUERY : What rate limits does the Acme API enforce?
  SCOPE : node_name=['tech_docs']
  NOTE  : the documented truth only
  ANSWER: [main_dataset] - Default: 100 requests per minute per API key.
          - Enterprise: can raise to 500 requests per minute per key.
...
          - No plan offers unlimited requests; provision multiple keys to shard traffic.

  QUERY : What did we promise customers about rate limits?
  SCOPE : node_name=['sales_calls']
  NOTE  : what the reps actually said
  ANSWER: [main_dataset] We promised to remove the 100 requests/minute cap for enterprise deals and to allow unlimited (no throttling) requests during the pilot — the AE said they'd include this in the proposal deck.

Stage 4 — the unfiltered cross-domain question pays off. Because both groups share one graph, the deliberately unscoped recall lines up the sales promises against the API reference and cites both sources. This is the question a dataset wall would make impossible.

  QUERY : Where do our sales promises contradict the technical documentation?
  SCOPE : everything (no filter, on purpose)
  NOTE  : a cross-domain question spanning both groups
  ANSWER: [main_dataset] Short answer — two direct contradictions:

          - Enterprise “remove rate limits entirely” (sales AE told Initech this) vs. the API docs stating Enterprise can raise the per-key limit only to 500 req/min and “No plan, including Enterprise, offers unlimited requests.” ...

          - “Unlimited requests during the pilot / we won’t throttle you” (AE promised, and agreed to put it in the proposal deck) vs. the API docs stating rate limits are enforced per API key ...

Stages 5 and 6 — datasets wall the domains off, and node sets still slice inside. The three remember() calls in Stage 5 print nothing; the [tech_docs] prefix on the first answer shows it came from that dataset alone, and Stage 6’s combined datasets plus node_name filter narrows the second to the Initech call only.

==============================================================================
(3) SEPARATE DATASETS (tech vs sales) — hard walls, node sets inside
==============================================================================

  QUERY : What rate limits does the Acme API enforce?
  SCOPE : datasets=['tech_docs']
  NOTE  : sales calls cannot leak in
  ANSWER: [tech_docs] - Default: 100 requests per minute per API key.
          - Enterprise: up to 500 requests per minute per API key.
...
          - No plan offers unlimited requests.

  QUERY : What was discussed with Initech?
  SCOPE : datasets=['sales_calls'] + node_name=['initech']
  NOTE  : one account's calls only
  ANSWER: [sales_calls] They discussed Initech’s bursty batch jobs and that the 100 requests/min cap would be a blocker; Acme offered to remove rate limits for an enterprise deal and provide unlimited requests during the pilot, and Initech asked that the unlimited pilot offer be included in the proposal deck.

Stage 7 — the footgun: an unscoped recall spans both datasets. The same rate-limit question without datasets returns one ANSWER per readable dataset, [tech_docs] and [sales_calls], so the rep’s promise is back in the result. Separating data at write time is not enough; the query has to choose its scope. The run then closes with the TAKEAWAY banner.

  QUERY : What rate limits does the Acme API enforce?
  SCOPE : everything (no dataset filter)
  NOTE  : spans both datasets again — scope must be chosen per query
  ANSWER: [tech_docs] - Default: 100 requests per minute per API key.
          - Enterprise: up to 500 requests per minute per API key.
...
          - No plan (including Enterprise) offers unlimited requests.
  ANSWER: [sales_calls] By default a 100 requests per minute cap is mentioned; Acme said they can remove rate limits for an enterprise deal and offered unlimited (no throttling) requests during the pilot.

==============================================================================
TAKEAWAY
==============================================================================
  Node sets are tags: soft groups inside one shared graph — scoped AND
  cross-domain questions both work. Datasets are walls: own permissions,
  isolated storage, independent forget — content never crosses over.
  Mixing docs and sales calls? Start with node sets; move the domains
  into separate datasets when they must never contaminate each other.

Before You Start

Complete Quickstart to understand basic operations
Ensure you have LLM Providers configured — every answer is a live LLM call
Use Ladybug (the default) or Neo4j as your graph store — both support node-set filtering out of the box
Leave ENABLE_BACKEND_ACCESS_CONTROL at its default, or set it to true: dataset scoping in recall() only walls datasets off in multi-user mode, which the default Ladybug and LanceDB stack turns on automatically
Run it from a checkout of the cognee repo: the script reads its two markdown documents and two transcripts from the sibling data/ folder
The script calls cognee.forget(everything=True) before each of its three sections, so point it at a scratch instance rather than memory you want to keep — see Forget

How It Works

Stage 1: Ingest Everything into One Dataset

    await cognee.remember(
        TECH_DOCS + [SALES_CALL_INITECH, SALES_CALL_HOOLI], self_improvement=False
    )

    answer = await cognee.recall(RATE_LIMIT_QUERY, query_type=SearchType.HYBRID_COMPLETION)

All four files land in the default dataset with no labels, and the rate-limit question retrieves from docs and sales calls alike. This is the layout that produces the client complaint the demo opens with — the documented 100-requests-per-minute limit competes with the rep’s “unlimited requests” promise in one retrieval pool.

Stage 2: Tag Every Source with Node Sets

    # An item can carry several tags at once: everything here is also tagged
    # "acme_api", so a shared grouping cuts across the docs/calls split.
    await cognee.remember(TECH_DOCS, node_set=["tech_docs", "acme_api"], self_improvement=False)
    await cognee.remember(
        [SALES_CALL_INITECH, SALES_CALL_HOOLI],
        node_set=["sales_calls", "acme_api"],
        self_improvement=False,
    )

Same dataset, same graph — but now every item carries node-set tags. Tags are soft and overlappable: the shared acme_api tag cuts across the docs/calls split, which no folder-style hierarchy could express.

Stage 3: Scope a Recall to One Tag

    docs_answer = await cognee.recall(
        RATE_LIMIT_QUERY,
        query_type=SearchType.HYBRID_COMPLETION,
        node_name=["tech_docs"],
    )

node_name restricts retrieval to the tagged subgraph, so the same question now answers from the documentation alone. The sales calls are still in the graph — they just don’t participate in this query.

Stage 4: Ask a Cross-Domain Question Unfiltered

    # The graph is still ONE graph — cross-domain questions work when you
    # want them to, by simply not filtering.
    cross_query = "Where do our sales promises contradict the technical documentation?"
    cross_answer = await cognee.recall(cross_query, query_type=SearchType.GRAPH_COMPLETION)

This is the payoff of tags over walls: because docs and calls share one graph, entities from both link to each other, and an unfiltered recall can draw on both sides to surface where the promises and the spec disagree. Choose node sets when you want both scoped and cross-domain questions to work.

Stage 5: Wall Off Tech and Sales into Datasets

    await cognee.remember(
        TECH_DOCS,
        dataset_name="tech_docs",
        node_set=["api_reference"],
        self_improvement=False,
    )
    await cognee.remember(
        SALES_CALL_INITECH,
        dataset_name="sales_calls",
        node_set=["initech"],
        self_improvement=False,
    )
    await cognee.remember(
        SALES_CALL_HOOLI,
        dataset_name="sales_calls",
        node_set=["hooli"],
        self_improvement=False,
    )

The third layout gives each domain its own dataset: graphs are built independently, permissions and storage are separate, and either side can be forgotten without touching the other. Node sets still work inside each dataset — here the sales dataset tags each call by account.

Stage 6: Combine Dataset and Node-Set Filters

    initech_query = "What was discussed with Initech?"
    initech_answer = await cognee.recall(
        initech_query,
        query_type=SearchType.HYBRID_COMPLETION,
        datasets=["sales_calls"],
        node_name=["initech"],
    )

The two mechanisms compose: datasets picks the wall, node_name picks the tag inside it, and the answer covers one account’s calls only.

Stage 7: Watch an Unscoped Recall Span Both Datasets

    # Caution: recall() without `datasets` spans ALL datasets you can read —
    # separating data at write time is not enough, queries must opt into the
    # scope they want.
    unscoped_answer = await cognee.recall(RATE_LIMIT_QUERY, query_type=SearchType.HYBRID_COMPLETION)

The demo closes on its one footgun: separating data at write time is not enough. recall() without datasets spans every dataset you can read, so the final answer draws on both sides again — queries must opt into the scope they want.

Run It

uv run python examples/demos/organizing_your_data/organizing_your_data_demo.py

Choosing a Layout

Start with the simplest layout that answers your queries cleanly. One dataset with no labels is fine while the corpus covers a single domain for a single audience. Reach for node sets the moment distinct content types share entities you still want linked — scoped questions stay clean, and cross-domain questions keep working. Move a domain into its own dataset when it must never contaminate the other’s answers, or when it needs different access control or retention — and remember the two compose, so datasets can carry node sets inside them.

NodeSets

How node-set labels are written and how node_name filters retrieval by them.

Datasets

What a dataset isolates — documents, permissions, and processing.

NodeSet Grouping

The minimal single-feature guide to tagging memories with node sets.

Agentic Procurement Decisions

A sibling demo where node-set scoping powers an agent’s research phase.

Getting Started

Core Concepts

Setup Configuration

Guides

Examples

CLI

Rust SDK

TypeScript SDK

OSS

What You’ll Build

Features in Play

What to Expect

Before You Start

How It Works

Stage 1: Ingest Everything into One Dataset

Stage 2: Tag Every Source with Node Sets

Stage 3: Scope a Recall to One Tag

Stage 4: Ask a Cross-Domain Question Unfiltered

Stage 5: Wall Off Tech and Sales into Datasets

Stage 6: Combine Dataset and Node-Set Filters

Stage 7: Watch an Unscoped Recall Span Both Datasets

Run It

Choosing a Layout

NodeSets

Datasets

NodeSet Grouping

Agentic Procurement Decisions

​What You’ll Build

​Features in Play

​What to Expect

​Before You Start

​How It Works

​Stage 1: Ingest Everything into One Dataset

​Stage 2: Tag Every Source with Node Sets

​Stage 3: Scope a Recall to One Tag

​Stage 4: Ask a Cross-Domain Question Unfiltered

​Stage 5: Wall Off Tech and Sales into Datasets

​Stage 6: Combine Dataset and Node-Set Filters

​Stage 7: Watch an Unscoped Recall Span Both Datasets

​Run It

​Choosing a Layout

NodeSets

Datasets

NodeSet Grouping

Agentic Procurement Decisions

What You’ll Build

Features in Play

What to Expect

Before You Start

How It Works

Stage 1: Ingest Everything into One Dataset

Stage 2: Tag Every Source with Node Sets

Stage 3: Scope a Recall to One Tag

Stage 4: Ask a Cross-Domain Question Unfiltered

Stage 5: Wall Off Tech and Sales into Datasets

Stage 6: Combine Dataset and Node-Set Filters

Stage 7: Watch an Unscoped Recall Span Both Datasets

Run It

Choosing a Layout