> ## Documentation Index > Fetch the complete documentation index at: https://docs.cognee.ai/llms.txt > Use this file to discover all available pages before exploring further. # Organizing Your Data > Ingest the same mixed corpus three ways — one pile, node sets, separate datasets — and watch how each layout changes what recall answers Your assistant's memory holds both your product documentation and your sales-call transcripts, and its answers have started blending the two — a question about API rate limits comes back quoting whatever a rep promised a customer last quarter. The memory is doing its job by linking related facts; the problem is that everything lives in one undifferentiated pile. ## What You'll Build Four small files — two technical documents (an API reference and an architecture guide) and two sales-call transcripts (where a rep promises "unlimited requests" and "instant sync") — are ingested three ways: everything in one dataset, one dataset with node-set tags, and two walled-off datasets with node sets inside each. The same questions run under every layout, and each recall prints as a comparable `QUERY` / `SCOPE` / `ANSWER` block, so you can see exactly what each layout changes: the unfiltered pile blends promise with spec, node sets scope a query to one group while still answering cross-domain questions, and datasets make leakage impossible. The run ends with the rule of thumb the demo exists to teach: node sets are tags, datasets are walls. The complete runnable demo — the script plus the four data files it reads — is [`examples/demos/organizing_your_data/`](https://github.com/topoteretes/cognee/tree/dev/examples/demos/organizing_your_data) — this page walks through its key moments rather than reproducing it. ## Features in Play * [NodeSets](/core-concepts/further-concepts/node-sets) — tags each source with soft, overlappable groups at write time; `node_name` scopes retrieval to a tag at read time * [Datasets](/core-concepts/further-concepts/datasets) — the hard boundary with its own permissions and storage that keeps sales transcripts out of technical answers entirely * [Remember](/core-concepts/main-operations/remember) — ingests the four bundled files under each layout's labels * [Recall](/core-concepts/main-operations/recall) — runs the same questions under every layout, scoped by `datasets` and `node_name` ## What to Expect The excerpts below are from a real run, trimmed, and follow the same seven stages as [How It Works](#how-it-works): that section explains what each stage does, this one shows the output each stage produces. Stages 2 and 5 are `remember()` calls that print nothing, so they appear together with the recall that first reveals their effect. Every recall prints as a `QUERY` / `SCOPE` / `NOTE` / `ANSWER` block, and each `ANSWER` line is prefixed with the dataset it came from, so the same question can be compared across scopes by eye. Every answer is a live LLM call, so the wording varies from run to run. **Stage 1 — one pile, one retrieval pool.** With no filter, the rate-limit question retrieves from the API reference and the sales calls alike. In this run the model resolved the conflict in favor of the documentation, but nothing in the query kept the rep's promise out of the pool; the `NOTE` line names the competition to watch for. ```text theme={null} ============================================================================== (1) ALL DATA IN ONE DATASET — the setup that confuses the LLM ============================================================================== QUERY : What rate limits does the Acme API enforce? SCOPE : everything (no filter) NOTE : docs and sales calls share one retrieval pool — the documented limit (100/min) competes with the rep's 'unlimited requests' promise ANSWER: [main_dataset] - Default: 100 requests per minute per API key. - Enterprise: can be raised to 500 requests per minute per key. ... - No plan offers unlimited requests. ``` **Stages 2 and 3 — node sets scope the same question two ways.** The tagging in Stage 2 prints nothing; its effect shows the moment Stage 3 scopes a recall to one tag. Both recalls run against one dataset and one graph. Scoped to `tech_docs`, the answer is the documented limit; scoped to `sales_calls` with the same call shape, it is what the rep said, with no mention of the spec. ```text theme={null} QUERY : What rate limits does the Acme API enforce? SCOPE : node_name=['tech_docs'] NOTE : the documented truth only ANSWER: [main_dataset] - Default: 100 requests per minute per API key. - Enterprise: can raise to 500 requests per minute per key. ... - No plan offers unlimited requests; provision multiple keys to shard traffic. QUERY : What did we promise customers about rate limits? SCOPE : node_name=['sales_calls'] NOTE : what the reps actually said ANSWER: [main_dataset] We promised to remove the 100 requests/minute cap for enterprise deals and to allow unlimited (no throttling) requests during the pilot — the AE said they'd include this in the proposal deck. ``` **Stage 4 — the unfiltered cross-domain question pays off.** Because both groups share one graph, the deliberately unscoped recall lines up the sales promises against the API reference and cites both sources. This is the question a dataset wall would make impossible. ```text theme={null} QUERY : Where do our sales promises contradict the technical documentation? SCOPE : everything (no filter, on purpose) NOTE : a cross-domain question spanning both groups ANSWER: [main_dataset] Short answer — two direct contradictions: - Enterprise “remove rate limits entirely” (sales AE told Initech this) vs. the API docs stating Enterprise can raise the per-key limit only to 500 req/min and “No plan, including Enterprise, offers unlimited requests.” ... - “Unlimited requests during the pilot / we won’t throttle you” (AE promised, and agreed to put it in the proposal deck) vs. the API docs stating rate limits are enforced per API key ... ``` **Stages 5 and 6 — datasets wall the domains off, and node sets still slice inside.** The three `remember()` calls in Stage 5 print nothing; the `[tech_docs]` prefix on the first answer shows it came from that dataset alone, and Stage 6's combined `datasets` plus `node_name` filter narrows the second to the Initech call only. ```text theme={null} ============================================================================== (3) SEPARATE DATASETS (tech vs sales) — hard walls, node sets inside ============================================================================== QUERY : What rate limits does the Acme API enforce? SCOPE : datasets=['tech_docs'] NOTE : sales calls cannot leak in ANSWER: [tech_docs] - Default: 100 requests per minute per API key. - Enterprise: up to 500 requests per minute per API key. ... - No plan offers unlimited requests. QUERY : What was discussed with Initech? SCOPE : datasets=['sales_calls'] + node_name=['initech'] NOTE : one account's calls only ANSWER: [sales_calls] They discussed Initech’s bursty batch jobs and that the 100 requests/min cap would be a blocker; Acme offered to remove rate limits for an enterprise deal and provide unlimited requests during the pilot, and Initech asked that the unlimited pilot offer be included in the proposal deck. ``` **Stage 7 — the footgun: an unscoped recall spans both datasets.** The same rate-limit question without `datasets` returns one `ANSWER` per readable dataset, `[tech_docs]` and `[sales_calls]`, so the rep's promise is back in the result. Separating data at write time is not enough; the query has to choose its scope. The run then closes with the `TAKEAWAY` banner. ```text theme={null} QUERY : What rate limits does the Acme API enforce? SCOPE : everything (no dataset filter) NOTE : spans both datasets again — scope must be chosen per query ANSWER: [tech_docs] - Default: 100 requests per minute per API key. - Enterprise: up to 500 requests per minute per API key. ... - No plan (including Enterprise) offers unlimited requests. ANSWER: [sales_calls] By default a 100 requests per minute cap is mentioned; Acme said they can remove rate limits for an enterprise deal and offered unlimited (no throttling) requests during the pilot. ============================================================================== TAKEAWAY ============================================================================== Node sets are tags: soft groups inside one shared graph — scoped AND cross-domain questions both work. Datasets are walls: own permissions, isolated storage, independent forget — content never crosses over. Mixing docs and sales calls? Start with node sets; move the domains into separate datasets when they must never contaminate each other. ``` ## Before You Start * Complete [Quickstart](/getting-started/quickstart) to understand basic operations * Ensure you have [LLM Providers](/setup-configuration/llm-providers) configured — every answer is a live LLM call * Use Ladybug (the default) or Neo4j as your [graph store](/setup-configuration/graph-stores) — both support node-set filtering out of the box * Leave `ENABLE_BACKEND_ACCESS_CONTROL` at its default, or set it to `true`: dataset scoping in `recall()` only walls datasets off in [multi-user mode](/core-concepts/multi-user-mode/multi-user-mode-overview), which the default Ladybug and LanceDB stack turns on automatically * Run it from a checkout of the cognee repo: the script reads its two markdown documents and two transcripts from the sibling `data/` folder * The script calls `cognee.forget(everything=True)` before each of its three sections, so point it at a scratch instance rather than memory you want to keep — see [Forget](/core-concepts/main-operations/forget) ## How It Works ### Stage 1: Ingest Everything into One Dataset ```python theme={null} await cognee.remember( TECH_DOCS + [SALES_CALL_INITECH, SALES_CALL_HOOLI], self_improvement=False ) answer = await cognee.recall(RATE_LIMIT_QUERY, query_type=SearchType.HYBRID_COMPLETION) ``` All four files land in the default dataset with no labels, and the rate-limit question retrieves from docs and sales calls alike. This is the layout that produces the client complaint the demo opens with — the documented 100-requests-per-minute limit competes with the rep's "unlimited requests" promise in one retrieval pool. ### Stage 2: Tag Every Source with Node Sets ```python theme={null} # An item can carry several tags at once: everything here is also tagged # "acme_api", so a shared grouping cuts across the docs/calls split. await cognee.remember(TECH_DOCS, node_set=["tech_docs", "acme_api"], self_improvement=False) await cognee.remember( [SALES_CALL_INITECH, SALES_CALL_HOOLI], node_set=["sales_calls", "acme_api"], self_improvement=False, ) ``` Same dataset, same graph — but now every item carries node-set tags. Tags are soft and overlappable: the shared `acme_api` tag cuts across the docs/calls split, which no folder-style hierarchy could express. ### Stage 3: Scope a Recall to One Tag ```python theme={null} docs_answer = await cognee.recall( RATE_LIMIT_QUERY, query_type=SearchType.HYBRID_COMPLETION, node_name=["tech_docs"], ) ``` `node_name` restricts retrieval to the tagged subgraph, so the same question now answers from the documentation alone. The sales calls are still in the graph — they just don't participate in this query. ### Stage 4: Ask a Cross-Domain Question Unfiltered ```python theme={null} # The graph is still ONE graph — cross-domain questions work when you # want them to, by simply not filtering. cross_query = "Where do our sales promises contradict the technical documentation?" cross_answer = await cognee.recall(cross_query, query_type=SearchType.GRAPH_COMPLETION) ``` This is the payoff of tags over walls: because docs and calls share one graph, entities from both link to each other, and an unfiltered recall can draw on both sides to surface where the promises and the spec disagree. Choose node sets when you want both scoped and cross-domain questions to work. ### Stage 5: Wall Off Tech and Sales into Datasets ```python theme={null} await cognee.remember( TECH_DOCS, dataset_name="tech_docs", node_set=["api_reference"], self_improvement=False, ) await cognee.remember( SALES_CALL_INITECH, dataset_name="sales_calls", node_set=["initech"], self_improvement=False, ) await cognee.remember( SALES_CALL_HOOLI, dataset_name="sales_calls", node_set=["hooli"], self_improvement=False, ) ``` The third layout gives each domain its own dataset: graphs are built independently, permissions and storage are separate, and either side can be forgotten without touching the other. Node sets still work inside each dataset — here the sales dataset tags each call by account. ### Stage 6: Combine Dataset and Node-Set Filters ```python theme={null} initech_query = "What was discussed with Initech?" initech_answer = await cognee.recall( initech_query, query_type=SearchType.HYBRID_COMPLETION, datasets=["sales_calls"], node_name=["initech"], ) ``` The two mechanisms compose: `datasets` picks the wall, `node_name` picks the tag inside it, and the answer covers one account's calls only. ### Stage 7: Watch an Unscoped Recall Span Both Datasets ```python theme={null} # Caution: recall() without `datasets` spans ALL datasets you can read — # separating data at write time is not enough, queries must opt into the # scope they want. unscoped_answer = await cognee.recall(RATE_LIMIT_QUERY, query_type=SearchType.HYBRID_COMPLETION) ``` The demo closes on its one footgun: separating data at write time is not enough. `recall()` without `datasets` spans every dataset you can read, so the final answer draws on both sides again — queries must opt into the scope they want. ## Run It ```bash theme={null} uv run python examples/demos/organizing_your_data/organizing_your_data_demo.py ``` ## Choosing a Layout Start with the simplest layout that answers your queries cleanly. One dataset with no labels is fine while the corpus covers a single domain for a single audience. Reach for node sets the moment distinct content types share entities you still want linked — scoped questions stay clean, and cross-domain questions keep working. Move a domain into its own dataset when it must never contaminate the other's answers, or when it needs different access control or retention — and remember the two compose, so datasets can carry node sets inside them. How node-set labels are written and how `node_name` filters retrieval by them. What a dataset isolates — documents, permissions, and processing. The minimal single-feature guide to tagging memories with node sets. A sibling demo where node-set scoping powers an agent's research phase.