> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognee.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Rust SDK Add

> add in cognee-rust: ingest text, files and URLs, deduplicate per dataset, and store files by content hash.

## What is the add operation

`add` is the first step of the lower-level pipeline. It ingests text, files and URLs into a dataset and stores them, ready for [cognify](https://docs.cognee.ai/rust/core-concepts/main-operations/legacy-operations/cognify.md). [remember](https://docs.cognee.ai/rust/core-concepts/main-operations/remember.md) runs it for you.

## What happens under the hood

Streams input, computes a content hash, deduplicates, and persists the data plus
metadata. Accepts text, file paths, and HTTP(S) URLs (fetched and routed by MIME
type). URL fetches refuse private, loopback, link-local, carrier-grade NAT
(`100.64.0.0/10`), multicast and unspecified addresses, including IPv4-mapped
IPv6 forms. This applies to IP-literal URLs, to hostnames that resolve only to
such addresses, and to every redirect hop. A blocked URL fails at once with an
invalid-URL error and is not retried. `HTTP_PROXY` / `HTTPS_PROXY` / `ALL_PROXY`
(and `NO_PROXY`) are honoured; when a proxy is used, the proxy resolves the
target hostname, so only IP-literal targets are checked. `data` rows are
dataset-scoped: a new row gets a random `uuid4` id, and re-adding content dedups only within the same dataset, owner and tenant (by content
hash), so the same content in two datasets is two rows, and a re-add after
delete gets a new id. Dataset ids stay deterministic (UUID5 of name, owner and
tenant). Upgrading runs a migration that splits a row shared by several
datasets into one row per dataset (the copies record the original id as
`legacy_id`, and the bindings' data records gain `dataset_id` and `legacy_id`);
stop every writer before upgrading.

## Loaders

Files are routed to a loader by their **sniffed** content type:
the `data` row stores the sniffed `original_extension` / `original_mime_type` (a local `.html` file keeps its path's),
non-UTF-8 bytes routed to the text loader (for example a PNG named `.txt`) fail
the add, and a file no loader claims (for example a `.zip`) fails with
`NoLoaderForFile` instead of falling back to the text loader.
`AddParams::preferred_loaders` is now honoured: its loader names
(`text_loader`, `pypdf_loader`, `image_loader`, …) are tried in order before
the default priority, and a name that is not registered is skipped. Images and audio
are accepted: the image loader (vision LLM) and audio loader (Whisper, when the
LLM provider has a transcriber) run at add, which stores the description or
transcript and keeps the raw file in `original_data_location`. When the
provider transcribes but its transcriber cannot be built (no API key, say), an
audio add fails with `audio needs a transcriber, which could not be built: …`
instead of an unsupported-type error. Re-adding the same audio into the same
dataset reuses the stored transcript while it is still in storage (even with no
transcriber configured any more), so sequential re-adds transcribe once. An add
into another dataset, or over a row holding raw audio (as a URL add stores it),
uses the transcript already stored for that file (see below); only a missing
transcript is transcribed again. Two concurrent adds of the same new file can still both
transcribe it. A fetched HTML
page (URL add) is stored raw and read by `text_loader`; a local
`.html` file is still text-extracted at add time.
Every file or URL add writes `external_metadata`
`{"_cognee": {"source_uri": <url or file:// URI>}}` merged over the item's own
metadata. Pipeline: [`cognee-ingestion`](https://github.com/topoteretes/cognee-rs/blob/main/crates/ingestion/) (`AddPipeline`).

## File storage

Stored files are named by content hash and written directly under
`data_root_directory`: extracted text as `text_<hash>.txt`, a
fetched page as `text_<hash>.html`, other fetched content as
`fetched_<hash>.<ext>`. The same content is stored once and shared by every row
that holds it, and delete removes it only when no row points at it any more.
Fetched content is rewritten in place on a re-add (write, then
rename), while a loader's text keeps the first version stored: a later add of
the same file, in any dataset, reuses it instead of running the loader (a
transcription or image description) again. A path that would leave the storage root is refused. While
an `add` runs, the files it stored are claimed, so a concurrent delete in
another dataset keeps them until the add's rows are saved (within one process).
Rows from builds before this change point at random `ab/cd/`
folders; they keep working as they are. This is a breaking change for code that
calls `LocalStorage` directly: `store`, `store_stream_dyn` and `create_writer`
now write at the relative path they are given and return it, overwriting a file
already there, and fail with `StorageError::InvalidPath` for a path outside the
storage root.

## Related

> [card] [Cognify](https://docs.cognee.ai/rust/core-concepts/main-operations/legacy-operations/cognify.md)
>
> The next pipeline step

> [card] [Loaders & Chunkers](https://docs.cognee.ai/rust/core-concepts/further-concepts/loaders-and-chunkers.md)
>
> How files are read

> [card] [Datasets](https://docs.cognee.ai/rust/core-concepts/further-concepts/datasets.md)
>
> Where added data lives


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.