Skip to main content

What is the add operation

add is the first step of the lower-level pipeline. It ingests text, files and URLs into a dataset and stores them, ready for cognify. remember runs it for you.

What happens under the hood

Streams input, computes a content hash, deduplicates, and persists the data plus metadata. Accepts text, file paths, and HTTP(S) URLs (fetched and routed by MIME type). URL fetches refuse private, loopback, link-local, carrier-grade NAT (100.64.0.0/10), multicast and unspecified addresses, including IPv4-mapped IPv6 forms. This applies to IP-literal URLs, to hostnames that resolve only to such addresses, and to every redirect hop. A blocked URL fails at once with an invalid-URL error and is not retried. HTTP_PROXY / HTTPS_PROXY / ALL_PROXY (and NO_PROXY) are honoured; when a proxy is used, the proxy resolves the target hostname, so only IP-literal targets are checked. data rows are dataset-scoped: a new row gets a random uuid4 id, and re-adding content dedups only within the same dataset, owner and tenant (by content hash), so the same content in two datasets is two rows, and a re-add after delete gets a new id. Dataset ids stay deterministic (UUID5 of name, owner and tenant). Upgrading runs a migration that splits a row shared by several datasets into one row per dataset (the copies record the original id as legacy_id, and the bindings’ data records gain dataset_id and legacy_id); stop every writer before upgrading.

Loaders

Files are routed to a loader by their sniffed content type: the data row stores the sniffed original_extension / original_mime_type (a local .html file keeps its path’s), non-UTF-8 bytes routed to the text loader (for example a PNG named .txt) fail the add, and a file no loader claims (for example a .zip) fails with NoLoaderForFile instead of falling back to the text loader. AddParams::preferred_loaders is now honoured: its loader names (text_loader, pypdf_loader, image_loader, …) are tried in order before the default priority, and a name that is not registered is skipped. Images and audio are accepted: the image loader (vision LLM) and audio loader (Whisper, when the LLM provider has a transcriber) run at add, which stores the description or transcript and keeps the raw file in original_data_location. When the provider transcribes but its transcriber cannot be built (no API key, say), an audio add fails with audio needs a transcriber, which could not be built: … instead of an unsupported-type error. Re-adding the same audio into the same dataset reuses the stored transcript while it is still in storage (even with no transcriber configured any more), so sequential re-adds transcribe once. An add into another dataset, or over a row holding raw audio (as a URL add stores it), uses the transcript already stored for that file (see below); only a missing transcript is transcribed again. Two concurrent adds of the same new file can still both transcribe it. A fetched HTML page (URL add) is stored raw and read by text_loader; a local .html file is still text-extracted at add time. Every file or URL add writes external_metadata {"_cognee": {"source_uri": <url or file:// URI>}} merged over the item’s own metadata. Pipeline: cognee-ingestion (AddPipeline).

File storage

Stored files are named by content hash and written directly under data_root_directory: extracted text as text_<hash>.txt, a fetched page as text_<hash>.html, other fetched content as fetched_<hash>.<ext>. The same content is stored once and shared by every row that holds it, and delete removes it only when no row points at it any more. Fetched content is rewritten in place on a re-add (write, then rename), while a loader’s text keeps the first version stored: a later add of the same file, in any dataset, reuses it instead of running the loader (a transcription or image description) again. A path that would leave the storage root is refused. While an add runs, the files it stored are claimed, so a concurrent delete in another dataset keeps them until the add’s rows are saved (within one process). Rows from builds before this change point at random ab/cd/ folders; they keep working as they are. This is a breaking change for code that calls LocalStorage directly: store, store_stream_dyn and create_writer now write at the relative path they are given and return it, overwriting a file already there, and fail with StorageError::InvalidPath for a path outside the storage root.

Cognify

The next pipeline step

Loaders & Chunkers

How files are read

Datasets

Where added data lives