> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognee.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Vector Stores
> Configure vector databases for embedding storage and semantic search in Cognee
Vector stores hold embeddings for semantic similarity search. They enable Cognee to find conceptually related content based on meaning rather than exact text matches.
**New to configuration?**
See the [Setup Configuration Overview](./overview) for the complete workflow:
install extras → create `.env` → choose providers → handle pruning.
For a complete, copy-paste `.env` block that combines this layer with a relational and a graph store, see [Store Configurations](/guides/store-configurations).
## Supported Providers
Cognee supports multiple vector store options through built-in providers and community-maintained adapters:
| Provider | Status | Notes |
| - | - | - |
| **LanceDB** | Built-in, default | File-based vector store, works out of the box |
| **PGVector** | Built-in extra | Postgres-backed vector storage with pgvector extension |
| **Turso** | Built-in extra | Local Turso database file (the Rust rewrite of SQLite); exact cosine search |
| **Neptune Analytics** | Built-in extra | Amazon Neptune Analytics hybrid solution |
| **ChromaDB** | Optional extra / adapter | HTTP server-based vector database; may require installing extra support |
| **Qdrant** | Community adapter | High-performance vector database and similarity search engine |
| **Redis** | Community adapter | Fast vector similarity search via Redis Search module |
| **FalkorDB** | Community adapter | Hybrid graph + vector database |
## Configuration
Community adapters must still be installed and registered in your application startup code before Cognee can use their provider value.
Use this shape for LanceDB.
```dotenv theme={null}
VECTOR_DB_PROVIDER="lancedb"
# Optional path or URL. Defaults to /databases/cognee.lancedb
VECTOR_DB_URL="/absolute/or/relative/path/to/cognee.lancedb"
# Optional. Defaults to true.
VECTOR_DB_SUBPROCESS_ENABLED="true"
# Optional. Max concurrent async RPCs in flight per subprocess worker.
# Defaults to 16. Must be > 0 (worker init raises ValueError otherwise).
SUBPROCESS_WORKER_MAX_INFLIGHT="16"
# Optional. Seconds an idle subprocess engine is kept alive before its worker
# is closed. Defaults to 600. Set to 0 to close at every dataset-context exit.
SUBPROCESS_IDLE_TTL_SECONDS="600"
```
Use this shape for PGVector.
```dotenv theme={null}
VECTOR_DB_PROVIDER="pgvector"
VECTOR_DB_HOST="localhost"
VECTOR_DB_PORT="5432"
VECTOR_DB_NAME="cognee_db"
VECTOR_DB_USERNAME="cognee"
VECTOR_DB_PASSWORD="cognee"
# Optional SQLAlchemy pool args JSON.
VECTOR_POOL_ARGS='{"pool_size": 2, "max_overflow": 2}'
```
If the explicit `VECTOR_DB_*` Postgres values are omitted, Cognee falls back to the relational `DB_HOST`, `DB_PORT`, `DB_NAME`, `DB_USERNAME`, and `DB_PASSWORD` settings. When backend access control is enabled, configure the explicit `VECTOR_DB_*` values.
Use this shape for Turso. Only local database files are supported.
```dotenv theme={null}
VECTOR_DB_PROVIDER="turso"
VECTOR_DB_URL="/absolute/path/to/cognee.turso.db"
```
`VECTOR_DB_URL` is optional; without it the file lives under `/databases/`. A `libsql://`, `http(s)://`, or `ws(s)://` URL is rejected with an error. Requires the `cognee[turso]` extra.
Use this shape for Neptune Analytics.
```dotenv theme={null}
VECTOR_DB_PROVIDER="neptune_analytics"
VECTOR_DB_URL="neptune-graph://"
# Required — no per-dataset database handler exists for Neptune Analytics
ENABLE_BACKEND_ACCESS_CONTROL="false"
```
AWS credentials are resolved through the environment or the default AWS SDK chain.
Use this shape for ChromaDB.
```dotenv theme={null}
VECTOR_DB_PROVIDER="chromadb"
VECTOR_DB_URL="http://localhost:8000"
VECTOR_DB_KEY=""
```
ChromaDB support may require installing the optional extra or adapter before setting `VECTOR_DB_PROVIDER="chromadb"`.
Use this shape for community adapters that connect through a URL.
Provider values:
* `qdrant`
* `redis`
* `pinecone`
* `falkor`
```dotenv theme={null}
VECTOR_DB_PROVIDER="qdrant"
VECTOR_DB_URL="http://localhost:6333"
VECTOR_DB_KEY=""
```
Use this shape for FalkorDB:
```dotenv theme={null}
VECTOR_DB_PROVIDER="falkor"
VECTOR_DB_URL="localhost"
VECTOR_DB_PORT="6379"
GRAPH_DATABASE_PROVIDER="falkor"
GRAPH_DATABASE_URL="localhost"
GRAPH_DATABASE_PORT="6379"
```
Use `VECTOR_DB_KEY` for Qdrant Cloud, Pinecone, or other authenticated deployments. Redis usually puts credentials in the URL. FalkorDB also needs `VECTOR_DB_PORT`, and hybrid graph + vector setups need the matching `GRAPH_DATABASE_*` variables.
Use this shape for Turbopuffer.
```dotenv theme={null}
TURBOPUFFER_API_KEY="your_api_key"
VECTOR_DATASET_DATABASE_HANDLER="turbopuffer"
# Optional: defaults to gcp-us-central1
TURBOPUFFER_REGION="gcp-us-central1"
```
This adapter uses custom `TURBOPUFFER_*` environment variables instead of the normal `VECTOR_DB_URL` / `VECTOR_DB_KEY` shape.
## Setup Guides
LanceDB is file-based and requires no additional setup. It's perfect for local development and single-user scenarios.
```dotenv theme={null}
VECTOR_DB_PROVIDER="lancedb"
# Optional, can be a path or URL. Defaults to /databases/cognee.lancedb
# VECTOR_DB_URL=/absolute/or/relative/path/to/cognee.lancedb
```
**Installation**: LanceDB is included by default with Cognee. No additional installation required.
**Data Location**: Vectors are stored in a local directory. Defaults under the Cognee system path if `VECTOR_DB_URL` is empty.
PGVector stores vectors inside your Postgres database using the pgvector extension.
```dotenv theme={null}
VECTOR_DB_PROVIDER="pgvector"
# If these are omitted, Cognee falls back to the relational DB settings
# (DB_HOST, DB_PORT, DB_NAME, DB_USERNAME, DB_PASSWORD).
VECTOR_DB_HOST="localhost"
VECTOR_DB_PORT="5432"
VECTOR_DB_NAME="cognee_db"
VECTOR_DB_USERNAME="cognee"
VECTOR_DB_PASSWORD="cognee"
```
**Installation**: Install the Postgres extras:
```bash theme={null}
pip install "cognee[postgres]"
# or for binary version
pip install "cognee[postgres-binary]"
```
**Docker Setup**: Use the built-in Postgres with pgvector:
```bash theme={null}
docker compose --profile postgres up -d
```
**Note**: If using your own Postgres, ensure `CREATE EXTENSION IF NOT EXISTS vector;` is available in the target database.
For Neon Postgres, run that extension statement once in the Neon database where Cognee stores vectors. Neon still uses the regular Cognee `pgvector` provider; configure the relational `DB_*` values and `DATABASE_CONNECT_ARGS` as described in [Relational Databases](/setup-configuration/relational-databases#neon-postgres), then set `VECTOR_DB_PROVIDER="pgvector"`.
When backend access control is enabled, configure the explicit `VECTOR_DB_HOST`, `VECTOR_DB_PORT`, `VECTOR_DB_NAME`, `VECTOR_DB_USERNAME`, and `VECTOR_DB_PASSWORD` values instead of relying on the relational DB fallback.
**Dedicated vs shared database.** PGVector either gets its own connection engine or borrows the relational one, and which of the two you get decides whether the vector store can be pruned on its own:
| Configuration | Result |
| - | - |
| All five `VECTOR_DB_*` values set and `VECTOR_DB_NAME` **different** from `DB_NAME` | **Dedicated.** Cognee builds its own engine against that database; collection tables live there, and `prune_system(vector=True, metadata=False)` drops only them. |
| `VECTOR_DB_NAME` resolves to the same database as `DB_NAME`, with `ENABLE_BACKEND_ACCESS_CONTROL="false"` | **Shared.** PGVector reuses the relational engine, collection tables sit beside the metadata tables, and a vector-only prune is [refused](#troubleshooting). |
| Any of the five values missing, with `ENABLE_BACKEND_ACCESS_CONTROL="false"` | **Shared.** Cognee logs a warning and falls back wholesale to the relational `DB_*` settings — setting `VECTOR_DB_NAME` alone does not give you a dedicated database. (With access control enabled, missing credentials raise instead of falling back.) |
If you previously ran with `ENABLE_BACKEND_ACCESS_CONTROL="false"` and a fully configured `VECTOR_DB_*` block whose `VECTOR_DB_NAME` differed from `DB_NAME`, Cognee wrote vectors into the **relational** database and now reads and writes the dedicated one — so existing vectors become invisible. See the [changelog](/changelog) for the upgrade options.
Turso stores vectors in a local database file on Turso, the Rust rewrite of SQLite. Each collection is one table, and similarity search is an exact cosine-distance scan (there is no approximate index).
```dotenv theme={null}
VECTOR_DB_PROVIDER="turso"
VECTOR_DB_URL="/absolute/path/to/cognee.turso.db"
```
**Installation**: Install the Turso extra:
```bash theme={null}
pip install "cognee[turso]"
```
This pulls in `pyturso`. If the extra is missing, selecting `VECTOR_DB_PROVIDER="turso"` raises an `ImportError` at engine creation with the install hint.
**Local files only**: Remote Turso databases are not supported. A `libsql://`, `http(s)://`, or `ws(s)://` value in `VECTOR_DB_URL` raises an error at engine creation, and `VECTOR_DB_KEY` is ignored. Any other value is treated as a local file path.
**Journal mode**: `TURSO_JOURNAL_MODE` is `wal` by default. Set it to `mvcc` to let concurrent writers commit in parallel; see [Turso Local](/guides/turso-local) for the trade-offs.
**Multi-user mode**: Setting `VECTOR_DB_PROVIDER="turso"` automatically selects the `turso` dataset database handler (you do not need to set `VECTOR_DATASET_DATABASE_HANDLER` yourself). With `ENABLE_BACKEND_ACCESS_CONTROL=True`, each dataset gets its own Turso database file named `{dataset_id}.turso.db` under `/databases/{user_id}/`. Deleting a dataset evicts its cached engine and removes that file.
Qdrant requires a running instance of the Qdrant server.
```dotenv theme={null}
VECTOR_DB_PROVIDER="qdrant"
VECTOR_DB_URL="http://localhost:6333"
```
**Installation**: Since Qdrant is a community adapter, you have to install the community package:
```bash theme={null}
pip install cognee-community-vector-adapter-qdrant
```
**Configuration**: To make sure Cognee uses Qdrant, you have to register it beforehand with the following line:
```python theme={null}
from cognee_community_vector_adapter_qdrant import register
register()
```
For more details on setting up Qdrant, visit the [more detailed description](/setup-configuration/community-maintained/qdrant) of this adapter.
**Docker Setup**: Start the Qdrant service:
```bash theme={null}
docker run -p 6333:6333 -p 6334:6334 \
-v "$(pwd)/qdrant_storage:/qdrant/storage:z" \
qdrant/qdrant
```
**Access**: Default port is 6333 for the database, and you can access the Qdrant dashboard at "localhost:6333/dashboard".
Redis can be used as a vector store through the Redis Search module, providing fast vector similarity search capabilities.
```dotenv theme={null}
VECTOR_DB_PROVIDER="redis"
VECTOR_DB_URL="redis://localhost:6379"
# VECTOR_DB_KEY is optional and not used by Redis
```
**Installation**: Since Redis is a community adapter, you have to install the community package:
```bash theme={null}
pip install cognee-community-vector-adapter-redis
```
**Configuration**: To make sure Cognee uses Redis, you have to register it beforehand with the following line:
```python theme={null}
from cognee_community_vector_adapter_redis import register
register()
```
You can also configure Redis programmatically:
```python theme={null}
from cognee import config
config.set_vector_db_config({
"vector_db_provider": "redis",
"vector_db_url": "redis://localhost:6379",
})
```
For more details on setting up Redis, visit the [more detailed description](/setup-configuration/community-maintained/redis) of this adapter.
**Docker Setup**: Start a Redis instance with Search module enabled:
```bash theme={null}
docker run -d --name redis -p 6379:6379 redis:8.0.2
```
Or use **Redis Cloud** with the Search module enabled: [Redis Cloud](https://redis.io/try-free)
**Connection URL Examples**:
* Local: `redis://localhost:6379`
* With authentication: `redis://user:password@localhost:6379`
* With SSL: `rediss://localhost:6380`
ChromaDB support is optional and may not be installed in your Cognee environment by default.
```dotenv theme={null}
VECTOR_DB_PROVIDER="chromadb"
VECTOR_DB_URL="http://localhost:8000"
VECTOR_DB_KEY=""
# ChromaDB does not currently support Cognee backend access control dataset routing.
ENABLE_BACKEND_ACCESS_CONTROL="False"
```
**Installation**: Install ChromaDB support before configuring `VECTOR_DB_PROVIDER=chromadb`:
```bash theme={null}
pip install "cognee[chromadb]"
```
If you are using ChromaDB through a community adapter package instead of a Cognee extra, install that adapter package and call its `register()` function before running Cognee vector operations.
**Docker Setup**: Start a ChromaDB server:
```bash theme={null}
docker run -p 8000:8000 chromadb/chroma
```
FalkorDB can serve as both graph and vector store, providing a hybrid solution.
```dotenv theme={null}
VECTOR_DB_PROVIDER="falkor"
VECTOR_DB_URL="localhost"
VECTOR_DB_PORT="6379"
```
**Installation**: Since FalkorDB is a community adapter, you have to install the community package:
```bash theme={null}
pip install cognee-community-hybrid-adapter-falkor
```
**Configuration**: To make sure Cognee uses FalkorDB, you have to register it beforehand with the following line:
```python theme={null}
from cognee_community_hybrid_adapter_falkor import register
register()
```
For more details on setting up FalkorDB, visit the [more detailed description](/setup-configuration/community-maintained/falkordb) of this adapter.
**Docker Setup**: Start the FalkorDB service:
```bash theme={null}
docker run -p 6379:6379 -p 3000:3000 -it --rm falkordb/falkordb:edge
```
**Access**: Default ports are 6379 (DB) and 3000 (UI).
Use Amazon Neptune Analytics as a hybrid vector + graph backend.
```dotenv theme={null}
VECTOR_DB_PROVIDER="neptune_analytics"
VECTOR_DB_URL="neptune-graph://"
# AWS credentials via environment or default SDK chain
# Required — no per-dataset database handler exists for Neptune Analytics
ENABLE_BACKEND_ACCESS_CONTROL="false"
```
**Installation**: Install Neptune extras:
```bash theme={null}
pip install "cognee[neptune]"
```
**Note**: URL must start with `neptune-graph://` and AWS credentials should be configured via environment variables or AWS SDK.
**Access control must be off.** Cognee registers no per-dataset database handler for Neptune Analytics, so with `ENABLE_BACKEND_ACCESS_CONTROL` left at its default the handler stays `lancedb` and the first dataset access raises `The selected vector dataset to database handler does not work with the configured vector database provider`.
## Important Considerations
Ensure `EMBEDDING_DIMENSIONS` matches your vector store collection/table schemas:
* PGVector column size
* LanceDB Vector size
* ChromaDB collection schema
A collection's vector width is fixed when the collection is created, so all of a dataset's vectors must come from one embedding model. Changing dimensions requires recreating collections.
**Cognee records the model per dataset and checks it once.** In multi-user mode (`ENABLE_BACKEND_ACCESS_CONTROL="true"`, the default) every dataset has a registry row, and Cognee writes the resolved embedding model and its width into that row's `vector_database_connection_info` JSON as two keys:
```json theme={null}
{
"embedding_model": "openai/text-embedding-3-large",
"embedding_dimensions": 3072
}
```
The record is written when the row is created — that is, when the dataset's databases are first provisioned — and the model is resolved exactly as the embedding engine resolves it, so a keyless install records the local `fastembed` model it actually embeds with rather than the stock config default.
The comparison then happens **once per dataset-context entry**, against the already-loaded row and after the dataset's vector configuration is bound. There is no extra call to the vector store on ordinary operations. Only the recorded width is compared, not the model name, so a swap to a different model of the same width passes the check — the vectors stay in place and recall degrades silently, as it did before.
**On a mismatch you get a `409`.** If the configured model's width differs from the recorded one, Cognee raises `EmbeddingDimensionMismatchError` (HTTP `409`) before any write or query reaches the store, naming the dataset, both models, and both widths:
```
Dataset 5c1e… was embedded with 'openai/text-embedding-3-large' (3072 dimensions),
but the configured embedding model 'openai/text-embedding-3-small' produces 1536.
A dataset's vectors must all come from one model.
```
Remediation is either of:
* Set `EMBEDDING_MODEL` back to the model that built the dataset, to keep using it as is.
* To move the dataset to the new model: restore the old `EMBEDDING_MODEL`, delete the dataset (`await cognee.forget(dataset="...")` or `cognee-cli forget`), then set the new model and ingest again. Deleting enters the same dataset context, so it raises the same `409` while the mismatch stands — the old value has to be back before `forget` will run. A full `await cognee.prune.prune_system()` goes through the store handlers instead of the dataset context and is not affected.
This replaces the store-level failure — an Arrow cast error or `expected N dimensions, not M`, which named neither the dataset nor the model — for datasets that carry the record. See [How do I determine EMBEDDING\_DIMENSIONS?](/setup-configuration/embedding-providers#how-do-i-determine-embedding_dimensions) for picking the value in the first place.
**Datasets created before the record existed are completed once from the store.** When a row carries no recorded width, Cognee asks the vector store for the width already in it — `get_stored_vector_size()`, implemented by LanceDB and PGVector — and records that width permanently (with `embedding_model` left `null`, since the model that built it is unknown; the error then reports it as "an earlier embedding model"). An empty store adopts the configured model instead, and a provider that does not implement the method is left to the store's own errors. If such a store holds two widths — the state a live `EMBEDDING_MODEL` change used to leave behind, where old collections keep the old width and new ones get the new one — the width most collections use is reported, the smaller one on a tie, and a warning is logged naming every width found. Re-embed that dataset to make it consistent.
**Single-user mode is unchanged.** With `ENABLE_BACKEND_ACCESS_CONTROL="false"` there is no per-dataset registry row, nothing is recorded, and an embedding-model change still surfaces as the vector store's own dimension error on first write.
**Pruning clears the record.** A vector prune that keeps the registry rows (`await cognee.prune.prune_system(metadata=False)`) deletes the vectors the record described, so it also drops the two recorded keys; the default `await cognee.prune.prune_system()` additionally drops the relational database and the rows with it. Either way the next use of each dataset records whatever model builds it anew, so prune-and-re-ingest remains a valid way to move a deployment to a new embedding model — and unlike `forget`, it is not blocked by the mismatch it resolves.
PGVector does **not** use a single `embeddings` table. Cognee creates one table per indexed field, named `{DataPointType}_{field}` — for example `DocumentChunk_text`, `Entity_name`, `EntityType_name`, and `TextSummary_text`. Each of these collection tables has exactly three columns:
| Column | Type | Description |
| - | - | - |
| `id` | `uuid` (primary key) | Matches the corresponding node id in the [graph store](/setup-configuration/graph-stores) |
| `payload` | `json` | Serialized data point — includes the embedded `text` plus reference scalars such as `document_id`, `document_name`, `chunk_index`, `source_chunk_id`, and `belongs_to_set` |
| `vector` | `vector(N)` | The embedding itself; `N` is your `EMBEDDING_DIMENSIONS` / model vector size |
When PGVector shares the relational Postgres database, these collection tables live in the same schema as the snake\_case relational metadata and provenance tables (`data`, `datasets`, `dataset_data`, `nodes`, and `edges`). If you configure PGVector with separate `VECTOR_DB_*` settings, the collection tables live in that vector database instead. When using the [Postgres graph store](/setup-configuration/graph-stores), graph data is stored in `graph_node` and `graph_edge`.
Cognee distinguishes vector collections by their PascalCase first letter. Text content is stored in the JSON `payload` and in `TEXT` columns; there are no fixed-width `varchar(255)` columns (the relational provenance `nodes.label` / `nodes.type` columns were migrated from `varchar(255)` to `TEXT`).
Every upsert into a LanceDB table appends a new fragment and leaves the superseded rows and table versions on disk, and LanceDB never reclaims them on its own. Cognee therefore compacts local LanceDB stores automatically. Nothing needs to be configured or deployed.
**When it runs.** Once per dataset at the end of every successful `cognify()` run, and so after every `remember()` that cognifies. A pass has two steps:
1. **Merge small fragments** into larger ones (up to 20,000 rows each). This step holds the vector write lock, because its commit would conflict with a concurrent upsert or delete.
2. **Prune superseded table versions.** A version is deleted only after the version that replaced it is older than a **300-second retention window**, so a reader that opened a table a moment ago keeps its snapshot. Pruning commits nothing and does not take the write lock, so writers are never blocked by it.
**Each pass is bounded.** One pass runs at most **4 compaction tasks** and removes at most **1,000 old versions**, counted across all tables of the store, not per table. Tables written since the last pass go first, and a rotating cursor moves through the rest so no table is skipped forever. A store with a large backlog therefore shrinks over several cognify runs instead of stalling one. These limits are fixed in code; there are no environment variables for them.
**Space from a pass is freed later.** Files a pass replaces can only be deleted once the retention window has passed, so the next `cognify()` on that store reclaims them. Nothing prunes in the background: pruning runs only inside a pass, so no prune can outlive the adapter or run while it closes.
**Both LanceDB modes are covered.** With `VECTOR_DB_SUBPROCESS_ENABLED=true` (the default) the two steps run inside the existing LanceDB subprocess worker. With it off they run in a thread.
**When it does not run:**
* **Remote stores** (`db://`, `http://`, `https://`, `s3://`, `gs://`, `az://`) are skipped, because there every rewrite is network transfer and cleanup means thousands of object deletes.
* **pylance and lancedb do not match.** Compaction is turned off for the adapter's lifetime. See [LanceDB compaction turned off](#lancedb-compaction-turned-off) below.
* **A pass is already running** on the same adapter. The second one returns at once with `{"skipped": "in_progress"}` instead of waiting.
**It never fails your run.** Compaction is best-effort: an error is logged, and for a single table it is recorded in the pass's stats while the other tables continue. The cognify run that triggered it still succeeds.
| Provider | Setup | Performance | Use Case |
| - | - | - | - |
| LanceDB | Zero setup | Good | Local development |
| PGVector | Postgres required | Excellent | Production with Postgres |
| Turso | `cognee[turso]` extra | Good | Embedded local file, no server |
| Neptune Analytics | AWS required | Excellent | Cloud hybrid solution |
| ChromaDB | Server required | Good | Dedicated vector store |
| Qdrant | Server required | Excellent | High-performance vector search |
| Redis | Server required | Excellent | Low-latency in-memory search |
| FalkorDB | Server required | Good | Hybrid graph + vector |
## Troubleshooting
With `ENABLE_BACKEND_ACCESS_CONTROL="false"`, a vector-only prune against a PGVector store that shares the relational Postgres database fails with:
```text theme={null}
SharedDatabasePruneError: PGVector cannot be pruned independently while it shares
the relational PostgreSQL database. Use prune_system(metadata=True) to delete the
shared database (this also deletes users, datasets and permissions), or set
VECTOR_DB_NAME to a different, dedicated database.
```
Cognee refuses rather than dropping every table in a database that also holds `data`, `datasets`, `dataset_data`, users, and permissions. Because [`cognee.prune.prune_system()`](/python-api/prune#prune-prune_system) defaults to `metadata=False`, the bare call is the one that hits this.
Either delete the shared database as a whole, which also removes datasets, data records, users and permissions:
```python theme={null}
await cognee.prune.prune_system(metadata=True)
```
Or move the vector store into a dedicated database by setting all five `VECTOR_DB_*` values with a distinct `VECTOR_DB_NAME`, per the table in the PGVector setup section above. Nothing is deleted when the error is raised — the vector step runs before graph deletion so the refusal aborts the prune first.
This issue is most commonly reported with LanceDB-backed search workloads.
`VECTOR_DB_SUBPROCESS_ENABLED` defaults to `true`. In high-fanout searches — for example, many queries across many datasets — this can create a large number of OS subprocesses. On macOS, that can quickly hit the default file descriptor limit (`ulimit -n`, often `256`) and surface as:
```text theme={null}
OSError: [Errno 24] Too many open files
```
Because the exception often appears inside LanceDB internals, it may not be obvious that the underlying issue is subprocess and file-descriptor exhaustion rather than a LanceDB data problem.
If this happens:
```dotenv theme={null}
VECTOR_DB_SUBPROCESS_ENABLED=false
```
And raise the shell limit before starting Cognee:
```bash theme={null}
ulimit -n 4096
```
This is especially relevant for audit-style or multi-dataset search runs where Cognee fans queries out across several datasets in parallel.
When `VECTOR_DB_SUBPROCESS_ENABLED` is `true`, concurrent async RPCs to the LanceDB subprocess run in parallel rather than serializing behind a single session lock. Each request carries its own id and is routed back to its own waiter, so concurrent add and search operations no longer queue behind one another.
`SUBPROCESS_WORKER_MAX_INFLIGHT` (default `16`) bounds how many async operations a worker runs at once, keeping the worker's memory footprint predictable. The value must be `> 0`; a zero or negative value fails worker initialization with a `ValueError` rather than silently degrading. Raise it for higher concurrency (for example, `SUBPROCESS_WORKER_MAX_INFLIGHT=64`), or set a large value to effectively remove the cap.
A per-call timeout or cancellation now resolves only that individual call and leaves the subprocess session running for other in-flight and future calls. The session is torn down only on genuine session-ending events (worker crash, shutdown, or respawn), which propagate a `SubprocessTransportError` to any calls still pending. Synchronous calls (such as the Kuzu graph backend) are unaffected and continue to run serially.
Subprocess engines are also no longer closed the moment a dataset context exits. `SUBPROCESS_IDLE_TTL_SECONDS` (default `600`) keeps an idle engine warm, and a background reaper thread closes it only after a full TTL with no use — so repeated or interactive queries against the same dataset reuse the live worker instead of paying for a close plus a respawn. Datasets with an active dataset-queue slot are skipped by the reaper, and non-subprocess engines (PGVector and other remote stores) are never reaped.
The trade-off: a kept-warm worker holds its memory and its PID (and, for the graph store's embedded Kuzu/Ladybug worker, its database file lock — LanceDB takes no exclusive file lock) for up to the TTL after its last use, with the total number of retained idle workers still bounded by `DATABASE_MAX_LRU_CACHE_SIZE`. Lower the TTL, or set `SUBPROCESS_IDLE_TTL_SECONDS=0` to close engines at each dataset-context exit as before, if you are memory-tight or need locks released promptly. See [Subprocess engine teardown coordination](/setup-configuration/permissions#subprocess-engine-teardown-coordination) for the full lifecycle.
Cognee's [automatic LanceDB compaction](#lancedb-storage-compaction) uses `pylance` to rewrite the tables `lancedb` writes, so the two must be on the same Lance core release line. If they are not, compaction turns itself off for the lifetime of the adapter and logs one of these warnings:
```text theme={null}
LanceDB compaction turned off: . Install the pylance release line matching
lancedb's bundled Lance core (see pyproject).
LanceDB compaction turned off: pylance cannot open the tables lancedb wrote ().
Install the pylance release line matching lancedb's bundled Lance core.
```
Ingestion and search keep working, but the store is no longer compacted and keeps growing on disk. To fix it, install the pair Cognee pins in `pyproject.toml` (currently `lancedb>=0.39.0,<0.40.0` and `pylance>=12.0.0,<12.1.0`), for example by reinstalling Cognee instead of upgrading `pylance` or `lancedb` on its own, then restart the process.
Other log lines from compaction:
* `Vector store compaction: N collection(s), fragments a -> b, ...`: the summary of one pass, including how many tasks and versions were left for later runs. It is logged at `info` when the pass did work and at `debug` otherwise.
* `Vector store compaction skipped: ...` and `Compaction skipped for collection '...'`: warnings for a pass, or one table in it, that failed. They are not fatal, and the next cognify run tries again.
## Community-Maintained Providers
Additional vector stores are available through community-maintained adapters:
* **[Qdrant](/setup-configuration/community-maintained/qdrant)** — Vector search engine with cloud and self-hosted options
* **[Redis](/setup-configuration/community-maintained/redis)** — Fast vector similarity search
* **[FalkorDB](/setup-configuration/community-maintained/falkordb)** — Hybrid vector and graph store
* **[Pinecone](/setup-configuration/community-maintained/pinecone)** — Managed vector database (requires separate install + registration)
* **[Turbopuffer](/setup-configuration/community-maintained/turbopuffer)** — High-performance vector database
* **Milvus, Weaviate, and more** — See [all community adapters](/setup-configuration/community-maintained/overview)
## Notes
* **Embedding Integration**: Vector stores use your embedding engine from the Embeddings section
* **Dimension Matching**: Keep `EMBEDDING_DIMENSIONS` consistent between embedding provider and vector store
* **Performance**: Local providers (LanceDB) are simpler but cloud providers offer better scalability
Configure embedding providers for vector generation
Set up graph databases for knowledge graphs
Return to setup configuration overview
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.