Skip to main content
Every value here resolves as defaults < config.json < .env < env < code (precedence), and can be set from code with the config API or cognee-cli config.

​
LLM

The llm section, registered by the LLM kind and shared by every LLM adapter (see Component config sections). Each key also answers to its Python flat name, the alias shown in brackets. The deep reference is CLI: LLM retries for retries and the cognee-llm rustdoc for the adapter. Adapter sections, registered by their factory and present only when it is: LLM_MAX_COMPLETION_TOKENS is a completion-token budget, and it does double duty: besides capping the option-less generate call, it is the LLM term in cognify’s chunk-size auto-calculation (min(<embedding engine limit>, llm_max_completion_tokens / 2) — Python’s get_max_chunk_tokens()). Lowering it for a small-context endpoint therefore shrinks the prompts cognify sends. On adapters that know their model’s documented output cap (Anthropic, Bedrock) the configured value is clamped to it first, as Python does via litellm’s model_cost.
Structured-output calls do not apply this ceiling — option-less ones send the built-in 16384 default. Cognify’s extraction calls send no cap at all (Python parity).
LLM_ARGS is the escape hatch for that last case. It takes a JSON object whose keys are merged into each chat and structured-output request body — the port of Python’s llm_config.llm_args and its {**self.llm_args, **kwargs} merge, so an explicitly-passed option always wins over an LLM_ARGS key of the same name. Its practical use is giving the extraction path more output headroom: because those calls send no max_tokens, the provider’s own default applies, and a low one (Baseten’s is 4096) truncates the structured JSON mid-object. The adapter detects that case (finish_reason: "length") and re-asks at a raised budget, but starting from a sufficient budget avoids the wasted first attempt.
Three details worth knowing. A bare max_tokens is folded into max_completion_tokens for reasoning models, and sampling params those models reject (temperature, top_p, frequency_penalty, presence_penalty) are dropped rather than passed through. A malformed value, or valid JSON that is not an object, is ignored with a warn! instead of aborting startup. And the vision/transcription path does not see it at all: it bypasses the merge. LLM_TEMPERATURE reaches every built-in provider (previously only Bedrock applied it). The OpenAI-compatible, Azure and Anthropic factories fold it into LLM_ARGS as temperature, as Python’s fold_sampling_params_into_llm_args does, so a temperature in LLM_ARGS itself wins. It is sent on every call that does not pick a temperature itself: requests whose options carry none (graph extraction) and option-less calls, plain or structured (search completions, feedback detection), which otherwise send 0.0. A temperature a call picks (summarization’s 0.3) wins, and the OpenAI-compatible and Azure adapters never send one to a reasoning model, which rejects it. Bedrock sends it as inferenceConfig.temperature on the same calls; an LLM_ARGS temperature is not used for it there, but passed through unchanged in additionalModelRequestFields like the rest of LLM_ARGS. Unset, these providers send a temperature from LLM_ARGS if it has one, on option-less calls too (where the built-in 0.0 used to shadow it). Otherwise extraction sends none and option-less calls send 0.0. Unlike Python, Rust does not pin 0.0 for local inference servers when the variable is unset. LLM_STREAMING has no effect. Python’s llm_streaming is likewise unread by its OpenAI and Azure adapters, and no Rust adapter streams: the Llm trait has no token-streaming call for a stream to reach. A fallback LLM (llm.fallback_provider/_model/_endpoint/_api_key, aliases llm_fallback_*) is configurable programmatically (no env binding) but not used yet. MOCK_LLM + cassettes power the offline benchmark — see performance/mock-benchmark.md.

​
Supported LLM providers

Several providers are OpenAI-compatible HTTP endpoints, so they route through the same adapter — differing only in base URL and litellm-style model-prefix stripping. LLM_API_KEY is required for every provider except bedrock (matching the Python SDK; for a local Ollama any non-empty value works). On bedrock an unset or empty LLM_API_KEY is a supported configuration — it falls through to SigV4 signing rather than erroring (see Bedrock below). The OpenAI-only request quirks are gated on the api.openai.com host, so they never fire against another endpoint. LLM_ENDPOINT always overrides the default when set on the OpenAI-compatible providers. Audio transcription (Whisper) is wired only for openai and custom/openai_compatible (which may expose /audio/transcriptions); ollama/mistral/gemini/bedrock get graceful no-audio. anthropic uses a native Messages-API adapter (not the OpenAI-compatible factory): it authenticates with x-api-key, hoists the system prompt into the top-level system field, and produces structured output via a forced tool_use. It requires LLM_API_KEY, strips an anthropic/ model prefix, honors LLM_ARGS, and supports image description (vision) but not audio transcription. The endpoint defaults to https://api.anthropic.com/v1 and deliberately ignores LLM_ENDPOINT/OPENAI_URL so a stray OPENAI_URL cannot misroute Anthropic traffic (matching Python, which passes no base URL). To route Anthropic traffic through a proxy / gateway, set the dedicated ANTHROPIC_BASE_URL (alias ANTHROPIC_API_BASE); other providers ignore it. azure reuses the OpenAI request path with Azure’s auth and URL conventions: it authenticates with the api-key header and appends an ?api-version=<v> query. Set LLM_PROVIDER=azure, LLM_API_KEY, LLM_API_VERSION (e.g. 2024-12-01-preview), and LLM_ENDPOINT to the deployment URL (https://<resource>.openai.azure.com/openai/deployments/<deployment>). Azure ignores the model in the request body for routing, but LLM_MODEL still selects the reasoning-model parameter shape, so for an o-series/gpt-5 deployment set LLM_MODEL to the underlying model name (a .../deployments/o3 style segment is also treated as reasoning), or pin it with LLM_REASONING.

​
Bedrock

LLM_PROVIDER=bedrock is a native adapter over Amazon Bedrock’s Converse API (POST {endpoint}/model/{modelId}/converse), not an OpenAI-compatible endpoint. Its AWS settings live in the aws config section (registered by the Bedrock LLM and embedding factories), so they can be set like any other section key: with set, in a config file or through the bindings, as aws.access_key_id or its flat alias aws_access_key_id, and so on. A field left unset there falls back to the AWS_* environment variable of the same name inside the adapter. The region has no env binding in the section (aws.region, alias aws_region_name); the adapter applies the env rungs of the region chain itself. The env names mirror the ones litellm reads, and several of them are not the boto3-standard spellings: Auth ladder. LLM_API_KEY, when set, is sent verbatim as Authorization: Bearer <key> and the request is not signed — no credential lookup runs, so a bearer-only deployment needs no AWS credentials. Otherwise AWS_BEARER_TOKEN_BEDROCK takes the same short-circuit. Only when both are absent does the request fall through to SigV4 (signing service bedrock), whose credentials resolve in this order:
  1. web-identity token + role + session name → STS AssumeRoleWithWebIdentity;
  2. role ARN → STS AssumeRole — skipped when the process is already running as that role, so an IRSA pod that already holds it does not assume itself;
  3. profile name → the shared-config profile;
  4. static access key + secret (plus session token when present);
  5. otherwise the default chain (env / shared config / SSO / ECS / IMDS).
Region and endpoint. The region resolves as: an explicitly configured region → a region embedded in a model ARN → AWS_REGION_NAME → AWS_REGION → the profile / default chain → the hard default us-west-2. The endpoint resolves as: an explicit API base → AWS_BEDROCK_RUNTIME_ENDPOINT → https://bedrock-runtime.{region}.amazonaws.com. LLM_ENDPOINT is not that API base: like the Anthropic path, the Bedrock factory deliberately passes no base URL, so a stray LLM_ENDPOINT cannot misroute Bedrock traffic. Use AWS_BEDROCK_RUNTIME_ENDPOINT for a custom runtime host. Converse only. Every model cognee ships routes to Converse after the model id is normalised (an ARN wrapper is unwrapped; cross-region inference prefixes such as eu./us./apac. and provisioned-throughput / context-window suffixes are stripped). Normalisation feeds routing and capability decisions only; the request URL always carries the model id exactly as configured. A model that routes to Bedrock’s legacy invoke API for chat is unsupported: construction fails with LlmError::FeatureNotSupported. (Embeddings are a separate path and do go over InvokeModel — see Embedding.) Capabilities. No streaming. No audio transcription. Image description (vision) is implemented over Converse image blocks — deliberately exceeding Python parity — but gated on a per-model capability table (caps.rs): an unlisted model is treated as text-only and returns LlmError::FeatureNotSupported. Structured output is capability-gated from the same table: models that advertise native structured output get Converse’s outputConfig.textFormat carrying the JSON schema, the rest get a synthetic json_tool_call tool (with toolChoice forced only for models that support it). Feature gating. The adapter and the embedding engine live behind the bedrock Cargo feature. It is non-default on the two leaf crates that carry the code (cognee-llm, cognee-embedding), but it is in the default feature list of cognee-components, cognee and cognee-cli (and of cognee-http-server), so every shipped binary has Bedrock available. Building with --no-default-features (or without bedrock) drops the whole AWS stack, and LLM_PROVIDER=bedrock then reports an unsupported provider.

​
Reasoning-model parameters

OpenAI reasoning families (o1*, o3*, o4*, gpt-5*) require a different request shape — max_completion_tokens instead of max_tokens, and no temperature/top_p/penalties. The adapter auto-detects this from the model name for any non-local endpoint (loopback and Ollama’s port keep the legacy shape). LLM_REASONING overrides that detection:
  • auto (default) — name/host auto-detection as above.
  • always — force the reasoning shape (e.g. a proxy that hides a reasoning model behind an opaque alias).
  • never — force the legacy max_tokens + temperature shape (e.g. a gateway serving a reasoning-named model that only accepts the legacy parameters).

​
Structured-output request shape

Nothing in the OpenAI-compatible protocol advertises which structured-output shape a server understands, so by default the adapter cascades: native tools, then legacy functions, then response_format: {"type": "json_object"}. The first shape that yields a parseable payload wins. The adapter keeps a per-mode miss counter and stops sending a mode that has produced nothing three calls running (see How retries and pacing interact). LLM_STRUCTURED_OUTPUT_MODE skips the discovery when you already know the answer:
  • auto (default) — cascade as above, bounded by the miss counter.
  • json_schema — try constrained decoding first, then cascade. See below.
  • tools — only native tool-calling.
  • functions — only the legacy functions/function_call pair.
  • json — only JSON mode.
Pick json when the server rejects tools/functions outright. If it merely ignores them — accepting the request and echoing JSON in content — leave it on auto: tool-calling mode already succeeds there in one request, and it is the only mode that puts the real JSON schema on the wire (JSON mode sends a prose example derived from the schema, which means weaker schema pressure). Pinning a mode means the other modes are never sent, and exhausting the pinned mode fails the call rather than falling through — the error names the pin. An unrecognised value falls back to auto.
Python’s LLM_INSTRUCTOR_MODE is not an alias for this. It covers similar ground but does not mean the same thing, so Rust does not read it. If a shared .env sets LLM_INSTRUCTOR_MODE, translate it to LLM_STRUCTURED_OUTPUT_MODE yourself.
The two differ in four ways:
  • When it applies. Python reads llm_instructor_mode only on its instructor structured-output framework (STRUCTURED_OUTPUT_FRAMEWORK=instructor). The default framework, litellm_native, ignores it. Aliasing it would make a value that does nothing in Python change behaviour in Rust.
  • Values. Python takes an instructor.Mode value: json_schema_mode, tool_call, tools_strict, markdown_json_mode, anthropic_tools, mistral_tools and so on, many of them specific to one provider. Only json_mode and function_call are also spelled the same in Rust; anything else falls back to auto here.
  • Empty value. Empty means the provider’s entry in Python’s static table (instructor_modes.py, for example json_schema_mode for OpenAI). In Rust, auto means the cascade.
  • Scope and pinning. Python’s mode applies to every adapter, and json_schema_mode is a pin. Rust’s knob applies only to the OpenAI-compatible and Azure providers (not Anthropic or Bedrock), and json_schema is not a pin (see below).
Rough translation: json_schema_mode → json_schema; tool_call / tools_strict / parallel_tool_call → tools; function_call → functions; json_mode / markdown_json_mode → json. json_schema — constrained decoding. json_schema is not a pin. It sends
ahead of the cascade, and falls back to it when the endpoint says no. On a backend that supports it the sampler cannot emit a token that would break the schema. It is opt-in because there is no capability table for an arbitrary OpenAI-compatible base URL, and some backends (Baseten’s gpt-oss-120b) answer HTTP 501 to a constrained request. The mode demotes rather than failing:
  1. strict: true with the all-required / additionalProperties: false schema;
  2. on rejection, the same envelope without strict, carrying the ordinary schema;
  3. on rejection again, out of the mode entirely: the tools → functions → json cascade runs exactly as under auto.
Only an outright refusal of the request shape moves that ladder — HTTP 501, or an HTTP 400 whose body names response_format / json_schema / schema / strict. Rate limits, other 5xx, network errors, model refusals and unrelated 400s leave the (process-lifetime, monotonic) memo untouched. The probe costs one request and is single-flighted per schema. A schema whose root is not an object never enters the mode, and a schema using a feature outside OpenAI’s strict subset (a map field, patternProperties, if/then/else) is refused and demoted. The Bedrock adapter does not read this knob; it picks Converse’s native outputConfig.textFormat.jsonSchema or a synthetic json_tool_call from its own capability table.
Ollama embeddings: set EMBEDDING_ENDPOINT explicitly when using EMBEDDING_PROVIDER=ollama. The Ollama embedder needs the /api/embed route, and when EMBEDDING_ENDPOINT is unset the embedding endpoint falls back to LLM_ENDPOINT, which for Ollama points at the /v1 chat base, so the embedder would POST to the wrong path. Set EMBEDDING_ENDPOINT=http://localhost:11434/api/embed.

​
Embedding

The embedding section is registered by the embedding kind and shared by every provider. Each provider’s factory registers the settings only it reads (see Component config sections). The Python flat name of each key is accepted as an alias, shown in brackets. Provider sections, registered by the provider’s factory and present only when it is compiled in: Provider values: onnx, fastembed, openai, openai_compatible, ollama, bedrock, mock, plus any provider a custom registry adds with ComponentRegistry::register_embedding. A provider whose factory is not registered (a misspelling, or a backend compiled out) is an error, never a silent ONNX fallback. MOCK_EMBEDDING diverges from the Python SDK, deliberately. Python’s embedding engines return [0.0] * dimensions under a truthy MOCK_EMBEDDING. Rust defaults to SHA-256-derived vectors instead, because zero vectors are silently discarded by cosine-distance KNN (the distance is NaN, and LanceDB filters NaN rows out), making retrieval vacuously empty. The mock engine is test-only. Set MOCK_EMBEDDING=zero to reproduce Python’s behaviour. EMBEDDING_PROVIDER=bedrock uses Amazon Bedrock’s InvokeModel API (POST {endpoint}/model/{modelId}/invoke) — Converse is never used here. Two model families are supported, selected from the normalised model id: Titan (amazon.titan-embed-text-v1, amazon.titan-embed-text-v2:0, amazon.titan-embed-image-v1), which takes one text per request, and Cohere (cohere.embed-*), which batches. Set EMBEDDING_MODEL to the Bedrock model id; EMBEDDING_DIMENSIONS drives the engine’s reported dimension() and is sent as dimensions on Titan v2, so it must match what the model returns. AWS configuration comes from the same AWS_* variables as the LLM path (see Bedrock), and EMBEDDING_API_KEY (falling back to LLM_API_KEY) is the Bedrock API key, which short-circuits to a bearer header with no SigV4; leaving it unset runs the ambient credential ladder. Returned vectors are always L2-normalised (server-side on Titan v2 via normalize: true, client-side for the others).
Bedrock embeddings and EMBEDDING_ENDPOINT: through Settings, EMBEDDING_ENDPOINT defaults to empty, and a Bedrock embedder inherits LLM_ENDPOINT only when the LLM provider is also bedrock. Direct users of EmbeddingConfig::default() start from the OpenAI base URL instead; the Bedrock engine ignores an endpoint that points at api.openai.com rather than POSTing Bedrock bodies there. Set it only to a real Bedrock runtime host (or use AWS_BEDROCK_RUNTIME_ENDPOINT).