Structured Output Backends

Structured output backends ensure reliable data extraction from LLM responses. Cognee supports three frameworks that convert LLM text into structured Pydantic models for knowledge graph extraction and other tasks.

New to configuration?See the Setup Configuration Overview for the complete workflow:install extras → create .env → choose providers → handle pruning.

Supported Frameworks

Cognee supports three structured output approaches:

LiteLLM Native — Validates responses into Pydantic models using LiteLLM’s own response_format, without the instructor dependency (default)
LiteLLM + Instructor — Provider-agnostic client with Pydantic coercion (opt-in)
BAML — DSL-based framework with type registry and guardrails (opt-in)

All three frameworks produce the same Pydantic-validated outputs, so your application code remains unchanged regardless of which backend you choose.

How It Works

Cognee uses a unified interface that abstracts the underlying framework:

from cognee.infrastructure.llm.LLMGateway import LLMGateway
await LLMGateway.acreate_structured_output(text, system_prompt, response_model)

The STRUCTURED_OUTPUT_FRAMEWORK environment variable determines which backend processes your requests, but the API remains identical. It defaults to litellm_native, so instructor is no longer in the default call path — set the variable explicitly to opt back into instructor or into baml.

Configuration

LiteLLM + Instructor
BAML
LiteLLM Native (Default)

An opt-in framework — no extra install needed, but you must select it explicitly since the default is litellm_native. Uses LiteLLM and the instructor library to coerce LLM responses into Pydantic models.

STRUCTURED_OUTPUT_FRAMEWORK=instructor

Optionally, control how the model is prompted for structured output:

# Override instructor mode (e.g. json_mode, tool_call, markdown_json_mode)
# Leave unset to use the provider's default — see "Instructor Modes" below.
LLM_INSTRUCTOR_MODE=json_schema_mode

BAML is an alternative structured output framework that uses a DSL-based type registry to extract data. It is particularly useful when small local models (such as Ollama models like llama3.1:8b or qwen3.5:0.8b) struggle to produce valid structured output with instructor, causing repeated InstructorRetryException errors.Installation: BAML requires a separate install:

pip install "cognee[baml]"

Configuration: BAML uses its own LLM settings, independent of the main LLM_* variables:

STRUCTURED_OUTPUT_FRAMEWORK=baml

# BAML-specific LLM settings (required)
BAML_LLM_PROVIDER=openai
BAML_LLM_MODEL=gpt-4o-mini
BAML_LLM_API_KEY=sk-...

# Optional BAML overrides
# BAML_LLM_ENDPOINT=https://api.openai.com/v1
# BAML_LLM_API_VERSION=
# BAML_LLM_TEMPERATURE=0.0

BAML_LLM_PROVIDER and BAML_LLM_MODEL accept the same provider names and model identifiers as the main LLM configuration. You can point BAML at a different model than your main LLM — for example, use a small Ollama model for general text generation while routing structured extraction through a cloud model.

If STRUCTURED_OUTPUT_FRAMEWORK=baml is set but the cognee[baml] extra is not installed, Cognee will raise an ImportError on startup. Run pip install "cognee[baml]" to resolve it.

Using BAML for Small Local Models

Small Ollama models (e.g. llama3.1:8b, qwen3.5:0.8b) often fail to produce valid JSON-structured output when using the instructor backend, resulting in repeated InstructorRetryException errors during cognify for types like KnowledgeGraph or SummarizedContent.Switching to BAML bypasses instructor entirely and uses BAML’s own extraction pipeline, which is more forgiving with smaller models:

# Main LLM — small local model via Ollama
LLM_PROVIDER=ollama
LLM_MODEL=llama3.1:8b
LLM_ENDPOINT=http://localhost:11434
LLM_API_KEY=ollama

# Use BAML for structured extraction (can point to a different, more capable model)
STRUCTURED_OUTPUT_FRAMEWORK=baml
BAML_LLM_PROVIDER=openai
BAML_LLM_MODEL=gpt-4o-mini
BAML_LLM_API_KEY=sk-...

LLM_ENDPOINT is the bare Ollama host, with no /v1 suffix. With STRUCTURED_OUTPUT_FRAMEWORK=baml it is not used for extraction at all — BAML builds its own client from the BAML_LLM_* settings — and the bare host is the form the default litellm_native backend needs if you turn BAML off, because LiteLLM routes Ollama over its native API and appends /api/generate itself. Add /v1 only for STRUCTURED_OUTPUT_FRAMEWORK="instructor", which talks to Ollama’s OpenAI-compatible API instead. See LLM Providers → Ollama for the full configuration.See the Local Setup guide for a complete Ollama configuration including embeddings.

LiteLLM Native is the default framework. It validates responses into Pydantic models using LiteLLM’s own response_format, without the instructor library. No extra install is needed beyond the base package, and no configuration is needed to use it — set the variable only if you want to pin the value explicitly:

STRUCTURED_OUTPUT_FRAMEWORK=litellm_native

It reuses the standard LLM_* settings (provider, model, API key, endpoint) — no separate configuration block. Behavior depends on the provider:

Schema-native path — For providers LiteLLM reports as schema-capable (OpenAI, Azure, Gemini, Mistral, Bedrock, and others), your Pydantic model is passed directly as response_format and the returned JSON is validated against it. This path has two tiers, strict and non-strict (see below).
JSON-object fallback — For providers without native schema support (Ollama, llama.cpp, custom OpenAI-compatible endpoints), Cognee requests a JSON object, injects the model’s JSON Schema into the prompt, and validates the result. Before validating, Cognee unwraps a markdown code fence if one wraps the whole reply — models on this path routinely answer with ```json … ``` even when asked not to. The unwrap is anchored to the entire payload, so JSON that merely contains backticks inside a value is left untouched. On a validation failure it feeds the error back to the model and retries so it can self-correct, up to 3 attempts before raising the last validation error.

Routing between the two paths is automatic per model (via litellm.supports_response_schema); you don’t select the path yourself.Strict and non-strict tiers on the schema-native path. Cognee first asks for strict schema enforcement, where the provider constrains decoding to your schema. Providers only enforce a restricted subset of JSON Schema, so a model they cannot enforce is rejected with a schema-related 400 — typically a discriminated union (which emits oneOf/discriminator) or a free-form dict field (which OpenAI 400s with 'additionalProperties' is required to be supplied and to be false). Rather than giving up on the native path there, Cognee retries the same request once with a non-strict json_schema payload: the raw model_json_schema() travels as guidance rather than as a decoding constraint, so those constructs are accepted, and conformance is still checked app-side by validating against your original Pydantic model. Only if the non-strict payload is also rejected does Cognee fall through to the JSON-object fallback.The demotion is remembered for the process. The (model, response model) pair is recorded in a module-level set, so the rejected strict request is paid once per process instead of on every call — later calls for that pair skip the strict attempt and go straight to non-strict. Models whose schema strict mode accepts (the majority, including KnowledgeGraph extraction) keep their strict, grammar-constrained guarantee untouched. Both tiers log a warning naming the model and response model when they are rejected.Output that comes back from either tier but fails Pydantic validation also routes to the JSON-object fallback, which has the error-feedback retry loop, rather than bubbling into the transport-level retry that would re-send the same prompt with no correction.

Neither the tiering nor the demotion cache has a configuration knob or environment variable — there is no setting that forces strict mode or disables the non-strict retry. The cache lives in process memory only, so it starts empty on every restart.

Audio and image transcription. The native adapter is not limited to structured output — it also backs create_transcript and transcribe_image, the two media entry points LLMGateway dispatches. On the default framework instructor is therefore absent from the entire gateway call path, not just from structured extraction. Setting STRUCTURED_OUTPUT_FRAMEWORK=instructor routes both media entry points back to the legacy per-provider adapters; baml has no transcription of its own and keeps using those adapters too.Audio goes to litellm.atranscription against TRANSCRIPTION_MODEL — a real speech-to-text endpoint — for every provider. OpenAI and Azure already worked that way and are unaffected. The change is visible on the others, whose legacy adapters each take their own route: Anthropic, Gemini, and Custom send a chat completion carrying the audio as an inline base64 file part; Ollama calls its OpenAI-compatible transcription endpoint; Mistral goes through the Mistral SDK. Bedrock and llama.cpp gain rather than change a path: their legacy adapters raise NotImplementedError for both audio and images, so media ingestion that used to fail outright now runs on the default framework. STRUCTURED_OUTPUT_FRAMEWORK=instructor still takes those provider-specific routes, but each now sends audio to TRANSCRIPTION_MODEL. Previously Anthropic, Gemini, Mistral, and Custom sent it to the chat model, and Ollama hardcoded whisper-1, ignoring TRANSCRIPTION_MODEL. To keep the chat-completion route on Anthropic, Gemini, or Custom, set TRANSCRIPTION_MODEL to a multimodal chat model; the default whisper-1 is not one.Transcription keyword arguments pass straight through to LiteLLM, so callers can still request response_format="verbose_json" and timestamp_granularities and read the segments back from the raw provider response — which is what the inline [HH:MM:SS] markers described in Loaders depend on. Images go through a multimodal completion against IMAGE_TRANSCRIBE_MODEL, or the chat model (LLM_MODEL) when that is unset, and fall back to the same "What's in this image?" prompt and 300-token cap the legacy adapters used when a caller passes neither. ImageLoader still passes its own IMAGE_TRANSCRIPTION_* overrides, so the image transcription settings behave identically.Model-name qualification. LiteLLM routes on a provider-qualified model name, while Cognee keeps provider and model as separate settings — so LLM_PROVIDER="ollama" with LLM_MODEL="phi4", a configuration that is valid for instructor, would reach LiteLLM as a bare phi4 and fail with litellm.BadRequestError: LLM Provider NOT provided before a request is even sent. On this path Cognee first asks LiteLLM to resolve LLM_MODEL on its own, and only when that lookup fails does it prefix the name with the configured provider:

`LLM_MODEL` (with `LLM_PROVIDER="ollama"`)	Sent to LiteLLM
`phi4`	`ollama/phi4`
`library/phi4`	`ollama/library/phi4`
`hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF`	`ollama/hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF`
`ollama/phi4`	`ollama/phi4` — already resolvable, untouched

Prefixing applies to ollama, anthropic, gemini, mistral, and bedrock. openai and azure are deliberately excluded: LiteLLM already resolves bare OpenAI model names, and Azure needs a deployment-specific form Cognee must not guess at. A slash does not by itself mean the name carries a provider — Ollama accepts namespaced tags and a GGUF pulled from Hugging Face keeps its full path — so slash-containing names go through the same check rather than being assumed routable. Anything LiteLLM already resolves is returned unchanged (gpt-4o, openai/gpt-5-mini, azure/gpt-4o-mini), so qualification cannot re-route a configuration that works today; the only cost is one extra local get_llm_provider lookup, with no network call. Any other provider is left as-is — if LiteLLM rejects its name, qualify LLM_MODEL yourself.TRANSCRIPTION_MODEL and IMAGE_TRANSCRIBE_MODEL are qualified by the same rule before they are handed to LiteLLM, so a provider-local speech-to-text or vision model needs no manual prefix either. On STRUCTURED_OUTPUT_FRAMEWORK=instructor, every legacy adapter that handles media (OpenAI, Azure, Anthropic, Gemini, Mistral, Ollama, and Custom) now receives both settings too — including Ollama, which previously ignored TRANSCRIPTION_MODEL and always sent the chat model for images.Qualification reads LLM_PROVIDER, so a namespaced model id needs the provider set explicitly. LLM_MODEL="library/phi4" on its own raises ProviderNotDeducibleError at configuration load, before qualification ever runs — library and hf.co are not provider prefixes Cognee recognises. See Provider Inference.

This is the framework Cognee uses when STRUCTURED_OUTPUT_FRAMEWORK is unset. Transient errors (including rate limits) are retried with backoff, while budget/quota exhaustion is surfaced as LLMPaymentRequiredError and is not retried. Content-policy violations fall back only when the standard fallback settings are configured (at minimum FALLBACK_MODEL and FALLBACK_API_KEY); otherwise Cognee raises ContentPolicyFilterError immediately.

Instructor Modes

When STRUCTURED_OUTPUT_FRAMEWORK=instructor, the instructor mode controls how Cognee asks the model for structured output — for example via the model’s native JSON-schema response, a plain JSON object, or a tool/function call. The value of LLM_INSTRUCTOR_MODE is passed directly to the instructor library’s Mode, so it must be one of instructor’s supported mode strings. LLM_INSTRUCTOR_MODE is empty by default. When it is unset, Cognee either applies a provider-specific mode or defers to the underlying Instructor/LiteLLM default, so in most cases you don’t need to set it at all:

`LLM_PROVIDER`	Behavior when `LLM_INSTRUCTOR_MODE` is unset
`openai`, `azure` with `gpt-5` models	`json_schema_mode`
`openai`, `azure` with other models	use Instructor/LiteLLM default
AWS Bedrock	`json_schema_mode`
`ollama`	`json_schema_mode`
`gemini`, `custom` (OpenAI-compatible), llama.cpp	`json_mode`
`anthropic`	`anthropic_tools`
`mistral`	`mistral_tools`

Common values you can set explicitly include json_schema_mode, json_mode, tool_call, and markdown_json_mode. Ollama’s default is json_schema_mode — it was json_mode in earlier Cognee versions. Ollama 0.5+ enforces the Pydantic schema as a decoder constraint instead of only being asked for “some JSON”, so local models fail first-attempt validation (and trigger InstructorRetryException) far less often. On an Ollama older than 0.5, which has no JSON-schema enforcement, set LLM_INSTRUCTOR_MODE=json_mode to restore the previous behavior.

Which mode for OpenAI models (e.g. the default openai/gpt-5.6-luna)? Leave LLM_INSTRUCTOR_MODE unset, or set json_schema_mode — Cognee applies json_schema_mode to any model whose id contains gpt-5, so the whole gpt-5.x family qualifies, and it is the recommended mode for OpenAI models that support native JSON-schema responses. Only override it when you point Cognee at a custom or local OpenAI-compatible endpoint that rejects JSON-schema responses; in that case try json_mode first, then markdown_json_mode or tool_call.

Setting Structured Output in a Script

You don’t have to use .env — the same settings can be configured directly in Python. Both the framework and the instructor mode are attributes on the internal LLMConfig.

set_llm_config
os.environ

Pass the exact attribute names (structured_output_framework, llm_instructor_mode) to cognee.config.set_llm_config():

import cognee

cognee.config.set_llm_config({
    "structured_output_framework": "instructor",  # or "baml", "litellm_native"
    "llm_instructor_mode": "json_mode",           # any instructor Mode string
})

Set the environment variables before importing cognee:

import os

os.environ["STRUCTURED_OUTPUT_FRAMEWORK"] = "instructor"
os.environ["LLM_INSTRUCTOR_MODE"] = "json_mode"

import cognee  # reads the variables on first config access

Switching to BAML at runtime via set_llm_config() does not initialize BAML’s client registry, which is built when the config is first constructed. To use BAML, set STRUCTURED_OUTPUT_FRAMEWORK=baml (and the BAML_LLM_* variables) in .env or via os.environ before importing cognee.

Important Notes

Unified Interface: Your application code uses the same acreate_structured_output() call regardless of framework
Provider Flexibility: LiteLLM + Instructor and LiteLLM Native reuse the standard LLM_* provider settings; BAML uses its own BAML_LLM_* block
Output Consistency: All three produce Pydantic-validated results
Performance: Framework choice doesn’t significantly impact performance

Troubleshooting

1 validation error for Response: content Input should be a valid string ... input_type=dict

This error appears during recall() / search() with completion search types such as GRAPH_COMPLETION, GRAPH_SUMMARY_COMPLETION, GRAPH_COMPLETION_COT, and RAG_COMPLETION.Cause. These search types ask the LLM for a plain-text answer (the retriever uses response_model=str). When the configured instructor mode doesn’t match what your model/provider actually supports, the model wraps its answer in a JSON object instead of returning plain text. The instructor backend then can’t coerce that dict into the expected string field, so Pydantic raises Input should be a valid string ... input_type=dict. This is common with OpenAI-compatible, custom, and local (Ollama / LM Studio) endpoints.Fixes:

Align the instructor mode with your provider. OpenAI/Azure gpt-4o/gpt-5 models work with the default json_schema_mode, and so does Ollama 0.5+, which now defaults to it as well. Endpoints that don’t support JSON-schema responses — including an Ollama older than 0.5 — usually need a different mode:
# Try one of these for OpenAI-compatible / local endpoints LLM_INSTRUCTOR_MODE=json_mode # LLM_INSTRUCTOR_MODE=markdown_json_mode # LLM_INSTRUCTOR_MODE=tool_call

Switch to BAML if a small/local model keeps wrapping answers in JSON. BAML bypasses instructor’s coercion and is more forgiving of loose model output:

STRUCTURED_OUTPUT_FRAMEWORK=baml
BAML_LLM_PROVIDER=openai
BAML_LLM_MODEL=gpt-4o-mini
BAML_LLM_API_KEY=sk-...
# For local/OpenAI-compatible endpoints — BAML takes an OpenAI-compatible
# base URL, so this one keeps the /v1:
# BAML_LLM_ENDPOINT=http://localhost:11434/v1
# BAML_LLM_API_KEY=ollama

Skip the LLM completion step to confirm retrieval works independently of model output formatting. Pass only_context=True to get back the LLM input rather than a generated answer — see Search Basics. If retrieval succeeds with only_context=True, the problem is the structured-output configuration above, not your graph.

1 validation error for SummarizedContent: description Input should be a valid string ... input_type=list

This error appeared during cognify() (chunk summarization) on small local models — Ollama and llama.cpp setups were the usual reporters.Cause. SummarizedContent has two fields: summary, which Cognee actually consumes, and description, which is unused and kept only for backwards compatibility. Both are still part of the JSON Schema handed to the model, so a model is free to fill description — and smaller models routinely answered it with a list of bullets instead of a string. Strict validation then failed the whole structured-output call, the retries exhausted, and an entire cognify() run died on a field nothing reads.Fix. Upgrade Cognee — description now coerces whatever the model returns instead of rejecting it: None becomes "", a list or tuple is joined with newlines, and anything else is stringified. summary, the field that is actually consumed, keeps strict validation, so a model that fails to produce a usable summary still surfaces as an error. This is a model-level change, so it applies to every structured-output backend; no configuration change is required.

AttributeError: type object 'str' has no attribute 'model_json_schema' (Gemini)

This error is raised from GeminiAdapter.acreate_structured_output during recall() / search() with completion search types (GRAPH_COMPLETION, RAG_COMPLETION, etc.).Cause. These search types request a plain-text answer with response_model=str. Gemini’s default instructor mode is json_mode, which handles str correctly. If you override LLM_INSTRUCTOR_MODE with a schema- or tool-based mode (json_schema_mode, tool_call, mistral_tools, …), Instructor tries to call str.model_json_schema() — a method that only exists on Pydantic models — and crashes.Fix — force json_mode for Gemini. Either leave LLM_INSTRUCTOR_MODE unset so Gemini falls back to its json_mode default, or set it explicitly:

LLM_PROVIDER="gemini"
LLM_MODEL="gemini/gemini-2.0-flash"
LLM_API_KEY="AIza..."
LLM_INSTRUCTOR_MODE="json_mode"

To set it in-script, pass it through set_llm_config — llm_instructor_mode is a valid key on the LLM config:

import cognee

cognee.config.set_llm_config({"llm_instructor_mode": "json_mode"})

cognee.config.set("llm_instructor_mode", "json_mode") raises InvalidConfigAttributeError: 'llm_instructor_mode' is not a valid attribute of the configuration — the generic config.set() only accepts a fixed set of keys. Use cognee.config.set_llm_config({...}) (or the LLM_INSTRUCTOR_MODE env var) instead.

See LLM Instructor Modes for the full list of modes and per-provider defaults.

ValueError: Unsupported type for BAML mapping: str | None

This error is raised with STRUCTURED_OUTPUT_FRAMEWORK=baml while BAML builds a dynamic type for a response model that has a PEP 604 optional field (X | None). In practice it surfaces during recall() / search() with completion search types such as GRAPH_COMPLETION, because the completion response model contains str | None fields — so recall fails on local/Ollama + BAML setups.Cause. BAML’s dynamic type builder previously recognized only typing.Union / typing.Optional. PEP 604 unions written as X | None have a different origin (types.UnionType), so they missed the Optional/Union branch and fell through to the unsupported-type error. typing.Optional[str] worked; the equivalent str | None did not.Fix. Upgrade Cognee — BAML now maps PEP 604 X | None unions the same way it maps typing.Optional, so optional fields work with either syntax. No configuration change is required.

Invalid JSON: expected value at line 1 column 1 (litellm_native)

This error is raised with STRUCTURED_OUTPUT_FRAMEWORK=litellm_native on the JSON-object fallback path — the path every provider without native schema support lands on (Ollama, llama.cpp, custom OpenAI-compatible endpoints, and any model LiteLLM does not report as schema-capable). It shows up during cognify() for response models such as SummarizedContent or KnowledgeGraph:

ValidationError: 1 validation error for SummarizedContent
  Invalid JSON: expected value at line 1 column 1
  [input_value='```json\n{\n  "summary":...\n}\n```', input_type=str]

Cause. The model wrapped otherwise-valid JSON in a markdown code fence. Pydantic’s model_validate_json() sees a backtick at column 1 and rejects the whole reply. The self-correction retry could not rescue it either — a model that fences once fences again — so all three attempts were spent and the run failed.Fix. Upgrade Cognee — the fallback path now strips a code fence that wraps the entire reply before validating, and leaves backticks inside JSON values alone. No configuration change is required.

LLM Providers

Configure LLM providers for text generation

Overview

Return to setup configuration overview

Custom Prompts

Learn about custom prompt configuration

Getting Started

Core Concepts

Setup Configuration

Guides

Cookbooks

Examples

CLI

OSS

Supported Frameworks

How It Works

Configuration

Instructor Modes

Setting Structured Output in a Script

Important Notes

Troubleshooting

LLM Providers

Overview

Custom Prompts

​Supported Frameworks

​How It Works

​Configuration

​Instructor Modes

​Setting Structured Output in a Script

​Important Notes

​Troubleshooting

LLM Providers

Overview

Custom Prompts

Supported Frameworks

How It Works

Configuration

Instructor Modes

Setting Structured Output in a Script

Important Notes

Troubleshooting