Structured Output Backends - Cognee Documentation

Structured output backends ensure reliable data extraction from LLM responses. Cognee supports two frameworks that convert LLM text into structured Pydantic models for knowledge graph extraction and other tasks.

New to configuration?See the Setup Configuration Overview for the complete workflow:install extras → create .env → choose providers → handle pruning.

Supported Frameworks

Cognee supports two structured output approaches:

LiteLLM + Instructor — Provider-agnostic client with Pydantic coercion (default)
BAML — DSL-based framework with type registry and guardrails

Both frameworks produce the same Pydantic-validated outputs, so your application code remains unchanged regardless of which backend you choose.

How It Works

Cognee uses a unified interface that abstracts the underlying framework:

from cognee.infrastructure.llm.LLMGateway import LLMGateway
await LLMGateway.acreate_structured_output(text, system_prompt, response_model)

The STRUCTURED_OUTPUT_FRAMEWORK environment variable determines which backend processes your requests, but the API remains identical.

Configuration

LiteLLM + Instructor (Default)
BAML

The default framework — no extra install needed. Uses LiteLLM and the instructor library to coerce LLM responses into Pydantic models.

STRUCTURED_OUTPUT_FRAMEWORK=instructor

Optionally, control how the model is prompted for structured output:

# Override instructor mode (e.g. json_mode, tool_call, markdown_json_mode)
# Leave unset to use the provider's default — see "Instructor Modes" below.
LLM_INSTRUCTOR_MODE=json_schema_mode

BAML is an alternative structured output framework that uses a DSL-based type registry to extract data. It is particularly useful when small local models (such as Ollama models like llama3.1:8b or qwen3.5:0.8b) struggle to produce valid structured output with instructor, causing repeated InstructorRetryException errors.Installation: BAML requires a separate install:

pip install "cognee[baml]"

Configuration: BAML uses its own LLM settings, independent of the main LLM_* variables:

STRUCTURED_OUTPUT_FRAMEWORK=baml

# BAML-specific LLM settings (required)
BAML_LLM_PROVIDER=openai
BAML_LLM_MODEL=gpt-4o-mini
BAML_LLM_API_KEY=sk-...

# Optional BAML overrides
# BAML_LLM_ENDPOINT=https://api.openai.com/v1
# BAML_LLM_API_VERSION=
# BAML_LLM_TEMPERATURE=0.0

BAML_LLM_PROVIDER and BAML_LLM_MODEL accept the same provider names and model identifiers as the main LLM configuration. You can point BAML at a different model than your main LLM — for example, use a small Ollama model for general text generation while routing structured extraction through a cloud model.

If STRUCTURED_OUTPUT_FRAMEWORK=baml is set but the cognee[baml] extra is not installed, Cognee will raise an ImportError on startup. Run pip install "cognee[baml]" to resolve it.

Using BAML for Small Local Models

Small Ollama models (e.g. llama3.1:8b, qwen3.5:0.8b) often fail to produce valid JSON-structured output when using the default instructor backend, resulting in repeated InstructorRetryException errors during cognify for types like KnowledgeGraph or SummarizedContent.Switching to BAML bypasses instructor entirely and uses BAML’s own extraction pipeline, which is more forgiving with smaller models:

# Main LLM — small local model via Ollama
LLM_PROVIDER=ollama
LLM_MODEL=llama3.1:8b
LLM_ENDPOINT=http://localhost:11434/v1
LLM_API_KEY=ollama

# Use BAML for structured extraction (can point to a different, more capable model)
STRUCTURED_OUTPUT_FRAMEWORK=baml
BAML_LLM_PROVIDER=openai
BAML_LLM_MODEL=gpt-4o-mini
BAML_LLM_API_KEY=sk-...

See the Local Setup guide for a complete Ollama configuration including embeddings.

Instructor Modes

When STRUCTURED_OUTPUT_FRAMEWORK=instructor, the instructor mode controls how Cognee asks the model for structured output — for example via the model’s native JSON-schema response, a plain JSON object, or a tool/function call. The value of LLM_INSTRUCTOR_MODE is passed directly to the instructor library’s Mode, so it must be one of instructor’s supported mode strings. LLM_INSTRUCTOR_MODE is empty by default. When it is unset, Cognee either applies a provider-specific mode or defers to the underlying Instructor/LiteLLM default, so in most cases you don’t need to set it at all:

`LLM_PROVIDER`	Behavior when `LLM_INSTRUCTOR_MODE` is unset
`openai`, `azure` with `gpt-5` models	`json_schema_mode`
`openai`, `azure` with other models	use Instructor/LiteLLM default
AWS Bedrock	`json_schema_mode`
`ollama`, `gemini`, `custom` (OpenAI-compatible), llama.cpp	`json_mode`
`anthropic`	`anthropic_tools`
`mistral`	`mistral_tools`

Common values you can set explicitly include json_schema_mode, json_mode, tool_call, and markdown_json_mode.

Which mode for OpenAI models (e.g. gpt-5-mini)? Leave LLM_INSTRUCTOR_MODE unset, or set json_schema_mode — Cognee applies json_schema_mode to gpt-5 models, and it is the recommended mode for OpenAI models that support native JSON-schema responses. Only override it when you point Cognee at a custom or local OpenAI-compatible endpoint that rejects JSON-schema responses; in that case try json_mode first, then markdown_json_mode or tool_call.

Important Notes

Unified Interface: Your application code uses the same acreate_structured_output() call regardless of framework
Provider Flexibility: Both frameworks support the same LLM providers
Output Consistency: Both produce identical Pydantic-validated results
Performance: Framework choice doesn’t significantly impact performance

Troubleshooting

`1 validation error for Response: content Input should be a valid string ... input_type=dict`

This error appears during recall() / search() with completion search types such as GRAPH_COMPLETION, GRAPH_SUMMARY_COMPLETION, GRAPH_COMPLETION_COT, and RAG_COMPLETION.Cause. These search types ask the LLM for a plain-text answer (the retriever uses response_model=str). When the configured instructor mode doesn’t match what your model/provider actually supports, the model wraps its answer in a JSON object instead of returning plain text. The instructor backend then can’t coerce that dict into the expected string field, so Pydantic raises Input should be a valid string ... input_type=dict. This is common with OpenAI-compatible, custom, and local (Ollama / LM Studio) endpoints.Fixes:

Align the instructor mode with your provider. OpenAI/Azure gpt-4o/gpt-5 models work with the default json_schema_mode. Endpoints that don’t support JSON-schema responses usually need a different mode:
```
# Try one of these for OpenAI-compatible / local endpoints
LLM_INSTRUCTOR_MODE=json_mode
# LLM_INSTRUCTOR_MODE=markdown_json_mode
# LLM_INSTRUCTOR_MODE=tool_call
```

Switch to BAML if a small/local model keeps wrapping answers in JSON. BAML bypasses instructor’s coercion and is more forgiving of loose model output:

STRUCTURED_OUTPUT_FRAMEWORK=baml
BAML_LLM_PROVIDER=openai
BAML_LLM_MODEL=gpt-4o-mini
BAML_LLM_API_KEY=sk-...
# For local/OpenAI-compatible endpoints:
# BAML_LLM_ENDPOINT=http://localhost:11434/v1
# BAML_LLM_API_KEY=ollama

Skip the LLM completion step to confirm retrieval works independently of model output formatting. Pass only_context=True to return the retrieved context directly — see Search Basics. If retrieval succeeds with only_context=True, the problem is the structured-output configuration above, not your graph.

LLM Providers

Configure LLM providers for text generation

Overview

Return to setup configuration overview

Custom Prompts

Learn about custom prompt configuration

​Supported Frameworks

​How It Works

​Configuration

​Instructor Modes

​Important Notes

​Troubleshooting

LLM Providers

Overview

Custom Prompts

Supported Frameworks

How It Works

Configuration

Instructor Modes

Important Notes

Troubleshooting