Setup Local Models with Ollama

Difficulty: Easy

Overview

In this tutorial, you’ll learn how to run local models with Cognee using Ollama . Running models locally allows for private, offline reasoning, and removes the need for cloud-based APIs.

By the end, you will have:

Installed Ollama and downloaded a supported local model.
Configured Cognee to use your local model via the Ollama API.
Created a basic script to verify everything is working.
Queried the local model through a simple Cognee pipeline.

What You’ll Learn

Ollama Setup: Install and run the Ollama CLI.
Model Download: Pull completion and embeddings models (like phi4 or sfr-embedding-mistral:latest) to run locally.
Cognee Config: Setup Cognee to use your local Ollama endpoint.

Here is a quick-start guide:

Step 1: Got to Ollama.com

Go to Ollama’s official website: 👉 https://ollama.com

Click on the Download button, choose your operation system and download the ollama installer

Step 2: Install Ollama

Install Ollama on your local system using the downloaded installer, then verify the installation by running


ollama --version

in your terminal.

Step 3: Choosing and downloading local models

Ollama supports many models. You need a large language model to serve completions and an embedding model to create embeddings from your textual data.

Pull the LLM model of your choice:


ollama pull YOUR_MODEL

Pull the embedding model of your choice:


ollama pull YOUR_EMBEDDING_MODEL

To verify the existence of your local models use:


ollama list

DISCLAIMER: Models below 32b parameters are unable to create the proper graph structure sometimes so we suggest to use models like ‘deepseek-r1:32b’ or ‘llama3.3’.

Step 4: Configure cognee to use local models

Setting up cognee environmental variables in your .env file in your project.


LLM_API_KEY="ollama"
LLM_MODEL = "YOUR_MODEL"
LLM_PROVIDER = "ollama"
LLM_ENDPOINT = "http://localhost:11434/v1"
EMBEDDING_PROVIDER="ollama"
EMBEDDING_MODEL="YOUR_EMBEDDING_MODEL"
EMBEDDING_ENDPOINT="http://localhost:11434/api/embeddings"
EMBEDDING_DIMENSIONS="DIMENSIONS_OF_YOUR_EMBEDDING_MODEL"
HUGGINGFACE_TOKENIZER="TOKENIZER_TO_YOUR_EMBEDDING_MODEL"

This tells Cognee to route requests to your locally running Ollama endpoints.

Step 5: Test cognee with a simple script

Here’s an example to test cognee with your local models. You can find the full example script here


import asyncio
import cognee
from cognee.shared.logging_utils import get_logger, ERROR
from cognee.api.v1.search import SearchType
 
# Prerequisites:
# 1. Copy `.env.template` and rename it to `.env`.
# 2. Add your OpenAI API key to the `.env` file in the `LLM_API_KEY` field:
#    LLM_API_KEY = "your_key_here"
 
 
async def main():
    # Create a clean slate for cognee -- reset data and system state
    await cognee.prune.prune_data()
    await cognee.prune.prune_system(metadata=True)
 
    # cognee knowledge graph will be created based on this text
    text = """
    Natural language processing (NLP) is an interdisciplinary
    subfield of computer science and information retrieval.
    """
 
    # Add the text, and make it available for cognify
    await cognee.add(text)
 
    # Run cognify and build the knowledge graph using the added text
    await cognee.cognify()
 
    # Query cognee for insights on the added text
    query_text = "Tell me about NLP"
    search_results = await cognee.search(query_type=SearchType.INSIGHTS, query_text=query_text)
    for result_text in search_results:
        print(result_text)
 
 
if __name__ == "__main__":
    logger = get_logger(level=ERROR)
    loop = asyncio.new_event_loop()
    asyncio.set_event_loop(loop)
    try:
        loop.run_until_complete(main())
    finally:
        loop.run_until_complete(loop.shutdown_asyncgens())

Summary

In this tutorial, you learned how to:

Install Ollama and run it as a local LLM server.
Pull and run your LLM and Embedding models.
Configure Cognee to use Ollama instead of cloud APIs.

Running models locally gives you full control over privacy, cost, and speed. Whether you’re prototyping or scaling production use-cases.

Join the Conversation!

Have questions or need more help? Join our community to connect with professionals, share insights, and get your questions answered!