Multimedia Processing

A minimal guide to ingesting non-text files — an audio recording and an image — into cognee. Use it when the knowledge you need lives in recordings, screenshots, charts, or scans rather than in documents.

Before You Start

Complete Quickstart to understand basic operations
Ensure you have LLM Providers configured — the audio file needs a provider with a transcription endpoint, and the image needs a vision model
Read Loaders for how cognee turns audio and images into the text it then processes
Have text_to_speech.mp3 and example.png in a multimedia_audio_image_processing_example_data/ directory next to the script — you can download the example media files from the cognee GitHub repository

Code in Action

import asyncio
import os
import pathlib

import cognee
from cognee import SearchType
from cognee.shared.logging_utils import ERROR, setup_logging


async def main():
    # Create a clean slate for cognee -- reset data and system state
    await cognee.forget(everything=True)

    # cognee knowledge graph will be created based on the text
    # and description of these files
    mp3_file_path = os.path.join(
        pathlib.Path(__file__).parent,
        "multimedia_audio_image_processing_example_data/text_to_speech.mp3",
    )
    png_file_path = os.path.join(
        pathlib.Path(__file__).parent,
        "multimedia_audio_image_processing_example_data/example.png",
    )

    # Remember the files and create knowledge graph memory
    await cognee.remember([mp3_file_path, png_file_path], self_improvement=False)

    # Query cognee for summaries of the data in the multimedia files
    search_results = await cognee.recall(
        query_type=SearchType.SUMMARIES,
        query_text="What is in the multimedia files?",
    )

    # Display search results
    for result_text in search_results:
        print(result_text)


if __name__ == "__main__":
    logger = setup_logging(log_level=ERROR)
    asyncio.run(main())

What Just Happened

Step 1: Locate the Media Files

# Create a clean slate for cognee -- reset data and system state
await cognee.forget(everything=True)

# cognee knowledge graph will be created based on the text
# and description of these files
mp3_file_path = os.path.join(
    pathlib.Path(__file__).parent,
    "multimedia_audio_image_processing_example_data/text_to_speech.mp3",
)
png_file_path = os.path.join(
    pathlib.Path(__file__).parent,
    "multimedia_audio_image_processing_example_data/example.png",
)

Forgetting everything first means the summaries you read at the end come only from this run. The two files are resolved relative to the script itself, inside its multimedia_audio_image_processing_example_data/ directory — swap in your own paths to ingest different media.

Step 2: Remember the Audio and the Image

# Remember the files and create knowledge graph memory
await cognee.remember([mp3_file_path, png_file_path], self_improvement=False)

remember() picks a loader per file from its type: the .mp3 goes to the audio loader for transcription, the .png to the image loader for vision transcription. From there both are ordinary text, so the same chunking, extraction, and summarization steps build one graph across the two files.

Step 3: Recall the Summaries

# Query cognee for summaries of the data in the multimedia files
search_results = await cognee.recall(
    query_type=SearchType.SUMMARIES,
    query_text="What is in the multimedia files?",
)

# Display search results
for result_text in search_results:
    print(result_text)

SearchType.SUMMARIES returns the summaries generated during ingestion rather than asking an LLM to answer the question, which makes it a direct way to see what cognee understood from each file. Everything printed here comes from the transcriptions, so it is also the quickest check that the audio and image were read correctly.

Advanced Usage

Get More Out of Images

Images are transcribed with an extraction-oriented prompt that asks for entities, their attributes, relationships, and any visible text — set IMAGE_EXTRACTION_ENABLED="false" to fall back to a short caption instead. With IMAGE_OCR_ENABLED="true" and pip install cognee[rapidocr], a local OCR pass appends the text it recognizes to the transcription, which helps on dense screenshots and scans. See Loaders for every image setting and its default.

Other Media Formats

The same call handles the rest of the audio extensions (.wav, .flac, .m4a, and more) and the image formats the vision loader claims. Video files take the video loader, which transcribes the audio track with inline [HH:MM:SS] timestamps; .mp4 and .webm work as-is, other containers need ffmpeg on your PATH.

Loaders

How each file type is turned into text before ingestion.

remember()

The full parameter surface of the ingestion call this guide uses.

SearchType

Every search mode, including when to prefer SUMMARIES.

Getting Started

Core Concepts

Setup Configuration

Guides

Examples

CLI

Rust SDK

TypeScript SDK

OSS

Before You Start

Code in Action

What Just Happened

Step 1: Locate the Media Files

Step 2: Remember the Audio and the Image

Step 3: Recall the Summaries

Advanced Usage

Loaders

remember()

SearchType

​Before You Start

​Code in Action

​What Just Happened

​Step 1: Locate the Media Files

​Step 2: Remember the Audio and the Image

​Step 3: Recall the Summaries

​Advanced Usage

Loaders

remember()

SearchType

Before You Start

Code in Action

What Just Happened

Step 1: Locate the Media Files

Step 2: Remember the Audio and the Image

Step 3: Recall the Summaries

Advanced Usage