Skip to Content
How-to GuidesCognee SDKS3 Storage

Using Amazon S3 as File Storage

Cognee can seamlessly read from and write to an Amazon S3 bucket instead of the local file-system. This is useful when you run Cognee in distributed environments (e.g. Kubernetes, ECS, EC2) where nodes do not share a common disk.

This guide walks you through the minimal steps to get up and running.

1. Prerequisites

  1. AWS Account – Create one if you don’t already have it.

If you are not the AWS administrator, ask your admin to create an access key for you or a role cognee can assume.

2. Create (or choose) an S3 bucket

Here is the official docs from AWS: Read 

For this guideline, we assume you have your cognee bucket:

s3://my-cognee-bucket

3. Collect your AWS credentials

If you are an admin you can generate a new Access key and Secret key for your IAM user in the AWS Console ( IAM ➜ Users ➜ Security credentials ). Otherwise, request them from your administrator.

4. Configure Cognee via environment variables

Create or update your .env (or export environment variables in your deployment manifests):

STORAGE_BACKEND=s3
STORAGE_BUCKET_NAME=my-cognee-bucket
AWS_ACCESS_KEY_ID=<your-access-key>
AWS_SECRET_ACCESS_KEY=<your-secret-key>
  • STORAGE_BACKEND tells Cognee to use S3 instead of the local disk.
  • STORAGE_BUCKET_NAME is the bucket you created/selected.
  • The AWS keys authenticate Cognee’s requests to S3.

5. Point Cognee’s directories to S3 paths

When you work programmatically you can pin the data and system directories to specific prefixes in your bucket:

import os
import cognee
 
bucket_name = os.getenv("STORAGE_BUCKET_NAME", "my-cognee-bucket")
 
# s3://my-cognee-bucket/data
data_directory_path = os.path.join("s3://", bucket_name, "data")
# s3://my-cognee-bucket/system
cognee_directory_path = os.path.join("s3://", bucket_name, "system")
 
cognee.config.data_root_directory(data_directory_path)
cognee.config.system_root_directory(cognee_directory_path)

That’s it — every file operation (read, write, and delete) now goes to your S3 bucket.

Join the Conversation!

Have questions? Join our community now to connect with professionals, share insights, and get your questions answered!


Alembic MigrationsDistributed Processing