Cognify

1. Classify documents
   • Turns raw Data rows into Document objects
   • Chooses the right document type (PDF, text, image, audio, etc.)
   • Attaches metadata and optional node sets
2. Check permissions
   • Verifies that the user has write access to the dataset
3. Extract chunks
   • Splits documents into DocumentChunks using a chunker
   • You can customize the chunk size and strategy — see Chunkers for details
   • Updates token counts in the relational DB
4. Extract graph
   • Calls the LLM to extract entities and relationships
   • Deduplicates nodes and edges, commits to the graph DB
5. Summarize text
   • Generates concise summaries per chunk
   • Stores them as TextSummary DataPoints for vector search
6. Add data points
   • Converts summaries and other DataPoints into graph + vector nodes
   • Embeds them in the vector store, persists in the graph DB

• Cognify always runs on a dataset
• You must have write access to the target dataset
• Permissions are enforced at pipeline start
• Each dataset maintains its own cognify status and token counts

• By default, .cognify processes all data in a dataset
• With incremental_loading=True, only new or updated files are processed
• Saves time and compute for large, evolving datasets

• Capture entities, names, nouns, and implied mentions exhaustively
• Form relationships as (start_node, relationship_name, end_node) triplets using explicit and inferred connections
• Avoid duplicates and overly generic terms

​

Python SDK

from cognee.infrastructure.databases.graph import get_graph_engine

graph_engine = await get_graph_engine()

# Returns all nodes and edges
nodes, edges = await graph_engine.get_graph_data()

# Inspect unique node types
node_types = {props.get("type") for _, props in nodes if props.get("type")}
print("Node types:", node_types)

# Inspect unique relationship names
relationship_names = {rel_name for _, _, rel_name, _ in edges}
print("Relationship names:", relationship_names)

• Nodes as (node_id: str, properties: dict)
• Edges as (source_id: str, target_id: str, relationship_name: str, properties: dict)

metrics = await graph_engine.get_graph_metrics()
# Returns: num_nodes, num_edges, mean_degree, edge_density,
#          num_connected_components, sizes_of_connected_components

metrics = await graph_engine.get_graph_metrics(include_optional=True)
print(metrics)

​

HTTP server mode

GET /api/v1/datasets/{dataset_id}/graph

1. Delete the dataset first, then re-add and re-cognify:
   # Clear existing processed data
   await cognee.datasets.empty_dataset(dataset_id=my_dataset.id)
   
   # Re-add source files
   await cognee.add(source_files, dataset_name="my_dataset")
   
   # Re-cognify with the updated schema
   await cognee.cognify()
   
2. Alternatively, use Memify for additive enrichment — it runs extraction and enrichment tasks over the existing graph without re-ingesting data. This is useful when you want to add new derived facts without reprocessing from scratch.

• Vector database contains embeddings for summaries and nodes
• Graph database contains entities and relationships
• Relational database tracks token counts and pipeline run status
• Your dataset is now ready for Search (semantic or graph-based)

1. Graph extraction — identifies entities and relationships from the chunk text
2. Summarization — generates a concise summary of the chunk

chunks          = ceil(document_tokens / chunk_size)
total_llm_calls = chunks × 2

chunk_size = min(embedding_model_max_tokens, llm_max_tokens ÷ 2)

• Increase chunk_size — fewer, larger chunks mean fewer calls:
  await cognee.cognify(chunk_size=4096)
  
• Skip summarization — use a custom pipeline that omits the summarize_text task, reducing calls to 1 per chunk.
• Enable rate limiting — set LLM_RATE_LIMIT_ENABLED=true to avoid bursting your provider quota when processing many chunks in parallel.