Web URL Ingestion

A guide to building a knowledge graph straight from a web page: pass an http(s) URL to remember(), control what gets extracted with CSS-selector rules, and visualize the result. Before you start:

Complete Quickstart to understand basic operations
Install the scraping extra: pip install cognee[scraping] (BeautifulSoup, Tavily, Playwright). Keenable needs no extra dependencies.

Code in Action

Step 1: Define Extraction Rules

Extraction rules tell the BeautifulSoup loader which parts of the page to keep. Each rule maps a name to a CSS selector (or XPath):

extraction_rules = {
    "title": {"selector": "title"},
    "headings": {"selector": "h1, h2, h3", "all": True},
    "links": {
        "selector": "a",
        "attr": "href",
        "all": True,
    },
    "paragraphs": {"selector": "p", "all": True},
}

Each rule supports:

Key	Description
`selector`	CSS selector to match elements.
`xpath`	XPath expression, as an alternative to `selector`.
`attr`	HTML attribute to extract (e.g. `href`) instead of the element text.
`all`	`True` extracts every matching element; `False` (default) only the first.
`join_with`	String used to join multiple extracted elements (default `" "`).

Rules are optional — without them the loader applies a comprehensive default set covering common HTML content areas (headings, paragraphs, articles, tables, code blocks, etc.). See Loaders for more preferred_loaders examples.

Step 2: Remember the URL

await cognee.remember(
    "https://en.wikipedia.org/api/rest_v1/page/html/Large_language_model",
    incremental_loading=False,
    preferred_loaders={"beautiful_soup_loader": {"extraction_rules": extraction_rules}},
    self_improvement=False,
)

remember() recognizes the URL, fetches the page, extracts content according to your rules, and builds the knowledge graph in one call.

Step 3: Visualize the Result

await cognee.visualize_graph("./web_url_example.html")

Open the HTML file in a browser to explore what was extracted — see Graph Visualization.

Choosing a crawler

Cognee can fetch pages with one of three backends:

Built-in BeautifulSoup crawler (default) — asynchronous HTTP requests, robots.txt compliance, rate limiting, and Playwright rendering for JavaScript-heavy pages.
Tavily — richer extraction from complex pages. Requires TAVILY_API_KEY.
Keenable — clean markdown extraction via the Keenable API. Requires KEENABLE_API_KEY.

When you ingest a URL through remember() or add(), the backend is picked from the environment in this order:

Tavily, if TAVILY_API_KEY is set
Keenable, if KEENABLE_API_KEY is set
The built-in crawler

If both keys are set, Tavily wins — unset TAVILY_API_KEY to route through Keenable. See Python API: add() for the crawler configuration options (tavily_config, soup_crawler_config).

Full Example

The script is available on our github. The complete flow — forget, remember a URL with extraction rules, and visualize — is in the following example:

Web URL content ingestion

import asyncio
from os import path

import cognee


async def main():
    await cognee.forget(everything=True)
    print("Data forgotten.")

    extraction_rules = {
        "title": {"selector": "title"},
        "headings": {"selector": "h1, h2, h3", "all": True},
        "links": {
            "selector": "a",
            "attr": "href",
            "all": True,
        },
        "paragraphs": {"selector": "p", "all": True},
    }

    await cognee.remember(
        "https://en.wikipedia.org/api/rest_v1/page/html/Large_language_model",
        incremental_loading=False,
        preferred_loaders={"beautiful_soup_loader": {"extraction_rules": extraction_rules}},
        self_improvement=False,
    )

    print("Knowledge graph created.")

    graph_visualization_path = path.join(
        path.dirname(__file__), ".artifacts", "web_url_example.html"
    )
    await cognee.visualize_graph(graph_visualization_path)


if __name__ == "__main__":
    asyncio.run(main())

Loaders

How Cognee turns files and pages into ingestable content

Graph Visualization

Render your knowledge graph to an interactive HTML file

Getting Started

Core Concepts

Setup Configuration

Guides

Examples

CLI

Rust SDK

TypeScript SDK

OSS

Code in Action

Step 1: Define Extraction Rules

Step 2: Remember the URL

Step 3: Visualize the Result

Choosing a crawler

Full Example

Loaders

Graph Visualization

​Code in Action

​Step 1: Define Extraction Rules

​Step 2: Remember the URL

​Step 3: Visualize the Result

​Choosing a crawler

​Full Example

Loaders

Graph Visualization

Code in Action

Step 1: Define Extraction Rules

Step 2: Remember the URL

Step 3: Visualize the Result

Choosing a crawler

Full Example