Web URL Ingestion

A guide to building a knowledge graph straight from a web page: pass an http(s) URL to remember(), control what gets extracted with CSS-selector rules, and visualize the result. Before you start:

Complete Quickstart to understand basic operations
Install the scraping extra: pip install cognee[scraping] (BeautifulSoup, Tavily, Playwright)

Code in Action

Step 1: Define Extraction Rules

Extraction rules tell the BeautifulSoup loader which parts of the page to keep. Each rule maps a name to a CSS selector (or XPath):

extraction_rules = {
    "title": {"selector": "title"},
    "headings": {"selector": "h1, h2, h3", "all": True},
    "links": {
        "selector": "a",
        "attr": "href",
        "all": True,
    },
    "paragraphs": {"selector": "p", "all": True},
}

Each rule supports:

Key	Description
`selector`	CSS selector to match elements.
`xpath`	XPath expression, as an alternative to `selector`.
`attr`	HTML attribute to extract (e.g. `href`) instead of the element text.
`all`	`True` extracts every matching element; `False` (default) only the first.
`join_with`	String used to join multiple extracted elements (default `" "`).

Rules are optional — without them the loader applies a comprehensive default set covering common HTML content areas (headings, paragraphs, articles, tables, code blocks, etc.). See Loaders for more preferred_loaders examples.

Step 2: Remember the URL

await cognee.remember(
    "https://en.wikipedia.org/api/rest_v1/page/html/Large_language_model",
    incremental_loading=False,
    preferred_loaders={"beautiful_soup_loader": {"extraction_rules": extraction_rules}},
    self_improvement=False,
)

remember() recognizes the URL, fetches the page, extracts content according to your rules, and builds the knowledge graph in one call.

Step 3: Visualize the Result

await cognee.visualize_graph("./web_url_example.html")

Open the HTML file in a browser to explore what was extracted — see Graph Visualization.

Choosing a crawler

By default Cognee uses its built-in BeautifulSoup-based crawler, which supports asynchronous HTTP requests, robots.txt compliance, rate limiting, and Playwright rendering for JavaScript-heavy pages. Set TAVILY_API_KEY in your environment to use the Tavily API instead for richer extraction from complex pages. See Python API: add() for the crawler configuration options (tavily_config, soup_crawler_config).

Full Example

The script is available on our github. The complete flow — forget, remember a URL with extraction rules, and visualize — is in the following example:

Web URL content ingestion

import asyncio
from os import path

import cognee


async def main():
    await cognee.forget(everything=True)
    print("Data forgotten.")

    extraction_rules = {
        "title": {"selector": "title"},
        "headings": {"selector": "h1, h2, h3", "all": True},
        "links": {
            "selector": "a",
            "attr": "href",
            "all": True,
        },
        "paragraphs": {"selector": "p", "all": True},
    }

    await cognee.remember(
        "https://en.wikipedia.org/api/rest_v1/page/html/Large_language_model",
        incremental_loading=False,
        preferred_loaders={"beautiful_soup_loader": {"extraction_rules": extraction_rules}},
        self_improvement=False,
    )

    print("Knowledge graph created.")

    graph_visualization_path = path.join(
        path.dirname(__file__), ".artifacts", "web_url_example.html"
    )
    await cognee.visualize_graph(graph_visualization_path)


if __name__ == "__main__":
    asyncio.run(main())

Loaders

How Cognee turns files and pages into ingestable content

Graph Visualization

Render your knowledge graph to an interactive HTML file

Getting Started

Core Concepts

Setup Configuration

Guides

Examples

CLI

Rust SDK

TypeScript SDK

OSS

Code in Action

Step 1: Define Extraction Rules

Step 2: Remember the URL

Step 3: Visualize the Result

Choosing a crawler

Full Example

Loaders

Graph Visualization

​Code in Action

​Step 1: Define Extraction Rules

​Step 2: Remember the URL

​Step 3: Visualize the Result

​Choosing a crawler

​Full Example

Loaders

Graph Visualization

Code in Action

Step 1: Define Extraction Rules

Step 2: Remember the URL

Step 3: Visualize the Result

Choosing a crawler

Full Example