> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cognee.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Web URL Ingestion

> Step-by-step guide to ingesting web page content with custom extraction rules

A guide to building a knowledge graph straight from a web page: pass an `http(s)` URL to `remember()`, control what gets extracted with CSS-selector rules, and visualize the result.

**Before you start:**

* Complete [Quickstart](getting-started/quickstart) to understand basic operations
* Install the scraping extra: `pip install cognee[scraping]` (BeautifulSoup, Tavily, Playwright)

## Code in Action

### Step 1: Define Extraction Rules

Extraction rules tell the BeautifulSoup loader which parts of the page to keep. Each rule maps a name to a CSS selector (or XPath):

```python theme={null}
extraction_rules = {
    "title": {"selector": "title"},
    "headings": {"selector": "h1, h2, h3", "all": True},
    "links": {
        "selector": "a",
        "attr": "href",
        "all": True,
    },
    "paragraphs": {"selector": "p", "all": True},
}
```

Each rule supports:

| Key         | Description                                                               |
| ----------- | ------------------------------------------------------------------------- |
| `selector`  | CSS selector to match elements.                                           |
| `xpath`     | XPath expression, as an alternative to `selector`.                        |
| `attr`      | HTML attribute to extract (e.g. `href`) instead of the element text.      |
| `all`       | `True` extracts every matching element; `False` (default) only the first. |
| `join_with` | String used to join multiple extracted elements (default `" "`).          |

Rules are optional — without them the loader applies a comprehensive default set covering common HTML content areas (headings, paragraphs, articles, tables, code blocks, etc.). See [Loaders](/core-concepts/further-concepts/loaders) for more `preferred_loaders` examples.

### Step 2: Remember the URL

```python theme={null}
await cognee.remember(
    "https://en.wikipedia.org/api/rest_v1/page/html/Large_language_model",
    incremental_loading=False,
    preferred_loaders={"beautiful_soup_loader": {"extraction_rules": extraction_rules}},
    self_improvement=False,
)
```

`remember()` recognizes the URL, fetches the page, extracts content according to your rules, and builds the knowledge graph in one call.

### Step 3: Visualize the Result

```python theme={null}
await cognee.visualize_graph("./web_url_example.html")
```

Open the HTML file in a browser to explore what was extracted — see [Graph Visualization](/guides/graph-visualization).

## Choosing a crawler

By default Cognee uses its built-in BeautifulSoup-based crawler, which supports asynchronous HTTP requests, robots.txt compliance, rate limiting, and Playwright rendering for JavaScript-heavy pages. Set `TAVILY_API_KEY` in your environment to use the Tavily API instead for richer extraction from complex pages. See [Python API: add()](/python-api/add) for the crawler configuration options (`tavily_config`, `soup_crawler_config`).

## Full Example

The script is available on our [github](https://github.com/topoteretes/cognee/blob/main/examples/demos/web_url_content_ingestion_example.py).

The complete flow — forget, remember a URL with extraction rules, and visualize — is in the following example:

<Accordion title="Web URL content ingestion">
  ```python theme={null}
  import asyncio
  from os import path

  import cognee


  async def main():
      await cognee.forget(everything=True)
      print("Data forgotten.")

      extraction_rules = {
          "title": {"selector": "title"},
          "headings": {"selector": "h1, h2, h3", "all": True},
          "links": {
              "selector": "a",
              "attr": "href",
              "all": True,
          },
          "paragraphs": {"selector": "p", "all": True},
      }

      await cognee.remember(
          "https://en.wikipedia.org/api/rest_v1/page/html/Large_language_model",
          incremental_loading=False,
          preferred_loaders={"beautiful_soup_loader": {"extraction_rules": extraction_rules}},
          self_improvement=False,
      )

      print("Knowledge graph created.")

      graph_visualization_path = path.join(
          path.dirname(__file__), ".artifacts", "web_url_example.html"
      )
      await cognee.visualize_graph(graph_visualization_path)


  if __name__ == "__main__":
      asyncio.run(main())
  ```
</Accordion>

<Columns cols={2}>
  <Card title="Loaders" icon="download" href="/core-concepts/further-concepts/loaders">
    How Cognee turns files and pages into ingestable content
  </Card>

  <Card title="Graph Visualization" icon="network" href="/guides/graph-visualization">
    Render your knowledge graph to an interactive HTML file
  </Card>
</Columns>