---
name: web-crawling
description: Fetch and clean page content from user-provided URLs with Crawl4AI.
---

# Web Crawling

Use Crawl4AI CLI for page extraction when the task is "fetch this URL and give me clean content".

## When to use this

- The user gives a specific URL and wants the page content.
- The page is a docs site, article, blog post, or product page.
- Raw HTML or `curl` output would be too noisy.
- The user wants markdown that is easier for an LLM to read.

Do not use this for general discovery across the web. Use web search first when the user needs to find URLs.

## Install

Crawl4AI currently needs Python 3.10+.

Recommended install:

```bash
uv tool install --python /opt/homebrew/bin/python3.11 crawl4ai
crawl4ai-setup
crawl4ai-doctor
```

What this does:

- `uv tool install ... crawl4ai` installs the CLI commands
- `crawl4ai-setup` installs the browser/runtime dependencies
- `crawl4ai-doctor` verifies the install with a real crawl

Installed commands:

- `crwl`
- `crawl4ai-setup`
- `crawl4ai-doctor`
- `crawl4ai-download-models`

## Default operation

Prefer `markdown-fit` first. It is the cleanest default for agent workflows.

```bash
crwl crawl <URL> -o markdown-fit
```

This usually strips more navigation and boilerplate than plain markdown.

## Useful modes

### 1. Best default: fit markdown

```bash
crwl crawl <URL> -o markdown-fit
```

Use for:

- docs pages
- articles
- product pages
- anything that should be concise and LLM-friendly

### 2. Full markdown

```bash
crwl crawl <URL> -o markdown
```

Use when:

- `markdown-fit` removes too much
- you want more structure, links, or surrounding context

### 3. JSON

```bash
crwl crawl <URL> -o json
```

Use when downstream code wants structured output instead of markdown.

## Important caveat

Question mode is not self-contained by default:

```bash
crwl crawl <URL> -o markdown-fit -q "What are the installation steps?"
```

This requires a default LLM provider to be configured. Without one, the CLI errors with:

```text
No default LLM provider configured.
Provider format: 'company/model'
```

So:

- use `markdown-fit` by default
- only use `-q/--question` if the machine already has an LLM provider configured

## Practical notes

- `markdown-fit` worked well on docs pages in local testing.
- It stayed fairly clean on simple pages like `example.com`.
- On very noisy pages such as Wikipedia, `markdown-fit` still leaves noticeable boilerplate. If the result is messy, try:
  - a more focused URL
  - plain `markdown`
  - fallback extraction with other tools if needed

## Examples

Fetch a docs page:

```bash
crwl crawl https://docs.crawl4ai.com/core/installation/ -o markdown-fit
```

Fetch a product or article page:

```bash
crwl crawl https://example.com -o markdown-fit
```

Get fuller page output:

```bash
crwl crawl https://docs.crawl4ai.com/core/installation/ -o markdown
```
