---
name: web-scraper
description: Scrape web pages on the server side. Supports static HTTP scraping (curl) and dynamic JavaScript rendering (Chrome print-to-pdf + PyMuPDF). Zero extra pip dependencies. Use when web_fetch returns empty or minimal content.
metadata: { "openclaw": { "emoji": "🕸️", "requires": { "bins": ["python3", "curl"] } } }
---

# Web Scraper

Server-side web page scraping with two modes. **No extra pip packages needed** -- uses only curl, Chrome, and PyMuPDF (already in the Docker image).

## When to Use

- `web_fetch` returned empty or very short content (common with SPA/JS-heavy sites)
- The user explicitly asks to scrape or crawl a URL

## Quick Start

```bash
# Static scrape (fast, curl + text extraction, works for most pages)
python3 {baseDir}/scripts/scrape.py "https://example.com"

# Dynamic scrape (Chrome renders JS, saves PDF, PyMuPDF extracts text)
python3 {baseDir}/scripts/scrape.py "https://www.bitmain.com/" --mode dynamic
```

## Options

| Option                   | Default  | Description                                                                        |
| ------------------------ | -------- | ---------------------------------------------------------------------------------- |
| `--mode static\|dynamic` | `static` | `static`: curl fetch + regex text extract. `dynamic`: Chrome PDF render + PyMuPDF. |
| `--timeout <seconds>`    | `15`     | Request/render timeout                                                             |
| `--max-chars <n>`        | `50000`  | Truncate output beyond this length                                                 |

## Decision Logic

1. **Try static mode first** (fast, < 2 seconds).
2. If the result `text` is very short (< 200 characters) or empty, the page likely needs JavaScript.
3. **Retry with `--mode dynamic`** to render JavaScript via headless Chrome.
4. If dynamic mode fails (no Chrome), report the error to the user.

## Output Format

JSON to stdout:

```json
{
  "status": "ok",
  "mode": "dynamic",
  "url": "https://www.bitmain.com/",
  "title": "BITMAIN",
  "text": "BITMAIN\nProducts\nBitcoin Miner S23 Hyd...",
  "length": 1779
}
```

## Modes Explained

### Static Mode (`--mode static`)

- Uses `curl` to fetch raw HTML
- Regex-based tag stripping for text extraction
- Fast (< 2 seconds), no Python dependencies needed
- Does NOT execute JavaScript
- Good for: documentation, blogs, news, static content

### Dynamic Mode (`--mode dynamic`)

- Chrome `--headless=new --print-to-pdf` renders the page including all JavaScript
- PyMuPDF (`fitz`) extracts text from the resulting PDF
- Slower (15-40 seconds) but captures dynamically generated content
- Requires `google-chrome` or `chromium` on PATH
- Good for: SPA (React/Vue/Angular), JS-rendered dashboards

## Examples

```bash
# Basic static scrape
python3 {baseDir}/scripts/scrape.py "https://docs.python.org/3/tutorial/index.html"

# SPA page (needs JavaScript rendering)
python3 {baseDir}/scripts/scrape.py "https://www.bitmain.com/" --mode dynamic

# Longer timeout for slow pages
python3 {baseDir}/scripts/scrape.py "https://slow-site.example.com" --mode dynamic --timeout 30
```

## Dependencies

| Tool                         | Required For             | Already Installed      |
| ---------------------------- | ------------------------ | ---------------------- |
| `python3`                    | Script runtime           | Yes (Docker image)     |
| `curl`                       | Static mode              | Yes (Docker image)     |
| `google-chrome` / `chromium` | Dynamic mode only        | Depends on image       |
| `PyMuPDF` (fitz)             | Dynamic mode PDF reading | Yes (requirements.txt) |
| `pdfminer`                   | Fallback PDF reading     | Yes (requirements.txt) |

## Security

- Only `http://` and `https://` URLs are allowed
- Requests to `localhost`, private IPs, and reserved addresses are blocked (SSRF protection)
- Output is truncated at `--max-chars` to prevent excessive memory usage
