![\"WebPeel](\".github/banner.svg\")
\n \n\n \n \n \n
\n ![\"npm](\"https://img.shields.io/npm/v/webpeel.svg?style=flat-square\")
\n ![\"npm](\"https://img.shields.io/npm/dm/webpeel.svg?style=flat-square\")
\n ![\"GitHub](\"https://img.shields.io/github/stars/webpeel/webpeel?style=flat-square\")
![\"License\"](\"https://img.shields.io/badge/license-WebPeel%20SDK-blue.svg?style=flat-square\"){kind=icon}
\n ![\"CI\"](\"https://github.com/webpeel/webpeel/actions/workflows/ci.yml/badge.svg\"){kind=icon}
\n
\n The web data platform for AI agents — fetch, search, crawl, extract, monitor, screenshot, and research any URL.\n
\n\n---\n\n## Quick Start\n\n```bash\nnpx webpeel \"https://example.com\" # Clean markdown\nnpx webpeel search \"AI trends 2025\" # Web search\nnpx webpeel crawl docs.example.com # Crawl entire site\n```\n\n[Get your free API key →](https://app.webpeel.dev/signup) · No credit card required · 500 requests/week free\n\n---\n\n## Why WebPeel\n\n- **65–98% token savings** — domain-specific extractors strip boilerplate, ads, and nav before content reaches your agent\n- **29 domain extractors** — purpose-built parsers for Reddit, Wikipedia, GitHub, Hacker News, YouTube, ArXiv, Amazon, and 22 more\n- **Zero-config Cloudflare bypass** — 4-layer escalation stack handles TLS fingerprinting, edge proxying, and cache fallback automatically\n\n---\n\n## Features\n\n| Feature | Command / API |\n|---------|---------------|\n| Fetch any URL | `webpeel \"url\"` |\n| Web search | `webpeel search \"query\"` |\n| Crawl sites | `webpeel crawl \"url\" --max-pages 50` |\n| Screenshots | `webpeel screenshot \"url\"` |\n| Monitor changes | `webpeel monitor \"url\" --interval 300` |\n| Browser actions | `--action 'click:.btn,wait:2000'` |\n| YouTube transcripts | auto-detected |\n| PDF extraction | auto-detected |\n| MCP server | `webpeel mcp` |\n| Schema extraction | `POST /v1/fetch` with `extract.schema` |\n| Research agent | `POST /v1/agent` |\n| Smart search | `POST /v1/search/smart` |\n\n---\n\n## MCP Integration\n\nGive Claude, Cursor, or any MCP-compatible agent the ability to browse the web in one config change.\n\n**Claude Desktop** (`~/.claude/claude_desktop_config.json`):\n```json\n{\n \"mcpServers\": {\n \"webpeel\": {\n \"command\": \"npx\",\n \"args\": [\"-y\", \"webpeel\", \"mcp\"],\n \"env\": {\n \"WEBPEEL_API_KEY\": \"wp_your_key_here\"\n }\n }\n }\n}\n```\n\n**Cursor / VS Code** (`.cursor/mcp.json` or `.vscode/mcp.json`):\n```json\n{\n \"mcpServers\": {\n \"webpeel\": {\n \"command\": \"npx\",\n \"args\": [\"-y\", \"webpeel\", \"mcp\"],\n \"env\": {\n \"WEBPEEL_API_KEY\": \"wp_your_key_here\"\n }\n }\n }\n}\n```\n\nAvailable MCP tools: `webpeel_read`, `webpeel_find`, `webpeel_see`, `webpeel_extract`, `webpeel_monitor`, `webpeel_act`, `webpeel_crawl`\n\n[Full MCP setup guide →](https://webpeel.dev/docs/mcp)\n\n---\n\n## API Example\n\n```bash\n# Fetch any page — returns clean markdown + metadata\ncurl \"https://api.webpeel.dev/v1/fetch?url=https://stripe.com/pricing\" \\\n -H \"Authorization: Bearer $WEBPEEL_API_KEY\"\n```\n\n```json\n{\n \"url\": \"https://stripe.com/pricing\",\n \"markdown\": \"# Stripe Pricing\\n\\n**Integrated per-transaction fees**...\",\n \"metadata\": {\n \"title\": \"Pricing & Fees | Stripe\",\n \"tokens\": 420,\n \"tokensOriginal\": 8200,\n \"savingsPct\": 94.9\n }\n}\n```\n\n[Full API reference →](https://webpeel.dev/docs/api)\n\n---\n\n## Token Efficiency\n\nWebPeel's 29 domain-specific extractors strip navigation, ads, sidebars, and boilerplate before sending content to your agent.\n\n| Site type | Raw HTML tokens | WebPeel tokens | Savings |\n|-----------|:--------------:|:--------------:|:-------:|\n| News article | 18,000 | 640 | **96%** |\n| Reddit thread | 24,000 | 890 | **96%** |\n| Wikipedia page | 31,000 | 2,100 | **93%** |\n| GitHub README | 5,200 | 1,800 | **65%** |\n| E-commerce product | 14,000 | 310 | **98%** |\n\nLess context used = lower costs + faster inference + longer agent chains.\n\n---\n\n## Security\n\nWebPeel is built with security-first principles:\n\n- **Helmet.js headers** — HSTS, X-Frame-Options, nosniff, XSS protection on all responses\n- **Webhook signing** — HMAC-SHA256 signatures on all outbound webhooks\n- **Audit logging** — every API call logged with IP, key, and action\n- **GDPR compliant** — `DELETE /v1/account` for full data erasure\n- **SSH hardened** — Fail2Ban, MaxAuthTries, key-only auth on all infrastructure\n\n[Security policy →](https://webpeel.dev/security)\n\n---\n\n## Links\n\n- 📖 [Documentation](https://webpeel.dev/docs) — Guides, references, and examples\n- 💰 [Pricing](https://webpeel.dev/pricing) — Plans and limits\n- 📝 [Blog](https://webpeel.dev/blog) — Tutorials, comparisons, and use cases\n- 📊 [Status](https://webpeel.dev/status) — Uptime and incidents\n- 🔒 [Security](https://webpeel.dev/security) — Security policy and disclosure\n- 📋 [SLA](https://webpeel.dev/sla) — Uptime commitments\n\n---\n\n## Contributing\n\nPull requests welcome! Please open an issue first to discuss major changes.\n\n1. Fork the repo\n2. Create your feature branch (`git checkout -b feat/my-feature`)\n3. Commit your changes (`git commit -m 'feat: add my feature'`)\n4. Push to the branch (`git push origin feat/my-feature`)\n5. Open a Pull Request\n\n---\n\n## License\n\n[WebPeel SDK License](LICENSE) — free for personal and commercial use with attribution. See LICENSE for full terms.\n\n\n Get started free →\n
\n","readmeFilename":"README.md"}