{"_id":"@aprimediet/webtools","name":"@aprimediet/webtools","dist-tags":{"latest":"1.0.0"},"versions":{"1.0.0":{"name":"@aprimediet/webtools","version":"1.0.0","type":"module","description":"Self-hosted web fetch / extract / search / crawl tools for the pi coding agent (Playwright + Readability, no MCP).","keywords":["pi-package"],"pi":{"extensions":["./index.ts"]},"dependencies":{"playwright":"^1.48.0","@mozilla/readability":"^0.5.0","jsdom":"^25.0.0","turndown":"^7.2.0","turndown-plugin-gfm":"^1.0.2"},"peerDependencies":{"@earendil-works/pi-coding-agent":"*","@earendil-works/pi-agent-core":"*","@earendil-works/pi-ai":"*","@earendil-works/pi-tui":"*","typebox":"*"},"gitHead":"7e9fbdb25ff45fae6f524a3b3d47540abd82a537","_id":"@aprimediet/webtools@1.0.0","_nodeVersion":"24.18.0","_npmVersion":"11.17.0","dist":{"integrity":"sha512-E7+FpgwUi7Fk4ZJMLKdsHK4END1jpZWAfLAETRx/tqhQ/tXuE5IL1Hr1JVyx3+SqUUzbeF6Mwwq4yZQzJX7JWg==","shasum":"6fc37b1f88635935a2f220ee04aa78501fac76ff","tarball":"https://registry.npmjs.org/@aprimediet/webtools/-/webtools-1.0.0.tgz","fileCount":9,"unpackedSize":40531,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIEI8qSZRjw5tHwB2En8UUFFBgTKg1VAJyoidNRZ1loRyAiEAvCaVuoOKsAGEP6D50LM1j4+bQ1cM20/QYQrxPCpJetE="}]},"_npmUser":{"name":"aditya.prima","email":"aprimediet@gmail.com"},"directories":{},"maintainers":[{"name":"aditya.prima","email":"aprimediet@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/webtools_1.0.0_1782401633335_0.6607330926235546"},"_hasShrinkwrap":false}},"time":{"created":"2026-06-25T15:33:53.191Z","1.0.0":"2026-06-25T15:33:53.493Z","modified":"2026-06-25T15:33:53.701Z"},"maintainers":[{"name":"aditya.prima","email":"aprimediet@gmail.com"}],"description":"Self-hosted web fetch / extract / search / crawl tools for the pi coding agent (Playwright + Readability, no MCP).","keywords":["pi-package"],"readme":"# @aprimediet/webtools\n\nSelf-hosted **web tools** for the [pi coding agent](https://www.npmjs.com/package/@earendil-works/pi-coding-agent): fetch, extract, search, and crawl — using only npm/npx/docker backends. **No MCP, no hosted scraping APIs.** The engine is **Playwright**; extraction is **Mozilla Readability + Turndown**; search drives the browser over a public engine's HTML.\n\n## Tools\n\n| Tool | Params | Returns |\n|---|---|---|\n| `web_fetch` | `url`, `render?`(auto/static/browser), `format?`(markdown/text/html), `timeoutMs?`, `maxBytes?` | page content as markdown/text/html + `{finalUrl,status,title,bytes,renderUsed,truncated}` |\n| `web_extract` | `url`, `html?`, `render?`, `format?`(markdown/text), `includeMetadata?`, `selectors?`(name→CSS) | main article markdown + `{title,byline,siteName,publishedTime,lang,excerpt,fields}` |\n| `web_search` | `query`, `limit?`, `engine?`(duckduckgo/bing/brave) | `[{title,url,snippet}]` scraped from the engine's HTML |\n| `web_crawl` | `startUrl`, `maxPages?`(20, cap 100), `maxDepth?`(2), `sameOrigin?`, `include?`/`exclude?`, `render?`, `format?`, `concurrency?`(3, cap 5) | combined per-page markdown + `{pageCount,pages[]}` (streams progress) |\n\n`render:auto` fetches statically and **escalates to the headless browser** if the page looks like a JS shell. The browser is launched **once** and reused, then closed on session shutdown.\n\n## Setup (self-hosted backends)\n\nThird-party libs (`playwright`, `@mozilla/readability`, `jsdom`, `turndown`, `turndown-plugin-gfm`) install automatically with the package. **Playwright browser binaries do not** — install them once:\n\n```bash\nnpx playwright install --with-deps chromium\n```\n\nOr point at a **Docker / remote browser** (zero local install) and set:\n\n```bash\nexport PLAYWRIGHT_CDP=http://localhost:9222         # Chrome DevTools Protocol endpoint\n# or\nexport PLAYWRIGHT_WS_ENDPOINT=ws://localhost:3000   # a Playwright server / browserless container\n```\n\n## Configuration (env + flags)\n\n| Env | Flag | Meaning |\n|---|---|---|\n| `WEBTOOLS_RENDER` | `--render-default` | default render mode (`auto`) |\n| `WEBTOOLS_SEARCH_ENGINE` | `--search-engine` | default engine (`duckduckgo`) |\n| `WEBTOOLS_USER_AGENT` | `--user-agent` | UA string |\n| `WEBTOOLS_ALLOW_PRIVATE` | `--allow-private` | allow private/loopback hosts (off by default) |\n| `WEBTOOLS_IGNORE_ROBOTS` | — | crawl ignores robots.txt (off by default) |\n| `PLAYWRIGHT_CDP` / `PLAYWRIGHT_WS_ENDPOINT` | — | connect to a remote/Docker browser |\n\n`/webtools` prints the current config and chosen browser backend.\n\n## Safety\n\n- **SSRF guard:** `web_fetch`/`web_extract`/`web_crawl` resolve the host and **block loopback/private/link-local ranges** (`127/8`, `10/8`, `172.16/12`, `192.168/16`, `169.254/16`, `::1`, `fc00::/7`, `*.local`) unless `WEBTOOLS_ALLOW_PRIVATE=1`.\n- **Caps:** per-request timeout (30s) and `maxBytes` (~5 MB); crawl caps per-page (30 KB) and total output (200 KB), honors robots.txt, rate-limits, and a concurrency cap.\n- **Search** scrapes a public engine's HTML, so it's best-effort: on a consent/captcha/changed page it **throws** a clear error (try another `engine`).\n\n## Install / run\n\n```bash\npi install npm:@aprimediet/webtools\npi list\n\n# Quick try without installing\npi -e ./extensions/webtools/index.ts\n\n# Hot-reload during dev\n/reload\n```\n\n## Layout\n\n```\nwebtools/                 # @aprimediet/webtools\n├── package.json          # pi manifest + deps\n├── index.ts              # factory: 4 tools + flags + /webtools + cleanup\n├── browser.ts            # SSRF guard + shared Playwright + fetchHtml\n├── extract.ts            # Readability + selectors\n├── search.ts             # Playwright SERP scraping (ddg/bing/brave)\n├── crawl.ts              # BFS crawler\n├── markdown.ts           # Turndown/jsdom HTML→markdown/text helpers\n└── config.ts             # env/flag-driven config\n```\n","readmeFilename":"README.md","_rev":"1-305242a46a863f5711506e999bb963ad"}