{"_id":"@autumnice/agent-crawl","_rev":"2-0183a6637b997ae11158873bab737bca","name":"@autumnice/agent-crawl","dist-tags":{"latest":"0.1.1"},"versions":{"0.1.0":{"name":"@autumnice/agent-crawl","version":"0.1.0","keywords":["crawler","browser","automation","ai","extraction","cli"],"author":{"name":"autumnice"},"license":"MIT","_id":"@autumnice/agent-crawl@0.1.0","maintainers":[{"name":"autumnice","email":"autumnice@163.com"}],"homepage":"https://github.com/autumnice/agent-crawl#readme","bugs":{"url":"https://github.com/autumnice/agent-crawl/issues"},"bin":{"agent-crawl":"dist/cli.js"},"dist":{"shasum":"021143c7f08b22fbcada6b0ff31ceea47f9a67a1","tarball":"https://registry.npmjs.org/@autumnice/agent-crawl/-/agent-crawl-0.1.0.tgz","fileCount":4,"integrity":"sha512-bcm8AG6NG375It4fSrzog73BRcLyPBksyGzdqWFnogxM60Y2ypuQyk4upDHneNE+JWCyhJku0q8lMCesVXoCXA==","signatures":[{"sig":"MEYCIQCVLrnfIkqlj++ZOnxos9C1ZtHGJMVV1q6DwNPjQcSdlQIhAJux/nurcNE2ivvFvriFSWnR4olo02aUNhq4CvrKK+8f","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":223380},"type":"module","engines":{"node":">=22"},"gitHead":"ca5348fe1e4c200ef96787d992bb1da3703742bf","scripts":{"dev":"tsx src/cli.ts","lint":"eslint src tests vitest.config.ts","test":"vitest run","build":"tsup","typecheck":"tsc --noEmit -p tsconfig.json && tsc --noEmit -p tsconfig.test.json","postinstall":"agent-browser install","format:check":"prettier --check \"README*.md\" \"docs/**/*.md\" \"package*.json\" \"tsconfig*.json\" \"*.config.ts\" \"src/**/*.ts\" \"tests/**/*.ts\" --ignore-path .gitignore","test:coverage":"vitest run --coverage","prepublishOnly":"npm run build"},"_npmUser":{"name":"autumnice","email":"autumnice@163.com"},"repository":{"url":"git+https://github.com/autumnice/agent-crawl.git","type":"git"},"_npmVersion":"11.5.2","description":"AI-assisted browser crawling CLI for structured extraction and file export","directories":{},"_nodeVersion":"22.18.0","dependencies":{"cheerio":"^1.2.0","exceljs":"^4.4.0","turndown":"^7.2.4","commander":"^14.0.3","agent-browser":"^0.26.0"},"publishConfig":{"access":"public","registry":"https://registry.npmjs.org"},"_hasShrinkwrap":false,"devDependencies":{"tsx":"^4.21.0","tsup":"^8.5.1","eslint":"^10.2.1","vitest":"^4.1.5","prettier":"^3.8.3","@eslint/js":"^10.0.1","typescript":"^6.0.3","@types/node":"^25.6.0","@types/turndown":"^5.0.6","typescript-eslint":"^8.59.1","@vitest/coverage-v8":"^4.1.5"},"_npmOperationalInternal":{"tmp":"tmp/agent-crawl_0.1.0_1778150906493_0.40490611583154035","host":"s3://npm-registry-packages-npm-production"}},"0.1.1":{"name":"@autumnice/agent-crawl","version":"0.1.1","description":"AI-assisted browser crawling CLI for structured extraction and file export","type":"module","license":"MIT","author":{"name":"autumnice"},"repository":{"type":"git","url":"git+https://github.com/autumnice/agent-crawl.git"},"publishConfig":{"registry":"https://registry.npmjs.org","access":"public"},"keywords":["crawler","browser","automation","ai","extraction","cli"],"bin":{"agent-crawl":"dist/cli.js"},"engines":{"node":">=22"},"scripts":{"dev":"tsx src/cli.ts","build":"tsup","test":"vitest run","test:coverage":"vitest run --coverage","lint":"eslint src tests vitest.config.ts","format:check":"prettier --check \"README*.md\" \"docs/**/*.md\" \"package*.json\" \"tsconfig*.json\" \"*.config.ts\" \"src/**/*.ts\" \"tests/**/*.ts\" --ignore-path .gitignore","typecheck":"tsc --noEmit -p tsconfig.json && tsc --noEmit -p tsconfig.test.json","prepublishOnly":"npm run build","postinstall":"agent-browser install"},"dependencies":{"agent-browser":"^0.26.0","cheerio":"^1.2.0","commander":"^14.0.3","exceljs":"^4.4.0","turndown":"^7.2.4"},"devDependencies":{"@eslint/js":"^10.0.1","@types/node":"^25.6.0","@types/turndown":"^5.0.6","@vitest/coverage-v8":"^4.1.5","eslint":"^10.2.1","prettier":"^3.8.3","tsup":"^8.5.1","tsx":"^4.21.0","typescript":"^6.0.3","typescript-eslint":"^8.59.1","vitest":"^4.1.5"},"_id":"@autumnice/agent-crawl@0.1.1","gitHead":"ca32d7635a0ca6cbd453bcf6d70f775e311f3b9b","bugs":{"url":"https://github.com/autumnice/agent-crawl/issues"},"homepage":"https://github.com/autumnice/agent-crawl#readme","_nodeVersion":"22.18.0","_npmVersion":"11.5.2","dist":{"integrity":"sha512-KWsJh7mzjYuYqaGw0cRBLmqIYmA/jYREl/6HkVaqr2Y6TF4FdKHKtevh7kZ+tEX3DXL7s4jvWmutI/eT0gD0wA==","shasum":"3aec327ce5f00a761f152cb2d55251013d5d54ea","tarball":"https://registry.npmjs.org/@autumnice/agent-crawl/-/agent-crawl-0.1.1.tgz","fileCount":4,"unpackedSize":223380,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEYCIQCwDkXbcDxZURXMtXcSGBaXxhYK/CTlSmEccP3ADoXNSQIhANyeRmveFAXRCIYyLmsI1zjDni7ClJk6yJnwXF2Ckt56"}]},"_npmUser":{"name":"autumnice","email":"autumnice@163.com"},"directories":{},"maintainers":[{"name":"autumnice","email":"autumnice@163.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/agent-crawl_0.1.1_1778204695946_0.5263012590377454"},"_hasShrinkwrap":false}},"time":{"created":"2026-05-07T10:48:26.394Z","modified":"2026-05-08T01:44:56.240Z","0.1.0":"2026-05-07T10:48:26.625Z","0.1.1":"2026-05-08T01:44:56.108Z"},"bugs":{"url":"https://github.com/autumnice/agent-crawl/issues"},"author":{"name":"autumnice"},"license":"MIT","homepage":"https://github.com/autumnice/agent-crawl#readme","keywords":["crawler","browser","automation","ai","extraction","cli"],"repository":{"type":"git","url":"git+https://github.com/autumnice/agent-crawl.git"},"description":"AI-assisted browser crawling CLI for structured extraction and file export","maintainers":[{"name":"autumnice","email":"autumnice@163.com"}],"readme":"# Agent-Crawl\n\nAI 辅助的浏览器采集 CLI。\n\n它和 `agent-e2e` 使用相似入口，但目标不同：`agent-e2e` 产出可回放测试配置，`agent-crawl` 面向网页内容采集、语义抽取、筛选和文件导出。\n\n## 快速开始\n\n```bash\nnpm install\nnpm run build\n\nagent-crawl config set llm.apiKey <your-key>\nagent-crawl https://example.com\n```\n\n也可以用环境变量：\n\n```bash\nexport AGENT_CRAWL_LLM_API_KEY=<your-key>\nexport AGENT_CRAWL_LLM_MODEL=gpt-4o\n```\n\n进入 Chat 后可以直接说：\n\n```text\n当前页面主列表里的详情内容你爬下，转成 markdown 格式，放到 ~/Downloads/details.md\n```\n\n```text\n当前列表页获取 id、姓名，然后到每条详情页里获取用户电话号码，导出 id、姓名、电话号码 三列 Excel\n```\n\n```text\n当前页下拉过滤浙江省、杭州市，再过滤市场销售。过滤后的列表每页 detail 都进去，在详情页工作经历里语义匹配教育或互联网直播相关经历，符合的整理成 Excel，列是 id、姓名、电话号码、工作单位\n```\n\n启动后会先输出访问诊断，例如 `public_static`、`public_dynamic`、`auth_required`、\n`auth_available`、`captcha_or_verify`、`permission_blocked` 或 `rate_limited`，并自动选择\nstatic fetch、Chrome Headless、browser session、handoff 或 fail-closed。普通用户不需要\n提前判断目标站点类型。\n\n## CLI\n\n```bash\nagent-crawl <url>\nagent-crawl chat <url>\nagent-crawl config\nagent-crawl config set llm.apiKey <key>\nagent-crawl config show\n```\n\n### 高级用法\n\n需要复用浏览器登录态、做 canary 或让外部 agent 精确控制浏览器时，再使用这些参数：\n\n```bash\nagent-browser profiles\nagent-crawl https://example.com/folder --headed\nagent-crawl https://example.com/folder --browser-state ~/.agent-crawl/auth/site.json\nagent-crawl https://example.com/folder --browser-auto-connect\nagent-crawl https://example.com/folder --browser-cdp 9222\n```\n\n这些模式适合短信验证码、第三方登录或用户已在 Chrome 中登录的站点。普通入口诊断到\n`auth_required` 时，会提示用户在打开的浏览器窗口完成登录，再回到 Chat 输入“继续”。\n`继续` 会重新运行访问诊断；只有 blocker 解除后才恢复采集。它们只复用用户已授权的\n浏览器状态；遇到验证码、登录失效、限流或访问控制仍会 fail closed，不会绕过验证。\n\nChat 中的 `auth_handoff` 会返回当前 URL/title、页面 blocker、`navigator.webdriver`、是否已连接 CDP，以及可直接重启的 `--headed` / `--browser-cdp` 命令建议。若 discovery 只看到登录页导航、账号菜单或其他低置信候选，pipeline 会停止抓详情并在 manifest 里记录 warning。\n\n浏览器会话的同源策略只用于 discovery/采集裁决，不会默认传给 `agent-browser --allowed-domains` 做网络资源过滤。很多登录态 SPA 会从独立 CDN、上传域或静态资源域加载编辑器、图片和正文通道；如果把 seed host 当成浏览器资源白名单，详情页可能表现为空白并在 DevTools 中出现 `ERR_BLOCKED_BY_CLIENT.Inspector`。\n\n部分高风控登录页会识别受控浏览器环境；如果人工点击登录后按钮变灰、无短信或无跳转，改用用户手动启动的 Chrome CDP 模式：\n\n```bash\nopen -na \"Google Chrome\" --args \\\n  --remote-debugging-address=127.0.0.1 \\\n  --remote-debugging-port=9222 \\\n  --user-data-dir=\"$HOME/.agent-crawl/chrome-cdp-profile\"\n\n# 在这个 Chrome 窗口里手动登录目标站点，然后：\nagent-crawl https://example.com/folder --browser-cdp 9222\n```\n\n如果要复用日常 Chrome 的 `Default` profile，需要用户先自行关闭 Chrome 并用 CDP 参数重新打开；不要由脚本强制关闭用户浏览器。\n\n默认数据目录是 `~/.agent-crawl`，可通过 `AGENT_CRAWL_HOME` 修改。每次会话会创建独立产物目录，Chat 里的 `write_markdown`、`write_excel`、`write_json` 工具会把文件落到用户指定路径或默认产物目录。\n\n## 当前边界\n\n第一版初始化的是基础架构和可运行 Chat 主链路：\n\n- 自动入口：`agent-crawl <url>` 会先运行 `AccessDiagnosis`，再自动选择 static\n  fetch、Chrome Headless、browser session、handoff 或 fail-closed。\n- 浏览器直连：基于 `agent-browser` 打开页面、读取 snapshot、点击、填写、滚动、读取页面文本。\n- 凭证隔离：启动参数中的密码只通过 `browser_fill_secret` 使用，不进入普通工具参数。\n- 登录 handoff：支持 `--headed` 下人工完成短信验证码/SSO/第三方登录，再由 agent 复用同一 browser session 继续采集。\n- 通用列表/详情发现：static href、browser href、SPA click probe 会合并成带 `role/confidence/blocker` 的发现合同；登录页、权限页、低置信导航和空白加载页 fail closed。\n- 页面证据：`read_current_page` 会抽取当前页面文本、markdown 和链接。\n- 访问 fallback：`resilient_fetch_page` 会按 curl 移动端 UA、Chrome Headless dump-dom、浏览器 session DOM 的顺序读取公开页面。\n- 正文证据：browser session DOM 等待正文候选区域稳定，ContentEvidence 保存候选根评分/原因/备选项；无 LLM transform 只有正文密度达标才会 high confidence。\n- 微信公众号合集：`fetch_wechat_album_articles` 可抓取 `mp.weixin.qq.com/mp/appmsgalbum` 前 N 篇文章正文；支持合并 Markdown，或每篇独立 Markdown + 标题/原文链接 Excel 索引。\n- 语义抽取：`extract_records` 使用 LLM 把页面内容转成结构化 records。\n- 语义筛选：`semantic_filter_records` 用 LLM 判断模糊条件，避免只能靠关键词。\n- 文件导出：支持 Markdown、JSON、Excel。\n\n访问 fallback 的边界是 fail closed：如果页面返回验证码、登录、限流或访问限制，工具会报告阻塞，不会绕过验证码、伪造身份或规避访问控制。\n\n低置信结果不会静默导出 Markdown。raw HTML、evidence 和 transform 可以保留用于复查，但 `confidence=low` 或空正文会阻止 Markdown 导出。\n\n后续要增强的是自动化 executor：分页遍历、列表项详情页展开、断点续跑、去重、限速、增量采集和更强的抽取 schema。\n\n## 开发\n\n```bash\nnpm run typecheck\nnpm run test\nnpm run lint\nnpm run build\n```\n\n## 架构\n\n详细说明：\n\n- [Architecture](docs/design/ARCHITECTURE.md)\n- [Content Evidence Pipeline](docs/design/CONTENT_EVIDENCE_PIPELINE.md)\n- [Auto Crawl Frontdoor](docs/design/AUTO_CRAWL_FRONTDOOR.md)\n","readmeFilename":"README.md"}