{"_id":"dom-miner","_rev":"5-8468a8dd41a699211367fc1291faf1ea","name":"dom-miner","dist-tags":{"latest":"0.1.4"},"versions":{"0.1.0":{"name":"dom-miner","version":"0.1.0","keywords":["playwright","qa","sitemap","dom","dom-miner","test-plan","agent","ai-testing","llm"],"license":"MIT","_id":"dom-miner@0.1.0","maintainers":[{"name":"latiasthinh","email":"thomas.ng4542@gmail.com"}],"bin":{"dom-miner":"dist/cli.js"},"dist":{"shasum":"a0c23d372d1a64f621a643310915c7dcd1dbe417","tarball":"https://registry.npmjs.org/dom-miner/-/dom-miner-0.1.0.tgz","fileCount":70,"integrity":"sha512-r5g+lUwrFrDz2vwiSEAxZRAznEBdp+R00ySoQyx2EnVopJxkvsbN1AaatxHEoVDoIAtKn1KNmqgTCZTowmJLeA==","signatures":[{"sig":"MEYCIQDKUOry/wLlnKPl+gZnsax9MnyKHoqpIEAX39UfQ0MnrAIhAOTjQoSkbKKlBo6xKEc3pQDsDtaIigpRyBXdULb7HnhW","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":236427},"main":"./dist/index.js","type":"module","types":"./dist/index.d.ts","engines":{"node":">=20"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js"}},"gitHead":"03d0c8d847c0a8486fc4860ca47266a9b1f31b5f","scripts":{"test":"node --test \"test/*.test.mjs\"","build":"tsc -p tsconfig.json","prepare":"npm run build","pretest":"npm run build","dom-miner":"node dist/cli.js","dom-miner:deep":"dom-miner deep","dom-miner:dump":"dom-miner dump","dom-miner:page":"dom-miner explore page","dom-miner:read":"dom-miner read","dom-miner:site":"dom-miner explore site","dom-miner:urls":"dom-miner explore urls","dom-miner:compact":"dom-miner compact","dom-miner:explore":"dom-miner explore","dom-miner:benchmark":"dom-miner benchmark"},"_npmUser":{"name":"latiasthinh","email":"thomas.ng4542@gmail.com"},"_npmVersion":"11.3.0","description":"dom-miner — mine any site into a compact DOM map for QA test plans and AI agents","directories":{},"_nodeVersion":"24.0.2","dependencies":{"dotenv":"^16.4.0 || ^17.0.0"},"_hasShrinkwrap":false,"devDependencies":{"typescript":"^5.9.0","@types/node":"^24.0.0","playwright-core":"^1.62.0"},"_npmOperationalInternal":{"tmp":"tmp/dom-miner_0.1.0_1785882107178_0.488415107709512","host":"s3://npm-registry-packages-npm-production"}},"0.1.1":{"name":"dom-miner","version":"0.1.1","keywords":["playwright","qa","sitemap","dom","dom-miner","test-plan","agent","ai-testing","llm"],"license":"MIT","_id":"dom-miner@0.1.1","maintainers":[{"name":"latiasthinh","email":"thomas.ng4542@gmail.com"}],"bin":{"dom-miner":"dist/cli.js"},"dist":{"shasum":"45bcb93a910cb56c0aaa6dd9dbff298326d2f264","tarball":"https://registry.npmjs.org/dom-miner/-/dom-miner-0.1.1.tgz","fileCount":70,"integrity":"sha512-GQVokOXVFqoxvHTEMOc3/wt/GLCsUFkbDDTkYEjPfC70UO9dK+O2C1vR+mO9fhQqtmI/CZGtIPPGHRvRqLZQgA==","signatures":[{"sig":"MEUCIAjFiRbyetNaiJdexlAwN1pDjTusjorP+lOJbXQzDfL1AiEAngT+VfUVnjCJFKwAF0N6NbItCx3ZLi1lRHbs1VI4O4Q=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":234496},"main":"./dist/index.js","type":"module","types":"./dist/index.d.ts","engines":{"node":">=20"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js"}},"gitHead":"29bd7cc181535222f27a65aed7d9ba0225a0be58","scripts":{"test":"node --test test/*.test.mjs","build":"tsc -p tsconfig.json","prepare":"npm run build","pretest":"npm run build","dom-miner":"node dist/cli.js","dom-miner:deep":"dom-miner deep","dom-miner:dump":"dom-miner dump","dom-miner:page":"dom-miner explore page","dom-miner:read":"dom-miner read","dom-miner:site":"dom-miner explore site","dom-miner:urls":"dom-miner explore urls","dom-miner:compact":"dom-miner compact","dom-miner:explore":"dom-miner explore","dom-miner:benchmark":"dom-miner benchmark"},"_npmUser":{"name":"latiasthinh","email":"thomas.ng4542@gmail.com"},"_npmVersion":"11.3.0","description":"dom-miner — mine any site into a compact DOM map for QA test plans and AI agents","directories":{},"_nodeVersion":"24.0.2","dependencies":{"dotenv":"^16.4.0 || ^17.0.0"},"_hasShrinkwrap":false,"devDependencies":{"typescript":"^5.9.0","@types/node":"^24.0.0","playwright-core":"^1.62.0"},"_npmOperationalInternal":{"tmp":"tmp/dom-miner_0.1.1_1785889389159_0.7902892708368936","host":"s3://npm-registry-packages-npm-production"}},"0.1.2":{"name":"dom-miner","version":"0.1.2","keywords":["playwright","qa","sitemap","dom","dom-miner","test-plan","agent","ai-testing","llm"],"license":"MIT","_id":"dom-miner@0.1.2","maintainers":[{"name":"latiasthinh","email":"thomas.ng4542@gmail.com"}],"bin":{"dom-miner":"dist/cli.js"},"dist":{"shasum":"f2afd9ded141eff8456f4e9742b7cfbc2e685b45","tarball":"https://registry.npmjs.org/dom-miner/-/dom-miner-0.1.2.tgz","fileCount":70,"integrity":"sha512-UgHLO7tjE++oKptQiRSZTCGxwHKEy5F5Z3v20K/XxMnzFdJ2Z4BuAVKRPak2sEAacgt+cya16UiwWCOKE0zaTw==","signatures":[{"sig":"MEUCIA/QPIKwqSleu+g3Veh7z148afCoHRxXNx1V6tL7LRHVAiEAjtpeAf6z4bKq3gkYqb8Esq3pq3N0ihjG6ku0YNf53E4=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":234496},"main":"./dist/index.js","type":"module","types":"./dist/index.d.ts","engines":{"node":">=20"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js"}},"gitHead":"29bd7cc181535222f27a65aed7d9ba0225a0be58","scripts":{"test":"node --test test/*.test.mjs","build":"tsc -p tsconfig.json","prepare":"npm run build","pretest":"npm run build","dom-miner":"node dist/cli.js","dom-miner:deep":"dom-miner deep","dom-miner:dump":"dom-miner dump","dom-miner:page":"dom-miner explore page","dom-miner:read":"dom-miner read","dom-miner:site":"dom-miner explore site","dom-miner:urls":"dom-miner explore urls","dom-miner:compact":"dom-miner compact","dom-miner:explore":"dom-miner explore","dom-miner:benchmark":"dom-miner benchmark"},"_npmUser":{"name":"latiasthinh","email":"thomas.ng4542@gmail.com"},"_npmVersion":"11.3.0","description":"dom-miner — mine any site into a compact DOM map for QA test plans and AI agents","directories":{},"_nodeVersion":"24.0.2","dependencies":{"dotenv":"^16.4.0 || ^17.0.0"},"_hasShrinkwrap":false,"devDependencies":{"typescript":"^5.9.0","@types/node":"^24.0.0","playwright-core":"^1.62.0"},"_npmOperationalInternal":{"tmp":"tmp/dom-miner_0.1.2_1785889406389_0.5453167287790046","host":"s3://npm-registry-packages-npm-production"}},"0.1.3":{"name":"dom-miner","version":"0.1.3","keywords":["playwright","qa","sitemap","dom","dom-miner","test-plan","agent","ai-testing","llm"],"license":"MIT","_id":"dom-miner@0.1.3","maintainers":[{"name":"latiasthinh","email":"thomas.ng4542@gmail.com"}],"bin":{"dom-miner":"dist/cli.js"},"dist":{"shasum":"c5bab7e15bde3b836d4b9eb9848aa355e3c2e4e1","tarball":"https://registry.npmjs.org/dom-miner/-/dom-miner-0.1.3.tgz","fileCount":78,"integrity":"sha512-fVvPWNQ8YFpUQFuZRbjtwREdrAW5XhftbVy9Ms02kAV5+TdK5Ymk9vZ7sCIeTKhcV3EjkqLVJ3y4x/SVqXJ/lg==","signatures":[{"sig":"MEUCIQDL7QaQ9e2ZOG1pgPrIvabOsJyYVpDpWIPD4XHuQ78jxAIgPXy0yYMQlmr2NRLYme6CzWKhbdXpGgR/NVv2GwhXHR4=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":258664},"main":"./dist/index.js","type":"module","types":"./dist/index.d.ts","engines":{"node":">=20"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js"}},"gitHead":"689a122792562a6f9c075c9f77425d871e64cc0b","scripts":{"test":"node --test test/*.test.mjs","build":"tsc -p tsconfig.json","prepare":"npm run build","pretest":"npm run build","dom-miner":"node dist/cli.js","dom-miner:deep":"dom-miner deep","dom-miner:dump":"dom-miner dump","dom-miner:page":"dom-miner explore page","dom-miner:read":"dom-miner read","dom-miner:site":"dom-miner explore site","dom-miner:urls":"dom-miner explore urls","dom-miner:compact":"dom-miner compact","dom-miner:explore":"dom-miner explore","dom-miner:benchmark":"dom-miner benchmark"},"_npmUser":{"name":"latiasthinh","email":"thomas.ng4542@gmail.com"},"_npmVersion":"11.3.0","description":"dom-miner — mine any site into a compact DOM map for QA test plans and AI agents","directories":{},"_nodeVersion":"24.0.2","dependencies":{"dotenv":"^16.4.0 || ^17.0.0"},"_hasShrinkwrap":false,"devDependencies":{"typescript":"^5.9.0","@types/node":"^24.0.0","playwright-core":"^1.62.0"},"_npmOperationalInternal":{"tmp":"tmp/dom-miner_0.1.3_1786010100460_0.20572618094287987","host":"s3://npm-registry-packages-npm-production"}},"0.1.4":{"name":"dom-miner","version":"0.1.4","description":"dom-miner — mine any site into a compact DOM map for QA test plans and AI agents","type":"module","bin":{"dom-miner":"dist/cli.js"},"main":"./dist/index.js","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js"}},"scripts":{"build":"tsc -p tsconfig.json","prepare":"npm run build","test":"node --test test/*.test.mjs","pretest":"npm run build","dom-miner":"node dist/cli.js","dom-miner:explore":"dom-miner explore","dom-miner:site":"dom-miner explore site","dom-miner:page":"dom-miner explore page","dom-miner:urls":"dom-miner explore urls","dom-miner:dump":"dom-miner dump","dom-miner:compact":"dom-miner compact","dom-miner:deep":"dom-miner deep","dom-miner:read":"dom-miner read","dom-miner:benchmark":"dom-miner benchmark"},"engines":{"node":">=20"},"dependencies":{"dotenv":"^16.4.0 || ^17.0.0"},"devDependencies":{"playwright-core":"^1.62.0","@types/node":"^24.0.0","typescript":"^5.9.0"},"keywords":["playwright","qa","sitemap","dom","dom-miner","test-plan","agent","ai-testing","llm"],"license":"MIT","_id":"dom-miner@0.1.4","gitHead":"689a122792562a6f9c075c9f77425d871e64cc0b","_nodeVersion":"24.0.2","_npmVersion":"11.3.0","dist":{"integrity":"sha512-tOZ7g85GQ5/3SgMwkqHTAtpfD476f6T3kX+H+IDVcZD4QYyPYv0hd87YECLLSiB1/GPrXG+0n7hCoLBVUj6meA==","shasum":"9a55d029ef3046b9b6ba560dc281c694ba0bed5a","tarball":"https://registry.npmjs.org/dom-miner/-/dom-miner-0.1.4.tgz","fileCount":78,"unpackedSize":258664,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEQCICZTK0vuBboYosKMyWUfeFQiPHFT4wumhWUpWNtM5FonAiAz6t527b8L+fV3zKj3j5CRedEtTW3PZcokW8jZ8CQ6qw=="}]},"_npmUser":{"name":"latiasthinh","email":"thomas.ng4542@gmail.com"},"directories":{},"maintainers":[{"name":"latiasthinh","email":"thomas.ng4542@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/dom-miner_0.1.4_1786010121965_0.21452640747046647"},"_hasShrinkwrap":false}},"time":{"created":"2026-08-04T22:21:47.005Z","modified":"2026-08-06T09:55:22.286Z","0.1.0":"2026-08-04T22:21:47.351Z","0.1.1":"2026-08-05T00:23:09.306Z","0.1.2":"2026-08-05T00:23:26.541Z","0.1.3":"2026-08-06T09:55:00.608Z","0.1.4":"2026-08-06T09:55:22.113Z"},"license":"MIT","keywords":["playwright","qa","sitemap","dom","dom-miner","test-plan","agent","ai-testing","llm"],"description":"dom-miner — mine any site into a compact DOM map for QA test plans and AI agents","maintainers":[{"name":"latiasthinh","email":"thomas.ng4542@gmail.com"}],"readme":"# dom-miner\n\n![dom-miner banner](dom-miner-banner.svg)\n\n**dom-miner** turns any website into a compact, structured DOM map — the perfect input for QA test-plan generation, AI agents, and test-case authoring. Skip the raw HTML noise. Get only the elements that matter, with numeric IDs, regions, and Playwright-ready locators.\n\n## Quick Start\n\n```bash\n# 1. Install\nnpm i -D dom-miner\n\n# 2. Initialize (npm scripts + gitignore)\nnpx dom-miner init\n\n# 3. Install browsers\nnpx playwright-core install chromium\n\n# 4. Explore a full site\nnpx dom-miner explore site --url https://example.com/ --top 40\n```\n\n## Explore Modes\n\n| Mode | Command | When |\n|------|---------|------|\n| **site** | `dom-miner explore site --url <home> --top 40` | Full site via sitemap/crawl — scope your test plan |\n| **page** | `dom-miner explore page --url <url>` | Single page dump — deep-dive one page |\n| **urls** | `dom-miner explore urls --urls-file list.txt` | Explicit URL set — map a feature slice |\n\n```bash\n# Full sitemap explore\nnpx dom-miner explore site --url https://example.com/ --top 50\n\n# Single page\nnpx dom-miner explore page --url https://example.com/about/ --stem example\n\n# URL set from file\nnpx dom-miner explore urls --urls-file ./module-urls.txt --stem example\n\n# URL set inline\nnpx dom-miner explore urls --stem example \\\n  --url https://example.com/super/ \\\n  --url https://example.com/investments/\n```\n\n## Sample Output\n\nA compact tree looks like this — clean, scannable, and token-efficient:\n\n```\nPage map: Example Site - About\nURL: https://example.com/about/\nNodes: text-holders 24, interactive 18 (visible 15, collapsed-nav 3)\n────────────────────────────────────────────────────────────\n[header]\n  text:heading1 \"About Us\"\n  text:p \"We build software that matters.\"\n  [1] link \"Home\" href=/\n  [2] link \"Products\" href=/products\n\n[nav]\n  [3] button \"Menu\"\n  [4] link \"About\" href=/about\n  [5] link \"Careers\" href=/careers (collapsed)\n  [6] link \"Team\" href=/about/team (collapsed)\n\n[main]\n  text:heading2 \"Our Mission\"\n  text:p \"To empower teams with tools that scale.\"\n  [7] link \"Read More\" href=/mission\n  [8] button \"Contact Sales\"\n  [9] textbox \"Email\"\n\n[footer]\n  text:p \"© 2024 Example Corp\"\n  [10] link \"Privacy\" href=/privacy\n  [11] link \"Terms\" href=/terms\n```\n\n## SPA / CSR Sites\n\ndom-miner reads the **live DOM** after Playwright navigation. For React/Vue/Angular apps:\n\n```bash\n# CSR/SPA mode with auto-settle + thin-shell retry\nnpx dom-miner explore site --url https://spa.example/ --spa --top 40\nnpx dom-miner explore page --url https://spa.example/dashboard --spa --ready \"main\" --with-deep\n```\n\n| Flag | Effect |\n|------|--------|\n| `--spa` | `load` wait, longer settle (≥3s), lazy scroll, thin-shell retry |\n| `--ready <css>` | Wait until selector is visible before read |\n| `--wait-until` | `commit` · `domcontentloaded` · `load` · `networkidle` |\n| `--scroll` | Scroll page to trigger lazy/intersection-observer content |\n| `--settle-ms` | Extra fixed delay after navigation |\n\nThin-shell auto-retry is **on by default** when the page looks empty after initial load.\n\n## URL List File Formats (`--urls-file`)\n\n| Extension | Format |\n|-----------|--------|\n| `.txt` | One URL per line (`#` comments OK) |\n| `.csv` | `url` column or first column |\n| `.json` | `string[]`, `{ urls: [...] }`, or dom-miner `urls-full.json` |\n\n## Batch JSON Files (`--batch-file`)\n\nFor exploring multiple URLs with optional authentication, pass a JSON file:\n\n```bash\ndom-miner explore urls --batch-file sites.json --stem myapp\n```\n\n**Format:**\n```json\n[\n  { \"url\": \"https://example.com/\" },\n  { \"url\": \"https://app.example.com/dashboard\", \"credential\": { \"username\": \"admin\", \"password\": \"secret\" } },\n  { \"url\": \"https://app.example.com/settings\", \"credential\": { \"username\": \"admin\", \"password\": \"secret\" } }\n]\n```\n\nOr as a single entry:\n```json\n{ \"url\": \"https://example.com/\", \"credential\": { \"username\": \"user\", \"password\": \"pass\" } }\n```\n\nWhen credentials are provided, dom-miner will:\n1. Navigate to the URL\n2. Auto-detect the login form (username + password fields)\n3. Fill credentials and submit\n4. Proceed with DOM exploration on the authenticated page\n\n**Supported batch formats:**\n- `{ url, credential? }` — single entry\n- `[{ url, credential? }, ...]` — array\n- `{ urls: [{ url, credential? }] }` — wrapped\n- `{ pages: [{ url, credential? }] }` — from dom-miner output\n\n## Global Authentication (`--username` / `--password`)\n\nFor authenticated exploration of an entire site or URL list, provide credentials once:\n\n```bash\n# Single URL with auth\ndom-miner explore page --url https://app.example.com/ -u admin -p secret --stem myapp\n\n# Multiple URLs with auth (credentials applied to all)\ndom-miner explore urls \\\n  --url https://app.example.com/dashboard \\\n  --url https://app.example.com/settings \\\n  -u admin -p secret --stem myapp\n\n# Full site with auth (discover + dump all pages)\ndom-miner explore site --url https://app.example.com/ -u admin -p secret --top 40\n\n# Batch file with global auth (global overrides per-URL credentials)\ndom-miner explore urls --batch-file sites.json -u admin -p secret --stem myapp\n```\n\nWhen `--username` / `--password` are provided:\n1. **Authenticate once** at the start (auto-detects login form, or use `--login-url`)\n2. **Reuse the session** for all URL exploration\n3. All pages are dumped with the authenticated browser context\n\n## Other Commands\n\n| Command | What |\n|---------|------|\n| `dom-miner init` | Add npm scripts + gitignore to current repo |\n| `dom-miner compact --url <u>` | Quick compact tree (stdout / `--out`) |\n| `dom-miner deep --url <u>` | Quick deep inventory |\n| `dom-miner benchmark --url <u>` | Compact vs deep metrics |\n| `dom-miner read --url <u>` | Compact, deep, or both |\n\n## Compact vs Deep\n\ndom-miner offers two read depths from the **same browser session**:\n\n| | Compact | Deep |\n|---|---|---|\n| **Purpose** | Test-plan scoping, multi-page mapping | TC authoring, codegen, locator healing |\n| **Output** | Pruned tree with regions + numeric IDs | Full interactive JSON with all hidden nodes |\n| **Tokens** | ~5–20× fewer than deep | Complete element inventory |\n| **Locators** | Lightweight hints | Every element gets a Playwright locator |\n| **Speed** | Fast read | Slower, but same browser state |\n\nYou don't need two browser stacks. One Playwright session → two read depths.\n\n## Artifacts\n\n```\ndata/dom-miner/<stem>/\n├── manifest.json            ← Site-level summary\n├── README.md                ← Human-readable page list\n└── <pageId>/\n    ├── compact.tree.txt     ← Agent-readable tree (the main output)\n    ├── compact.json         ← Structured compact data\n    ├── deep.json            ← Full inventory (if --with-deep)\n    └── observe-meta.json    ← Timing, tokens, metrics\n\ntest-output/sitemap/<stem>-urls-full.json  ← Full URL inventory\n```\n\n## Cloudflare / Bot Walls\n\nWhen a page is blocked (Cloudflare challenge, bot detection):\n- Evidence dump is kept for debugging\n- Page is marked `blocked` in `manifest.json`\n- Excluded from `pageCount` and token totals\n- Expand is skipped (clicks won't help behind a wall)\n\n## Library API\n\n```ts\nimport {\n  discoverSiteUrls,\n  dumpPagesToData,\n  loadUrlsFromFile,\n  runCompactObserve,\n  runDeepInventory,\n} from 'dom-miner';\n```\n\n| Export | Use |\n|--------|-----|\n| `discoverSiteUrls(homeUrl, opts)` | Sitemap discovery + ranking |\n| `dumpPagesToData({ root, stem, urls, ... })` | Batch page dump loop |\n| `runCompactObserve(page)` | Compact DOM read |\n| `runDeepInventory(page)` | Deep inventory read |\n| `loadUrlsFromFile(path)` | Parse .txt / .csv / .json URL lists |\n| `stemFromUrl(url)` | Derive output folder name from URL |\n| `pageIdFromUrl(url)` | Derive page directory name from URL |\n| `sanitizeStem(str)` | Sanitize user-supplied stem |\n\n## Environment Variables\n\n| Variable | Purpose |\n|----------|---------|\n| `MINE_URL` | Default `--url` |\n| `TEST_BASE_URL` | Fallback default `--url` |\n| `DOM_MINER_ROOT` | Force project root (default: walk-up from cwd) |\n\n## Develop\n\n```bash\nnpm install\nnpm run build\nnpm test        # 35 tests, zero dependencies\n```\n\nTypeScript source in `src/`, compiled to `dist/`. Library entry at `src/index.ts`, CLI entry at `src/cli.ts`.\n\n## License\n\nMIT\n","readmeFilename":"README.md"}