{"_id":"@derped/docexplorer","_rev":"2-402efa545eb290e475659694b1517271","name":"@derped/docexplorer","dist-tags":{"latest":"0.1.1"},"versions":{"0.1.0":{"name":"@derped/docexplorer","version":"0.1.0","keywords":["documentation","crawler","cli","markdown","openrouter","docs","scraper"],"author":"","license":"MIT","_id":"@derped/docexplorer@0.1.0","maintainers":[{"name":"derped","email":"derpedyea@pm.me"}],"bin":{"docexplorer":"dist/docexplorer.js"},"dist":{"shasum":"31a7fe7ef2ae251ae6b8451cf3e1afcf950805fe","tarball":"https://registry.npmjs.org/@derped/docexplorer/-/docexplorer-0.1.0.tgz","fileCount":5,"integrity":"sha512-lGtGxeSaNFD/ac4BwZ6/3qHFMXZDF5IP5w67C/HxdHAA8o43bwFyT2edHdvEz85blwStjm3SZhcPOB2ou2JN5Q==","signatures":[{"sig":"MEUCIEhUyQOaregUCYWubUvvcSggOSeEVVMPQuppfXQ6p1uwAiEA8Nd1Zqg2LNDNf+Mln5FJP28jpXp1CdznykH6r/PrI+M=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":59038},"main":"dist/docexplorer.js","type":"module","engines":{"node":">=18.0.0"},"gitHead":"2c1936d7a21df16b2bd6f0926beecfb96f601ba2","scripts":{"build":"tsc -p tsconfig.json","docexplorer":"bun src/docexplorer.ts","prepublishOnly":"bun run build"},"_npmUser":{"name":"derped","email":"derpedyea@pm.me"},"repository":{"url":"","type":"git"},"_npmVersion":"10.9.2","description":"CLI tool for crawling external documentation sites, converting pages to Markdown via OpenRouter, and materializing the docs into each of your local projects","directories":{},"_nodeVersion":"23.10.0","dependencies":{"ai":"*","ora":"^9.0.0","zod":"*","chalk":"5","cheerio":"*","cli-table3":"^0.6.5","lucide-react":"^0.554.0","@inquirer/prompts":"^8.0.1","@openrouter/ai-sdk-provider":"*"},"_hasShrinkwrap":false,"devDependencies":{"typescript":"*","@types/node":"*"},"_npmOperationalInternal":{"tmp":"tmp/docexplorer_0.1.0_1763495766066_0.8833054213709903","host":"s3://npm-registry-packages-npm-production"}},"0.1.1":{"name":"@derped/docexplorer","version":"0.1.1","description":"CLI tool for crawling external documentation sites, converting pages to Markdown via OpenRouter, and materializing the docs into each of your local projects","type":"module","main":"dist/docexplorer.js","bin":{"docexplorer":"dist/docexplorer.js"},"scripts":{"docexplorer":"bun src/docexplorer.ts","build":"tsc -p tsconfig.json","prepublishOnly":"bun run build"},"keywords":["documentation","crawler","cli","markdown","openrouter","docs","scraper"],"author":"","license":"MIT","repository":{"type":"git","url":""},"engines":{"node":">=18.0.0"},"dependencies":{"@inquirer/prompts":"^8.0.1","@openrouter/ai-sdk-provider":"*","ai":"*","chalk":"5","cheerio":"*","cli-table3":"^0.6.5","lucide-react":"^0.554.0","ora":"^9.0.0","zod":"*"},"devDependencies":{"typescript":"*","@types/node":"*"},"_id":"@derped/docexplorer@0.1.1","gitHead":"31a48adf4a80a4a9fed4929ca93654e9b521f89f","_nodeVersion":"23.10.0","_npmVersion":"10.9.2","dist":{"integrity":"sha512-uobSMx8yOaGaPgKPHt68dqjzhTJ82NScg+mEnV3yq/9mt8KMpeENlvwDpZdB2nywM2qXvci8qS3xTc0ZN0Gupg==","shasum":"16d3b1e8fffb3d128293f7d48ee3f4a583545fe6","tarball":"https://registry.npmjs.org/@derped/docexplorer/-/docexplorer-0.1.1.tgz","fileCount":5,"unpackedSize":59377,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIFpqAXha4wCmqn06GQgBqW36kS1Sr5Yzn52sUpaprEmZAiEAjLg+neV5NUOjBeDN333RdWdkOKqCGCJQo8Ou3r0+gmE="}]},"_npmUser":{"name":"derped","email":"derpedyea@pm.me"},"directories":{},"maintainers":[{"name":"derped","email":"derpedyea@pm.me"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/docexplorer_0.1.1_1763497091414_0.8457111311403653"},"_hasShrinkwrap":false}},"time":{"created":"2025-11-18T19:56:05.967Z","modified":"2025-11-18T20:18:11.774Z","0.1.0":"2025-11-18T19:56:06.258Z","0.1.1":"2025-11-18T20:18:11.600Z"},"license":"MIT","keywords":["documentation","crawler","cli","markdown","openrouter","docs","scraper"],"repository":{"type":"git","url":""},"description":"CLI tool for crawling external documentation sites, converting pages to Markdown via OpenRouter, and materializing the docs into each of your local projects","maintainers":[{"name":"derped","email":"derpedyea@pm.me"}],"readme":"# docexplorer\n\nCLI tool for crawling external documentation sites, converting pages to Markdown via OpenRouter, and materializing the docs into each of your local projects. It supports a system-wide local cache and can sync with a remote backend to share docsets across your team.\n\n## Installation\n\n### Prerequisites\n\n- Node.js / Bun installed\n- An OpenRouter API key\n\n### Install from npm (Recommended)\n\nInstall globally using npm:\n\n```bash\nnpm install -g @derped/docexplorer\n```\n\nThis makes a global `docexplorer` command available on your PATH. You can then run `docexplorer ...` from any project directory.\n\n### Development Installation\n\nIf you want to build from source or contribute:\n\n#### Install dependencies\n\n```bash\nbun install\n```\n\n#### Build the CLI\n\n```bash\nbun run build\n```\n\nThis compiles TypeScript into `dist/docexplorer.js`, which is what the CLI entry point uses.\n\n#### Optional: Install globally (development)\n\nFrom the project root:\n\n```bash\nnpm link\n```\n\nThis makes a global `docexplorer` command available on your PATH. You can then run `docexplorer ...` from any project directory.\n\nIf you do not want to link globally, you can always run via Bun directly from this repo:\n\n```bash\nbun run docexplorer -- <command> [...args]\n```\n\n## Quick Start (Interactive Mode)\n\nSimply run the tool without arguments to enter the interactive menu:\n\n```bash\nbun run docexplorer\n```\n\nThis will guide you through indexing, listing, pulling, pushing, and configuring the tool.\n\n## Configuration\n\n### OpenRouter API Key\n\n`docexplorer` needs an OpenRouter API key to call the model that converts HTML into Markdown and infers docs path prefixes.\n\n1. **Environment variable (takes precedence)**\n   - Set `OPENROUTER_API_KEY` in your shell.\n\n2. **Stored user config (persistent)**\n   - Run `docexplorer config` (or `set-api-key`) to save it to `~/.mineperial/docsexplorer/config.json`.\n\n### Backend URL (Optional)\n\nTo share docsets with a team or across machines, you can configure a backend URL (pointing to a running `docgrabber-backend` instance).\n\n- **Environment variable**: `DOCEXPLORER_BACKEND_URL`\n- **Default**: `https://docexplorer.derped.dev`\n\n### Other Environment Variables\n\n- `OPENROUTER_MODEL`\n  - Default: `openai/gpt-oss-safeguard-20b`\n  - Controls the model used for HTML→Markdown conversion.\n- `DOCEXPLORER_CONCURRENCY`\n  - Default: `5`, maximum `16`\n  - Controls parallel processing of pages.\n- `DOCEXPLORER_MAX_PAGES`\n  - Default: `20000`\n  - Limit on the number of pages to crawl per site.\n\n## Cache and Docs Layout\n\n### Global Cache\nCached docsets are stored in:\n- POSIX: `~/.mineperial/docsexplorer/cache`\n- Windows: `C:\\Users\\<you>\\.mineperial\\docsexplorer\\cache`\n\n### Project Docs\nWhen running `index` or `pull`, docs are written to:\n```text\n<your-project>/\n  .mineperial/\n    docs/\n      <docName>/\n        ... Markdown tree ...\n```\n\n## CLI Commands\n\n### `index`\nCrawl a site and create a local docset.\n\n```bash\ndocexplorer index <name> <url> [pathPrefix] [--force]\n```\n- `name`: Friendly name (e.g. `nextjs`).\n- `url`: Base URL (e.g. `https://nextjs.org/docs`).\n- `pathPrefix`: Optional path to restrict crawling (e.g. `/docs`).\n- `--force`: Re-crawl even if cached.\n\n### `list`\nList all docsets in your local global cache.\n\n```bash\ndocexplorer list\n```\n\n### `list-remote`\nList docsets available on the configured backend.\n\n```bash\ndocexplorer list-remote [query] [--limit <n>] [--offset <n>]\n```\n\n### `pull`\nInstall a docset into your current project. Checks local cache first, then tries to download from the backend.\n\n```bash\ndocexplorer pull <docId-or-name>\n```\n\n### `push`\nUpload all locally cached docsets to the configured backend.\n\n```bash\ndocexplorer push\n```\n\n### `config`\nOpen the interactive configuration menu to set API keys, defaults, and limits.\n\n```bash\ndocexplorer config\n```\n\n### `set-api-key`\nQuickly set the OpenRouter API key in the persistent config.\n\n```bash\ndocexplorer set-api-key <apiKey>\n```\n\n## Development\n\n- Source code: `src/`\n- Entry point: `src/docexplorer.ts`\n- Build: `bun run build`\n","readmeFilename":"README.md"}