{"_id":"@beeranked/ai-crawler-checker","_rev":"2-788ab94cab6fa308a5936e8626867688","name":"@beeranked/ai-crawler-checker","dist-tags":{"latest":"0.1.1"},"versions":{"0.1.0":{"name":"@beeranked/ai-crawler-checker","version":"0.1.0","keywords":["seo","robots.txt","ai","crawlers","gptbot","llms.txt","aeo","generative-engine-optimization"],"license":"MIT","_id":"@beeranked/ai-crawler-checker@0.1.0","maintainers":[{"name":"beerankedonline","email":"beeranked.backup@gmail.com"}],"homepage":"https://beeranked.online/ai-crawler-checker","bugs":{"url":"https://github.com/BeeRanked/ai-crawler-checker/issues"},"bin":{"ai-crawler-checker":"bin/cli.js"},"dist":{"shasum":"691c691568d9a8cc57d489d9f72bd1d1a05a185f","tarball":"https://registry.npmjs.org/@beeranked/ai-crawler-checker/-/ai-crawler-checker-0.1.0.tgz","fileCount":10,"integrity":"sha512-IivBmxSJA2bJNew7t1bg7kOQPCO7BPZpxVbAZoya0XZmylAZkeCNII7TlLzzfn3G4Yw5z6UXIx3c+o5TxnAgEQ==","signatures":[{"sig":"MEUCIQDm1834zDwVXRWOIdZtxOhtr54mRCsx8PXFMw4ksyrgIgIgBR+tJwuAx57vH3FQll/CwQSS9fhK6dB0nGCFa5t6hhI=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":28853},"main":"src/index.js","type":"module","engines":{"node":">=18"},"exports":{".":"./src/index.js","./robots":"./src/robots.js","./catalog":"./src/catalog.js"},"gitHead":"c44875702251b2b4bfce50231c780cd4cb8cff3d","scripts":{"test":"node --test"},"_npmUser":{"name":"beerankedonline","email":"beeranked.backup@gmail.com"},"repository":{"url":"git+https://github.com/BeeRanked/ai-crawler-checker.git","type":"git"},"_npmVersion":"9.2.0","description":"See which AI crawlers a site's robots.txt allows or blocks, split into crawlers that affect AI answers and crawlers that affect model training.","directories":{},"_nodeVersion":"20.19.2","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"_npmOperationalInternal":{"tmp":"tmp/ai-crawler-checker_0.1.0_1786496504880_0.4501636895126755","host":"s3://npm-registry-packages-npm-production"}},"0.1.1":{"name":"@beeranked/ai-crawler-checker","version":"0.1.1","description":"See which AI crawlers a site's robots.txt allows or blocks, split into crawlers that affect AI answers and crawlers that affect model training.","type":"module","main":"src/index.js","exports":{".":"./src/index.js","./catalog":"./src/catalog.js","./robots":"./src/robots.js"},"bin":{"ai-crawler-checker":"bin/cli.js"},"scripts":{"test":"node --test"},"keywords":["seo","robots.txt","ai","crawlers","gptbot","llms.txt","aeo","generative-engine-optimization"],"license":"MIT","engines":{"node":">=18"},"repository":{"type":"git","url":"git+https://github.com/BeeRanked/ai-crawler-checker.git"},"homepage":"https://beeranked.online/ai-crawler-checker","publishConfig":{"access":"public"},"gitHead":"708bbc539a95eee24dd650bd3054de52065a70e0","bugs":{"url":"https://github.com/BeeRanked/ai-crawler-checker/issues"},"_id":"@beeranked/ai-crawler-checker@0.1.1","_nodeVersion":"20.19.2","_npmVersion":"9.2.0","dist":{"integrity":"sha512-Nzqb3o85KmVpm1JcyfYIZPfcfCwg5LcAIMVerFYhKmE1yrwkQMs7EhzRyzHprwB/9hUz7hZeJHnipfpwndWcIA==","shasum":"e7113e18a9f39e754d5bcc3e73f176466cbda599","tarball":"https://registry.npmjs.org/@beeranked/ai-crawler-checker/-/ai-crawler-checker-0.1.1.tgz","fileCount":10,"unpackedSize":28853,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEYCIQCxajMJMZ+m0T/3XIx8SOORdrh+Ccch7pqni7ZhHFDh3QIhAJjfQtegaWKrLleuNLwNOuOaQ1t4Piz88kdGkXr1RcXg"}]},"_npmUser":{"name":"beerankedonline","email":"contact@beeranked.online"},"directories":{},"maintainers":[{"name":"beerankedonline","email":"contact@beeranked.online"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/ai-crawler-checker_0.1.1_1786501524894_0.7018661936113868"},"_hasShrinkwrap":false}},"time":{"created":"2026-08-12T01:01:44.726Z","modified":"2026-08-12T02:25:25.232Z","0.1.0":"2026-08-12T01:01:45.050Z","0.1.1":"2026-08-12T02:25:25.032Z"},"bugs":{"url":"https://github.com/BeeRanked/ai-crawler-checker/issues"},"license":"MIT","homepage":"https://beeranked.online/ai-crawler-checker","keywords":["seo","robots.txt","ai","crawlers","gptbot","llms.txt","aeo","generative-engine-optimization"],"repository":{"type":"git","url":"git+https://github.com/BeeRanked/ai-crawler-checker.git"},"description":"See which AI crawlers a site's robots.txt allows or blocks, split into crawlers that affect AI answers and crawlers that affect model training.","maintainers":[{"name":"beerankedonline","email":"contact@beeranked.online"}],"readme":"# ai-crawler-checker\n\nSee which AI crawlers a website's `robots.txt` allows or blocks, and understand\nthe difference that actually matters: crawlers that decide whether you can be\n**cited in AI answers** (ChatGPT search, Claude, Perplexity, Gemini grounding,\nSiri) versus crawlers that **train models** on your content. Blocking the wrong\ngroup quietly removes you from AI results while doing nothing you intended.\n\nNo browser, no API key, two small HTTP requests. Runs as a library, a CLI, or a\nself-hosted HTTP endpoint.\n\n## Why the two groups are separate\n\nA lot of sites paste a big \"block the AI bots\" list into `robots.txt` to stay\nout of model training, and accidentally block the crawlers that put them in AI\nanswers. Those are different bots from the same companies. This tool reads the\n`robots.txt`, checks every documented crawler against it (following RFC 9309),\nand tells you which group each verdict falls in, with a plain-language summary\nof what it costs you.\n\nEvery crawler in the catalog is documented by the operator that runs it, and\neach entry links to that operator's own page as its source.\n\n## Install\n\n```bash\nnpm install @beeranked/ai-crawler-checker\n# or run it once without installing:\nnpx @beeranked/ai-crawler-checker example.com\n```\n\nRequires Node 18 or newer (uses the built-in `fetch`).\n\n## CLI\n\n```bash\nai-crawler-checker example.com\nai-crawler-checker https://example.com --json\n```\n\n## Library\n\n```js\nimport { checkCrawlers } from '@beeranked/ai-crawler-checker';\n\nconst report = await checkCrawlers('example.com');\nconsole.log(report.summary);   // { answerVisible, answerTotal, trainingAllowed, trainingTotal }\nconsole.log(report.findings);  // [{ level: 'bad' | 'warn' | 'good' | 'info', text }]\nconsole.log(report.bots);      // per-crawler verdicts with sources\n```\n\nOn Node versions without a global `fetch`, pass one:\n\n```js\nimport { checkCrawlers } from '@beeranked/ai-crawler-checker';\nimport { fetch } from 'undici';\nawait checkCrawlers('example.com', { fetch });\n```\n\nThe robots.txt primitives are exported too, if you only want the parser:\n\n```js\nimport { parseRobots, groupFor, verdict } from '@beeranked/ai-crawler-checker/robots';\n```\n\n## Self-host as an HTTP endpoint\n\n`worker.js` is a ready Cloudflare Worker:\n\n```bash\nnpm install\ncp wrangler.toml.example wrangler.toml\nnpx wrangler deploy\n```\n\nCurrent wrangler (v4) needs Node 22 or newer; on Node 18 or 20, deploy with\n`npx wrangler@3 deploy` instead.\n\nThen `POST /` with `{ \"url\": \"https://example.com\" }`, or `GET /?url=...`, and\nyou get the full JSON report. It runs comfortably on the Cloudflare Workers free\ntier.\n\n## The crawler catalog\n\nThe catalog (`src/catalog.js`) is meant to stay current as operators publish or\nrename agents. A pull request that adds or updates a crawler should link to the\noperator's own documentation as the source; entries that cannot be sourced are\nnot accepted, because the whole point is that the table is true.\n\n## Hosted version\n\nPrefer to paste a URL and get a visual report? There is a free hosted version at\n[beeranked.online/ai-crawler-checker](https://beeranked.online/ai-crawler-checker),\nfrom the team that maintains this project.\n\n## License\n\nMIT. See [LICENSE](LICENSE).\n","readmeFilename":"README.md"}