{"_id":"@datafast/ai-crawl","_rev":"9-f56fe295c2af8f550733cdd4042102ac","name":"@datafast/ai-crawl","dist-tags":{"latest":"1.0.9"},"versions":{"1.0.1":{"name":"@datafast/ai-crawl","version":"1.0.1","keywords":["datafast","ai","crawler","llm","analytics","middleware"],"author":"","license":"MIT","_id":"@datafast/ai-crawl@1.0.1","maintainers":[{"name":"mcnaveen","email":"funmemcn@gmail.com"},{"name":"marclou94","email":"marc.louvion@gmail.com"}],"dist":{"shasum":"d7258553e109a20277f72c09c402564606329f91","tarball":"https://registry.npmjs.org/@datafast/ai-crawl/-/ai-crawl-1.0.1.tgz","fileCount":6,"integrity":"sha512-y6u/kELdMMg2neCbSvu6vcrz5HY1Nn7WXucENWA2rDJbTVQvjKWKn9VqvFuY+r/4HzSsoVYZBanwud+TlWkq7Q==","signatures":[{"sig":"MEYCIQDRzPicPH6bSkDq3mGyjguAyi5vD2yP3qEsUekRRDnaywIhAP9XgJhexkgZN2wsaklx7dXv81bNvwtTb3R4PECaq2lI","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":52380},"main":"dist/index.js","types":"dist/index.d.ts","module":"dist/index.mjs","gitHead":"62cd9a5ee4884d217806bdc100899245705a1ba5","scripts":{"dev":"tsup src/index.ts --format cjs,esm --dts --watch","build":"tsup src/index.ts --format cjs,esm --dts --clean","deploy":"npm run build && npm version patch --no-git-tag-version && npm publish","typecheck":"tsc --noEmit"},"_npmUser":{"name":"marclou94","email":"marc.louvion@gmail.com"},"_npmVersion":"11.4.2","description":"Server-side AI crawler tracking for DataFast","directories":{},"_nodeVersion":"24.3.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"tsup":"^8.0.0","typescript":"^5.3.0"},"_npmOperationalInternal":{"tmp":"tmp/ai-crawl_1.0.1_1780997802538_0.26600708377981075","host":"s3://npm-registry-packages-npm-production"}},"1.0.2":{"name":"@datafast/ai-crawl","version":"1.0.2","keywords":["datafast","ai","crawler","llm","analytics","middleware"],"author":"","license":"MIT","_id":"@datafast/ai-crawl@1.0.2","maintainers":[{"name":"mcnaveen","email":"funmemcn@gmail.com"},{"name":"marclou94","email":"marc.louvion@gmail.com"}],"dist":{"shasum":"468e732cfb311787500875aa5f823030d9eb6cc2","tarball":"https://registry.npmjs.org/@datafast/ai-crawl/-/ai-crawl-1.0.2.tgz","fileCount":6,"integrity":"sha512-qLuLHeYSpAd23aopzbQwno5gP9SewJ+wp1+sGQxt4NfSTBAGB1il1cw9OSmgQJHmfSLICtFijO7bVl1NeXRkQg==","signatures":[{"sig":"MEQCIApOWBwgsWTqbfRb1uT/kiFRGKSgYoauf1dKKCrVYwPVAiBreG+cD9opn2pWhK6OwHA4iScSx/LKcH1zTyIH6LDXcA==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":55775},"main":"dist/index.js","types":"dist/index.d.ts","module":"dist/index.mjs","gitHead":"387ab84fcadbf112b446567dd80e7284eb476d1b","scripts":{"dev":"tsup src/index.ts --format cjs,esm --dts --watch","build":"tsup src/index.ts --format cjs,esm --dts --clean","deploy":"npm run build && npm version patch --no-git-tag-version && npm publish","typecheck":"tsc --noEmit"},"_npmUser":{"name":"marclou94","email":"marc.louvion@gmail.com"},"_npmVersion":"11.4.2","description":"Server-side AI crawler tracking for DataFast","directories":{},"_nodeVersion":"24.3.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"tsup":"^8.0.0","typescript":"^5.3.0"},"_npmOperationalInternal":{"tmp":"tmp/ai-crawl_1.0.2_1780997947644_0.162776966989725","host":"s3://npm-registry-packages-npm-production"}},"1.0.3":{"name":"@datafast/ai-crawl","version":"1.0.3","keywords":["datafast","ai","crawler","llm","analytics","middleware"],"author":"","license":"MIT","_id":"@datafast/ai-crawl@1.0.3","maintainers":[{"name":"mcnaveen","email":"funmemcn@gmail.com"},{"name":"marclou94","email":"marc.louvion@gmail.com"}],"dist":{"shasum":"5bcb11f7bcbe4979e60cb8fd2e29765975a1cdcb","tarball":"https://registry.npmjs.org/@datafast/ai-crawl/-/ai-crawl-1.0.3.tgz","fileCount":6,"integrity":"sha512-pf0fmrAzvzH/0CUZwwmrWnG0hLXsEvi9TH46o2wBi2avs4lks+khnK19I+2GtgdpuDwYgfIUOXyUgmi5Av4SWw==","signatures":[{"sig":"MEQCIHtd+An/NLqteXc/VW8hROW3FY9QRNB5JSAO+FhkyEpjAiAOBfVc3vqbg3Bx2YzCf3F1RRa2bN+3sXq/oNQbH4ljVw==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":67350},"main":"dist/index.js","types":"dist/index.d.ts","module":"dist/index.mjs","gitHead":"d1b2a546346f5fc8f60078afe015fbc383866abd","scripts":{"dev":"tsup src/index.ts --format cjs,esm --dts --watch","build":"tsup src/index.ts --format cjs,esm --dts --clean","deploy":"npm run build && npm version patch --no-git-tag-version && npm publish","typecheck":"tsc --noEmit"},"_npmUser":{"name":"marclou94","email":"marc.louvion@gmail.com"},"_npmVersion":"11.4.2","description":"Server-side AI crawler tracking for DataFast","directories":{},"_nodeVersion":"24.3.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"tsup":"^8.0.0","typescript":"^5.3.0"},"_npmOperationalInternal":{"tmp":"tmp/ai-crawl_1.0.3_1781508936968_0.880967933601781","host":"s3://npm-registry-packages-npm-production"}},"1.0.4":{"name":"@datafast/ai-crawl","version":"1.0.4","keywords":["datafast","ai","crawler","llm","analytics","middleware"],"author":"","license":"MIT","_id":"@datafast/ai-crawl@1.0.4","maintainers":[{"name":"mcnaveen","email":"funmemcn@gmail.com"},{"name":"marclou94","email":"marc.louvion@gmail.com"}],"dist":{"shasum":"5a758fe9a9b1703766d0a9937031a859f4dc6c6d","tarball":"https://registry.npmjs.org/@datafast/ai-crawl/-/ai-crawl-1.0.4.tgz","fileCount":6,"integrity":"sha512-+cC9tLL/FUEpEIcO0u+1r7ycDSC73Kpq02ypz23uuFziuFbvnpkYBhEWFzw2iYdO3TUc+q/+GrdBh3zpaEKp5A==","signatures":[{"sig":"MEUCIQCt6rF8H80XSNp2MNn36AaG0+D0uNNnmZwTFU1dZI4wEgIgeVH3KJNE2H8NBJqy3xGGL1I3PDgnPKHxhOGVpjCLMVs=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":67235},"main":"dist/index.js","types":"dist/index.d.ts","module":"dist/index.mjs","gitHead":"8d38c420240d944c3d8cce22b0f6bb7274dab7cf","scripts":{"dev":"tsup src/index.ts --format cjs,esm --dts --watch","build":"tsup src/index.ts --format cjs,esm --dts --clean","deploy":"npm run build && npm version patch --no-git-tag-version && npm publish","typecheck":"tsc --noEmit"},"_npmUser":{"name":"marclou94","email":"marc.louvion@gmail.com"},"_npmVersion":"11.4.2","description":"Server-side AI crawler tracking for DataFast","directories":{},"_nodeVersion":"24.3.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"tsup":"^8.0.0","typescript":"^5.3.0"},"_npmOperationalInternal":{"tmp":"tmp/ai-crawl_1.0.4_1781765165176_0.9680565104145988","host":"s3://npm-registry-packages-npm-production"}},"1.0.5":{"name":"@datafast/ai-crawl","version":"1.0.5","keywords":["datafast","ai","crawler","llm","analytics","middleware"],"author":"","license":"MIT","_id":"@datafast/ai-crawl@1.0.5","maintainers":[{"name":"mcnaveen","email":"funmemcn@gmail.com"},{"name":"marclou94","email":"marc.louvion@gmail.com"}],"dist":{"shasum":"62011aa9acd01a5caab22fd31e7da5e193465c89","tarball":"https://registry.npmjs.org/@datafast/ai-crawl/-/ai-crawl-1.0.5.tgz","fileCount":6,"integrity":"sha512-y0rieHoij585C8raLYJ4eB8DBzj36nBCT3o28BBbbTy8kEjA3JzwmndKbh4mZlyjY4IKSKBzR3RLfWSZT+Gdew==","signatures":[{"sig":"MEYCIQD4+hWX4LXf4XwQB1yUgHisXTyR9Rc94l/nZEKSamGYqgIhAMeQLDG4gTQWhji9To3whzHunZdUONxXevemcl09lYme","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":67235},"main":"dist/index.js","types":"dist/index.d.ts","module":"dist/index.mjs","gitHead":"8d38c420240d944c3d8cce22b0f6bb7274dab7cf","scripts":{"dev":"tsup src/index.ts --format cjs,esm --dts --watch","build":"tsup src/index.ts --format cjs,esm --dts --clean","deploy":"npm run build && npm version patch --no-git-tag-version && npm publish","typecheck":"tsc --noEmit"},"_npmUser":{"name":"marclou94","email":"marc.louvion@gmail.com"},"_npmVersion":"11.4.2","description":"Server-side AI crawler tracking for DataFast","directories":{},"_nodeVersion":"24.3.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"tsup":"^8.0.0","typescript":"^5.3.0"},"_npmOperationalInternal":{"tmp":"tmp/ai-crawl_1.0.5_1781765854860_0.6849377207467986","host":"s3://npm-registry-packages-npm-production"}},"1.0.6":{"name":"@datafast/ai-crawl","version":"1.0.6","keywords":["datafast","ai","crawler","llm","analytics","middleware"],"author":"","license":"MIT","_id":"@datafast/ai-crawl@1.0.6","maintainers":[{"name":"mcnaveen","email":"funmemcn@gmail.com"},{"name":"marclou94","email":"marc.louvion@gmail.com"}],"dist":{"shasum":"53bbd8030701d118fe5a8f155942fb881879fbdf","tarball":"https://registry.npmjs.org/@datafast/ai-crawl/-/ai-crawl-1.0.6.tgz","fileCount":6,"integrity":"sha512-xGsZyPff/ikd7kYFZ4wM7gLnrnIXfSUZtJQbBCVcCoMn6M+V8xFnD0BbUtiAWXhKHW22ivADefYQSzrCjGhatQ==","signatures":[{"sig":"MEUCIFlXfcFBQW8psjH/4I8dbOPz+iXqbOlFMmTteurQlmNiAiEAxWiiP10znkm4forYR5WEtJqNcgfaOTtJa8EOAkzx8C8=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":69228},"main":"dist/index.js","types":"dist/index.d.ts","module":"dist/index.mjs","gitHead":"505c07df63b600867563f0aa8ee69e35cd933638","scripts":{"dev":"tsup src/index.ts --format cjs,esm --dts --watch","build":"tsup src/index.ts --format cjs,esm --dts --clean","deploy":"npm run build && npm version patch --no-git-tag-version && npm publish","typecheck":"tsc --noEmit"},"_npmUser":{"name":"marclou94","email":"marc.louvion@gmail.com"},"_npmVersion":"11.4.2","description":"Server-side AI crawler tracking for DataFast","directories":{},"_nodeVersion":"24.3.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"tsup":"^8.0.0","typescript":"^5.3.0"},"_npmOperationalInternal":{"tmp":"tmp/ai-crawl_1.0.6_1782990040154_0.6379968639910141","host":"s3://npm-registry-packages-npm-production"}},"1.0.7":{"name":"@datafast/ai-crawl","version":"1.0.7","keywords":["datafast","ai","crawler","llm","analytics","middleware"],"author":"","license":"MIT","_id":"@datafast/ai-crawl@1.0.7","maintainers":[{"name":"mcnaveen","email":"funmemcn@gmail.com"},{"name":"marclou94","email":"marc.louvion@gmail.com"}],"dist":{"shasum":"6b6537fa99e391a46124adab7e93e541801c4196","tarball":"https://registry.npmjs.org/@datafast/ai-crawl/-/ai-crawl-1.0.7.tgz","fileCount":6,"integrity":"sha512-lsW0UsJgiasTmqofyROBJEa0UiXHIRt5jsjugRNdYnbHDEJ7t2MI9uCvMrPFTR93U1MMuGuxQbMXZSnu/yTE4A==","signatures":[{"sig":"MEYCIQDpAE5vGLor4mtfRwvRNFJfBi0wmAbZlZRKkZpNS5gRMgIhAJMmLmH9+jDaE68lJPXsHezFLrEQnQlvNvSj2b3xBnsG","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":69231},"main":"dist/index.js","types":"dist/index.d.ts","module":"dist/index.mjs","gitHead":"b0df6fdf5d132c3db2c7908e7c14858d68ddcb68","scripts":{"dev":"tsup src/index.ts --format cjs,esm --dts --watch","build":"tsup src/index.ts --format cjs,esm --dts --clean","deploy":"npm run build && npm version patch --no-git-tag-version && npm publish","typecheck":"tsc --noEmit"},"_npmUser":{"name":"marclou94","email":"marc.louvion@gmail.com"},"_npmVersion":"11.4.2","description":"Server-side AI crawler tracking for DataFast","directories":{},"_nodeVersion":"24.3.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"tsup":"^8.0.0","typescript":"^5.3.0"},"_npmOperationalInternal":{"tmp":"tmp/ai-crawl_1.0.7_1783316204244_0.653518133981573","host":"s3://npm-registry-packages-npm-production"}},"1.0.8":{"name":"@datafast/ai-crawl","version":"1.0.8","keywords":["datafast","ai","crawler","llm","analytics","middleware"],"author":"","license":"MIT","_id":"@datafast/ai-crawl@1.0.8","maintainers":[{"name":"mcnaveen","email":"funmemcn@gmail.com"},{"name":"marclou94","email":"marc.louvion@gmail.com"}],"dist":{"shasum":"7638413d8bbbd5646dac63dadc4f706dd6e69552","tarball":"https://registry.npmjs.org/@datafast/ai-crawl/-/ai-crawl-1.0.8.tgz","fileCount":6,"integrity":"sha512-sfH36660dKUzdlWbzQcI47O8uOz2gy/FGnJXfmwV1rFJfkwukdRz/ofDlwBSWr+EP4MV7udLkjTVm8xu2SmnwQ==","signatures":[{"sig":"MEUCIEoHE+iiFrzwLMZDfECsHRlYPJh34yT9J4Qz/u1glO/yAiEAu7xwSIDMXmYp0EB5KsD/LN5C5RqzWr1w22NrzMUY+fw=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":71782},"main":"dist/index.js","types":"dist/index.d.ts","module":"dist/index.mjs","gitHead":"421d05466ee5fa1353aee62201dec555d78c3782","scripts":{"dev":"tsup src/index.ts --format cjs,esm --dts --watch","test":"npm run build && node --test test/*.test.mjs","build":"tsup src/index.ts --format cjs,esm --dts --clean","deploy":"npm run build && npm version patch --no-git-tag-version && npm publish","typecheck":"tsc --noEmit"},"_npmUser":{"name":"marclou94","email":"marc.louvion@gmail.com"},"_npmVersion":"11.4.2","description":"Server-side AI crawler tracking for DataFast","directories":{},"_nodeVersion":"24.3.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"tsup":"^8.0.0","typescript":"^5.3.0"},"_npmOperationalInternal":{"tmp":"tmp/ai-crawl_1.0.8_1784070759529_0.0883817431391245","host":"s3://npm-registry-packages-npm-production"}},"1.0.9":{"name":"@datafast/ai-crawl","version":"1.0.9","description":"Server-side AI crawler tracking for DataFast","main":"dist/index.js","module":"dist/index.mjs","types":"dist/index.d.ts","scripts":{"build":"tsup src/index.ts --format cjs,esm --dts --clean","dev":"tsup src/index.ts --format cjs,esm --dts --watch","test":"npm run build && node --test test/*.test.mjs","typecheck":"tsc --noEmit","deploy":"npm run build && npm version patch --no-git-tag-version && npm publish"},"keywords":["datafast","ai","crawler","llm","analytics","middleware"],"author":"","license":"MIT","devDependencies":{"tsup":"^8.0.0","typescript":"^5.3.0"},"publishConfig":{"access":"public"},"_id":"@datafast/ai-crawl@1.0.9","gitHead":"48a029da9f8780ffa1bccd2af69d1b7c6737a227","_nodeVersion":"24.3.0","_npmVersion":"11.4.2","dist":{"integrity":"sha512-S+EUH3l2jpJMXdaxTEtL84PTv1WiOsA61tSnI4Y8QT+NWDUHVFpNZtDiLiO7Nw+GtuK71XB5PzhxHgoDGI66hA==","shasum":"63428dc38ebbe48188bd4970f7adf35281633e1d","tarball":"https://registry.npmjs.org/@datafast/ai-crawl/-/ai-crawl-1.0.9.tgz","fileCount":6,"unpackedSize":72759,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEQCIG/mlp3QhqiaGmtsonXrN6PoFwGvy20ZTIU9l9lgCGaiAiAHJKJTa/5nDrzkOItUM2I2pkH4MOx1jHVRJ0SDh/2cjQ=="}]},"_npmUser":{"name":"marclou94","email":"marc.louvion@gmail.com"},"directories":{},"maintainers":[{"name":"mcnaveen","email":"funmemcn@gmail.com"},{"name":"marclou94","email":"marc.louvion@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/ai-crawl_1.0.9_1784098614088_0.8369781870729978"},"_hasShrinkwrap":false}},"time":{"created":"2026-06-09T09:36:42.313Z","modified":"2026-07-15T06:56:54.352Z","1.0.1":"2026-06-09T09:36:42.683Z","1.0.2":"2026-06-09T09:39:07.783Z","1.0.3":"2026-06-15T07:35:37.169Z","1.0.4":"2026-06-18T06:46:05.302Z","1.0.5":"2026-06-18T06:57:35.000Z","1.0.6":"2026-07-02T11:00:40.296Z","1.0.7":"2026-07-06T05:36:44.378Z","1.0.8":"2026-07-14T23:12:39.676Z","1.0.9":"2026-07-15T06:56:54.234Z"},"license":"MIT","keywords":["datafast","ai","crawler","llm","analytics","middleware"],"description":"Server-side AI crawler tracking for DataFast","maintainers":[{"name":"mcnaveen","email":"funmemcn@gmail.com"},{"name":"marclou94","email":"marc.louvion@gmail.com"}],"readme":"[![DataFast logo](https://datafa.st/logoAndName-small.png)](https://datafa.st)\n\n# @datafast/ai-crawl\n\nServer-side AI crawler tracking for DataFast.\n\nUse this package to see when known AI assistants, AI search systems, or training crawlers request pages on your website. It is not a replacement for DataFast web analytics and does not track human visitors, sessions, conversions, attribution, or browser pageviews.\n\nFull setup guide: [DataFast bot traffic tracking docs](https://datafa.st/docs/bot-traffic-tracking)\n\nCrawler directory: [See all crawlers tracked by DataFast](https://datafa.st/crawlers)\n\n## Installation\n\n```bash\nnpm install @datafast/ai-crawl\n```\n\n## How It Works\n\nThe package runs on your server, middleware, edge function, or worker. For each incoming request, it checks the user agent and path locally. If the request is not a likely AI/search crawler document request, nothing is sent to DataFast.\n\nWhen a likely crawler requests a document page, the package sends a small event to DataFast's AI crawl endpoint. In runtimes that support background tasks, such as Vercel and Cloudflare, it uses `waitUntil` so your page response does not wait for DataFast.\n\nCrawler-facing discovery files are tracked when requested by known crawlers. This includes `robots.txt`, `llms.txt`, `llms-full.txt`, and sitemap XML files.\n\nThe package is only a cost and safety pre-filter. DataFast's servers are the source of truth for provider classification, category, IP verification, confidence, and crawler list updates. This means DataFast can update crawler user agents and IP ranges without requiring you to upgrade this package for every crawler change.\n\nStatus code is optional. The most important signal is the page the AI tried to crawl. Status code is captured only when the backend exposes it without adding latency.\n\nImportant: when you pass a Vercel, Cloudflare, or Worker context to `trackAICrawlerRequest`, do not `await` it. The package schedules the tracking request with `waitUntil` and returns immediately.\n\n## Tutorials\n\n### Next.js / Vercel Proxy\n\nCreate or update `proxy.ts` (previously called `middleware.ts` / `middleware.js` in older Next.js versions):\n\n```ts\n// proxy.ts\nimport {\n  NextResponse,\n  type NextFetchEvent,\n  type NextRequest,\n} from \"next/server\";\nimport { trackAICrawlerRequest } from \"@datafast/ai-crawl\";\n\nexport function proxy(request: NextRequest, event: NextFetchEvent) {\n  trackAICrawlerRequest(request, event, {\n    websiteId: \"dfid_your_tracking_id\",\n  });\n\n  return NextResponse.next();\n}\n\nexport const config = {\n  matcher: [\"/((?!api|_next/static|_next/image|favicon.ico).*)\"],\n};\n```\n\nThis is non-blocking. The package schedules the DataFast request with `event.waitUntil`, then your app continues returning the page.\n\nNext.js Proxy runs before the final page response, so status code is usually unknown and will be stored as `0`.\n\n### Cloudflare Pages\n\nInstall the package, then create a Pages Functions middleware file:\n\n```ts\n// functions/_middleware.ts\nimport { trackAICrawlerRequest } from \"@datafast/ai-crawl\";\n\nexport async function onRequest(context) {\n  trackAICrawlerRequest(context.request, context, {\n    websiteId: \"dfid_your_tracking_id\",\n  });\n\n  return context.next();\n}\n```\n\nCloudflare Pages provides `context.waitUntil`, so the DataFast request runs in the background while `context.next()` continues to your page or static asset.\n\nDo not `await trackAICrawlerRequest(...)` here. Call it, then immediately return `context.next()`.\n\nFor better Cloudflare billing/performance hygiene, also exclude obvious static assets from Function invocation when your framework does not generate this automatically. Add `_routes.json` to your build output:\n\n```json\n{\n  \"version\": 1,\n  \"include\": [\"/*\"],\n  \"exclude\": [\n    \"/assets/*\",\n    \"/static/*\",\n    \"/_next/*\",\n    \"/favicon.ico\",\n    \"/*.css\",\n    \"/*.js\",\n    \"/*.png\",\n    \"/*.jpg\",\n    \"/*.svg\",\n    \"/*.ico\",\n    \"/*.woff2\"\n  ]\n}\n```\n\nThe package still filters static assets locally, but `_routes.json` prevents Cloudflare from invoking your Function for the most obvious asset requests. Do not exclude crawler-facing files such as `/robots.txt`, `/llms.txt`, `/llms-full.txt`, or `/sitemap.xml` if you want to see crawler activity on those files.\n\nCloudflare docs: [Pages middleware](https://developers.cloudflare.com/pages/functions/middleware/), [Pages Functions API](https://developers.cloudflare.com/pages/functions/api-reference/), [routing and `_routes.json`](https://developers.cloudflare.com/pages/functions/routing/).\n\n### Cloudflare Workers\n\nWrap your Worker `fetch` handler:\n\n```ts\nimport { withAICrawlerTracking } from \"@datafast/ai-crawl\";\n\nexport default {\n  fetch: withAICrawlerTracking(\n    async (request: Request, env: Env, ctx: ExecutionContext) => {\n      return fetch(request);\n    },\n    {\n      websiteId: \"dfid_your_tracking_id\",\n    },\n  ),\n};\n```\n\nThis captures status code because the wrapper sees the `Response` your handler already created. It still uses `ctx.waitUntil` when available, so the tracking request is best-effort and non-blocking.\n\n### Custom Docker, Cloud Run, or reverse proxies\n\nMost frameworks expose the public request URL automatically. If your runtime instead builds `request.url` with an internal hostname such as `localhost` or `0.0.0.0`, set `publicOrigin` once:\n\n```ts\ntrackAICrawlerRequest(request, context, {\n  websiteId: \"dfid_your_tracking_id\",\n  publicOrigin: \"https://example.com\",\n});\n```\n\nDataFast keeps the original path and query string, replaces only the internal origin, and still validates the resulting public hostname against your website configuration.\n\n### Express\n\n```ts\nimport express from \"express\";\nimport { createExpressAICrawlerMiddleware } from \"@datafast/ai-crawl\";\n\nconst app = express();\n\napp.use(\n  createExpressAICrawlerMiddleware({\n    websiteId: \"dfid_your_tracking_id\",\n  }),\n);\n```\n\nThe Express middleware attaches a `finish` listener and sends status code after the response has already been sent to the crawler. It calls `next()` immediately and does not wait for DataFast before your app continues.\n\n### Hono\n\n```ts\nimport { Hono } from \"hono\";\nimport { trackAICrawlerResponse } from \"@datafast/ai-crawl\";\n\nconst app = new Hono();\n\napp.use(\"*\", async (c, next) => {\n  await next();\n\n  trackAICrawlerResponse(c.req.raw, c.res, c.executionCtx, {\n    websiteId: \"dfid_your_tracking_id\",\n  });\n});\n```\n\nThis is useful on Cloudflare Workers or other Hono runtimes where you can access the final response and execution context.\n\n### Generic Request / Response Handler\n\nIf your backend gives you a standard `Request` and `Response`, use:\n\n```ts\nimport { trackAICrawlerResponse } from \"@datafast/ai-crawl\";\n\nexport async function handleRequest(request, context) {\n  const response = await yourAppHandler(request);\n\n  trackAICrawlerResponse(request, response, context, {\n    websiteId: \"dfid_your_tracking_id\",\n  });\n\n  return response;\n}\n```\n\nIf your backend only gives you the request before the app response exists, use request-only tracking:\n\n```ts\nimport { trackAICrawlerRequest } from \"@datafast/ai-crawl\";\n\nexport function middleware(request, context) {\n  trackAICrawlerRequest(request, context, {\n    websiteId: \"dfid_your_tracking_id\",\n  });\n\n  return next();\n}\n```\n\nRequest-only tracking is enough to know which page the AI tried to crawl. Response-aware tracking only adds status code when it is easy to get.\n\n## Optional Request Authentication\n\nCreate a Bot traffic token in your website's Bot traffic settings, then add it to the same server-side configuration:\n\n```ts\ntrackAICrawlerRequest(request, event, {\n  websiteId: \"dfid_your_tracking_id\",\n  authToken: process.env.DATAFAST_BOT_TOKEN,\n});\n```\n\nKeep the token on your server and out of browser code. Token validation only affects request acceptance after you enable **Reject unauthenticated requests** in Bot traffic settings. Rotating a token invalidates the previous token immediately, and deleting it automatically turns request authentication off.\n\n## Filtering\n\nBy default, the package tracks every relevant crawler category it recognizes, including AI answer fetchers, search/indexing crawlers, training crawlers, and other AI crawler traffic.\n\nYou can opt out of specific categories:\n\n```ts\ntrackAICrawlerRequest(request, event, {\n  websiteId: \"dfid_your_tracking_id\",\n\n  // Optional category opt-outs. All default to false.\n  disableAnswerFetch: true,\n  disableSearchCrawlers: true,\n  disableTrainingCrawlers: true,\n  disableOtherCrawlers: true,\n});\n```\n\nFor backward compatibility, `includeSearchCrawlers: false` also disables search/indexing crawlers. Prefer `disableSearchCrawlers: true` for new installs.\n\n## What Gets Tracked\n\nThe tracker only sends likely AI/search crawler document requests to DataFast. Normal human traffic, API routes, framework internals, and generic static assets such as CSS, JavaScript, images, fonts, maps, PDFs, and media files are ignored before any network request is made.\n\nCrawler-facing files are intentionally not treated as generic static assets. DataFast tracks known crawler requests to files such as `/robots.txt`, `/llms.txt`, `/llms-full.txt`, and sitemap XML files, because those files are often requested specifically by search bots, AI assistants, and AI infrastructure.\n\nIt stores provider, agent, category, page, hostname, status code when available, source, and confidence.\n\n```txt\nOpenAI / ChatGPT-User  -> answer_fetch\nOpenAI / OAI-SearchBot -> search_index\nOpenAI / GPTBot        -> training\nGooglebot              -> search_index\nGoogle-NotebookLM      -> answer_fetch\nGoogle-Extended        -> training\nGoogle-CloudVertexBot  -> training\nPerplexityBot          -> search_index\nClaudeBot              -> training\nBingbot                -> search_index\nApplebot               -> training\nBytespider             -> training\nCCBot                  -> training\nxAI-SearchBot          -> answer_fetch\nGrok-DeepSearch        -> answer_fetch\nGrokBot / xAI-Bot      -> ai_crawler\n```\n\n## License\n\nMIT\n","readmeFilename":"README.md"}