{"_id":"@casoon/astro-crawler-policy","_rev":"4-5ff411960cbf1632ecf6e81168dc39ce","name":"@casoon/astro-crawler-policy","dist-tags":{"latest":"0.1.2"},"versions":{"0.1.0":{"name":"@casoon/astro-crawler-policy","version":"0.1.0","keywords":["astro","astro-integration","astro-component","withastro","robots.txt","llms.txt","crawler","crawler-policy","ai-crawler","bot","seo","ai","sitemap"],"license":"MIT","_id":"@casoon/astro-crawler-policy@0.1.0","maintainers":[{"name":"jseidel","email":"joern.seidel@casoon.de"}],"homepage":"https://github.com/casoon/astro-crawler-policy#readme","bugs":{"url":"https://github.com/casoon/astro-crawler-policy/issues"},"dist":{"shasum":"b4a4dc37c922a6e9a5c4dbe07c5c3b284f60e90e","tarball":"https://registry.npmjs.org/@casoon/astro-crawler-policy/-/astro-crawler-policy-0.1.0.tgz","fileCount":23,"integrity":"sha512-hzhAvVOLeBYXveoMMpdnT8E/wVxYpgJPvC4tjv/q1HCjMjMr+ki23wW5oLDWAbmpEJvSlcfNjmJPfBCZEuzFEQ==","signatures":[{"sig":"MEUCIQCtvFDM1C2IOeM84yYC3enyiW7MXiWaVwig0mDaO+FMYAIgFmE5zEB+TFMw4KWULPzYv2Xyu0UROa51VArpKh+kalw=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":43667},"main":"./dist/index.js","type":"module","types":"./dist/index.d.ts","module":"./dist/index.js","engines":{"node":">=18.0.0"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js"}},"gitHead":"086a99f00347d46aa26727ead99df693524b1487","scripts":{"test":"vitest run","build":"tsc","prepublishOnly":"npm test && npm run build"},"_npmUser":{"name":"jseidel","email":"joern.seidel@casoon.de"},"repository":{"url":"git+https://github.com/casoon/astro-crawler-policy.git","type":"git"},"_npmVersion":"11.6.2","description":"Policy-first crawler control for Astro — generates robots.txt and llms.txt with presets, per-bot rules, AI crawler registry, and build-time audits.","directories":{},"_nodeVersion":"24.11.1","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"vitest":"^4.1.4","typescript":"^5.8.3","@types/node":"^25.5.2"},"peerDependencies":{"astro":">=4.0.0"},"peerDependenciesMeta":{"astro":{"optional":true}},"_npmOperationalInternal":{"tmp":"tmp/astro-crawler-policy_0.1.0_1775727830095_0.7624931369622066","host":"s3://npm-registry-packages-npm-production"},"deprecated":"Nicht mehr gepflegt – bitte @casoon/astro-site-files verwenden (erzeugt robots.txt)"},"0.1.1":{"name":"@casoon/astro-crawler-policy","version":"0.1.1","keywords":["astro","astro-integration","astro-component","withastro","robots.txt","llms.txt","crawler","crawler-policy","ai-crawler","bot","seo","ai","sitemap"],"license":"MIT","_id":"@casoon/astro-crawler-policy@0.1.1","maintainers":[{"name":"jseidel","email":"joern.seidel@casoon.de"}],"homepage":"https://github.com/casoon/astro-crawler-policy#readme","bugs":{"url":"https://github.com/casoon/astro-crawler-policy/issues"},"dist":{"shasum":"1bb44652516e33e84d1fdaa58933b2251b404690","tarball":"https://registry.npmjs.org/@casoon/astro-crawler-policy/-/astro-crawler-policy-0.1.1.tgz","fileCount":23,"integrity":"sha512-UvpD93C38MjnKxfxb7SZEUuWlL9/F6KPi7oh7wFfRyzhY2etD30L32Nc891wQG/INWm8rpYbjk0+Hi0hxXKvXg==","signatures":[{"sig":"MEUCIGPTTLPHtrT4iU4AFBrcIn3++MH03jGjFviZkh7f3pIAAiEApy67QNOJ0aZ7yrOyBsA4scygksSzSTzSJ1Vlukw7DxE=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":44416},"main":"./dist/index.js","type":"module","types":"./dist/index.d.ts","module":"./dist/index.js","engines":{"node":">=18.0.0"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js"}},"gitHead":"2e932f5a54940a81114862f5fce6ae377ac16908","scripts":{"test":"vitest run","build":"tsc","prepublishOnly":"npm test && npm run build"},"_npmUser":{"name":"jseidel","email":"joern.seidel@casoon.de"},"repository":{"url":"git+https://github.com/casoon/astro-crawler-policy.git","type":"git"},"_npmVersion":"11.6.2","description":"Policy-first crawler control for Astro — generates robots.txt and llms.txt with presets, per-bot rules, AI crawler registry, and build-time audits.","directories":{},"_nodeVersion":"24.11.1","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"vitest":"^4.1.4","typescript":"^5.8.3","@types/node":"^25.5.2"},"peerDependencies":{"astro":">=4.0.0"},"peerDependenciesMeta":{"astro":{"optional":true}},"_npmOperationalInternal":{"tmp":"tmp/astro-crawler-policy_0.1.1_1775730728179_0.6899916649716544","host":"s3://npm-registry-packages-npm-production"},"deprecated":"Nicht mehr gepflegt – bitte @casoon/astro-site-files verwenden (erzeugt robots.txt)"},"0.1.2":{"name":"@casoon/astro-crawler-policy","version":"0.1.2","keywords":["astro","astro-integration","astro-component","withastro","robots.txt","llms.txt","crawler","crawler-policy","ai-crawler","bot","seo","ai","sitemap"],"license":"MIT","_id":"@casoon/astro-crawler-policy@0.1.2","maintainers":[{"name":"jseidel","email":"joern.seidel@casoon.de"}],"homepage":"https://github.com/casoon/astro-crawler-policy#readme","bugs":{"url":"https://github.com/casoon/astro-crawler-policy/issues"},"dist":{"shasum":"70974062a6f9271c8ed43e7c494e262d76fd8570","tarball":"https://registry.npmjs.org/@casoon/astro-crawler-policy/-/astro-crawler-policy-0.1.2.tgz","fileCount":23,"integrity":"sha512-ceHvd8qOdGO+maMKF87n+HTN2xNDwN4BEUvphFo+mxw6oKgxEksa5EzJF5vpYRJkboMoxYm9mRwRiNJAhIOHig==","signatures":[{"sig":"MEUCIAtBojJhyWb5a988MboiMop3Y4p4RnWI/LfJq6lKWBIUAiEArUxyU6uOn3QerHoogHPSsIi/M+oAMEsGSVoGaqCO9Yg=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":45093},"main":"./dist/index.js","type":"module","types":"./dist/index.d.ts","module":"./dist/index.js","engines":{"node":">=18.0.0"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js"}},"gitHead":"8719933b83d6298b92971cf55e23a1658d9c01c4","scripts":{"test":"vitest run","build":"tsc","prepublishOnly":"npm test && npm run build"},"_npmUser":{"name":"jseidel","email":"joern.seidel@casoon.de"},"repository":{"url":"git+https://github.com/casoon/astro-crawler-policy.git","type":"git"},"_npmVersion":"11.6.2","description":"Policy-first crawler control for Astro — generates robots.txt and llms.txt with presets, per-bot rules, AI crawler registry, and build-time audits.","directories":{},"_nodeVersion":"24.11.1","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"vitest":"^4.1.4","typescript":"^5.8.3","@types/node":"^25.5.2"},"peerDependencies":{"astro":">=4.0.0"},"peerDependenciesMeta":{"astro":{"optional":true}},"_npmOperationalInternal":{"tmp":"tmp/astro-crawler-policy_0.1.2_1776093801261_0.7961046212733056","host":"s3://npm-registry-packages-npm-production"},"deprecated":"Nicht mehr gepflegt – bitte @casoon/astro-site-files verwenden (erzeugt robots.txt)"}},"time":{"created":"2026-04-09T09:43:50.025Z","modified":"2026-09-16T06:08:14.513Z","0.1.0":"2026-04-09T09:43:50.219Z","0.1.1":"2026-04-09T10:32:08.327Z","0.1.2":"2026-04-13T15:23:21.424Z"},"bugs":{"url":"https://github.com/casoon/astro-crawler-policy/issues"},"license":"MIT","homepage":"https://github.com/casoon/astro-crawler-policy#readme","keywords":["astro","astro-integration","astro-component","withastro","robots.txt","llms.txt","crawler","crawler-policy","ai-crawler","bot","seo","ai","sitemap"],"repository":{"url":"git+https://github.com/casoon/astro-crawler-policy.git","type":"git"},"description":"Policy-first crawler control for Astro — generates robots.txt and llms.txt with presets, per-bot rules, AI crawler registry, and build-time audits.","maintainers":[{"name":"jseidel","email":"joern.seidel@casoon.de"}],"readme":"# @casoon/astro-crawler-policy\n\nPolicy-first crawler control for Astro. Generates `robots.txt` (and optionally `llms.txt`) from a typed configuration at build time.\n\n## What it does\n\n- Generates `robots.txt` from a typed configuration — no manual file editing required\n- Applies one of five built-in **presets** covering the most common use cases\n- Supports **content signals** (`search`, `ai-input`, `ai-train`) for newer crawler directives\n- Includes a **bot registry** with 13 known crawlers for per-bot and group-based rules\n- **Merges** the generated output with an existing `public/robots.txt` (replace / prepend / append)\n- Runs **build-time audits** that warn about common misconfigurations\n- Optionally generates **`llms.txt`** — a markdown summary of the AI content policy\n- Supports **environment-specific overrides** (e.g. lockdown on staging)\n\nThis plugin renders crawler policy. It does not enforce blocking at the network, WAF, or edge layer.\n\n## Installation\n\n```sh\nnpm install @casoon/astro-crawler-policy\n```\n\n## Quick start\n\n```ts\n// astro.config.ts\nimport { defineConfig } from 'astro/config';\nimport crawlerPolicy from '@casoon/astro-crawler-policy';\n\nexport default defineConfig({\n  site: 'https://example.com',\n  integrations: [\n    crawlerPolicy({\n      preset: 'citationFriendly',\n      sitemaps: ['/sitemap-index.xml']\n    })\n  ]\n});\n```\n\nThe plugin hooks into `astro:build:done` and writes `dist/robots.txt`. With just these two options you get sensible defaults: search engines allowed, verified AI bots allowed for citation, AI training bots blocked.\n\n## Presets\n\nPresets are the primary way to express intent. Each preset sets default content signals and group-level rules.\n\n| Preset | Search | AI citation | AI training | Unknown AI |\n|---|---|---|---|---|\n| `seoOnly` | allow | disallow | disallow | disallow |\n| `citationFriendly` *(default)* | allow | allow | disallow | disallow |\n| `openToAi` | allow | allow | allow | allow |\n| `blockTraining` | allow | allow | disallow | disallow |\n| `lockdown` | disallow | disallow | disallow | disallow |\n\n`citationFriendly` allows bots that do citation or summarization but blocks bots whose only purpose is training data collection (GPTBot, Google-Extended, CCBot, Bytespider, Applebot-Extended). Bots with mixed roles like ClaudeBot are allowed.\n\n`blockTraining` goes further and blocks every bot with any training category, including mixed bots like ClaudeBot and meta-externalagent.\n\n`lockdown` adds a global `User-agent: * / Disallow: /` rule, overriding everything.\n\n## Content signals\n\nContent signals are non-standard directives appended to the wildcard `User-agent: *` block:\n\n```\nUser-agent: *\nContent-Signal: search=yes, ai-input=yes, ai-train=no\nAllow: /\n```\n\nThey communicate intent to crawlers that support them. The three signals map to:\n\n| Signal | Meaning |\n|---|---|\n| `search` | Indexing for traditional search engines |\n| `aiInput` | Using content as input for AI responses (citation, summarization) |\n| `aiTrain` | Using content as AI training data |\n\nThe directive name and signal keys follow the [contentsignals.org](https://contentsignals.org) specification (proposed IETF aipref standard). Google Search Console may flag them as unrecognised directives — the audit system emits an `info` message when they are present.\n\nEach preset sets default values for all three signals. You can override them individually:\n\n```ts\ncrawlerPolicy({\n  preset: 'citationFriendly',\n  contentSignals: {\n    aiTrain: true  // override just this one; search and aiInput come from the preset\n  }\n})\n```\n\n## Groups and per-bot rules\n\nRules are resolved in layers, from least to most specific:\n\n1. **Preset** — sets group-level defaults\n2. **`groups`** — overrides for entire bot categories\n3. **`bots`** — overrides for individual bots by registry ID\n\nA bot's final action is the most specific rule that applies to it. An explicit entry in `bots` always wins over a `groups` setting.\n\n```ts\ncrawlerPolicy({\n  preset: 'citationFriendly',\n\n  // Override an entire group\n  groups: {\n    searchEngines: 'allow',  // default\n    verifiedAi: 'allow',     // default\n    unknownAi: 'disallow'    // default\n  },\n\n  // Override individual bots (takes precedence over groups)\n  bots: {\n    GPTBot: 'disallow',   // blocks this bot even if verifiedAi is 'allow'\n    ClaudeBot: 'allow'    // allows this bot even if verifiedAi were 'disallow'\n  }\n})\n```\n\nThe three groups are:\n- **`searchEngines`** — bots with category `search` (Googlebot, Bingbot)\n- **`verifiedAi`** — verified bots with AI categories (`ai-search`, `ai-input`, `ai-training`)\n- **`unknownAi`** — unverified bots or bots with category `unknown-ai`\n\nWhen a bot's action resolves to `'inherit'` (no group or preset covers it), the bot is omitted from the output.\n\n## Custom rules\n\nFor anything not covered by the preset or registry, use `rules` to add raw robots.txt directives:\n\n```ts\ncrawlerPolicy({\n  rules: [\n    {\n      userAgent: '*',\n      disallow: ['/admin/', '/internal/'],\n      crawlDelay: 2\n    },\n    {\n      userAgent: 'Slurp',\n      disallow: ['/']\n    }\n  ]\n})\n```\n\nA `userAgent: '*'` rule in `rules` is merged with the wildcard block that the preset generates — it does not create a second `User-agent: *` section.\n\nAvailable fields per rule:\n\n| Field | Type | Description |\n|---|---|---|\n| `userAgent` | `string \\| string[]` | One or more User-agent values |\n| `allow` | `string[]` | Paths to allow |\n| `disallow` | `string[]` | Paths to disallow |\n| `crawlDelay` | `number` | Crawl-delay in seconds |\n| `comment` | `string` | Inline comment above the rule |\n\n## Merge strategy\n\nWhen a `public/robots.txt` already exists, the merge strategy controls how it is combined with the generated output.\n\n| Strategy | Result |\n|---|---|\n| `prepend` *(default)* | Generated output first, then existing file |\n| `append` | Existing file first, then generated output |\n| `replace` | Generated output only, existing file ignored |\n\n```ts\ncrawlerPolicy({\n  mergeStrategy: 'prepend'\n})\n```\n\nUse `prepend` to let the generated policy take precedence. Use `append` to keep hand-written rules at the top. Use `replace` when you want full control from config and no manual overrides.\n\n## Environment overrides\n\nThe plugin detects the current environment from these variables, in order:\n\n1. `CONTEXT` (Netlify)\n2. `DEPLOYMENT_ENVIRONMENT`\n3. `NODE_ENV`\n4. Falls back to `'production'`\n\nUse `env` to apply different settings per environment:\n\n```ts\ncrawlerPolicy({\n  preset: 'citationFriendly',\n  env: {\n    staging: { preset: 'lockdown' },\n    preview: { preset: 'lockdown' }\n  }\n})\n```\n\nAny option can be overridden per environment. Nested objects (`contentSignals`, `bots`, `groups`) are merged — not replaced — with the base config.\n\n## Output files\n\n```ts\ncrawlerPolicy({\n  output: {\n    robotsTxt: true,  // default — writes dist/robots.txt\n    llmsTxt: true     // opt-in — writes dist/llms.txt\n  }\n})\n```\n\n### llms.txt\n\nWhen `output.llmsTxt: true` is set, the plugin generates `dist/llms.txt` alongside `robots.txt`. The file is a Markdown summary of the AI content policy — which crawlers are allowed or blocked, what signals are active, and where the sitemap is:\n\n```md\n# example.com\n\n> AI content access policy for example.com.\n> Generated by @casoon/astro-crawler-policy (preset: citationFriendly).\n\n## Content Policy\n\n- Search indexing: allowed\n- AI citation and summarization: allowed\n- AI training data collection: not allowed\n\n## AI Systems\n\n### Allowed\n- OAI-SearchBot (OpenAI)\n- ClaudeBot (Anthropic)\n- claude-web (Anthropic)\n- PerplexityBot (Perplexity)\n- meta-externalagent (Meta)\n- Amazonbot (Amazon)\n- Googlebot (Google)\n- Bingbot (Microsoft)\n\n### Blocked\n- GPTBot (OpenAI)\n- Google-Extended (Google)\n- CCBot (Common Crawl)\n- Bytespider (ByteDance)\n- Applebot-Extended (Apple)\n\n## Sitemaps\n\n- https://example.com/sitemap-index.xml\n```\n\n## Debug mode\n\nSet `debug: true` to print the resolved configuration to the build log:\n\n```ts\ncrawlerPolicy({ debug: true })\n```\n\nBuild output:\n\n```\n[@casoon/astro-crawler-policy] [debug] registry version: 2026-04-09\n[@casoon/astro-crawler-policy] [debug] environment: production\n[@casoon/astro-crawler-policy] [debug] preset: citationFriendly\n[@casoon/astro-crawler-policy] [debug] content signals: search=yes, aiInput=yes, aiTrain=no\n[@casoon/astro-crawler-policy] [debug] bot: GPTBot → disallow\n[@casoon/astro-crawler-policy] [debug] bot: OAI-SearchBot → allow\n...\n[@casoon/astro-crawler-policy] [debug] sitemap: https://example.com/sitemap-index.xml\n```\n\n## Bot registry\n\nThe following bots are known and can be referenced by ID in `bots: {}`:\n\n| ID | Provider | Categories | Group |\n|---|---|---|---|\n| `GPTBot` | OpenAI | ai-training | verifiedAi |\n| `OAI-SearchBot` | OpenAI | ai-search, ai-input | verifiedAi |\n| `ClaudeBot` | Anthropic | ai-input, ai-training | verifiedAi |\n| `claude-web` | Anthropic | ai-input | verifiedAi |\n| `Google-Extended` | Google | ai-training | verifiedAi |\n| `CCBot` | Common Crawl | ai-training | verifiedAi |\n| `PerplexityBot` | Perplexity | ai-search, ai-input | verifiedAi |\n| `Bytespider` | ByteDance | ai-training | verifiedAi |\n| `meta-externalagent` | Meta | ai-input, ai-training | verifiedAi |\n| `Amazonbot` | Amazon | ai-search, ai-input | verifiedAi |\n| `Applebot-Extended` | Apple | ai-training | verifiedAi |\n| `Googlebot` | Google | search | searchEngines |\n| `Bingbot` | Microsoft | search | searchEngines |\n\n## Extending the registry\n\nThe built-in registry covers the most common crawlers. To support bots not yet listed, use `extraBots`:\n\n```ts\ncrawlerPolicy({\n  extraBots: [\n    {\n      id: 'MyCustomBot',\n      provider: 'Acme Corp',\n      userAgents: ['MyCustomBot/1.0'],\n      categories: ['ai-training'],\n      verified: true\n    }\n  ],\n  bots: {\n    MyCustomBot: 'disallow'\n  }\n})\n```\n\nExtra bots participate in group rules, per-bot overrides, audit checks, and `llms.txt` output — the same as built-in bots.\n\n**Keeping the registry up to date:** The registry ships as part of the package. As new crawlers emerge, updates are released as patch versions. Run `npm update @casoon/astro-crawler-policy` to get the latest bot data. The `REGISTRY_VERSION` export contains the date of the last registry update.\n\n## Audit warnings\n\nThe plugin emits warnings and info messages during the build:\n\n| Code | Level | Condition |\n|---|---|---|\n| `MISSING_SITE_URL` | warn | No `site` set in Astro config |\n| `NO_SITEMAP` | info | No sitemaps configured |\n| `DUPLICATE_USER_AGENT_RULE` | warn | Two rules share the same User-agent |\n| `UNLOCKED_NON_PRODUCTION_ENVIRONMENT` | warn | Staging/preview not globally blocked |\n| `NON_STANDARD_DIRECTIVES` | info | Content signals may trigger GSC syntax warnings |\n| `AI_INPUT_WITHOUT_ALLOWED_BOTS` | warn | `aiInput` enabled but all AI bots blocked |\n| `UNKNOWN_BOT_ID` | warn | A bot ID in `bots: {}` is not in the registry |\n| `GROUP_BOT_OVERRIDE_CONFLICT` | info | Bot override contradicts its group rule |\n\nAudit settings:\n\n```ts\ncrawlerPolicy({\n  audit: {\n    warnOnMissingSitemap: true,  // default\n    warnOnConflicts: true        // default\n  }\n})\n```\n\n## Programmatic usage\n\nThe core modules are exported for use outside of the Astro integration:\n\n```ts\nimport {\n  compilePolicy,\n  renderRobotsTxt,\n  renderLlmsTxt,\n  auditPolicy,\n  defaultRegistry,\n  REGISTRY_VERSION\n} from '@casoon/astro-crawler-policy';\n\nconst policy = compilePolicy({\n  options: { preset: 'citationFriendly', sitemaps: ['/sitemap-index.xml'] },\n  site: 'https://example.com',\n  environment: 'production'\n});\n\nconst robotsTxt = renderRobotsTxt(policy);\nconst llmsTxt = renderLlmsTxt(policy, 'https://example.com');\nconst issues = auditPolicy(policy, { site: 'https://example.com', registry: defaultRegistry });\n```\n\n## Generated output examples\n\n### citationFriendly (default)\n\n```ts\ncrawlerPolicy({\n  preset: 'citationFriendly',\n  sitemaps: ['/sitemap-index.xml']\n})\n```\n\n```\n# Generated by @casoon/astro-crawler-policy\n# preset: citationFriendly\n\nUser-agent: *\nContent-Signal: search=yes, ai-input=yes, ai-train=no\nAllow: /\n\nUser-agent: GPTBot\nDisallow: /\n\nUser-agent: OAI-SearchBot\nAllow: /\n\nUser-agent: ClaudeBot\nAllow: /\n\nUser-agent: claude-web\nAllow: /\n\nUser-agent: Google-Extended\nDisallow: /\n\nUser-agent: CCBot\nDisallow: /\n\nUser-agent: PerplexityBot\nAllow: /\n\nUser-agent: Bytespider\nDisallow: /\n\nUser-agent: meta-externalagent\nAllow: /\n\nUser-agent: Amazonbot\nAllow: /\n\nUser-agent: Applebot-Extended\nDisallow: /\n\nUser-agent: Googlebot\nAllow: /\n\nUser-agent: Bingbot\nAllow: /\n\nSitemap: https://example.com/sitemap-index.xml\n```\n\n### seoOnly\n\n```ts\ncrawlerPolicy({ preset: 'seoOnly' })\n```\n\n```\n# Generated by @casoon/astro-crawler-policy\n# preset: seoOnly\n\nUser-agent: *\nContent-Signal: search=yes, ai-input=no, ai-train=no\nAllow: /\n\nUser-agent: GPTBot\nDisallow: /\n\nUser-agent: OAI-SearchBot\nDisallow: /\n\nUser-agent: ClaudeBot\nDisallow: /\n\nUser-agent: claude-web\nDisallow: /\n\nUser-agent: Google-Extended\nDisallow: /\n\nUser-agent: CCBot\nDisallow: /\n\nUser-agent: PerplexityBot\nDisallow: /\n\nUser-agent: Bytespider\nDisallow: /\n\nUser-agent: meta-externalagent\nDisallow: /\n\nUser-agent: Amazonbot\nDisallow: /\n\nUser-agent: Applebot-Extended\nDisallow: /\n\nUser-agent: Googlebot\nAllow: /\n\nUser-agent: Bingbot\nAllow: /\n```\n\n### lockdown (staging/preview)\n\n```ts\ncrawlerPolicy({\n  env: {\n    staging: { preset: 'lockdown' },\n    preview: { preset: 'lockdown' }\n  }\n})\n```\n\nWhen `CONTEXT=staging` or `NODE_ENV=staging`:\n\n```\n# Generated by @casoon/astro-crawler-policy\n# preset: lockdown\n\nUser-agent: *\nContent-Signal: search=no, ai-input=no, ai-train=no\nDisallow: /\n```\n\n---\n\n> This tool only works for crawlers and AI bots that actually respect robots.txt. Respect, however, is rare these days.\n","readmeFilename":"README.md"}