{"_id":"@ammit/llms-txt","name":"@ammit/llms-txt","dist-tags":{"latest":"0.1.0"},"versions":{"0.1.0":{"name":"@ammit/llms-txt","version":"0.1.0","description":"Generate llms.txt files for any website. Make your site AI-friendly.","keywords":["llms","llms.txt","ai","agents","crawler","markdown","cli"],"license":"MIT","author":{"name":"Amit Kosti"},"repository":{"type":"git","url":"git+https://github.com/ammit/llms-txt.git"},"homepage":"https://github.com/ammit/llms-txt","bugs":{"url":"https://github.com/ammit/llms-txt/issues"},"type":"module","bin":{"llms-txt":"dist/index.js"},"main":"./dist/index.js","scripts":{"build":"tsup","dev":"tsup --watch","lint":"tsc --noEmit","test":"vitest run","prepublishOnly":"npm run build"},"dependencies":{"@mozilla/readability":"^0.5.0","commander":"^13.0.0","jsdom":"^26.0.0","ora":"^8.2.0","robots-parser":"^3.0.1","turndown":"^7.2.0","turndown-plugin-gfm":"^1.0.2"},"devDependencies":{"@types/jsdom":"^21.1.7","@types/node":"^22.0.0","@types/turndown":"^5.0.5","tsup":"^8.4.0","typescript":"^5.7.0","vitest":"^3.0.0"},"engines":{"node":">=20"},"_id":"@ammit/llms-txt@0.1.0","gitHead":"5d40a971b839526a9618207d607747e1f683d5e2","_nodeVersion":"22.16.0","_npmVersion":"10.9.2","dist":{"integrity":"sha512-99pvk1s3JYjTk8DjUSQUNhM3PyP2dRQl3p427gwn4kbkgiWuiyc/cwi1p3PMLktdd7Mb6HK/kkH4IWAFUoIrDQ==","shasum":"954acdd4ad1b055c60780b9a71a611e698eda5b3","tarball":"https://registry.npmjs.org/@ammit/llms-txt/-/llms-txt-0.1.0.tgz","fileCount":4,"unpackedSize":20943,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEQCIBjh5JJt+947W7BhmZGqLDWeyDd/bhCWRiH5p9SJBg+CAiB/8fLgif9XgVpE6nukblyPd+/S5/wCn9t+124Fi93ZzA=="}]},"_npmUser":{"name":"ammit","email":"akosti92@gmail.com"},"directories":{},"maintainers":[{"name":"ammit","email":"akosti92@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/llms-txt_0.1.0_1775479452607_0.9976114414681432"},"_hasShrinkwrap":false}},"time":{"created":"2026-04-06T12:44:12.201Z","0.1.0":"2026-04-06T12:44:12.747Z","modified":"2026-04-06T12:44:12.961Z"},"maintainers":[{"name":"ammit","email":"akosti92@gmail.com"}],"description":"Generate llms.txt files for any website. Make your site AI-friendly.","homepage":"https://github.com/ammit/llms-txt","keywords":["llms","llms.txt","ai","agents","crawler","markdown","cli"],"repository":{"type":"git","url":"git+https://github.com/ammit/llms-txt.git"},"author":{"name":"Amit Kosti"},"bugs":{"url":"https://github.com/ammit/llms-txt/issues"},"license":"MIT","readme":"# llms-txt\n\nGenerate `llms.txt` files for any website. Make your site AI-friendly.\n\n[llms.txt](https://llmstxt.org/) is a standard that helps AI agents understand your website. It converts your pages into clean, token-efficient markdown that LLMs can consume directly, reducing token usage by ~90% compared to raw HTML.\n\nOnly 5-15% of websites have `llms.txt` today. This tool generates one for you automatically.\n\n## Install\n\n```bash\nnpm install -g @ammit/llms-txt\n```\n\nOr run directly:\n\n```bash\nnpx @ammit/llms-txt https://example.com\n```\n\n## Usage\n\n```bash\n# Generate llms.txt for a site\nllms-txt https://docs.anthropic.com\n\n# Set crawl depth\nllms-txt https://example.com --depth 3\n\n# Output to files\nllms-txt https://example.com -o ./llms.txt\n\n# Include/exclude URL patterns\nllms-txt https://example.com --include \"/docs/**\" --exclude \"/blog/**\"\n\n# Generate llms-full.txt (all content bundled)\nllms-txt https://example.com --full\n```\n\n## What it does\n\n1. Discovers pages via `sitemap.xml` and link following\n2. Extracts clean content using [Readability](https://github.com/mozilla/readability) (strips nav, ads, scripts)\n3. Converts to markdown via [Turndown](https://github.com/mixmark-io/turndown)\n4. Outputs a standard `llms.txt` index and optional `llms-full.txt` bundle\n\n## Output format\n\nThe generated `llms.txt` follows the [llms.txt standard](https://llmstxt.org/):\n\n```markdown\n# Example Docs\n\n> Documentation for the Example platform.\n\n## Getting Started\n\n- [Quick Start](https://example.com/docs/quickstart): Set up your first project in 5 minutes\n- [Installation](https://example.com/docs/install): System requirements and install steps\n\n## API Reference\n\n- [Authentication](https://example.com/docs/api/auth): API keys and OAuth setup\n- [Endpoints](https://example.com/docs/api/endpoints): Complete REST API reference\n```\n\n## Options\n\n| Flag | Description | Default |\n|------|-------------|---------|\n| `--depth, -d` | Max crawl depth | `3` |\n| `--output, -o` | Output file path | stdout |\n| `--full` | Also generate `llms-full.txt` | `false` |\n| `--include` | URL patterns to include (glob) | all |\n| `--exclude` | URL patterns to exclude (glob) | none |\n| `--rate` | Requests per second | `2` |\n| `--concurrency, -c` | Parallel requests | `5` |\n| `--json` | Output as JSON | `false` |\n| `--verbose, -v` | Show detailed skip/fetch logging | `false` |\n| `--timeout` | Fetch timeout in milliseconds | `10000` |\n\n## How it works\n\n```\nURL --> Sitemap/Link Discovery --> Content Extraction --> Markdown Conversion --> llms.txt\n```\n\n- **Discovery**: Checks `sitemap.xml` first, falls back to recursive link following\n- **Extraction**: Mozilla Readability isolates main content, removes chrome\n- **Conversion**: Turndown produces clean GFM markdown\n- **Assembly**: Groups pages by URL path into sections, generates descriptions\n\n## Contributing\n\nContributions welcome. Please open an issue first to discuss what you'd like to change.\n\n## License\n\n[MIT](LICENSE)\n","readmeFilename":"README.md","_rev":"1-bd6d3f4fb2b125c421e516b40b284e87"}