{"_id":"@25xcodes/llmstxt-parser","name":"@25xcodes/llmstxt-parser","dist-tags":{"latest":"1.2.0"},"versions":{"1.2.0":{"name":"@25xcodes/llmstxt-parser","version":"1.2.0","description":"High-quality llms.txt parser and validator for the llmstxt.org specification","type":"module","main":"./dist/index.cjs","module":"./dist/index.js","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.cjs"}},"scripts":{"build":"tsup","dev":"tsup --watch","test":"vitest run","test:watch":"vitest","prepublishOnly":"npm run build"},"keywords":["llms.txt","llmstxt","llm","parser","validator","markdown","ai","rag","mcp","model-context-protocol"],"author":"","license":"MIT","repository":{"type":"git","url":"git+https://github.com/kiarashplusplus/webmcp-tooling-suite.git","directory":"packages/llmstxt-parser"},"homepage":"https://llmstxt.org","devDependencies":{"@vitest/coverage-v8":"^4.0.0","tsup":"^8.0.0","typescript":"^5.3.0","vitest":"^4.0.0"},"_id":"@25xcodes/llmstxt-parser@1.2.0","gitHead":"06b246443edf450eb842ec3e38e47b339debb92e","bugs":{"url":"https://github.com/kiarashplusplus/webmcp-tooling-suite/issues"},"_nodeVersion":"20.19.6","_npmVersion":"10.8.2","dist":{"integrity":"sha512-0aMV021iA3F7i3kavhvAkv/EgJynpr1xkmvvA1JtimKcXdDYKZD/0EdwMMUK3FIxXpN+qT1Wm8Dg+nVTNg9WiQ==","shasum":"426c9557ec641b927a18bdb8de1c744597e9ac5e","tarball":"https://registry.npmjs.org/@25xcodes/llmstxt-parser/-/llmstxt-parser-1.2.0.tgz","fileCount":8,"unpackedSize":111918,"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@25xcodes%2fllmstxt-parser@1.2.0","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIQCKVg/Vc0vlEagf6OuYEZSnEiNHc0cQ8zmhyQ3ssLoClgIgOa9Mi9ec7xaSIwOIMUbNirY7EwdJ+p6xnQJJtPp/zT0="}]},"_npmUser":{"name":"25xcodes","email":"kiarash@25x.codes"},"directories":{},"maintainers":[{"name":"25xcodes","email":"kiarash@25x.codes"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/llmstxt-parser_1.2.0_1764957269058_0.11905463675601102"},"_hasShrinkwrap":false}},"time":{"created":"2025-12-05T17:54:28.999Z","1.2.0":"2025-12-05T17:54:29.196Z","modified":"2025-12-05T17:54:29.615Z"},"maintainers":[{"name":"25xcodes","email":"kiarash@25x.codes"}],"description":"High-quality llms.txt parser and validator for the llmstxt.org specification","homepage":"https://llmstxt.org","keywords":["llms.txt","llmstxt","llm","parser","validator","markdown","ai","rag","mcp","model-context-protocol"],"repository":{"type":"git","url":"git+https://github.com/kiarashplusplus/webmcp-tooling-suite.git","directory":"packages/llmstxt-parser"},"bugs":{"url":"https://github.com/kiarashplusplus/webmcp-tooling-suite/issues"},"license":"MIT","readme":"# @25xcodes/llmstxt-parser\n\n[![npm version](https://img.shields.io/npm/v/@25xcodes/llmstxt-parser.svg)](https://www.npmjs.com/package/@25xcodes/llmstxt-parser)\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)\n\nA high-quality TypeScript library for parsing and validating [llms.txt](https://llmstxt.org) files.\n\n## Features\n\n- 🚀 **Zero dependencies** — Works in Node.js and browsers\n- 📝 **Full TypeScript support** — Comprehensive type definitions\n- ✅ **Validation** — Per the llmstxt.org specification\n- 🔍 **Discovery** — Auto-discover llms.txt from well-known paths\n- 🎯 **RAG-ready** — Utilities for embedding and indexing\n- 📊 **Token estimation** — Approximate token counts for LLMs\n\n## Installation\n\n```bash\nnpm install @25xcodes/llmstxt-parser\n```\n\n## Quick Start\n\n```typescript\nimport { parseLLMSTxt, validateLLMSTxt } from '@25xcodes/llmstxt-parser'\n\nconst markdown = `\n# My Project\n\n> A brief description of the project.\n\n## Documentation\n- [Getting Started](https://example.com/docs/start): Quick start guide\n- [API Reference](https://example.com/docs/api): Full API docs\n`\n\n// Parse the document\nconst doc = parseLLMSTxt(markdown)\nconsole.log(doc.title)    // \"My Project\"\nconsole.log(doc.summary)  // \"A brief description of the project.\"\nconsole.log(doc.links)    // [{ title: \"Getting Started\", ... }, ...]\n\n// Validate the document\nconst result = validateLLMSTxt(doc)\nconsole.log(result.valid)  // true\nconsole.log(result.score)  // 95\n```\n\n## API Reference\n\n### Parsing\n\n#### `parseLLMSTxt(markdown: string): LLMSTxtDocument`\n\nParse an llms.txt markdown string into a structured document.\n\n```typescript\nimport { parseLLMSTxt } from '@25xcodes/llmstxt-parser'\n\nconst doc = parseLLMSTxt(markdown)\n\nconsole.log(doc.title)     // H1 title (required)\nconsole.log(doc.summary)   // Blockquote summary (optional)\nconsole.log(doc.sections)  // Array of sections with links\nconsole.log(doc.links)     // All links (flattened)\nconsole.log(doc.raw)       // Original markdown\n```\n\n### Validation\n\n#### `validateLLMSTxt(doc: LLMSTxtDocument): LLMSTxtValidationResult`\n\nValidate a parsed document against the llmstxt.org specification.\n\n```typescript\nimport { parseLLMSTxt, validateLLMSTxt } from '@25xcodes/llmstxt-parser'\n\nconst doc = parseLLMSTxt(markdown)\nconst result = validateLLMSTxt(doc)\n\nif (!result.valid) {\n  console.log('Errors:', result.errors)\n  console.log('Warnings:', result.warnings)\n}\n\nconsole.log('Score:', result.score) // 0-100\n```\n\n#### `parseAndValidate(markdown: string)`\n\nParse and validate in one call.\n\n```typescript\nimport { parseAndValidate } from '@25xcodes/llmstxt-parser'\n\nconst { document, validation } = parseAndValidate(markdown)\n```\n\n### Fetching\n\n#### `fetchLLMSTxt(urlOrDomain: string, options?: FetchOptions): Promise<LLMSTxtDocument>`\n\nFetch and parse llms.txt from a URL or domain.\n\n```typescript\nimport { fetchLLMSTxt } from '@25xcodes/llmstxt-parser'\n\n// Fetch from a specific URL\nconst doc = await fetchLLMSTxt('https://example.com/llms.txt')\n\n// Or discover from a domain (tries well-known paths)\nconst doc = await fetchLLMSTxt('example.com')\n\n// With CORS proxy for browser environments\nconst doc = await fetchLLMSTxt('example.com', {\n  corsProxy: 'https://my-cors-proxy.workers.dev'\n})\n```\n\n**Options:**\n\n| Option | Type | Default | Description |\n|--------|------|---------|-------------|\n| `timeout` | `number` | `10000` | Request timeout in ms |\n| `checkFull` | `boolean` | `true` | Also check for llms-full.txt |\n| `corsProxy` | `string` | — | CORS proxy URL |\n| `fetch` | `typeof fetch` | `globalThis.fetch` | Custom fetch function |\n\n#### `discoverLLMSTxtFiles(domain: string, options?: FetchOptions): Promise<DiscoveredFile[]>`\n\nDiscover all available llms.txt files for a domain.\n\n```typescript\nimport { discoverLLMSTxtFiles } from '@25xcodes/llmstxt-parser'\n\nconst files = await discoverLLMSTxtFiles('example.com')\n// [\n//   { url: 'https://example.com/llms.txt', type: 'standard' },\n//   { url: 'https://example.com/llms-full.txt', type: 'full' }\n// ]\n```\n\n### Utilities\n\n#### `estimateTokens(doc: LLMSTxtDocument): TokenEstimate`\n\nEstimate token count for LLM context.\n\n```typescript\nimport { parseLLMSTxt, estimateTokens } from '@25xcodes/llmstxt-parser'\n\nconst doc = parseLLMSTxt(markdown)\nconst tokens = estimateTokens(doc)\n\nconsole.log(`Total: ~${tokens.total} tokens`)\ntokens.bySection.forEach(s => {\n  console.log(`  ${s.section}: ~${s.tokens}`)\n})\n```\n\n#### `toRAGFormat(doc: LLMSTxtDocument): string`\n\nConvert document to plain text format for RAG systems.\n\n```typescript\nimport { parseLLMSTxt, toRAGFormat } from '@25xcodes/llmstxt-parser'\n\nconst doc = parseLLMSTxt(markdown)\nconst ragText = toRAGFormat(doc)\n\n// Use for embedding or context injection\n```\n\n#### `extractLinksForIndex(doc: LLMSTxtDocument): RAGLinkEntry[]`\n\nExtract structured link data for vector databases.\n\n```typescript\nimport { parseLLMSTxt, extractLinksForIndex } from '@25xcodes/llmstxt-parser'\n\nconst doc = parseLLMSTxt(markdown)\nconst links = extractLinksForIndex(doc)\n\nfor (const link of links) {\n  await vectorDb.insert({\n    id: link.id,\n    content: link.embedContent,\n    metadata: { url: link.url, section: link.section }\n  })\n}\n```\n\n## Types\n\n### `LLMSTxtDocument`\n\n```typescript\ninterface LLMSTxtDocument {\n  title: string           // H1 title (required)\n  summary?: string        // Blockquote summary\n  sections: LLMSTxtSection[]\n  links: LLMSTxtLink[]    // All links (flattened)\n  raw: string             // Original markdown\n  sourceUrl?: string      // If fetched remotely\n  isFull?: boolean        // If llms-full.txt\n}\n```\n\n### `LLMSTxtSection`\n\n```typescript\ninterface LLMSTxtSection {\n  heading: string         // Section title\n  level: 2 | 3            // H2 or H3\n  content?: string        // Description text\n  links: LLMSTxtLink[]    // Links in this section\n}\n```\n\n### `LLMSTxtLink`\n\n```typescript\ninterface LLMSTxtLink {\n  title: string           // Link text\n  url: string             // URL\n  description?: string    // Description after link\n  section?: string        // Parent section name\n  optional?: boolean      // Marked as optional\n}\n```\n\n### `LLMSTxtValidationResult`\n\n```typescript\ninterface LLMSTxtValidationResult {\n  valid: boolean          // No errors\n  score: number           // 0-100\n  errors: LLMSTxtValidationError[]\n  warnings: LLMSTxtValidationWarning[]\n}\n```\n\n## Well-Known Paths\n\nThe library checks these paths when discovering llms.txt:\n\n```typescript\nimport { LLMSTXT_PATHS } from '@25xcodes/llmstxt-parser'\n\n// ['/llms.txt', '/llms-full.txt', '/.well-known/llms.txt']\n```\n\n## llms.txt Specification\n\nThis library implements the [llmstxt.org](https://llmstxt.org) specification:\n\n- **H1 Title** (required): `# Project Name`\n- **Summary** (recommended): `> Brief description`\n- **Sections** (optional): `## Section Name`\n- **Links**: `- [Title](url): Description`\n\n### Example llms.txt\n\n```markdown\n# FastHTML\n\n> FastHTML is a python library for creating server-rendered hypermedia applications.\n\n## Docs\n- [Quick start](https://fastht.ml/docs/quickstart): Get started in 5 minutes\n- [API Reference](https://fastht.ml/docs/api): Full API documentation\n\n## Examples\n- [Todo App](https://github.com/example/todo): Complete CRUD example\n\n## Optional\n- [Starlette docs](https://starlette.io): Underlying framework docs. Optional.\n```\n\n## License\n\nMIT\n","readmeFilename":"README.md","_rev":"1-3ceb7f56b626740d3f0dc1929f493baa"}