{"_id":"@6digit/silktext","_rev":"6-18022079a3d80e936c43c44c19318320","name":"@6digit/silktext","dist-tags":{"latest":"0.1.5"},"versions":{"0.1.0":{"name":"@6digit/silktext","version":"0.1.0","keywords":["crawler","markdown","web-scraping","convex","readability","turndown","sitemap","robots.txt"],"author":{"name":"6Digit"},"license":"MIT","_id":"@6digit/silktext@0.1.0","maintainers":[{"name":"cocporn","email":"cocporn@gmail.com"}],"homepage":"https://github.com/6digit-studio/silktext#readme","bugs":{"url":"https://github.com/6digit-studio/silktext/issues"},"dist":{"shasum":"8b6a445f42a5919fe486c7566f5bb9c2e3916012","tarball":"https://registry.npmjs.org/@6digit/silktext/-/silktext-0.1.0.tgz","fileCount":31,"integrity":"sha512-59e+vKbhlz4FZjH9uH5sqDOANjTMbEgNbgin8tb+pR6fr3j1MDyYqFN/24j+4CqhSic+02574xnWjqG4UqVhRw==","signatures":[{"sig":"MEUCIQCQRuIKRLHpPMCuZ2jr0ELV9d7qVrLLsH9BLnJsIpM22AIgVYy6bFN0cL+NvBZwBErJucCpFGhlleVG+k8WPJjiIhU=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":64821},"main":"./dist/index.js","type":"module","types":"./dist/index.d.ts","engines":{"node":">=18.0.0"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js"},"./renderers/apify":{"types":"./dist/renderers/apify.d.ts","import":"./dist/renderers/apify.js"},"./renderers/firecrawl":{"types":"./dist/renderers/firecrawl.d.ts","import":"./dist/renderers/firecrawl.js"}},"gitHead":"7fe7bed243e33159828514d3690709e6c8349538","scripts":{"dev":"tsc --watch","lint":"tsc --noEmit","test":"node --test tests/*.test.js","build":"tsc","prepublishOnly":"npm run build && npm test"},"_npmUser":{"name":"cocporn","email":"cocporn@gmail.com"},"repository":{"url":"git+https://github.com/6digit-studio/silktext.git","type":"git"},"_npmVersion":"11.4.2","description":"Lightweight, runtime-safe crawling → clean Markdown","directories":{},"sideEffects":false,"_nodeVersion":"23.2.0","dependencies":{"turndown":"^7.1.2"},"publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"typescript":"^5.3.0","@types/node":"^20.0.0","@types/jsdom":"^21.1.6","@types/turndown":"^5.0.4"},"optionalDependencies":{"jsdom":"^24.0.0","cheerio":"^1.0.0","sitemapper":"^3.3.7","@mozilla/readability":"^0.5.0"},"_npmOperationalInternal":{"tmp":"tmp/silktext_0.1.0_1755163127680_0.03173966643688919","host":"s3://npm-registry-packages-npm-production"}},"0.1.1":{"name":"@6digit/silktext","version":"0.1.1","keywords":["crawler","markdown","web-scraping","convex","readability","turndown","sitemap","robots.txt"],"author":{"name":"6Digit"},"license":"MIT","_id":"@6digit/silktext@0.1.1","maintainers":[{"name":"cocporn","email":"cocporn@gmail.com"}],"homepage":"https://github.com/6digit-studio/silktext#readme","bugs":{"url":"https://github.com/6digit-studio/silktext/issues"},"dist":{"shasum":"5428171721dbfdbee3a9ce76594640b9b7a9167a","tarball":"https://registry.npmjs.org/@6digit/silktext/-/silktext-0.1.1.tgz","fileCount":31,"integrity":"sha512-cj7C5Fy7naCukWLKkmHqufc2oerQH5m1F6JVWN7c31WBbyNlwKSGVlPxbFEtWZzYE3v5aa4QsSho1scqjh2+mw==","signatures":[{"sig":"MEQCIEHvj+F7u2dPzTP2LirNyH7Kmhsh8KNQ6zBufYji5XDeAiBdoRcLMEr3ROMxNaPitWhXD5+BL38+01gKyfPo6+zFcg==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":66568},"main":"./dist/index.js","type":"module","types":"./dist/index.d.ts","engines":{"node":">=18.0.0"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js"},"./renderers/apify":{"types":"./dist/renderers/apify.d.ts","import":"./dist/renderers/apify.js"},"./renderers/firecrawl":{"types":"./dist/renderers/firecrawl.d.ts","import":"./dist/renderers/firecrawl.js"}},"gitHead":"7fe7bed243e33159828514d3690709e6c8349538","scripts":{"dev":"tsc --watch","lint":"tsc --noEmit","test":"node --test tests/*.test.js","build":"tsc","prepublishOnly":"npm run build && npm test"},"_npmUser":{"name":"cocporn","email":"cocporn@gmail.com"},"repository":{"url":"git+https://github.com/6digit-studio/silktext.git","type":"git"},"_npmVersion":"11.4.2","description":"Lightweight, runtime-safe crawling → clean Markdown","directories":{},"sideEffects":false,"_nodeVersion":"23.2.0","dependencies":{"turndown":"^7.1.2"},"publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"typescript":"^5.3.0","@types/node":"^20.0.0","@types/jsdom":"^21.1.6","@types/turndown":"^5.0.4"},"optionalDependencies":{"jsdom":"^24.0.0","cheerio":"^1.0.0","sitemapper":"^3.3.7","@mozilla/readability":"^0.5.0"},"_npmOperationalInternal":{"tmp":"tmp/silktext_0.1.1_1755164105000_0.02852724384400629","host":"s3://npm-registry-packages-npm-production"}},"0.1.2":{"name":"@6digit/silktext","version":"0.1.2","keywords":["crawler","markdown","web-scraping","convex","readability","turndown","sitemap","robots.txt"],"author":{"name":"6digit"},"license":"MIT","_id":"@6digit/silktext@0.1.2","maintainers":[{"name":"cocporn","email":"cocporn@gmail.com"}],"homepage":"https://github.com/6digit-studio/silktext#readme","bugs":{"url":"https://github.com/6digit-studio/silktext/issues"},"dist":{"shasum":"cccd7796943a44417f93ef8ca201518d0d40cfa8","tarball":"https://registry.npmjs.org/@6digit/silktext/-/silktext-0.1.2.tgz","fileCount":39,"integrity":"sha512-YoQj3Mxp85/g9tqSROrIg2a8azEs1B97EyaHv7MFqr5fz0h24GoJF9pglZNhwfGO/nTOBTS3nQyAC8q4hz50tg==","signatures":[{"sig":"MEUCIQCxAPFKyv2Y83ONmFPs+iMtB4o3XZBZQUgK/i++Rvy6lQIgTXz4GXN49mF/hsxRw+Bz/znCMeX+h8l9T/IrhcCryT4=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":77404},"main":"./dist/index.js","type":"module","types":"./dist/index.d.ts","engines":{"node":">=18.0.0"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js"},"./convex":{"types":"./dist/convex.d.ts","import":"./dist/convex.js"},"./renderers/apify":{"types":"./dist/renderers/apify.d.ts","import":"./dist/renderers/apify.js"},"./renderers/firecrawl":{"types":"./dist/renderers/firecrawl.d.ts","import":"./dist/renderers/firecrawl.js"}},"gitHead":"71e1085055e6c3e600fa46fd31053775e2c4c2b9","scripts":{"dev":"tsc --watch","lint":"tsc --noEmit","test":"node --test tests/*.test.js","build":"tsc","prepublishOnly":"npm run build && npm test"},"_npmUser":{"name":"cocporn","email":"cocporn@gmail.com"},"repository":{"url":"git+https://github.com/6digit-studio/silktext.git","type":"git"},"_npmVersion":"11.4.2","description":"Lightweight, runtime-safe crawling → clean Markdown","directories":{},"sideEffects":false,"_nodeVersion":"23.2.0","dependencies":{"turndown":"^7.1.2"},"publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"typescript":"^5.3.0","@types/node":"^20.0.0","@types/jsdom":"^21.1.6","@types/turndown":"^5.0.4"},"optionalDependencies":{"jsdom":"^24.0.0","cheerio":"^1.0.0","sitemapper":"^3.3.7","@mozilla/readability":"^0.5.0"},"_npmOperationalInternal":{"tmp":"tmp/silktext_0.1.2_1755169838615_0.8164564525330231","host":"s3://npm-registry-packages-npm-production"}},"0.1.3":{"name":"@6digit/silktext","version":"0.1.3","keywords":["crawler","markdown","web-scraping","convex","readability","turndown","sitemap","robots.txt"],"author":{"name":"6digit"},"license":"MIT","_id":"@6digit/silktext@0.1.3","maintainers":[{"name":"cocporn","email":"cocporn@gmail.com"}],"homepage":"https://github.com/6digit-studio/silktext#readme","bugs":{"url":"https://github.com/6digit-studio/silktext/issues"},"dist":{"shasum":"eb5e9d455d681c46a149f02836863e36c62d8214","tarball":"https://registry.npmjs.org/@6digit/silktext/-/silktext-0.1.3.tgz","fileCount":43,"integrity":"sha512-zswzZLATeWtqbT5IwPGObLj9eYqb85XveA84l59r0pRej7OdHgaB6x3WIvTU72xQh21n22Vw2p1KF2YGp8rZSQ==","signatures":[{"sig":"MEYCIQDde+hdswPXGjUhTzrcxTdu/NNbAgeQfRPRwaTXu4lBkwIhAOL2TsBb6fw8czZnFtgDgpPbtLewWiuDtYCl5unhRY9Y","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":95039},"main":"./dist/index.js","type":"module","types":"./dist/index.d.ts","engines":{"node":">=18.0.0"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js"},"./convex":{"types":"./dist/convex.d.ts","import":"./dist/convex.js"},"./renderers/apify":{"types":"./dist/renderers/apify.d.ts","import":"./dist/renderers/apify.js"},"./renderers/firecrawl":{"types":"./dist/renderers/firecrawl.d.ts","import":"./dist/renderers/firecrawl.js"}},"gitHead":"71e1085055e6c3e600fa46fd31053775e2c4c2b9","scripts":{"dev":"tsc --watch","lint":"tsc --noEmit","test":"node --test tests/*.test.js","build":"tsc","prepublishOnly":"npm run build && npm test"},"_npmUser":{"name":"cocporn","email":"cocporn@gmail.com"},"repository":{"url":"git+https://github.com/6digit-studio/silktext.git","type":"git"},"_npmVersion":"11.4.2","description":"Lightweight, runtime-safe crawling → clean Markdown","directories":{},"sideEffects":false,"_nodeVersion":"23.2.0","dependencies":{"turndown":"^7.1.2"},"publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"typescript":"^5.3.0","@types/node":"^20.0.0","@types/jsdom":"^21.1.6","@types/turndown":"^5.0.4"},"optionalDependencies":{"jsdom":"^24.0.0","cheerio":"^1.0.0","sitemapper":"^3.3.7","@mozilla/readability":"^0.5.0"},"_npmOperationalInternal":{"tmp":"tmp/silktext_0.1.3_1755219769818_0.2575503692259473","host":"s3://npm-registry-packages-npm-production"}},"0.1.4":{"name":"@6digit/silktext","version":"0.1.4","keywords":["crawler","markdown","web-scraping","convex","readability","turndown","sitemap","robots.txt"],"author":{"name":"6digit"},"license":"MIT","_id":"@6digit/silktext@0.1.4","maintainers":[{"name":"cocporn","email":"cocporn@gmail.com"}],"homepage":"https://github.com/6digit-studio/silktext#readme","bugs":{"url":"https://github.com/6digit-studio/silktext/issues"},"dist":{"shasum":"c624fae1dde2a976d51ed5a85e1d546228dd566a","tarball":"https://registry.npmjs.org/@6digit/silktext/-/silktext-0.1.4.tgz","fileCount":43,"integrity":"sha512-b9cDXnz7Gdp4cNhd+2OMLVFPSCE7AoartahZlW8dKmGUXAZevAC07m5LnxgjtzWGJAfqchU854fsOl9c+kZedQ==","signatures":[{"sig":"MEYCIQCJxQGPJSlRtO3dl9sUHfRMa7nuQsnImhElkpfea8TELgIhAILQQ6eZXjpyiWz9kIUnKGzlUaBDTcKIFsRxR9gM667F","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":95154},"main":"./dist/index.js","type":"module","types":"./dist/index.d.ts","engines":{"node":">=18.0.0"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js"},"./convex":{"types":"./dist/convex.d.ts","import":"./dist/convex.js"},"./simple-crawl":{"types":"./dist/simple-crawl.d.ts","import":"./dist/simple-crawl.js"},"./renderers/apify":{"types":"./dist/renderers/apify.d.ts","import":"./dist/renderers/apify.js"},"./renderers/firecrawl":{"types":"./dist/renderers/firecrawl.d.ts","import":"./dist/renderers/firecrawl.js"}},"gitHead":"71e1085055e6c3e600fa46fd31053775e2c4c2b9","scripts":{"dev":"tsc --watch","lint":"tsc --noEmit","test":"node --test tests/*.test.js","build":"tsc","prepublishOnly":"npm run build && npm test"},"_npmUser":{"name":"cocporn","email":"cocporn@gmail.com"},"repository":{"url":"git+https://github.com/6digit-studio/silktext.git","type":"git"},"_npmVersion":"11.4.2","description":"Lightweight, runtime-safe crawling → clean Markdown","directories":{},"sideEffects":false,"_nodeVersion":"23.2.0","dependencies":{"turndown":"^7.1.2"},"publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"typescript":"^5.3.0","@types/node":"^20.0.0","@types/jsdom":"^21.1.6","@types/turndown":"^5.0.4"},"optionalDependencies":{"jsdom":"^24.0.0","cheerio":"^1.0.0","sitemapper":"^3.3.7","@mozilla/readability":"^0.5.0"},"_npmOperationalInternal":{"tmp":"tmp/silktext_0.1.4_1755220035968_0.0487169464747641","host":"s3://npm-registry-packages-npm-production"}},"0.1.5":{"name":"@6digit/silktext","version":"0.1.5","description":"Lightweight, runtime-safe crawling → clean Markdown","type":"module","main":"./dist/index.js","types":"./dist/index.d.ts","publishConfig":{"access":"public"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js"},"./convex":{"types":"./dist/convex.d.ts","import":"./dist/convex.js"},"./renderers/firecrawl":{"types":"./dist/renderers/firecrawl.d.ts","import":"./dist/renderers/firecrawl.js"},"./renderers/apify":{"types":"./dist/renderers/apify.d.ts","import":"./dist/renderers/apify.js"},"./simple-crawl":{"types":"./dist/simple-crawl.d.ts","import":"./dist/simple-crawl.js"}},"sideEffects":false,"scripts":{"build":"tsc","dev":"tsc --watch","test":"node --test tests/*.test.js","lint":"tsc --noEmit","prepublishOnly":"npm run build && npm test"},"keywords":["crawler","markdown","web-scraping","convex","readability","turndown","sitemap","robots.txt"],"author":{"name":"6digit"},"license":"MIT","repository":{"type":"git","url":"git+https://github.com/6digit-studio/silktext.git"},"engines":{"node":">=18.0.0"},"dependencies":{"turndown":"^7.1.2"},"optionalDependencies":{"jsdom":"^24.0.0","@mozilla/readability":"^0.5.0","cheerio":"^1.0.0","sitemapper":"^3.3.7"},"devDependencies":{"@types/node":"^20.0.0","@types/turndown":"^5.0.4","@types/jsdom":"^21.1.6","typescript":"^5.3.0"},"_id":"@6digit/silktext@0.1.5","gitHead":"71e1085055e6c3e600fa46fd31053775e2c4c2b9","bugs":{"url":"https://github.com/6digit-studio/silktext/issues"},"homepage":"https://github.com/6digit-studio/silktext#readme","_nodeVersion":"23.2.0","_npmVersion":"11.4.2","dist":{"integrity":"sha512-brr3QAQsYztbYCQT2qqJJWUqJQxmpo0wuhGdD6YmjTZcksJ/DdIiOSt577BGPgeNU0afk9p8RceFGAXKLSdEXw==","shasum":"320f3d43f92ab9ede198ea8d0e34ec9c1c49c84f","tarball":"https://registry.npmjs.org/@6digit/silktext/-/silktext-0.1.5.tgz","fileCount":43,"unpackedSize":95745,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIBMl8JeYfgTofJOv/RN/q1F+FGkf+85Wz1lZxwQ+4+fjAiEAnylNbHRZBvkgMBnCjhjzov3qfK50G9Ube61w/l6HElE="}]},"_npmUser":{"name":"cocporn","email":"cocporn@gmail.com"},"directories":{},"maintainers":[{"name":"cocporn","email":"cocporn@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/silktext_0.1.5_1755221397543_0.5102936595620333"},"_hasShrinkwrap":false}},"time":{"created":"2025-08-14T09:18:47.586Z","modified":"2025-08-15T01:29:57.939Z","0.1.0":"2025-08-14T09:18:47.887Z","0.1.1":"2025-08-14T09:35:05.240Z","0.1.2":"2025-08-14T11:10:38.768Z","0.1.3":"2025-08-15T01:02:49.980Z","0.1.4":"2025-08-15T01:07:16.162Z","0.1.5":"2025-08-15T01:29:57.724Z"},"bugs":{"url":"https://github.com/6digit-studio/silktext/issues"},"author":{"name":"6digit"},"license":"MIT","homepage":"https://github.com/6digit-studio/silktext#readme","keywords":["crawler","markdown","web-scraping","convex","readability","turndown","sitemap","robots.txt"],"repository":{"type":"git","url":"git+https://github.com/6digit-studio/silktext.git"},"description":"Lightweight, runtime-safe crawling → clean Markdown","maintainers":[{"name":"cocporn","email":"cocporn@gmail.com"}],"readme":"# @6digit/silktext\n\n[![npm version](https://badge.fury.io/js/@6digit%2Fsilktext.svg)](https://www.npmjs.com/package/@6digit/silktext)\n[![TypeScript](https://img.shields.io/badge/TypeScript-Ready-blue.svg)](https://www.typescriptlang.org/)\n[![ESM](https://img.shields.io/badge/ESM-Native-green.svg)](https://nodejs.org/api/esm.html)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n\n> Lightweight, runtime-safe crawling → clean Markdown\n\nA TypeScript library for crawling websites and extracting clean Markdown content, designed to work in any JavaScript runtime (Node 18+, Bun, Deno, Convex, Workers).\n\n## Features\n\n- **Runtime-agnostic**: Works everywhere - Convex, Node, Bun, Deno, Workers\n- **Clean extraction**: Converts HTML to clean, consistent Markdown\n- **Smart crawling**: Respects robots.txt, rate limits, and crawl depth\n- **Flexible strategies**: Heuristic (fast) or Readability (thorough) extraction\n- **No heavy deps**: Core has minimal dependencies; heavy libs are optional\n- **TypeScript-first**: Full type safety with clear data contracts\n- **SPA-aware**: Detects thin HTML and supports renderer adapters (coming soon)\n\n## Installation\n\n```bash\nnpm install @6digit/silktext\n```\n\n## Quick Start\n\n### One-liner crawl\n\n```typescript\nimport { crawlSite } from \"@6digit/silktext\";\n\nawait crawlSite({ \n  startUrls: [\"https://example.com\"], \n  maxDepth: 2, \n  onPage: (page) => console.log(page.markdown) \n});\n```\n\n### Two-liner fetch → extract\n\n```typescript\nimport { fetchHtml, extract } from \"@6digit/silktext\";\n\nconst resp = await fetchHtml({ fetch: globalThis.fetch }, { url: \"https://example.com\" });\nconst page = await extract(resp, { strategy: \"heuristic\" });\nconsole.log(page.markdown);\n```\n\n### Extract HTML directly\n\n```typescript\nimport { extractHtml } from \"@6digit/silktext\";\n\nconst page = await extractHtml(htmlString, \"https://example.com\");\nconsole.log(page.metadata.title, page.markdown);\n```\n\n## Convex Usage\n\nsilktext works in both Convex runtime modes:\n\n### Default Convex Runtime (recommended)\nFast and lightweight with heuristic extraction:\n\n```typescript\nimport { action } from \"convex/_generated/server\";\nimport { crawlSite } from \"@6digit/silktext\";\n\nexport const crawlDocs = action({\n  args: {},\n  handler: async (ctx) => {\n    const pages: string[] = [];\n    \n    await crawlSite({\n      startUrls: [\"https://docs.example.com\"],\n      maxDepth: 2,\n      sameOrigin: true,\n      onPage: (page) => {\n        pages.push(`# ${page.metadata.title}\\n\\n${page.markdown}`);\n      }\n    });\n    \n    return pages.join(\"\\n\\n---\\n\\n\");\n  }\n});\n```\n\n### Node.js Runtime in Convex\nFor advanced extraction with Readability:\n\n```typescript\n\"use node\";  // Enable Node.js runtime\n\nimport { action } from \"convex/_generated/server\";\nimport { crawlSite } from \"@6digit/silktext\";\n\nexport const crawlWithReadability = action({\n  args: {},\n  handler: async (ctx) => {\n    const pages: string[] = [];\n    \n    await crawlSite({\n      startUrls: [\"https://article-site.com\"],\n      maxDepth: 1,\n      onPage: async (page) => {\n        // Can use Readability strategy in Node runtime\n        const enhanced = await extract(page, { strategy: \"readability\" });\n        pages.push(enhanced.markdown);\n      }\n    });\n    \n    return pages;\n  }\n});\n```\n\n## API Reference\n\n### `crawlSite(options)`\n\nCrawl a website starting from one or more URLs.\n\n```typescript\ninterface CrawlOptions {\n  startUrls: string[];              // URLs to start crawling from\n  maxDepth?: number;                // Max crawl depth (default: 1)\n  maxPages?: number;                // Max pages to crawl (default: 100)\n  sameOrigin?: boolean;             // Stay on same host (default: true)\n  allow?: (url: string) => boolean; // Custom allow filter\n  deny?: (url: string) => boolean;  // Custom deny filter\n  userAgent?: string;               // Custom user agent\n  rate?: {                          // Rate limiting\n    perHostRps?: number;            // Requests per second per host (default: 1)\n    concurrent?: number;            // Concurrent requests (default: 4)\n  };\n  onPage?: (page: ExtractResult, ctx: CrawlContext) => Promise<void> | void;\n}\n```\n\n### `fetchHtml(http, request)`\n\nFetch HTML from a URL with proper headers and redirect handling.\n\n```typescript\ninterface FetchRequest {\n  url: string;\n  headers?: Record<string, string>;\n}\n\nconst response = await fetchHtml(\n  { fetch: globalThis.fetch }, \n  { url: \"https://example.com\" }\n);\n```\n\n### `extract(response, options)`\n\nExtract clean Markdown and metadata from HTML.\n\n```typescript\ninterface ExtractOptions {\n  strategy?: \"heuristic\" | \"readability\";  // Default: \"heuristic\"\n  langHint?: string;                        // Language hint for extraction\n}\n\nconst result = await extract(fetchResponse, { strategy: \"heuristic\" });\n```\n\n### `extractHtml(html, url, options)`\n\nExtract directly from an HTML string.\n\n```typescript\nconst result = await extractHtml(\n  \"<html>...</html>\", \n  \"https://example.com\",\n  { strategy: \"readability\" }\n);\n```\n\n## Extraction Strategies\n\n### Heuristic (default)\n- Fast, lightweight extraction\n- Removes navigation, ads, and other noise\n- Finds main content using common selectors\n- **Convex-compatible** - uses native DOMParser in Convex/browser\n- Falls back to regex-based extraction in Node without jsdom\n\n### Readability\n- Uses Mozilla's Readability algorithm\n- Better for complex article layouts\n- **NOT Convex-compatible** - requires dynamic imports\n- Only available in Node.js with jsdom installed\n- More accurate for news/blog content\n\n## Convex Runtime Compatibility\n\nsilktext is designed for maximum Convex compatibility with clear runtime boundaries:\n\n### Default Convex Runtime (V8 Isolate)\nThe default Convex runtime provides a limited DOM environment. In this mode:\n\n✅ **What works:**\n- Heuristic extraction using native `DOMParser` (when available)\n- Turndown for HTML→Markdown conversion (works with strings)\n- All crawling features\n- URL normalization and rate limiting\n- Basic metadata extraction\n\n⚠️ **Limitations:**\n- No `jsdom` or heavy DOM libraries\n- Readability strategy not available\n- Falls back to regex-based extraction if DOMParser unavailable\n\n### Convex with \"use node\" Runtime\nAdd `\"use node\";` at the top of your action file for full Node.js capabilities:\n\n✅ **Additional features:**\n- Readability extraction strategy via dynamic `import('jsdom')`\n- Full DOM parsing with jsdom\n- All Node.js-specific features\n\n**Note:** The heuristic extraction works in both modes. Turndown accepts HTML strings directly, so most content extraction works without a full DOM.\n\n## Handling SPAs and Thin HTML\n\nSingle-page applications often return minimal HTML shells like:\n```html\n<div id=\"app\"></div>\n```\n\nsilktext handles this gracefully:\n- **Default behavior**: Returns minimal extracted content (won't throw)\n- **With renderer**: Use the optional `renderer` parameter in `crawlSite`\n- **Coming soon**: Pre-built renderer adapters for Firecrawl, Playwright, etc.\n\nExample with custom renderer:\n```typescript\nawait crawlSite({\n  startUrls: ['https://spa-site.com'],\n  renderer: {\n    render: async (url) => {\n      // Your rendering logic here (e.g., Playwright, Puppeteer)\n      return { html: renderedHtml };\n    }\n  },\n  onPage: (page) => console.log(page.markdown)\n});\n```\n\n## Data Types\n\n### ExtractResult\n\n```typescript\ninterface ExtractResult {\n  url: string;                    // Normalized URL\n  markdown: string;                // Clean Markdown content\n  metadata: {\n    title?: string;                // Page title\n    description?: string;          // Meta description\n    og?: Record<string, string>;  // Open Graph tags\n    twitter?: Record<string, string>; // Twitter Card tags\n    jsonLd?: unknown[];            // JSON-LD structured data\n    links?: string[];              // Discovered links\n  };\n}\n```\n\n## Features\n\n### URL Normalization\n- Lowercases hostnames\n- Removes tracking parameters (utm_*, fbclid, gclid, etc.)\n- Strips URL fragments\n- Sorts query parameters\n\n### Politeness\n- Respects rate limits (1 RPS per host by default)\n- Concurrent request limiting\n- Per-host token bucket rate limiting\n\n### Content Extraction\n- Removes scripts, styles, navigation, footers\n- Preserves main content structure\n- Converts to clean Markdown with Turndown\n- Extracts all metadata (OG, Twitter, JSON-LD)\n\n## License\n\nMIT","readmeFilename":"README.md"}