{"_id":"@firecrawl/firecrawl-convex","_rev":"3-dfd9662cc3d37fb614980ac688af25a0","name":"@firecrawl/firecrawl-convex","dist-tags":{"latest":"0.1.1"},"versions":{"0.1.0":{"name":"@firecrawl/firecrawl-convex","version":"0.1.0","keywords":["convex","component","firecrawl","scraping","crawler","web-search","rag"],"author":"","license":"MIT","_id":"@firecrawl/firecrawl-convex@0.1.0","maintainers":[{"name":"abimaelmartell","email":"abimex@gmail.com"},{"name":"mogery","email":"mo.geryy@gmail.com"},{"name":"tomsideguide","email":"tom@sideguide.dev"},{"name":"rakramprakash","email":"rakshithramprakash@gmail.com"},{"name":"hello_sideguide","email":"hello@sideguide.dev"}],"homepage":"https://github.com/firecrawl/firecrawl-convex#readme","bugs":{"url":"https://github.com/firecrawl/firecrawl-convex/issues"},"dist":{"shasum":"40b5630ec16a55ca70c4f464067d049a13a66f0f","tarball":"https://registry.npmjs.org/@firecrawl/firecrawl-convex/-/firecrawl-convex-0.1.0.tgz","fileCount":81,"integrity":"sha512-VK60tp3fI6m1AWwY2ocFaHuQmrHiq6DflUE0/n1QIMbbW2ase9U1WGxBPaQyKrCILkbr/w/CXa1tCqgeKBwg9A==","signatures":[{"sig":"MEUCIAkUYyhBTX0FWH7LnNUHHBaZk989G6lekHhrDFA1SgKLAiEAskVYXRUabM+AT1k0Txq0268rly2pAvJAFOeXBwHCZo0=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":386258},"main":"eslint.config.js","type":"module","types":"./dist/client/index.d.ts","module":"./dist/client/index.js","exports":{".":{"types":"./dist/client/index.d.ts","default":"./dist/client/index.js"},"./test":"./src/test.ts","./package.json":"./package.json","./convex.config":{"types":"./dist/component/convex.config.d.ts","default":"./dist/component/convex.config.js"},"./convex.config.js":{"types":"./dist/component/convex.config.d.ts","default":"./dist/component/convex.config.js"},"./_generated/component":{"types":"./dist/component/_generated/component.d.ts"},"./_generated/component.js":{"types":"./dist/component/_generated/component.d.ts"}},"gitHead":"4e53ca1a7483cbb728481479b58fcd8e51c3859d","scripts":{"dev":"convex dev --start 'npm run dev:build'","lint":"eslint .","test":"vitest run --typecheck","alpha":"npm version prerelease --preid alpha && npm publish --tag alpha && git push --follow-tags","build":"tsc --project ./tsconfig.build.json","clean":"rm -rf dist *.tsbuildinfo","predev":"convex init && npm run build:codegen","verify":"npm run build && npm run test && npm run typecheck && npm run lint","prepare":"npm run build","release":"npm version patch && npm publish && git push --follow-tags","version":"(npm whoami || npm login) && vim -c 'normal o' -c 'normal o## '$npm_package_version CHANGELOG.md && prettier -w CHANGELOG.md && git add CHANGELOG.md","dev:mock":"node example/mock-firecrawl.mjs","dev:build":"chokidar 'tsconfig*.json' 'src/**/*.ts' -i '**/*.test.ts' -c 'npm run build:codegen' --initial","typecheck":"tsc --noEmit && tsc -p example && tsc -p example/convex","preversion":"npm ci && npm run build:clean && npm run test && npm run lint && npm run typecheck","test:watch":"vitest --typecheck --clearScreen false","build:clean":"npm run clean && npm run build:codegen","build:codegen":"npx convex codegen --component-dir ./src/component && npm run build","test:coverage":"vitest run --coverage --coverage.reporter=text","prepublishOnly":"npm run clean && npm run build"},"_npmUser":{"name":"rakramprakash","email":"rakshithramprakash@gmail.com"},"repository":{"url":"git+https://github.com/firecrawl/firecrawl-convex.git","type":"git"},"_npmVersion":"11.19.0","description":"Firecrawl component for Convex: scrape, map, and search the web, and run durable crawls with reactive progress.","directories":{"example":"example"},"_nodeVersion":"26.7.0","dependencies":{"convex-helpers":"^0.1.122"},"_hasShrinkwrap":false,"devDependencies":{"convex":"1.43.0","eslint":"9.39.4","vitest":"4.1.4","globals":"^17.5.0","prettier":"3.8.3","@eslint/js":"9.39.4","typescript":"6.0.3","@types/node":"^24.12.2","convex-test":"0.0.55","chokidar-cli":"3.0.0","@edge-runtime/vm":"^5.0.0","typescript-eslint":"8.58.2","@convex-dev/eslint-plugin":"^2.0.0"},"peerDependencies":{"convex":"^1.43.0"},"_npmOperationalInternal":{"tmp":"tmp/firecrawl-convex_0.1.0_1786082192740_0.6526722318563714","host":"s3://npm-registry-packages-npm-production"}},"0.1.1":{"name":"@firecrawl/firecrawl-convex","version":"0.1.1","keywords":["convex","component","firecrawl","scraping","crawler","web-search","rag"],"license":"MIT","_id":"@firecrawl/firecrawl-convex@0.1.1","maintainers":[{"name":"abimaelmartell","email":"abimex@gmail.com"},{"name":"mogery","email":"mo.geryy@gmail.com"},{"name":"tomsideguide","email":"tom@sideguide.dev"},{"name":"rakramprakash","email":"rakshithramprakash@gmail.com"},{"name":"hello_sideguide","email":"hello@sideguide.dev"}],"homepage":"https://github.com/firecrawl/firecrawl-convex#readme","bugs":{"url":"https://github.com/firecrawl/firecrawl-convex/issues"},"dist":{"shasum":"6568da5fc945642152f8225a01810a42ca60b077","tarball":"https://registry.npmjs.org/@firecrawl/firecrawl-convex/-/firecrawl-convex-0.1.1.tgz","fileCount":80,"integrity":"sha512-fecepodOAcKLUKFD+4dyqRfU8ATn3y15nYtayyAmmQAV0FT/6sVwKNY0+rpq8/NKTLrEpuHr+cE0htNQ+ssU+g==","signatures":[{"sig":"MEYCIQCqK4BvW8OOlJptinr26JvkfnDRt9KPJPMum+SkhMrp5AIhAJkQunT1JVDtwSJSlMlOS4+p+MZUWsUrQd31e/4MS0TN","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":384608},"type":"module","types":"./dist/client/index.d.ts","module":"./dist/client/index.js","exports":{".":{"types":"./dist/client/index.d.ts","default":"./dist/client/index.js"},"./test":"./src/test.ts","./package.json":"./package.json","./convex.config":{"types":"./dist/component/convex.config.d.ts","default":"./dist/component/convex.config.js"},"./convex.config.js":{"types":"./dist/component/convex.config.d.ts","default":"./dist/component/convex.config.js"},"./_generated/component":{"types":"./dist/component/_generated/component.d.ts"},"./_generated/component.js":{"types":"./dist/component/_generated/component.d.ts"}},"gitHead":"d4056f1e70b6a459ed88df2bb97fa2016816a751","scripts":{"dev":"convex dev --start 'npm run dev:build'","lint":"eslint .","test":"vitest run --typecheck","alpha":"npm version prerelease --preid alpha && npm publish --tag alpha && git push --follow-tags","build":"tsc --project ./tsconfig.build.json","clean":"rm -rf dist *.tsbuildinfo","predev":"convex init && npm run build:codegen","verify":"npm run build && npm run test && npm run typecheck && npm run lint","prepare":"npm run build","release":"npm version patch && npm publish && git push --follow-tags","version":"(npm whoami || npm login) && vim -c 'normal o' -c 'normal o## '$npm_package_version CHANGELOG.md && prettier -w CHANGELOG.md && git add CHANGELOG.md","dev:mock":"node example/mock-firecrawl.mjs","dev:build":"chokidar 'tsconfig*.json' 'src/**/*.ts' -i '**/*.test.ts' -c 'npm run build:codegen' --initial","typecheck":"tsc --noEmit && tsc -p example && tsc -p example/convex","preversion":"npm ci && npm run build:clean && npm run test && npm run lint && npm run typecheck","test:watch":"vitest --typecheck --clearScreen false","build:clean":"npm run clean && npm run build:codegen","build:codegen":"npx convex codegen --component-dir ./src/component && npm run build","test:coverage":"vitest run --coverage --coverage.reporter=text","prepublishOnly":"npm run clean && npm run build"},"_npmUser":{"name":"rakramprakash","email":"rakshithramprakash@gmail.com"},"repository":{"url":"git+https://github.com/firecrawl/firecrawl-convex.git","type":"git"},"_npmVersion":"11.19.0","description":"Firecrawl component for Convex: scrape, map, and search the web, and run durable crawls with reactive progress.","directories":{},"_nodeVersion":"26.7.0","dependencies":{"convex-helpers":"^0.1.122"},"_hasShrinkwrap":false,"devDependencies":{"convex":"1.43.0","eslint":"9.39.4","vitest":"4.1.4","globals":"^17.5.0","prettier":"3.8.3","@eslint/js":"9.39.4","typescript":"6.0.3","@types/node":"^24.12.2","convex-test":"0.0.55","chokidar-cli":"3.0.0","@edge-runtime/vm":"^5.0.0","typescript-eslint":"8.58.2","@convex-dev/eslint-plugin":"^2.0.0"},"peerDependencies":{"convex":"^1.43.0"},"_npmOperationalInternal":{"tmp":"tmp/firecrawl-convex_0.1.1_1786082810725_0.13945286798156786","host":"s3://npm-registry-packages-npm-production"}}},"time":{"created":"2026-08-07T05:56:32.596Z","modified":"2026-08-13T05:17:41.543Z","0.1.0":"2026-08-07T05:56:32.895Z","0.1.1":"2026-08-07T06:06:50.857Z"},"bugs":{"url":"https://github.com/firecrawl/firecrawl-convex/issues"},"license":"MIT","homepage":"https://github.com/firecrawl/firecrawl-convex#readme","keywords":["convex","component","firecrawl","scraping","crawler","web-search","rag"],"repository":{"url":"git+https://github.com/firecrawl/firecrawl-convex.git","type":"git"},"description":"Firecrawl component for Convex: scrape, map, and search the web, and run durable crawls with reactive progress.","maintainers":[{"email":"abimex@gmail.com","name":"abimaelmartell"},{"email":"mo.geryy@gmail.com","name":"mogery"},{"email":"tom@sideguide.dev","name":"tomsideguide"},{"email":"gaurav@firecrawl.dev","name":"chadha93-firecrawl"},{"email":"rakshithramprakash@gmail.com","name":"rakramprakash"},{"email":"hello@sideguide.dev","name":"hello_sideguide"}],"readme":"# Firecrawl for Convex\n\n[![npm version](https://img.shields.io/npm/v/@firecrawl/firecrawl-convex.svg)](https://www.npmjs.com/package/@firecrawl/firecrawl-convex)\n\nScrape, map, and search the web from Convex functions, and run **durable crawls**\nwhose progress and pages live in your Convex database — so your UI subscribes to\na crawl instead of polling for it.\n\n```ts\nconst firecrawl = new FirecrawlClient(components.firecrawl);\n\n// One-shot\nconst page = await firecrawl.scrape(ctx, \"https://firecrawl.dev\", {\n  formats: [\"markdown\"],\n});\n\n// Durable: returns immediately, pages stream into your database\nconst { crawlId } = await firecrawl.startCrawl(ctx, {\n  url: \"https://docs.firecrawl.dev\",\n  options: { limit: 50 },\n  onComplete: internal.myModule.indexCrawledPages,\n});\n```\n\n## What you get\n\n| | |\n| --- | --- |\n| `scrape` | One URL → markdown, HTML, screenshot, summary, structured JSON |\n| `map` | Every URL on a site, fast |\n| `search` | Web search, optionally scraping each result |\n| `startCrawl` | A whole site, tracked in your database: reactive status, pages as they arrive, a completion callback |\n\nCrawls are the reason this is a component rather than a few `fetch` calls. A\ncrawl of a large site takes minutes and outlives any single action, so the\ncomponent owns a `crawls` row and a `pages` table, advances them from Firecrawl\nwebhooks (with a poll watchdog behind them), and exposes plain Convex queries.\nYour client gets live progress through the normal subscription mechanism.\n\n## Install\n\n```sh\nnpm install @firecrawl/firecrawl-convex\n```\n\nAdd the component to your app, wiring the API key through typed component env:\n\n```ts\n// convex/convex.config.ts\nimport { defineApp } from \"convex/server\";\nimport { v } from \"convex/values\";\nimport firecrawl from \"@firecrawl/firecrawl-convex/convex.config\";\n\nconst app = defineApp({\n  env: {\n    FIRECRAWL_API_KEY: v.string(),\n    FIRECRAWL_WEBHOOK_SECRET: v.optional(v.string()),\n  },\n});\n\napp.use(firecrawl, {\n  // Mounts the webhook route at <your-site>/firecrawl/webhook.\n  // Required for crawls in webhook mode.\n  httpPrefix: \"/firecrawl/\",\n  env: {\n    FIRECRAWL_API_KEY: app.env.FIRECRAWL_API_KEY,\n    FIRECRAWL_WEBHOOK_SECRET: app.env.FIRECRAWL_WEBHOOK_SECRET,\n  },\n});\n\nexport default app;\n```\n\nThen set the key on your deployment:\n\n```sh\nnpx convex env set FIRECRAWL_API_KEY fc-your-key\n# Recommended: from the Firecrawl dashboard → Advanced → webhook secret\nnpx convex env set FIRECRAWL_WEBHOOK_SECRET whsec-your-secret\nnpx convex dev\n```\n\nGet a key at [firecrawl.dev](https://firecrawl.dev).\n\n## Scrape, map, search\n\nCall the component from your own actions. Keeping a wrapper in your app is where\nauthentication, authorization, and rate limiting belong — components can't see\n`ctx.auth`. The [example app](example/convex/example.ts) shows the full pattern:\na `requireUser` gate on every paid endpoint, and an app-owned crawl → user table\nchecked before any crawl can be read, cancelled, or deleted.\n\n```ts\n// convex/web.ts\nimport { v } from \"convex/values\";\nimport { FirecrawlClient } from \"@firecrawl/firecrawl-convex\";\nimport { action } from \"./_generated/server\";\nimport { components } from \"./_generated/api\";\n\nconst firecrawl = new FirecrawlClient(components.firecrawl);\n\nexport const scrapePage = action({\n  args: { url: v.string() },\n  handler: async (ctx, args) => {\n    await requireUser(ctx);\n    return await firecrawl.scrape(ctx, args.url, {\n      formats: [\"markdown\", { type: \"json\", prompt: \"Extract the pricing table\" }],\n      onlyMainContent: true,\n      maxAge: 3_600_000, // reuse Firecrawl's cache for an hour\n    });\n  },\n});\n\nexport const siteUrls = action({\n  args: { url: v.string() },\n  handler: (ctx, args) => firecrawl.map(ctx, args.url, { limit: 500 }),\n});\n\nexport const searchWeb = action({\n  args: { query: v.string() },\n  handler: (ctx, args) =>\n    firecrawl.search(ctx, args.query, {\n      limit: 5,\n      scrapeOptions: { formats: [\"markdown\"] },\n    }),\n});\n```\n\nOption names match the [Firecrawl v2 API](https://docs.firecrawl.dev/api-reference/v2-introduction)\nand are passed through untouched, so the Firecrawl docs are the reference for\nwhat they do. The typed surface covers the common options; a few enterprise and\nniche ones (`profile`, `threatProtection`, `auditMetadata`, search `enterprise`)\nare deliberately left out of the types. Those, and anything Firecrawl ships\nbefore this package catches up, go through `extra`:\n\n```ts\nawait firecrawl.scrape(ctx, url, { extra: { threatProtection: { mode: \"off\" } } });\n```\n\n`scrape`, `map`, and `search` return the API response as-is (validated as\n`v.any()` at the component boundary) rather than a re-modelled shape, so a new\nresponse field is available the day Firecrawl ships it. The `FirecrawlClient`\nmethods give you TypeScript types over those responses.\n\n## Durable crawls\n\n```ts\nexport const crawlDocs = action({\n  args: { url: v.string() },\n  handler: async (ctx, args) => {\n    const userId = await requireUser(ctx);\n    return await firecrawl.startCrawl(ctx, {\n      url: args.url,\n      options: {\n        limit: 100,\n        includePaths: [\"^/docs/.*\"],\n        scrapeOptions: { formats: [\"markdown\"], onlyMainContent: true },\n      },\n      onComplete: internal.web.onCrawlComplete,\n      context: { userId },\n    });\n  },\n});\n```\n\n`startCrawl` returns `{ crawlId, jobId }` right away. From there:\n\n```ts\n// Live status: total, completed, pageCount, creditsUsed, error\nexport const crawlProgress = query({\n  args: { crawlId: v.string() },\n  handler: (ctx, args) => firecrawl.getCrawl(ctx, args.crawlId),\n});\n\n// Pages as they land — works with usePaginatedQuery\nexport const crawlPages = query({\n  args: { crawlId: v.string(), paginationOpts: paginationOptsValidator },\n  handler: (ctx, args) => firecrawl.listPages(ctx, args),\n});\n```\n\n```tsx\nfunction CrawlView({ crawlId }: { crawlId: string }) {\n  const crawl = useQuery(api.web.crawlProgress, { crawlId });\n  const { results } = usePaginatedQuery(\n    api.web.crawlPages,\n    { crawlId },\n    { initialNumItems: 25 },\n  );\n  return (\n    <>\n      <progress value={crawl?.pageCount ?? 0} max={crawl?.total ?? 1} />\n      <ul>{results.map((p) => <li key={p._id}>{p.url}</li>)}</ul>\n    </>\n  );\n}\n```\n\n### The completion callback\n\n`onComplete` takes an **internal mutation** of your app, run exactly once when\nthe crawl reaches a terminal state. `context` comes back untouched, so you can\ncarry a user id, a document id, whatever.\n\n```ts\nexport const onCrawlComplete = internalMutation({\n  args: {\n    crawlId: v.string(),\n    jobId: v.optional(v.string()),\n    status: v.union(v.literal(\"completed\"), v.literal(\"failed\"), v.literal(\"cancelled\")),\n    pageCount: v.number(),\n    unstored: v.optional(v.number()),\n    error: v.optional(v.string()),\n    context: v.optional(v.any()),\n  },\n  handler: async (ctx, args) => {\n    if (args.status !== \"completed\") return;\n    if (args.unstored) console.warn(`${args.unstored} pages were too large to store`);\n    // e.g. hand the pages to an embedding pipeline\n    await ctx.scheduler.runAfter(0, internal.rag.indexCrawl, {\n      crawlId: args.crawlId,\n      userId: args.context?.userId,\n    });\n  },\n});\n```\n\n### Webhook mode vs poll mode\n\n| | webhook (default) | poll |\n| --- | --- | --- |\n| How pages arrive | Firecrawl pushes `crawl.page` events; a slow watchdog poll catches anything dropped | the component polls the status endpoint, backing off to 30s |\n| Requires | `httpPrefix` mounted, and a deployment Firecrawl can reach over the internet | nothing |\n| Use it when | normal cloud deployments | local dev, self-hosted behind a firewall |\n\n```ts\nawait firecrawl.startCrawl(ctx, { url, mode: \"poll\" });\n```\n\nA local Convex deployment isn't reachable from Firecrawl's servers, so use\n`mode: \"poll\"` there — or the mock server described below, which delivers\nwebhooks to your local deployment for you.\n\nDeliveries are checked twice: the `X-Firecrawl-Signature` HMAC (whenever\n`FIRECRAWL_WEBHOOK_SECRET` is set) and a per-crawl token the component hands\nFirecrawl when it registers the webhook. A delivery failing either check is\nrejected with 401, and nothing is written.\n\n### Other crawl operations\n\n```ts\nawait firecrawl.getCrawlByJobId(ctx, jobId);       // look up by Firecrawl's id\nawait firecrawl.listCrawls(ctx, { status: \"scraping\", limit: 20 });\nawait firecrawl.getPage(ctx, { crawlId, url });\nawait firecrawl.cancelCrawl(ctx, crawlId);         // action\nawait firecrawl.deleteCrawl(ctx, crawlId);         // mutation: crawl + its pages\nawait firecrawl.resumeCrawl(ctx, crawlId);         // mutation: see below\n```\n\nThe component stops checking on a crawl after ~250 status checks (roughly two\nhours of polling, or a day of webhook watchdog) and finalizes it as `failed`\nwith an explanatory error, so subscribers and `onComplete` are never left\nwaiting on a job that will never report. If the job really is still running on\nFirecrawl, `resumeCrawl` picks tracking back up where it left off.\n\nPass `storeContent: false` to `startCrawl` to record only URLs and metadata —\nuseful when you just want the callback, or when you re-fetch content elsewhere.\n\n## Good to know\n\n- **Errors** are `ConvexError`s carrying `{ code, status, path, message }`, so\n  you can branch on `error.data.status === 402` (out of credits) or\n  `429` (rate limited). Transient failures (408, 425, 429, 5xx) are retried\n  three times with backoff, honoring `Retry-After`.\n- **Document limits.** Convex documents cap at 1MB, so every page is budgeted\n  in UTF-8 bytes across the whole document before it's written. Text and link\n  lists are truncated; a screenshot, extracted `json`, or `changeTracking` blob\n  that doesn't fit is dropped whole; oversized `metadata` falls back to its\n  essential keys. Any of that sets `truncated: true` on the page. If Firecrawl\n  returns pages that still can't be stored, the count shows up as `unstored` on\n  the crawl and in the `onComplete` payload — never silently. For very large\n  corpora, consider `storeContent: false` plus your own storage.\n- **Credits** show up as `creditsUsed` on the crawl row and in each page's\n  `metadata`.\n- **Self-hosted Firecrawl:** declare `FIRECRAWL_API_URL` in the component env\n  and point it at your instance.\n- **Runtime:** everything runs in the Convex runtime — no `\"use node\"`, no\n  bundled SDK. Requests go straight to the v2 REST API.\n\n## Testing\n\nRegister the component in your own tests:\n\n```ts\nimport { convexTest } from \"convex-test\";\nimport firecrawl from \"@firecrawl/firecrawl-convex/test\";\nimport schema from \"./schema\";\n\nconst modules = import.meta.glob(\"./**/*.*s\");\n\nexport function initConvexTest() {\n  process.env.FIRECRAWL_API_KEY = \"fc-test-key\";\n  const t = convexTest(schema, modules);\n  firecrawl.register(t);\n  return t;\n}\n```\n\nThen stub `fetch` to return canned Firecrawl responses — see\n[`example/convex/example.test.ts`](example/convex/example.test.ts).\n\n## Try it locally\n\nThe [`example/`](example) app exercises every entry point, and\n[`example/mock-firecrawl.mjs`](example/mock-firecrawl.mjs) stands in for the\nFirecrawl API — including signed webhook deliveries — so you can watch a crawl\nprogress without spending credits:\n\n```sh\nnpm install\n\n# terminal 1\nFIRECRAWL_WEBHOOK_SECRET=whsec-mock npm run dev:mock\n\n# terminal 2\nnpx convex env set FIRECRAWL_API_KEY fc-mock-key\nnpx convex env set FIRECRAWL_API_URL http://127.0.0.1:4242\nnpx convex env set FIRECRAWL_WEBHOOK_SECRET whsec-mock\nnpm run dev\n\n# terminal 3\nnpx convex env set DEMO_ALLOW_ANONYMOUS true   # local CLI demo only\nnpx convex run example:startCrawl '{\"url\":\"https://mock.test\",\"limit\":3}'\nnpx convex run example:myCrawls '{}'\nnpx convex run example:reports '{}'\n```\n\nSwap in a real key (and drop `FIRECRAWL_API_URL`) to hit the live API. See\n[`example/README.md`](example/README.md) for the full list of commands.\n\n## Development\n\n```sh\nnpm run dev         # component codegen + build watcher + convex dev\nnpm test            # vitest, including type tests\nnpm run typecheck\nnpm run lint\n```\n\n`npm run dev` runs the three steps the\n[authoring docs](https://docs.convex.dev/components/authoring) describe, in\norder: component codegen, package build, then `convex dev` for the example app.\n\n## License\n\nMIT\n","readmeFilename":"README.md"}