{"_id":"@delali/narsil-embeddings-transformers","_rev":"4-30f201ef7b6159c3194046db0a76f82c","name":"@delali/narsil-embeddings-transformers","dist-tags":{"latest":"0.2.0"},"versions":{"0.1.5":{"name":"@delali/narsil-embeddings-transformers","version":"0.1.5","keywords":["narsil","embedding","transformers","transformers.js","vector-search","search-engine","huggingface"],"author":{"url":"https://sondelali.com","name":"Delali"},"license":"Apache-2.0","_id":"@delali/narsil-embeddings-transformers@0.1.5","maintainers":[{"name":"assetcorp","email":"delalify@gmail.com"}],"homepage":"https://github.com/assetcorp/narsil","bugs":{"url":"https://github.com/assetcorp/narsil/issues"},"dist":{"shasum":"6c6e4a939bfe337b2e9e031e950cfb0a311e824a","tarball":"https://registry.npmjs.org/@delali/narsil-embeddings-transformers/-/narsil-embeddings-transformers-0.1.5.tgz","fileCount":5,"integrity":"sha512-1hocTGazExK0uXDdMwJDCia6rq/02RvcC6MoIxpZ/ZYYJ5hcLm7c48PhhucVJZN6IgBESo8DgmU8fi3ClYqr5Q==","signatures":[{"sig":"MEYCIQCtlyQMXpOIxHTkvzYpM9FkFLKo7QbZc6SepfPDBPwSAwIhAMQRp23j+/vvN9oAlPWITBtyJyHKKGVSAkCPGw7RyVbu","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":26102},"type":"module","engines":{"node":">=22"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.mjs"}},"gitHead":"df97d8d139e08f42200ae86c6c5b83d135e8bbe3","scripts":{"lint":"biome check src/","test":"vitest run","build":"tsup","typecheck":"tsc --noEmit","test:watch":"vitest","test:coverage":"vitest run --coverage"},"_npmUser":{"name":"assetcorp","email":"delalify@gmail.com"},"repository":{"url":"git+https://github.com/assetcorp/narsil.git","type":"git","directory":"packages/embeddings-transformers"},"_npmVersion":"10.9.4","description":"Transformers.js embedding adapter for Narsil search engine","directories":{},"sideEffects":false,"_nodeVersion":"22.22.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"tsup":"8.5.1","vitest":"4.1.2","typescript":"5.9.3","@types/node":"22.15.18","@biomejs/biome":"2.4.4","@huggingface/transformers":"3.8.1"},"peerDependencies":{"@huggingface/transformers":">=3.0.0 <4.0.0"},"_npmOperationalInternal":{"tmp":"tmp/narsil-embeddings-transformers_0.1.5_1775573570424_0.5413022924982025","host":"s3://npm-registry-packages-npm-production"}},"0.1.6":{"name":"@delali/narsil-embeddings-transformers","version":"0.1.6","keywords":["narsil","embedding","transformers","transformers.js","vector-search","search-engine","huggingface"],"author":{"url":"https://sondelali.com","name":"Delali"},"license":"Apache-2.0","_id":"@delali/narsil-embeddings-transformers@0.1.6","maintainers":[{"name":"assetcorp","email":"delalify@gmail.com"}],"homepage":"https://github.com/assetcorp/narsil","bugs":{"url":"https://github.com/assetcorp/narsil/issues"},"dist":{"shasum":"c47468df4e125d39c25d0ab73f0ef4161398d049","tarball":"https://registry.npmjs.org/@delali/narsil-embeddings-transformers/-/narsil-embeddings-transformers-0.1.6.tgz","fileCount":5,"integrity":"sha512-N6L6JhCJsV30QzfZtio7BIljX/SOoj5G7Wb/2/5CyjO2rpGYneYgd69Dzjzk+cXbGuob5L73+9rtaClI7YyGhw==","signatures":[{"sig":"MEUCIBvhUHBQqrXuwRQ1hhTwGVM6ftKMDNyUS9SJv7MNsW0aAiEA2H5Thq22EyaSUYm9mpL1uRY1+H0ViQrQyjPKZGcSF6w=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@delali%2fnarsil-embeddings-transformers@0.1.6","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":26101},"type":"module","_from":"file:delali-narsil-embeddings-transformers-0.1.6.tgz","engines":{"node":">=22"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.mjs"}},"scripts":{"lint":"biome check src/","test":"vitest run","build":"tsup","typecheck":"tsc --noEmit","test:watch":"vitest","test:coverage":"vitest run --coverage"},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:bd00d837-852c-4d20-a1db-df21eeaf1cc1"}},"_resolved":"/tmp/ddd18c463f5d8021b29a2a96c873fb3c/delali-narsil-embeddings-transformers-0.1.6.tgz","_integrity":"sha512-N6L6JhCJsV30QzfZtio7BIljX/SOoj5G7Wb/2/5CyjO2rpGYneYgd69Dzjzk+cXbGuob5L73+9rtaClI7YyGhw==","repository":{"url":"git+https://github.com/assetcorp/narsil.git","type":"git","directory":"packages/embeddings-transformers"},"_npmVersion":"11.11.0","description":"Transformers.js embedding adapter for Narsil search engine","directories":{},"sideEffects":false,"_nodeVersion":"24.14.1","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"tsup":"8.5.1","vitest":"4.1.2","typescript":"5.9.3","@types/node":"22.15.18","@biomejs/biome":"2.4.4","@huggingface/transformers":"3.8.1"},"peerDependencies":{"@huggingface/transformers":">=3.0.0 <4.0.0"},"_npmOperationalInternal":{"tmp":"tmp/narsil-embeddings-transformers_0.1.6_1775573888053_0.8145503911828476","host":"s3://npm-registry-packages-npm-production"}},"0.1.7":{"name":"@delali/narsil-embeddings-transformers","version":"0.1.7","keywords":["narsil","embedding","transformers","transformers.js","vector-search","search-engine","huggingface"],"author":{"url":"https://sondelali.com","name":"Delali"},"license":"Apache-2.0","_id":"@delali/narsil-embeddings-transformers@0.1.7","maintainers":[{"name":"assetcorp","email":"delalify@gmail.com"}],"homepage":"https://github.com/assetcorp/narsil","bugs":{"url":"https://github.com/assetcorp/narsil/issues"},"dist":{"shasum":"4f74677a5f2194c4977d803751ea36d23076dd54","tarball":"https://registry.npmjs.org/@delali/narsil-embeddings-transformers/-/narsil-embeddings-transformers-0.1.7.tgz","fileCount":5,"integrity":"sha512-aovIA50EelGbFt1VatOyZo48r35EZBIMGEl1iD33Sijl4pT3gR+JsZR2Mp9iKrbGx0ONcwgPY9roo9ESybT4MQ==","signatures":[{"sig":"MEUCIHhV01XAbZ7//hOGtDUYt+e0xCdNU/JjtHGITC0WHM6eAiEA0LgpbiCvjWhjrsc8Ma3/U4nrD/ifC76oaEvUGyXJTJA=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@delali%2fnarsil-embeddings-transformers@0.1.7","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":26256},"type":"module","_from":"file:delali-narsil-embeddings-transformers-0.1.7.tgz","engines":{"node":">=22"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.mjs"}},"scripts":{"lint":"biome check src/","test":"vitest run","build":"tsup","typecheck":"pnpm run typecheck:src && pnpm run typecheck:tests","test:watch":"vitest","test:coverage":"vitest run --coverage","typecheck:src":"tsc --noEmit","typecheck:tests":"tsc --noEmit -p tsconfig.test.json"},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:bd00d837-852c-4d20-a1db-df21eeaf1cc1"}},"_resolved":"/tmp/26e563018d534e8acfe9b5a52f02b08d/delali-narsil-embeddings-transformers-0.1.7.tgz","_integrity":"sha512-aovIA50EelGbFt1VatOyZo48r35EZBIMGEl1iD33Sijl4pT3gR+JsZR2Mp9iKrbGx0ONcwgPY9roo9ESybT4MQ==","repository":{"url":"git+https://github.com/assetcorp/narsil.git","type":"git","directory":"packages/embeddings-transformers"},"_npmVersion":"11.17.0","description":"Transformers.js embedding adapter for Narsil search engine","directories":{},"sideEffects":false,"_nodeVersion":"24.19.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"tsup":"8.5.1","vitest":"4.1.2","typescript":"5.9.3","@types/node":"22.15.18","@biomejs/biome":"2.4.4","@huggingface/transformers":"3.8.1"},"peerDependencies":{"@huggingface/transformers":">=3.0.0 <4.0.0"},"_npmOperationalInternal":{"tmp":"tmp/narsil-embeddings-transformers_0.1.7_1788127992676_0.39093478809049476","host":"s3://npm-registry-packages-npm-production"}},"0.2.0":{"_id":"@delali/narsil-embeddings-transformers@0.2.0","bugs":{"url":"https://github.com/assetcorp/narsil/issues"},"dist":{"shasum":"1703e52c0087c84de440bf088ae856a45cb51531","tarball":"https://registry.npmjs.org/@delali/narsil-embeddings-transformers/-/narsil-embeddings-transformers-0.2.0.tgz","fileCount":5,"integrity":"sha512-E0HKVSPmrq1Wnp5ccpyFBcYOYfm52JydXzeRs4bn6TrK0CMFKBCyLqh/nwHV2UdKoURtFaGeZwBlLTjxOSEBeQ==","signatures":[{"sig":"MEYCIQD2OJdemQaWgvUKDCFN8DVbcFn3pHJz/JRWZjdbeBDuYQIhAIQC6R1yNjG//2dU1yfxLgm3TGlnlNU8xSAT/p/8FTZU","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"},{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEQCIBlpymVguDfNUzSgSijM91BBqKWrNX4anhbAbJbIKTEpAiAD65S8XiV56+qXv5StOXMs01snWjV6kJaEzLap2dx1MA=="}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@delali%2fnarsil-embeddings-transformers@0.2.0","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":27353},"name":"@delali/narsil-embeddings-transformers","type":"module","_from":"file:delali-narsil-embeddings-transformers-0.2.0.tgz","author":{"url":"https://sondelali.com","name":"Delali"},"engines":{"node":">=22"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.mjs"}},"license":"Apache-2.0","scripts":{"lint":"biome check src/","test":"vitest run","build":"tsup","typecheck":"pnpm run typecheck:src && pnpm run typecheck:tests","test:watch":"vitest","test:coverage":"vitest run --coverage","typecheck:src":"tsc --noEmit","typecheck:tests":"tsc --noEmit -p tsconfig.test.json"},"version":"0.2.0","_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:bd00d837-852c-4d20-a1db-df21eeaf1cc1"}},"homepage":"https://narsil.sondelali.com","keywords":["narsil","embedding","transformers","transformers.js","vector-search","search-engine","huggingface"],"_resolved":"/tmp/7204f14915b6a1f818e656de794697f3/delali-narsil-embeddings-transformers-0.2.0.tgz","_integrity":"sha512-E0HKVSPmrq1Wnp5ccpyFBcYOYfm52JydXzeRs4bn6TrK0CMFKBCyLqh/nwHV2UdKoURtFaGeZwBlLTjxOSEBeQ==","repository":{"url":"git+https://github.com/assetcorp/narsil.git","type":"git","directory":"packages/embeddings-transformers"},"_npmVersion":"11.19.0","description":"Transformers.js embedding adapter for Narsil search engine","directories":{},"maintainers":[{"name":"assetcorp","email":"delalify@gmail.com"}],"sideEffects":false,"_nodeVersion":"24.21.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"tsup":"8.5.1","vitest":"4.1.2","typescript":"5.9.3","@types/node":"22.15.18","@biomejs/biome":"2.4.4","@huggingface/transformers":"4.3.0"},"peerDependencies":{"@huggingface/transformers":">=3.0.0 <5.0.0"},"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/narsil-embeddings-transformers_0.2.0_1790544259200_0.16129491951195618"}}},"time":{"created":"2026-04-07T14:52:50.350Z","modified":"2026-09-27T21:24:19.771Z","0.1.5":"2026-04-07T14:52:50.584Z","0.1.6":"2026-04-07T14:58:08.191Z","0.1.7":"2026-08-30T22:13:12.809Z","0.2.0":"2026-09-27T21:24:19.276Z"},"bugs":{"url":"https://github.com/assetcorp/narsil/issues"},"author":{"url":"https://sondelali.com","name":"Delali"},"license":"Apache-2.0","homepage":"https://narsil.sondelali.com","keywords":["narsil","embedding","transformers","transformers.js","vector-search","search-engine","huggingface"],"repository":{"url":"git+https://github.com/assetcorp/narsil.git","type":"git","directory":"packages/embeddings-transformers"},"description":"Transformers.js embedding adapter for Narsil search engine","maintainers":[{"name":"assetcorp","email":"delalify@gmail.com"}],"readme":"# @delali/narsil-embeddings-transformers\n\nA [Transformers.js](https://huggingface.co/docs/transformers.js) embedding adapter for the [Narsil](https://github.com/assetcorp/narsil) search engine. The adapter computes embeddings in Node.js or in the browser through ONNX Runtime, so every text stays on the machine that embeds it. It implements Narsil's `EmbeddingAdapter` interface and accepts any Hugging Face model that the `feature-extraction` pipeline loads.\n\n## Installation\n\n```bash\npnpm add @delali/narsil-embeddings-transformers @huggingface/transformers\n```\n\n`@huggingface/transformers` is a peer dependency, so install it alongside this package. The adapter accepts any 3.x or 4.x release, which is the range `>=3.0.0 <5.0.0`.\n\n## Quick start\n\n```typescript\nimport { createTransformersEmbedding } from '@delali/narsil-embeddings-transformers'\n\nconst embedding = createTransformersEmbedding({\n  dimensions: 384,\n})\n\nconst vector = await embedding.embed('a red panda eating bamboo', 'document')\nconsole.log(vector.length) // 384\n```\n\nThe factory function returns a synchronous adapter object. The underlying model loads lazily on the first `embed()` or `embedBatch()` call, and subsequent calls reuse the same pipeline instance.\n\n## Choosing a model\n\nThe default model is `Xenova/all-MiniLM-L6-v2`, a 384-dimensional sentence embedding model that works well for general-purpose text similarity. Here are common alternatives:\n\n| Model | Dimensions | Size (q8) | Use case |\n| ----- | ---------- | --------- | ---------- |\n| `Xenova/all-MiniLM-L6-v2` | 384 | ~23 MB | General-purpose, good balance of speed and quality |\n| `Xenova/bge-base-en-v1.5` | 768 | ~65 MB | Higher quality English embeddings, requires prefix |\n| `Xenova/bge-small-en-v1.5` | 384 | ~23 MB | Smaller BGE variant for English |\n| `Xenova/multilingual-e5-small` | 384 | ~50 MB | Multilingual support, requires prefix |\n| `Xenova/gte-small` | 384 | ~23 MB | Strong general-purpose alternative |\n\nSet the `dimensions` config value to match the output dimensionality of your chosen model. If you set it incorrectly, the adapter will throw an error on the first embedding call.\n\n```typescript\nconst embedding = createTransformersEmbedding({\n  model: 'Xenova/bge-base-en-v1.5',\n  dimensions: 768,\n})\n```\n\n### Browser vs Node.js\n\nAll models work in both environments. In the browser, models are downloaded from the Hugging Face Hub and cached in the browser's Cache API. In Node.js, models are cached on disk at `~/.cache/huggingface/`. The first call triggers the download; subsequent calls load from cache.\n\n## Document and query prefixes\n\nSome models (BGE, E5, and instruction-tuned models) require specific text prefixes for documents and queries. Configure these with `documentPrefix` and `queryPrefix`:\n\n```typescript\nconst embedding = createTransformersEmbedding({\n  model: 'Xenova/bge-base-en-v1.5',\n  dimensions: 768,\n  documentPrefix: 'Represent this sentence: ',\n  queryPrefix: 'Represent this sentence for searching relevant passages: ',\n})\n```\n\nFor E5 models:\n\n```typescript\nconst embedding = createTransformersEmbedding({\n  model: 'Xenova/multilingual-e5-small',\n  dimensions: 384,\n  documentPrefix: 'passage: ',\n  queryPrefix: 'query: ',\n})\n```\n\nThe adapter prepends the appropriate prefix based on the `purpose` argument (`'document'` or `'query'`) passed to `embed()` and `embedBatch()`.\n\n## Device and quantization\n\nControl where inference runs and at what precision:\n\n```typescript\nconst embedding = createTransformersEmbedding({\n  dimensions: 384,\n  device: 'webgpu',  // 'wasm' | 'webgpu' | 'cpu'\n  dtype: 'fp32',     // 'fp32' | 'fp16' | 'q8' | 'q4'\n})\n```\n\n- **`device`**: Defaults to auto-detection by Transformers.js. Use `'webgpu'` for GPU acceleration in supported browsers. Use `'cpu'` for Node.js environments.\n- **`dtype`**: The adapter uses `'fp32'` by default, so a text gets the same vector from `embed` and from `embedBatch`. Under a quantised setting such as `'q8'` or `'q4'`, you download the smaller models in the table above and inference is faster, at a small cost in quality. Under a quantised setting the model computes one scale for the numbers of a whole padded batch, so `embedBatch` embeds each text on its own to return the vector that `embed` returns for it. A batch then takes as long as embedding the same texts one at a time.\n\n## Download progress\n\nTrack model download progress for a better loading experience:\n\n```typescript\nconst embedding = createTransformersEmbedding({\n  dimensions: 384,\n  progress: (data) => {\n    console.log('Download progress:', data)\n  },\n})\n```\n\nThe `progress` callback is passed through to the Transformers.js `pipeline()` function as `progress_callback`. The callback data includes status, file name, and download percentage when available.\n\n## Integration with Narsil\n\nThe adapter plugs into Narsil's embedding configuration for automatic vector generation on insert and text-based vector search on query:\n\n```typescript\nimport { createNarsil } from '@delali/narsil'\nimport { createTransformersEmbedding } from '@delali/narsil-embeddings-transformers'\n\nconst embeddingAdapter = createTransformersEmbedding({\n  dimensions: 384,\n})\n\nconst narsil = createNarsil({\n  embedding: embeddingAdapter,\n})\n\nconst index = await narsil.createIndex({\n  name: 'articles',\n  schema: {\n    title: 'string',\n    body: 'string',\n    titleVector: 'vector[384]',\n  },\n  embedding: {\n    fields: {\n      titleVector: ['title', 'body'],\n    },\n  },\n})\n\nawait index.insert({\n  title: 'Introduction to Vector Search',\n  body: 'Vector search finds similar items by comparing numerical representations...',\n})\n\nconst results = await index.query({\n  vector: {\n    field: 'titleVector',\n    text: 'how does semantic search work',\n    limit: 10,\n  },\n})\n```\n\nWhen you insert a document, Narsil automatically generates embeddings for the `titleVector` field by concatenating the `title` and `body` source fields and passing them through the adapter. When you query with `text` instead of a raw vector, Narsil embeds the query text using the same adapter with the `'query'` purpose.\n\n## Shutdown and cleanup\n\nRelease the ONNX session and free memory by calling `shutdown()`:\n\n```typescript\nawait embeddingAdapter.shutdown()\n```\n\nIf the adapter is passed to a Narsil instance, calling `narsil.shutdown()` will shut down the embedding adapter automatically.\n\n## API reference\n\n### `createTransformersEmbedding(config)`\n\nReturns an object conforming to Narsil's `EmbeddingAdapter` interface.\n\n### Config options\n\n| Option | Type | Default | Description |\n| ------ | ---- | ------- | ----------- |\n| `dimensions` | `number` | **(required)** | Output dimensionality of the model. Must match the model's actual output size. |\n| `model` | `string` | `'Xenova/all-MiniLM-L6-v2'` | Hugging Face model identifier for the `feature-extraction` pipeline. |\n| `dtype` | `string` | `'fp32'` | Model precision: `'fp32'`, `'fp16'`, `'q8'`, or `'q4'`. Under a quantised setting, `embedBatch` embeds each text on its own. |\n| `device` | `'wasm' \\| 'webgpu' \\| 'cpu'` | auto-detect | Inference backend. Omit to let Transformers.js pick the best available. |\n| `pooling` | `'mean' \\| 'cls'` | `'mean'` | Token pooling strategy for generating a single vector from token-level outputs. |\n| `normalize` | `boolean` | `true` | Whether to L2-normalize output vectors. |\n| `documentPrefix` | `string` | `''` | Text prepended to input when `purpose` is `'document'`. |\n| `queryPrefix` | `string` | `''` | Text prepended to input when `purpose` is `'query'`. |\n| `progress` | `(data: unknown) => void` | - | Callback for model download progress events. |\n| `pipelineOptions` | `Record<string, unknown>` | - | Additional options passed through to the Transformers.js `pipeline()` constructor. |\n\n### Returned adapter methods\n\n| Method | Signature | Description |\n| ------ | --------- | ----------- |\n| `embed` | `(input: string, purpose: 'document' \\| 'query', signal?: AbortSignal) => Promise<Float32Array>` | Embed a single string. |\n| `embedBatch` | `(inputs: string[], purpose: 'document' \\| 'query', signal?: AbortSignal) => Promise<Float32Array[]>` | Embed multiple strings in a single model forward pass. |\n| `dimensions` | `readonly number` | The configured output dimensionality. |\n| `shutdown` | `() => Promise<void>` | Release the model pipeline and free resources. |\n\n## License\n\nApache-2.0\n","readmeFilename":"README.md"}