{"_id":"llm-splitter","_rev":"8-41a64027443c74d5d502fadc16d50e68","name":"llm-splitter","dist-tags":{"latest":"0.3.0"},"versions":{"0.1.1":{"name":"llm-splitter","version":"0.1.1","keywords":["llm","splitter","chunking","text","vectorization"],"author":{"name":"Ryan Roemer","email":"ryan.roemer@nearform.com"},"license":"MIT","_id":"llm-splitter@0.1.1","maintainers":[{"name":"simoneb","email":"simone.busoli@gmail.com"}],"homepage":"https://github.com/nearform/llm-splitter#readme","bugs":{"url":"https://github.com/nearform/llm-splitter/issues"},"dist":{"shasum":"fe618c4360691153f803fcaf229f25ae74c24ef2","tarball":"https://registry.npmjs.org/llm-splitter/-/llm-splitter-0.1.1.tgz","fileCount":12,"integrity":"sha512-AFIX8SFR8JNt7qmA8PJvo0stnDQategbVtjkedf1jpsDr9Mr6EoBVEDEe48dSWCweZXOCzJJbzS07KJrQs4XJQ==","signatures":[{"sig":"MEYCIQChGNWiKMRp0kcRj6ssrudPLaq4ItSABwLr95zBRiktbwIhAMiAUqdGN9rs4IuGk8obawxnFJFAEJLz6LTjEcaiNqIT","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/llm-splitter@0.1.1","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":34953},"main":"dist/index.js","type":"module","types":"dist/index.d.ts","gitHead":"09dbe67dd9468d6dab5edf2ba8ddf6bb2127efee","private":false,"scripts":{"lint":"eslint .","test":"node --loader ts-node/esm/transpile-only --test","build":"tsc --project tsconfig.json","check":"npm run lint && npm run test && npm run check-format","format":"prettier --write . && eslint --cache --fix .","prepare":"husky","prebuild":"rm -rf dist","test:only":"npm run test -- --test-only","check-lint":"eslint .","check-format":"prettier --check ."},"_npmUser":{"name":"simoneb","email":"simone.busoli@gmail.com"},"repository":{"url":"git+https://github.com/nearform/llm-splitter.git","type":"git"},"_npmVersion":"10.8.2","description":"Efficient, configurable text chunking utility for LLM vectorization. Returns rich chunk metadata.","directories":{},"lint-staged":{"*.{js,jsx}":"eslint --cache --fix"},"_nodeVersion":"20.19.4","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"husky":"^9.1.7","eslint":"^9.30.1","globals":"^16.3.0","ts-node":"^10.9.2","prettier":"^3.6.2","tiktoken":"^1.0.21","@eslint/js":"^9.30.0","typescript":"^5.0.0","@types/node":"^22.16.5","lint-staged":"^16.1.0","@commitlint/cli":"^19.8.1","eslint-config-prettier":"^10.1.5","eslint-plugin-prettier":"^5.5.1","@commitlint/config-conventional":"^19.8.1"},"_npmOperationalInternal":{"tmp":"tmp/llm-splitter_0.1.1_1753430590506_0.42472719008276716","host":"s3://npm-registry-packages-npm-production"}},"0.2.0":{"name":"llm-splitter","version":"0.2.0","keywords":["llm","splitter","chunking","text","vectorization"],"author":{"name":"Ryan Roemer","email":"ryan.roemer@nearform.com"},"license":"MIT","_id":"llm-splitter@0.2.0","maintainers":[{"name":"simoneb","email":"simone.busoli@gmail.com"},{"name":"eamonn.frisby","email":"eamonn.frisby@nearform.com"},{"name":"donal.moynihan","email":"donal.moynihan@nearform.com"},{"name":"sociablesteve","email":"steve.goode@nearform.com"},{"name":"ryan.roemer","email":"ryan@loose-bits.com"},{"name":"elizibin","email":"elizibin@gmail.com"}],"homepage":"https://github.com/nearform/llm-splitter#readme","bugs":{"url":"https://github.com/nearform/llm-splitter/issues"},"dist":{"shasum":"714d8ee8a444d09431aab6d3db80f60c32e13fef","tarball":"https://registry.npmjs.org/llm-splitter/-/llm-splitter-0.2.0.tgz","fileCount":12,"integrity":"sha512-Yqi947Vk5Ps2YqhOV8K+RR6bseLhZLIVfovpWJH5cT7GE4Pca8/3iny/3oQ47scD7SfQd3whhxMfc+KgxTgDHA==","signatures":[{"sig":"MEYCIQDodQ0FSVHEAyyWqdKcK0s10wvdMT7RCPaDiSigM63hAgIhAOBNmVugsz1rx3G2QzSEqqKZPCTATvK1s54WMXft7HyY","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/llm-splitter@0.2.0","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":43366},"main":"dist/index.js","type":"module","types":"dist/index.d.ts","gitHead":"4f9e3600aed4de3ca63a8b6e48c1cdada870ac78","private":false,"scripts":{"lint":"eslint .","test":"node --loader ts-node/esm/transpile-only --test","build":"tsc --project tsconfig.json","check":"npm run lint && npm run test && npm run check-format","format":"prettier --write . && eslint --cache --fix .","prepare":"husky","prebuild":"rm -rf dist","test:only":"npm run test -- --test-only","check-lint":"eslint .","check-format":"prettier --check ."},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:dae0bca3-d788-4c88-926f-5e89b61ee335"}},"repository":{"url":"git+https://github.com/nearform/llm-splitter.git","type":"git"},"_npmVersion":"11.6.1","description":"Efficient, configurable text chunking utility for LLM vectorization. Returns rich chunk metadata.","directories":{},"lint-staged":{"*.{js,jsx}":"eslint --cache --fix"},"_nodeVersion":"22.20.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"husky":"^9.1.7","eslint":"^9.30.1","globals":"^16.3.0","ts-node":"^10.9.2","prettier":"^3.6.2","tiktoken":"^1.0.21","@eslint/js":"^9.30.0","typescript":"^5.0.0","@types/node":"^24.1.0","lint-staged":"^16.1.0","@commitlint/cli":"^20.0.0","typescript-eslint":"^8.45.0","eslint-config-prettier":"^10.1.5","eslint-plugin-prettier":"^5.5.1","@commitlint/config-conventional":"^20.0.0"},"_npmOperationalInternal":{"tmp":"tmp/llm-splitter_0.2.0_1759766413740_0.8483213977637292","host":"s3://npm-registry-packages-npm-production"}},"0.3.0":{"name":"llm-splitter","private":false,"version":"0.3.0","description":"Efficient, configurable text chunking utility for LLM vectorization. Returns rich chunk metadata.","license":"MIT","author":{"name":"Ryan Roemer","email":"ryan.roemer@nearform.com"},"repository":{"type":"git","url":"git+https://github.com/nearform/llm-splitter.git"},"homepage":"https://github.com/nearform/llm-splitter#readme","bugs":{"url":"https://github.com/nearform/llm-splitter/issues"},"keywords":["llm","splitter","chunking","text","vectorization"],"type":"module","main":"src/index.js","types":"dist/index.d.ts","scripts":{"prebuild":"rm -rf dist","build":"tsc --project tsconfig.json","prepack":"npm run build","check":"npm run check:lint && npm run check:types && npm run check:format && npm run test","check:format":"prettier --check .","check:lint":"eslint .","check:types":"tsc --project tsconfig.check.json","format":"prettier --write . && eslint --cache --fix .","test":"node --test \"test/*.test.js\"","test:only":"node --test --test-only \"test/*.test.js\"","dep:check":"npx npm-check-updates"},"devDependencies":{"@changesets/changelog-github":"^1.0.0","@changesets/cli":"^3.0.1","@eslint/js":"^10.0.1","@types/node":"^26.2.0","eslint":"^10.8.1","eslint-config-prettier":"^10.1.8","eslint-plugin-prettier":"^5.5.6","prettier":"^3.9.6","tiktoken":"^1.0.22","typescript":"^7.0.2"},"publishConfig":{"access":"public"},"_id":"llm-splitter@0.3.0","_integrity":"sha512-SXi9RUmxdo8b7e1Pt7w5mmwPn6LCJc6Bhtndap0aKKHQxbfImW2B3GL4VEhysO3p/+4BQ6CIvruVl8+zwOh2vw==","_resolved":"/home/runner/work/_temp/changeset-pack-9585353226-1787699742404/packages/llm-splitter-0.3.0.tgz","_from":"file:../../_temp/changeset-pack-9585353226-1787699742404/packages/llm-splitter-0.3.0.tgz","_nodeVersion":"24.19.0","_npmVersion":"12.0.2","dist":{"integrity":"sha512-SXi9RUmxdo8b7e1Pt7w5mmwPn6LCJc6Bhtndap0aKKHQxbfImW2B3GL4VEhysO3p/+4BQ6CIvruVl8+zwOh2vw==","shasum":"93d5ac8584d2ec9d0b7e74d58b5f88e853268a30","tarball":"https://registry.npmjs.org/llm-splitter/-/llm-splitter-0.3.0.tgz","fileCount":9,"unpackedSize":44783,"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/llm-splitter@0.3.0","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEQCIARCfHS5DAKX29Nq+jLctGFw9BymQlk2/vR9lRXpA6pCAiBx9jEBVYoSlCU5/Seovj06SbvZiElOabfwuZZShuiUEg=="}]},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:3310a6ff-a81b-4f91-b442-7bba24ce5dc8"}},"directories":{},"maintainers":[{"name":"simoneb","email":"simone.busoli@gmail.com"},{"name":"eamonn.frisby","email":"eamonn.frisby@nearform.com"},{"name":"donal.moynihan","email":"donal.moynihan@nearform.com"},{"name":"gbdev13","email":"contato@gbdev.me"},{"name":"sociablesteve","email":"steve.goode@nearform.com"},{"name":"ryan.roemer","email":"ryan@loose-bits.com"},{"name":"elizibin","email":"elizibin@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/llm-splitter_0.3.0_1787699748955_0.7514623027865721"},"_hasShrinkwrap":false}},"time":{"created":"2025-07-25T08:03:10.419Z","modified":"2026-08-25T23:15:49.473Z","0.1.1":"2025-07-25T08:03:10.675Z","0.2.0":"2025-10-06T16:00:13.942Z","0.3.0":"2026-08-25T23:15:49.092Z"},"bugs":{"url":"https://github.com/nearform/llm-splitter/issues"},"author":{"name":"Ryan Roemer","email":"ryan.roemer@nearform.com"},"license":"MIT","homepage":"https://github.com/nearform/llm-splitter#readme","keywords":["llm","splitter","chunking","text","vectorization"],"repository":{"type":"git","url":"git+https://github.com/nearform/llm-splitter.git"},"description":"Efficient, configurable text chunking utility for LLM vectorization. Returns rich chunk metadata.","maintainers":[{"name":"simoneb","email":"simone.busoli@gmail.com"},{"name":"eamonn.frisby","email":"eamonn.frisby@nearform.com"},{"name":"donal.moynihan","email":"donal.moynihan@nearform.com"},{"name":"gbdev13","email":"contato@gbdev.me"},{"name":"sociablesteve","email":"steve.goode@nearform.com"},{"name":"ryan.roemer","email":"ryan@loose-bits.com"},{"name":"elizibin","email":"elizibin@gmail.com"}],"readme":"# `llm-splitter`\n\n[![npm version](https://badgen.net/npm/v/llm-splitter?icon=npm)](https://www.npmjs.com/package/llm-splitter)\n[![GitHub release](https://badgen.net/github/release/nearform/llm-splitter?icon=github)](https://github.com/nearform/llm-splitter)\n[![GitHub CI](https://badgen.net/github/checks/nearform/llm-splitter?icon=github)](https://github.com/nearform/llm-splitter)\n\nA JavaScript library for splitting text into configurable chunks with overlap support.\n\n## Features\n\n- 📖 **Paragraph-aware**: fits whole paragraphs per chunk where the token budget allows\n- 🧠 **Any tokenizer**: bring your own splitter — `tiktoken`, words, sentences, characters\n- 📊 **Positions that add up**: every chunk carries `start`/`end`, and they cover the source\n  with no gaps, so you can store embeddings without storing text\n- 📝 **Typed**: authored in JS with JSDoc annotations; ships `.d.ts` for TypeScript consumers\n\n## Installation\n\n```sh\n$ npm install llm-splitter\n```\n\nRuns on any currently supported Node.js release and evergreen browsers.\n\n## Usage\n\n```js\nimport { split, getChunk } from \"llm-splitter\";\n```\n\nTypeScript consumers can also import the types. `Chunk` is the element type of the array\n`split()` returns; `SplitOptions` is its second argument, useful for typing a wrapper.\n\n```ts\nimport type { Chunk, SplitOptions } from \"llm-splitter\";\n```\n\n## API\n\n### `split(input, options)`\n\nSplits text into chunks based on a custom splitter function.\n\nEach chunk carries positional data (`start` and `end`) that can retrieve the chunk string\n(or array of strings) later via `getChunk()`. This is for the common scenario of storing\nembeddings for a chunk in a database (e.g. `pgvector`) without also storing the chunk text,\nyet still being able to recover it from the original input.\n\n#### Parameters\n\n- `input` (string|string[]) - The text or array of texts to split\n- `options` (object) - Configuration options\n  - `chunkSize` (number) - Maximum number of tokens per chunk (default: `512`)\n  - `chunkOverlap` (number) - Number of overlapping tokens between chunks (default: `0`)\n  - `chunkStrategy` (string) - Grouping preference for chunks (default: `\"character\"`)\n  - `splitter` (function) - Function to split text into tokens (default: character-by-character)\n\nNotes:\n\n- `input` must be a `string` or an array whose elements are all strings; anything else throws `TypeError`.\n- `chunkSize` must be a positive integer ≥ 1.\n- `chunkOverlap` must be a non-negative integer ≥ 0, and less than `chunkSize`.\n- `splitter` must return an array of strings. A non-array return throws `TypeError`; an element\n  that is neither a string nor an object throws plain `Error`, as do invalid options.\n- `splitter` functions may **omit** text but must not **mutate** it. Splitting on spaces is\n  fine (`(t) => t.split(\" \")`); uppercasing the results is not. A mutating splitter throws\n  when a token can't be located — but it can also anchor at a wrong position with no error,\n  so don't rely on it failing loudly (see \"Multibyte / Unicode Strings\").\n- Zero-length tokens are skipped: they anchor nowhere and don't count toward `chunkSize`.\n- Array element boundaries are always token boundaries — a token never spans two elements.\n- Input with no anchorable content yields no chunks: `split(\"\")` and `split([])` return `[]`,\n  as does a whitespace-only input under `chunkStrategy: \"paragraph\"`.\n- Sample `splitter` functions:\n  - Character: `text => text.split('')` (default)\n  - Word: `text => text.split(/\\s+/)`\n  - Sentence: `text => text.split(/[.!?]+/)`\n  - Line: `text => text.split(/\\n/)`\n\n#### Returns\n\nAn array of chunk objects:\n\n```js\n{\n  text: string | string[], // The chunk text\n  start: number,           // Start position in the original text\n  end: number              // End position in the original text\n}\n```\n\n`text` follows the input you passed: `split(str)` gives chunks whose `text` is a `string`,\n`split(arr)` gives chunks whose `text` is a `string[]`. TypeScript consumers get that\nnarrowing automatically; `getChunk` narrows the same way. Passing a value typed\n`string | string[]` still works and still returns the union.\n\n#### Examples\n\n**Chunk size and overlap:**\n\n```js\nconst text = \"Hello world! This is a test.\";\nconst chunks = split(text, {\n  chunkSize: 10,\n  chunkOverlap: 2,\n});\n\n// =>\n[\n  { text: \"Hello worl\", start: 0, end: 10 },\n  { text: \"rld! This \", start: 8, end: 18 },\n  { text: \"s is a tes\", start: 16, end: 26 },\n  { text: \"est.\", start: 24, end: 28 },\n];\n```\n\n**Word-based splitting:**\n\n```js\nconst text = \"Hello world! This is a test.\";\nconst chunks = split(text, {\n  chunkSize: 3,\n  chunkOverlap: 1,\n  splitter: (text) => text.split(/\\s+/),\n});\n\n// =>\n[\n  { text: \"Hello world! This\", start: 0, end: 17 },\n  { text: \"This is a\", start: 13, end: 22 },\n  { text: \"a test.\", start: 21, end: 28 },\n];\n```\n\n**Array of strings:**\n\n```js\nconst texts = [\"Hello world!\", \"This is a test.\"];\nconst chunks = split(texts, {\n  chunkSize: 5,\n  splitter: (text) => text.split(\" \"),\n});\n\n// =>\n[\n  { text: [\"Hello world!\", \"This is a \"], start: 0, end: 22 },\n  { text: [\"test.\"], start: 22, end: 27 },\n];\n```\n\n**Paragraph chunking**\n\nThe default `chunkStrategy: \"character\"` fits as many tokens as it can into each chunk.\n`chunkStrategy: \"paragraph\"` instead fits as many whole _paragraphs_ (delimited by `\\n\\n` or\na string array boundary) as it can. When the current chunk already holds a complete paragraph\nand the next one wouldn't fit, the chunk is emitted early so that paragraph can start a fresh\none — which tends to keep more context together for RAG and similar uses.\n\nWhole paragraphs are a _preference_, not a guarantee. A paragraph is still split across\nchunks when:\n\n- it has more tokens than `chunkSize` on its own, or\n- `chunkOverlap > 0` and tokens carried over from the previous chunk leave too little room.\n  Carried-over tokens don't count as a paragraph boundary. If keeping paragraphs whole\n  matters more than overlap context, use `chunkOverlap: 0`.\n\n<details>\n  <summary>See example...</summary>\n\n```js\n// Mix of paragraphs across array items and within items with `\\n\\n` marker.\nconst texts = [\n  \"Who has seen the wind?\\n\\nNeither I nor you.\",\n  \"But when the leaves hang trembling,\",\n  \"The wind is passing through.\",\n  \"Who has seen the wind?\\n\\nNeither you nor I.\",\n  \"But when the trees bow down their heads,\",\n  \"The wind is passing by.\",\n];\nconst chunks = split(texts, {\n  chunkSize: 20,\n  chunkOverlap: 2,\n  chunkStrategy: \"paragraph\",\n  splitter: (text) => text.split(/\\s+/),\n});\n\n// =>\n[\n  {\n    text: [\n      \"Who has seen the wind?\\n\\nNeither I nor you.\",\n      \"But when the leaves hang trembling,\",\n      \"The wind is passing through.\",\n    ],\n    start: 0,\n    end: 105,\n  },\n  {\n    text: [\n      \"passing through.\",\n      \"Who has seen the wind?\\n\\nNeither you nor I.\",\n      \"But when the trees bow down their heads,\",\n    ],\n    start: 89,\n    end: 187,\n  },\n  {\n    text: [\"their heads,\", \"The wind is passing by.\"],\n    start: 175,\n    end: 210,\n  },\n];\n```\n\n</details>\n\n### `getChunk(input, start, end)`\n\nExtracts a chunk of text from the original input by position. For array `input` the positions\nare treated as if all elements were concatenated into one long string, so the returned result\nis an array whose first and/or last element may be a substring of that item's text.\n\n#### Parameters\n\n- `input` (string|string[]) - The original input text or array of texts\n- `start` (number) - Start position in the original text\n- `end` (number) - End position in the original text\n\n#### Returns\n\n- `string` - For single string input\n- `string[]` - For array of strings input\n\nNotes:\n\n- Positions are clamped, not validated: a range outside the input returns `\"\"` (string) or\n  `[]` (array) rather than throwing, and `start`/`end` below `0` are treated as `0`. These are\n  _not_ `String.prototype.slice` semantics — `getChunk(\"hello\", 0, -2)` is `\"\"`, while\n  `\"hello\".slice(0, -2)` is `\"hel\"`.\n- Every element of an array `input` must be a string, whether or not it falls inside\n  `[start, end)`; a non-string element anywhere throws `TypeError`.\n\n#### Examples\n\n```js\nconst text = \"Hello world! This is a test.\";\nconst chunk = getChunk(text, 0, 12);\n// =>\n(\"Hello world!\");\n\nconst texts = [\"Hello world!\", \"This is a test.\"];\nconst chunk = getChunk(texts, 0, 16);\n// =>\n[\"Hello world!\", \"This\"];\n```\n\n## Advanced Usage\n\n### Custom Splitter Functions\n\nBeyond the one-liners listed under `split()`'s parameters, the interesting case is a real\ntokenizer.\n\n#### TikToken\n\nSplit using the TikToken tokenizer with the commonly used `text-embedding-ada-002` model.\n\n<details>\n  <summary>See example...</summary>\n\n```js\nimport tiktoken from \"tiktoken\";\n\n// Create a tokenizer for a specific model\nconst tokenizer = tiktoken.encoding_for_model(\"text-embedding-ada-002\");\nconst td = new TextDecoder();\n\n// Create a token splitter function\nconst tokenSplitter = (text) =>\n  Array.from(tokenizer.encode(text)).map((token) =>\n    td.decode(tokenizer.decode([token])),\n  );\n\nconst text = \"Hello world! This is a test.\";\nconst chunks = split(text, {\n  chunkSize: 3,\n  chunkOverlap: 1,\n  splitter: tokenSplitter,\n});\n\n// Don't forget to free the tokenizer when done\ntokenizer.free();\n\n// =>\n[\n  { text: \"Hello world!\", start: 0, end: 12 },\n  { text: \"! This is\", start: 11, end: 20 },\n  { text: \" is a test\", start: 17, end: 27 },\n  { text: \" test.\", start: 22, end: 28 },\n];\n```\n\n</details>\n\n### Chunk Coverage and Positions\n\n`start` and `end` index the source as one continuous run of UTF-16 code units. For an array\ninput that is the elements concatenated **with no separator**, so the total length is the sum\nof the element lengths, _not_ the array's own `length`.\n\nCoverage is lossless from `chunks[0].start` onward: every code unit in\n`[chunks[0].start, totalLength)` belongs to at least one chunk,\n`chunks[i].end >= chunks[i+1].start` for every adjacent pair (`>=` because `chunkOverlap` may\nmake them overlap), and the last chunk's `end` is exactly `totalLength`. So \"which chunk owns\nposition 12?\" always has an answer — which is the point, for RAG citations, highlighting, and\nre-chunking.\n\nWhat that costs you:\n\n- **Chunk ends may carry trailing whitespace.** Code units a splitter dropped are absorbed\n  into the _previous_ chunk by extending its `end`, so a chunk's `text` can end in `\"\\n\\n\"`.\n  Trim it if you don't want it — the reverse isn't possible without re-reading the source.\n- **Code units before `chunks[0].start` are uncovered.** Leading whitespace in paragraph mode\n  has no previous chunk to extend back into. This is the only gap.\n- **Offsets are code units, not characters.** A typical emoji occupies two, a CJK character\n  one, and a boundary can land inside a surrogate pair.\n\n### Multibyte / Unicode Strings\n\nTokenizers that split byte streams without regard to character boundaries are problematic for\nmultibyte text (as noted by\n[other text splitting libraries](https://js.langchain.com/docs/how_to/split_by_token/)). When\n`tiktoken` decodes a token straddling a multi-byte sequence, the result contains U+FFFD\nreplacement characters — and `llm-splitter` still maps that part back to a `start`/`end` in the\noriginal input, by searching the source for it.\n\nThat search is exact when a part's decoded length equals the source span it consumed:\n\n- ✅ **Byte-preserving splitters** — `text.split('')`, `text.split(/\\s+/)`, sentence and line\n  regexes, and `tiktoken` (cl100k, ada-002, gpt-4o), which substitutes exactly one U+FFFD per\n  undecodable byte.\n- ⚠️ **Tokenizers that normalize during decode** — `gte-small`, `bge-small`, and uncased\n  BERT-style WordPiece, typically loaded via `@huggingface/transformers`. Lowercasing, accent\n  stripping, and `##` prefixes make a decoded part longer than the span it consumed, so the\n  cursor overshoots and later parts throw or land in the wrong place. It's the _model_'s\n  tokenizer config that decides this, not the runtime.\n- ❌ **Mutating splitters** — rewriting token content is unsupported and can fail quietly.\n  `split()` throws when a part is nowhere in the source, but a lowercased `\"hi\"` will happily\n  anchor on some later `h` with no error.\n\nFor an affected tokenizer, chunk with a 1:1 tokenizer (tiktoken is a common choice) even if\nyour embedding model is from elsewhere. Failing that, apply the same normalization to the input\nand split the normalized text, accepting that positions then index that text rather than your\noriginal. Padding decoded parts back to source length is not enough — it repairs the cursor\narithmetic, not the mutation.\n\n#### Known limitations\n\nA search is inference, so even a ✅ splitter can anchor a part a code unit or two early — when\na multi-character delimiter it dropped contains a copy of the part that follows, or when your\nsource itself holds a literal U+FFFD, common in scraped and mojibake-recovered text. Those\ncharacters join the following chunk instead of the preceding one; coverage and\n`chunk.text === getChunk(input, start, end)` still hold, and `chunkOverlap` softens the effect.\nA part with nothing positionable in it at all — every code unit a U+FFFD or a combining mark —\nis dropped, its source absorbed into the neighboring chunk.\n\nSingle-character delimiters and character-class regexes (`/\\s+/`, `/[.!?]+/`) can't reach any of\nthis: a part never contains a character the splitter splits on. Neither can `tiktoken` or\n`text.split('')`, which drop nothing between parts.\n\n#### Token undercounting\n\nBecause unanchorable parts are dropped, a chunk may hold more semantic tokens than `chunkSize`\nspecifies. On 10MB of blog content with `tiktoken`, 99.6% of parts anchored on an exact match\nat the cursor. If your downstream has a hard token limit (an embedding API's max tokens, say),\napply a small `chunkSize` discount.\n\n#### Example\n\nEmoji, paragraph mode, and overlap together — note the leading `\\n` appears in no chunk,\nbecause paragraph mode strips leading whitespace and the first chunk has no previous chunk to\nextend back into.\n\n<details>\n  <summary>See example...</summary>\n\n```js\nconst text = `\nA noiseless 🤫 patient spider, 🕷️\nI mark'd where on a little 🏔️ promontory it stood isolated,\nMark'd how to explore 🔍 the vacant vast 🌌 surrounding,\n`;\n\nconst chunks = split(text, {\n  chunkSize: 15,\n  chunkOverlap: 2,\n  chunkStrategy: \"paragraph\",\n  splitter: tokenSplitter, // from examples above\n});\n\nconsole.log(JSON.stringify(chunks, null, 2));\n// =>\n[\n  {\n    text: \"A noiseless 🤫 patient spider, 🕷️\\nI mark'd where on\",\n    start: 1,\n    end: 53,\n  },\n  {\n    text: \" where on a little 🏔️ promontory it stood isolated,\\nMark'd\",\n    start: 44,\n    end: 103,\n  },\n  {\n    text: \"Mark'd how to explore 🔍 the vacant vast 🌌 surrounding,\\n\",\n    start: 97,\n    end: 154,\n  },\n];\n```\n\n</details>\n\n## License\n\nMIT\n","readmeFilename":"README.md"}