{"_id":"persian-normalize","name":"persian-normalize","dist-tags":{"latest":"1.0.0"},"versions":{"1.0.0":{"name":"persian-normalize","version":"1.0.0","description":"Normalize Persian and Arabic-script text. Fold ZWNJ, Arabic yeh/kaf, three digit sets and diacritics into one comparable form — or clean text for storage without destroying it.","type":"module","main":"./src/index.js","module":"./src/index.js","types":"./src/index.d.ts","exports":{".":{"types":"./src/index.d.ts","import":"./src/index.js","default":"./src/index.js"}},"sideEffects":false,"engines":{"node":">=18"},"scripts":{"test":"node --test test/"},"keywords":["persian","farsi","arabic","normalize","normalization","zwnj","half-space","nim-fasele","rtl","unicode","text","search","nlp","tokenize","diacritics","persian-digits","arabic-digits","i18n"],"author":{"name":"Qatreh","email":"qatreh.ai@gmail.com","url":"https://qatrehai.ir"},"license":"MIT","homepage":"https://qatrehai.ir/blog/persian-chatbot-text-normalization-en","repository":{"type":"git","url":"git+https://github.com/qatrehai/persian-normalize.git"},"bugs":{"url":"https://github.com/qatrehai/persian-normalize/issues"},"_id":"persian-normalize@1.0.0","gitHead":"742efea004d8bf0fd1ff69191eb4e07a10a75647","_nodeVersion":"20.18.0","_npmVersion":"10.8.2","dist":{"integrity":"sha512-QM//RujW2hb6EPeMItFbg1f91T86J6+xKHDZ/bFTqPuYHi1feq/ImHAgmsIqKUOzUlCtprBYimVXUhd3PUzWzA==","shasum":"ddcef634085647ef341f7e491006122240b2820f","tarball":"https://registry.npmjs.org/persian-normalize/-/persian-normalize-1.0.0.tgz","fileCount":5,"unpackedSize":18560,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEQCIGuKNNceyC1t6LjmttehSXjBXvW3a3pghzhBl0N0+hK2AiBmZ7JqqbAQNQFLePFUoSS/McI0aXZ36FZWnziK/ZXOyA=="}]},"_npmUser":{"name":"qatrehai","email":"qatreh.ai@gmail.com"},"directories":{},"maintainers":[{"name":"qatrehai","email":"qatreh.ai@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/persian-normalize_1.0.0_1788881848256_0.9080961237427858"},"_hasShrinkwrap":false}},"time":{"created":"2026-09-08T15:37:27.850Z","1.0.0":"2026-09-08T15:37:28.392Z","modified":"2026-09-08T15:37:28.603Z"},"maintainers":[{"name":"qatrehai","email":"qatreh.ai@gmail.com"}],"description":"Normalize Persian and Arabic-script text. Fold ZWNJ, Arabic yeh/kaf, three digit sets and diacritics into one comparable form — or clean text for storage without destroying it.","homepage":"https://qatrehai.ir/blog/persian-chatbot-text-normalization-en","keywords":["persian","farsi","arabic","normalize","normalization","zwnj","half-space","nim-fasele","rtl","unicode","text","search","nlp","tokenize","diacritics","persian-digits","arabic-digits","i18n"],"repository":{"type":"git","url":"git+https://github.com/qatrehai/persian-normalize.git"},"author":{"name":"Qatreh","email":"qatreh.ai@gmail.com","url":"https://qatrehai.ir"},"bugs":{"url":"https://github.com/qatrehai/persian-normalize/issues"},"license":"MIT","readme":"# persian-normalize\n\nMake Persian and Arabic-script text comparable. Zero dependencies, ESM, TypeScript types included.\n\n```bash\nnpm install persian-normalize\n```\n\n## The problem\n\nIn Persian, one word can be written several ways that look identical to a reader and are completely different to a computer:\n\n```js\nconst a = \"چت‌بات\";   // with a zero-width non-joiner (U+200C)\nconst b = \"چت بات\";   // with an ordinary space\n\na === b                 // false\na.length === b.length   // true  ← the trap: length tells you nothing\n```\n\nNone of these variants is a spelling mistake. All appear in published Persian, and all are encoded distinctly in Unicode:\n\n| Variation | Example |\n|---|---|\n| Zero-width non-joiner | `می‌رود` · `می رود` · `میرود` |\n| Persian yeh vs Arabic yeh | `ی` (U+06CC) vs `ي` (U+064A) |\n| Persian keheh vs Arabic kaf | `ک` (U+06A9) vs `ك` (U+0643) |\n| Heh vs teh marbuta | `شرکه` vs `شرکة` |\n| Three digit sets | `۱۴۰۳` · `١٤٠٣` · `1403` |\n| Diacritics and tatweel | `مَدرسه` · `مدـــرسه` |\n\nA system that recognises one form of `چت‌بات` fails three users in four — and it fails **silently**, because nothing crashes. It just returns the wrong answer.\n\n## Two jobs, opposite settings\n\nThe library separates the two things people constantly conflate.\n\n### `cleanText` — for text you store or display\n\nRepairs what a wrong keyboard layout produced, drops decoration, tidies spacing. **Keeps the text readable.**\n\n```js\nimport { cleanText } from \"persian-normalize\";\n\ncleanText(\"مي‌كنم\")     // \"می‌کنم\"    Arabic yeh and kaf repaired\ncleanText(\"شركة\")       // \"شرکه\"      teh marbuta fixed\ncleanText(\"مَدرسه\")     // \"مدرسه\"     harakat dropped\ncleanText(\"١٤٠٣\")       // \"۱۴۰۳\"      Arabic-Indic digits to Persian\ncleanText(\"آموزش\")      // \"آموزش\"     the madda is correct Persian — untouched\n```\n\nThe ZWNJ survives, because deleting it turns `می‌رود` into `میرود`, which is a different and worse spelling.\n\n### `foldForSearch` — for text you compare\n\nFlattens every variant into one form. The output is for matching, not for reading.\n\n```js\nimport { foldForSearch, equals } from \"persian-normalize\";\n\nfoldForSearch(\"چت‌بات\")      // \"چت بات\"\nfoldForSearch(\"۱۴۰۳\")        // \"1403\"\nfoldForSearch(\"سلام، دنیا!\") // \"سلام دنیا\"\n\nequals(\"چت‌بات\", \"چت بات\")   // true\nequals(\"شرکة\", \"شرکه\")       // true\nequals(\"۱۴۰۳\", \"1403\")       // true\nequals(\"آموزش\", \"سازمان\")    // false — different words stay different\n```\n\n**The ZWNJ becomes a space, not nothing.** Delete it and `چت‌بات` becomes `چتبات`, which still does not equal `چت بات`. Turn it into a space and both spellings arrive at the same two tokens.\n\n## Matching without the substring trap\n\nThe other half of the problem. Persian attaches prefixes and suffixes freely, so a plain `includes()` finds words that are not there:\n\n```js\n\"آموزش سازمانی دارید؟\".includes(\"زمان\")   // true — «زمان» hides inside «سازمانی»\n```\n\nThat one line routes a question about *enterprise training* to a page about *project timelines*, and the answer reads perfectly well. Nobody notices.\n\n```js\nimport { containsWord, tokenize } from \"persian-normalize\";\n\ncontainsWord(\"آموزش سازمانی دارید؟\", \"زمان\")   // false ✓\ncontainsWord(\"دوره آموزشی ما\", \"آموزش\")        // true  — long enough to prefix-match\ncontainsWord(\"قیمت چت‌بات چقدر است\", \"چت بات\")  // true  — phrase, across spellings\n\ntokenize(\"چت‌بات سازمانی!\")                     // [\"چت\", \"بات\", \"سازمانی\"]\n```\n\nOnly the final token of a phrase may match by prefix, and only when it is at least `minPrefix` characters (default 5) — so `آموزش` still matches `آموزشی`, while `زمان` no longer matches `سازمانی`.\n\n## API\n\n| Function | Purpose |\n|---|---|\n| `cleanText(s, opts?)` | Repair for storage or display |\n| `foldForSearch(s, opts?)` | Flatten for comparison |\n| `equals(a, b)` | Compare ignoring spelling variation |\n| `tokenize(s)` | Folded tokens |\n| `containsWord(haystack, needle, opts?)` | Whole-word or phrase match |\n| `ZWNJ` | The U+200C character |\n\n```ts\ncleanText(input, {\n  digitsToLatin?: boolean,          // ۱۲۳ -> 123        default false\n  arabicDigitsToPersian?: boolean,  // ١٢٣ -> ۱۲۳        default true\n  diacritics?: boolean,             // strip harakat     default true\n  collapseSpaces?: boolean,         //                   default true\n})\n\nfoldForSearch(input, { keepPunctuation?: boolean })   // default false\ncontainsWord(haystack, needle, { minPrefix?: number }) // default 5\n```\n\n## Notes\n\n**This is not `String.prototype.normalize()`.** Unicode NFC/NFKC compose and decompose characters, but Persian yeh and Arabic yeh are *separate letters with separate meanings*, not two encodings of one character. Unicode will not merge them, and it should not. Script folding is an application-level decision.\n\n**Normalize your reference strings too**, at start-up. A keyword list typed by a developer on one layout and a query typed by a user on another will otherwise never meet, however good the folding is on the input side.\n\n**It is not only chatbots.** Anywhere Persian text is compared: product search, customer-name lookup, address matching, deduplication. In a database, two spellings of one company name remain two separate records.\n\n## Tests\n\nZero dependencies, including dev ones.\n\n```bash\nnode --test test/\n```\n\nEvery case came from a real failure in a production Persian assistant, which is why the negative assertions — the things that must *not* fold together — carry as much weight as the positive ones.\n\n## Background\n\nThe reasoning behind each step, and how the failures were found:\n[Why Your Persian Chatbot Answers the Wrong Question](https://qatrehai.ir/blog/persian-chatbot-text-normalization-en)\n\n## Licence\n\nMIT © [Qatreh](https://qatrehai.ir) — an AI team in Karaj, Iran.\n\nFound a variant this misses? Please open an issue — that is exactly the contribution this needs.\n","readmeFilename":"README.md","_rev":"1-8714d7cc27cd014398c9d29be8e7e769"}