{"_id":"@aid-on/fuzztok","name":"@aid-on/fuzztok","dist-tags":{"latest":"1.0.0"},"versions":{"1.0.0":{"name":"@aid-on/fuzztok","version":"1.0.0","description":"高速・軽量なファジートークン推定ライブラリ - Fast and lightweight fuzzy token estimation library with CJK support","main":"dist/index.js","types":"dist/index.d.ts","module":"dist/index.mjs","exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.mjs","require":"./dist/index.js"}},"scripts":{"build":"tsup","dev":"tsup --watch","type-check":"tsc --noEmit","lint":"eslint src --ext .ts","test":"vitest","test:coverage":"vitest --coverage","prepublishOnly":"npm run build && npm run type-check"},"keywords":["token","estimation","llm","fuzzy","cjk","japanese","chinese","korean","multilingual","typescript"],"author":{"name":"aid-on"},"license":"MIT","devDependencies":{"@types/node":"^20.0.0","@typescript-eslint/eslint-plugin":"^6.0.0","@typescript-eslint/parser":"^6.0.0","eslint":"^8.0.0","tsup":"^8.0.0","typescript":"^5.0.0","vitest":"^1.0.0","@vitest/coverage-v8":"^1.0.0"},"engines":{"node":">=16.0.0"},"publishConfig":{"access":"public"},"repository":{"type":"git","url":"git+https://github.com/Aid-On/fuzztok.git"},"homepage":"https://github.com/Aid-On/fuzztok#readme","bugs":{"url":"https://github.com/Aid-On/fuzztok/issues"},"_id":"@aid-on/fuzztok@1.0.0","gitHead":"36ff1c564778a95c52a8ee889267ab1c5e1e7918","_nodeVersion":"20.10.0","_npmVersion":"10.2.3","dist":{"integrity":"sha512-tsAxcdRK22gZTANit95/hARbQug2QdkwvD2MvOIjRIL62xhrej8iAH43EG9g4l2UXQr/fqx4tSSYYHjgXde+Vw==","shasum":"e7a992bd79eccdc8b6a767a8346027bbc7746ab1","tarball":"https://registry.npmjs.org/@aid-on/fuzztok/-/fuzztok-1.0.0.tgz","fileCount":9,"unpackedSize":93162,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEYCIQCzHH1w7ZAvSTzjtByKqKF/a2CSTeOKd53qA6uofjySHgIhAJqDrqP/EtGu3hdPoUEbUAd3H0FWzsPfw2+rhhqw/tJr"}]},"_npmUser":{"name":"aid-on","email":"hiromi.motodera@aid-on.org"},"directories":{},"maintainers":[{"name":"aid-on","email":"hiromi.motodera@aid-on.org"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/fuzztok_1.0.0_1754058274328_0.060566093745909155"},"_hasShrinkwrap":false}},"time":{"created":"2025-08-01T14:24:34.262Z","1.0.0":"2025-08-01T14:24:34.506Z","modified":"2025-08-01T14:24:34.739Z"},"maintainers":[{"name":"aid-on","email":"hiromi.motodera@aid-on.org"}],"description":"高速・軽量なファジートークン推定ライブラリ - Fast and lightweight fuzzy token estimation library with CJK support","homepage":"https://github.com/Aid-On/fuzztok#readme","keywords":["token","estimation","llm","fuzzy","cjk","japanese","chinese","korean","multilingual","typescript"],"repository":{"type":"git","url":"git+https://github.com/Aid-On/fuzztok.git"},"author":{"name":"aid-on"},"bugs":{"url":"https://github.com/Aid-On/fuzztok/issues"},"license":"MIT","readme":"# @aid-on/fuzztok\n\nFast and lightweight fuzzy token estimation library with CJK support\n\n[日本語版 README](./README.ja.md)\n\n## Features\n\n- **🚀 High Performance**: Optimized for speed and low memory usage\n- **🌏 CJK Support**: Advanced support for Chinese, Japanese, and Korean text\n- **🔧 Flexible Architecture**: Dependency injection pattern for model configurations\n- **📊 Detailed Analysis**: Character type breakdown and composition analysis\n- **⚡ Batch Processing**: Support for batch estimation and streaming text\n- **💰 Cost Calculation**: Built-in token-to-cost conversion utilities\n- **🐛 Debug Tools**: Visualization tools for estimation breakdown\n\n## Installation\n\n```bash\nnpm install @aid-on/fuzztok\n```\n\n## Quick Start\n\n```javascript\nimport { createSimpleFuzzyEstimator } from '@aid-on/fuzztok';\n\n// Configure models\nconst modelConfigs = {\n  'gpt-3.5-turbo': {\n    charsPerToken: 4,\n    overhead: 10,\n    cjkTokensPerChar: 1.2,\n    mixedTextMultiplier: 1.05,\n    numberTokensPerChar: 3.5,\n    symbolTokensPerChar: 2.5,\n    whitespaceHandling: 'compress'\n  }\n};\n\n// Create estimator\nconst estimator = createSimpleFuzzyEstimator(modelConfigs, 'gpt-3.5-turbo');\n\n// Simple estimation\nconst tokens = estimator.estimate('Hello, world! こんにちは！');\nconsole.log(\\`Estimated tokens: \\${tokens}\\`);\n\n// Detailed estimation\nconst detailed = estimator.estimateDetailed('Hello, world! こんにちは！');\nconsole.log(detailed);\n```\n\n## API Reference\n\n### Core Classes\n\n#### `FuzzyTokenEstimator`\n\nMain estimation engine with dependency injection for model configurations.\n\n```typescript\nconstructor(\n  modelProvider: ModelConfigProvider,\n  options?: {\n    fallbackConfig?: FuzzyModelConfig;\n    defaultModel?: string;\n  }\n)\n```\n\n**Methods:**\n- `estimate(text: string, modelName?: string): number` - Simple token count\n- `estimateDetailed(text: string, modelName?: string): EstimationResult` - Detailed analysis\n- `estimatePayload(payload: TextPayload): number` - Estimate from text payload\n- `estimateBatch(texts: string[], modelName?: string): EstimationResult[]` - Batch processing\n\n#### `CharacterClassifier`\n\nUtility for character type detection and text analysis.\n\n```typescript\n// Static methods\nCharacterClassifier.isCJKCharacter(char: string): boolean\nCharacterClassifier.getCharacterType(char: string): CharacterType\nCharacterClassifier.analyzeTextComposition(text: string): TextComposition\n```\n\n### Configuration\n\n#### `FuzzyModelConfig`\n\n```typescript\ninterface FuzzyModelConfig extends BaseTokenConfig {\n  cjkTokensPerChar: number;           // CJK characters per token\n  mixedTextMultiplier: number;        // Mixed text adjustment factor\n  numberTokensPerChar?: number;       // Number tokenization rate\n  symbolTokensPerChar?: number;       // Symbol tokenization rate\n  whitespaceHandling?: 'ignore' | 'count' | 'compress';\n}\n```\n\n### Factory Functions\n\n```typescript\n// Using ModelConfigProvider\ncreateFuzzyEstimator(\n  modelProvider: ModelConfigProvider,\n  options?: ConfigOptions\n): FuzzyTokenEstimator\n\n// Using simple config object\ncreateSimpleFuzzyEstimator(\n  modelConfigs: Record<string, FuzzyModelConfig>,\n  defaultModel?: string\n): FuzzyTokenEstimator\n```\n\n## Advanced Usage\n\n### Custom Model Provider\n\n```javascript\nimport { FuzzyTokenEstimator } from '@aid-on/fuzztok';\n\nclass CustomModelProvider {\n  getConfig(modelName) {\n    // Fetch from database, API, etc.\n    return {\n      charsPerToken: 4,\n      overhead: 10,\n      cjkTokensPerChar: 1.2,\n      mixedTextMultiplier: 1.05\n    };\n  }\n  \n  getSupportedModels() {\n    return ['custom-model-1', 'custom-model-2'];\n  }\n}\n\nconst estimator = new FuzzyTokenEstimator(new CustomModelProvider());\n```\n\n### Cost Calculation\n\n```javascript\nimport { TokenCostCalculator } from '@aid-on/fuzztok';\n\nclass MyCostProvider {\n  getCost(model) {\n    return { input: 0.0015, output: 0.002 }; // per 1K tokens\n  }\n}\n\nconst calculator = new TokenCostCalculator(new MyCostProvider());\nconst cost = calculator.calculate('gpt-3.5-turbo', 1000, 500);\nconsole.log(cost.formattedTotal); // \"$2.25\"\n```\n\n### Streaming Support\n\n```javascript\nasync function* textStream() {\n  yield \"Hello \";\n  yield \"world \";\n  yield \"こんにちは！\";\n}\n\nfor await (const result of estimator.estimateStream(textStream())) {\n  console.log(\\`Chunk: \\${result.chunk}, Tokens: \\${result.tokens}, Total: \\${result.total}\\`);\n}\n```\n\n## CJK Support\n\nThis library provides comprehensive support for CJK text:\n\n- **Chinese**: Simplified and Traditional Chinese characters\n- **Japanese**: Hiragana, Katakana, and Kanji\n- **Korean**: Hangul syllables and compatibility characters\n- **Extended Unicode**: CJK Extension A-G, compatibility forms, and more\n\n## License\n\nMIT\n\n## Contributing\n\nIssues and pull requests are welcome on [GitHub](https://github.com/Aid-On/fuzztok).\n","readmeFilename":"README.md","_rev":"1-484df1ca8eccecca059c7e139c8e183c"}