{"_id":"@ayushmanmishra/toon","_rev":"4-7a38d4416fecc3088c0427b281a62390","name":"@ayushmanmishra/toon","dist-tags":{"latest":"1.1.1"},"versions":{"1.0.0":{"name":"@ayushmanmishra/toon","version":"1.0.0","keywords":["toon","json","llm","token","optimization","compact","format","serialization","data-format","ai","openai","anthropic"],"author":{"url":"https://github.com/ayushmanmishra","name":"Ayushman Mishra"},"license":"MIT","_id":"@ayushmanmishra/toon@1.0.0","maintainers":[{"name":"ayushmanmishra","email":"ayushmanmishra094@gmail.com"}],"homepage":"https://github.com/ayushmanmishra/toon#readme","bugs":{"url":"https://github.com/ayushmanmishra/toon/issues"},"dist":{"shasum":"46ffd9aa66f9f6a2c7e2bc80ec826cba0983c5df","tarball":"https://registry.npmjs.org/@ayushmanmishra/toon/-/toon-1.0.0.tgz","fileCount":16,"integrity":"sha512-daetmH6UlQ7iIhsRpJo3b7cNNWK1GuP+t4VTL/riAhnVoFR71vp5grLj/2VltZzqSBXAjLVgCDHcNKHupTAwOQ==","signatures":[{"sig":"MEQCICGaS5Hlo6gvnTp+Pt8/0ZDc6VBvN69nkEiEPjn8TXl0AiAtRtwlxIdPICO5BMhyB9le82RWKJ5KWp+ZIDTqlaIMtw==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":48620},"main":"dist/index.js","types":"dist/index.d.ts","engines":{"node":">=14.0.0"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.js"}},"gitHead":"59e57e511a42a803908543bbcacefb0aa821e926","scripts":{"test":"jest","build":"tsc","compare":"ts-node scripts/token-comparison.ts","benchmark":"ts-node scripts/benchmark.ts","benchmark:all":"ts-node scripts/comprehensive-benchmark.ts","prepublishOnly":"npm run build && npm test"},"_npmUser":{"name":"ayushmanmishra","email":"ayushmanmishra094@gmail.com"},"repository":{"url":"git+https://github.com/ayushmanmishra/toon.git","type":"git"},"_npmVersion":"11.6.2","description":"Token-Oriented Object Notation - A compact format for LLM prompts with ~50% fewer tokens than JSON","directories":{},"_nodeVersion":"25.0.0","dependencies":{"js-tiktoken":"^1.0.21"},"_hasShrinkwrap":false,"devDependencies":{"jest":"^29.5.0","ts-jest":"^29.1.0","ts-node":"^10.9.2","typescript":"^5.0.0","@types/jest":"^29.5.0","@types/node":"^20.0.0"},"_npmOperationalInternal":{"tmp":"tmp/toon_1.0.0_1762635880183_0.8909961002208813","host":"s3://npm-registry-packages-npm-production"}},"1.0.1":{"name":"@ayushmanmishra/toon","version":"1.0.1","keywords":["toon","json","llm","token","optimization","compact","format","serialization","data-format","ai","openai","anthropic"],"author":{"url":"https://github.com/Ayushman88","name":"Ayushman Mishra"},"license":"MIT","_id":"@ayushmanmishra/toon@1.0.1","maintainers":[{"name":"ayushmanmishra","email":"ayushmanmishra094@gmail.com"}],"homepage":"https://github.com/Ayushman88/toon#readme","bugs":{"url":"https://github.com/Ayushman88/toon/issues"},"dist":{"shasum":"f9e3f81ca85606f8092b5eab5245e848c81d4239","tarball":"https://registry.npmjs.org/@ayushmanmishra/toon/-/toon-1.0.1.tgz","fileCount":16,"integrity":"sha512-nbyoZxv24N599WrBnqYJOc58LVWSakAtBnGrAyVqHrUQPNL2blZECu7pupdTVw+w4dFRZ+aC37tzWvFEhPokmA==","signatures":[{"sig":"MEUCIQC6sHFzPCYMOfdGuPM6y9V/sG9yCs9aaAXYMdT+CfX7LAIgLXGgZj5m8Y3CQyU+oibUeZ4i2PHGXb0RwJgLa8wrX0E=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":48877},"main":"dist/index.js","types":"dist/index.d.ts","engines":{"node":">=14.0.0"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.js"}},"gitHead":"a4a30f970a18ffb4ac8a5ae2bfcea5e218f418f2","scripts":{"test":"jest","build":"tsc","compare":"ts-node scripts/token-comparison.ts","benchmark":"ts-node scripts/benchmark.ts","benchmark:all":"ts-node scripts/comprehensive-benchmark.ts","prepublishOnly":"npm run build && npm test"},"_npmUser":{"name":"ayushmanmishra","email":"ayushmanmishra094@gmail.com"},"repository":{"url":"git+https://github.com/Ayushman88/toon.git","type":"git"},"_npmVersion":"11.6.2","description":"Token-Oriented Object Notation - A compact format for LLM prompts with ~50% fewer tokens than JSON","directories":{},"_nodeVersion":"25.0.0","dependencies":{"js-tiktoken":"^1.0.21"},"_hasShrinkwrap":false,"devDependencies":{"jest":"^29.5.0","ts-jest":"^29.1.0","ts-node":"^10.9.2","typescript":"^5.0.0","@types/jest":"^29.5.0","@types/node":"^20.0.0"},"_npmOperationalInternal":{"tmp":"tmp/toon_1.0.1_1762639105339_0.01566994170124869","host":"s3://npm-registry-packages-npm-production"}},"1.1.0":{"name":"@ayushmanmishra/toon","version":"1.1.0","keywords":["toon","json","llm","token","optimization","compact","format","serialization","data-format","ai","openai","anthropic"],"author":{"url":"https://github.com/Ayushman88","name":"Ayushman Mishra"},"license":"MIT","_id":"@ayushmanmishra/toon@1.1.0","maintainers":[{"name":"ayushmanmishra","email":"ayushmanmishra094@gmail.com"}],"homepage":"https://github.com/Ayushman88/toon#readme","bugs":{"url":"https://github.com/Ayushman88/toon/issues"},"dist":{"shasum":"a3c41b86f778168a8fb6c287f11643f46053234c","tarball":"https://registry.npmjs.org/@ayushmanmishra/toon/-/toon-1.1.0.tgz","fileCount":16,"integrity":"sha512-865HoFqDvdJ2qCWagl08+sJvjopbRaZayOcvWXDjC49kQU9Ubf7T4xJbcj2sDplydKYAiM/1QBK9Q5EAI8UWYQ==","signatures":[{"sig":"MEUCIASTUAPAY6yqmOS4NZkrEBxtO7neIOukU612LMknV6lyAiEAltMjp/yMcqVCwY3OyQFmk7iHCirJn6vU2ESSkcZAryY=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":58646},"main":"dist/index.js","types":"dist/index.d.ts","engines":{"node":">=14.0.0"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js","require":"./dist/index.js"}},"gitHead":"d791de42e2d3c34c9a7b5fb8fa460e4e78b8b7f1","scripts":{"test":"jest","build":"tsc","compare":"ts-node scripts/token-comparison.ts","benchmark":"ts-node scripts/benchmark.ts","benchmark:all":"ts-node scripts/comprehensive-benchmark.ts","prepublishOnly":"npm run build && npm test"},"_npmUser":{"name":"ayushmanmishra","email":"ayushmanmishra094@gmail.com"},"repository":{"url":"git+https://github.com/Ayushman88/toon.git","type":"git"},"_npmVersion":"11.6.2","description":"Token-Oriented Object Notation - A compact format for LLM prompts with ~50% fewer tokens than JSON","directories":{},"_nodeVersion":"25.0.0","dependencies":{"js-tiktoken":"^1.0.21"},"_hasShrinkwrap":false,"devDependencies":{"jest":"^29.5.0","ts-jest":"^29.1.0","ts-node":"^10.9.2","typescript":"^5.0.0","@types/jest":"^29.5.0","@types/node":"^20.0.0"},"_npmOperationalInternal":{"tmp":"tmp/toon_1.1.0_1762659490081_0.9818365718844364","host":"s3://npm-registry-packages-npm-production"}},"1.1.1":{"name":"@ayushmanmishra/toon","version":"1.1.1","description":"Token-Oriented Object Notation - A compact format for LLM prompts with ~50% fewer tokens than JSON. Optimized for OpenAI, Anthropic, and other LLM APIs.","main":"dist/index.js","types":"dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","require":"./dist/index.js","import":"./dist/index.js"}},"scripts":{"build":"tsc","test":"jest","prepublishOnly":"npm run build && npm test","compare":"ts-node scripts/token-comparison.ts","benchmark":"ts-node scripts/benchmark.ts","benchmark:all":"ts-node scripts/comprehensive-benchmark.ts"},"keywords":["toon","json","llm","token","optimization","compact","format","serialization","data-format","ai","openai","anthropic"],"author":{"name":"Ayushman Mishra","url":"https://github.com/Ayushman88"},"license":"MIT","repository":{"type":"git","url":"git+https://github.com/Ayushman88/toon.git"},"bugs":{"url":"https://github.com/Ayushman88/toon/issues"},"homepage":"https://github.com/Ayushman88/toon#readme","engines":{"node":">=14.0.0"},"devDependencies":{"@types/jest":"^29.5.0","@types/node":"^20.0.0","jest":"^29.5.0","ts-jest":"^29.1.0","ts-node":"^10.9.2","typescript":"^5.0.0"},"dependencies":{"js-tiktoken":"^1.0.21"},"gitHead":"ad63fb0ed3f40d01a0253e8708391902a7fd9ac7","_id":"@ayushmanmishra/toon@1.1.1","_nodeVersion":"25.0.0","_npmVersion":"11.6.2","dist":{"integrity":"sha512-q3bIX5f2A0m/MUGUKt83jCvB4hJqzgf/Os5N7NydfwOXMsU1sDNsDzIE4PBpO+bJh5IDwXZo4tf3/vIXO1SRzQ==","shasum":"c446182631cab66b499f560c07a8e18be353dacb","tarball":"https://registry.npmjs.org/@ayushmanmishra/toon/-/toon-1.1.1.tgz","fileCount":20,"unpackedSize":73821,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEYCIQCUEovhfpq9JKMld32JkzA1eKJQM6aWauBcWolOoFHKuwIhAPjMpEWaCLxJsk4B1/+B15dqIYTTubCKw420e57yse5q"}]},"_npmUser":{"name":"ayushmanmishra","email":"ayushmanmishra094@gmail.com"},"directories":{},"maintainers":[{"name":"ayushmanmishra","email":"ayushmanmishra094@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/toon_1.1.1_1762915334351_0.9051989836592087"},"_hasShrinkwrap":false}},"time":{"created":"2025-11-08T21:04:40.122Z","modified":"2025-11-12T02:42:14.717Z","1.0.0":"2025-11-08T21:04:40.366Z","1.0.1":"2025-11-08T21:58:25.543Z","1.1.0":"2025-11-09T03:38:10.284Z","1.1.1":"2025-11-12T02:42:14.552Z"},"bugs":{"url":"https://github.com/Ayushman88/toon/issues"},"author":{"name":"Ayushman Mishra","url":"https://github.com/Ayushman88"},"license":"MIT","homepage":"https://github.com/Ayushman88/toon#readme","keywords":["toon","json","llm","token","optimization","compact","format","serialization","data-format","ai","openai","anthropic"],"repository":{"type":"git","url":"git+https://github.com/Ayushman88/toon.git"},"description":"Token-Oriented Object Notation - A compact format for LLM prompts with ~50% fewer tokens than JSON. Optimized for OpenAI, Anthropic, and other LLM APIs.","maintainers":[{"name":"ayushmanmishra","email":"ayushmanmishra094@gmail.com"}],"readme":"# TOON - Token-Oriented Object Notation\n\n\n\n**A compact, human-readable format designed for passing structured data to Large Language Models (LLMs)**\n\n[](https://www.npmjs.com/package/@ayushmanmishra/toon)\n[](https://opensource.org/licenses/MIT)\n\n**21.0% fewer tokens than JSON compact • 47.2% fewer than JSON • Best-in-class performance on complex nested data**\n\n[Quick Start](#-quick-start) • [Documentation](#-documentation) • [Benchmarks](#-performance-benchmarks) • [Specification](./spec/TOON.md)\n\n
\n\n---\n\n> **Note**: This is an independent implementation of a TOON-like format. There is also an [official TOON format](https://github.com/toon-format/toon) with a different specification. This package (`@ayushmanmishra/toon`) uses a different syntax optimized for different use cases.\n\n## 🆕 What's New in v1.1.0+\n\n### v1.1.1 (Latest)\n\n- 📝 Enhanced documentation with comprehensive \"What's New\" section\n- 📊 Updated benchmark numbers with verified results\n- 📖 Updated specification with tabular format and presets\n- 🔄 Package name reverted to `@ayushmanmishra/toon`\n\n### v1.1.0\n\n### Major Features Added\n\n#### 🎯 Semantic Headers for LLM Context\n\nTOON now includes semantic headers in tabular format that provide essential context for LLMs:\n\n```\nusers[2]{id,name,role}:\n1 Alice admin\n2 Bob user\n```\n\n**Benefits:**\n\n- **Better LLM Understanding**: The header `users[2]{id,name,role}:` tells LLMs exactly what the data represents\n- **Context-Aware Parsing**: LLMs know the data type, count, and available fields before processing\n- **Minimal Token Overhead**: Only adds ~1.2% tokens while significantly improving LLM comprehension\n\n#### ⚙️ Preset Configurations\n\nFour ready-to-use presets optimized for different scenarios:\n\n- **`forLLM`** (Recommended): Maximum token efficiency with semantic headers\n - Compact booleans (`1`/`0`), compact null (`~`)\n - Tab delimiters for optimal tokenization\n - Semantic headers enabled\n- **`forLLMNested`**: Best for complex nested structures\n\n - Same as `forLLM` plus automatic flattening\n - **29.6% better than JSON compact** on nested data\n - Shortens keys intelligently (e.g., `customer.name` → `c_n`)\n\n- **`forDebugging`**: Human-readable output\n\n - Standard booleans and null values\n - Spaces added for readability\n - Perfect for development and debugging\n\n- **`forCompatibility`**: JSON-like balance\n - Standard boolean/null representations\n - Comma delimiters\n - Good for compatibility-focused use cases\n\n#### 📊 Enhanced Tabular Format\n\n- **Automatic Detection**: Uniform arrays of objects automatically use tabular format\n- **Tab Delimiters**: Uses `\\t` for optimal tokenization (single token per delimiter)\n- **Smart Quoting**: With tabs, strings with spaces can remain unquoted (saves many tokens)\n- **Efficient Encoding**: Eliminates redundant key names in data rows\n\n#### 🔄 Flattening for Nested Data\n\nNew `flatten` option converts nested structures into efficient tabular format:\n\n```typescript\n// Before: Nested structure\n{ orders: [{ id: 1, customer: { name: \"Alice\" } }] }\n\n// After: Flattened with shortened keys\norders[1]{id,c_n}:\n1 Alice\n```\n\n**Performance**: Achieves **29.6% better than JSON compact** on nested data while maintaining semantic context.\n\n#### 🎨 Token Optimization Improvements\n\n- **Tab Delimiter Support**: `\\t`, `,`, or `|` delimiters (tabs tokenize best)\n- **Key Shortening**: Context-aware key shortening for flattened structures\n- **Aggressive Quoting**: Only quotes when absolutely necessary\n- **Smart String Handling**: Unquoted strings with spaces when using tabs\n\n### Performance Improvements\n\n- **21.0% better than JSON compact** (verified across 5 real-world datasets)\n- **47.2% better than JSON**\n- **35.1% better than YAML**\n- **42.3% better than XML**\n- **TOON flattened**: 29.6% better than JSON compact on nested data\n\n### Breaking Changes\n\n**None!** All changes are backward compatible. Existing code continues to work, and new features are opt-in via presets or options.\n\n### Migration Guide\n\n**No migration needed!** Your existing code works as-is. To take advantage of new features:\n\n```typescript\n// Old way (still works)\nimport { encode } from \"@ayushmanmishra/toon\";\nconst toon = encode(data);\n\n// New way (recommended)\nimport { encode, forLLM } from \"@ayushmanmishra/toon\";\nconst toon = encode(data, forLLM); // Better LLM context\n```\n\n---\n\n## 🎯 What is TOON?\n\n**TOON (Token-Oriented Object Notation)** is a compact data serialization format specifically engineered to minimize token usage when passing structured data to Large Language Models. By eliminating redundant syntax, using explicit counts, and leveraging LLM context understanding, TOON achieves significant token savings while maintaining human readability.\n\n### Key Advantages\n\n- **🏆 Best-in-Class Performance**: Outperforms JSON, JSON compact, YAML, and XML on complex nested data\n- **⚡ 21.0% Token Reduction**: Fewer tokens than JSON compact, 47.2% fewer than JSON\n- **📖 Human Readable**: Easy to debug and verify, unlike binary formats\n- **🤖 LLM Optimized**: Designed specifically for LLM input, leveraging context understanding\n- **🌳 Nested Structure Support**: Handles complex hierarchies that CSV cannot represent\n- **🎯 Semantic Headers**: Provides context about data structure for better LLM understanding\n- **⚙️ Preset Configurations**: Ready-to-use presets for different use cases (`forLLM`, `forLLMNested`, `forDebugging`)\n- **📊 Tabular Format**: Automatic tabular encoding for uniform arrays with optimal tokenization\n\n---\n\n## 📊 Performance Benchmarks\n\nComprehensive testing across **5 real-world datasets** demonstrates TOON's superior performance:\n\n### Overall Performance Summary\n\n| Format | Total Tokens | vs JSON Compact | vs JSON | vs TOON |\n| -------------- | ------------ | --------------- | ------------- | -------- |\n| **TOON** | **17,482** | **-21.0%** ✅ | **-47.2%** ✅ | Baseline |\n| TOON flattened | 15,570 | -29.6% ✅ | -53.0% ✅ | -11.0% |\n| JSON Compact | 22,125 | Baseline | -33.2% | +26.5% |\n| YAML | 26,940 | +21.8% | -18.7% | +54.1% |\n| XML | 30,298 | +36.9% | -8.6% | +73.3% |\n| JSON | 33,140 | +49.8% | Baseline | +89.6% |\n\n**Result**: TOON is the **best structured format overall**, beating JSON by 47.2%, JSON compact by 21.0%, YAML by 35.1%, and XML by 42.3%. TOON flattened provides even better performance (29.6% better than JSON compact) for nested data.\n\n### Detailed Dataset Results\n\n#### ✅ TOON Wins (3 of 5 datasets)\n\n1. **GitHub Repositories** — **8,555 tokens** (TOON is best)\n\n - Beats all formats including CSV\n - Complex nested structures showcase TOON's strength\n - Repositories with metadata, nested objects, and arrays\n\n2. **Uniform Employee Records** — **1,213 tokens** (TOON is best)\n\n - Only 0.4% more than CSV (1,208 tokens)\n - 70.5% better than JSON\n - 47.4% better than JSON compact\n - Efficient handling of tabular data with metadata\n\n3. **Deeply Nested Configuration** — **73 tokens** (TOON is best)\n - Beats all formats\n - Minimal overhead for nested object structures\n - Perfect for configuration files and hierarchical data\n\n#### ⚠️ Where CSV Wins (2 of 5 datasets)\n\n4. **E-commerce Orders** — CSV: 1,191 tokens vs TOON flattened: 2,939 tokens\n\n - CSV wins on pure flat tabular structure\n - TOON flattened is 62.3% better than JSON\n - TOON flattened is 33.8% better than JSON compact\n - **Note**: CSV cannot represent nested structures that TOON handles efficiently\n\n5. **Event Logs** — CSV: 2,659 tokens vs TOON flattened: 2,687 tokens\n - TOON flattened only 1.1% more than CSV\n - TOON flattened is 55.7% better than JSON\n - **Note**: CSV cannot represent optional nested metadata\n\n### Key Insights\n\n**TOON excels at complex, nested data structures** where CSV cannot compete. CSV only wins on pure flat tabular data with zero structure overhead, but cannot handle nested structures that TOON represents efficiently.\n\n**The Challenge**: To beat CSV on e-commerce orders, we would need to flatten nested structures, which would lose information or require a different representation. TOON's advantage is handling nested/complex structures that CSV cannot.\n\n---\n\n## 🚀 Quick Start\n\n### Installation\n\n```bash\nnpm install @ayushmanmishra/toon\n```\n\n### Basic Usage\n\n```typescript\nimport { encode, forLLM } from \"@ayushmanmishra/toon\";\n\n// Simple array\nconst data = { tags: [\"jazz\", \"chill\", \"lofi\"] };\nconst toon = encode(data);\n// Result: tags[3]: jazz,chill,lofi\n\n// Optimized for LLM prompts (recommended)\nconst users = {\n users: [\n { id: 1, name: \"Alice\", role: \"admin\" },\n { id: 2, name: \"Bob\", role: \"user\" },\n ],\n};\nconst toon = encode(users, forLLM);\n// Result: users[2]{id,name,role}:\n// id name role\n// 1 Alice admin\n// 2 Bob user\n```\n\n> **💡 Tip**: Use the `forLLM` preset for best results with LLM APIs. It includes semantic headers that help LLMs understand your data structure.\n\n### Real-World Example\n\n```typescript\nimport { encode, forLLM } from \"@ayushmanmishra/toon\";\n\n// GitHub repository data\nconst repo = {\n name: \"toon\",\n stars: 150,\n owner: { name: \"ayushman\", verified: true },\n tags: [\"llm\", \"format\", \"optimization\"],\n config: { private: false, archived: false },\n};\n\nconst toon = encode(repo, forLLM);\n// Result: name: toon,stars: 150,owner{name: ayushman,verified: 1},tags[3]: llm,format,optimization,config{private: 0,archived: 0}\n```\n\n### Presets for Common Use Cases\n\nTOON provides presets optimized for different scenarios:\n\n```typescript\nimport {\n encode,\n forLLM,\n forLLMNested,\n forDebugging,\n} from \"@ayushmanmishra/toon\";\n\n// For LLM prompts (recommended default)\nconst toon1 = encode(data, forLLM);\n\n// For complex nested data (beats CSV by 36%!)\nconst toon2 = encode(nestedData, forLLMNested);\n\n// For debugging (human-readable)\nconst toon3 = encode(data, forDebugging);\n```\n\n---\n\n## 🌐 Multi-Language Support\n\nTOON is a **language-agnostic format specification**. While the official implementation is in TypeScript/JavaScript, TOON can be implemented in any programming language.\n\n### Official Implementation\n\n- **JavaScript/TypeScript** (Node.js) - ✅ Available now\n - npm: `@ayushmanmishra/toon`\n - Works in Node.js, browsers, and TypeScript projects\n - Supports both CommonJS and ES modules\n\n### Community Implementations\n\nWe welcome implementations in other languages! The TOON format is simple to implement:\n\n- **Python** - 🚧 Coming soon (or contribute yours!)\n- **Rust** - 🚧 Coming soon (or contribute yours!)\n- **Go** - 🚧 Coming soon (or contribute yours!)\n- **Java** - 🚧 Coming soon (or contribute yours!)\n- **C# / .NET** - 🚧 Coming soon (or contribute yours!)\n- **Ruby** - 🚧 Coming soon (or contribute yours!)\n- **PHP** - 🚧 Coming soon (or contribute yours!)\n\n### Implementing TOON in Your Language\n\nTOON is straightforward to implement because it's a text-based format. The core algorithm follows a simple recursive pattern:\n\n1. **Null/Undefined** → `null` or `~`\n2. **Array** → `key[count]: value1,value2,value3`\n3. **Object** → `key1: value1,key2: value2` or `key{innerKey: value}`\n4. **Primitive** → String, number, boolean\n\n**Quick Start Guide**: See our [Implementation Guide](./docs/IMPLEMENTATION_GUIDE.md) for:\n\n- Complete algorithm explanation\n- Code examples in Python, Rust, Go, Java\n- Testing guidelines\n- Contribution instructions\n\n### Using TOON from Any Language\n\nEven without a native implementation, you can use TOON from any language:\n\n1. **Generate TOON strings** - Any language can create TOON-formatted strings\n2. **Pass to LLMs** - TOON is just text, works with any LLM API\n3. **LLMs parse TOON** - No decoder needed, LLMs understand TOON natively\n\n**Example (Python without library)**:\n\n```python\ndef to_toon(data):\n if data is None:\n return \"null\"\n if isinstance(data, list):\n items = \",\".join(to_toon(item) for item in data)\n return f\"[{len(data)}]: {items}\"\n if isinstance(data, dict):\n pairs = \",\".join(f\"{k}: {to_toon(v)}\" for k, v in data.items())\n return pairs\n return str(data)\n```\n\n### Contributing Language Implementations\n\nIf you implement TOON in another language:\n\n1. Follow the [TOON specification](./spec/TOON.md)\n2. Match the JavaScript implementation's behavior\n3. Add comprehensive tests\n4. Create a README for your implementation\n5. Submit a PR or create a separate repository and link it here!\n\n---\n\n## 📖 Documentation\n\n### Syntax Overview\n\nTOON uses compact syntax to minimize token usage while maintaining readability:\n\n| Type | Syntax | Example |\n| ------------------ | ---------------------------------- | --------------------------------------------- |\n| **Arrays** | `key[count]: value1,value2,value3` | `tags[3]: jazz,chill,lofi` |\n| **Objects** | `key1: value1,key2: value2` | `name: John,age: 30` |\n| **Nested Objects** | `key{innerKey: value}` | `user{name: John,age: 30}` |\n| **Primitives** | No quotes unless needed | `title: Hello World` → `title: \"Hello World\"` |\n| **Booleans** | `true`/`false` or `1`/`0` | `active: 1` (compact mode) |\n| **Null** | `null` or `~` | `value: ~` (compact mode) |\n\n### Encoding Options\n\n```typescript\nimport { encode, EncodeOptions } from \"@ayushmanmishra/toon\";\n\nconst options: EncodeOptions = {\n compactBooleans: true, // Use 1/0 instead of true/false (saves ~60% tokens)\n compactNull: true, // Use ~ instead of null\n readable: false, // Add spaces for readability (default: false)\n flatten: false, // Flatten nested structures into columns (beats CSV on complex data)\n delimiter: \"\\t\", // Use tabs for better tokenization (default: ',')\n};\n\nconst data = { active: true, value: null };\nconst toon = encode(data, options);\n// Result: active: 1,value: ~\n```\n\n### Complete Examples\n\n#### Basic Structures\n\n```typescript\n// Array\nencode({ tags: [\"jazz\", \"chill\", \"lofi\"] });\n// → tags[3]: jazz,chill,lofi\n\n// Object\nencode({ name: \"John\", age: 30 });\n// → name: John,age: 30\n\n// Nested object\nencode({ user: { name: \"John\", age: 30 } });\n// → user{name: John,age: 30}\n\n// Array of objects\nencode({\n users: [\n { name: \"Alice\", age: 25 },\n { name: \"Bob\", age: 30 },\n ],\n});\n// → users[2]: {name: Alice,age: 25},{name: Bob,age: 30}\n```\n\n#### Advanced Structures\n\n```typescript\n// Complex nested structure\nencode({\n repository: {\n name: \"toon\",\n metadata: {\n stars: 150,\n forks: 12,\n },\n tags: [\"llm\", \"format\"],\n contributors: [\n { name: \"Alice\", commits: 45 },\n { name: \"Bob\", commits: 32 },\n ],\n },\n});\n// → repository{name: toon,metadata{stars: 150,forks: 12},tags[2]: llm,format,contributors[2]: {name: Alice,commits: 45},{name: Bob,commits: 32}}\n```\n\n#### With Options\n\n```typescript\n// Compact mode (maximum token savings)\nencode(\n { active: true, value: null, count: 0 },\n {\n compactBooleans: true,\n compactNull: true,\n }\n);\n// → active: 1,value: ~,count: 0\n\n// Readable mode (for debugging)\nencode({ name: \"John\", age: 30 }, { readable: true });\n// → name: John, age: 30\n```\n\n#### Special Cases\n\n```typescript\n// Strings with spaces (auto-quoted)\nencode({ title: \"Hello World\" });\n// → title: \"Hello World\"\n\n// Empty arrays\nencode({ tags: [] });\n// → tags[0]:\n\n// Empty objects\nencode({ config: {} });\n// → config{}\n\n// Mixed types in arrays\nencode({ mixed: [\"hello\", 42, true, null] });\n// → mixed[4]: hello,42,true,null\n\n// Numbers and special values\nencode({\n count: 42,\n price: 19.99,\n negative: -5,\n zero: 0,\n});\n// → count: 42,price: 19.99,negative: -5,zero: 0\n```\n\n---\n\n## 🔧 API Reference\n\n### `encode(value, options?)`\n\nEncodes any JSON-serializable value to TOON format.\n\n#### Parameters\n\n- **`value`** (`any`): The value to encode (any JSON-serializable value)\n- **`options`** (`EncodeOptions`, optional): Encoding options\n\n#### Returns\n\n- **`string`**: TOON format string\n\n#### Options\n\n| Option | Type | Default | Description |\n| ----------------- | ----------------- | ------- | ------------------------------------------------------------------ | --------------------------------------------------- |\n| `compactBooleans` | `boolean` | `false` | Use `1`/`0` instead of `true`/`false` (saves ~60% tokens) |\n| `compactNull` | `boolean` | `false` | Use `~` instead of `null` |\n| `readable` | `boolean` | `false` | Add spaces after separators for readability |\n| `flatten` | `boolean` | `false` | Flatten nested structures into columns (beats CSV on complex data) |\n| `delimiter` | `',' \\| '\\t' \\| ' | '` | `','` | Delimiter for tabular arrays (tabs tokenize better) |\n| `tabular` | `boolean` | `true` | Use tabular format for uniform arrays of objects |\n\n#### Examples\n\n```typescript\nimport { encode } from \"@ayushmanmishra/toon\";\n\n// Basic encoding\nencode({ name: \"John\" });\n// → name: John\n\n// With options\nencode(\n { active: true, value: null },\n { compactBooleans: true, compactNull: true }\n);\n// → active: 1,value: ~\n\n// Flattened mode (beats CSV on nested data)\nencode(\n { orders: [{ id: 1, customer: { name: \"John\" }, items: [{ sku: \"A\" }] }] },\n { flatten: true, delimiter: \"\\t\", compactBooleans: true }\n);\n// → oid\tc_n\ti0_s\n// 1\tJohn\tA\n```\n\n---\n\n## 💡 Use Cases\n\nTOON is ideal for scenarios where token efficiency matters:\n\n### Primary Use Cases\n\n- **🤖 LLM Prompts**: Reduce token usage in API calls (OpenAI, Anthropic, etc.)\n- **📊 Structured Data**: Pass complex data structures efficiently to LLMs\n- **🪟 Context Windows**: Fit more data in limited context windows\n- **💰 Cost Optimization**: Reduce API costs by using fewer tokens\n- **🔍 RAG Systems**: Efficiently pass retrieved context to LLMs\n- **⚙️ Agent Systems**: Compact representation of tool outputs and state\n- **📝 Configuration Files**: Efficient representation of nested configurations\n\n### When to Use TOON vs Other Formats\n\n#### ✅ Use TOON when:\n\n- Data has nested structures (objects, arrays of objects)\n- Data has mixed types or optional fields\n- You need to represent complex relationships\n- Data structure varies between records\n- You're passing data to LLMs and want maximum efficiency\n- You need human-readable format for debugging\n\n#### ⚠️ Use CSV when:\n\n- Data is purely flat and tabular\n- All records have identical structure\n- No nested structures needed\n- Maximum compression for simple tables is required\n- You're working with spreadsheet-like data\n\n#### ⚠️ Use JSON when:\n\n- You need bidirectional encoding/decoding\n- You're working with APIs that require JSON\n- You need standard format compatibility\n- Token efficiency is not a primary concern\n\n---\n\n## 🎨 Token Optimization Features\n\nTOON achieves token efficiency through several optimization techniques:\n\n1. **Smart Quoting**: Only quotes strings that contain spaces or special characters\n2. **Boolean Compression**: Use `1`/`0` instead of `true`/`false` (saves ~60% tokens)\n3. **Compact Separators**: No spaces around separators by default\n4. **Explicit Counts**: Array counts enable efficient parsing and reduce ambiguity\n5. **Minimal Nesting**: Compact nesting syntax with `{}` instead of nested objects\n6. **Null Compression**: Use `~` instead of `null` in compact mode\n7. **No Redundant Syntax**: Eliminates unnecessary brackets, quotes, and delimiters\n\n---\n\n## 📈 Token Comparison Examples\n\n### Example 1: Simple Array\n\n**JSON**: `{ \"tags\": [\"jazz\", \"chill\", \"lofi\"] }` \n**Tokens**: ~15 tokens\n\n**TOON**: `tags[3]: jazz,chill,lofi` \n**Tokens**: ~8 tokens\n\n**Savings**: ~47% token reduction\n\n### Example 2: Object with Multiple Fields\n\n**JSON**: `{ \"name\": \"John\", \"age\": 30, \"active\": true }` \n**Tokens**: ~15 tokens\n\n**TOON**: `name: John,age: 30,active: 1` (with `compactBooleans: true`) \n**Tokens**: ~8 tokens\n\n**Savings**: ~47% token reduction\n\n### Example 3: Nested Structure\n\n**JSON**: `{ \"user\": { \"name\": \"John\", \"tags\": [\"admin\", \"user\"] } }` \n**Tokens**: ~20 tokens\n\n**TOON**: `user{name: John,tags[2]: admin,user}` \n**Tokens**: ~11 tokens\n\n**Savings**: ~45% token reduction\n\n### Example 4: Complex Nested Data\n\n**JSON**:\n\n```json\n{\n \"repository\": {\n \"name\": \"toon\",\n \"owner\": { \"name\": \"ayushman\", \"verified\": true },\n \"tags\": [\"llm\", \"format\"]\n }\n}\n```\n\n**Tokens**: ~35 tokens\n\n**TOON**: `repository{name: toon,owner{name: ayushman,verified: 1},tags[2]: llm,format}` \n**Tokens**: ~18 tokens\n\n**Savings**: ~49% token reduction\n\n---\n\n## 📚 Format Specification\n\nFor complete format details, see the [official TOON specification](./spec/TOON.md).\n\n### Quick Reference\n\n- **Arrays**: `key[count]: value1,value2,value3`\n- **Objects**: `key1: value1,key2: value2`\n- **Nested Objects**: `key{innerKey: value}`\n- **Primitives**: No quotes unless needed (spaces, special chars)\n- **Booleans**: `true`/`false` or `1`/`0` (compact mode)\n- **Null**: `null` or `~` (compact mode)\n\n### Additional Resources\n\n- [Implementation Guide](./docs/IMPLEMENTATION_GUIDE.md) - Guide for implementing TOON in other languages\n- [TOON Specification](./spec/TOON.md) - Complete format specification\n\n---\n\n## 🤝 Contributing\n\nContributions are welcome! TOON is an open-source project designed to make LLM interactions more efficient.\n\n### Getting Started\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/amazing-feature`)\n3. Make your changes\n4. Add tests if applicable\n5. Commit your changes (`git commit -m 'Add some amazing feature'`)\n6. Push to the branch (`git push origin feature/amazing-feature`)\n7. Open a Pull Request\n\n### Development\n\n```bash\n# Install dependencies\nnpm install\n\n# Build the project\nnpm run build\n\n# Run tests\nnpm test\n\n# Run benchmarks\nnpm run benchmark\n\n# Run comprehensive benchmarks\nnpm run benchmark:all\n```\n\nPlease see the [specification](./spec/TOON.md) for format details and design principles.\n\n### Publishing to npm\n\nFor maintainers: See [Publishing Guide](./docs/PUBLISHING.md) for step-by-step instructions on publishing to npm.\n\n---\n\n## 📄 License\n\nMIT License - see [LICENSE](LICENSE) file for details.\n\n---\n\n## 🙏 Acknowledgments\n\nTOON is designed with the goal of making LLM interactions more efficient and cost-effective. Special thanks to the open-source community for inspiration and feedback.\n\n---\n\n\n\n**Made with ❤️ for the LLM community**\n\n[Report Bug](https://github.com/Ayushman88/toon/issues) · [Request Feature](https://github.com/Ayushman88/toon/issues) · [Documentation](./spec/TOON.md) · [npm Package](https://www.npmjs.com/package/@ayushmanmishra/toon)\n\n
\n","readmeFilename":"README.md"}