{"_id":"@darshitakhanna15/tokenist","name":"@darshitakhanna15/tokenist","dist-tags":{"latest":"1.0.0"},"versions":{"1.0.0":{"name":"@darshitakhanna15/tokenist","version":"1.0.0","description":"Tokenist: Open-Source Local Token Optimizer Layer for AI Agents","main":"dist/index.js","bin":{"tokenist":"dist/index.js","tokenist-init":"dist/cli/init.js"},"scripts":{"build":"tsc && ts-node src/core/copy_assets.ts","start":"node dist/index.js","benchmark":"ts-node src/core/benchmarkReal.ts","init":"ts-node src/cli/init.ts","export":"ts-node src/cli/export.ts","prepublishOnly":"npm run build"},"keywords":["mcp","model-context-protocol","token-optimization","context-window","claude-code","antigravity","tree-sitter","ast-compression","ai-agent"],"author":{"name":"Tokenist Team"},"license":"MIT","devDependencies":{"@types/node":"^18.19.130","@types/react":"^19.2.17","@types/vscode":"^1.80.0","react":"^19.2.7","ts-node":"^10.9.2","typescript":"^5.9.3"},"dependencies":{"@modelcontextprotocol/sdk":"^1.29.0","@xenova/transformers":"^1.4.2","ignore":"^7.0.6","tree-sitter-wasms":"^0.1.13","web-tree-sitter":"^0.20.8"},"_id":"@darshitakhanna15/tokenist@1.0.0","_nodeVersion":"20.20.2","_npmVersion":"10.8.2","dist":{"integrity":"sha512-vKVETMrowCl9dgVeiAjk+O95NF8ljUm591unWB/tQkReKiZaqEoDoFWR4qU+zNZOBD10l2egdiURR75xDFmg7Q==","shasum":"10894168af30bcc9d16a62faa5195731359acfca","tarball":"https://registry.npmjs.org/@darshitakhanna15/tokenist/-/tokenist-1.0.0.tgz","fileCount":63,"unpackedSize":373879,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEQCIGmrKC8A7dmFfc46z1EOlbUSv+dKqlHy84ZV8FE7sx3kAiBrv3DoyK2j2K7O3UsVfnVCK6J4Y3eRyUMzLnVu6V6U2Q=="}]},"_npmUser":{"name":"darshitakhanna15","email":"darshitakhanna@gmail.com"},"directories":{},"maintainers":[{"name":"darshitakhanna15","email":"darshitakhanna@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/tokenist_1.0.0_1785924551429_0.4233869366831311"},"_hasShrinkwrap":false}},"time":{"created":"2026-08-05T10:09:11.275Z","1.0.0":"2026-08-05T10:09:11.603Z","modified":"2026-08-05T10:09:11.829Z"},"maintainers":[{"name":"darshitakhanna15","email":"darshitakhanna@gmail.com"}],"description":"Tokenist: Open-Source Local Token Optimizer Layer for AI Agents","keywords":["mcp","model-context-protocol","token-optimization","context-window","claude-code","antigravity","tree-sitter","ast-compression","ai-agent"],"author":{"name":"Tokenist Team"},"license":"MIT","readme":"# Tokenist ⚡\n> **Open-Source Local Token Optimizer Layer for AI Coding Agents**\n\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)\n[![Node.js Version](https://img.shields.io/badge/node-%3E%3D18.0.0-brightgreen.svg)](https://nodejs.org/)\n[![MCP Spec](https://img.shields.io/badge/MCP-1.0.0-blue.svg)](https://modelcontextprotocol.io/)\n[![Token Savings](https://img.shields.io/badge/Token_Savings-Up_to_84%25-success.svg)](#benchmark--performance)\n\n**Tokenist** is an ultra-fast, local Model Context Protocol (MCP) server that sits between AI agent platforms (Claude Code, Antigravity, Kiro, OpenCode, Continue.dev) and your codebase. \n\nBy analyzing Abstract Syntax Trees (ASTs) via `web-tree-sitter`, PageRank dependency centrality graphs, and hybrid semantic search, Tokenist **reduces token consumption by up to 84%** while preserving complete syntactic integrity and core API signatures.\n\n---\n\n## 🚀 Key Features\n\n* **🌳 Multi-Tier AST Code Compression**: 3 intelligent compression levels (`normal`, `medium`, `aggressive`) using Tree-Sitter WASM parsers for TypeScript, JavaScript, Python, Go, Rust, Java, and C#.\n* **🕸️ PageRank Dependency Centrality Scoring**: Power iteration graph analysis ranks high-impact architectural interfaces so foundational modules are kept at full detail while leaf utilities are aggressively compressed.\n* **⚡ SHA-256 Content-Hash AST Cache**: Instant `<15ms` cache hits with SHA-256 invalidation that eliminates NFS/Docker `mtime` staleness issues.\n* **🔍 BM25 + Semantic Hybrid Search**: Local ONNX `all-MiniLM-L6-v2` vector embeddings combined with BM25 keyword matching for context retrieval.\n* **📌 Query-Adaptive Symbol Pinning**: Automatically extracts function, class, and variable identifiers from developer prompts to pin relevant target files.\n* **📊 Analytics Dashboard & Telemetry**: Built-in HTTP dashboard and terminal cost estimator tracking reclaimed tokens and dollar savings across GPT-4o, Claude 3.5/3.7, and Gemini models.\n* **🔌 Universal Platform Integration**: Out-of-the-box MCP transport adapters for Claude Code, Antigravity, Kiro, OpenCode, Continue.dev, and Gemini CLI.\n\n---\n\n## 📦 Quickstart\n\n### 1. Initialize in your repository\nRun the initialization wizard in your project root:\n```bash\nnpx tokenist init\n```\nThis generates a `.tokenist.json` configuration file tailored to your project structure.\n\n### 2. Connect to your AI Agent\n\n#### 🤖 Claude Code / Claude Desktop\nAdd Tokenist to your `claude_desktop_config.json` or `.mcp/config.json`:\n```json\n{\n  \"mcpServers\": {\n    \"tokenist\": {\n      \"command\": \"npx\",\n      \"args\": [\"-y\", \"tokenist\"]\n    }\n  }\n}\n```\n\n#### 🌌 Antigravity / Gemini IDE Platform\nAdd to your workspace `.gemini/mcp.json`:\n```json\n{\n  \"mcpServers\": {\n    \"tokenist\": {\n      \"command\": \"npx\",\n      \"args\": [\"-y\", \"tokenist\"]\n    }\n  }\n}\n```\n\n#### 🎯 Kiro / Continue.dev (HTTP Transport)\nStart Tokenist in HTTP transport mode:\n```bash\nnpx tokenist --transport http --port 3334\n```\n\n---\n\n## 🛠️ MCP Tools Reference\n\nTokenist exposes 5 high-performance MCP tools:\n\n| MCP Tool Name | Description | Key Parameters |\n|---|---|---|\n| `fetch_optimized_repository_map` | Crawls and compresses an entire workspace folder into a token-budgeted context map. | `dirPath`, `tier` (`normal`\\|`medium`\\|`aggressive`), `maxTokenBudget`, `semanticQuery`, `activePrompt` |\n| `read_optimized_file` | Reads and compresses a single source file at a specified compression tier. | `filePath`, `tier` (`normal`\\|`medium`\\|`aggressive`) |\n| `get_file_diff_context` | Extracts uncommitted git diff hunks with configurable context lines. | `filePath`, `contextLines` |\n| `reindex_workspace` | Force-flushes AST caches and re-indexes PageRank dependency graphs. | *None* |\n| `get_token_savings_report` | Displays accumulated token volume savings and estimated cash savings. | *None* |\n\n---\n\n## 📊 Benchmark & Performance\n\nTested on a real-world TypeScript project (49 files, ~63,000 BPE tokens):\n\n| Compression Tier | Raw Volume | Compressed Volume | Token Reduction | Average Latency |\n|---|---|---|---|---|\n| **Normal** (Clean & Licenses) | 63,066 tokens | 62,804 tokens | **0.42%** | 0.8ms / file |\n| **Medium** (Control Blocks Collapsed) | 63,066 tokens | 34,120 tokens | **45.89%** | 1.2ms / file |\n| **Aggressive** (Function Signature Map) | 63,066 tokens | 9,982 tokens | **84.16%** | 1.8ms / file |\n\n*Cost Savings:* At $3.00/1M tokens (Claude Sonnet), aggressive tier reduces context cost from **$0.189 to $0.029 per request** (Saving **~$160/month** for active developers).\n\n---\n\n## ⚙️ Configuration (`.tokenist.json`)\n\nCustomize compression behaviors by creating a `.tokenist.json` in your workspace root:\n\n```json\n{\n  \"defaultTier\": \"aggressive\",\n  \"maxTokenBudget\": 8000,\n  \"enableSemanticRanking\": true,\n  \"gitAwareTiers\": true,\n  \"pricingModel\": \"claude-sonnet-4\",\n  \"alwaysInclude\": [\n    \"src/types/**/*.ts\",\n    \"src/server.ts\"\n  ],\n  \"preserveCommentPatterns\": [\"TODO\", \"FIXME\", \"@api\", \"@public\"],\n  \"cacheStrategy\": \"disk\",\n  \"maxCacheSizeMb\": 20\n}\n```\n\n---\n\n## 🔠 Supported Languages\n\nTokenist provides native Tree-Sitter AST grammars for:\n- **TypeScript** (`.ts`, `.tsx`)\n- **JavaScript** (`.js`, `.jsx`, `.mjs`)\n- **Python** (`.py`)\n- **Go** (`.go`)\n- **Rust** (`.rs`)\n- **Java** (`.java`)\n- **C#** (`.cs`)\n\n---\n\n## 📄 License\n\nDistributed under the MIT License. See `LICENSE` for more information.\n","readmeFilename":"README.md","_rev":"1-1c79b852e9a2d8740056079d34e5260a"}