![\"toolscore\"](\"assets/logo-light.svg\"){kind=logo}
\n \n \n
\n ![\"npm](\"https://img.shields.io/npm/v/%40aryan_mangod%2Ftoolscore.svg\")
\n ![\"GitHub](\"https://img.shields.io/github/v/release/Aryan-MP/toolscore?sort=semver\")
![\"License:](\"https://img.shields.io/badge/license-MIT-informational.svg\"){kind=icon}
\n
See how real models actually use your MCP tools. Selection accuracy, parameter accuracy, confusion pairs, graded A to F.
\n\nStatic linters check that your tool descriptions *exist*. toolscore checks that they *work*: it generates realistic tasks for your toolset (with ground-truth labels built in by construction), runs them against a real model, and tells you which tools the model confuses, which parameters it gets wrong, and what to fix.\n\n## Why\n\nAgent toolsets fail silently. Research on production agents shows tool-selection accuracy collapses as toolsets grow (down to roughly 13% on large tool menus), and a wrong parameter cascades into a wrong final answer most of the time. Audits of the MCP ecosystem grade the overwhelming majority of published tools **F**, mostly for descriptions a model cannot disambiguate.\n\nThe teams who fix this today do it by hand: building custom eval harnesses, iterating on tool descriptions until selection accuracy climbs. One documented case took tool selection from 60% to 100% purely by rewriting descriptions. toolscore turns that loop into one command.\n\n## Install\n\n```bash\n# No Node.js required. Standalone binary, checksum-verified at install time.\ncurl -fsSL https://raw.githubusercontent.com/Aryan-MP/toolscore/main/install.sh | bash\n\n# Or, if you are already in a Node/npm project\nnpm install -g @aryan_mangod/toolscore\n```\n\n## Quick start\n\n```bash\n# Optional: interactive setup, saves defaults to .toolscorerc.json\nnpx @aryan_mangod/toolscore init\n\n# Scan your MCP server (BYOK, uses your ANTHROPIC_API_KEY)\nnpx @aryan_mangod/toolscore scan --stdio \"node ./my-mcp-server.js\"\n\n# No API key? Use your Claude subscription via the Claude Code CLI\nnpx @aryan_mangod/toolscore scan --stdio \"node ./my-mcp-server.js\" --provider claude-code\n\n# Or try the pipeline against the bundled flawed demo server, fully offline\nnpm run demo\n\n# Then let a model rewrite the descriptions it got wrong, and prove the delta\nnpx @aryan_mangod/toolscore fix --stdio \"node ./my-mcp-server.js\" --provider claude-code\n```\n\n`scan` and `fix` show live progress (connecting, generating tasks, running, scoring) and print nothing but clean JSON on stdout when run with `--json`, so they are safe to pipe in CI.\n\nReal output from a real model (Claude, via `--provider claude-code`) scanning the bundled demo server:\n\n```\n toolscore · tsx demo/flawed-server.ts\n 7 tools · 21 tasks · claude-code/default-prompted\n\n Grade: B+ (89/100)\n\n ├─ Tool selection 86%\n └─ Parameter accuracy 94% (when the right tool was chosen)\n\n Lexical baseline: 0% (bag-of-words matcher on the same tasks)\n\n Confusion pairs (expected → actually called)\n find_content → search_docs ×1\n add_document → create_note ×1\n get_user → fetch_account ×1\n```\n\nThe demo server has three deliberately planted flaws: three near-identical search tools, a vague note/document split, and two user-lookup tools with interchangeable descriptions. The model stumbled on **exactly those three**.\n\n## How it works\n\n1. **Discover.** Connects to your MCP server over stdio and lists its tools.\n2. **Generate.** Creates realistic user tasks from each tool, so the correct answer is known by construction: pick tool T, write a request that objectively requires T with concrete parameter values embedded in the text. Extra tasks deliberately target your most confusable tool pairs (ranked by description similarity), hunting where failures actually live instead of sampling randomly.\n3. **Run.** Presents the **full** toolset and each task to the model, records which tool it picks and with what arguments.\n4. **Score.** Selection accuracy (60% weight) plus parameter accuracy among correct selections (40% weight) produce a 0 to 100 score and an A to F grade, plus a confusion-pair diagnosis pointing at the exact tools to fix.\n\n### The lexical baseline: keeping the benchmark honest\n\nA tool benchmark can silently flatter you: if generated tasks reuse your tools' vocabulary (\"search the docs for…\" leading to `search_docs`), any model passes by keyword matching and the grade means nothing. toolscore defends against this two ways:\n\n- Task generation is **anti-leakage by instruction**: tasks describe the user's goal in the user's own words and must never echo a tool's name or distinctive vocabulary.\n- Every scan also runs a **bag-of-words matcher** over the same tasks. If that dumb baseline scores high, the report warns you that the grade is inflated. (On the demo scan above: model 86%, baseline 0%. The tasks genuinely require understanding intent.)\n\n### `toolscore fix`\n\n`fix` runs a scan, has the model rewrite the descriptions of every tool involved in a failure, then **re-runs the identical task set** against the rewritten toolset and shows the before and after delta, including whether the resolved confusion pairs actually resolved, and whether the new descriptions cheated by echoing task vocabulary (the lexical baseline is compared too). You get copy-pasteable descriptions with a verified score improvement, or an honest report that they did not help.\n\n### Reproducibility\n\nGenerated task sets are cached in `.toolscore/`, keyed by a hash of your toolset's schemas. Same server, same task set, same model produces comparable grades run to run, which is what makes regression gating in CI meaningful. Pass `--regenerate` after changing your tools.\n\n## CLI reference\n\n```\ntoolscore init interactive setup, saves defaults to .toolscorerc.json\ntoolscore scan --stdio \"