{"_id":"@agentkit-js/evals-runner","_rev":"3-3b20eb6150539fc676a2e35d9dc287ce","name":"@agentkit-js/evals-runner","dist-tags":{"latest":"0.1.0"},"versions":{"0.1.0":{"name":"@agentkit-js/evals-runner","version":"0.1.0","keywords":["agentkit","evaluation","benchmark","model-evaluation","pareto","longmemeval","paired-statistics"],"author":{"name":"agentkit-js contributors"},"license":"Apache-2.0","_id":"@agentkit-js/evals-runner@0.1.0","maintainers":[{"name":"tellerlin","email":"lintao_mailbox@163.com"}],"homepage":"https://github.com/telleroutlook/agentkit-js/tree/main/packages/evals-runner#readme","bugs":{"url":"https://github.com/telleroutlook/agentkit-js/issues"},"bin":{"agentkit-evals":"dist/cli.js"},"dist":{"shasum":"2f756c481ddd250520fcff7767e9c11bd707888e","tarball":"https://registry.npmjs.org/@agentkit-js/evals-runner/-/evals-runner-0.1.0.tgz","fileCount":91,"integrity":"sha512-z+CumdoL9SIlxxYyg7xoGvsTwrbunI2qms29einzq05BM+97bwhVCrD4kmf/240TLdEUuNGuBwbE+iol4ACEjg==","signatures":[{"sig":"MEUCIQDK3Wj7ORDUWP5Q71s1OFjgh7KjwasDNzs/9xPRDuektQIgODwYG3aZKEO1qpKSZHCC5SjEy4UE1QTvWbk8E/us3jY=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@agentkit-js%2fevals-runner@0.1.0","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":420769},"main":"./dist/index.js","type":"module","types":"./dist/index.d.ts","engines":{"node":">=20.0.0"},"exports":{".":{"types":"./dist/index.d.ts","import":"./dist/index.js"},"./stats":{"types":"./dist/stats/index.d.ts","import":"./dist/stats/index.js"},"./suites":{"types":"./dist/suites/index.d.ts","import":"./dist/suites/index.js"}},"gitHead":"c629f75b91a8855e107cb0130c858b10e8ea6e79","scripts":{"test":"vitest run","build":"tsc -p tsconfig.json","clean":"rm -rf dist .turbo","typecheck":"tsc -p tsconfig.json --noEmit"},"_npmUser":{"name":"tellerlin","email":"lintao_mailbox@163.com"},"deprecated":"This package has moved to @wasmagent/evals-runner. Please update your dependency.","repository":{"url":"git+https://github.com/telleroutlook/agentkit-js.git","type":"git","directory":"packages/evals-runner"},"_npmVersion":"10.9.8","description":"Evaluation runner for agentkit-js — multi-model multi-axis benchmark harness with paired statistics + Pareto report. Targets the gaps in evomerge-style single-axis evaluation: long-context recall, multi-turn memory, agent trajectory, latency, cost.","directories":{},"_nodeVersion":"22.22.3","dependencies":{"zod":"^3.23.0","@agentkit-js/core":"workspace:*","@agentkit-js/devtools":"workspace:*"},"publishConfig":{"access":"public","provenance":true},"_hasShrinkwrap":false,"devDependencies":{"vitest":"^2.1.0","typescript":"^5.7.0","@types/node":"^22.0.0","@agentkit-js/kernel-quickjs":"workspace:*"},"_npmOperationalInternal":{"tmp":"tmp/evals-runner_0.1.0_1781508321009_0.16713379109957605","host":"s3://npm-registry-packages-npm-production"}}},"time":{"created":"2026-06-15T07:25:20.880Z","modified":"2026-07-04T01:07:19.455Z","0.1.0":"2026-06-15T07:25:21.138Z"},"bugs":{"url":"https://github.com/telleroutlook/agentkit-js/issues"},"author":{"name":"agentkit-js contributors"},"license":"Apache-2.0","homepage":"https://github.com/telleroutlook/agentkit-js/tree/main/packages/evals-runner#readme","keywords":["agentkit","evaluation","benchmark","model-evaluation","pareto","longmemeval","paired-statistics"],"repository":{"url":"git+https://github.com/telleroutlook/agentkit-js.git","type":"git","directory":"packages/evals-runner"},"description":"Evaluation runner for agentkit-js — multi-model multi-axis benchmark harness with paired statistics + Pareto report. Targets the gaps in evomerge-style single-axis evaluation: long-context recall, multi-turn memory, agent trajectory, latency, cost.","maintainers":[{"email":"lintao_mailbox@163.com","name":"tellerlin"},{"email":"hyin.sybase@gmail.com","name":"haining-yin"}],"readme":"# @agentkit-js/evals-runner\n\n> Multi-model multi-axis evaluation harness for any OpenAI-compatible\n> endpoint. Built on the agentkit-js scorer + EventLog + RunsAggregator\n> primitives, plus a TypeScript port of the canonical paired-statistics\n> machinery (McNemar exact / Wilson CI / paired bootstrap / G1 gate).\n\nagentkit-js stays **independent**: this package adds an evaluation layer\non top of the runtime's primitives, but agentkit's runtime keeps no\nknowledge of any specific model author or research line. Use it the same\nway you'd use `lm-evaluation-harness` — just point at a base URL.\n\n## Why this exists\n\nThe accuracy-only benchmarks academic projects ship (GSM8K, IFEval,\nMMLU, HumanEval) ignore the things production model selection actually\nneeds:\n\n- **Long-context recall** — does the model retrieve a fact from the\n  middle of a 16K document?\n- **Multi-turn memory** — can it use a 28-turn dialog history?\n- **Trajectory quality** — does an agent loop recover from a failed step?\n- **Latency under budget** — accuracy is irrelevant if p95 wall blows\n  the budget.\n- **Cost per correct answer** — the only axis that decides deployment.\n\nSix reference suites cover those gaps. Plus the statistical primitives\nthat turn a 6-item run into a defensible claim:\n\n- `mcnemarExact(b, c)` — exact paired test\n- `wilsonCI(s, n)` — proportion CI\n- `pairedBootstrap(cand, base)` — distribution-free delta CI\n- `buildG1Report(seeds[])` — pooled-across-seeds gate matching the\n  ≥3-seed discipline standard in the field\n\n## Install\n\n```bash\nnpm install @agentkit-js/evals-runner @agentkit-js/core\n```\n\n## Quick start (CLI)\n\n```bash\n# List the 6 reference suites:\nagentkit evals list\n\n# Run multi-turn memory + cost-per-correct against 2 models, 3 seeds:\nagentkit evals run \\\n  --suite=multi-turn-memory,cost-per-correct \\\n  --models=\"qwen2.5:0.5b@http://localhost:11434/v1,gpt-4o-mini@https://api.openai.com/v1\" \\\n  --seeds=0,1,2 \\\n  --report-file=./eval-report.md\n```\n\nMarkdown output: a Pareto-flagged summary table, a per-suite item × model\nmatrix, a configuration footer. Drop into a PR or commit message verbatim.\n\n## Programmatic use\n\n```ts\nimport {\n  runEvaluation,\n  multiTurnMemorySuite,\n  costPerCorrectSuite,\n  renderReportMarkdown,\n} from \"@agentkit-js/evals-runner\";\n\nconst report = await runEvaluation({\n  models: [\n    {\n      id: \"qwen2.5:0.5b\",\n      baseUrl: \"http://localhost:11434/v1\",\n      apiKey: \"ollama\",\n      pricePer1MInput: 0,    // local — free\n      pricePer1MOutput: 0,\n    },\n    {\n      id: \"gpt-4o-mini\",\n      baseUrl: \"https://api.openai.com/v1\",\n      apiKey: process.env.OPENAI_API_KEY,\n      pricePer1MInput: 0.15,\n      pricePer1MOutput: 0.60,\n    },\n  ],\n  suites: [multiTurnMemorySuite, costPerCorrectSuite],\n  seeds: [0, 1, 2],\n});\n\nconsole.log(renderReportMarkdown(report));\n```\n\n## Reference suites\n\n| Suite                    | What it measures                                                       |\n| ------------------------ | ---------------------------------------------------------------------- |\n| `multi-turn-memory`      | LongMemEval-style 6-item conversation-history recall, 5 categories     |\n| `long-context-recall`    | Needle-in-haystack at 10% / 50% / 90% depth in a ~16K-token document   |\n| `cost-per-correct`       | Same items as multi-turn-memory; reports USD per passing answer        |\n| `tool-sequence`          | 3-step JSON-encoded tool-call plans matched against an expected order  |\n| `agent-trajectory`       | Plan + reasoning emission scored by trajectory validity + length       |\n| `latency-under-budget`   | Multi-turn memory under a 2 s wall-clock + 256-token budget per item   |\n\nAll 6 suites use **synthetic / hand-built fixtures**. None of the items\noverlap with publicly published training corpora (GSM8K / MMLU / IFEval /\nHumanEval / Alpaca etc.) — that is a deliberate choice so a model fine-\ntuned on those public benchmarks does NOT silently leak through. Adding\nyour own suites is encouraged; see `BenchmarkSuite` in `src/types.ts`.\n\n## Statistical primitives\n\n```ts\nimport {\n  mcnemarExact,\n  wilsonCI,\n  pairedBootstrap,\n  buildG1Report,\n} from \"@agentkit-js/evals-runner/stats\";\n\n// Exact McNemar paired test:\nconst { p } = mcnemarExact(/* b */ 25, /* c */ 5);\n// → { p: 3.249e-4, b: 25, c: 5, n: 30 }\n\n// Wilson CI on a binomial proportion:\nconst [lo, hi] = wilsonCI(/* successes */ 50, /* total */ 100);\n// → [0.40383, 0.59617]\n\n// G1 gate over ≥3 seeds:\nconst g1 = buildG1Report(\"v1.2 vs baseline\", [seed0, seed1, seed2]);\n// → { passes: true, pooled: { mcnemarP: 1e-12, ... }, ... }\n```\n\nAll primitives have parity tests against scipy reference values\n(`src/stats/index.test.ts`).\n\n## What is NOT in this package\n\n- **Auto-quantize / auto-merge / auto-train.** This is an evaluation\n  harness, not a model-tuning tool. It tells you which model wins; what\n  you do with that information lives in your tuning pipeline.\n- **GSM8K / MMLU / IFEval / HumanEval suites.** These are common\n  training-data leakage vectors. Use `lm-evaluation-harness` if you need\n  them; this package focuses on axes those benchmarks don't cover.\n- **A leaderboard SaaS.** The output is markdown + JSON; ship it\n  yourself.\n\n## Design points\n\n- **Provider-agnostic.** A `ModelSpec` is `{ id, baseUrl, modelId,\n  apiKey }` — point at Ollama, OpenRouter, vLLM, AI Gateway, OpenAI,\n  Anthropic-compat, or anything else.\n- **Deterministic by default.** `temperature=0`. Three seeds enforced.\n  Reports `σ across seeds` so you can see when a model's \"win\" is noise.\n- **Pareto-first reporting.** The default summary flags\n  non-dominated models on `(meanAcc, totalCostUsd, p95WallMs)` per\n  suite — because in real selection you don't want one number, you\n  want the deployment trade-off surface.\n\n## Roadmap (v0.2)\n\n- IRT subset selection — pick the most informative N items from a\n  larger pool. Lets you publish 50-item numbers as confidently as 500.\n- Conformal CI — distribution-free CI for use when bootstrap\n  assumptions don't hold.\n- Per-model concurrency hooks for clouds with high parallelism budgets.\n- Real `ToolCallingAgent` loop in `agent-trajectory` (currently\n  string-presence heuristic).\n\n## See also\n\n- [`docs/guides/evals-runner.md`](https://github.com/telleroutlook/agentkit-js/blob/main/docs/guides/evals-runner.md)\n  — full guide with Ollama / OpenRouter / Gateway recipes.\n- [`docs/guides/openai-compat-recipes.md`](https://github.com/telleroutlook/agentkit-js/blob/main/docs/guides/openai-compat-recipes.md)\n  — same model-spec format used by the rest of agentkit.\n","readmeFilename":"README.md"}