{"_id":"@ai-craft/agent-bench","_rev":"2-54899293c2dbde2b141ef11053680cd8","name":"@ai-craft/agent-bench","dist-tags":{"latest":"0.1.0"},"versions":{"0.1.0":{"name":"@ai-craft/agent-bench","version":"0.1.0","license":"MIT","_id":"@ai-craft/agent-bench@0.1.0","maintainers":[{"name":"volkz","email":"delacruzd93@gmail.com"}],"dist":{"shasum":"4b49c01670b9ed82ef18302231d5e10ecc5db4d1","tarball":"https://registry.npmjs.org/@ai-craft/agent-bench/-/agent-bench-0.1.0.tgz","fileCount":433,"integrity":"sha512-l7e08ZfvcbybjnOW0qboLWTCJKLEabOC3zH1p/7qIGCOdTB3SIuFIXGRqUS3s2ZFviahxzqultbF9PjtOQu+7w==","signatures":[{"sig":"MEQCIE0HjjC3vw9aJT6lBI5GBqodkELg4IVkAUC3IrWMQdt0AiA8SCtOuM+CC1YNeXOAEkgx2ZzNcRATWz6EZECQVL/ebQ==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":5068224},"main":"./index.js","type":"module","_from":"file:ai-craft-agent-bench-0.1.0.tgz","engines":{"node":">=22"},"_npmUser":{"name":"volkz","email":"delacruzd93@gmail.com"},"_resolved":"/private/var/folders/_j/tzygz83s12v4rnxgchnxz55m0000gp/T/221be6e504bc94125ee342a15179f2f3/ai-craft-agent-bench-0.1.0.tgz","_integrity":"sha512-l7e08ZfvcbybjnOW0qboLWTCJKLEabOC3zH1p/7qIGCOdTB3SIuFIXGRqUS3s2ZFviahxzqultbF9PjtOQu+7w==","_npmVersion":"10.9.2","description":"Unified runnable benchmark harness for craftflow agent runtime packages","directories":{},"_nodeVersion":"22.14.0","dependencies":{"@ai-craft/agent-llm":"0.1.1","@ai-craft/agent-loop":"0.1.1","@ai-craft/tokentracker":"0.1.0","@ai-craft/agent-workflow":"0.0.3"},"publishConfig":{"access":"public"},"_hasShrinkwrap":false,"_npmOperationalInternal":{"tmp":"tmp/agent-bench_0.1.0_1784450392535_0.7757141458737247","host":"s3://npm-registry-packages-npm-production"},"deprecated":"internal-only, not published going forward"}},"time":{"created":"2026-07-19T08:39:52.407Z","modified":"2026-08-12T11:22:48.471Z","0.1.0":"2026-07-19T08:39:52.739Z"},"license":"MIT","description":"Unified runnable benchmark harness for craftflow agent runtime packages","maintainers":[{"name":"volkz","email":"delacruzd93@gmail.com"}],"readme":"# @ai-craft/agent-bench\n\nUnified, runnable benchmark harness for `@ai-craft` workflows. Runs craftflow\norchestration and the agent runtime end-to-end via `@ai-craft/agent-workflow`\nin-process, then scores results on task success rate, cost, latency, and token\nefficiency.\n\n---\n\n## Installation\n\nThis is a workspace-only package in the `@ai-craft` monorepo. It is not\npublished to npm as a standalone dependency. Run from the monorepo root:\n\n```bash\nnvm use 22.14.0\npnpm install\n```\n\n---\n\n## Quick Start\n\nRun the built-in smoke suite in deterministic (CI-safe) mode:\n\n```bash\nagent-bench run --suite smoke --mode deterministic --out docs/benchmarks/\n```\n\nCompare two scorecards to detect regressions:\n\n```bash\nagent-bench compare \\\n  --baseline docs/benchmarks/2026-06-23-smoke-agent-workflow-deterministic.json \\\n  --candidate docs/benchmarks/2026-06-24-smoke-agent-workflow-deterministic.json\n```\n\nList available suites and adapters:\n\n```bash\nagent-bench list\n```\n\n---\n\n## CLI Commands\n\n| Command | Description |\n|---------|-------------|\n| `run` | Run a benchmark suite against a harness adapter |\n| `compare` | Compare two JSON scorecards and report per-task deltas |\n| `list` | Print all available suites and adapters |\n\n### `agent-bench run`\n\n```\nagent-bench run --suite <id> --mode <deterministic|live> [OPTIONS]\n\nRequired:\n  --suite <id>       Suite to run (e.g. smoke, orchestration)\n  --mode <mode>      \"deterministic\" (scripted, CI-safe) or \"live\" (real provider)\n\nOptional:\n  --adapter <id>     Harness adapter to use (default: agent-workflow)\n  --out <dir>        Output directory for JSON/Markdown reports\n                     (default: docs/benchmarks/)\n  --git-sha <sha>    Git SHA to embed in scorecard metadata\n```\n\n### `agent-bench compare`\n\n```\nagent-bench compare --baseline <path.json> --candidate <path.json>\n\n  --baseline <path>   Path to baseline scorecard JSON\n  --candidate <path>  Path to candidate scorecard JSON\n\nExits 0 if no regressions detected; exits 1 if any regression found.\nA regression is: success rate drops, OR cost increases >10%, OR latency\nincreases >20% on any task.\n```\n\n### `agent-bench list`\n\n```\nagent-bench list\n```\n\nPrints all registered suites and adapters with short descriptions.\n\n---\n\n## Suites\n\n### `smoke`\n\nOne BUILD happy-path task in deterministic mode. Verifies the harness runs\nend-to-end and produces a complete `Scorecard` with `successRate: 1`.\n\n### `orchestration`\n\nFour BUILD branch-coverage tasks targeting craftflow gate paths:\n\n| Task | Scenario |\n|------|----------|\n| `reject-at-approval` | Plan rejected at the approval gate |\n| `remfix-loop` | REM-FIX sub-loop triggered and resolved |\n| `verifier-fail-gate` | Integration verifier returns FAIL; workflow holds |\n| `failure-stop` | Terminal FAIL propagated to workflow stop |\n\nAll run in deterministic mode (scripted LLM, no external API calls).\n\n---\n\n## Programmatic API\n\n```typescript\nimport {\n  runBench,\n  scoreTask,\n  buildScorecard,\n  compareScorecards,\n  writeReport,\n} from '@ai-craft/agent-bench';\n```\n\n### `runBench(opts): Promise<RawRun[]>`\n\nRuns all tasks in a `BenchSuite` against a `HarnessAdapter`. Isolates each\ntask's usage ledger via a unique `runId` and temporary `AICRAFT_USAGE_LOG`\npath. Returns one `RawRun` per task.\n\n```typescript\nconst runs = await runBench({ suite: smokeSuite, adapter, mode: 'deterministic' });\n```\n\n### `scoreTask(rawRun, expect): Promise<TaskScore>`\n\nJoins a `RawRun` with a task's `expect` criteria and reads the isolated\ntokentracker ledger to produce a `TaskScore` with cost, latency, and token\ntotals.\n\n### `buildScorecard(scored, meta): Scorecard`\n\nAggregates an array of `{ rawRun, task, score }` entries into a `Scorecard`\nwith summary statistics: `successRate`, `totalUsdActual`, `meanLatencyMs`,\n`totalTokens`, `passCount`, `failCount`.\n\n### `compareScorecards(baseline, candidate): ScorecardDelta`\n\nProduces per-task and summary deltas. Flags regressions where:\n- task success flips from true to false\n- per-task cost increases more than 10%\n- per-task latency increases more than 20%\n\nSets `hasRegression: true` and `regressionCount` on the summary when any\nthreshold is breached.\n\n---\n\n## Metrics\n\n| Metric | Description | Source |\n|--------|-------------|--------|\n| `successRate` | Fraction of tasks that met their `expect` criteria | `expect.status` match (deterministic) or `verifyCommand` exit 0 (live) |\n| `cost` (USD) | Actual spend per task, net of cache savings | `summarizeUsageLedger` from `@ai-craft/tokentracker` |\n| `latencyMs` | Wall-clock time for the adapter run | `Date.now()` delta around `adapter.run()` |\n| `tokens` | Input + output + total token counts | Summed from raw `AiCraftUsageLedgerRecord[]` |\n| `retries` | Re-review / re-hunt / re-verify loop counts | `artifact.telemetry` from `WorkflowArtifact` |\n\n---\n\n## Modes\n\n### Deterministic\n\nUses a `makeScriptedLLM` factory backed by `createFake` from\n`@ai-craft/agent-llm`. Per-agent scripted `StreamEvent[][]` arrays control\nevery model turn. No network calls, no cost — CI-safe.\n\nSuitable for: harness correctness tests, regression gates, local development.\n\n### Live\n\nUses a real `LLMClient` (`createAnthropic` / `createOpenAI` / `createFoundry`).\nTask success is determined by `expect.verifyCommand` exit code. Requires\n`--provider` plus valid credentials. Prints an estimated-cost warning before\nexecuting.\n\nSuitable for: end-to-end quality validation, real-token cost profiling.\n\nLive mode requires Node 22.14.0 and provider credentials\n(`ANTHROPIC_API_KEY` or `~/.config/ai-agent/credentials.json`).\n\n---\n\n## Development\n\nNode 22.14.0 is required. Use the `NX_DAEMON=false` prefix for all NX\ncommands to avoid daemon-related interference.\n\n```bash\n# Run tests\nNX_DAEMON=false pnpm exec nx test agent-bench -- --run\n\n# Typecheck source\nNX_DAEMON=false pnpm exec nx run agent-bench:typecheck\n\n# Typecheck specs\nNX_DAEMON=false pnpm exec nx run agent-bench:typecheck-spec\n\n# Build (esbuild CLI bin + type declarations)\nNX_DAEMON=false pnpm exec nx build agent-bench\n```\n\nTest count: 39 tests across 6 spec files covering scripted LLM, runner ledger\nisolation, scorer math, compare regression detection, report output, and the\norchestration suite.\n\n---\n\n## Spec\n\n`docs/ai/specs/0008-agent-bench.md` (SPEC-0008-agent-bench, status: approved)\n\n---\n\n## License\n\nSee root `LICENSE` file.\n","readmeFilename":"README.md"}