{"_id":"@danypops/pi-eval-harness","_rev":"3-698d083bf5db5c0d862a287bf08ddda0","name":"@danypops/pi-eval-harness","dist-tags":{"latest":"0.3.0"},"versions":{"0.1.0":{"name":"@danypops/pi-eval-harness","version":"0.1.0","keywords":["pi","pi-extension","testing","eval","agent-evaluation","tool-call"],"license":"MIT","_id":"@danypops/pi-eval-harness@0.1.0","maintainers":[{"name":"danypops","email":"danypops@protonmail.com"}],"homepage":"https://github.com/DanyPops/pi-integral/tree/main/packages/pi-eval-harness#readme","bugs":{"url":"https://github.com/DanyPops/pi-integral/issues"},"dist":{"shasum":"a06d32c0dcf1217b7dc3f30e5f74eef19679af55","tarball":"https://registry.npmjs.org/@danypops/pi-eval-harness/-/pi-eval-harness-0.1.0.tgz","fileCount":18,"integrity":"sha512-DL91H8Fds1PuzuR2d03dExnVl99miQ8TugURH8Wvyqr6pB30L1S9gOWcGLogK9ffn+mp8KPJiqVEFMcyqDU2GA==","signatures":[{"sig":"MEUCID63/m2LDNreQ4qQNVv3EP4KhqHFDOxcsX+vBhrv57wpAiEAl/DGTxHb+L9T9MHoYaP1S5tQcQZkYULwLfv4J5BXpR8=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":30099},"main":"./dist/index.js","type":"module","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","default":"./dist/index.js"},"./package.json":"./package.json"},"gitHead":"af476ee37b408cb5e3f18b5a4d5dea5cd72cf08f","scripts":{"test":"bun run build && bun test test","build":"rm -rf dist && tsc -p tsconfig.build.json","format":"biome format --write .","typecheck":"tsc --noEmit"},"_npmUser":{"name":"danypops","email":"danypops@protonmail.com"},"repository":{"url":"git+https://github.com/DanyPops/pi-integral.git","type":"git","directory":"packages/pi-eval-harness"},"_npmVersion":"11.14.1","description":"Scores a real agent run's own tool-call behavior -- AND/OR tool-call matching, graduated checker composition, and turn/tool-call/token-usage rollups -- over Pi's own real AgentSessionEvent stream (e.g. from @danypops/pi-process-harness).","directories":{},"sideEffects":false,"_nodeVersion":"22.22.2","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"typebox":"*","bun-types":"latest","typescript":"^5.9.2","@types/node":"^22.0.0","@biomejs/biome":"^2.5.6","@earendil-works/pi-ai":"*","@danypops/pi-process-harness":"workspace:*","@earendil-works/pi-coding-agent":"*"},"peerDependencies":{"@earendil-works/pi-coding-agent":"*"},"_npmOperationalInternal":{"tmp":"tmp/pi-eval-harness_0.1.0_1787501741519_0.48419773095797436","host":"s3://npm-registry-packages-npm-production"}},"0.2.0":{"name":"@danypops/pi-eval-harness","version":"0.2.0","keywords":["pi","pi-extension","testing","eval","agent-evaluation","tool-call"],"license":"MIT","_id":"@danypops/pi-eval-harness@0.2.0","maintainers":[{"name":"danypops","email":"danypops@protonmail.com"}],"homepage":"https://github.com/DanyPops/pi-integral/tree/main/packages/pi-eval-harness#readme","bugs":{"url":"https://github.com/DanyPops/pi-integral/issues"},"dist":{"shasum":"06f7918ac02e893d0af6345b2389eb7d00484b79","tarball":"https://registry.npmjs.org/@danypops/pi-eval-harness/-/pi-eval-harness-0.2.0.tgz","fileCount":30,"integrity":"sha512-LeirzcYdv+RTtJRfvPW8sbWx/1lgxFsRZ/WgA7oCLVZozuFF8ObgW43zi4VinjoPCYYA8tE+prEpiaUon3znDQ==","signatures":[{"sig":"MEUCICmQ58O7I6e1LTK7o+/P/kH9qIWlPoXtJr35xblluA5XAiEApfSa/mP4T/cIMFP4lX4OYm+oSw+eo7DuEY7kgkJ5HH4=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@danypops%2fpi-eval-harness@0.2.0","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"unpackedSize":66685},"main":"./dist/index.js","type":"module","types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","default":"./dist/index.js"},"./package.json":"./package.json"},"gitHead":"af13442512f2273c74ae8bb49bad86135af65221","scripts":{"test":"bun run build && bun test test","build":"rm -rf dist && tsc -p tsconfig.build.json","format":"biome format --write .","typecheck":"tsc --noEmit"},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:67dcb60d-b1e0-4202-ad7e-ad5f4c8aee69"}},"repository":{"url":"git+https://github.com/DanyPops/pi-integral.git","type":"git","directory":"packages/pi-eval-harness"},"_npmVersion":"12.0.2","description":"Scores a real agent run's own tool-call behavior -- AND/OR tool-call matching, graduated checker composition, and turn/tool-call/token-usage rollups -- over Pi's own real AgentSessionEvent stream (e.g. from @danypops/pi-process-harness).","directories":{},"sideEffects":false,"_nodeVersion":"24.19.0","publishConfig":{"access":"public"},"_hasShrinkwrap":false,"devDependencies":{"typebox":"*","bun-types":"latest","typescript":"^5.9.2","@types/node":"^22.0.0","@biomejs/biome":"^2.5.6","@earendil-works/pi-ai":"*","@danypops/pi-process-harness":"0.2.0","@earendil-works/pi-coding-agent":"*"},"peerDependencies":{"@earendil-works/pi-coding-agent":"*"},"_npmOperationalInternal":{"tmp":"tmp/pi-eval-harness_0.2.0_1787508712002_0.2498414544075871","host":"s3://npm-registry-packages-npm-production"}},"0.3.0":{"name":"@danypops/pi-eval-harness","version":"0.3.0","description":"Scores a real agent run's own tool-call behavior -- AND/OR tool-call matching, graduated checker composition, and turn/tool-call/token-usage rollups -- over Pi's own real AgentSessionEvent stream (e.g. from @danypops/pi-process-harness).","license":"MIT","type":"module","main":"./dist/index.js","sideEffects":false,"types":"./dist/index.d.ts","exports":{".":{"types":"./dist/index.d.ts","default":"./dist/index.js"},"./package.json":"./package.json"},"scripts":{"build":"rm -rf dist && tsc -p tsconfig.build.json","test":"bun run build && bun test test","typecheck":"tsc --noEmit","format":"biome format --write ."},"repository":{"type":"git","url":"git+https://github.com/DanyPops/pi-integral.git","directory":"packages/pi-eval-harness"},"homepage":"https://github.com/DanyPops/pi-integral/tree/main/packages/pi-eval-harness#readme","bugs":{"url":"https://github.com/DanyPops/pi-integral/issues"},"peerDependencies":{"@earendil-works/pi-coding-agent":"*"},"devDependencies":{"@biomejs/biome":"^2.5.6","@danypops/pi-process-harness":"0.2.0","@earendil-works/pi-ai":"*","@earendil-works/pi-coding-agent":"*","@types/node":"^22.0.0","bun-types":"latest","typebox":"*","typescript":"^5.9.2"},"publishConfig":{"access":"public"},"keywords":["pi","pi-extension","testing","eval","agent-evaluation","tool-call"],"gitHead":"bedff020ed6af4c72d880442c25e36c47e4cd1cd","_id":"@danypops/pi-eval-harness@0.3.0","_nodeVersion":"24.19.0","_npmVersion":"12.0.2","dist":{"integrity":"sha512-KDFBglwAtAP5cVhoXomq/vpb53f0i8t+vl1fwsBIKYWEiq+RlS59hpC4yceHFqQbSA3Px6g5N6X9oA9klG4dOg==","shasum":"cbbb0c800558c1d8849f7438c3fc728c1a18feed","tarball":"https://registry.npmjs.org/@danypops/pi-eval-harness/-/pi-eval-harness-0.3.0.tgz","fileCount":30,"unpackedSize":70271,"attestations":{"url":"https://registry.npmjs.org/-/npm/v1/attestations/@danypops%2fpi-eval-harness@0.3.0","provenance":{"predicateType":"https://slsa.dev/provenance/v1"}},"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCICFotKs71Pfivqv712RFEudiyM4x4RuYCqx4ij+tRTOxAiEAgirX6FruH0n+M0AmtqcVSZyBI5OVI/dbOEriWmJZWlQ="}]},"_npmUser":{"name":"GitHub Actions","email":"npm-oidc-no-reply@github.com","trustedPublisher":{"id":"github","oidcConfigId":"oidc:67dcb60d-b1e0-4202-ad7e-ad5f4c8aee69"}},"directories":{},"maintainers":[{"name":"danypops","email":"danypops@protonmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/pi-eval-harness_0.3.0_1787511393496_0.7311039241719426"},"_hasShrinkwrap":false}},"time":{"created":"2026-08-23T16:15:41.339Z","modified":"2026-08-23T18:56:33.992Z","0.1.0":"2026-08-23T16:15:41.666Z","0.2.0":"2026-08-23T18:11:52.203Z","0.3.0":"2026-08-23T18:56:33.649Z"},"bugs":{"url":"https://github.com/DanyPops/pi-integral/issues"},"license":"MIT","homepage":"https://github.com/DanyPops/pi-integral/tree/main/packages/pi-eval-harness#readme","keywords":["pi","pi-extension","testing","eval","agent-evaluation","tool-call"],"repository":{"type":"git","url":"git+https://github.com/DanyPops/pi-integral.git","directory":"packages/pi-eval-harness"},"description":"Scores a real agent run's own tool-call behavior -- AND/OR tool-call matching, graduated checker composition, and turn/tool-call/token-usage rollups -- over Pi's own real AgentSessionEvent stream (e.g. from @danypops/pi-process-harness).","maintainers":[{"name":"danypops","email":"danypops@protonmail.com"}],"readme":"# @danypops/pi-eval-harness\n\nScores a real agent run's own tool-call behavior over Pi's real `AgentSessionEvent` stream --\nAND/OR tool-call matching, graduated checker composition, and turn/tool-call/token-usage\nrollups. Ported from Alef's own `packages/core/eval` evaluation framework, adapted to operate\ndirectly on the real event union any `@danypops/pi-process-harness` (or `pi --mode rpc`) run\nalready produces, instead of a bespoke OTel span format.\n\n## Usage\n\n```ts\nimport { deriveTurns, expectsAll, extractToolExecutions, summarizeRunUsage } from \"@danypops/pi-eval-harness\";\n\nconst executions = extractToolExecutions(sessionEvents);\n// [{ toolCallId, toolName, args, result, isError }, ...] in completion order\n\nconst checker = expectsAll([{ tool: \"search_code\", target: { pattern: \"TODO\" } }]);\nconst result = await checker.check({ executions });\n// { pass, score, errors }\n\nconst turns = deriveTurns(sessionEvents);\nconst usage = summarizeRunUsage(turns);\n// { turns, tokensIn, tokensOut, cacheReadTokens, costUsd, toolCalls, toolNames }\n```\n\n## Scope\n\n- `extractToolExecutions` -- pairs `tool_execution_start`/`tool_execution_end` by `toolCallId`\n  into one real completed call per pair, in completion order.\n- `matchesToolCall`/`describeToolCall` -- whether one completed execution satisfies a `ToolCall`\n  expectation (tool name, target args, produced output), and a human-readable description of it.\n- `expectsAll`/`expectsAny`/`all` -- `Checker`s with AND/OR/composed semantics and graduated\n  `CheckerResult` scoring (0.0 hard fail, 1.0 full pass).\n- `deriveTurns`/`summarizeRunUsage` -- one `Turn` per real `turn_end` event (model, token usage,\n  cost, ordered tool-call names), rolled up into whole-run totals.\n\n## Testing your own Checkers: fixture self-test discipline\n\nPorted from Alef's own `Evaluation.fixture`/`FixtureSet` discipline: a `Checker` must be proven\ncorrect against a small, hand-authored, known-good `ToolExecution[]` fixture -- with **zero**\nlive process spawns or LLM calls -- before it is ever trusted against a real run. This package's\nown test suite follows exactly that pattern (see `test/checker.test.ts` and `test/tool-call.test.ts`):\nbuild a `ToolExecution` fixture by hand, call `checker.check({ executions })` directly, assert the\nexact `{ pass, score, errors }` you expect. No helper wraps this -- the whole point is that\n`Checker.check()` is already a pure, synchronous-or-trivially-awaitable function; wrapping it\nwould only hide the assertion, not simplify it.\n\nA `Checker` that only ever gets exercised against a real, expensive `pi-process-harness` run has\nno fast, deterministic proof it is correct in isolation -- write the fixture test first.\n\n## Real live-LLM smoke test (opt-in, real cost -- never run automatically)\n\n`scripts/real-llm-smoke-test.ts` spawns a genuine `pi` process against a real model (no faux\nprovider, no scripted tool calls) and runs the captured trace through this package's own\nmatching/rollup functions -- confirming they handle a real model's own event shape, not just the\nfaux provider's conveniences every other test here relies on. Run explicitly:\n\n```sh\nbun scripts/real-llm-smoke-test.ts\n```\n\nRequires real provider credentials already configured in your own ambient Pi profile (this\nscript deliberately runs with `isolatedHome: false`) and incurs a real, billed API call.\n\n## License\n\nMIT\n","readmeFilename":"README.md"}