{"_id":"@arthurcarlson/evalgate","name":"@arthurcarlson/evalgate","dist-tags":{"latest":"1.0.0"},"versions":{"1.0.0":{"name":"@arthurcarlson/evalgate","version":"1.0.0","description":"LLM prompt eval gate for CI: run YAML-defined eval cases against your prompts on every PR, post a pass/fail report, and fail the build when outputs regress.","type":"module","license":"MIT","engines":{"node":">=20"},"bin":{"evalgate":"dist/cli.js"},"scripts":{"build":"npm run build:cli && npm run build:action","build:cli":"tsup","build:action":"ncc build src/action.ts -o dist --no-cache --no-source-map-register","test":"vitest run","test:watch":"vitest","prepublishOnly":"npm run build && npm test"},"keywords":["evals","llm","prompt testing","ci","github action","anthropic","openai","prompt engineering","regression testing"],"author":{"name":"Arthur Carlson","url":"https://linkedin.com/in/arthurcarlsonn"},"repository":{"type":"git","url":"git+https://github.com/arthurcarlsonn/evalgate.git"},"dependencies":{"@actions/core":"^1.11.1","@actions/github":"^6.0.0","ajv":"^8.17.1","fast-glob":"^3.3.2","yaml":"^2.6.1","zod":"^3.24.1"},"devDependencies":{"@types/node":"^22.10.0","@vercel/ncc":"^0.38.3","tsup":"^8.3.5","typescript":"^5.7.2","vitest":"^2.1.8"},"_id":"@arthurcarlson/evalgate@1.0.0","gitHead":"2c6910f2af3e80e12a4804689c7f2e0f7fa2a1ca","bugs":{"url":"https://github.com/arthurcarlsonn/evalgate/issues"},"homepage":"https://github.com/arthurcarlsonn/evalgate#readme","_nodeVersion":"22.22.0","_npmVersion":"10.9.4","dist":{"integrity":"sha512-Uo3v8RVskhWS6J/5uOe4Fs7qLZPWwkgJK5ZgjonrEaDzwm07jyLDu84P/E6TyoK4CK2o8oZnmX4S8mvUj6drNw==","shasum":"4c8a7b3ad2d3f72144c4ce240d716d6f2bbd7b5b","tarball":"https://registry.npmjs.org/@arthurcarlson/evalgate/-/evalgate-1.0.0.tgz","fileCount":7,"unpackedSize":2093888,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEQCIHZF2rpeEZ2qoKZ+4Ilgl0hGecT+I3sX0pSY6NigWbj2AiBrMdVYK2txHtfav4WLU6HZ5eRV/3tHRioRFfVVVI8Haw=="}]},"_npmUser":{"name":"arthurcarlson","email":"arthurcarlsondesouzabarbosa@gmail.com"},"directories":{},"maintainers":[{"name":"arthurcarlson","email":"arthurcarlsondesouzabarbosa@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/evalgate_1.0.0_1783215610851_0.36426511808689255"},"_hasShrinkwrap":false}},"time":{"created":"2026-07-05T01:40:10.740Z","1.0.0":"2026-07-05T01:40:11.003Z","modified":"2026-07-05T01:40:11.179Z"},"maintainers":[{"name":"arthurcarlson","email":"arthurcarlsondesouzabarbosa@gmail.com"}],"description":"LLM prompt eval gate for CI: run YAML-defined eval cases against your prompts on every PR, post a pass/fail report, and fail the build when outputs regress.","homepage":"https://github.com/arthurcarlsonn/evalgate#readme","keywords":["evals","llm","prompt testing","ci","github action","anthropic","openai","prompt engineering","regression testing"],"repository":{"type":"git","url":"git+https://github.com/arthurcarlsonn/evalgate.git"},"author":{"name":"Arthur Carlson","url":"https://linkedin.com/in/arthurcarlsonn"},"bugs":{"url":"https://github.com/arthurcarlsonn/evalgate/issues"},"license":"MIT","readme":"# evalgate\n\nUnit tests exist for code; prompts get vibes. **evalgate** runs YAML-defined eval cases against your LLM prompts on every pull request, posts a pass/fail report as a sticky PR comment, and fails the build when outputs regress. A one-word prompt change that breaks tone, format, or a JSON contract becomes a red X on the PR.\n\n[GitHub Marketplace badge — pending listing]\n\n## 60-second quickstart\n\n**1.** Copy the workflow:\n\n```yaml\n# .github/workflows/evals.yml\nname: Prompt evals\non: pull_request\npermissions:\n  contents: read\n  pull-requests: write\njobs:\n  evalgate:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - uses: arthurcarlsonn/evalgate@v1\n        with:\n          config: evals/**/*.eval.yaml\n        env:\n          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}\n          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}\n```\n\n**2.** Add one eval file:\n\n```yaml\n# evals/support.eval.yaml\ndefaults:\n  provider: anthropic\n  model: claude-haiku-4-5\n  temperature: 0\n\ncases:\n  - id: refund-policy-tone\n    prompt: |\n      You are the support agent for {{brand}}.\n      Customer asks: \"{{question}}\"\n      Reply in under 120 words.\n    vars:\n      brand: GlowLab\n      question: What is your refund policy?\n    assert:\n      - type: contains\n        value: \"30 days\"\n      - type: not_contains\n        value: lifetime guarantee\n      - type: llm_judge\n        rubric: Reply is polite, mentions the refund window, and does not invent policies.\n```\n\n**3.** Add `ANTHROPIC_API_KEY` to your repo secrets. Open a PR — evalgate comments with the table and gates the merge.\n\n## Try it locally, free\n\n```bash\nnpx @arthurcarlson/evalgate run --dry-run --config \"evals/**/*.eval.yaml\"\n```\n\n`--dry-run` uses the mock provider: zero API calls, zero cost. Canned outputs are read from `mocks/<case-id>.txt` next to your eval file, so you can develop assertions offline. The `examples/` folder in this repo runs green out of the box:\n\n```bash\nnpx @arthurcarlson/evalgate run --dry-run --config \"examples/evals/*.eval.yaml\"\n```\n\n## Config reference\n\n```yaml\ndefaults:            # optional, applies to every case\n  provider: anthropic   # anthropic | openai | openrouter | mock\n  model: claude-haiku-4-5\n  temperature: 0        # 0-2, default 0\n  max_tokens: 1024\n\ncases:\n  - id: my-case            # required, unique, [a-zA-Z0-9._-]\n    prompt: \"inline {{var}}\"   # OR prompt_file: path/relative/to/this/file\n    vars: { var: value }       # {{var}} is literal string replacement;\n                               # unknown variables fail before any API call\n    provider: openai           # per-case overrides of any default\n    model: gpt-4o-mini\n    temperature: 0.2\n    max_tokens: 512\n    assert:                    # at least one; case passes only if all pass\n      - type: contains\n        value: \"...\"\n```\n\n## Assertions\n\n| Type | Passes when |\n|------|-------------|\n| `contains` | Output includes the value (case-sensitive) |\n| `not_contains` | Output does not include the value |\n| `icontains` | Case-insensitive contains |\n| `regex` | JS regex (as string) matches |\n| `json_valid` | Output parses as JSON (one surrounding code fence is stripped) |\n| `json_schema` | Parsed JSON validates against `schema_file` (ajv, draft 2020-12) |\n| `max_chars` | Output length ≤ value |\n| `llm_judge` | Judge model (temperature 0, fixed template) answers `PASS`; anything else fails with the returned reason |\n\n## Action inputs and outputs\n\n| Input | Default | |\n|---|---|---|\n| `config` | `evals/**/*.eval.yaml` | Eval file glob |\n| `min_pass_rate` | `100` | Gate: fail the run below this pass rate (%) |\n| `comment` | `true` | Post/update the sticky PR comment |\n| `max_cases` | `50` | Cost guard: hard cap on cases per run |\n| `fail_on_missing_key` | `true` | Fail fast when a needed provider key is missing |\n\nOutputs: `pass_rate`, `passed`, `failed`, `total`. The report also lands in the job summary.\n\n## Providers\n\nKeys are read from env only — never from inputs or flags — and masked with `core.setSecret`:\n\n| Provider | Env var |\n|---|---|\n| `anthropic` | `ANTHROPIC_API_KEY` |\n| `openai` | `OPENAI_API_KEY` |\n| `openrouter` | `OPENROUTER_API_KEY` |\n| `mock` | none |\n\n## Cost notes\n\nDefaults are chosen to keep runs cheap: temperature 0, haiku-class model, `max_cases` cap at 50, concurrency 4, retries only on 429/5xx. A 20-case suite on a haiku-class model typically completes in well under 90 seconds for pennies.\n\n## Troubleshooting\n\n- **\"Missing provider key(s)\"** — add the secret to the workflow `env:` block. Set `fail_on_missing_key: false` to run anyway (those cases will fail at runtime).\n- **No PR comment** — the job needs `permissions: pull-requests: write` and `GITHUB_TOKEN` in `env:`; on non-PR events the comment is skipped with a notice.\n- **Gate failed but you disagree** — lower `min_pass_rate`, or fix the prompt. That's the point.\n\n## CLI\n\n```\nevalgate run [--config <glob>] [--dry-run] [--min-pass-rate <n>] [--max-cases <n>] [--json]\n```\n\nExit codes: `0` gate passed · `1` runtime/config error · `2` gate failed.\n\n## License\n\nMIT © Arthur Carlson\n","readmeFilename":"README.md","_rev":"1-a6c0b5724088373cca0c4e1ca5464990"}