---
title: "Evals"
description: "Define repeatable scored checks for an Eve agent with defineEval and run them with eve eval."
---

An eval is a scored check that runs your agent against real sessions and grades the result. Use it to catch regressions when you change a prompt or a tool: compare the output against expected text, JSON, SQL, or behavior, and optionally ship the results to Braintrust.

## `defineEval`

Eve discovers evals under the app-root `evals/` directory, in `.eval.ts` files. The file path is the eval's identity, so you don't author an `id` or `name`.

```text
my-agent/
├── agent/
├── evals/
│   ├── smoke.eval.ts
│   └── weather.eval.ts
└── package.json
```

```ts title="evals/weather.eval.ts"
import { defineEval } from "eve/evals";
import { Checks } from "eve/evals/checks";
import { Run } from "eve/evals/scores";

export default defineEval({
  description: "Basic message and tool-usage coverage for the weather agent.",
  cases: [
    { id: "brooklyn-forecast", input: "What is the weather in Brooklyn?", expected: "Sunny" },
  ],
  checks: [Checks.didNotFail(), Checks.toolCalled("get_weather")],
  scores: [Run.didNotFail()],
});
```

Every eval needs `scores` (an empty array is fine) and either `cases` or `load`. The rest are optional: `description`, `task`, `checks`, `requires`, `model`, `thresholds`, `modelOptions`, `tags`, `metadata`, `maxConcurrency`, `timeoutMs`, `reporters`. The init template adds `evals/**/*.ts` to `tsconfig.json`, so your eval code type-checks alongside the app.

`model` is only required when a model-backed scorer (one of the `Autoevals` wrappers) is present without its own per-scorer model override — and it's the scorer model, not the agent's. Eve only uses it for model-backed scoring, never to swap out the agent under test. Pass a string id (e.g. `"anthropic/claude-opus-4.8"`) to route through the Vercel AI Gateway, or hand it an AI SDK model instance to use that directly.

## Checks and scores

Evals grade a case on two distinct tiers:

- **Checks are hard assertions.** Any failed check marks the case failed and `eve eval` exits non-zero. Use them for the things that must hold — the run completed, the right tool ran, the output parses.
- **Scores are soft data.** They land in reports and artifacts, and a below-threshold score marks the case `scored` — visible but not fatal, unless you pass `--strict`.

Both exist at the eval level (applied to every case) and the case level (appended to the eval's). Built-in checks live on `eve/evals/checks`:

| Check                                                         | Asserts                                                                           |
| ------------------------------------------------------------- | --------------------------------------------------------------------------------- |
| `Checks.completed()`                                          | The run did not fail and did not park on unanswered HITL input                    |
| `Checks.waiting()`                                            | The run parked on HITL input (for approval-shaped evals)                          |
| `Checks.didNotFail()`                                         | No terminal failure and no `turn.failed`/`step.failed` events (parked runs pass)  |
| `Checks.messageIncludes(token)`                               | Joined assistant text contains `token` (string or RegExp)                         |
| `Checks.outputEquals(value)` / `Checks.outputMatches(schema)` | Deep equality / Standard Schema (e.g. Zod) validation of the parsed output        |
| `Checks.toolCalled(name, opts?)`                              | A matching tool call happened (`input`, `output`, `isError`, `times` constraints) |
| `Checks.toolNotCalled(name)`                                  | No call to `name`                                                                 |
| `Checks.toolOrder([...names])`                                | Tool names appear in order (other calls may interleave)                           |
| `Checks.noFailedActions()`                                    | No tool, subagent, or skill action reported a failure                             |
| `Checks.subagentCalled(name, opts?)`                          | A subagent delegation happened (`remoteUrl`, `output` constraints)                |
| `Checks.event(predicate, label)`                              | Escape hatch: any predicate over the typed event stream                           |

Matcher options accept a literal (objects partial-deep-match), a RegExp, or a function — return a boolean to act as a predicate, or return a value to compare against (handy for runner-assigned values like environment-provided URLs):

```ts
Checks.toolCalled("bash", { input: { command: /^pwd/ }, isError: false, times: 1 });
Checks.subagentCalled("weather", {
  remoteUrl: () => process.env.WEATHER_AGENT_URL!,
  output: /72F/,
});
```

A custom check is a plain function receiving `{ case, result, target }` and returning `{ name, passed, message? }`.

Note that `result.status` is `"waiting"` whenever the session is left open for a next message — the normal end state of a successful turn. Parking on unanswered HITL input is tracked separately as `result.derived.parked`, which is what `Checks.completed()` and `Checks.waiting()` key off.

## Cases and tasks

A case is either a data case or a scripted case.

Data cases pair an `input` with an `expected`, plus optional `checks`, `scores`, `requires`, tags, and metadata. `input` can be a string or an object. `expected` is optional, which is handy when you only care about behavior. List cases inline, or load them dynamically:

```ts
import { defineEval } from "eve/evals";
import { loadYaml } from "eve/evals/loaders";
import { Text } from "eve/evals/scores";

export default defineEval({
  async load() {
    const doc = await loadYaml("evals/data/cases.yaml");
    return (doc.evals as readonly { task: string; prompt: string; sql: string }[]).map((row) => ({
      id: row.task,
      input: row.prompt,
      expected: row.sql,
    }));
  },
  scores: [Text.exact()],
});
```

The loaders (`loadJson`, `loadYaml` from `eve/evals/loaders`) resolve paths relative to the app root. They're meant for fixtures, not runtime agent code. For the task itself, `task.prompt` covers the single-string case, `task.messages` sets up a static multi-turn conversation, `task.run` gives you imperative control flow, and `task.parseOutput` lets you compare a transformed result rather than the raw final message.

Scripted cases define their own `run(ctx)` and do not need `input`. Use them for smoke-style behavior: multi-turn branching, HITL approvals, structured output, attachments, and multiple independent sessions.

```ts
import { defineEval } from "eve/evals";
import { Checks } from "eve/evals/checks";

export default defineEval({
  checks: [Checks.didNotFail()],
  scores: [],
  cases: [
    {
      id: "approve-tool",
      async run({ session }) {
        await session.send("run `pwd`");
        session.expectInputRequests({ toolName: "bash" });
        const turn = await session.respondAll("approve");
        return turn.message;
      },
      checks: [Checks.toolCalled("bash", { input: { command: /pwd/ } })],
    },
  ],
});
```

`ctx.session` is the primary `EveEvalSession`; `ctx.newSession()` creates another independent session against the same target. `session.send()` accepts the same input as `ClientSession.send()`, `session.sendFile(text, path, mediaType?)` attaches a local file as a data URL, and `turn.expectOk()` throws only when the turn ended failed. Events from every eval session are captured in the case result and artifacts. `ctx.log(message)` records debug lines into the case artifact; `--verbose` also streams them to stdout as cases run.

## Targets and requirements

An eval target is always an HTTP URL. `eve eval` starts a local dev server, while `eve eval --url <url>` runs against an existing server or deployment. The runner polls `/eve/v1/health`, verifies `/eve/v1/info`, and exposes the live target as `ctx.target`:

```ts
export default defineEval({
  requires: ["mockModels", "devRoutes"],
  scores: [],
  cases: [
    {
      id: "heartbeat",
      async run({ target }) {
        const { sessionIds } = await target.dispatchSchedule("heartbeat");
        const session = await target.attachSession(sessionIds[0]!);
        return session.events;
      },
    },
  ],
});
```

`target.fetch(path, init)` performs an authenticated fetch against the target for channel/webhook ingress. `target.dispatchSchedule(id)` uses the dev-only schedule route, and `target.attachSession(sessionId, { startIndex? })` consumes one turn from a session created by a channel or schedule so its events participate in checks and scores.

Use `requires` to declare assumptions the runner verifies before executing a case:

| Requirement    | Means                                                                 |
| -------------- | --------------------------------------------------------------------- |
| `"mockModels"` | `/eve/v1/info` reports the deterministic mock model adapter is active |
| `"devRoutes"`  | `/eve/v1/info` reports dev-only routes are mounted                    |
| `"env:NAME"`   | The eval process has environment variable `NAME` set                  |

Eval-level requirements apply to every case; case-level requirements append. Unmet requirements produce a visible `skipped` verdict and do not affect the exit code. Pass `--no-skips` when a CI leg must prove full coverage. For local targets, `--mock-models` starts the dev server with deterministic authored models; it is invalid with `--url` because remote target capabilities are discovered, not set by the runner.

## Scorers and rubrics

Scorers live in namespaces on `eve/evals/scores`. Pick the cheapest one that captures what "correct" means here. The deterministic scorers run instantly for free; an LLM judge runs once per case and burns tokens, so save it for when nothing simpler will do.

| Need                                                       | Use                                                                                           |
| ---------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| Grade agent behavior (run succeeded, used the right tools) | `Run.didNotFail()`, `Run.usedTool(name, opts?)`, `Run.usedNoTools()`, `Run.maxToolCalls(max)` |
| Exact string match                                         | `Text.exact()`, `Text.includes()`                                                             |
| Fuzzy text match (typos, whitespace)                       | `Text.levenshtein()`                                                                          |
| Exact JSON match                                           | `Json.deepEqual()`                                                                            |
| Exact SQL match (after normalization)                      | `Sql.exactNormalized()`                                                                       |
| LLM-judged factual correctness vs an expected answer       | `Autoevals.factuality()`                                                                      |
| LLM-judged summary quality                                 | `Autoevals.summary()`                                                                         |
| LLM-judged SQL semantic equivalence                        | `Autoevals.sql()`                                                                             |
| LLM-judged free-form criteria (no `expected` to match)     | `Autoevals.closedQA({ criteria: "..." })`                                                     |

Each scorer gets the flattened `input`, `output`, and `expected` strings along with the full case and task result — including derived facts: typed tool calls (name, input, output, error state), subagent calls, HITL input requests, and whether the run parked. `Run.usedTool` accepts the same matcher options as `Checks.toolCalled`, so you can grade tool-input quality fractionally where a check would assert it absolutely. Return `null` from a scorer to skip a case.

### Thresholds

By default a scorer has to hit an exact match to pass. `thresholds` loosens that, mapping each scorer name to the minimum score you'll accept:

```ts
defineEval({
  cases: [{ id: "hello", input: "Hello", expected: "Hello" }],
  scores: [Run.didNotFail(), Text.includes()],
  thresholds: { "run.didNotFail": 1, "text.includes": 0.5 },
});
```

`maxConcurrency` caps parallelism and `timeoutMs` bounds each case. Leave them off and `eve eval` runs up to 8 cases per eval at once. For provider-specific scorer-model settings, use `modelOptions.providerOptions`. Individual Autoevals scorers can also take their own `model` / `modelOptions`, which win over the eval default.

## Reporters

Eval-level reporters live on `eve/evals/reporters`. `Braintrust(...)` uploads eval results to Braintrust experiments, and `JUnit({ filePath })` writes JUnit XML for CI annotations. Eve still runs and scores everything itself; reporters just ship the results out. Checks log as binary scores under a `check:` prefix so experiments diff check regressions the same way they diff score regressions. Eval and case `metadata` ride along to reporters, and the CLI prints a console summary by default.

## Running evals

```bash
eve eval                       # run all discovered evals locally
eve eval weather smoke         # run selected evals
eve eval --url https://<app>   # target a remote app instead of a local host
eve eval --mock-models         # local dev target uses deterministic mock models
eve eval --tag fast            # only cases (or evals) carrying a tag
eve eval --case brooklyn-forecast  # only specific case ids
eve eval --strict              # below-threshold scores also fail the exit code
eve eval --no-skips            # unmet requirements fail instead of skipping
eve eval --junit .eve/junit.xml  # write JUnit XML
eve eval --list                # print discovered evals and cases without running
eve eval --verbose             # stream per-case ctx.log lines to stdout
eve eval --json                # machine-readable output
eve eval --skip-report         # skip eval-defined reporters (e.g. Braintrust)
```

Exit codes: `0` when every case passed its checks (and thresholds, under `--strict`), `1` when any case failed (a failed check, an execution error, or a strict threshold miss), `2` for configuration errors. Each run drops per-eval artifacts under `.eve/evals/<timestamp>-<eval-id>/`, including per-case check results and verdicts.

## A good baseline

Most apps do fine with a small smoke eval. Assert behavior with `Checks.didNotFail()` plus one or two content checks, keep case fixtures in `evals/data/`, and only reach for Braintrust once you actually need shared result review or experiment history. In CI, run `eve eval --strict` so threshold misses fail the build too.

## What to read next

- [Tools](../tools): the surface most evals assert on
- [instrumentation.ts](./instrumentation): tracing and debugging
