---
title: "Assertions"
description: "Scoped methods, value assertions, the matcher mini-language, and gate vs soft severity."
---

Assertions are how an eval grades what its `test(t)` function produced. Each one records a result and returns a chainable handle. The runner reads every recorded result to compute the verdict, so a single run reports every failing assertion rather than dying on the first. There are two deterministic surfaces: scoped methods and `t.check` for grading a specific value. For model-graded assertions, see [Judge](./judge).

## Scoped assertions

Scoped assertions take no explicit value and gate by default. Assertions on `t` inspect the whole run after `test` finishes. Session assertions snapshot that session when called, and turn assertions inspect one immutable response. A scope is **parked** when it paused on unanswered human-in-the-loop (HITL) input.

| Assertion                                             | Asserts                                                                       |
| ----------------------------------------------------- | ----------------------------------------------------------------------------- |
| `t.succeeded()`                                       | The run did not fail and did not park on unanswered HITL input                |
| `t.parked()`                                          | The run cleanly parked on HITL input                                          |
| `t.messageIncludes(token)`                            | Joined assistant text contains `token` (string or RegExp)                     |
| `turn.outputEquals(value)` / `.outputMatches(schema)` | Deep equality or Standard Schema validation of turn/session structured output |
| `t.calledTool(name, opts?)`                           | A matching tool call completed (`input`, `output`, `status`, `count`)         |
| `t.loadedSkill(skill, opts?)`                         | Sugar for `t.calledTool("load_skill", { input: { skill }, ...opts })`         |
| `t.notCalledTool(name)`                               | No request for `name` in any lifecycle state                                  |
| `t.toolOrder([...names])`                             | Tool requests appear in order                                                 |
| `t.usedNoTools()`                                     | No tool calls at all                                                          |
| `t.maxToolCalls(n)`                                   | At most `n` tool calls                                                        |
| `t.noFailedActions()`                                 | No tool, subagent, or skill action reported a failure                         |
| `t.calledSubagent(name, opts?)`                       | A subagent delegation happened (`remoteUrl`, `output` constraints)            |
| `t.event(type, opts?)` / `t.notEvent(type, opts?)`    | Typed event presence, data, and count matching                                |
| `t.eventOrder([...matchers])`                         | Matching event groups occur in order                                          |
| `t.eventsSatisfy(label, predicate)`                   | Escape hatch: any predicate over the typed event stream                       |

`succeeded()` accepts both a closed session and a healthy session left open for the next user message; it rejects protocol failures and unanswered HITL. `parked()` requires a clean HITL park. Structured output assertions live on turns and independent sessions, where the output is unambiguous (see the [output schema guide](../guides/client/output-schema)).

```ts
await t.send("What is the weather in Brooklyn?");
t.succeeded();
t.calledTool("get_weather");
```

The same vocabulary narrows naturally in multi-turn and externally-created session evals:

```ts
const first = await t.send("Call get_weather for Brooklyn");
first.calledTool("get_weather", { count: 1 });

const attached = await t.target.attachSession(sessionId);
attached.succeeded();
attached.messageIncludes("Sunny");
```

`t.calledTool` and `t.usedNoTools` are mutually exclusive; assert one or the other, never both in the same run.

## Value assertions with `t.check`

`t.check(value, assertion)` grades an explicit value against a builder from `eve/evals/expect`. The value can be `t.reply`, a turn's `.message`, parsed JSON, or any local you computed:

```ts
import { includes, equals, matches, satisfies, similarity } from "eve/evals/expect";

t.check(t.reply, includes(/sunny/i)); // substring or RegExp (gate)
t.check(parsed, equals({ city: "Brooklyn" })); // deep structural equality (gate)
t.check(parsed, matches(WeatherSchema)); // Standard Schema, e.g. Zod (gate)
t.check(t.reply, similarity("Sunny, 72F")); // fuzzy 0–1 Levenshtein (soft)
t.check(
  latencyMs,
  satisfies((value) => value < 1_000, "latency under one second"),
);
```

| Builder                | Scores                                                  | Default |
| ---------------------- | ------------------------------------------------------- | ------- |
| `includes(value)`      | coerced string contains a substring or matches a RegExp | gate    |
| `equals(value)`        | deep structural equality                                | gate    |
| `matches(schema)`      | validates against a Standard Schema                     | gate    |
| `similarity(expected)` | normalized Levenshtein similarity, 1 = identical        | soft    |
| `satisfies(fn, label)` | custom boolean predicate                                | gate    |

Pick the cheapest builder that captures what "correct" means. When exact match is too strict but a judge model is overkill, `similarity` is the middle ground. For nuanced grading, reach for the [judge](./judge).

## The matcher mini-language

`t.calledTool` and `t.calledSubagent` take matcher objects. Tools accept `{ input, output, status, count }`; subagents accept `{ remoteUrl, output, status, count }`. Calls match `status: "completed"` by default; use `"pending"`, `"failed"`, or `"rejected"` explicitly for lifecycle checks. `count` is the exact number of calls matching every supplied constraint.

Matcher values accept a literal (objects partial-deep-match), a RegExp, or a predicate function that returns a boolean:

```ts
t.calledTool("bash", { input: { command: /^pwd/ }, count: 1 });

t.calledTool("echo", { output: (value) => String(value).includes(marker) });

parked.calledTool("guarded", { status: "pending", count: 1 });
t.calledTool("guarded", { output: /approved/, count: 1 });

t.calledSubagent("weather", {
  remoteUrl: (value) => value === process.env.WEATHER_AGENT_URL,
  output: /72F/,
});
```

`requireInputRequest` uses the same matcher language for `input`, `prompt`, and `display`. Its `optionIds` matcher receives option ids in request order; a literal array must match that complete ordered list exactly:

```ts
const request = session.requireInputRequest({
  toolName: "ask_question",
  optionIds: ["red", "blue"],
});
```

## Run state and derived facts

Beyond the raw `t.events` stream, the runner derives typed facts the assertions read: tool calls (name, input, output, lifecycle status), subagent calls, and HITL input requests. A turn that leaves the session open for a next message is the normal end state of a successful turn; parking on unanswered HITL input is tracked separately.

Typed event matching covers presence, absence, exact counts, partial event data, and ordering:

```ts
turn.notEvent("result.completed");
turn.eventOrder([
  { type: "subagent.called", data: { name: "researcher" }, count: 2 },
  { type: "subagent.completed", data: { subagentName: "researcher" }, count: 2 },
]);
```

When a protocol invariant needs cross-event correlation, `eventsSatisfy` remains the escape hatch:

```ts
t.eventsSatisfy("assistant reply includes the marker", (events) =>
  events.some((e) => e.type === "message.completed" && e.data.message?.includes(marker)),
);
```

## Required preconditions

Recorded assertions never throw and are not awaitable. When later control flow depends on a value, use `await t.require(value, assertion)`. It records a gate, returns the original value when it passes, and stops the test body without adding a duplicate execution error when it fails:

```ts
await t.require(
  sessionIds,
  satisfies((ids) => ids.length > 0, "dispatch started a session"),
);
await t.target.attachSession(sessionIds[0]!);
```

Use the matching `require*` lookups when dependent code needs protocol data:

```ts
const call = turn.requireToolCall("search");
const request = session.requireInputRequest({ toolName: "guarded" });
```

## Severity

Every assertion returns a chainable handle. Severity rides on the assertion, so there is no separate thresholds map to keep in sync.

- `.gate(threshold?)` is hard. A miss marks the eval `failed` and `eve eval` exits non-zero.
- `.soft(threshold?)` is tracked data. A below-threshold miss marks the eval `scored`, fatal only under `--strict`. With no threshold, it is tracked-only and never fails.
- `.atLeast(threshold)` is soft with a bar (equivalent to `.soft(threshold)`).

The defaults are chosen so you rarely set severity. Run-level methods and `includes`/`equals`/`matches` are gates; `similarity` and every `t.judge.*` assertion are soft. Annotate only when you deviate:

```ts
t.calledTool("get_weather").soft(); // record the tool call as a metric, don't gate
t.check(t.reply, similarity("Sunny")).atLeast(0.8); // gate the fuzzy match under --strict
t.check(t.reply, includes("error")).soft(); // track without failing the build
```

## What to read next

- [Judge](./judge): LLM-graded assertions with thresholds
- [Cases](./cases): where assertions attach
- [Running evals](./running): how verdicts map to exit codes
