Evals

Evals run fixed cases against an agent and record whether its turns, tools, events, and output meet a contract. The Evals guide covers the regression workflow.

Eval discovery and case IDs

Eval files live under the project-root evals/ directory and end in .eval.ts or .eval.js. The path under that directory becomes the eval ID, so evals/readiness.eval.ts becomes readiness, evals/builds/api.eval.ts becomes builds/api, and evals/builds/index.eval.ts becomes builds.

Export one defineEval that sets exactly one of test or cases. A case ID adds one path segment to the file ID:

export default defineEval({
  tags: ["smoke"],
  cases: [
    {
      id: "checkout",
      async test(t) {
        await t.send("Review checkout PR 42.");
        t.succeeded();
      },
    },
    {
      id: "search",
      async test(t) {
        await t.send("Review search PR 7.");
        t.succeeded();
      },
    },
  ],
});

Those cases are prs/checkout and prs/search when the file is evals/prs.eval.ts. Case IDs must be unique single path segments.

A file can also export an array of defineEval calls, named with zero-padded indexes such as sql/0000.

iterations repeats one datapoint 1 to 100 times. With iterations: 3, weather/nyc runs as weather/nyc/1 through weather/nyc/3, and selecting weather/nyc runs all three.

Eval configuration

Every suite needs evals/evals.config.ts with maxConcurrency:

import { defineEvalConfig } from "@grok/agent-sdk/evals";

export default defineEvalConfig({
  maxConcurrency: 20,
  judge: { model: "gpt-5.4-mini" },
});

Option	Contract
maxConcurrency	Required. Concurrent authored datapoints, from 1 to 200
timeoutMs	Per-case timeout. A case or file value wins, then the CLI flag, then this config
judge	Default model for t.judge
reporters	Objects notified as cases and runs complete
maxPlaygroundRuns	Server-side batches kept in playground history. Defaults to 20

A case can override description, tags, timeoutMs, iterations, judge, reporters, and metadata. Case metadata merges over file metadata, and case reporters add to the file's reporters.

Send turns in a case

await t.send(message, options?) runs one turn and waits until it finishes or fails. Sends in one test share a session.

The returned turn has the assistant message, sessionId, events, ordered toolCalls, ok, and a one-based index. Assertions on a turn inspect only that turn, and assertions on t inspect the whole case. The case also exposes t.reply, t.events, t.turns, t.sessionId, t.iteration, t.iterations, and the timeout t.signal.

On the first send only, workspaceFiles seeds the session workspace from a map of relative paths to content, and workspaceDir sets an absolute working directory for a new local session.

turn.expectOk() throws when the turn failed, so later steps that depend on it don't run.

Trajectory assertions

Assertions record failures without stopping the test.

Assertion	Checks
t.succeeded()	Run finished without failure
t.calledTool(name, matcher?), t.notCalledTool(name)	A matching tool request occurred, or none did
t.loadedSkill(name)	Agent opened a named skill
t.toolOrder(names), t.eventOrder(matchers)	Tool requests or event groups appeared in relative order
t.usedNoTools(), t.maxToolCalls(max)	No tool was requested, or the count stayed under the bound
t.noFailedActions()	No tool result failed
t.calledSubagent(name, matcher?)	Matching subagent call occurred
t.taggedArtifact(kind?, predicate?)	Matching durable artifact was tagged
t.event(type, matcher?), t.notEvent(type, matcher?)	A matching session event occurred, or none did
t.eventsSatisfy(label, predicate)	Predicate accepted the event stream
t.check(value, expectation)	Value met an expectation builder
t.score(name, value)	Recorded a numeric score from 0 to 1

t.requireToolCall, t.requireInputRequest, and t.require return the matched value and stop the rest of the test when nothing matches.

Tool and event matchers

Tool matchers can narrow by input, output, status, and count. Values can be literals, regular expressions, or predicates. Object literals match partially:

t.calledTool("inspect_pr", {
  input: { verdict: "review" },
  count: 1,
});

Tool status is completed, failed, pending, or rejected. Subagent matchers accept output, status, count, and callId. Event matchers accept data and count.

Expectation builders

includes(string | RegExp), equals(value), matches(schema), and satisfies(predicate, label) are hard gates by default, and similarity(expected) is a soft score. matches accepts Standard Schema implementations such as Zod. normalizedSimilarity(value, expected) returns the same 0-to-1 value that similarity uses.

Severity and verdicts

Every assertion returns a severity handle:

t.calledTool("search").soft();
t.check(t.reply, similarity("Expected answer")).atLeast(0.8);
t.judge.closedQA("names the failing step").gate(1);

• .gate(threshold?) fails the case on a miss.
• .soft(threshold?) records a value and never fails the case.
• .atLeast(threshold) sets a soft bar, and a miss gives the case a scored verdict.

Case verdicts are passed, failed, scored, or skipped. A scored case fails the run only under --strict.

Judges

Judges grade what deterministic checks can't express:

t.judge.factuality(expected).atLeast(0.8);
t.judge.summarizes(expected).atLeast(0.8);
t.judge.closedQA("names the root cause").gate(1);
t.judge.sql(expected).atLeast(0.9);

Each judge grades t.reply unless you pass { on } with another value. Set judge options in the project config, the eval, the case, or the call, and the nearest value wins.

t.judge.model(prompt) returns a raw model reply for custom grading. Agent output is untrusted input, so wrap it with fenceUntrusted and include EVAL_JUDGE_INJECTION_GUARD in the judge prompt.

Judge host

A judge needs a judge model and a Cursor credential from agent-sdk login, CURSOR_API_KEY, or CURSOR_SERVICE_ACCOUNT_KEY. Without a usable credential the runner skips the case, and without a judge model it fails the case and names where to set one.

Datasets and fixtures

Load committed JSON, JSONL, or YAML relative to the project root:

import { defineEval, includes } from "@grok/agent-sdk/evals";
import { loadYaml } from "@grok/agent-sdk/evals/loaders";

const rows = await loadYaml<Array<{ prompt: string; expected: string }>>(
  "evals/data/cases.yaml",
);

export default rows.map(row =>
  defineEval({
    async test(t) {
      await t.send(row.prompt);
      t.check(t.reply, includes(row.expected));
    },
  })
);

Reporters and results

Built-in reporters include JUnit({ filePath, suiteName? }) and Artifacts({ dir }). A custom reporter can implement onRunStart, onEvalComplete, and onRunComplete.

JSON output has run totals and one result per case. Each result has the case ID, verdict, assertions, session ID, inputs, final text, tool calls, metrics, logs, duration, metadata, tags, and any error or skip reason.

Each local run writes summary.json, results.jsonl, and evals/<case-id>.json under evals/<stamp>/ in the project state directory. --state-root doesn't move them, and --artifacts <dir> or --no-artifacts does.

Select cases

agent-sdk eval --dir . --list
agent-sdk eval --dir .
agent-sdk eval --dir . builds/checkout
agent-sdk eval --dir . --tag smoke

An ID matches itself and its descendants. Repeated IDs or tags combine with OR, and a case must match both groups when you pass both. Run evals covers runs on a server.

Suite summary cases

A case tagged EVAL_SUMMARY_TAG aggregates its file, so selecting it by ID or with --tag summary also runs every other case in that file.

import { defineEval, EVAL_SUMMARY_TAG } from "@grok/agent-sdk/evals";

export default defineEval({
  cases: [
    {
      id: "checkout",
      async test(t) {
        await t.send("Review checkout PR 42.");
        t.succeeded();
      },
    },
    {
      id: "overall",
      tags: [EVAL_SUMMARY_TAG],
      async test(t) {
        await t.send("Summarize the checkout review.");
        t.succeeded();
      },
    },
  ],
});