Evals

An eval sends a fixed input to the real agent and checks which tools ran and what the answer looks like. To improve an agent and lock each win with an eval, follow Hillclimbing.

Evals run real tools, so guard any writes first.

Protect a tool decision

The case below passes only if the agent calls the inspection tool and doesn't approve.

// evals/readiness.eval.ts
import { defineEval, includes } from "@grok/agent-sdk/evals";

export default defineEval({
  tags: ["smoke"],
  async test(t) {
    await t.send("Is https://github.com/acme/checkout/pull/42 ready to approve?");

    t.succeeded();
    t.calledTool("inspect_pr");
    t.notCalledTool("approve_pr");
    t.check(t.reply, includes(/ready|blocked|approve/i));
  },
});

Evals live in evals/ beside bot/. Set how many cases run at once:

// evals/evals.config.ts
import { defineEvalConfig } from "@grok/agent-sdk/evals";

export default defineEvalConfig({
  maxConcurrency: 20,
});

agent-sdk eval --dir . --list
agent-sdk eval --dir . --tag smoke

Check answer shape or quality

When the output has a contract, check its shape:

import { matches } from "@grok/agent-sdk/evals";
import { z } from "zod";

const verdict = JSON.parse(t.reply ?? "{}");
t.check(verdict, matches(z.object({ ready: z.boolean(), blockers: z.array(z.string()) })));

When correctness depends on meaning, use a judge. Judges covers models and grader options.

t.judge.factuality("The lint step failed on src/sidebar.ts.").atLeast(0.8);

Test a conversation

Several t.send calls in one case share a session. An assertion on a returned turn checks only that turn, and an assertion on t checks the whole conversation.

const first = await t.send("Inspect PR 42.");
first.calledTool("inspect_pr");

const followUp = await t.send("Now summarize only the blockers.");
followUp.usedNoTools();
t.check(followUp.message, includes(/blocker/i));
t.succeeded();

Pin realistic inputs

Seed the files the agent reads instead of depending on your checkout:

await t.send("Review pr/diff.patch and report blockers.", {
  workspaceFiles: {
    "pr/diff.patch": [
      "diff --git a/app/routes/search.ts b/app/routes/search.ts",
      "+res.send(`<h1>Results for ${req.query.q}</h1>`);",
    ].join("\n"),
  },
});

Save a GitHub pull request's webhook payloads as fixtures:

agent-sdk github replay https://github.com/acme/checkout/pull/42 \
  --events '*' --dry-run --fixtures-dir fixtures/github

Datasets and fixtures covers loaders for larger sets.

Run and debug a regression suite

Run one case while you iterate, then the smoke set before review:

agent-sdk eval --dir . readiness
agent-sdk eval --dir . --tag smoke

Each run writes a folder under evals/ in the project state directory, or under --artifacts <dir> if you pass it. Open evals/<case-id>.json in that folder to see a failed case's input, assertions, tool calls, and final text. Model turns need a Cursor credential.

Run evals in CI

# CURSOR_API_KEY comes from the CI secret store.
agent-sdk eval --dir . --json --no-stream \
  --junit reports/evals.xml \
  --artifacts reports/evals \
  > reports/evals.json

Run --tag smoke on pull requests and the full suite on a schedule. The CLI reference covers every flag.

Keep side effects out of eval sessions

Eval sessions carry purpose: "eval". Check it where the tool writes, and return the decision without the external action:

async execute({ findings }, ctx) {
  if (ctx.session.purpose === "eval") {
    return { posted: false, reason: "eval", count: findings.length };
  }

  return await postFindings(findings);
}

The tool call stays in the trajectory, so the eval can still assert that the agent chose to post.