Hillclimbing

Hillclimbing improves an agent one measured change at a time. Hold the inputs fixed, change one thing, and keep it only when the same run gets better without moving the behavior you froze.

A coding agent in your checkout can run this loop by following the bdk-hillclimb skill.

Run one measured round

First agree on the agent, one to three fixed inputs, the metric to improve, and the behavior that must not change, such as the output shape, side effects, and existing evals.

Then run one round:

1. Run the agent the way a user would, on saved or local inputs, without deploying or posting live.
2. Read the trajectory and name the main failure in one sentence.
3. Change one lever.
4. Rerun the same fixtures.
5. Compare correctness, efficiency, and contract fit.
6. Keep, revert, or narrow the change.

Don't swap a fixture mid-round, or the comparison means nothing.

Choose the smallest lever

Start with the earliest place that can remove the failure. Prepare evidence on the host before the turn, then change the shape of that evidence, then the instructions or skills, then remove or gate tools that invite wandering, and change the framework only when the agent can't express the fix.

Fixing the host or evidence often improves quality and latency together, because the model searches less for ground truth.

Match the fix to the failure

When the agent picks the wrong tool, such as a PR reviewer that calls approve_pr before inspect_pr, pin one PR and tighten only the instruction that owns that decision, so you know which edit fixed it.

t.calledTool("inspect_pr");
t.notCalledTool("approve_pr");

When a run wanders or runs long, it often repeats reads or makes the model find evidence the host already has. Move that preparation ahead of the turn, and keep the change only if the answer still finds the same issue.

before: 14 tool calls, 92 seconds, 6 repeated reads
after:   5 tool calls, 31 seconds, 0 repeated reads

When the answer is weak, freeze the public shape, compare both runs against known-good findings, and edit the instruction or skill that owns the missing judgment.

fixture: saved checkout PR diff
success: report both gold findings with file paths
freeze:  findings JSON shape; no review post; existing smoke evals

Lock every kept win

Each kept change needs an eval that would have failed before it, such as a tool-choice assertion, a maxToolCalls bound, an output-shape check, or a quality score.

agent-sdk eval --dir . --tag smoke

Don't loosen an existing eval so that a new change passes. If an older eval starts failing, the change broke something that used to work, and you should fix the change instead.