# Systematic Debug Workflow

A step-by-step process for diagnosing and fixing issues in ADK agents. Follow these steps in order — each step narrows the problem space.

Most diagnostic CLI commands use `--format json` for structured, parseable output. Make sure `adk dev` is running in the background, once the project is validated.

---

## The Debug Loop

```
1. VALIDATE  → adk check --format json
2. REPRODUCE → adk chat --single "msg" --format json
3. LOGS      → adk logs error --format json
4. TRACES    → adk traces --format json
5. CLASSIFY  → Runtime error? LLM behavior? Config issue?
6. FIX       → Apply targeted fix
7. VERIFY    → Re-test with adk chat, confirm clean output
8. PREVENT   → Write a regression eval
```

---

## Step 1: Validate

Before debugging runtime behavior, rule out project-level issues.

```bash
adk check --format json
```

This catches:
- Invalid `agent.config.ts`
- Schema mismatches in actions, tools, workflows
- Invalid entity names
- Missing dependencies
- Import errors

If `adk check` reports errors, fix them first. Many "runtime" issues are actually build/config problems.

---

## Step 2: Reproduce

Create a minimal reproduction with a single message.

```bash
adk chat --single "the message that triggers the bug" --format json
```

**Why minimal?** A single message isolates the problem. Most often, multi-turn conversations add noise — the issue might be in turn 3 but caused by context from turn 1. That being said, some bugs are only reproducible by going through many turns.

**Save the conversation ID** from the JSON output — you'll need it for trace queries and follow up conversations.

If the issue requires multi-turn context:

```bash
# First message
adk chat --single "Setup message" --format json
# Note the conversation_id from output

# Trigger the bug
adk chat --single "Bug-triggering message" --conversation-id <id> --format json
```

---

## Step 3: Check Logs

Quick scan for errors and warnings.

```bash
# Errors only
adk logs error --format json

# Errors and warnings from the last hour
adk logs warning since=1h --format json

# Stream live while reproducing
adk logs --follow --format json
```

Logs give you a high-level view. If the error is obvious here (startup failure, integration auth error, missing config), fix it directly.

If logs don't explain the issue, move to traces.

---

## Step 4: Inspect Traces

Traces show everything the agent did during a conversation turn.

```bash
# All recent traces
adk traces --format json

# For a specific conversation
adk traces conversation=<id> --format json
```

**Read spans in order.** Look for:

1. **`think` spans** — Does the model's reasoning make sense? Does it understand the user's intent?
2. **`tool_call` spans** — Was the right tool called? Did it succeed? Are the parameters correct?
3. **`code_execution_exception` spans** — Any runtime errors? Read the stack trace.
4. **`end` span** — Did the turn complete normally?

**Find the divergence point:** Compare what happened vs what should have happened. The first span that doesn't match expectations is usually where the bug is.

---

## Step 5: Classify the Failure

Based on what you found in logs and traces, classify the issue:

### Runtime Error
The code threw an exception or returned an error.
- **Traces show:** `code_execution_exception` or `tool_call` with `success: false`
- **Next:** See `common-failures.md` for specific patterns and fixes

### LLM Behavior
The model made a bad decision — wrong tool, hallucinated values, refusal, or looping.
- **Traces show:** `think` span with incorrect reasoning, or `tool_call` with wrong tool/params
- **Next:** See `llm-debugging.md` for diagnosis and fixes

### Config Issue
Missing or incorrect configuration — integration auth, agent.json, environment.
- **Traces show:** May not have traces at all, or errors mentioning config/auth
- **Logs show:** Startup errors, auth failures, missing config warnings
- **Next:** See `common-failures.md` sections on config, integration failures, and agent.json vs agent.local.json

---

## Step 6: Fix

Apply a targeted fix based on the classification:

| Classification | Where to Fix | Reference |
|---------------|-------------|-----------|
| Runtime error in handler | Action/tool/workflow handler code | `common-failures.md` |
| Schema mismatch | Input/output schema definitions | `common-failures.md` § Build Errors |
| Wrong tool selected | Tool descriptions, instructions | `llm-debugging.md` § Wrong Tool |
| Hallucinated params | Input schemas, validation | `llm-debugging.md` § Hallucinated Parameters |
| Model refusal | Instructions, tool descriptions | `llm-debugging.md` § Refusal |
| Model looping | `onBeforeTool` guard, output clarity | `llm-debugging.md` § Looping |
| Integration failure | Dev Console config, `adk add` | `common-failures.md` § Integration Failures |
| Config issue | `agent.json`, `agent.local.json` | `common-failures.md` § Config Confusion |

---

## Step 7: Verify

Re-run the exact reproduction from Step 2:

```bash
adk chat --single "the message that triggered the bug" --format json
```

Then confirm:

```bash
# No errors in logs
adk logs error --format json

# Traces show correct behavior
adk traces --format json
```

Check that:
- The bot responds correctly
- The right tools are called with the right parameters
- No error spans in the traces
- No new warnings in logs

---

## Step 8: Prevent with Evals

After fixing, write a regression eval so the bug can't come back. Use the **adk-evals** skill for full guidance.

Quick template:

```typescript
import { Eval } from '@botpress/adk'

export default new Eval({
  name: 'fix-description-here',
  type: 'regression',
  tags: ['bugfix'],

  conversation: [
    {
      user: 'the message that triggered the bug',
      assert: {
        response: [
          { not_contains: 'error' },
          { llm_judge: 'Response correctly handles the scenario' },
        ],
        tools: [{ called: 'expectedTool' }],
      },
    },
  ],
})
```

Run it:

```bash
adk evals fix-description-here --format json
```

Tag regression evals as `type: 'regression'` so they run in CI and catch future regressions.

---

## When to Escalate

If you've gone through the loop and still can't resolve the issue, gather this information for a bug report:

```bash
# Project validation
adk check --format json > check-output.json

# Recent errors
adk logs error --format json > error-logs.json

# Traces from the failing conversation
adk traces conversation=<id> --format json > traces.json
```

Include:
1. **Reproduction steps** — exact messages/events that trigger the issue
2. **`adk check` output** — project state
3. **Error logs** — relevant error entries
4. **Trace output** — full trace with spans for the failing conversation
5. **Expected vs actual behavior** — what should happen vs what does happen
6. **ADK version** — `adk --version`

Let the human know and suggest reaching out to support with the above information.
