# Traces & Logs

Traces and logs are the primary debugging interface for ADK agents. Every conversation turn, tool call, LLM reasoning step, and error is recorded as a trace with spans. The CLI provides structured access to all of it.

## CLI Debugging Tools

Most commands support `--format json` for structured output — **always use it** when consuming output programmatically. The exception is `adk dev`, which has no `--format` flag: use `adk dev --non-interactive` to get a structured NDJSON event stream instead.

| Command | Purpose | Options / tokens |
|---------|---------|-----------------|
| `adk check --format json` | Offline project validation — catches config/schema issues before runtime | — |
| `adk logs --format json` | Browse or stream log entries | `--follow`, `--summary`, level filters (`error`, `warning`) |
| `adk traces --format json` | Query traces with span details — filter by workflow, action, trigger, conversation, error, time | `workflow=`, `action=`, `trigger=`, `conversation=`, `trace=`, `error`, `since=`, `until=`, `limit=`, `--follow`, `--include-llm` |
| `adk workflows runs --format json` | List or inspect durable workflow runs by id, name, or status | `name=`, `status=`, `limit=`, `nextToken=`, `<wrkflow_id>` |
| `adk chat --single "msg" --format json` | Send a test message and get structured response | `--conversation-id` |
| `adk dev --non-interactive` | Start dev server with structured NDJSON event stream (no TUI) | — |

## Querying Logs

Logs give a high-level view of what happened. Start here before diving into traces.

```bash
adk logs --format json                          # last 50 entries
adk logs error --format json                    # errors only
adk logs warning --format json                  # warnings and above
adk logs warning since=1h --format json         # warnings from the last hour
adk logs error limit=10 --format json           # last 10 errors
adk logs --follow --format json                 # stream live as NDJSON
adk logs --summary --format json                # single snapshot summary
```

### When to use logs vs traces

- **Logs** — quick error scanning, startup issues, integration failures, deployment problems
- **Traces** — detailed per-conversation analysis, tool call inspection, LLM reasoning review

## Querying Traces

Traces provide the full picture of a conversation turn: every span in order. The command takes positional filter tokens (not flags) for slicing the trace store.

```bash
adk traces --format json                                    # recent traces (default limit 20)
adk traces error --format json                              # error traces only
adk traces workflow=<name> --format json                    # traces for a workflow
adk traces conversation=<id> --format json                  # traces for a conversation
adk traces trace=<id> --include-llm --format json           # drill into one trace with LLM content
adk traces since=1h limit=50 --format json                  # last hour, up to 50
adk traces error --follow --format json                     # stream errors live (NDJSON)
```

Combine tokens freely (`adk traces error workflow=onboarding since=1h`). `--format` only accepts `json`; omit it for the default text output.

### Trace Structure

A trace represents one unit of agent activity. Each trace contains:

- **Trace ID** — unique identifier
- **Timestamp** — when the trace was recorded
- **Conversation ID** — which conversation it belongs to
- **Type** — the kind of activity
- **Spans** — ordered list of sub-events within the trace

### Span Types

Spans are the atomic units inside a trace. Each span has a `type` field:

| Span Type | Key Fields | What It Means |
|-----------|-----------|---------------|
| `tool_call` | `tool_name`, `input`, `output`, `success`, `error` | A tool was invoked. Check `success` to see if it worked. |
| `code_execution_exception` | `message`, `stackTrace` | Code error in the LLMz sandbox. Read `stackTrace` to find the source. |
| `think` | `content` | LLM reasoning step — shows why the model made a decision. |
| `end` | — | Conversation turn completed. |

### Reading a Trace

When analyzing a trace:

1. Look at spans in chronological order
2. Find where behavior diverges from expected
3. For `tool_call` spans: check `success`, read `error` if false, verify `input` matches expectations
4. For `think` spans: understand model reasoning before a tool call
5. For `code_execution_exception`: read the stack trace, correlate with source code

## Reproducing Issues with `adk chat`

Use `adk chat` to send test messages and generate fresh traces:

```bash
# Single message with structured output
adk chat --single "Hello" --format json

# Continue an existing conversation
adk chat --single "Follow-up message" --conversation-id <id> --format json
```

The JSON output includes the bot's response and conversation ID, which you can use to query traces for that specific conversation and send subsequent messages.

## Programmatic Trace Access: `onTrace` Hook

For monitoring traces in code, use the `onTrace` hook in conversation handlers. This is useful for custom logging, performance monitoring, and automated error detection. The Botpress ADK also has an OTLP endpoint that can be enabled with the `--otlp` flag on `adk dev`, if the current environment is setup for OTLP traces.

### Basic Trace Logging

```typescript
// src/conversations/extensions/logging.ts
import { Autonomous } from "@botpress/runtime";

export const onTraceLogging: Autonomous.Hooks["onTrace"] = ({ trace, iteration }) => {
  if (trace.type === "code_execution_exception") {
    console.error(`Code Execution Error: ${trace.message}`, trace.stackTrace);
  }

  if (trace.type === "tool_call" && !trace.success) {
    console.error(
      `Error during tool call to "${trace.tool_name}" with input "${JSON.stringify(trace.input)}":`,
      trace.error
    );
  }
};
```

### Comprehensive Trace Logging

```typescript
export const onTraceLogging: Autonomous.Hooks["onTrace"] = ({ trace, iteration }) => {
  switch (trace.type) {
    case "code_execution_exception":
      console.error(`[CODE ERROR] ${trace.message}`, trace.stackTrace);
      break;
    case "tool_call":
      if (trace.success) {
        console.log(`[TOOL SUCCESS] ${trace.tool_name}`, trace.output);
      } else {
        console.error(`[TOOL ERROR] ${trace.tool_name}`, trace.error);
      }
      break;
    case "think":
      console.debug(`[THINK] ${trace.content}`);
      break;
    default:
      console.log(`[TRACE] ${trace.type}`, trace);
  }
};
```

### Using in a Conversation

```typescript
import { Conversation } from "@botpress/runtime";
import { onTraceLogging } from "./extensions/logging";

export default new Conversation({
  channel: "*",
  handler: async ({ execute }) => {
    await execute({
      instructions: `You are a helpful assistant...`,
      hooks: {
        onTrace: (props) => onTraceLogging!(props),
      },
    });
  },
});
```

### Performance Monitoring

```typescript
export const makePerformanceMonitor = () => {
  const startTime = Date.now();
  const toolStartTimes = new Map<string, number>();
  const toolMetrics = new Map<string, number[]>();

  const onBeforeTool: Autonomous.Hooks["onBeforeTool"] = async ({ tool }) => {
    toolStartTimes.set(tool.name, Date.now());
  };

  const onAfterTool: Autonomous.Hooks["onAfterTool"] = async ({ tool }) => {
    const start = toolStartTimes.get(tool.name);
    const duration = start ? Date.now() - start : 0;
    const metrics = toolMetrics.get(tool.name) || [];
    metrics.push(duration);
    toolMetrics.set(tool.name, metrics);
    console.log(`[PERF] ${tool.name}: ${duration}ms`);
  };

  const onTrace: Autonomous.Hooks["onTrace"] = ({ trace }) => {
    if (trace.type === "end") {
      console.log(`[PERF] Total conversation: ${Date.now() - startTime}ms`);
      console.log(`[PERF] Tool metrics:`, Object.fromEntries(toolMetrics));
    }
  };

  return { onBeforeTool, onAfterTool, onTrace };
};
```

### Hook Reference

| Hook | Parameters | When It Fires |
|------|-----------|--------------|
| `onBeforeTool` | `{ iteration, tool, input, controller }` | Before each tool call |
| `onAfterTool` | `{ tool, output }` | After each tool call |
| `onTrace` | `{ trace, iteration }` | On every trace event |
