# ADK JSON Schemas

Defines the JSON structures used by the ADK eval-driven iteration system. Adapted from skill-creator for CASRE context.

---

## evals.json

Defines test prompts and assertions for an artifact. Located at `skills/aw-adk/evals/evals.json` or `<artifact>/evals/evals.json`.

```json
{
  "artifact_name": "payments-agent",
  "artifact_type": "agent",
  "evals": [
    {
      "id": 1,
      "prompt": "Create an agent for payments processing in the revex/memberships namespace",
      "expected_output": "Agent file with Identity, Core Mission, Critical Rules, Process, Deliverables sections",
      "files": [],
      "expectations": [
        "The agent has a Core Mission section with 2+ sentences",
        "Frontmatter includes name, description, tools, model, category, squad",
        "Agent file is placed at correct registry path",
        "At least 2 colocated eval files created"
      ]
    }
  ]
}
```

**Fields:**
- `artifact_name`: Name of the artifact being tested
- `artifact_type`: One of: command, agent, skill, rule, eval
- `evals[].id`: Unique integer identifier
- `evals[].prompt`: The task to execute (realistic user request)
- `evals[].expected_output`: Human-readable success description
- `evals[].files`: Optional input file paths
- `evals[].expectations`: Verifiable assertions (graded by agents/grader.md)

---

## eval_metadata.json

Per-eval directory metadata. Located at `<workspace>/iteration-N/<eval-name>/eval_metadata.json`.

```json
{
  "eval_id": 1,
  "eval_name": "create-payments-agent",
  "prompt": "Create an agent for payments processing in the revex/memberships namespace",
  "assertions": [
    "The agent has a Core Mission section with 2+ sentences",
    "Frontmatter includes name, description, tools, model"
  ]
}
```

---

## grading.json

Output from the grader agent. Located at `<run-dir>/grading.json`.

```json
{
  "expectations": [
    {
      "text": "The agent has a Core Mission section",
      "passed": true,
      "evidence": "Found '## Core Mission' at line 42 with 3 sentences"
    }
  ],
  "summary": {
    "passed": 8,
    "failed": 2,
    "total": 10,
    "pass_rate": 0.80
  },
  "execution_metrics": {
    "tool_calls": { "Read": 5, "Write": 2, "Bash": 3 },
    "total_tool_calls": 10,
    "total_steps": 6,
    "errors_encountered": 0
  },
  "timing": {
    "executor_duration_seconds": 45.0,
    "total_duration_seconds": 52.0
  },
  "claims": [
    {
      "claim": "Agent scores B-Tier (65/100)",
      "type": "quality",
      "verified": true,
      "evidence": "Rubric scoring confirms total = 65"
    }
  ],
  "eval_feedback": {
    "suggestions": [],
    "overall": "Assertions cover structure well. Consider adding behavioral checks."
  }
}
```

**Important:** The viewer depends on exact field names: `text`, `passed`, `evidence` in expectations (not `name`/`met`/`details`).

---

## timing.json

Wall clock timing. Located at `<run-dir>/timing.json`.

```json
{
  "total_tokens": 84852,
  "duration_ms": 23332,
  "total_duration_seconds": 23.3
}
```

**How to capture:** When a subagent task completes, the notification includes `total_tokens` and `duration_ms`. Save immediately — this data is not persisted elsewhere.

---

## benchmark.json

Aggregated results. Located at `<workspace>/iteration-N/benchmark.json`. Generated by `scripts/aggregate-benchmark.py`.

```json
{
  "metadata": {
    "artifact_name": "payments-agent",
    "iteration_dir": "payments-agent-workspace/iteration-1",
    "evals_run": ["create-payments-agent", "score-minimal-agent"],
    "total_runs": 4
  },
  "runs": [
    {
      "eval_id": 1,
      "eval_name": "create-payments-agent",
      "configuration": "with_artifact",
      "run_number": 1,
      "result": {
        "pass_rate": 0.85,
        "passed": 6,
        "failed": 1,
        "total": 7,
        "time_seconds": 42.5,
        "tokens": 3800,
        "errors": 0
      },
      "expectations": [
        { "text": "...", "passed": true, "evidence": "..." }
      ]
    }
  ],
  "run_summary": {
    "with_artifact": {
      "pass_rate": { "mean": 0.85, "stddev": 0.05 },
      "time_seconds": { "mean": 45.0, "stddev": 12.0 },
      "tokens": { "mean": 3800, "stddev": 400 }
    },
    "without_artifact": {
      "pass_rate": { "mean": 0.35, "stddev": 0.08 },
      "time_seconds": { "mean": 32.0, "stddev": 8.0 },
      "tokens": { "mean": 2100, "stddev": 300 }
    },
    "delta": {
      "pass_rate": "+0.500",
      "time_seconds": "+13.0",
      "tokens": "+1700"
    }
  },
  "notes": [
    "Without-artifact runs consistently fail on eval placement checks (0% pass rate)"
  ]
}
```

**Important:** The viewer reads `configuration` (not `config`) and `result.pass_rate` (not top-level `pass_rate`).

---

## comparison.json

Output from blind comparator. Located at `<grading-dir>/comparison.json`.

```json
{
  "winner": "A",
  "reasoning": "Artifact A has stronger Identity section with concrete traits...",
  "rubric": {
    "A": { "dimensions": { "1_frontmatter": 8, "2_identity": 9 }, "total": 74, "tier": "B" },
    "B": { "dimensions": { "1_frontmatter": 7, "2_identity": 5 }, "total": 61, "tier": "B" }
  },
  "output_quality": {
    "A": { "score": 74, "strengths": ["..."], "weaknesses": ["..."] },
    "B": { "score": 61, "strengths": ["..."], "weaknesses": ["..."] }
  }
}
```

---

## feedback.json

Human review feedback. Downloaded from eval-viewer. Located at `<workspace>/iteration-N/feedback.json`.

```json
{
  "reviews": [
    {
      "run_id": "create-payments-agent-with_artifact",
      "feedback": "Identity section is great but Code Examples are too generic",
      "timestamp": "2026-04-22T10:30:00Z"
    }
  ],
  "status": "complete"
}
```

Empty feedback means the reviewer thought it was fine.
