# Wall-E QA scenario library

**Item B of testing deep-dive (2026-04-26).** Scaffold for cross-cutting
agent behavior scenarios — a markdown-driven complement to the JSON
benchmarks in `wall-e/eval/benchmarks/`.

## When to write a scenario vs a benchmark

- **Use `eval/benchmarks/`** for measurable single-task behaviors:
  scoring is JSON-rubric-driven, runs in the eval loop, dashboards
  results. Examples: HumanEval+ coding tasks, chat-eval factuality
  questions, coding-001..N orchestrator tasks.

- **Use `qa/scenarios/`** (this dir) for cross-cutting behaviors that
  are hard to formalize as JSON rubrics: persona consistency across
  many turns, channel handoff (Slack→iMessage mid-thread context
  preservation), instruction follow-through over a long session.
  Scoring is LLM-as-judge or human review.

Mirrors OpenClaw's `qa/scenarios/agents/`, `qa/scenarios/channels/`,
`qa/scenarios/character/` taxonomy.

## Layout

```
wall-e/qa/
├── README.md                          # this file
├── scenarios/
│   ├── agents/
│   │   └── instruction-followthrough.md      # initial scenario
│   ├── channels/
│   │   └── handoff-context-preservation.md   # initial scenario
│   └── character/
│       └── persona-consistency.md            # initial scenario
└── scenario-format.md                 # frontmatter contract + how to write one
```

## Running a scenario

The runner is intentionally NOT shipped in this scaffold — running
scenarios needs LLM-as-judge wiring + reports/ dir + CI integration.
This first cut ships:

1. The directory structure (so authors have a home)
2. `scenario-format.md` — the contract for new scenarios
3. 3 initial scenarios as templates

When the runner is added (next deep-dive on testing infrastructure):
- `npm run qa` → walks scenarios/, runs each via chat() + LLM judge
- `npm run qa -- --category channels` → run only channel scenarios
- `npm run qa -- --live` → use real LLM (otherwise mock)

## Why scaffold-first

The high-value bug is "we have no place for cross-cutting scenarios."
Adding a runner without scenarios is putting the cart before the horse.
Adding scenarios without a runner means at least the SCENARIOS exist as
documented expectations — they can be hand-evaluated until the runner
catches up.
