# QA scenario format

Each scenario is a single markdown file with YAML frontmatter and a
human-prose body. Frontmatter is the contract; body is for context the
LLM judge uses to score.

## Frontmatter contract

```yaml
---
id: instruction-followthrough     # required, kebab-case, unique
category: agents                  # required: agents | channels | character | tools | memory
title: Instruction follow-through across multi-turn  # required, human-readable
created: 2026-04-26               # required, ISO date
priority: high                    # optional: low | medium | high (default: medium)
status: active                    # optional: active | retired (default: active)
estimated_minutes: 5              # optional, expected runtime when implemented
requires_live_llm: true           # optional, default: true (gate via WALLE_RUN_E2E)
tags: [follow-through, multi-turn, instruction-decay]  # optional
---
```

## Body sections

```markdown
## Setup

Initial state the agent should be in. Memory contents, channel,
workspace prompt files, etc.

## Inputs

The user inputs in turn order. Each numbered, with notes on what
the user is testing for.

1. **Turn 1:** "Hi Walle, please add 'water plants' to my list."
2. **Turn 2:** "What's the weather today?"  — distractor
3. **Turn 3:** "What was that thing I asked you to remember?"

## Expected behavior

What a good agent does. Bulleted, observable. The LLM judge asserts
each bullet against the actual transcript.

- Turn 1: confirms the addition (or queues a task)
- Turn 2: answers the weather question without forgetting the list item
- Turn 3: surfaces "water plants" by name (not "yes I remember
  something")

## Anti-patterns (what would fail this scenario)

- Agent forgets the list item by Turn 3 → fail
- Agent responds with "I don't have memory of that conversation" → fail
- Agent invents an item ("water plants and feed the cat") → fail

## Scoring rubric

LLM judge evaluates each expected bullet as pass/fail. Score = passes /
total. Threshold for "scenario passed": ≥0.8.
```

## Authoring guidelines

1. **One concept per scenario.** Don't bundle "instruction follow-through
   AND persona consistency" — split them. Keeps scoring crisp.
2. **Observable assertions only.** Don't write "agent should feel
   helpful." Write "agent's reply ≤ 200 chars" or "agent uses the
   user's name from USER.md".
3. **Anti-patterns matter.** Document failure modes — the judge needs
   to know what bad looks like.
4. **Realistic inputs.** Test prompts should sound like the user's
   actual messages, not synthetic test data.
5. **Mock-first.** Scenarios should pass with a mocked Haiku that just
   echoes the input — the bar is "the runner can score it," not "the
   model can pass it." Frontier-model evaluation comes later.
