# AI Evaluation Framework (R10 — v3.46+)

> **Reference for `ai-prompt` skill** — how to evaluate AI skills against expected outputs.

## When This Reference Applies

- You are about to ship a new prompt or change an existing one
- You need to compare two versions of the same skill (`v1.0.0` vs `v1.1.0`)
- You want to benchmark the same prompt across providers/models (OpenAI vs Claude vs Gemini)
- The user asks "how do I test my prompt", "how do I prevent regression", "how do I A/B prompts"

For one-shot manual testing during development, the eval framework is overkill — call the skill directly. Use this framework when you want **reproducible, scored, auditable** runs.

---

## The 4 entities

### `AiEvalDataset` — collection of test cases

```csharp
class AiEvalDataset : BaseEntity, IAuditableEntity
{
    public string Code { get; }                  // kebab-case stable id, e.g. "support-classification-v1"
    public string Name { get; }
    public string? Description { get; }
    public IReadOnlyCollection<AiEvalDatasetItem> Items { get; }
}
```

**Provider/model agnostic** : the same dataset can be replayed against multiple skills, providers and models to compare quality and cost.

### `AiEvalDatasetItem` — one test case

```csharp
class AiEvalDatasetItem
{
    public Guid DatasetId { get; }
    public string Label { get; }                 // Human-readable name for the case
    public string VariablesJson { get; }         // Inputs to feed into the skill
    public string ExpectedOutputJson { get; }    // Expected result (for grading)
}
```

### `AiEvaluation` — one run

```csharp
class AiEvaluation : BaseEntity, IAuditableEntity
{
    public Guid SkillId { get; }                 // Which skill was tested
    public Guid DatasetId { get; }               // Which dataset was replayed
    public AiEvaluationStatus Status { get; }    // Running / Completed / Failed
    public DateTime StartedAt { get; }
    public DateTime? CompletedAt { get; }
    public int ProcessedItems { get; }
    public int TotalItems { get; }
    public int PassedItems { get; }              // Items whose actual output matched expected
    public double? AverageScore { get; }         // [0..1], null while in progress
    public IReadOnlyCollection<AiEvalResult> Results { get; }
}

enum AiEvaluationStatus { Running = 0, Completed = 1, Failed = 2 }
```

Service methods : `RecordResult(AiEvalResult)`, `MarkCompleted()`, `MarkFailed(reason)`. The aggregate computes `AverageScore` from the per-item `Score` values when completed.

### `AiEvalResult` — per-item outcome

Records actual output vs expected, the score (0..1), pass/fail flag, and any tokens/cost incurred.

---

## Workflow

```
1. Curate dataset
   └─ AiEvalDataset.Create("support-classification-v1", "Support classification — 50 cases")
      └─ AddItem("simple billing question", { "ticket": "..." }, { "category": "billing" })
      └─ AddItem("urgent outage report",   { "ticket": "..." }, { "category": "incident" })
      └─ … 48 more items

2. Run evaluation against current skill version
   └─ AiEvaluation.Create(skillId, datasetId, totalItems: 50)
      └─ For each item: invoke skill, grade output, RecordResult(actual, expected, score)
      └─ MarkCompleted()
      └─ Stored AverageScore = 0.86 (43/50 passed)

3. Iterate — change prompt / blocks / model
   └─ New AiEvaluation against same dataset
   └─ Compare AverageScore : 0.86 → 0.92 (+0.06)

4. Decide
   └─ If new score significantly higher : promote prompt version
   └─ If equal or lower : revert
```

---

## DO / DON'T

| ✅ DO | ❌ DON'T |
|---|---|
| Build datasets from real production examples (anonymised) | Use synthetic / hallucinated test cases as the only ones |
| Stabilise dataset.Code — never rename, only add new datasets | Mutate dataset items after first run (compromises history) |
| Grade with structured comparison (JSON match, key fields, semantic distance) | Grade with substring search / regex |
| Track cost per evaluation — reject prompt changes that 3× cost for marginal score | Optimise for score alone |
| Run evals as part of the prompt-change PR review | Ship prompt changes without an eval baseline |
| Keep `AverageScore` as the single headline metric | Multiply scoring criteria — pick one and stick to it |

---

## Wiring into a release pipeline

A prompt change should run the relevant evaluation as part of CI:

1. PR opens with a new prompt version (`v1.1.0`).
2. CI invokes a Command that creates an `AiEvaluation(skillId, datasetId, totalItems)` against `v1.1.0`.
3. CI compares `AverageScore` of `v1.1.0` vs the latest passing eval of `v1.0.0`.
4. PR is annotated with the delta. Score regression > 5% → PR blocked.

---

## Reference source files (read-only)

- `D:\01 - projets\SmartStack.app\features\IA-Workflow\src\SmartStack.Domain\AI\Evaluations\AiEvalDataset.cs`
- `D:\01 - projets\SmartStack.app\features\IA-Workflow\src\SmartStack.Domain\AI\Evaluations\AiEvalDatasetItem.cs`
- `D:\01 - projets\SmartStack.app\features\IA-Workflow\src\SmartStack.Domain\AI\Evaluations\AiEvaluation.cs`
- `D:\01 - projets\SmartStack.app\features\IA-Workflow\src\SmartStack.Domain\AI\Evaluations\AiEvalResult.cs`

See also [ai-agent-modes.md](ai-agent-modes.md) for `MaxIterations` / `QualityThreshold` (used by `SelfCorrection` mode in conjunction with eval scores).
