# Grader References — Human-Graded Gold Standard

> Human expert grades for validating LLM grader accuracy.

## Purpose

These YAML files contain **human-graded reference samples** — real LLM responses
paired with expert human scores across all three dimensions (Task Completion,
Code Correctness, Doc Coverage). The grader validation tool
(`pnpm grader-validate`) runs the current grader on these responses and compares
its scores against the human grades.

## File format

Each YAML file contains an array of reference-graded samples:

```yaml
- taskId: groq-blog-queries
  area: groq
  response: |
    <the LLM's actual response text from an eval run>
  rubrics:
    - dimension: task-completion
      rubricText: |
        <the full rubric text used for grading>
      humanScore: 75
      notes: "Correct GROQ syntax but missed ordering clause"
    - dimension: code-correctness
      rubricText: |
        <the full rubric text>
      humanScore: 60
    - dimension: doc-coverage
      rubricText: |
        <the full rubric text>
      humanScore: 80
  gradedBy: "engineer-name"
  gradedAt: "2026-03-10"
```

## Grading protocol

To minimize subjectivity when creating reference grades:

1. **Use the same rubric text** the LLM grader uses (copy from the expanded task
   file or the eval results JSON)
2. **Two independent graders per sample**, reconcile disagreements
3. **Focus on edge cases** — samples where the grader is likely to be wrong
   (subtle bugs, plausible but incorrect code, partially correct responses)
4. **Score on the 0–100 scale** matching the rubric bands:
   - 0–19: Failed / missing critical info
   - 20–49: Attempted but fundamentally wrong
   - 50–79: Partial / mostly correct
   - 80–100: Complete / fully correct
5. **Include notes** explaining your reasoning, especially for non-obvious
   scores. These help diagnose grader failures.

## How to extract responses from eval results

```bash
# View a specific gold test response from eval results:
python3 -c "
import json
with open('results/latest/eval-results.json') as f:
    data = json.load(f)
for r in data['results']['results']:
    desc = r.get('testCase',{}).get('description','')
    if 'groq-blog-queries' in desc.lower() and '(gold)' in desc:
        print(r['response']['output'][:2000])
        break
"
```

## Coverage goals

Start with 2–3 reference-graded samples per feature area, covering:

- A high-quality response (scores mostly 80+)
- A medium-quality response (scores 40–70)
- A low-quality or incorrect response (scores 0–30)

This gives the validation tool a range of quality levels to check the grader's
discrimination power across the full scale.

## Related

- [Grader Reliability Plan](../../../docs/archive/exec-plans/grader-reliability.md)
  — Phase 2
- [Rubric Templates](../../config/rubrics.yaml) — the rubrics used for grading
