# PR Review System Quality Control Prompt

You are a senior software architect tasked with quality-controlling an AI-powered PR review system. Your goal is to analyze the system holistically, identify gaps between intent and implementation, and provide actionable feedback.

## System Overview

This is a **parallel orchestrator PR review system** that:
1. An orchestrator AI analyzes a PR and delegates to specialist agents
2. Specialist agents (security, quality, logic, codebase-fit) perform deep reviews
3. A finding-validator agent validates all findings against actual code
4. The orchestrator synthesizes results into a final verdict

**Key Design Principles (from vision document):**
- Evidence-based validation (NOT confidence-based)
- Pattern-triggered mandatory exploration (6 semantic triggers)
- Understand intent BEFORE looking for issues
- The diff is the question, not the answer

---

## FILES TO EXAMINE

### Vision & Architecture
- `docs/PR_REVIEW_99_TRUST.md` - The vision document defining 99% trust goal

### Orchestrator Prompts
- `apps/desktop/prompts/github/pr_parallel_orchestrator.md` - Main orchestrator prompt
- `apps/desktop/prompts/github/pr_followup_orchestrator.md` - Follow-up review orchestrator

### Specialist Agent Prompts
- `apps/desktop/prompts/github/pr_security_agent.md` - Security review agent
- `apps/desktop/prompts/github/pr_quality_agent.md` - Code quality agent
- `apps/desktop/prompts/github/pr_logic_agent.md` - Logic/correctness agent
- `apps/desktop/prompts/github/pr_codebase_fit_agent.md` - Codebase fit agent
- `apps/desktop/prompts/github/pr_finding_validator.md` - Finding validator agent

### Implementation Code
- `apps/desktop/src/main/ai/runners/github/parallel-orchestrator-reviewer.ts` - Orchestrator implementation
- `apps/desktop/src/main/ai/runners/github/parallel-followup-reviewer.ts` - Follow-up implementation
- `apps/desktop/src/main/ai/runners/github/models.ts` - Schema definitions (ReviewFinding, VerificationEvidence, etc.)
- `apps/desktop/src/main/ai/runners/github/sdk-utils.ts` - Vercel AI SDK utilities for running agents
- `apps/desktop/src/main/ai/runners/github/review-tools.ts` - Tools available to review agents
- `apps/desktop/src/main/ai/runners/github/context-gatherer.ts` - Gathers PR context (files, callers, dependents)

### Models & Configuration
- `apps/desktop/src/main/ai/runners/github/models.ts` - Data models
- `apps/desktop/src/main/ai/tools/models.ts` - Tool models

---

## ANALYSIS TASKS

### 1. Vision Alignment Check
Compare the implementation against `PR_REVIEW_99_TRUST.md`:

- [ ] **Evidence-based validation**: Is the system truly evidence-based or does it still use confidence scores anywhere?
- [ ] **6 Mandatory Triggers**: Are all 6 semantic triggers properly defined and enforced?
  1. Output contract changed
  2. Input contract changed
  3. Behavioral contract changed
  4. Side effect contract changed
  5. Failure contract changed
  6. Null/undefined contract changed
- [ ] **Phase 0 (Understand Intent)**: Is it mandatory? Is it enforced before delegation?
- [ ] **Phase 1 (Trigger Detection)**: Is it mandatory? Does it output explicit trigger analysis?
- [ ] **Bounded Exploration**: Is exploration limited to depth 1 (direct callers only)?

### 2. Prompt Quality Analysis
For each agent prompt, check:

- [ ] Does it explain WHAT to look for?
- [ ] Does it explain HOW to verify findings?
- [ ] Does it require evidence (code snippets, line numbers)?
- [ ] Does it define when to STOP exploring?
- [ ] Does it distinguish between "in scope" and "out of scope"?
- [ ] Does it handle the "no issues found" case properly?

### 3. Schema Enforcement
Check `models.ts`:

- [ ] Is `VerificationEvidence` required (not optional) on all finding types?
- [ ] Does `VerificationEvidence` require:
  - `code_examined` (actual code, not description)
  - `line_range_examined` (specific lines)
  - `verification_method` (how it was verified)
- [ ] Are there any finding types that bypass evidence requirements?

### 4. Information Flow
Trace how information flows:

- [ ] PR Context → Orchestrator: What context is provided?
- [ ] Orchestrator → Specialists: Are triggers passed? Are known callers passed?
- [ ] Specialists → Validator: Are all findings validated?
- [ ] Validator → Final Output: Are false positives properly dismissed?

### 5. False Positive Prevention
Check mechanisms to prevent false positives:

- [ ] Do specialists verify issues exist before reporting?
- [ ] Does the validator re-read the actual code?
- [ ] Are "missing X" claims (missing error handling, etc.) verified?
- [ ] Are dismissed findings tracked for transparency?

### 6. Log Analysis (ATTACH LOGS BELOW)
When reviewing logs, check:

- [ ] Did the orchestrator output PR UNDERSTANDING before delegating?
- [ ] Did the orchestrator output TRIGGER DETECTION before delegating?
- [ ] Were triggers passed to specialists in delegation prompts?
- [ ] Did specialists actually explore when triggers were present?
- [ ] Were findings validated with real code evidence?
- [ ] Were any false positives caught by the validator?

---

## SPECIFIC QUESTIONS TO ANSWER

1. **Trigger System Effectiveness**: Did the trigger detection system correctly identify semantic contract changes? Were there any missed triggers or false triggers?

2. **Exploration Quality**: When exploration was mandated by a trigger, did specialists explore effectively? Did they stop at the right time?

3. **Evidence Quality**: Are the `code_examined` fields in findings actual code snippets or just descriptions? Are line numbers accurate?

4. **False Positive Rate**: How many findings were dismissed as false positives? What caused them?

5. **Missing Issues**: Based on your understanding of the PR, were there any issues that SHOULD have been caught but weren't?

6. **Prompt Gaps**: Are there any scenarios not covered by the current prompts?

7. **Schema Gaps**: Are there any ways findings could bypass evidence requirements?

---

## OUTPUT FORMAT

Provide your analysis in this structure:

```markdown
## Executive Summary
[2-3 sentences on overall system health]

## Vision Alignment Score: X/10
[Brief explanation]

## Critical Issues (Must Fix)
1. [Issue]: [Description] → [Suggested Fix]
2. ...

## High Priority Improvements
1. [Improvement]: [Why it matters] → [How to implement]
2. ...

## Medium Priority Improvements
1. ...

## Low Priority / Nice to Have
1. ...

## Log Analysis Findings
### What Worked Well
- ...

### What Didn't Work
- ...

### Specific Recommendations from Log Analysis
1. ...

## Questions for the Team
1. [Question that needs human input]
2. ...
```

---

## ATTACH LOGS BELOW

Paste the PR review debug logs here for analysis:

```
[PASTE LOGS HERE]
```

---

## IMPORTANT NOTES

- Focus on **systemic issues**, not one-off bugs
- Prioritize issues that cause **false positives** (annoying) over false negatives (missed issues)
- Consider **language-agnostic** design - the system should work for any codebase
- Think about **edge cases**: empty PRs, huge PRs, refactor-only PRs, CSS-only PRs
- The goal is **99% trust** - developers should trust the review enough to act on it immediately
