# Fact Validator Implementation Plan

> **Note (2026-01-28)**: This is a historical planning document. The core `on_finish` hook
> infrastructure has been implemented and refactored since this plan was written.
> Implementation has moved from `src/check-execution-engine.ts` to a dedicated module at
> `src/engine/on-finish/orchestrator.ts` and `src/engine/on-finish/utils.ts`. Method names
> like `executeCheckInline()` and `handleOnFinishHooks()` mentioned below no longer exist.
> The test file references may also be outdated. See the actual source files for current
> implementation details.

## 🎯 CURRENT STATUS: 50% COMPLETE (Phases 1-4 done, Phases 5-6 pending)

**Last Updated:** 2025-10-16 (see note above for 2026 status)

### Implementation Status

| Component | Status | Files |
|-----------|--------|-------|
| **Phase 1: Infrastructure** | ✅ COMPLETE | `src/types/config.ts`, `src/config.ts` |
| **Phase 2: Execution** | ✅ COMPLETE | `src/engine/on-finish/orchestrator.ts`, `src/engine/on-finish/utils.ts` |
| **Phase 3: Configuration** | ✅ COMPLETE | `examples/fact-validator.yaml` |
| **Phase 4: Documentation** | ✅ COMPLETE | `docs/failure-routing.md`, `docs/dependencies.md`, `docs/foreach-dependency-propagation.md` |
| **Phase 5: Testing** | 🚧 PARTIAL | `tests/engine/foreach-on-finish.engine.test.ts`, `tests/unit/on-finish-validation.test.ts` |
| **Phase 6: Deployment** | ⏳ PENDING | - |

### Test Results

✅ **16 on_finish-specific tests passing** (as of 2026-01-28)

### What's Working

- ✅ `on_finish` hook infrastructure (types, validation, detection)
- ✅ Full execution logic (`run`, `goto_js`, error handling, logging)
- ✅ Complete fact validator example configuration
- ✅ Memory-based retry tracking
- ✅ forEach iteration with aggregation
- ✅ Dynamic routing based on validation results

### Quick Test

```bash
npm run build
./dist/index.js --config examples/fact-validator.yaml --event issue_opened --cli
```

---

## Executive Summary

This plan implements an AI-powered fact validation system for GitHub issue and comment assistants. Before posting responses, the system:

1. **Extracts** all factual claims from the AI-generated response
2. **Validates** each fact individually using AI with code/documentation search
3. **Aggregates** validation results to determine overall accuracy
4. **Retries** the original assistant with fact-checking context if validation fails
5. **Posts** the response only after facts are verified (or max attempts reached)

**Key Feature**: Introduces a new `on_finish` routing hook on the forEach check itself to handle aggregation and routing after ALL dependent checks complete ALL iterations.

**Configuration**: Controlled by `ENABLE_FACT_VALIDATION` environment variable (enabled by default in workflows).

## Critical Design Decision: `on_finish` on forEach Check

### The Winning Argument: Multiple Dependents in forEach

The `on_finish` hook **MUST** be on the forEach check itself, not on dependent checks.

**Why?** Consider this scenario:

```yaml
checks:
  extract-facts:
    type: ai
    forEach: true
    # Outputs: [fact1, fact2, fact3]

  validate-security:
    depends_on: [extract-facts]
    # Runs 3 times, validates security aspects

  validate-technical:
    depends_on: [extract-facts]
    # Runs 3 times, validates technical aspects

  validate-format:
    depends_on: [extract-facts]
    # Runs 3 times, validates format/style
```

**With `on_finish` on forEach check:**
```yaml
extract-facts:
  forEach: true
  on_finish:  # ✅ Runs ONCE after ALL dependents complete ALL iterations
    run: [aggregate-all-validations]
    goto_js: |
      const securityValid = outputs.history['validate-security'].every(r => r.is_valid);
      const technicalValid = outputs.history['validate-technical'].every(r => r.is_valid);
      const formatValid = outputs.history['validate-format'].every(r => r.is_valid);

      if (!securityValid || !technicalValid || !formatValid) {
        return 'retry-with-context';
      }
      return null;
```

**This is the ONLY way to aggregate across ALL dependent checks!**

If `on_finish` were on dependent checks, each would trigger separately and there would be no single point to aggregate results from all dependents.

## User Requirements Met

- ✅ Fact validator enabled conditionally via environment variable
- ✅ Enabled by default in GitHub workflow
- ✅ Extract facts after AI generates response
- ✅ forEach validation of each fact individually
- ✅ **NEW**: `on_finish` hook on forEach check for post-iteration aggregation and routing
- ✅ Check if all facts valid after forEach completes
- ✅ Route back to original assistant with context if invalid (max 2 attempts)
- ✅ Track attempts in memory to prevent infinite loops
- ✅ Liquid templates check attempt count and fact data from memory
- ✅ Second pass includes fact validation context for AI to self-correct

## Architecture

### High-Level Flow

```mermaid
graph TD
    A[Issue/Comment Trigger] --> B[init-fact-validation-memory]
    B --> C[issue-assistant or comment-assistant]
    C --> D{ENABLE_FACT_VALIDATION?}
    D -->|No| E[post-direct-response]
    D -->|Yes| F[extract-facts: forEach with on_finish]
    F --> G{Any facts to validate?}
    G -->|No facts| E
    G -->|Yes| H[validate-fact: depends_on extract-facts]
    H --> I[forEach Iterations: fact 1...N]
    I --> J[on_finish triggers on extract-facts]
    J --> K[aggregate-validations via on_finish.run]
    K --> L{on_finish.goto_js evaluates}
    L -->|All valid| M[return null → proceed]
    L -->|Invalid + attempt < 2| N[increment attempt, return issue-assistant]
    L -->|Invalid + attempt >= 1| O[return null → give up]
    M --> P[post-verified-response]
    N --> C
    O --> Q[post-unverified-warning]

    style F fill:#ffeb3b,stroke:#f57c00,stroke-width:3px
    style J fill:#ffeb3b,stroke:#f57c00,stroke-width:3px

    classDef newFeature fill:#4caf50,stroke:#2e7d32,stroke-width:2px
    class F,J newFeature
```

**Legend:**
- 🟡 **Yellow boxes**: Components involving the new `on_finish` hook
- 🟢 **Green**: New features requiring implementation

### Detailed Flow with on_finish

```mermaid
graph TD
    A[issue-assistant generates response] --> B[extract-facts: AI with forEach]
    B --> C[validate-fact runs N times]
    C --> D1[Iteration 1]
    C --> D2[Iteration 2]
    C --> DN[Iteration N]
    D1 --> E[All iterations complete]
    D2 --> E
    DN --> E
    E --> F[on_finish.run executes]
    F --> G[aggregate-validations runs]
    G --> H[on_finish.goto_js evaluates]
    H -->|allValid = true| I[return null]
    H -->|allValid = false, attempt = 0| J[increment, return issue-assistant]
    H -->|allValid = false, attempt >= 1| K[return null]
    I --> L[post-verified-response]
    J --> A
    K --> M[post-unverified-warning]

    style E fill:#ffeb3b,stroke:#f57c00,stroke-width:3px
    style F fill:#ffeb3b,stroke:#f57c00,stroke-width:3px
```

## Complete Implementation

### 1. Environment Variable Configuration

Add to `.visor.yaml`:
```yaml
env:
  ENABLE_FACT_VALIDATION: "${{ env.ENABLE_FACT_VALIDATION }}"
```

Add to GitHub workflow (`.github/workflows/visor.yml`):
```yaml
env:
  ENABLE_FACT_VALIDATION: "true"  # Enable by default
```

### 2. Memory Initialization

```yaml
checks:
  init-fact-validation:
    type: memory
    operation: set
    key: fact_validation_attempt
    value: 0
    namespace: fact-validation
    on: [issue_opened, issue_comment]
    if: "env.ENABLE_FACT_VALIDATION === 'true'"
```

### 3. Update Assistants with Retry Context

```yaml
  issue-assistant:
    type: ai
    group: dynamic
    schema: issue-assistant
    depends_on: [init-fact-validation]
    prompt: |
      {% if memory.has('fact_validation_issues', 'fact-validation') %}
      ## ⚠️ Previous Fact Validation Issues
      The following facts were incorrect in your previous response:

      {% assign issues = memory.get('fact_validation_issues', 'fact-validation') %}
      {% for issue in issues %}
      - **{{ issue.claim }}** - {{ issue.issue }}
        - Evidence: {{ issue.evidence }}
        {% if issue.correction %}
        - Correct information: {{ issue.correction }}
        {% endif %}
      {% endfor %}

      Please correct these facts in your new response.
      {% endif %}

      You are a GitHub issue assistant for the {{ event.repository.fullName }} repository.

      [Rest of your existing issue-assistant prompt...]
    on: [issue_opened]

  comment-assistant:
    type: ai
    group: dynamic
    schema: issue-assistant
    depends_on: [init-fact-validation]
    prompt: |
      {% if memory.has('fact_validation_issues', 'fact-validation') %}
      ## ⚠️ Previous Fact Validation Issues
      [Same as above...]
      {% endif %}

      You are a GitHub comment assistant for {{ event.repository.fullName }}.

      [Rest of your existing comment-assistant prompt...]
    on: [issue_comment]
```

### 4. Fact Extraction (forEach with on_finish)

```yaml
  extract-facts:
    type: ai
    group: fact-validation
    schema: plain
    depends_on: [issue-assistant, comment-assistant]
    prompt: |
      Extract all factual claims from the following assistant response.

      Response:
      ```
      {{ outputs['issue-assistant'].text || outputs['comment-assistant'].text }}
      ```

      Extract facts in these categories:
      1. **Technical Facts**: How code/features work
      2. **Configuration Facts**: Settings, files, configuration
      3. **Documentation Facts**: References to docs, guides, examples
      4. **Version Facts**: Versions, releases, compatibility
      5. **Feature Facts**: Feature availability or behavior
      6. **Process Facts**: Workflows, procedures, best practices

      Return a JSON array of fact objects:
      ```json
      [
        {
          "id": "fact-1",
          "category": "Configuration Facts",
          "claim": "The default config file is .visor.yaml",
          "context": "Mentioned when explaining Visor setup",
          "verifiable": true
        }
      ]
      ```

      Return ONLY the JSON array. If no verifiable facts, return: []
    transform_js: |
      try {
        const parsed = JSON.parse(output);
        const verifiable = parsed.filter(f => f.verifiable === true);
        return verifiable;
      } catch (e) {
        log('Failed to parse facts:', e);
        return [];
      }
    forEach: true  # ← Makes this a forEach check

    # ✅ on_finish runs ONCE after ALL dependents complete
    on_finish:
      # First, run aggregation
      run: [aggregate-validations]

      # Then, routing decision based on aggregated results
      goto_js: |
        const allValid = memory.get('all_facts_valid', 'fact-validation');
        const attempt = memory.get('fact_validation_attempt', 'fact-validation') || 0;

        log('Fact validation complete - allValid:', allValid, 'attempt:', attempt);

        if (allValid) {
          log('All facts valid, proceeding to post');
          return null;  // Continue to post checks
        }

        if (attempt >= 1) {
          log('Max attempts reached, giving up');
          return null;  // Continue to warning check
        }

        // Retry with fact validation context
        log('Facts invalid, retrying assistant with context');
        memory.increment('fact_validation_attempt', 1, 'fact-validation');
        return event.name === 'issue_opened' ? 'issue-assistant' : 'comment-assistant';

      goto_event: "{{ event.event_name }}"

    on: [issue_opened, issue_comment]
    if: "env.ENABLE_FACT_VALIDATION === 'true'"
```

### 5. Fact Validation (Dependent Check)

```yaml
  validate-fact:
    type: ai
    group: fact-validation
    schema: plain
    depends_on: [extract-facts]
    prompt: |
      Verify this factual claim using available MCP tools.

      Fact to verify:
      - **Category**: {{ outputs['extract-facts'].category }}
      - **Claim**: {{ outputs['extract-facts'].claim }}
      - **Context**: {{ outputs['extract-facts'].context }}

      ## Verification Process

      1. **Use MCP tools** to search code, read docs, check config
      2. **Search relevant files**:
         - Config facts → .visor.yaml, defaults/.visor.yaml, examples/
         - Feature facts → docs/, src/
         - Doc facts → Verify links and file references
      3. **Cross-reference** with examples and patterns

      Return JSON:
      ```json
      {
        "fact_id": "{{ outputs['extract-facts'].id }}",
        "claim": "{{ outputs['extract-facts'].claim }}",
        "is_valid": true,
        "confidence": "high",
        "evidence": "Found in defaults/.visor.yaml:10",
        "correction": null,
        "sources": ["defaults/.visor.yaml", "docs/configuration.md"]
      }
      ```

      Be thorough. If unsure, mark confidence as "low".
      Return ONLY the JSON object.
    transform_js: |
      try {
        return JSON.parse(output);
      } catch (e) {
        log('Failed to parse validation:', e);
        return {
          fact_id: outputs['extract-facts'].id,
          claim: outputs['extract-facts'].claim,
          is_valid: false,
          confidence: 'low',
          evidence: 'Failed to parse validation response',
          correction: null,
          sources: []
        };
      }
    on: [issue_opened, issue_comment]
    if: "env.ENABLE_FACT_VALIDATION === 'true'"
```

### 6. Aggregate Validation Results

```yaml
  aggregate-validations:
    type: script
    namespace: fact-validation
    content: |
      // Get ALL validation results from forEach iterations
      const validations = outputs.history['validate-fact'];

      log('Aggregating', validations.length, 'validation results');

      // Analyze results
      const invalid = validations.filter(v => !v.is_valid);
      const lowConfidence = validations.filter(v => v.confidence === 'low');
      const allValid = invalid.length === 0 && lowConfidence.length === 0;

      // Store results
      memory.set('all_facts_valid', allValid, 'fact-validation');
      memory.set('validation_results', validations, 'fact-validation');
      memory.set('invalid_facts', invalid, 'fact-validation');
      memory.set('low_confidence_facts', lowConfidence, 'fact-validation');

      // Store issues for retry context
      if (!allValid) {
        const issues = [...invalid, ...lowConfidence].map(v => ({
          claim: v.claim,
          issue: v.is_valid ? 'low confidence' : 'incorrect',
          evidence: v.evidence,
          correction: v.correction
        }));
        memory.set('fact_validation_issues', issues, 'fact-validation');
      }

      return {
        total: validations.length,
        valid: validations.filter(v => v.is_valid && v.confidence !== 'low').length,
        invalid: invalid.length,
        low_confidence: lowConfidence.length,
        all_valid: allValid,
        summary: allValid
          ? 'All facts validated successfully'
          : `Found ${invalid.length} invalid and ${lowConfidence.length} low-confidence facts`
      };
    on: [issue_opened, issue_comment]
```

### 7. Post Verified Response

```yaml
  post-verified-response:
    type: github
    tags: [github]
    depends_on: [extract-facts]
    on: [issue_opened, issue_comment]
    if: |
      env.ENABLE_FACT_VALIDATION === 'true' &&
      memory.get('all_facts_valid', 'fact-validation') === true
    op: comment.create
    value: |
      {{ outputs['issue-assistant'].text || outputs['comment-assistant'].text }}
```

### 8. Post Unverified Warning

```yaml
  post-unverified-warning:
    type: github
    tags: [github]
    depends_on: [extract-facts]
    on: [issue_opened, issue_comment]
    if: |
      env.ENABLE_FACT_VALIDATION === 'true' &&
      memory.get('all_facts_valid', 'fact-validation') === false &&
      memory.get('fact_validation_attempt', 'fact-validation') >= 1
    op: comment.create
    value: |
      ⚠️ **Fact Validation Warning**

      I attempted to respond to your {{ event.name === 'issue_opened' ? 'issue' : 'comment' }}, but could not verify all factual claims after validation.

      **Issues Found:**
      {% assign invalid = memory.get('invalid_facts', 'fact-validation') %}
      {% for fact in invalid %}
      - **{{ fact.claim }}**: {{ fact.evidence }}
        {% if fact.correction %}
        - Correction: {{ fact.correction }}
        {% endif %}
      {% endfor %}

      {% assign lowConf = memory.get('low_confidence_facts', 'fact-validation') %}
      {% if lowConf.size > 0 %}

      **Low Confidence:**
      {% for fact in lowConf %}
      - **{{ fact.claim }}**: {{ fact.evidence }}
      {% endfor %}
      {% endif %}

      I recommend having a human team member review this to provide accurate information.
```

### 9. Direct Post (Validation Disabled)

```yaml
  post-direct-response:
    type: github
    tags: [github]
    depends_on: [issue-assistant, comment-assistant]
    on: [issue_opened, issue_comment]
    if: "env.ENABLE_FACT_VALIDATION !== 'true'"
    op: comment.create
    value: |
      {{ outputs['issue-assistant'].text || outputs['comment-assistant'].text }}
```

## on_finish Hook Implementation Requirements

### Type Definitions (`src/types/config.ts`)

```typescript
interface CheckConfig {
  type: string;
  forEach?: boolean;
  on_finish?: RoutingAction;  // NEW: Only valid on forEach checks
  on_success?: RoutingAction;
  on_fail?: RoutingAction;
  // ... other fields
}

interface RoutingAction {
  run?: string[];
  run_js?: string;
  goto?: string;
  goto_js?: string;
  goto_event?: string;
  retry?: RetryConfig;
}
```

### Execution Engine (`src/check-execution-engine.ts`)

**Key behaviors to implement:**

1. **Trigger Condition**:
   - Only on checks with `forEach: true`
   - Triggers AFTER all dependent checks complete ALL their iterations
   - Does NOT trigger if forEach array is empty

2. **Execution Order**:
   ```
   forEach check executes once → outputs array
   ↓
   All dependent checks execute N times (forEach propagation)
   ↓
   on_finish.run executes (checks in order)
   ↓
   on_finish.run_js evaluates (additional dynamic checks)
   ↓
   on_finish.goto_js evaluates (routing decision)
   ↓
   If goto returned, jump to ancestor check
   ```

3. **Context Available**:
   ```javascript
   {
     step: { id: 'extract-facts', tags: [...], group: '...' },
     attempt: 1,
     loop: 2,
     outputs: {
       'extract-facts': [...],        // Array of items
       'validate-fact': [...],        // Array of ALL results
     },
     outputs.history: {
       'extract-facts': [[...], ...], // Cross-loop history
       'validate-fact': [[...], ...],
     },
     forEach: {
       total: 3,
       successful: 3,
       failed: 0,
       items: [...]
     },
     memory,
     pr,
     files,
     env
   }
   ```

4. **Error Handling**:
   - If `on_finish.run` checks fail, mark forEach check as failed
   - If `goto_js` throws error, fallback to static `goto` or skip routing
   - Clear error if `on_finish` used on non-forEach check

5. **Loop Safety**:
   - Count `on_finish.goto` toward `max_loops`
   - Track routing transitions
   - Abort with clear message if max_loops exceeded

### Schema Validation (`src/config.ts`)

```typescript
// Validation rules:
// 1. on_finish only allowed on checks with forEach: true
// 2. on_finish.goto must be ancestor (same as on_success/on_fail)
// 3. on_finish.goto_js must return string or null
// 4. on_finish.run must be array of valid check IDs

if (check.on_finish && !check.forEach) {
  throw new Error(
    `Check '${checkId}' has on_finish but forEach is not true. ` +
    `on_finish is only valid on forEach checks.`
  );
}
```

### Documentation Updates

1. **`docs/failure-routing.md`**: Add `on_finish` section ✅
2. **`docs/dependencies.md`**: Explain `on_finish` with forEach ✅
3. **`examples/fact-validator.yaml`**: Complete working example ✅
4. **`docs/foreach-dependency-propagation.md`**: Update with `on_finish` ✅

## Testing Strategy

### Unit Tests

1. **on_finish Trigger**:
   - Triggers after last forEach iteration
   - Does NOT trigger for empty arrays
   - Does NOT trigger on non-forEach checks

2. **on_finish Execution Order**:
   - `run` executes before `goto`
   - `run_js` evaluates after `run`
   - `goto_js` evaluates last

3. **on_finish Context**:
   - Has access to `outputs.history` of all dependents
   - Has `forEach` stats (total, successful, failed)
   - Has correct `attempt` and `loop` counters

4. **Error Cases**:
   - Error if `on_finish` on non-forEach check
   - Handle `goto_js` errors gracefully
   - Handle failed `on_finish.run` checks

### Integration Tests

1. **Full Fact Validation Flow**:
   - Issue opened → assistant → extract → validate → aggregate → post
   - All facts valid → direct post
   - Some facts invalid → retry → validate → post

2. **Retry Logic**:
   - Invalid facts → retry once with context → validate → post
   - Invalid facts → retry → still invalid → warning post

3. **Edge Cases**:
   - No facts extracted → direct post
   - All facts low confidence → retry
   - Validation disabled → direct post

4. **Multiple Dependents**:
   - forEach with multiple dependent checks
   - Aggregate across all dependents
   - Route based on combined results

### E2E Tests

1. **GitHub Issue Opened**:
   - Real issue → assistant response → fact validation → post
2. **Comment on Issue**:
   - Comment → response → invalid facts → retry → post
3. **Validation Disabled**:
   - Comment → response → direct post (no validation)

## Quick Reference

### Configuration Checklist

- [x] Add `ENABLE_FACT_VALIDATION` to workflow env (default: "true") ✅
- [x] Implement `on_finish` hook in execution engine ✅
- [x] Add type definitions for `on_finish` ✅
- [x] Add schema validation for `on_finish` ✅
- [x] Add memory initialization check ✅
- [x] Add fact extraction AI check with `forEach` and `on_finish` ✅
- [x] Add fact validation AI check (depends on extract-facts) ✅
- [x] Add aggregation memory check ✅
- [x] Add response posting checks (verified/unverified/direct) ✅
- [x] Update issue-assistant with retry context ✅
- [x] Update comment-assistant with retry context ✅

**Status:** All core configuration items complete. See `examples/fact-validator.yaml` for reference.

### Check Names

```yaml
init-fact-validation        # Memory: Initialize attempt counter
issue-assistant             # AI: Generate issue response
comment-assistant           # AI: Generate comment response
extract-facts               # AI: Extract facts (forEach: true, on_finish)
validate-fact               # AI: Validate each fact (depends on extract-facts)
aggregate-validations       # Memory: Collect validation results (run by on_finish)
post-verified-response      # GitHub: Post response (all facts valid)
post-unverified-warning     # GitHub: Post warning (max attempts reached)
post-direct-response        # GitHub: Post directly (validation disabled)
```

### Memory Keys (namespace: fact-validation)

```yaml
fact_validation_attempt     # number: Current attempt (0, 1, 2)
all_facts_valid             # boolean: True if all facts validated
validation_results          # array: All validation results
invalid_facts               # array: Facts that failed validation
low_confidence_facts        # array: Facts with low confidence
fact_validation_issues      # array: Issues for retry context
```

### Liquid Template Examples

```liquid
{# Check if fact validation is enabled #}
{% if env.ENABLE_FACT_VALIDATION == 'true' %}

{# Get current attempt #}
{% assign attempt = "fact_validation_attempt" | memory_get: "fact-validation" %}

{# Check if all facts are valid #}
{% assign allValid = "all_facts_valid" | memory_get: "fact-validation" %}

{# Get invalid facts #}
{% assign invalidFacts = "invalid_facts" | memory_get: "fact-validation" %}
{% for fact in invalidFacts %}
  - {{ fact.claim }}: {{ fact.evidence }}
{% endfor %}

{% endif %}
```

### JavaScript Examples

```javascript
// In on_finish.goto_js
const allValid = memory.get('all_facts_valid', 'fact-validation');
const attempt = memory.get('fact_validation_attempt', 'fact-validation') || 0;

// Get all validation results from forEach iterations
const results = outputs.history['validate-fact'];
const allResultsValid = results.every(r => r.is_valid);

// Decide whether to retry
if (!allValid && attempt < 1) {
  memory.increment('fact_validation_attempt', 1, 'fact-validation');
  return event.name === 'issue_opened' ? 'issue-assistant' : 'comment-assistant';
}
return null;
```

## Rollout Plan

### Phase 1: on_finish Hook Implementation (3-4 days)
- Implement TypeScript types
- Implement execution engine logic
- Add schema validation
- Write unit tests
- Update documentation

### Phase 2: Fact Validator Configuration (2-3 days)
- Create complete .visor.yaml configuration
- Implement all checks
- Test forEach and on_finish integration
- Write integration tests

### Phase 3: Testing & Refinement (2-3 days)
- E2E testing with real scenarios
- Tune prompts for better fact extraction
- Tune confidence thresholds
- Performance testing

### Phase 4: Documentation & Examples (1-2 days)
- Complete user documentation
- Add example configurations
- Create debugging guide
- Write best practices

### Phase 5: Deployment (1 day)
- Enable in staging
- Monitor metrics
- Enable in production
- Set up alerts

**Total**: 9-13 days

## Success Metrics

1. **Accuracy Rate**: % of responses with all facts validated correctly
2. **Validation Coverage**: % of responses that go through validation
3. **Retry Rate**: % requiring retry after failed validation
4. **False Positive Rate**: % of valid facts marked invalid
5. **Performance**: Average latency added by validation
6. **User Satisfaction**: Feedback on response accuracy

## Risk Assessment

| Risk | Probability | Impact | Mitigation |
|------|-------------|--------|------------|
| High latency | High | Medium | Configurable enable/disable, optimize prompts |
| False positives | Medium | High | Thorough testing, tune thresholds |
| Infinite loops | Low | High | Hard limit (max 2 attempts), memory tracking |
| API quota exhaustion | Medium | Medium | Rate limiting, caching |

## Conclusion

This fact validation system significantly improves AI response accuracy while maintaining flexibility. The key innovation is the `on_finish` hook on forEach checks, enabling clean aggregation and routing after all dependent validations complete.

### Key Benefits

1. **on_finish Hook**: Clean primitive for post-forEach processing
2. **Self-Correcting AI**: Retry with validation context
3. **Granular Validation**: Each fact validated independently
4. **Safe Loops**: Memory tracking prevents recursion
5. **Flexible**: Easy to enable/disable per environment

---

## 📋 IMPLEMENTATION TASK LIST

### ✅ PHASE 1: Core `on_finish` Hook Infrastructure (COMPLETED)

**Priority: P0 (CRITICAL) - Status: ✅ DONE**

- [x] **1.1** Create `OnFinishConfig` interface in `src/types/config.ts`
  - [x] Add `run?: string[]` field
  - [x] Add `goto?: string` field
  - [x] Add `goto_event?: EventTrigger` field
  - [x] Add `goto_js?: string` field
  - [x] Add `run_js?: string` field
  - [x] Add `on_finish?: OnFinishConfig` to `CheckConfig`

- [x] **1.2** Add schema validation in `src/config.ts`
  - [x] Validate `on_finish` only allowed on `forEach: true` checks
  - [x] Add clear error messages
  - [x] Fix missing 'memory' check type

- [x] **1.3** Implement detection & triggering in `src/check-execution-engine.ts`
  - [x] Create `handleOnFinishHooks()` method
  - [x] Detect forEach checks with `on_finish`
  - [x] Verify all dependents completed
  - [x] Build context (outputs, forEach stats, memory, PR)
  - [x] Skip empty forEach arrays

- [x] **1.4** Write comprehensive tests
  - [x] Unit tests for validation (3 tests)
  - [x] E2E tests for integration (4 tests)
  - [x] Verify no regressions (full suite)

**Result:** ✅ All 1426 tests passing, infrastructure solid

---

### ✅ PHASE 2: Complete `on_finish` Execution (COMPLETED)

**Priority: P0 (CRITICAL - BLOCKING) - Status: ✅ DONE**

- [x] **2.1** Detection and context building ✅
  - [x] Detect `on_finish` configuration
  - [x] Build forEach stats
  - [x] Prepare outputs context
  - [x] Create memory helpers

- [x] **2.2** Implement `executeCheckInline()` method ✅
  - [x] Extract/refactor `executeNamedCheckInline` from `executeWithRouting()`
  - [x] Make it a class method accessible to `handleOnFinishHooks()`
  - [x] Handle check dependencies
  - [x] Update results map
  - [x] Support event override (`goto_event`)
  - [x] Add error handling
  - **Files modified:** `src/check-execution-engine.ts` (lines 283-470)

- [x] **2.3** Implement `on_finish.run` execution ✅
  - [x] Iterate through `onFinish.run` array
  - [x] Call `executeCheckInline()` for each check
  - [x] Execute in order (sequential)
  - [x] Propagate errors properly
  - [x] Log execution start/complete
  - **Files modified:** `src/check-execution-engine.ts` (lines 625-668)

- [x] **2.5** Implement `on_finish.goto_js` evaluation ✅
  - [x] Compile JS expression with Sandbox
  - [x] Evaluate with full context
  - [x] Extract goto target (string or null)
  - [x] Call `executeCheckInline()` with target
  - [x] Support `goto_event` override
  - [x] Handle null return (no routing)
  - [x] Add loop safety checks
  - **Files modified:** `src/check-execution-engine.ts` (lines 670-748)

- [x] **2.6** Implement static `on_finish.goto` execution ✅
  - [x] Check for static goto string
  - [x] Call `executeCheckInline()` with target
  - [x] Support `goto_event` override
  - **Files modified:** `src/check-execution-engine.ts` (lines 710-713)

- [x] **2.7** Add comprehensive error handling ✅
  - [x] Try-catch around `run` execution
  - [x] Try-catch around `goto_js` evaluation
  - [x] Fallback to static `goto` on `goto_js` error
  - [x] Clear error messages
  - [x] Log all errors

- [x] **2.8** Add execution logging ✅
  - [x] Log `on_finish` start
  - [x] Log each check execution
  - [x] Log routing decisions
  - [x] Log completion
  - [x] Debug output for context

- [x] **2.9** Write tests for execution ✅
  - [x] Unit test: `on_finish.run` executes in order
  - [x] Unit test: `on_finish.goto_js` routes correctly
  - [x] Unit test: Error handling works
  - [x] E2E test: Full routing flow
  - [x] E2E test: Retry with memory
  - **Files:** `tests/unit/on-finish-validation.test.ts`, `tests/e2e/foreach-on-finish.test.ts`
  - **Result:** 1438 tests passing, 0 failures

**Total Phase 2: COMPLETED** ✅

---

### ✅ PHASE 3: Fact Validator Configuration (COMPLETED)

**Priority: P1 (HIGH) - Status: ✅ DONE**

- [x] **3.1** Create memory initialization check ✅
  - [x] Type: `memory`
  - [x] Operation: `set`
  - [x] Key: `fact_validation_attempt`
  - [x] Value: `0`
  - [x] Namespace: `fact-validation`
  - [x] Add `if` condition for `ENABLE_FACT_VALIDATION`
  - **File:** `examples/fact-validator.yaml`

- [x] **3.2** Update issue-assistant prompt ✅
  - [x] Add Liquid template check for `memory.has('fact_validation_issues')`
  - [x] Display previous validation failures
  - [x] Format correction context clearly
  - [x] Test template rendering
  - **File:** `examples/fact-validator.yaml`

- [x] **3.3** Update comment-assistant prompt ✅
  - [x] Add same Liquid template logic as issue-assistant
  - [x] Ensure consistency in error display
  - [x] Test template rendering
  - **File:** `examples/fact-validator.yaml`

- [x] **3.4** Create extract-facts check ✅
  - [x] Type: `command` (demo) / `ai` (production)
  - [x] Schema: `plain`
  - [x] Depends on: `[issue-assistant, comment-assistant]`
  - [x] Write comprehensive extraction prompt
  - [x] Add `transform_js` to parse and filter facts
  - [x] Set `forEach: true`
  - [x] Add `on_finish` configuration:
    - [x] `run: [aggregate-validations]`
    - [x] `goto_js`: Check validation results and route
    - [x] `goto_event`: Pass through event
  - [x] Add `if` condition for `ENABLE_FACT_VALIDATION`
  - **File:** `examples/fact-validator.yaml`

- [x] **3.5** Create validate-fact check ✅
  - [x] Type: `command` (demo) / `ai` (production)
  - [x] Schema: `plain`
  - [x] Depends on: `[extract-facts]`
  - [x] Write validation prompt with MCP tools
  - [x] Instructions for using code search
  - [x] Instructions for reading documentation
  - [x] Add `transform_js` to parse validation result
  - [x] Handle parse errors gracefully
  - [x] Add `if` condition for `ENABLE_FACT_VALIDATION`
  - **File:** `examples/fact-validator.yaml`

- [x] **3.6** Create aggregate-validations check ✅
  - [x] Type: `script`
  - [x] Namespace: `fact-validation`
  - [x] Use `content` to:
    - [x] Read all validation results from `outputs.history['validate-fact']`
    - [x] Calculate `all_facts_valid`
    - [x] Store `invalid_facts`
    - [x] Store `low_confidence_facts`
    - [x] Store `fact_validation_issues` for retry
    - [x] Return summary object
  - **File:** `examples/fact-validator.yaml`

- [x] **3.7** Create post-verified-response check ✅
  - [x] Type: `logger` (demo) / `github` (production)
  - [x] Depends on: `[extract-facts]`
  - [x] Operation: `comment.create`
  - [x] Add `if` condition: `all_facts_valid === true`
  - [x] Value: Original assistant response
  - **File:** `examples/fact-validator.yaml`

- [x] **3.8** Create post-unverified-warning check ✅
  - [x] Type: `logger` (demo) / `github` (production)
  - [x] Depends on: `[extract-facts]`
  - [x] Operation: `comment.create`
  - [x] Add `if` condition: `all_facts_valid === false && attempt >= 1`
  - [x] Value: Warning with validation issues
  - [x] Format invalid and low-confidence facts
  - **File:** `examples/fact-validator.yaml`

- [x] **3.9** Create post-direct-response check ✅
  - [x] Type: `logger` (demo) / `github` (production)
  - [x] Depends on: `[issue-assistant, comment-assistant]`
  - [x] Operation: `comment.create`
  - [x] Add `if` condition: `ENABLE_FACT_VALIDATION !== 'true'`
  - [x] Value: Direct assistant response
  - **File:** `examples/fact-validator.yaml` (commented out in demo)

- [x] **3.10** Add environment configuration ✅
  - [x] Add to configuration: `env.ENABLE_FACT_VALIDATION`
  - [x] Test environment variable passing
  - **File:** `examples/fact-validator.yaml`

**Total Phase 3: COMPLETED** ✅

**Deliverable:** Complete working example at `examples/fact-validator.yaml` (474 lines) demonstrating:
- ✅ Memory-based retry tracking
- ✅ Conditional assistant prompts with validation context
- ✅ forEach fact extraction
- ✅ Parallel fact validation (N iterations)
- ✅ on_finish aggregation with `run` and `goto_js`
- ✅ Smart routing based on validation results
- ✅ Multiple posting paths (verified/unverified/direct)

**Test Command:** `./dist/index.js --config examples/fact-validator.yaml --event issue_opened --cli`

---

### ✅ PHASE 4: Documentation & Examples (COMPLETED)

**Priority: P2 (MEDIUM) - Status: ✅ DONE**

- [x] **4.1** Update `docs/failure-routing.md` ✅
  - [x] Add `on_finish` section (260 lines)
  - [x] Explain when it triggers
  - [x] Show examples with forEach
  - [x] Document context available
  - **File:** `docs/failure-routing.md` (lines 160-420)

- [x] **4.2** Update `docs/dependencies.md` ✅
  - [x] Explain `on_finish` with forEach propagation
  - [x] Show dependency chain examples
  - [x] Document best practices
  - **File:** `docs/dependencies.md` (lines 83-315)

- [x] **4.3** Complete working example ✅
  - [x] Created `examples/fact-validator.yaml` (474 lines)
  - [x] Added comprehensive comments
  - [x] Tested and verified working
  - **File:** `examples/fact-validator.yaml`

- [x] **4.4** Update `docs/foreach-dependency-propagation.md` ✅
  - [x] Add `on_finish` section (372 lines)
  - [x] Show how it fits in propagation
  - [x] Updated with complete lifecycle
  - **File:** `docs/foreach-dependency-propagation.md` (lines 69-441)

**Total Phase 4: COMPLETED** ✅

**Deliverables:**
- 864 lines of documentation added across 3 core docs
- Complete working example with inline documentation
- Cross-references and best practices throughout
- Common pitfalls and debugging tips included

---

### ✅ PHASE 5: Testing & Refinement (PARTIALLY COMPLETE)

**Priority: P1 (HIGH) - Status: 🚧 50% DONE**

- [x] **5.1** E2E test: Full fact validation flow ✅
  - [x] Test with real issue scenarios (3 tests)
  - [x] Verify extraction works
  - [x] Verify validation works
  - [x] Verify retry works
  - [x] Verify posting works
  - **File:** `tests/e2e/foreach-on-finish.test.ts`

- [x] **5.2** E2E test: Retry flow ✅
  - [x] Create issue with invalid facts (3 tests)
  - [x] Verify assistant retries
  - [x] Verify correction context passed
  - [x] Verify max attempts enforced
  - **File:** `tests/e2e/foreach-on-finish.test.ts`

- [x] **5.3** E2E test: Empty facts ✅
  - [x] Test response with no facts (3 tests)
  - [x] Verify direct posting
  - [x] Verify no validation runs
  - **File:** `tests/e2e/foreach-on-finish.test.ts`

- [x] **5.4** E2E test: Validation disabled ✅
  - [x] Set `ENABLE_FACT_VALIDATION=false` (2 tests)
  - [x] Verify direct posting
  - [x] Verify no validation runs
  - **File:** `tests/e2e/foreach-on-finish.test.ts`
  - **Result:** 20 tests passing (11 new + 9 existing)

- [ ] **5.5** Prompt tuning: Fact extraction
  - [ ] Test with various response types
  - [ ] Tune categories
  - [ ] Reduce false positives
  - [ ] Test edge cases
  - **Estimated effort:** 3-4 hours

- [ ] **5.6** Prompt tuning: Fact validation
  - [ ] Tune MCP tool usage
  - [ ] Improve evidence quality
  - [ ] Reduce false negatives
  - [ ] Test confidence thresholds
  - **Estimated effort:** 3-4 hours

- [ ] **5.7** Performance testing
  - [ ] Measure latency added
  - [ ] Identify bottlenecks
  - [ ] Optimize slow operations
  - [ ] Test with concurrent requests
  - **Estimated effort:** 2-3 hours

- [ ] **5.8** Integration testing
  - [ ] Test with real repository
  - [ ] Test with real issues/comments
  - [ ] Monitor API quota usage
  - [ ] Collect metrics
  - **Estimated effort:** 2-3 hours

**Total Phase 5 Effort:** 16-22 hours

---

### 🚀 PHASE 6: Deployment & Monitoring (NOT STARTED)

**Priority: P3 (LOW) - Status: ⏳ 0% DONE**

- [ ] **6.1** Set up staging environment
  - [ ] Deploy to staging
  - [ ] Enable fact validation
  - [ ] Monitor for errors
  - **Estimated effort:** 1 hour

- [ ] **6.2** Collect baseline metrics
  - [ ] Accuracy rate
  - [ ] Validation coverage
  - [ ] Retry rate
  - [ ] Performance impact
  - **Estimated effort:** 2-3 hours

- [ ] **6.3** Set up monitoring alerts
  - [ ] High error rate
  - [ ] High latency
  - [ ] API quota warnings
  - [ ] Infinite loop detection
  - **Estimated effort:** 1-2 hours

- [ ] **6.4** Production rollout
  - [ ] Gradual rollout (10% → 50% → 100%)
  - [ ] Monitor metrics
  - [ ] Adjust configuration as needed
  - **Estimated effort:** 2-3 hours

- [ ] **6.5** Post-deployment monitoring
  - [ ] Track success metrics
  - [ ] Collect user feedback
  - [ ] Identify improvement areas
  - **Estimated effort:** Ongoing

**Total Phase 6 Effort:** 6-9 hours

---

## 📊 OVERALL PROGRESS SUMMARY

| Phase | Priority | Status | Progress | Effort Remaining |
|-------|----------|--------|----------|------------------|
| **Phase 1: Infrastructure** | P0 | ✅ Done | 100% | 0 hours |
| **Phase 2: Execution** | P0 | ✅ Done | 100% | 0 hours |
| **Phase 3: Configuration** | P1 | ✅ Done | 100% | 0 hours |
| **Phase 4: Documentation** | P2 | ✅ Done | 100% | 0 hours |
| **Phase 5: Testing** | P1 | 🚧 Partial | 50% | 8-11 hours |
| **Phase 6: Deployment** | P3 | ⏳ Not Started | 0% | 6-9 hours |
| **TOTAL** | - | - | **75%** | **14-20 hours** |

---

## 🎯 IMPLEMENTATION PROGRESS

### ✅ Week 1: Complete Execution Layer (P0) - DONE

1. ✅ **Task 2.2** - Implement `executeCheckInline()` (BLOCKER) - COMPLETED
2. ✅ **Task 2.3** - Implement `on_finish.run` execution - COMPLETED
3. ✅ **Task 2.5** - Implement `on_finish.goto_js` evaluation - COMPLETED
4. ✅ **Task 2.9** - Write execution tests - COMPLETED
5. ✅ Verify all tests passing, no regressions - COMPLETED (1438 tests passing)

**Deliverable:** ✅ Fully functional `on_finish` hook with `run` and `goto_js` execution

### ✅ Week 2: Build Fact Validator (P1) - DONE

6. ✅ **Tasks 3.1-3.3** - Memory init + assistant prompt updates - COMPLETED
7. ✅ **Tasks 3.4-3.5** - Extract and validate fact checks - COMPLETED
8. ✅ **Task 3.6** - Aggregation check - COMPLETED
9. ✅ **Tasks 3.7-3.10** - Response posting + env config - COMPLETED
10. ✅ Manual testing of full flow - COMPLETED

**Deliverable:** ✅ Complete fact validator configuration at `examples/fact-validator.yaml`

### 🔄 Week 3: Test & Document (P1-P2) - NEXT

11. ⏳ **Phase 5.1-5.4** - E2E tests (additional scenarios)
12. ⏳ **Phase 5.5-5.6** - Prompt tuning (for production AI usage)
13. ⏳ **Phase 4.1-4.4** - Core documentation
14. ⏳ **Phase 4.5-4.6** - User guides

**Deliverable:** Tested and documented fact validator

---

## ✅ CRITICAL BLOCKERS - RESOLVED

1. ✅ **Task 2.2: `executeCheckInline()` method** - COMPLETED
2. ✅ **Tasks 2.3 + 2.5: `run` and `goto_js`** - COMPLETED
3. ✅ **Phase 2 completion** - COMPLETED

**Status:** All critical blockers have been resolved. The core infrastructure is complete and tested.
