# DevForgeAI Audit Module

Adversarial phase auditing for the DevForgeAI framework. Stops Claude Code agents from claiming work they didn't do.

## Why this exists

DevForgeAI's existing five enforcement layers (per-phase reference loading, CLI gates, Execute-Verify-Record triplets, fresh-context subagents, hook-based write protection) are all strong defense-in-depth — but each one has a seam where the agent itself decides whether the gate is satisfied. Token bias lives in the model's priors, so in-band enforcement is bias asking bias for permission.

This module adds a sixth layer that fires **out-of-band**:

- A separate model acts as adversarial judge.
- The judge runs from a Claude Code **hook**, not a subagent — fires deterministically, agent cannot bypass.
- Every "satisfied" verdict requires **citations** (file + line range + verbatim quote).
- Citations are **deterministically verified** against the actual filesystem before being accepted.
- Optional **consensus mode** runs multiple judges from different model families in parallel.

## Install

```bash
pip install pydantic pyyaml requests python-dotenv
```

`python-dotenv` is optional but recommended.

## Quickstart

Add your API key to `.env` in the project root:

```
ANTHROPIC_API_KEY=sk-ant-...
```

That's it. The default config uses `claude-haiku-4-5` via the Anthropic Messages API. No `audit.yaml` needed unless you want to override defaults.

Wire into Claude Code by merging `examples/settings.json` into `.claude/settings.json`:

```json
{
  "hooks": {
    "PostToolUse": [{
      "matcher": "Bash",
      "hooks": [{ "type": "command", "command": "python -m devforgeai.audit.hook" }]
    }]
  }
}
```

The hook runs after every Bash command but exits 0 immediately unless the command matches `devforge phase-complete <phase>`. Audit cost is paid only at phase boundaries.

## Per-phase setup

```
.devforgeai/phases/<phase_name>/
  requirements.json    # list of Requirement objects (see examples/)
  sources.json         # list of source file paths the agent was given
  artifacts/           # populated by the agent during the phase
  consensus.yaml       # OPTIONAL: enables multi-judge consensus mode
```

## Single-judge mode (default)

When no `consensus.yaml` is present, the hook runs a single judge using `.devforgeai/audit.yaml`. Defaults to Anthropic Haiku — fast, cheap, reliable for structured output. See `examples/audit.yaml` for the full set of options.

**The shared-priors tradeoff.** Using Claude Haiku to judge Claude Sonnet/Opus agents weakens the "different model family" mitigation: shared training data, shared RLHF signals, correlated bias on borderline calls. The citation verifier remains the unfakeable layer regardless, but consensus mode (below) is the proper fix for high-stakes work.

## Consensus mode (recommended for high-stakes phases)

Drop a `consensus.yaml` in the phase directory:

```yaml
rule: all  # or 'majority' or 'any'

judges:
  - label: anthropic-haiku
    provider: anthropic
    model: claude-haiku-4-5
    # api_key auto-loaded from ANTHROPIC_API_KEY

  - label: openai-mini
    provider: openai
    endpoint: https://api.openai.com
    model: gpt-4.1-mini
    api_key_env: OPENAI_API_KEY
```

That's the entire opt-in. Phases without a `consensus.yaml` continue to use the single-judge path. Per-phase opt-in lets you reserve consensus mode for the work that warrants it (auth flows, data-integrity boundaries, compliance-relevant artifacts) without paying the cost on routine refactors.

### How consensus works

Judges run **in parallel** via a thread pool. For each judge:

1. The auditor sends the same artifacts and requirements to that judge's endpoint.
2. The judge returns its verdicts.
3. The citation verifier checks every "satisfied" citation against the actual filesystem. Fabricated citations downgrade the verdict to `missing_evidence` per-judge before consensus is applied.

Then the consensus rule decides:

| Rule | Approves when… | Errored judges count as… | Use when… |
| --- | --- | --- | --- |
| `all` | every judge approves | non-approve | high-stakes; fail-closed default |
| `majority` | more than half approve | non-approve | you have 3+ judges and can tolerate one being wrong |
| `any` | at least one approves | irrelevant | almost never; bypasses the point |

**Errored judges (network failure, auth failure, malformed output) never count as approvals** under any rule. This is deliberate — you can't be confident about a verdict you didn't get.

### What the agent sees on consensus failure

```
AUDIT FAIL: phase 'auth_implementation' rejected by consensus (rule=all)

3 judge(s) ran: 1 approved, 2 rejected, 0 errored.

  [APPROVE] anthropic-haiku (anthropic:claude-haiku-4-5): all 5 requirement(s) verified
  [REJECT]  openai-mini (openai:gpt-4.1-mini): 2 requirement(s) unsatisfied
             [missing_evidence] REQ-AUTH-002: token validation function not invoked in middleware
             [missing_evidence] REQ-AUTH-004: refresh-token endpoint missing rate limit
  [REJECT]  local-qwen (openai:qwen2.5-coder:7b): 1 requirement(s) unsatisfied
             [missing_evidence] REQ-AUTH-002: middleware imports validator but does not call it

Disagreements (2 requirement(s)):
  REQ-AUTH-002:
    anthropic-haiku: satisfied
    openai-mini: missing_evidence
    local-qwen: missing_evidence
  REQ-AUTH-004:
    anthropic-haiku: satisfied
    openai-mini: missing_evidence
    local-qwen: satisfied

The phase cannot advance. Address the rejected requirements with concrete,
verifiable evidence in the artifacts under .devforgeai/phases/auth_implementation/artifacts/,
then re-run `devforge phase-complete auth_implementation`.
```

The disagreement table is the high-signal output: when one judge approves a requirement that two others reject, that's exactly the shared-priors situation consensus mode is designed to catch.

### Cost notes

Each judge in the consensus list multiplies the per-audit cost. With Haiku + GPT-4.1-mini, you're roughly doubling cost — still single-digit cents per high-stakes phase. Adding a local Ollama judge as a third costs nothing per call but adds 5-15s of wall-clock latency since judges run in parallel and the slowest sets the audit time.

For a typical DevForgeAI workflow with 5-10 phases where 1-2 are high-stakes, total cost is still under $0.10 per workflow run.

## Switching the default judge to non-Anthropic

Either edit `.devforgeai/audit.yaml`:

```yaml
provider: openai
endpoint: https://api.openai.com
model: gpt-4.1-mini
```

Or use environment variables:

```bash
export DEVFORGEAI_AUDIT_PROVIDER=openai
export DEVFORGEAI_AUDIT_ENDPOINT=https://api.openai.com
export DEVFORGEAI_AUDIT_MODEL=gpt-4.1-mini
export DEVFORGEAI_AUDIT_API_KEY=sk-...
```

The `openai` provider uses OpenAI-compatible Chat Completions, which works with OpenAI, OpenRouter, Mistral, Ollama (`http://localhost:11434`), vLLM, LM Studio, Together, Groq, Fireworks, and most local inference servers.

## Architecture

```
devforgeai/audit/
├── hook.py              entry point — routes to single or consensus path
├── auditor.py           LLM client — dispatches anthropic vs openai-compat
├── consensus.py         multi-judge orchestration with parallel execution
├── citation_verifier.py the unfakeable layer — filesystem-grounded
├── prompts.py           audit prompt template
├── schemas.py           Pydantic models (Requirement, Verdict, AuditResult)
└── config.py            config loader (yaml + .env + env vars)
```

Three trust zones, deliberately separated:

| Module | Trusts the LLM? | Why |
| --- | --- | --- |
| `auditor.py` | Yes | Asks judge for verdicts |
| `consensus.py` | Per-judge only | Independent verification of each judge before applying rule |
| `citation_verifier.py` | No | Re-checks every claim against the actual files |
| `hook.py` | No | Decides exit code from post-verification verdict only |

## Audit log format

Every audit writes a JSON record to `.devforgeai/audit_log/`. Single-judge logs:

```
.devforgeai/audit_log/<phase>_<timestamp>.json
```

Consensus logs:

```
.devforgeai/audit_log/<phase>_<timestamp>_consensus.json
```

Consensus logs include each judge's full verdict, citation failures, and the disagreement table. This is your forensic trail for governance reviews — you can demonstrate that AI-generated artifacts went through evidence-based, multi-judge review rather than vibes-based approval.

## Known limitations / extension points

1. **Shared priors with a single Anthropic judge.** Discussed above. Mitigate with consensus mode for critical phases.

2. **Path traversal in `sources.json` is not deeply validated.** The verifier blocks traversal in citations; `sources.json` is trusted input. If your phase definitions come from untrusted sources, harden `_load_sources` in `hook.py`.

3. **No streaming or incremental audit.** Whole-phase audits at boundaries. Step-level audits would catch skipping earlier but cost more.

4. **No partial credit.** Any non-satisfied verdict fails the phase. Add a `severity` field to Requirements if you want "advance with caveats" workflows.

5. **No weighted consensus.** All judges have equal weight under `majority`. If you want "Haiku alone can't approve, but two cheap judges can override" semantics, the rule layer needs extending.

6. **No automatic retry on transient errors.** A flaky network blip on one judge call fails the audit. For CI environments with unreliable connectivity, wrap `_run_one_judge` in a retry decorator.

## Generating Fixtures Automatically

Authoring a phase fixture (`requirements.json` + `sources.json` + `artifacts/`) by hand takes ~30 minutes per story. The `generate-audit-fixture` CLI command reduces that to ~30 seconds.

```
devforgeai-validate generate-audit-fixture <STORY_ID> [--phase=NN] [--base=<git-ref>] [--force]
```

### What gets auto-generated

| Output | Source |
| --- | --- |
| `requirements.json` | Parsed from the story's `## Acceptance Criteria` (XML blocks preferred, markdown fallback). One `Requirement` per AC. `success_criteria` are populated from `<then>`, `<approach>`, and DoD items keyword-matching the AC. |
| `sources.json` | Story file (always) + ADRs referenced (`ADR-NNN`) + context files mentioned by name (`tech-stack.md`, etc.) + prior story refs (capped) + recent QA recommendation archives for this story (capped). |
| `artifacts/` | If the story declares files in `## Source Files Guidance`, ONLY those (focused fixture). Otherwise, falls back to `git diff --name-only <base>..HEAD`. Files preserve their project-relative paths under `artifacts/`. Files exceeding 200 KB are skipped. |

### Flags

- `--phase=NN` — append `-phaseNN` to the fixture directory name. Useful for per-phase audits (Dev Complete, AC verify, etc.). Omit for a story-level fixture.
- `--base=<git-ref>` — base ref for `git diff` artifact discovery. Default: `git merge-base HEAD main`. If the default fails (e.g., on a detached HEAD or unusual branch topology), the command exits with a clear error asking you to pass `--base` explicitly.
- `--force` — overwrite an existing fixture directory. Without `--force`, the command refuses and exits 1.
- `--project-root=<dir>` — defaults to `.`.

### Required review (the output is NOT authoritative)

Every successful run prints this reminder:

> NOTE: requirements.json was auto-translated from the story's AC blocks. Review it
> and tighten any vague success_criteria — automated translation cannot fully capture
> nuance. The judge will verify whatever is written here, so vague criteria → vague verdicts.

Open the generated `requirements.json` and check each entry. Common edits:

- Replace abstract criteria like `"Code follows clean architecture"` with concrete ones like `"OrderService class exposes 4 public methods, all with type hints"`.
- Add citation hints when the AC's `<then>` clause is vague.
- Trim `success_criteria` lists that exceed ~4 entries (the judge gets diluted attention).

### Example

```bash
devforgeai-validate generate-audit-fixture STORY-672 --phase=03 --base=main --force
```

Output:

```
Audit fixture generated at devforgeai/audit/phases/STORY-672-phase03
  requirements: 5 entries
  sources:      8 files
  artifacts:    4 files (87 KB)
  git base:     main

NOTE: requirements.json was auto-translated from the story's AC blocks...
```

The next `devforgeai-validate phase-complete STORY-672 --phase=03` will trigger the audit hook against this fixture.

## Testing without burning API credits

For smoke testing the wiring without hitting live APIs, point at mock servers. See the previous version of this README for an example HTTP mock; the same pattern works for both Anthropic and OpenAI-compatible endpoints (each judge calls its own configured endpoint, so you can mock one and use a real one for the other).

## License

Match this to DevForgeAI's main license.
