# The Six Audits

You perform **six distinct audits**, each producing findings that feed into your final referee report.

---

### Audit 1: Code Audit

**Purpose:** Identify coding errors, logic gaps, and implementation problems.

**Checklist:**

- [ ] **Missing value handling**: How are NAs/missing values treated in the cleaning stage? Are they dropped, imputed, or ignored? Is this documented and justified?
- [ ] **Merge diagnostics**: After any merge/join, are there checks for (a) expected row counts, (b) unmatched observations, (c) duplicates created?
- [ ] **Variable construction**: Do constructed variables (dummies, logs, interactions) match their intended definitions?
- [ ] **Loop/apply logic**: Are there off-by-one errors, incorrect indexing, or iteration over wrong dimensions?
- [ ] **Filter conditions**: Do `filter()`, `keep if`, or `[condition]` statements correctly implement the stated sample restrictions?
- [ ] **Package/function behavior**: Are functions being used correctly? (e.g., `lm()` vs `felm()` fixed effects handling)

**Action:** Document each issue with file path, line number (if applicable), and explanation of why it matters.

---

### Audit 2: Cross-Language Replication

**Purpose:** Exploit orthogonality of hallucination errors across languages to catch bugs through independent replication.

**Protocol:**

1. **Identify the primary language** of the analysis (R, Stata, or Python)
2. **Create replication scripts** in the other two languages:
   - If primary is **R** → create Stata and Python replication scripts
   - If primary is **Stata** → create R and Python replication scripts
   - If primary is **Python** → create R and Stata replication scripts
3. **Name replication scripts clearly:**
   ```
   code/replication/
   ├── referee2_replicate_main_results.do      # Stata replication
   ├── referee2_replicate_main_results.R       # R replication
   ├── referee2_replicate_main_results.py      # Python replication
   ├── referee2_replicate_event_study.do
   ├── referee2_replicate_event_study.R
   └── ...
   ```
4. **Run all three implementations** and compare results:
   - Point estimates must match to 6+ decimal places
   - Standard errors must match (accounting for degrees of freedom conventions)
   - Sample sizes must be identical
   - Any constructed variables (residuals, fitted values, etc.) must match

**What discrepancies reveal:**
- **Different point estimates**: Likely a coding error in one implementation
- **Different standard errors**: Check clustering, robust SE specifications, or DoF adjustments
- **Different sample sizes**: Check missing value handling, merge behavior, or filter conditions
- **Different significance levels**: Usually a standard error issue

**Deliverable:**
1. Named replication scripts saved to `code/replication/`
2. A comparison table showing results from all three languages, with discrepancies highlighted and diagnosed

---

### Audit 3: Directory & Replication Package Audit

**Purpose:** Ensure the project is organized for eventual public release as a replication package.

**Checklist:**

- [ ] **Folder structure**: Is there clear separation between `/data/raw`, `/data/clean`, `/code`, `/output`, `/docs`?
- [ ] **Relative paths**: Are ALL file paths relative to the project root? Absolute paths (`C:\Users\...` or `/Users/scott/...`) are automatic failures.
- [ ] **Naming conventions**:
  - Variables: Are names informative? (`treatment_intensity` not `x1`)
  - Datasets: Do names reflect contents? (`county_panel_2000_2020.dta` not `data2.dta`)
  - Scripts: Is execution order clear? (`01_clean.R`, `02_merge.R`, `03_estimate.R`)
- [ ] **Master script**: Is there a single script that runs the entire pipeline from raw data to final output?
- [ ] **README**: Does `/code/README.md` explain how to run the replication?
- [ ] **Dependencies**: Are required packages/libraries documented with versions?
- [ ] **Seeds**: Are random seeds set for any stochastic procedures?

**Scoring:** Assign a replication readiness score (1-10) with specific deficiencies noted.

---

### Audit 4: Output Automation Audit

**Purpose:** Verify that tables and figures are programmatically generated, not manually created.

**Checklist:**

- [ ] **Tables**: Are regression tables generated by code (e.g., `stargazer`, `esttab`, `statsmodels`)? Or are they manually typed into LaTeX/Word?
- [ ] **Figures**: Are figures saved programmatically with code (e.g., `ggsave()`, `graph export`, `plt.savefig()`)? Or are they manually exported?
- [ ] **In-text numbers**: Are key statistics (N, means, coefficients mentioned in text) pulled programmatically or hardcoded?
- [ ] **Reproducibility test**: If you re-run the code, do you get *exactly* the same outputs (byte-identical files)?

**Deductions:**
- Manual table entry: Major concern
- Manual figure export: Minor concern
- Hardcoded in-text statistics: Major concern
- Non-reproducible outputs: Major concern

---

### Audit 5: Empirical Methods Audit

**Purpose:** Verify that the analytical methods — whatever they are — are coherent, correctly implemented, and properly interpreted. This audit adapts to the paper's methodology.

**Step 1: Identify the methodological paradigm.** Before applying any checklist, determine which approach(es) the paper uses:

| Paradigm | Examples |
|----------|----------|
| **Causal inference / Econometrics** | DiD, IV, RDD, RCT, synthetic control, matching |
| **Experiments (lab/online)** | Randomized experiments, A/B tests, within-subjects designs |
| **Computational modelling / Simulation** | Agent-based models, Monte Carlo, optimization, game theory |
| **Machine learning / NLP** | Classification, prediction, LLMs, embeddings, fine-tuning |
| **Survey / Psychometrics** | Likert scales, SEM, factor analysis, conjoint |
| **Qualitative / Mixed methods** | Interviews, case studies, thematic analysis, mixed designs |
| **MCDM / Decision analysis** | AHP, TOPSIS, PROMETHEE, multi-objective optimization |
| **Theoretical / Mathematical** | Proofs, axioms, mechanism design, formal models |

**Step 2: Apply the paradigm-specific checklist(s).** Most papers use one primary paradigm; some combine multiple. Apply ALL relevant checklists.

---

#### 5A. Causal Inference / Econometrics

- [ ] **Identification strategy**: Is the source of variation clearly stated? Is it plausible?
- [ ] **Estimating equation**: Does the code implement what the paper/documentation claims?
- [ ] **Standard errors**: Clustered at the appropriate level? Sufficient clusters (>50)? Heteroskedasticity addressed?
- [ ] **Fixed effects**: Correct? Collinear with treatment?
- [ ] **Controls**: Appropriate? Any "bad controls" (post-treatment variables)?
- [ ] **Sample definition**: Who is in and why? Restrictions justified?
- [ ] **Parallel trends** (if DiD): Pre-trends evidence? Pre-treatment tests?
- [ ] **First stage** (if IV): Shown? F-statistic reported?
- [ ] **Balance** (if RCT/RD): Balance tests shown?
- [ ] **Magnitude plausibility**: Effect size reasonable given priors?

#### 5B. Experiments (Lab / Online)

- [ ] **Randomisation**: Properly implemented? Stratified? Block-randomized?
- [ ] **Power analysis**: Was a pre-registered power analysis conducted? Is the sample large enough?
- [ ] **Pre-registration**: Is the experiment pre-registered? Do analyses match the pre-analysis plan?
- [ ] **Demand effects / Experimenter bias**: Could participants guess the hypothesis? Blinding?
- [ ] **Manipulation checks**: Do they verify the treatment worked as intended?
- [ ] **Attention checks**: Were inattentive participants filtered?
- [ ] **Multiple comparisons**: Are p-values corrected for multiple testing?
- [ ] **Effect sizes**: Reported alongside significance? Cohen's d or equivalent?
- [ ] **Ecological validity**: How well does the lab setting map to the real world?
- [ ] **Attrition**: Differential dropout between conditions?

#### 5C. Computational Modelling / Simulation

- [ ] **Model specification**: Are assumptions clearly stated and justified?
- [ ] **Parameter calibration**: Where do parameter values come from? Empirically grounded or arbitrary?
- [ ] **Sensitivity analysis**: How sensitive are results to parameter choices?
- [ ] **Convergence**: Do simulations converge? How many iterations/runs?
- [ ] **Seed reproducibility**: Are random seeds set and reported?
- [ ] **Validation**: Is the model validated against known data or analytical solutions?
- [ ] **Boundary conditions**: Are edge cases and extreme parameter values tested?
- [ ] **Computational correctness**: Does the code implement the stated model?

#### 5D. Machine Learning / NLP

- [ ] **Train/test split**: Proper held-out test set? No data leakage?
- [ ] **Baselines**: Are appropriate baselines compared?
- [ ] **Metrics**: Are evaluation metrics appropriate for the task and data distribution?
- [ ] **Hyperparameter tuning**: Described? Separate validation set used?
- [ ] **Cross-validation**: K-fold or equivalent?
- [ ] **Statistical significance**: Are differences between models tested? Confidence intervals?
- [ ] **Ablation**: Are component contributions isolated?
- [ ] **Data contamination**: Could training data overlap with test data or evaluation benchmarks?
- [ ] **Prompt sensitivity** (if LLM): Are results robust to prompt variation?
- [ ] **Reproducibility**: Model weights, code, data shared?

#### 5E. Survey / Psychometrics

- [ ] **Construct validity**: Are scales validated? Cronbach's alpha or equivalent reported?
- [ ] **Sampling**: Probability vs convenience sample? Representativeness discussed?
- [ ] **Response rate**: Reported? Non-response bias addressed?
- [ ] **Common method bias**: Single vs multiple sources? Harman's test or marker variable?
- [ ] **Scale anchoring**: Are Likert scales appropriately anchored and labeled?
- [ ] **Factor structure**: Exploratory/confirmatory factor analysis conducted?
- [ ] **Social desirability**: Could responses be biased by self-presentation?
- [ ] **Missing data**: How handled? MCAR/MAR/MNAR assessment?

#### 5F. Qualitative / Mixed Methods

- [ ] **Research design**: Is the qualitative approach (grounded theory, case study, etc.) clearly stated?
- [ ] **Sampling logic**: Theoretical/purposive sampling justified?
- [ ] **Data saturation**: Addressed? How determined?
- [ ] **Coding process**: Inter-rater reliability? Codebook provided?
- [ ] **Reflexivity**: Researcher positionality discussed?
- [ ] **Triangulation**: Multiple data sources or methods?
- [ ] **Integration** (if mixed): How are qualitative and quantitative components integrated?
- [ ] **Transferability**: Is the scope of generalization appropriate?

#### 5G. MCDM / Decision Analysis

- [ ] **Criteria selection**: How were criteria chosen? Justified?
- [ ] **Weighting method**: Appropriate? Sensitivity to weight changes?
- [ ] **Normalization**: Method appropriate for the data type?
- [ ] **Rank reversal**: Tested? Method known to be susceptible?
- [ ] **Consistency**: (if AHP) Consistency ratio reported and acceptable?
- [ ] **Alternatives**: Is the set of alternatives appropriate and complete?
- [ ] **Stakeholder involvement**: Are decision-makers involved in the process?
- [ ] **Comparison**: Are results compared across multiple MCDM methods?

#### 5H. Theoretical / Mathematical

- [ ] **Assumptions**: Clearly stated? Reasonable? Necessary?
- [ ] **Proof correctness**: Are proofs logically valid? Any gaps?
- [ ] **Generality**: How general are the results? What breaks if assumptions are relaxed?
- [ ] **Constructiveness**: Are existence proofs constructive? Can the results be computed?
- [ ] **Relation to existing results**: Are results properly compared to existing theorems?
- [ ] **Examples/counterexamples**: Are the results illustrated?

---

**Deliverable:** List of methodological concerns with severity ratings, organized by the relevant paradigm checklist(s).

---

### Audit 6: Novelty & Literature Assessment

**Purpose:** Independently verify that the paper's claimed contributions are genuinely novel and correctly positioned relative to the existing literature. This audit catches the case where the user's work unknowingly overlaps with or has been pre-empted by existing papers.

**Why this matters for your own work:** It is easy to be blind to competing work, especially recent preprints, concurrent submissions, or papers in adjacent fields. A hostile reviewer WILL find these. Better to find them yourself first.

**Protocol:**

1. **Extract claimed contributions**: Identify every explicit contribution claim in the paper (typically in the introduction and conclusion). Record the exact language and page references.

2. **Launch a Novelty & Literature sub-agent** using the Task tool with `subagent_type: general-purpose`. Provide it with:
   - The paper's exact claimed contributions (with page references)
   - The research question
   - The key methods used
   - The field/domain
   - The papers the author cites as most closely related

   The sub-agent's task is to independently search the literature for:
   - Papers that have already made the **same** contribution (pre-empting)
   - Papers that have made a **very similar** contribution in a different context
   - Concurrent/simultaneous work making the same point
   - Papers the author **should have cited** but didn't
   - Entire literature streams the author may have overlooked

   **Include this Standard Forbid-List in the sub-agent prompt** (per `~/.claude/rules/subagent-prompt-discipline.md`):

   ```
   ## Scope of action — DO NOT do these things

   This sub-agent has a narrow scope: search the literature, classify
   each claimed contribution, and return findings in your final
   response. Do NOT do any of the following:

   - Do NOT modify the paper under review.
   - Do NOT run `git add`, `git commit`, `git push`, or any other git
     write command.
   - Do NOT edit `.context/`, `MEMORY.md`, `CLAUDE.md`, `README.md`,
     or any project documentation.
   - Do NOT edit the project's `.bib` file. List candidate citations
     in your findings; the orchestrator decides what to add.
   - Do NOT create files outside your final response.

   Return findings only. The orchestrator handles consolidation.
   ```

3. **Classify each contribution:**

| Level | Symbol | Meaning |
|-------|--------|---------|
| **Novel** | 🟢 | No prior work found that pre-empts this |
| **Incremental** | 🟡 | Prior work exists in a different context; this extends it |
| **Overlapping** | 🟠 | Substantial overlap with existing work; unclear what is truly new |
| **Pre-empted** | 🔴 | An existing paper has already made this contribution |

4. **Assess literature positioning:**
   - Is the literature review adequate for the target venue?
   - Are the most relevant competitors cited and clearly differentiated?
   - Are there important omissions that a reviewer would flag?

**Red flags:**
- The author avoids citing the most directly relevant prior work
- The "contribution" is a methodological tweak with no new substantive insight
- The literature review cites only tangentially related work, not direct competitors
- The contribution boils down to "we did X but with different data" without theoretical justification for why the new context matters

**Deliverable:**

```markdown
### Novelty Assessment

**Overall verdict:** [Novel / Incremental / Overlapping / Pre-empted]

| Claimed Contribution | Novelty | Key Prior Work | Gap |
|---------------------|---------|---------------|-----|
| [Contribution 1] | 🟢/🟡/🟠/🔴 | [Closest paper] | [What's different] |

### Missing Citations
[Papers that should be cited but aren't]

### Literature Gaps
[Streams of literature the paper overlooks]

### Positioning Recommendation
[How to sharpen the contribution claim]
```

**Important:** If the sub-agent finds pre-empting work (🔴), this is a **major concern** and should be flagged prominently. It is far better to discover this during self-review than to have a referee point it out.
