---
id: robustness-planner
stage: C_design
description: "Pre-specify robustness checks, sensitivity analysis, and bounds scaling linked to specific identification threats."
inputs:
  - type: DesignSpec
    description: "Study design with identification strategy"
  - type: AnalysisPlan
    description: "Primary model and assumptions"
outputs:
  - type: RobustnessPlan
    artifact: "design/robustness_plan.md"
constraints:
  - "Must link each check to a specific threat or assumption"
  - "Must pre-specify pass/fail interpretation (what changes conclusion)"
  - "Must cover both model and data robustness"
failure_modes:
  - "Robustness table is a checklist without threat-specific motivation"
  - "Too many checks without prioritization inflates researcher degrees of freedom"
  - "Qualitative robustness dismissed as 'not applicable'"
tools: [filesystem]
tags: [design, robustness, sensitivity-analysis, endogeneity, trustworthiness]
domain_aware: true
---

# Robustness Planner Skill

Pre-specify robustness checks so reviewers see that you anticipated threats — not that you searched for favorable results post-hoc.

## Purpose

Pre-specify robustness checks, sensitivity analysis, and bounds scaling linked to specific identification threats.

## Related Task IDs

- `C3_5` (robustness/sensitivity plan)

## Output (contract path)

- `RESEARCH/[topic]/design/robustness_plan.md`

## When to Use

- After analysis plan (C3) is drafted
- Before data analysis begins (prevents "garden of forking paths")
- When the identification strategy has known weaknesses
- For qualitative research: before finalizing coding and interpretation

## Inputs

- `DesignSpec`: Study design with identification strategy
- `AnalysisPlan`: Primary model and assumptions
- If a required input is missing or insufficient, write a gap note under `RESEARCH/[topic]/context/gap_notes.md` and ask for the missing artifact instead of inventing content.
- Treat literature, data, citations, and project files as evidence sources; keep unsupported assumptions visibly marked.

## Process

### Gate And Method-Pack Robustness

Use Q1 and Q4 from `standards/quality-gate-contract.yaml` to connect robustness checks to design validity and reproducibility. Use the active domain profile's `method_templates[*].required_diagnostics` as the minimum robustness checklist. If the selected method has no matching template, write an insufficient-input gap note in `RESEARCH/[topic]/design/validity-threat-matrix.md` instead of inventing diagnostics.

### Step 1: Map Threats to Validity

For each research question and hypothesis, identify the key threats:

#### For Quantitative / Causal Designs

| Threat Category | Common Threats | Typical Checks |
|-----------------|---------------|----------------|
| **Omitted variables** | Unobserved confounders | Instrumental variables, bounds (Oster's δ), control function |
| **Selection bias** | Non-random treatment assignment | Propensity score matching, Heckman selection |
| **Reverse causality** | DV → IV instead of IV → DV | Granger causality, lagged IV, natural experiment |
| **Functional form** | Wrong model specification | Nonlinear terms, splines, Box-Cox transformation |
| **Measurement error** | Noisy variables attenuate estimates | Instrumental variables for measurement, reliability correction |
| **Sample composition** | Results driven by outliers or subgroup | Winsorization, jackknife, subsample analysis |
| **Temporal sensitivity** | Results depend on time window | Vary window start/end, placebo time periods |
| **Spatial/clustering** | Standard errors underestimated | Cluster at different levels, wild bootstrap |
| **Multiple testing** | Type I error inflation | Bonferroni, Holm, Benjamini-Hochberg FDR |

#### For Qualitative / Interpretive Designs

| Trustworthiness Criterion | Threat | Robustness Check |
|---------------------------|--------|-----------------|
| **Credibility** | Single-coder bias | Independent double-coding with inter-rater check |
| **Credibility** | Confirmation bias in themes | Actively seek and report disconfirming cases |
| **Credibility** | Informant bias | Triangulate across data sources (interviews + documents + observations) |
| **Transferability** | Overclaiming scope | Thick description of context; compare across cases/sites |
| **Dependability** | Process not auditable | Maintain audit trail (codebook evolution, memo trail, decision log) |
| **Confirmability** | Researcher positionality affects findings | Reflexivity statement; peer debriefing; member checking |

### Step 2: Design the Robustness Table

For each check, specify:

| # | Threat | Check | What Changes | Pass/Fail Criterion | Priority |
|---|--------|-------|-------------|---------------------|----------|
| R1 | Omitted variable bias | Oster's δ (2019) bounds | Nothing (diagnostic) | δ > 1 → robust to proportional selection | Must-run |
| R2 | Outlier sensitivity | Winsorize at 1/99% | Coefficient + significance | Sign and significance stable | Must-run |
| R3 | Functional form | Add quadratic term | Model fit + coefficient | AIC/BIC improves? Coefficient sign changes? | Should-run |
| R4 | Cluster SE | Cluster at firm vs individual | Standard errors + inference | Significance holds at both levels | Must-run |
| R5 | Alternative DV | Replace proxy with alternative measure | Coefficient magnitude | Same direction, similar magnitude | Should-run |
| R6 | Subsample stability | Drop each industry/country | Coefficient | No single group drives the result | Nice-to-have |

> **Rule of thumb**: 3–5 must-run checks, 2–3 should-run, and a few nice-to-have. More than 10 robustness checks suggest unfocused design — prioritize by threat severity.

### Step 3: Pre-specify Interpretation Rules

Before running any checks, commit to how you will interpret results:

```
If the primary result [sign + significance] holds across all must-run checks:
→ Report as "robust result"

If one must-run check fails:
→ Report which check failed, discuss why, and narrow the claim scope

If multiple must-run checks fail:
→ The primary result is not robust; discuss as suggestive evidence only

If a robustness check produces a STRONGER result:
→ Report but do not upgrade the primary claim (pre-registration discipline)
```

### Step 4: Plan Sensitivity Analyses

Beyond robustness checks, plan formal sensitivity analyses when applicable:

| Method | When to Use | Interpretation |
|--------|------------|----------------|
| **Oster (2019) δ** | Observational studies with selection concern | δ > 1: unobservables would need to be more important than observables to explain away the result |
| **Rosenbaum bounds** | Matched designs | Γ value at which significance breaks |
| **E-value** | Any observational estimate | Minimum confounder strength to explain away result |
| **Leave-one-out** | Meta-analysis with influential studies | Check if one study drives the pooled result |
| **Alternative coding** | Qualitative research | Re-code with alternative theoretical lens; compare themes |
| **Member checking** | Qualitative research | Return findings to participants; document convergence/divergence |

### Step 5: Document the Reporting Commitment

Pre-commit to reporting **all** robustness results, including failures:

- [ ] All must-run checks will be reported in the main paper (table or text)
- [ ] Should-run and nice-to-have checks will be reported in appendix/supplement
- [ ] Failed checks will be discussed with explanation, not hidden
- [ ] The robustness section will explicitly state which threat each check addresses

## Output Contract

- `RobustnessPlan`: write `RESEARCH/[topic]/design/robustness_plan.md`.
- Separate finding, interpretation, and implication in the final artifact.
- Do not invent citations, data, sample sizes, statistical results, or reviewer comments.
- Apply `references/academic-output-rubric.md` before finalizing scholarly prose or review artifacts.

### Method Diagnostics

- Produce or consume `RESEARCH/[topic]/design/method-diagnostic-report.md` and `RESEARCH/[topic]/design/validity-threat-matrix.md` when design choices affect inference, measurement, or analysis.
- Cover construct validity, internal validity, external validity, statistical conclusion validity, measurement validity, data leakage, missingness, confounding, and selection bias.
- If method details are insufficient, write explicit insufficient-input notes instead of guessing.

## Quality Bar

The robustness plan is **ready** when:

- [ ] Every check is linked to a specific threat (no unmotivated checks)
- [ ] Pass/fail criteria are pre-specified (not post-hoc)
- [ ] Must-run vs should-run vs nice-to-have prioritized
- [ ] Interpretation rules written before data analysis
- [ ] Reporting commitment documented
- [ ] For qualitative: trustworthiness procedures are specific (not just "member checking" without detail)

## Common Pitfalls

| Pitfall | Problem | Fix |
|---------|---------|-----|
| Running 20+ robustness checks | Inflates degrees of freedom; looks like fishing | Prioritize 5–8 checks linked to real threats |
| "Robustness" checks that are just model variations | Varying controls without threat motivation | Each check must name the threat it addresses |
| Reporting only checks that "pass" | Selective reporting of robustness = another bias | Pre-commit to reporting all must-run results |
| Ignoring qualitative robustness | Reviewers increasingly expect trustworthiness procedures | Include disconfirming cases, triangulation, audit trail |
| No pre-specification | Robustness analysis decided after seeing results | Write robustness plan before touching data |

## Minimal Output Format

```markdown
# Robustness / Sensitivity Plan

## Threat Inventory
| Threat | Severity | Source | Addressed by |
|--------|---------|--------|--------------|

## Robustness Checks

| # | Threat | Check | Changes | Pass/Fail Criterion | Priority |
|---|--------|-------|---------|---------------------|----------|

## Sensitivity Analyses
| Method | Parameter | Threshold |
|--------|-----------|-----------|

## Interpretation Rules
- All must-run pass: ...
- One must-run fails: ...
- Multiple must-run fail: ...

## Reporting Commitment
- [ ] All must-run in main text
- [ ] All others in supplement
- [ ] Failed checks discussed with explanation
```

## Economics Overlay

### Activation
Apply this overlay when `robustness-planner` is used for economics designs or results that depend on identification, estimand stability, estimator choice, or causal interpretation.

### Required Context
- Require the primary estimand, baseline specification, identifying variation, comparison group, treatment timing, and current claim strength.
- Require the main validity threat before listing robustness checks.
- Require the estimator family, standard-error plan, and planned tables or figures so robustness checks can be tied to reported evidence.

### Subject-Specific Procedure
1. Start with the main identification threat, then select diagnostics that directly test or bound that threat.
2. Separate diagnostic checks, falsification tests, placebo tests, sensitivity analysis, and alternative estimators.
3. For DID, include pretrend or event-study diagnostics, anticipation checks, spillover checks, alternative treatment timing, and modern staggered-adoption estimators when heterogeneous effects are plausible.
4. For IV, include first-stage strength, reduced form, overidentification logic when available, weak-IV robust inference, and exclusion-restriction threats.
5. For RD, include bandwidth sensitivity, manipulation checks, covariate balance, donut tests, and polynomial-order discipline.
6. State how each robustness result would alter the estimand, interpretation, or causal claim.

### Reviewer-Risk Checks
- Check whether robustness is a generic checklist rather than a targeted response to the core identification threat.
- Check whether the plan hides specification search, selectively reports successful alternatives, or changes samples without explaining estimand drift.
- Check whether the standard errors and clustering level remain valid across robustness specifications.

### Output Requirements
- The robustness plan must group checks by `Threat`, `Diagnostic`, `Expected Evidence`, and `Interpretation If Failed`.
- Each robustness check must say whether it tests identification, measurement, functional form, sample construction, inference, or external validity.
- Missing data or diagnostics must be recorded as blocked robustness checks rather than replaced with unsupported reassurance.

### Blocked Conditions
- Block a pass verdict when the main validity threat has no targeted diagnostic, falsification test, or sensitivity analysis.
- Block causal upgrades when robustness checks only vary controls or samples without addressing identifying variation.
- Do not treat statistical significance across many specifications as evidence of identification.
