# Migrations runbook

Operational reference for the three-layer migration safety system. Read this before running a migration in prod, and again when something has gone wrong.

## The three layers (recap)

| Layer | What | Where defined |
|---|---|---|
| 1 — Process | Expand-and-contract: never combine "add + backfill + drop" in one migration. Two PRs, two deploys. | `CLAUDE.md` |
| 2 — Tooling | `scripts/migrate-with-backup.sh` — pg_dump before every `prisma migrate`, abort if dump fails. | `scripts/migrate-with-backup.sh` |
| 3 — In-migration SQL | Pre-flight count + abort-on-orphan + sidecar backup table before any column drop. | `CLAUDE.md` |

Recovery never depends on a single layer. PITR (if your provider has it) is a fourth, continuous layer underneath.

## Routine: running a migration

### Local (against `.env.testing` or `.env`)

```bash
# Always use these — they wrap pg_dump around prisma migrate.
npm run db:migrate                  # = prisma migrate dev (creates a new migration)
npm run db:migrate:deploy           # = prisma migrate deploy (applies pending migrations)
npm run db:backup                   # dump only, no migration (manual snapshot)
```

Backups land in `.backups/pre-migrate-<mode>-<YYYYMMDD-HHMMSS>-<git-sha>.sql.gz`. The directory is gitignored. Prune old ones manually when convenient.

If your local `pg_dump` is older than the server, the wrapper auto-falls-back to a Docker container with `postgres:16`. Override the image if your server is on a different major:

```bash
PG_DUMP_DOCKER_IMAGE=postgres:15 PG_DUMP_VIA_DOCKER=1 npm run db:backup
```

### Production (Vercel build)

The Vercel `build` script still runs `prisma migrate deploy` directly — the *prod backup* is decoupled from the Vercel build and runs on GitHub Actions instead (see `.github/workflows/backup-on-migration.yml`). Whenever a commit on `master` or `implementation` touches `prisma/migrations/**` or `prisma/schema.prisma`, the workflow runs the same wrapper in `backup-only` mode against `PROD_DATABASE_URL` and uploads the dump as a GHA artifact.

**One-time setup before the workflow can run**:

1. Repo → Settings → Secrets and variables → Actions → "New repository secret"
2. Name: `PROD_DATABASE_URL`. Value: full Postgres URL of prod (with credentials).
3. Confirm the workflow file is on `master`. The first migration commit afterward will trigger it.

## Recovery

### Find the right backup

**Local backups** — listed by timestamp + commit SHA:

```bash
ls -lt .backups/
```

**Prod backups (GHA artifacts)**:

- Web UI: repo → **Actions** tab → "Backup prod DB before migration deploys" workflow → click the run for the relevant commit → scroll to "Artifacts" at the bottom → download the `.zip`.
- CLI:
  ```bash
  gh run list --workflow=backup-on-migration.yml --limit 10
  gh run download <run-id> -n prod-db-backup-<commit-sha>
  ```

GHA artifacts are retained for 90 days by default (configurable in the workflow file).

### Restore commands

The dump format is identical regardless of where it came from (local or GHA), so the restore commands work for both. Decompress with `gunzip -c` (portable; macOS `zcat` does NOT decompress `.gz`).

**1. Inspect a dump without restoring:**

```bash
gunzip -c .backups/pre-migrate-deploy-<ts>-<sha>.sql.gz | head -50
gunzip -c .backups/pre-migrate-deploy-<ts>-<sha>.sql.gz | grep -E '^(CREATE TABLE|INSERT INTO "Project")' | head
```

**2. Restore to a fresh scratch database (for forensics, dry-run, or recovering a specific row):**

```bash
# Create a scratch DB (psql must point at a server you control; do NOT do this against prod)
createdb scratch_restore
# Or via psql: psql -c 'CREATE DATABASE scratch_restore' "$ADMIN_DB_URL"

gunzip -c .backups/pre-migrate-deploy-<ts>-<sha>.sql.gz \
  | psql "postgresql://USER:PASS@HOST:PORT/scratch_restore"
```

Now you have an exact copy of the pre-migration state. Inspect, copy specific rows back to prod, or run the migration against the scratch DB to verify what *would* happen.

**3. Full prod rollback (last resort — see provider section below first):**

```bash
# DESTRUCTIVE — drops the existing schema and replaces it with the dumped contents.
# Only do this if PITR is unavailable AND you've decided a full rollback is the right call.
# The dump uses --no-owner --no-privileges, so Postgres roles are NOT modified.

# 1. Take a current dump first (in case rollback itself goes wrong)
DATABASE_URL="$PROD_DATABASE_URL" npm run db:backup

# 2. Drop and recreate public schema
psql "$PROD_DATABASE_URL" -c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;'

# 3. Restore from the pre-migration dump
gunzip -c .backups/pre-migrate-deploy-<ts>-<sha>.sql.gz | psql "$PROD_DATABASE_URL"

# 4. Re-run prisma migrate so the migrations table reflects current state
DATABASE_URL="$PROD_DATABASE_URL" npx prisma migrate resolve --applied <last-good-migration>
```

### Provider-specific recovery (preferred over manual restore when available)

| Provider | Recovery feature | Notes |
|---|---|---|
| Neon | Branch from PITR | Console → Branches → "Create branch from a previous state" → pick a timestamp before the bad migration. Update `DATABASE_URL` to the new branch. Free tier: 7-day retention. |
| Vercel Postgres | Same as Neon (it's Neon) | Use the Vercel dashboard or Neon console. |
| Supabase | Daily snapshots (paid tiers) | Dashboard → Database → Backups. |
| RDS / Aurora | Automated snapshots + PITR | AWS console. PITR up to backup retention period. |
| Self-hosted | Whatever you set up | Hopefully `pg_basebackup` + WAL archiving. |

**Rule of thumb**: provider PITR > GHA artifact > local `.backups/`. Try them in that order — PITR is closest to "the exact moment before things broke," GHA artifacts are the moment before deploy, local backups are whoever ran the last migration on their laptop.

## Dry-running a migration before merging

Before merging a destructive migration PR, restore a fresh GHA artifact (or a `pg_dump` you took manually against prod) into a scratch DB and run the migration there. Check row counts before/after. If the wrapper aborts on orphan check, the migration is broken — fix the SQL.

```bash
# Pull most recent prod backup
gh run download $(gh run list --workflow=backup-on-migration.yml --limit 1 --json databaseId -q '.[0].databaseId') \
  -n prod-db-backup-<sha> -D /tmp/prod-snapshot

# Spin up a scratch DB and restore
createdb migration_dryrun
gunzip -c /tmp/prod-snapshot/*.sql.gz | psql "postgresql://localhost/migration_dryrun"

# Apply the pending migration
DATABASE_URL="postgresql://localhost/migration_dryrun" \
  bash scripts/migrate-with-backup.sh deploy

# Spot-check: did the rows you expected get backfilled?
psql "postgresql://localhost/migration_dryrun" -c 'SELECT count(*) FROM "Project" WHERE ...;'
```

If the dry-run looks right, merge. If it doesn't, fix the migration and try again.

## Security

**Treat dump files like prod credentials.** They contain:

- Hashed passwords (still attackable offline)
- OAuth tokens stored in `Integration.metadata` — these are live credentials for Vercel, GitHub, Neon, etc. Anyone with the dump can impersonate the integration.
- User PII (emails, names, anything stored in user-facing tables)

**Therefore:**

- `.backups/` is gitignored and must stay that way. Never commit a `.sql.gz` from this directory.
- GHA artifacts inherit the repo's access — only collaborators on the repo can download them. If you make the repo public or add an external collaborator, they get historical artifacts too. Audit before changing access.
- When sharing a dump for debugging, scrub the sensitive tables first (`Integration`, `Account`, `Session`, anything with tokens). A quick scrub:
  ```bash
  gunzip -c backup.sql.gz | grep -v '^COPY public."Integration"' | gzip > backup-scrubbed.sql.gz
  ```
  (This is a blunt instrument — it drops the entire `Integration` COPY block. Inspect before sharing.)
- Rotate any tokens that may have been exposed if a dump leaks. Vercel/GitHub/Neon integration tokens can all be revoked from their respective dashboards.

## Adding support for a new DB engine

Today the wrapper supports Postgres only. To add MySQL/MongoDB/etc.:

1. Add a case branch in `scripts/migrate-with-backup.sh` matching the URL scheme (`mysql://`, `mongodb://`, etc.).
2. Use the appropriate dump tool (`mysqldump`, `mongodump`).
3. Update the sanity check to recognize the new dump format.
4. Update this runbook with restore commands for the new engine.

The orchestration logic (dump → validate → migrate, abort on dump failure) stays identical.
