# Deployment Guide

Complete guide for deploying the 4Runr Gateway to production.

## Table of Contents

1. [Prerequisites](#prerequisites)
2. [Pre-Deployment Checklist](#pre-deployment-checklist)
3. [Environment Setup](#environment-setup)
4. [Database Setup](#database-setup)
5. [Deployment Methods](#deployment-methods)
6. [Post-Deployment Verification](#post-deployment-verification)
7. [Rollback Procedures](#rollback-procedures)
8. [Scaling](#scaling)

## Prerequisites

### Infrastructure Requirements

- **PostgreSQL 12+**: For persistent storage
- **Redis 6+**: For queue, rate limiting, and idempotency
- **Docker & Docker Compose**: For containerized deployment (recommended)
- **Node.js 18+**: For direct deployment (alternative)

### Security Requirements

- Strong database passwords
- Secure JWT secrets (min 32 characters)
- Encryption keys for sensitive data
- SSL/TLS certificates for HTTPS
- Firewall rules configured

### Network Requirements

- Port 3000: Gateway API (or configure reverse proxy)
- Port 5432: PostgreSQL (internal only)
- Port 6379: Redis (internal only)

## Pre-Deployment Checklist

- [ ] All environment variables configured
- [ ] Database migrations tested
- [ ] Secrets generated and stored securely
- [ ] Health checks configured
- [ ] Monitoring and alerting set up
- [ ] Backup strategy in place
- [ ] Rollback plan documented
- [ ] Load testing completed
- [ ] Security audit performed

## Environment Setup

### 1. Generate Secure Secrets

```bash
# Generate JWT secret
openssl rand -base64 32

# Generate encryption key
openssl rand -base64 32

# Generate API keys (if using memory mode)
openssl rand -hex 32
```

### 2. Create Production Environment File

```bash
cd infra
cp env.example .env.production
```

Edit `.env.production` with production values:

```bash
NODE_ENV=production
GATEWAY_PERSISTENCE=postgres
DATABASE_URL=postgresql://user:strong-password@db-host:5432/4runr_gateway
REDIS_URL=redis://redis-host:6379

AUTH_ENABLED=true
AUTH_MODE=mixed
JWT_SECRET=<generated-secret>
JWT_ALG=HS256

RATE_LIMIT_ENABLED=true
RATE_LIMIT_MAX=100
RATE_LIMIT_WINDOW_MS=60000

SENTINEL_STORE_PLAIN=false
SENTINEL_MODE=live
SENTINEL_SHIELD_MODE=enforce

LOG_LEVEL=warn
```

### 3. Store Secrets Securely

**DO NOT** commit secrets to version control. Use:
- Environment variable management (AWS Secrets Manager, HashiCorp Vault, etc.)
- Docker secrets
- Kubernetes secrets
- CI/CD secret management

## Database Setup

### 1. Create Database

```sql
CREATE DATABASE 4runr_gateway;
CREATE USER 4runr WITH PASSWORD 'strong-password';
GRANT ALL PRIVILEGES ON DATABASE 4runr_gateway TO 4runr;
```

### 2. Run Migrations

```bash
cd apps/gateway
pnpm db:generate
pnpm db:migrate
```

Or via Docker:
```bash
docker-compose -f infra/docker-compose.prod.yml run --rm gateway pnpm db:migrate
```

### 3. Seed Initial Data (Optional)

```bash
# Seed API key
pnpm seed:api-key
```

## Deployment Methods

### Method 1: Docker Compose (Recommended)

**Best for**: Single server deployments, small to medium scale

1. **Build and start:**
```bash
cd infra
docker-compose -f docker-compose.prod.yml up -d --build
```

2. **Verify services:**
```bash
docker-compose -f docker-compose.prod.yml ps
docker-compose -f docker-compose.prod.yml logs -f gateway
```

3. **Check health:**
```bash
curl http://localhost:3000/health
curl http://localhost:3000/ready
```

### Method 2: Kubernetes

**Best for**: Large scale, high availability

1. **Create namespace:**
```bash
kubectl create namespace 4runr-gateway
```

2. **Create secrets:**
```bash
kubectl create secret generic gateway-secrets \
  --from-literal=jwt-secret='your-secret' \
  --from-literal=database-url='postgresql://...' \
  --namespace=4runr-gateway
```

3. **Deploy:**
```bash
kubectl apply -f k8s/gateway-deployment.yaml
kubectl apply -f k8s/gateway-service.yaml
```

### Method 3: Direct Node.js

**Best for**: Development, custom deployments

1. **Build:**
```bash
cd apps/gateway
pnpm install
pnpm build
```

2. **Start:**
```bash
NODE_ENV=production node dist/index.js
```

Or use PM2:
```bash
pm2 start dist/index.js --name gateway
pm2 save
pm2 startup
```

## Post-Deployment Verification

### 1. Health Checks

```bash
# Liveness
curl http://your-server/health
# Expected: {"status":"ok"}

# Readiness
curl http://your-server/ready
# Expected: {"ready":true,"checks":{...}}

# Startup
curl http://your-server/startup
# Expected: {"started":true}
```

### 2. API Functionality

```bash
# Create a run
curl -X POST http://your-server/api/runs \
  -H "x-api-key: your-key" \
  -H "Content-Type: application/json" \
  -d '{"name": "Test Run", "input": {}}'

# List runs
curl http://your-server/api/runs \
  -H "x-api-key: your-key"

# Check metrics
curl http://your-server/metrics
```

### 3. Monitor Logs

```bash
# Docker
docker-compose -f infra/docker-compose.prod.yml logs -f gateway

# Kubernetes
kubectl logs -f deployment/gateway -n 4runr-gateway

# PM2
pm2 logs gateway
```

### 4. Check Metrics

Access Prometheus metrics at `/metrics` and verify:
- HTTP request counts
- Run creation/start/completion rates
- Error rates
- Queue job metrics
- Rate limit hits

## Rollback Procedures

### Quick Rollback (Docker Compose)

```bash
cd infra

# Stop current version
docker-compose -f docker-compose.prod.yml down

# Switch to previous version
git checkout <previous-version-tag>

# Rebuild and start
docker-compose -f docker-compose.prod.yml up -d --build
```

### Database Rollback

If migrations need to be rolled back:

```bash
cd apps/gateway
pnpm prisma migrate resolve --rolled-back <migration-name>
```

**Warning**: Only rollback migrations if you're certain no data depends on them.

### Configuration Rollback

```bash
# Restore previous environment file
cp .env.production.backup .env.production

# Restart services
docker-compose -f docker-compose.prod.yml restart gateway
```

## Scaling

### Horizontal Scaling

The gateway is stateless and can be scaled horizontally:

```bash
# Docker Compose (multiple instances)
docker-compose -f docker-compose.prod.yml up -d --scale gateway=3

# Kubernetes
kubectl scale deployment gateway --replicas=3 -n 4runr-gateway
```

### Load Balancing

Use a load balancer (nginx, HAProxy, AWS ALB) in front of multiple gateway instances:

```nginx
upstream gateway {
    least_conn;
    server gateway1:3000;
    server gateway2:3000;
    server gateway3:3000;
}
```

### Database Scaling

- Use connection pooling (Prisma handles this)
- Consider read replicas for read-heavy workloads
- Monitor connection counts

### Redis Scaling

- Use Redis Cluster for high availability
- Configure Redis Sentinel for failover
- Monitor memory usage

## Monitoring & Alerting

### Key Metrics to Monitor

- **HTTP Request Rate**: Requests per second
- **Error Rate**: 4xx and 5xx errors
- **Response Time**: p50, p95, p99 latencies
- **Queue Depth**: Number of jobs waiting
- **Database Connections**: Active connections
- **Memory Usage**: Container/node memory
- **CPU Usage**: Container/node CPU

### Recommended Alerts

- Error rate > 1%
- Response time p95 > 1s
- Queue depth > 1000
- Database connections > 80% of max
- Memory usage > 80%
- Health check failures

## Backup & Recovery

### Database Backups

```bash
# Daily backup script
pg_dump -h db-host -U 4runr 4runr_gateway > backup-$(date +%Y%m%d).sql

# Restore
psql -h db-host -U 4runr 4runr_gateway < backup-20231119.sql
```

### Configuration Backups

```bash
# Backup environment file
cp .env.production .env.production.backup

# Backup docker-compose
cp docker-compose.prod.yml docker-compose.prod.yml.backup
```

## Security Hardening

### Production Security Checklist

- [ ] HTTPS enabled (TLS 1.2+)
- [ ] Security headers configured
- [ ] Secrets stored securely (not in code)
- [ ] Database access restricted (firewall)
- [ ] Redis access restricted (firewall)
- [ ] Rate limiting enabled
- [ ] Authentication required
- [ ] Error messages masked in production
- [ ] Logging configured (no sensitive data)
- [ ] Regular security updates

### SSL/TLS Setup

Use a reverse proxy (nginx, Traefik) for SSL termination:

```nginx
server {
    listen 443 ssl;
    server_name api.yourdomain.com;
    
    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;
    
    location / {
        proxy_pass http://gateway:3000;
    }
}
```

## Troubleshooting

See [TROUBLESHOOTING.md](./TROUBLESHOOTING.md) for common deployment issues.

## Support

For deployment issues:
- Check logs: `docker-compose logs gateway`
- Verify health: `curl http://localhost/health`
- Review metrics: `curl http://localhost/metrics`
- See troubleshooting guide

