# Troubleshooting Guide

Common issues and solutions for the 4Runr Gateway.

## Table of Contents

1. [Startup Issues](#startup-issues)
2. [Database Issues](#database-issues)
3. [Authentication Issues](#authentication-issues)
4. [Queue Issues](#queue-issues)
5. [Rate Limiting Issues](#rate-limiting-issues)
6. [Performance Issues](#performance-issues)
7. [Error Messages](#error-messages)

## Startup Issues

### Gateway Won't Start

**Symptoms:**
- Container exits immediately
- "Cannot find module" errors
- Port already in use

**Solutions:**

1. **Check logs:**
```bash
docker-compose logs gateway
# or
pm2 logs gateway
```

2. **Port conflict:**
```bash
# Check what's using port 3000
lsof -i :3000
# or
netstat -tulpn | grep 3000

# Change port in docker-compose.yml or .env
PORT=3001
```

3. **Missing dependencies:**
```bash
# Rebuild
docker-compose build --no-cache gateway

# Or reinstall
pnpm install
```

4. **Prisma client not generated:**
```bash
cd apps/gateway
pnpm db:generate
```

### Health Check Failing

**Symptoms:**
- `/health` returns 500 or times out
- Container marked as unhealthy

**Solutions:**

1. **Check if server is actually running:**
```bash
curl http://localhost:3000/health
```

2. **Check dependencies:**
```bash
curl http://localhost:3000/ready
# Check which dependencies are failing
```

3. **Review startup logs:**
```bash
docker-compose logs gateway | grep -i error
```

## Database Issues

### Cannot Connect to Database

**Symptoms:**
- "Connection refused" errors
- "Authentication failed" errors
- Timeout errors

**Solutions:**

1. **Verify database is running:**
```bash
docker-compose ps postgres
# or
psql -h localhost -U 4runr -d 4runr_gateway
```

2. **Check connection string:**
```bash
echo $DATABASE_URL
# Should be: postgresql://user:password@host:port/database
```

3. **Verify network connectivity:**
```bash
# From gateway container
docker-compose exec gateway ping postgres
```

4. **Check firewall rules:**
```bash
# Ensure port 5432 is accessible
telnet db-host 5432
```

### Migration Errors

**Symptoms:**
- "Migration failed" errors
- Schema mismatch errors

**Solutions:**

1. **Check migration status:**
```bash
cd apps/gateway
pnpm prisma migrate status
```

2. **Reset database (development only):**
```bash
pnpm prisma migrate reset
```

3. **Apply pending migrations:**
```bash
pnpm db:migrate
```

4. **Manual migration:**
```bash
pnpm prisma migrate dev --name migration-name
```

### Prisma Client Not Found

**Symptoms:**
- "Cannot find module '@prisma/client'"
- "Prisma client not generated"

**Solutions:**

1. **Generate Prisma client:**
```bash
cd apps/gateway
pnpm db:generate
```

2. **Verify generation:**
```bash
ls -la node_modules/.prisma/client
```

3. **Rebuild:**
```bash
pnpm clean
pnpm build
```

## Authentication Issues

### API Key Not Working

**Symptoms:**
- 401 Unauthorized errors
- "Invalid API key" messages

**Solutions:**

1. **Verify API key format:**
```bash
# Check header name (should be x-api-key)
curl -H "x-api-key: your-key" http://localhost:3000/api/runs
```

2. **Check if key exists in database:**
```bash
# PostgreSQL mode
psql -h localhost -U 4runr -d 4runr_gateway -c "SELECT * FROM api_keys;"

# Memory mode - check API_KEY env var
echo $API_KEY
```

3. **Verify AUTH_ENABLED:**
```bash
# Should be true
echo $AUTH_ENABLED
```

4. **Check key is not revoked:**
```bash
psql -h localhost -U 4runr -d 4runr_gateway -c "SELECT * FROM api_keys WHERE revoked = false;"
```

### JWT Authentication Failing

**Symptoms:**
- "Invalid token" errors
- "Token expired" errors

**Solutions:**

1. **Verify JWT_SECRET matches:**
```bash
echo $JWT_SECRET
# Must match the secret used to sign the token
```

2. **Check token expiration:**
```bash
# Decode JWT (use jwt.io or similar)
# Verify 'exp' claim is in the future
```

3. **Verify algorithm:**
```bash
# Check JWT_ALG matches token algorithm
echo $JWT_ALG
```

4. **Check token format:**
```bash
# Should be: Authorization: Bearer <token>
curl -H "Authorization: Bearer your-token" http://localhost:3000/api/runs
```

## Queue Issues

### Jobs Not Processing

**Symptoms:**
- Runs stuck in "queued" status
- No job activity

**Solutions:**

1. **Check Redis connection:**
```bash
# Verify Redis is running
docker-compose ps redis
redis-cli ping
```

2. **Check REDIS_URL:**
```bash
echo $REDIS_URL
# Should be: redis://host:port
```

3. **Verify queue initialization:**
```bash
# Check logs for queue initialization
docker-compose logs gateway | grep -i queue
```

4. **Check queue status:**
```bash
# Access Redis and check queue
redis-cli
> KEYS bull:run-execution:*
> LLEN bull:run-execution:waiting
```

### Jobs Failing

**Symptoms:**
- Jobs marked as "failed"
- Error logs in processor

**Solutions:**

1. **Check job error logs:**
```bash
docker-compose logs gateway | grep -i "job.*fail"
```

2. **Verify processor context:**
```bash
# Check if Sentinel/SSE context is set
docker-compose logs gateway | grep -i "processor context"
```

3. **Check run store:**
```bash
# Verify runs can be retrieved
curl http://localhost:3000/api/runs -H "x-api-key: your-key"
```

## Rate Limiting Issues

### Rate Limits Too Strict

**Symptoms:**
- 429 errors too frequently
- Legitimate requests blocked

**Solutions:**

1. **Increase limits:**
```bash
RATE_LIMIT_MAX=100
RATE_LIMIT_WINDOW_MS=60000
```

2. **Check per-route limits:**
- Read operations: Full limit
- Write operations: 75% of limit
- Expensive operations: 50% of limit

3. **Verify Redis is working:**
```bash
# Rate limiting uses Redis
redis-cli ping
```

4. **Check identity tracking:**
```bash
# Rate limits are per-identity (API key, JWT sub, or IP)
# Verify same identity isn't being used for multiple clients
```

### Rate Limits Not Working

**Symptoms:**
- No 429 errors even with high load
- Rate limiting disabled

**Solutions:**

1. **Verify RATE_LIMIT_ENABLED:**
```bash
echo $RATE_LIMIT_ENABLED
# Should be true
```

2. **Check Redis connection:**
```bash
# Rate limiting requires Redis
redis-cli ping
```

3. **Verify middleware is registered:**
```bash
# Check logs for rate limit middleware
docker-compose logs gateway | grep -i "rate limit"
```

## Performance Issues

### Slow Response Times

**Symptoms:**
- High latency
- Timeout errors

**Solutions:**

1. **Check database performance:**
```bash
# Monitor database connections
psql -c "SELECT count(*) FROM pg_stat_activity;"

# Check slow queries
psql -c "SELECT * FROM pg_stat_statements ORDER BY total_time DESC LIMIT 10;"
```

2. **Check queue depth:**
```bash
# High queue depth indicates bottleneck
redis-cli LLEN bull:run-execution:waiting
```

3. **Monitor resource usage:**
```bash
# CPU and memory
docker stats 4runr-gateway
# or
top -p $(pgrep -f "node.*gateway")
```

4. **Check for memory leaks:**
```bash
# Monitor memory over time
watch -n 1 'docker stats --no-stream 4runr-gateway | grep gateway'
```

### High Memory Usage

**Symptoms:**
- Container OOM errors
- Memory usage growing over time

**Solutions:**

1. **Check for memory leaks:**
```bash
# Use Node.js memory profiling
node --inspect dist/index.js
```

2. **Review queue job retention:**
```bash
# Check queue config for job retention
# Reduce removeOnComplete/removeOnFail if too many jobs retained
```

3. **Limit concurrent jobs:**
```bash
# Configure queue concurrency
# In queue config, set concurrency limit
```

4. **Increase container memory:**
```bash
# In docker-compose.yml
deploy:
  resources:
    limits:
      memory: 2G
```

## Error Messages

### "Cannot find module '@prisma/client'"

**Solution:**
```bash
cd apps/gateway
pnpm db:generate
```

### "Prisma client not generated yet"

**Solution:**
```bash
cd apps/gateway
pnpm db:generate
pnpm build
```

### "Queue system not available (Redis required)"

**Solution:**
- Verify Redis is running: `docker-compose ps redis`
- Check REDIS_URL: `echo $REDIS_URL`
- Gateway will fall back to synchronous execution

### "Security validation failed"

**Solution:**
- Check JWT_SECRET length (min 32 chars)
- Verify no insecure default values
- See security validation logs for details

### "Database connection required"

**Solution:**
- Verify DATABASE_URL is set
- Check database is accessible
- In development, can use `GATEWAY_PERSISTENCE=memory`

### "Rate limit exceeded"

**Solution:**
- Wait for rate limit window to reset
- Check retry_after_ms in error response
- Consider increasing RATE_LIMIT_MAX if legitimate

## Getting Help

### Collecting Diagnostic Information

```bash
# Health status
curl http://localhost:3000/health
curl http://localhost:3000/ready

# Metrics
curl http://localhost:3000/metrics

# Logs
docker-compose logs gateway --tail=100

# Environment
docker-compose exec gateway env | grep -E "(NODE_ENV|DATABASE|REDIS|AUTH|RATE)"

# Database status
psql -h localhost -U 4runr -d 4runr_gateway -c "SELECT version();"

# Redis status
redis-cli INFO
```

### Log Levels

Increase log verbosity for debugging:

```bash
LOG_LEVEL=debug
```

**Warning**: Debug logs can be verbose. Use only for troubleshooting.

## Common Patterns

### Service Won't Start After Update

1. Check for breaking changes in changelog
2. Verify environment variables still valid
3. Check database migrations needed
4. Review dependency updates

### Intermittent Errors

1. Check for resource limits (CPU, memory)
2. Verify network stability
3. Check database connection pool
4. Review queue job processing

### Production Issues

1. Check error logs first
2. Verify health endpoints
3. Review metrics for anomalies
4. Check recent deployments
5. Verify dependencies (DB, Redis) are healthy

