# Phase 5: Testing & Validation

This document outlines the testing strategy and test suites for the 4Runr Gateway.

## Overview

Phase 5 includes:
- **5.1 Integration Tests** (1 day) - Test against real services
- **5.2 Load Testing** (1 day) - Performance and stress testing
- **5.3 End-to-End Testing** (0.5-1 day) - Complete workflow validation

## 5.1 Integration Tests

### Test Suites

All integration tests are located in `apps/gateway/src/__tests__/integration/`:

1. **PostgreSQL Persistence** (`postgres-persistence.test.ts`)
   - CRUD operations
   - Data persistence across restarts
   - Status updates

2. **Authentication** (`authentication.test.ts`)
   - API key authentication
   - JWT authentication
   - Mixed mode authentication
   - Public endpoint access

3. **Idempotency** (`idempotency.test.ts`)
   - Duplicate request handling
   - Key expiration
   - Cross-route idempotency

4. **Sentinel Integration** (`sentinel.test.ts`)
   - Policy enforcement
   - Kill switch
   - Event streaming
   - Audit trails

5. **Rate Limiting** (`rate-limiting.test.ts`)
   - Per-tenant rate limiting
   - Rate limit recovery
   - Bypass for health endpoints
   - Different limits per route

6. **End-to-End Workflows** (`e2e-workflow.test.ts`)
   - Complete run lifecycle
   - Error scenarios
   - Integration with all features

### Running Integration Tests

```bash
# Setup test environment
cd apps/gateway
bash scripts/setup-test-env.sh

# Run all integration tests
pnpm test:integration

# Run specific test suite
pnpm test -- postgres-persistence
pnpm test -- authentication
pnpm test -- idempotency
pnpm test -- sentinel
pnpm test -- rate-limiting
pnpm test -- e2e-workflow
```

### Prerequisites

1. **Server Running**: Gateway must be running (via docker-compose or manually)
2. **PostgreSQL**: For persistence tests
3. **Redis**: For idempotency and queue tests
4. **Environment Variables**:
   - `TEST_SERVER_URL` (default: http://localhost:3000)
   - `TEST_API_KEY` (default: test-key-ce67dabb0d4e27e3d72877c921b89cae)
   - `TEST_DATABASE_URL` (optional, uses DATABASE_URL if not set)
   - `TEST_REDIS_URL` (optional, uses REDIS_URL if not set)

## 5.2 Load Testing

### k6 Load Tests

Load tests are located in `apps/gateway/load-tests/`:

1. **Basic Load Test** (`k6-basic.js`)
   - Ramp up: 10 → 50 users
   - Tests: Health, List, Create, Get
   - Duration: ~3 minutes

2. **Rate Limit Test** (`k6-rate-limit.js`)
   - Rapid ramp to 100 users
   - Tests rate limiting enforcement
   - Duration: ~50 seconds

### Running Load Tests

```bash
# Install k6 (if not installed)
# See load-tests/README.md for installation instructions

# Set environment variables
export GATEWAY_URL="http://localhost:3000"
export API_KEY="test-key-ce67dabb0d4e27e3d72877c921b89cae"

# Run basic load test
cd apps/gateway
pnpm load-test

# Run rate limit test
pnpm load-test:rate-limit

# Or directly with k6
k6 run load-tests/k6-basic.js
k6 run load-tests/k6-rate-limit.js
```

### Performance Targets

- **Response Time**: p95 < 500ms, p99 < 1s
- **Error Rate**: < 1%
- **Throughput**: Varies by endpoint
- **Memory**: No leaks, stable usage

### Memory Leak Testing

Monitor memory during load tests:

```bash
# Run load test and monitor memory
k6 run load-tests/k6-basic.js &
PID=$!
while kill -0 $PID 2>/dev/null; do
  ps -p $PID -o rss=,vsz=,pcpu=
  sleep 5
done
```

## 5.3 End-to-End Testing

End-to-end tests are included in the integration test suite (`e2e-workflow.test.ts`).

### Test Scenarios

1. **Complete Run Workflow**
   - Create → Start → Monitor → Complete
   - Verifies all status transitions
   - Checks for output and completion time

2. **Workflow with Authentication**
   - Verifies auth is required
   - Tests with valid/invalid credentials

3. **Workflow with Sentinel**
   - Verifies Sentinel monitoring
   - Tests policy enforcement

4. **Workflow with Queue**
   - Tests multiple runs in queue
   - Verifies priority handling

5. **Error Scenarios**
   - Invalid run ID
   - Validation errors
   - Cancellation

### Running E2E Tests

```bash
# Run all E2E tests
pnpm test -- e2e-workflow

# Or run all integration tests (includes E2E)
pnpm test:integration
```

## Test Helpers

### `helpers/test-server.ts`

Provides utilities for making HTTP requests to the test server:
- `initTestServer()` - Initialize test configuration
- `authenticatedRequest()` - Make authenticated requests
- `jwtAuthenticatedRequest()` - Make JWT-authenticated requests
- `unauthenticatedRequest()` - Make unauthenticated requests
- `waitForServer()` - Wait for server to be ready

## Success Criteria

### Integration Tests
- ✅ All integration tests pass
- ✅ Edge cases covered
- ✅ Tests are isolated and clean up after themselves

### Load Testing
- ✅ System handles expected load (50+ concurrent users)
- ✅ Response times acceptable (p95 < 500ms)
- ✅ No memory leaks (stable memory usage)
- ✅ Rate limiting effective under load

### End-to-End Testing
- ✅ Complete workflows work end-to-end
- ✅ All features integrated correctly
- ✅ Error scenarios handled gracefully

## Continuous Integration

For CI/CD, ensure:
1. Test services are available (PostgreSQL, Redis)
2. Gateway server is started before tests
3. Tests clean up after themselves
4. Load tests are optional (can be run separately)

## Next Steps

After Phase 5 completion:
- Review test coverage
- Add missing edge cases
- Optimize based on load test results
- Document performance baselines

