# Load Orchestrator

The Load Orchestrator is a powerful tool for running distributed load tests using WebdriverIO. It coordinates multiple test instances across different machines using MQTT for communication.

## Architecture

The load orchestrator consists of several key components:

1. **MQTT Broker** (load-orchestrator.js)
   - Runs on port 1883
   - Handles client registration and coordination
   - Manages test session states
   - Handles file transfers and report generation

2. **Load Service** (load-service.js)
   - WebdriverIO service that connects test runners to the orchestrator
   - Manages worker states and synchronization
   - Handles test execution flow

3. **Store Server**
   - Simple key-value store for sharing data between workers
   - Used for worker configuration and state management

## Configuration

### WebdriverIO Config Options

The load test configuration is specified in your WebdriverIO config file (e.g., wdio.conf.load*.js). Here are the key configuration options:

```javascript
export const config = {
  // Load Service Configuration
  services: [
    [LoadService, {
      loadTestData: loadTestData,           // Test data configuration
      instances: 15,                        // Number of parallel instances to run
      loadOrchestrator: 'mqtt://127.0.0.1:1883',  // MQTT broker URL
      resetSession: true,                   // Whether to reset session state
      storePort: 3333                      // Port for the store server
    }]
  ],

  // Test Users Configuration
  testUsers: ["user1", "user2", ...],      // Array of test users
  testUserData: {                          // User details
    "user1": {
      "firstname": "First",
      "lastname": "Last"
    },
    ...
  },

  // Reporter Configuration
  reporters: [
    ['allure', {
      outputDir: 'allure-results-loadXXXXXX',
      disableReportGeneration: true,
      cleanBefore: true,
      ...
    }],
    [EuiReporter, {
      loggerOptions: {
        reportDir: 'jsonReport-loadXXXXXX'
      },
      publishers: [
        {
          type: MqttFilePublisher,
          config: {
            loadOrchestrator: 'mqtt://127.0.0.1:1883',
            sessionUuid: loadTestData.sessionConfig.sessionUuid,
            publish: 'always',
            dirPath: 'allure-results-loadXXXXXX'
          }
        }
      ]
    }]
  ]
}
```

### Test Data Configuration

The test data configuration (testdata.json) defines the load test parameters:

```javascript
{
  "sessionConfig": {
    "sessionUuid": "unique-session-id",    // Unique identifier for the test session
    "resetSession": true,                  // Whether to reset session state
    "users": 30                            // Total number of users for the test
  },
  "testConfig": {
    "rampUpDelay": 1000,                  // Delay between starting each user (ms)
    "waitTime": 500                        // Wait time between test iterations
  },
  "tasks": [                              // Test data for each user
    {
      "title": "Task 1",
      "url": "task1-url",
      ...
    }
  ]
}
```

## Test Synchronization

To synchronize test execution across workers, use the following pattern in your test files:

```javascript
describe('Load Test', () => {
  it('should wait for login ramp up', async () => {
    let waitTime = testData.testConfig.rampUpDelay * dataIndex;
    await browser.deductedWait(waitTime);
  });

  it('should sync after login', async () => {
    // Set sync point for orchestration
    global.workerConfig.syncpoint = 'syncAfterLogin';
  });

  // More test steps...
});
```

Key sync points should be added after critical operations like:
- Login completion
- Navigation to key pages
- Before/after high-load operations
- Before test completion

## Error Handling

The load orchestrator includes built-in error handling:

1. **Cancelling a Test Session**
   ```bash
   # Linux/Mac
   ./cancel-load-test-session.sh <sessionUuid>
   
   # Windows
   cancel-load-test-session.cmd <sessionUuid>
   ```

2. **Error States**
   - When an error occurs, the session is marked as in error state
   - All workers are notified to stop testing
   - Report generation is skipped
   - File cleanup still occurs

## Best Practices

1. **Resource Management**
   - Set appropriate instance counts based on available resources
   - Use ramp-up delays to prevent login storms
   - Monitor system resources during test execution

2. **Test Design**
   - Add appropriate sync points to coordinate test execution
   - Include proper error handling and cleanup
   - Use debug logging for troubleshooting

3. **Configuration**
   - Use unique session IDs for each test run
   - Configure appropriate timeouts and thresholds
   - Set up proper reporting paths and cleanup

## Debugging

The load orchestrator provides several debugging features:

1. **Debug Logging**
   ```javascript
   // In your test
   console.log('Received message on topic:', topic, 'Message:', messageStr);
   ```

2. **Network Logging**
   ```javascript
   // After key operations
   browser.logPuppeteerNetwork();
   ```

3. **State Monitoring**
   - Monitor worker states through MQTT messages
   - Check sync points and coordination
   - Review session state and error conditions 

## File Publishing and Report Generation

### MqttFilePublisher Configuration

The MqttFilePublisher is responsible for sending test results and reports to the load orchestrator. Configure it in your WebdriverIO config:

```javascript
reporters: [
  ['allure', {
    outputDir: 'allure-results-load202501',
    disableReportGeneration: true,
    cleanBefore: true,
    disableWebdriverStepsReporting: true,
    disableWebdriverScreenshotsReporting: false,
    addConsoleLogs: true,
    openReport: false
  }],
  [EuiReporter, {
    loggerOptions: {
      reportDir: 'jsonReport-load202501-linux4dev-local'
    },
    publishers: [
      {
        type: MqttFilePublisher,
        config: {
          loadOrchestrator: 'mqtt://127.0.0.1:1883',  // MQTT broker URL
          sessionUuid: loadTestData.sessionConfig.sessionUuid,
          publish: 'always',                          // When to publish results
          dirPath: 'allure-results-load202501'        // Directory containing results
        }
      }
    ]
  }]
]
```

### File Handling Process

The load orchestrator handles test result files in the following way:

1. **File Reception**
   - Files are received through MQTT messages
   - Each file is stored in a session-specific directory
   - The orchestrator tracks expected and received file counts
   - Acknowledgments are sent back to clients for each file

2. **File Types**
   - Binary files (e.g., .zip, .png, .jpg) are stored directly
   - Text files (.txt, .json) are automatically zipped for efficiency
   - Each file type is handled appropriately during processing

3. **Report Generation**
   ```javascript
   {
     // S3 Configuration for report storage
     objectstore: {
       prefixRootKeyWithTime: true,        // Add timestamp to S3 key
       s3rootKey: '/your/path',            // Base path in S3
       dirPath: 'allure-results-load202501' // Local report directory
     },
     // Vault configuration for S3 credentials
     vaultConfig: {
       endpoint: 'https://vault.example.com',
       namespace: '/your/namespace',
       testusersPath: '/path/to/users'
     }
   }
   ```

4. **Processing Steps**
   - Files are collected from all test instances
   - Text files are consolidated into a single zip
   - Allure report is generated from all results
   - Original files are cleaned up
   - If configured, report is published to S3:
     - Credentials are fetched from vault
     - Report is uploaded with timestamp
     - Local files are cleaned up

5. **Error Handling**
   - Failed transfers are retried up to 3 times
   - File acknowledgments ensure delivery
   - If session is in error state:
     - Report generation is skipped
     - Files are cleaned up immediately
     - Error state is logged
