Cluster Management

Scale your AI workforce across multiple machines — Mac Minis, VPS, cloud instances, or any combination.

Overview

The Cluster system enables horizontal scaling of your AI agent workforce. Instead of running all agents on a single machine, you can distribute them across multiple worker nodes — each running one or more agents that report back to a central dashboard.

This is useful when:

• You have more agents than a single machine can handle
• You want geographic distribution (agents closer to users)
• You need hardware isolation (GPU nodes for voice, separate machines for browser automation)
• You want fault tolerance (agents survive if one machine goes down)
• You're scaling from a single Mac Mini to a fleet of machines

Architecture

┌─────────────────────────────────────┐
│         Control Plane               │
│   (Enterprise Dashboard Server)     │
│                                     │
│  ┌──────────┐  ┌────────────────┐   │
│  │ Dashboard │  │  Cluster API   │   │
│  │   (UI)    │  │  (REST + SSE)  │   │
│  └──────────┘  └────────────────┘   │
│         │              │            │
│  ┌──────────────────────────┐       │
│  │     Shared Database      │       │
│  │  (Postgres / Supabase)   │       │
│  └──────────────────────────┘       │
└─────────────┬───────────────────────┘
              │ HTTP (heartbeat, status)
     ┌────────┼────────┐
     │        │        │
┌────┴───┐ ┌──┴───┐ ┌──┴───┐
│Worker 1│ │Worker│ │Worker│
│Mac Mini│ │ VPS  │ │ AWS  │
│Agent A │ │AgentB│ │AgentC│
│Agent D │ │AgentE│ │AgentF│
└────────┘ └──────┘ └──────┘
    

Key Concepts

Adding Worker Nodes

There are 3 ways to add a worker node, all from the dashboard UI:

Method 1: Manual Registration

Best for: Machines that already have AgenticMail installed and running.

1. Go to Operations > Cluster in the sidebar
2. Click Add Worker Node
3. Select the Manual Registration tab
4. Enter the node name, host IP/hostname, and port
5. Click Test Connection to verify reachability
6. Click Add Node

The node will appear in the cluster and start receiving heartbeats when the agent process has WORKER_NODE_ID set.

Method 2: SSH Deploy

Best for: Fresh machines where you want the dashboard to handle everything.

1. Click Add Worker Node > Deploy via SSH tab
2. Enter the SSH host, username, and optionally paste a private key
3. Optionally specify which agent IDs to deploy
4. Click Deploy Worker

The dashboard will SSH into the machine, install Node.js, PM2, and AgenticMail, write the environment file, and start the agent processes. The node auto-registers on startup.

Method 3: Setup Script

Best for: Machines you can't SSH into from the dashboard (firewalled, air-gapped, or you prefer manual control).

1. Click Add Worker Node > Setup Script tab
2. Enter a name and port
3. Click Generate Setup Script
4. Copy the script
5. SSH into the target machine yourself and paste/run the script
6. Edit ~/.agenticmail/worker.env to set your DATABASE_URL
7. Start agents with pm2 start "agenticmail-enterprise agent --id <ID>"

Environment Variables

These environment variables control worker node behavior:

Monitoring & Health

Real-Time Status

The Cluster page shows live status for every node via Server-Sent Events (SSE). No polling — updates appear instantly when:

• A node registers or re-registers
• A heartbeat is received (every 30s)
• A node goes offline (no heartbeat for 90s)
• Node agent list changes

Node Statuses

Stats Cards

The top of the Cluster page shows aggregate stats:

• Total Nodes — All registered nodes (online + offline)
• Online — Currently heartbeating nodes
• Running Agents — Total agents across all online nodes
• Total CPUs — Aggregate CPU cores across online nodes
• Total Memory — Aggregate RAM across online nodes

Node Detail & Actions

Click any node card to see full details:

• Platform info — OS, architecture, CPU count, memory
• Agent list — All agents running on this node with names and roles
• Capabilities — What the node supports (browser, voice, etc.)
• Ping — Test connection from dashboard to node (shows latency)
• Restart Agents — Send restart signal to all agents on this node

Load Balancing

When deploying a new agent, the system can automatically select the best node:

• Least-loaded — Node with fewest running agents
• Capability-matching — If the agent needs "voice" or "browser", only nodes with those capabilities are considered
• API endpoint: GET /api/engine/cluster/best-node?capabilities=voice,browser

Database Sharing

All worker nodes must connect to the same database as the control plane. This is how agents share state, memory, tasks, and configuration.

Connection pool settings are auto-optimized per node via the smartDbConfig() helper. Each node maintains its own small connection pool (3 connections max).

Networking Requirements

If nodes are behind NAT or firewalls, only outbound from worker to control plane is strictly required. The test-connection and restart features need inbound access.

Security Considerations

• No authentication on cluster API (yet) — The heartbeat and status endpoints are unauthenticated for simplicity. Only expose the control plane's API behind a reverse proxy with authentication, or restrict by IP.
• SSH keys — For SSH deploy, keys are used once and not persisted. Prefer SSH agent forwarding or temporary keys.
• Database credentials — Each worker node needs DATABASE_URL. Use environment variables, never commit credentials.
• TLS — Use HTTPS between workers and control plane in production. Set ENTERPRISE_URL=https://....
• Network segmentation — Place worker nodes and control plane on the same VPN/VPC for internal traffic.

Edge Cases & Troubleshooting

Node keeps showing "offline"

• Check ENTERPRISE_URL is correct and reachable from the worker
• Check WORKER_NODE_ID is set in the agent's environment
• Check firewall rules — worker must be able to POST to ENTERPRISE_URL/api/engine/cluster/heartbeat/...
• Check agent logs: pm2 logs agent-name

Duplicate node IDs

If two machines use the same WORKER_NODE_ID, they'll overwrite each other's registration. Use unique IDs per machine.

Agent appears on wrong node

Each agent process reports its WORKER_NODE_ID on startup. If you move an agent between machines, restart it on the new machine — it will re-register under the new node.

Control plane restarts

All node data is persisted in the cluster_nodes database table. On restart, nodes load from DB as "offline" and transition to "online" when the next heartbeat arrives (within 30s).

Worker restarts

PM2 auto-restarts crashed agent processes. On restart, the agent re-registers with the control plane within seconds.

Network partition

If a worker loses connectivity to the control plane, it continues running agents normally. It just stops reporting status. When connectivity resumes, the next heartbeat restores the "online" status.

Database failover

All nodes connect to the same database. If the database goes down, all nodes are affected. Use a cloud provider with automatic failover (Supabase, Neon, RDS Multi-AZ).

API Reference

Register Node (POST /api/engine/cluster/register)

{
  "nodeId": "mac-mini-office",    // Required, unique, 2-64 chars, alphanumeric + .-_
  "name": "Office Mac Mini",       // Optional display name
  "host": "192.168.1.50",          // Required, IP or hostname
  "port": 3101,                    // Required, 1-65535
  "platform": "darwin",            // Optional, auto-detected
  "arch": "arm64",                 // Optional, auto-detected
  "cpuCount": 10,                  // Optional, auto-detected
  "memoryMb": 16384,               // Optional, auto-detected
  "version": "0.5.324",            // Optional
  "agents": ["agent-uuid-1"],      // Optional, list of agent IDs
  "capabilities": ["browser", "voice"] // Optional
}
  

Heartbeat (POST /api/engine/cluster/heartbeat/:nodeId)

{
  "agents": ["agent-uuid-1"],  // Current agent list
  "cpuUsage": 0.45,            // Optional, 0-1
  "memoryUsage": 0.62          // Optional, 0-1
}