# HTTP Clustering

By default, the `okdb` CLI runs HTTP serving across **multiple worker processes** so that
request/response latency is never starved by background work (indexing, view rebuilds,
compaction, async catch-up). The processing tier runs in a single **primary** process; HTTP +
SSE are served by **N passive worker processes** that share the listen socket.

This is a **CLI-only default**. The embedded library (`new OKDB(...).http.listen(port)`) stays
single-process unless you opt in — see [Embedded clustering](#embedded-clustering).

---

## Process model

```
                ┌──────────────────────────────────────────────┐
                │ primary  (full role — engines, async          │
                │          processors, compaction:'active', bus)│
                │  • runs ALL processing / indexing / compaction │
                │  • HTTP-SILENT (never serves requests itself)  │
                │  • opens its OKDB BEFORE forking (seeds the     │
                │    shared __tokenSecret so workers agree)       │
                └───────────────┬──────────────────────────────┘
                                │ forks N
        ┌───────────────┬───────┴───────┬───────────────┐
        ▼               ▼               ▼               ▼
   ┌─────────┐    ┌─────────┐     ┌─────────┐     ┌─────────┐
   │ worker  │    │ worker  │ ... │ worker  │     │ worker  │   passive role:
   │ HTTP+SSE│    │ HTTP+SSE│     │ HTTP+SSE│     │ HTTP+SSE│   engines:false
   └─────────┘    └─────────┘     └─────────┘     └─────────┘   asyncProcessors:false
        └───────────────┴── shared listen socket ──┴───────────┘ compaction:'passive'
                                                                  bus:true
```

- **Primary = processor.** Exactly one process runs the processing pool, async processors, and
  compaction. It is HTTP-silent; it never serves requests. (The role flag is the belt; the
  `OKDBLock` proc-lease is the backstop — no worker ever processes.)
- **Workers = passive HTTP servers.** Each forked worker opens a **passive** OKDB
  (`engines:false`, `asyncProcessors:false`, `compaction:'passive'`, `bus:true`) and serves
  HTTP + SSE on the shared socket. Workers do real reads/writes — inline index/schema/FK/TTL
  updates run synchronously inside the writer regardless of role — but never run background
  processing. The primary's async processors pick up the changelog via the UDP bus poke.
- **`workers:1`** (or `--no-cluster`) is byte-identical to the pre-cluster single-process path.

Worker writes update indexes correctly; the primary's processors then index FTS / refresh
views / run the materializer / etc. from the shared changelog.

---

## The `workers` knob

The CLI resolves the worker count from the first source present, highest priority first:

| Priority | Source              | Notes                                          |
| -------- | ------------------- | ---------------------------------------------- |
| 1        | `--no-cluster`      | Forces `workers:1` (single-process escape).    |
| 2        | `--workers N`       | Explicit flag.                                 |
| 3        | `OKDB_HTTP_WORKERS` | Environment variable.                          |
| 4        | `http.workers`      | `.kdbconfig` config file.                      |
| 5        | _default_           | `max(1, cores − 1)` where `cores = os.cpus()`. |

A non-integer or `< 1` explicit value clamps to `1` with a warning. The default reserves one
core for the processing primary and serves HTTP from the rest; on a single-core box it collapses
to one process.

```bash
okdb                       # default: max(1, cores − 1) HTTP workers
okdb --workers 4           # 4 workers
okdb --no-cluster          # single process (back-compat)
OKDB_HTTP_WORKERS=3 okdb   # 3 workers
```

```json
// .kdbconfig
{ "http": { "workers": 4 } }
```

---

## shmbuf requirement (hard gate)

Multi-process clustering is **only safe with the native `shmbuf` binding**. The per-env Hybrid
Logical Clock and changelog clock counter live in a **cross-process shared-memory segment**
(`shmbuf`). Without the native addon, `shmbuf` falls back to an in-process `SharedArrayBuffer`
that cannot coordinate clocks across forked workers — two processes could mint the same HLC /
clock value and corrupt causal ordering.

Therefore, **if `shmbuf` is in fallback mode at startup, the CLI forces `workers:1`** regardless
of any flag, env var, or config, and logs a warning. This gate is non-negotiable.

Check readiness with `okdb doctor`:

```
✓ cluster: ready (7 workers)                                         # native present
✓ cluster: unavailable (shmbuf native missing) — running single-process   # fallback
```

---

## SSE / log coherence boundary

The cluster exposes two distinct event domains. Knowing which is which explains what an SSE
client on a given worker will and won't see:

- **Data changes are cluster-wide.** Each worker runs a **change-feed**: it subscribes to the
  UDP bus `SYSTEM_POKE` signal and, per poke, reads the shared changelog (`getChanges`) to
  reconstruct data-change events (item put/remove, index, view, type). A write committed on
  worker A surfaces in worker B's SSE within the poke window. The bus carries the **signal**;
  the **data** is read from shared LMDB — the changelog is the single source of truth.
- **Processing progress is primary-owned, forwarded over IPC.** Ephemeral processing events
  (view-rebuild progress, FTS/index lifecycle, processor lifecycle) originate only in the
  primary (the sole processor). The primary forwards a curated set to all workers over
  **cluster IPC**, plus a workers / processing status snapshot. Workers serve
  `/api/processing/status` and `/api/cluster/status` from that snapshot.
- **Worker-local logs stay local.** A log line emitted inside worker A is **not** replicated to
  worker B's SSE. Logs are per-process; only data changes (via the changelog) and primary
  progress (via IPC) are coherent across the cluster.

`/api/cluster/status` returns the cluster shape (primary pid, worker pids, uptime).

---

## Lifecycle

- **Worker death → primary respawns** it with bounded backoff (`min(4000, 250·2^failures)` ms)
  and a crash-loop guard (≥ 5 deaths / 10 s per slot stops respawning that slot).
- **Primary death → workers exit** (no orphan listeners or locks), via `cluster` disconnect plus
  a POSIX `ppid===1` backstop.
- **Shutdown** drains in-flight requests, then closes envs, with a 3000 ms hard-exit backstop.

---

## Embedded clustering

The **library default is single-process** and unchanged. `new OKDB(...).http.listen(port)` with
no `workers` option returns a plain single-process `http.Server`. To opt an embedded app into
clustering, pass `workers` (and a passive-worker OKDB factory) to `http.listen`:

```js
okdb.http.listen(port, {
    workers: 4,
    primaryOkdb: okdb, // full-role processor, opened before listen() (open-before-fork)
    makeOkdb: async () => {
        const w = new OKDB(dbPath, {
            engines: false,
            asyncProcessors: false,
            compaction: 'passive',
            bus: true,
        });
        await w.open();
        return w;
    },
});
```

The same shmbuf requirement applies — only cluster an embedded app on a host with the native
`shmbuf` binding.
