# Workers — background-work placement over the root

The worker population is OKDB's **inter-process placement layer**: a set of
generic worker processes forked over one LMDB root that claim and execute
background work — processor drains, queue jobs, function runs, and (FTS)
snapshot bootstraps. There is no intra-process executor (the thread/process
pool was removed in the runtime-model rework, P4): whoever holds a claim
drains it **on its own loop in bounded quanta**, yielding between chunks.

> **See also:** [Processors](processors.md) for the change-subscription **modes**
> (`single`/`fanout`/`inline`), registration, and bootstrap — the _guarantee_ each
> processor declares. This doc covers _where_ that work runs (population & placement);
> processors.md covers _what_ a processor is.

## Quick start

```javascript
const db = new OKDB('./data');
await db.open();

// Declare workers desired state. Root-keyed and idempotent: any instance may call
// it any number of times; one population results.
db.workers.ensure({ scale: 'auto' }); // default: min 0, backlog-driven

// Tear down: scale:0 persists across restarts.
await db.workers.destroy();
```

**Zero-config embedded behavior is unchanged** — no `ensure()` call means the
instance self-processes exactly as before (end-of-tick async on its own loop,
leases held uncontended).

## `workers.ensure({scale})`

| `scale` value      | Meaning                                                                                 |
| ------------------ | --------------------------------------------------------------------------------------- |
| `'auto'` (default) | Supervisor runs; starts at 0 workers; forks on backlog, retires to 0 when idle          |
| `{ min, max }`     | Supervisor runs; always keeps at least `min` workers; elastic ceiling `max`             |
| `0`                | Hard off — no supervisor, no workers (serverless, test runners, PID-limited containers) |

```javascript
// Auto-scaling (recommended default)
db.workers.ensure({ scale: 'auto' });

// Fixed floor with elastic ceiling
db.workers.ensure({ scale: { min: 2, max: 8 } });

// Hard off
db.workers.ensure({ scale: 0 });
```

Workers are **homogeneous and anonymous**: every worker handles every job class
(processing drains, queue definitions, function runs, bootstrap jobs). Any
worker may claim any work unit — the claim IS the dispatcher.

## Deployment shapes

| shape                   | how                                                                                                                                                                     |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| embedded, self-process  | default — nothing to configure                                                                                                                                          |
| embedded + workers      | `db.workers.ensure({scale:'auto'})` — instance stops self-claiming when workers are up; resumes when they retire                                                        |
| embedded + HTTP workers | `db.http.listen(port, { workers: N })` after `open()` — forks okdb's own passive HTTP worker entry (child-entry mode; your app never re-executes; built-in routes only) |
| CLI server              | HTTP cluster (primary + N HTTP workers) is the `bin/okdb` default; add workers alongside                                                                                |

## How placement works

- **`processing: 'auto'`** (the default) resolves against the workers
  desired-state:
    - `scale:0` or no workers → self-process (today's behavior)
    - `scale:'auto'` at 0 live workers → self-process; forks first worker → stop claiming; population retires to 0 → resume self-claiming
    - `scale:{min,max}` → stop claiming immediately (workers are starting up)
      A configured-but-dead population is **re-forked** by any `'auto'` instance
      (recovery restores separation rather than absorbing the work).
- Explicit `processing: 'main'` self-processes regardless of workers;
  `'none'` never processes. The legacy `'threads'`/`'processes'` values warn
  once and map to `'auto'` (the executor they selected was removed in P4).
- **Supervisor**: a per-root `OKDBLock` lease elects one forker. The holder
  forks/retires children to match desired-state, respawns crashes (backoff
  250 ms→4 s, crash-loop guard 5-in-10 s), and recycles wedged workers (stats
  heartbeat silent > 3× interval → SIGKILL + respawn). Supervisor death → its
  children die with it (parent-death watch); any surviving instance acquires the
  lease and re-forks from durable state.
- **Shmbuf hard gate**: `workers.ensure({ scale: non-zero })` throws
  `WORKERS_SHMBUF_REQUIRED` when the shmbuf native binding is missing (the
  fallback shim is process-local; a multi-process write topology would corrupt
  HLC/clock ordering). A persisted desired > 0 read by a fallback-mode instance
  is treated as 0 + a warning.

## Job classes

| class        | mechanism                                                                                                                                                                                                                                                                    |
| ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `processing` | the existing per-(env,type) lease + retry; bus `POKE {env,type}` wakes the holder                                                                                                                                                                                            |
| `queue`      | the verb is the placement: `queue.process(type, handler)` — in-process consumer; `queue.worker(type, module)` — the SHARED POOL (durable definition, declarative: no workers yet → jobs wait until `workers.ensure()`); `queue.spawn(type, module)` — dedicated forked child |
| `functions`  | durable request/response rows (`~fn:requests`/`~fn:responses`) + PROC bus fast path (`OKDB_FN_LEGACY_POOL=1` forces the legacy runner pool)                                                                                                                                  |
| `bootstrap`  | snapshot bootstraps route as one-shot worker jobs (v1: FTS only — durable per-index meta); ineligible/closure bootstraps run owner-local                                                                                                                                     |

## Admission & elasticity

Per worker: `maxLeases` (default 32, `OKDB_WORKERS_MAX_LEASES`), loop-lag gate —
p99 > 200 ms (`OKDB_WORKERS_LAG_GATE_MS`) for 2 windows → stop claiming (resume
below half the gate for 2 windows).

The supervisor probes every `OKDB_WORKERS_SUPERVISE_MS` (default 10 s) and
scales on four backlog signals, **all read from durable state** (never this
instance's in-memory processor lag, which is self-cancelling on a self-claiming
supervisor and inflates on a claim-gated one):

- changelog lag — per-type heads vs the persisted `~proc:state` cursors
  (high `OKDB_WORKERS_LAG_HIGH` default 10 000, low `OKDB_WORKERS_LAG_LOW`
  default 1 000)
- queue depth (`OKDB_WORKERS_QUEUE_HIGH` default 100, `_LOW` default 10)
- function backlog — pending `~fn:requests` rows (`OKDB_WORKERS_FN_HIGH`
  default 10)
- pending routed-bootstrap job rows (any pending row counts as backlog)

Scale-up respects `maxWorkers` (for `'auto'`: cores−1 or
`OKDB_WORKERS_AUTO_MAX`) and holds at +1 until the last-forked worker comes
online; scale-down floor is `minWorkers` (0 for `'auto'` — the population fully
retires when idle). Cooldown: `OKDB_WORKERS_SCALE_COOLDOWN_MS` default 5 000 ms.

## Ops

- Workers are registry citizens: `kind:'worker'` rows under their supervisor,
  with memory/CPU stats **and their processor states** (reported over the
  stats IPC channel) — the registry's `processorsView()` shows the
  claim-holder's row as the truth for each 1-of-N key, so the UI says
  "online on worker pid N", not "paused". Decommission/recommission/kill route
  through the supervisor's reconcile loop (durable slot records).
- Status honesty: a `processing:'auto'` instance that released its claims
  reports `claimGated: true` (rendered as `delegated`, never `paused`) and
  `heldBy: {pid, hostname}` from the live lease record; a worker-routed
  bootstrap surfaces `progress: {phase:'bootstrap', placement:'worker',
state, claimedBy}`.
- Per-drain deadlines (`drainDeadlineMs`, default 60 s) mark a slow processor
  errored (lease retained, in-place retry); a truly wedged worker (silent stats)
  is recycled by the supervisor.
- Runtime mode switching: `processor.setMode(logicalKey, mode)` /
  `POST /api/processors/:logicalKey/mode` — see `docs/processors.md`.
