# Processors (custom change-driven workers)

A **processor** maintains derived state from a source type's change log. You register one per type you want to react to; the framework owns the hard parts:

- a **lease** so exactly one process runs each processor (others wait and take over on failure),
- a durable **cursor** so it resumes where it left off after a restart,
- **batching**, post-commit **scheduling** (+ cross-process wake-up over the bus),
- **pause/resume**, and **error recovery**.

OKDB's own features (indexes, views, FTS, materializer, embeddings, time-machine) are built on this primitive — and it's a public extension point you can use directly.

---

## Modes — where and when your handler runs

| mode     | runs                                         | use when                                                                                             |
| -------- | -------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| `inline` | synchronously inside the writer's commit     | derived state must be consistent the instant the write returns, and the work is cheap                |
| `async`  | post-commit, batched, on the main event loop | default for light/moderate derived work                                                              |
| `worker` | post-commit, batched, on a **worker thread** | per-event work is CPU-heavy (parse / tokenize / encode / compress) and you want it off the main loop |

---

## Registering a processor

```javascript
const stop = env.processor.register('Order', {
    mode: 'async',
    name: 'order-stats', // shown in admin / status
    cursorKey: 'order-stats', // durable resume key — unique per processor
    originMode: 'all', // 'self' | 'remote' | 'all'
    handler: async (changes, info) => {
        for (const c of changes) {
            // c.type, c.key, c.action ('put' | 'remove'), c.clock, c.value?
        }
    },
});

// Control surface:
await stop(); // unregister + release the lease
stop.status(); // live status { state, lastClock, lag, mode, ... }
stop.pause();
stop.resume();
```

`changes` are change-log records (`{ type, key, action, clock, ... }`). In `async` mode with the default `hydrateValues: true`, `put` records also carry `.value` (the current document). In `worker` mode you hydrate yourself — see below.

`originMode` filters by who wrote the change: `self` (this instance's writes only), `remote` (other instances/processes), or `all`.

---

## `worker` mode and `workerKey` (sharing)

`worker` mode runs your handler on a worker thread. It requires a **resolvable handler module** — a file path and export name, **not a closure** (closures can't cross the thread boundary; the worker loads the module in its own isolate):

```javascript
env.processor.register('Doc', {
    mode: 'worker',
    workerKey: `myfeature:${env.name}`, // optional — see below
    cursorKey: 'myfeature:Doc',
    originMode: 'all',
    handler: { module: require.resolve('./my-handler.js'), export: 'handle' },
});
```

**`workerKey` controls sharing:**

- **Omit it** → a **dedicated** worker for this processor (the key defaults to the processor's own id). Best for isolation, or for genuinely parallel CPU work across separate threads.
- **Set the same key** on several processors → they **share one** worker thread (and one OKDB instance), processing drains serially. This is what FTS does: every FTS-indexed type registers a `worker`-mode processor with `workerKey: 'fts:<env>'`, so the whole env's FTS indexing runs on a single shared worker. All processors sharing a key **must declare the same handler module/export** (validated at registration).

The worker is **lazily spawned** on the first real work, **torn down after `workerIdleMs`** of idle (default 30 s), and **respawns automatically** (the cursor is durable, so no state is lost). If the worker dies or a single drain exceeds `drainTimeoutMs` (default 60 s), the in-flight work is retried — the cursor only advances on success, so processing is **at-least-once**; make handlers idempotent.

---

## Writing a worker handler module

The module loads in a fresh worker isolate, so it must be **self-contained** and **replay any state it needs from durable sources** (disk / the env) — in-process registrations made on the main thread are not visible to it.

```javascript
// my-handler.js
module.exports = {
    // optional — called once at worker startup (open derived sub-DBs, read config, …)
    async init({ env }) {
        this.store = env.openDB?.('my-derived');
    },

    // called per drain; `changes` are RAW (not hydrated) — read values via info.env
    async handle(changes, info) {
        const env = info.env; // the worker's own OKDBEnv — real reads/writes
        for (const c of changes) {
            if (c.action === 'put') {
                const doc = c.value ?? env.get(c.type, c.key); // hydrate on demand
                /* ...update your derived state via env / your sub-DBs... */
            } else if (c.action === 'remove') {
                /* ...handle deletion... */
            }
        }
    },
};

// A plain function export also works:
//   module.exports = async function handle(changes, info) { ... }
// register with export: 'handle' (or the function's name); it must resolve to a function.
```

Rules, stated plainly:

- **Resolvable module + export**, never a closure.
- The worker has its **own OKDB** open on the same path (background features off). It shares the underlying LMDB storage with the main process, so your writes are visible to it — and to readers/queries — immediately on commit.
- **Raw changes**: in worker mode the framework does not pre-hydrate; read `info.env.get(c.type, c.key)` when you need the document.
- **You don't manage the cursor** — the main thread owns it and advances it only when your `handle` resolves. Throwing makes the framework retry the range (at-least-once) → write **idempotent** handlers.
- **Be worker-portable**: read schemas / config / definitions from disk in `init`, not from closures.

---

## Cross-process behaviour

Each processor takes a lease (the `'proc'` lock). With several processes open on the same env, exactly one runs each processor; the others wait and take over if the holder dies. Worker sharing is per-process — the lease-holding process's worker does the work, so there is still a single writer to your derived store.

---

## Cookbook

- **A dedicated CPU-heavy indexer** — `mode: 'worker'`, no `workerKey`.
- **Several indexers sharing one thread/OKDB (like FTS)** — `mode: 'worker'`, same `workerKey`.
- **An immediate-consistency derived field** — `mode: 'inline'` (cheap work only; it runs on the write path).
- **A light aggregate on the main loop** — `mode: 'async'`.

> Note: a `worker` processor adds first-index latency after idle (the lazy OKDB spawn ≈ a few hundred ms). Derived state is eventually-consistent; if you need a deterministic "caught up" point, wait on `stop.status().lastClock` reaching the source clock.
