# Queue

OKDB has a built-in durable job queue.

Queue data lives in the **current environment**:

- `okdb.queue` uses the **default** environment
- `okdb.env('myenv').queue` uses that **custom** environment

Jobs survive restart, are stored as normal OKDB records, and can be listed, edited, retried, and processed by workers.

No Redis, no RabbitMQ. Just OKDB.

---

## Quick start

```javascript
const OKDB = require('./okdb');

async function main() {
    const okdb = new OKDB('./db');
    await okdb.open();

    // process() runs the consumer IN THIS process (it takes a closure). For a consumer that
    // runs on the worker population or a dedicated child, use worker(type, module) /
    // spawn(type, module) — those cross a process boundary, so they take a module path.
    const worker = okdb.queue.process(
        'send-email',
        async (payload, ctx) => {
            await ctx.markProgress('sending email');

            await sendEmail(payload.to, payload.subject, payload.orderId);

            await ctx.okdb.put('email_log', payload.orderId, {
                to: payload.to,
                sentAt: Date.now(),
                jobId: ctx.jobId,
            });
        },
        {
            concurrency: 4,
            ttl: 30_000,
        },
    );

    const jobId = await okdb.queue.enqueue(
        'send-email',
        {
            to: 'alice@example.com',
            subject: 'Your order shipped',
            orderId: 'o42',
        },
        {
            max_tries: 3,
            retry_delay: 1000,
            backoff_multiplier: 2,
        },
    );

    console.log('enqueued', jobId);

    // later
    await worker.stop();
    await okdb.close();
}
```

---

## Environments

Queue data is **not** stored in a special dedicated `queue/` environment.

Instead:

- `okdb.queue.enqueue(...)` stores queue records in `default`
- `customEnv.queue.enqueue(...)` stores queue records in `customEnv`

Example:

```javascript
const okdb = new OKDB('./db');
await okdb.open();

const jobsEnv = await okdb.createEnvironment('jobs');

await okdb.queue.enqueue('default-task', { a: 1 });
await jobsEnv.queue.enqueue('custom-task', { b: 2 });
```

This is useful when you want separate queues per environment.

---

## Enqueuing jobs

```javascript
const id = await okdb.queue.enqueue('process-image', {
    fileId: 'f1',
    userId: 'u1',
});

console.log(id); // UUID string
```

### Signature

```javascript
await okdb.queue.enqueue(type, payload, options);
```

- `type` — non-empty job type string
- `payload` — any serialisable value; stored separately from job metadata
- `options` — scheduling / retry / routing metadata

### Enqueue options

```javascript
await okdb.queue.enqueue(
    'process-image',
    { fileId: 'f1' },
    {
        when: Date.now() + 5000, // schedule in the future
        priority: 10,
        tags: ['images', 'urgent'],
        bucket: 'gpu-workers',
        bucket_tokens: 1,
        max_tries: 3, // null = unlimited
        retry_delay: 2000,
        backoff_multiplier: 2.0,
        cron: '0 * * * *',
    },
);
```

Notes:

- `priority` is an integer
- `tags` can be a string or array of strings
- `bucket` and `bucket_tokens` integrate with token buckets
- `max_tries: 1` means “try once, then fail permanently”
- `max_tries: null` means unlimited retries

---

## Processing jobs with `process(...)`

Use a normal in-process consumer like this:

```javascript
const worker = okdb.queue.process(
    'send-email',
    async (payload, ctx) => {
        await ctx.markProgress('validating');

        if (!payload.to) throw new Error('missing recipient');

        await ctx.markProgress('sending');
        await sendEmail(payload.to, payload.subject);

        await ctx.okdb.put('email_log', payload.orderId, {
            to: payload.to,
            sentAt: Date.now(),
            jobId: ctx.jobId,
        });
    },
    {
        concurrency: 2,
        ttl: 30_000,
        onPermanentFail: async (job, err) => {
            console.error('permanently failed', job.id, err.message);
        },
    },
);
```

### Handler signature

The handler receives:

```javascript
async (payload, ctx) => { ... }
```

Where `ctx` contains:

- `ctx.okdb` — the root `OKDB` instance
- `ctx.jobId` — queue job id
- `ctx.heartbeat()` — extend the claim TTL for long-running jobs
- `ctx.markProgress(message)` — update the job's `progress` field

These local worker helpers now map to the same public queue API that remote workers can use:

- `okdb.queue.claim(type, options)`
- `okdb.queue.markJobHeartbeat(jobId, claimId, ttl?)`
- `okdb.queue.markJobProgress(jobId, claimId, message)`
- `okdb.queue.markJobComplete(jobId, claimId, result?)`
- `okdb.queue.markJobFail(jobId, claimId, error, code?)`

### Placement: `process` vs `worker` vs `spawn`

The three verbs split on **where the consumer runs** — and therefore on whether they take a closure or a module:

| Verb                         | Runs in                                     | 2nd arg                                | Lifetime                   |
| ---------------------------- | ------------------------------------------- | -------------------------------------- | -------------------------- |
| `queue.process(type, fn)`    | **this** Node process                       | a **closure**                          | dies with this instance    |
| `queue.worker(type, module)` | the **worker population** (other processes) | a **module path** / `{module, export}` | durable; the fleet runs it |
| `queue.spawn(type, module)`  | a **dedicated forked child**                | a **module path**                      | tied to the child          |

A closure can't cross a process boundary, so `worker()`/`spawn()` require a module. Passing a closure to `worker()` is an error that points you at `process()`.

`process(...)` is great for I/O-heavy jobs but gives no isolation from CPU-heavy work — use `spawn()` (dedicated child) or `worker()` (population) for that.

---

## Worker options

### No polling — event-driven wakeup

A consumer does **not** poll. An idle lane sleeps on a wake handle and resumes immediately when work appears: a local `enqueue` (same process), a job finishing (frees a `max_concurrency` slot), or a cross-process queue write (the UDP bus POKE). The only timer is a long **backstop** (default 30 s, `OKDB_QUEUE_IDLE_MS`) that exists solely to cover a dropped bus POKE and to wake for time-scheduled jobs — the lane caps its sleep at the soonest future `when`, so delayed jobs still fire on time.

### `pollInterval`

Optional. When set, it **caps the idle backstop** for this consumer (an upper bound on wakeup latency if a bus POKE is ever dropped). It is no longer a poll cadence — leave it unset to get the default 30 s backstop with instant event-driven pickup.

```javascript
{
    pollInterval: 100;
}
```

### `concurrency`

How many jobs to process in parallel **inside the same process**.

```javascript
{
    concurrency: 4;
}
```

This is implemented as multiple independent claim/handle lanes.

### `ttl`

Claim timeout in ms.

```javascript
{
    ttl: 30_000;
}
```

If your worker dies or stops heartbeating, expired running jobs are re-queued by reconciliation.

### `onPermanentFail`

Called after a job permanently fails.

---

## Remote / external workers

If your workers run outside the embedded Node.js process, use the public queue lifecycle methods directly or call the equivalent HTTP endpoints.

### Public queue methods

```javascript
const claimed = await okdb.queue.claim('send-email', { ttl: 30_000 });
if (!claimed) return;

await okdb.queue.markJobProgress(claimed.id, claimed.claim_id, 'sending');

try {
    await sendEmail(claimed.payload.to, claimed.payload.subject);
    await okdb.queue.markJobComplete(claimed.id, claimed.claim_id, { delivered: true });
} catch (err) {
    await okdb.queue.markJobFail(claimed.id, claimed.claim_id, err, 'send_failed');
}
```

### Claim ownership contract

Remote lifecycle operations require the active `claim_id` returned by `claim(...)`.

- only the current claim holder may heartbeat, write progress, complete, or fail the job
- stale workers are rejected once the claim changes or expires
- this prevents duplicate workers from mutating the same running job

### HTTP endpoints for remote workers

The queue HTTP API now exposes the full remote-worker lifecycle:

- `POST /api/queue/claim/:type`
- `POST /api/queue/job/:id/heartbeat`
- `POST /api/queue/job/:id/progress`
- `POST /api/queue/job/:id/complete`
- `POST /api/queue/job/:id/fail`

Example:

```text
// claim
POST /api/queue/claim/send-email
{ "options": { "ttl": 30000 } }

// progress
POST /api/queue/job/<id>/progress
{ "claim_id": "<claim>", "progress": "sending" }

// complete
POST /api/queue/job/<id>/complete
{ "claim_id": "<claim>", "result": { "delivered": true } }

// fail
POST /api/queue/job/<id>/fail
{ "claim_id": "<claim>", "code": "send_failed", "message": "smtp timeout" }
```

---

## Spawned workers

If you want queue handling in a **separate Node process**, use `spawn(...)`:

```javascript
const spawned = await okdb.queue.spawn('send-email', './workers/email.js', {
    concurrency: 4,
    pollInterval: 100,
});

// later
await spawned.stop();
```

The module must export a function:

```javascript
module.exports = async function (payload, ctx) {
    await ctx.okdb.put('email_log', payload.orderId, {
        sentAt: Date.now(),
        jobId: ctx.jobId,
    });
};
```

The child process opens the same OKDB path and receives the same `ctx.okdb` semantics as a normal worker.

---

## Job lifecycle

```text
pending → running → done
                └→ pending (retry)
                └→ failed
```

In more detail:

- `pending` — ready to be claimed
- `running` — claimed by a worker and currently in-flight
- `done` — completed successfully
- `failed` — permanently failed after all tries are exhausted

When a running job's claim expires, reconciliation moves it back to `pending`.

---

## Heartbeats and reconciliation

For long-running jobs, call `ctx.heartbeat()` periodically:

```javascript
okdb.queue.process(
    'import',
    async (payload, ctx) => {
        for (let i = 0; i < 10; i++) {
            await processChunk(i);
            await ctx.heartbeat();
            await ctx.markProgress(`chunk ${i + 1}/10`);
        }
    },
    { ttl: 5000 },
);
```

You can also manually trigger reconciliation:

```javascript
await okdb.queue._reconcile();
```

The admin UI also exposes reconcile actions.

---

## Listing jobs

```javascript
const { items, cursor } = await okdb.queue.list({
    type: 'send-email',
    status: 'pending',
    sort: 'when', // 'when' | 'priority'
    direction: 'asc',
    limit: 50,
    cursor: null,
});
```

Supported filters/options:

- `type`
- `status`
- `bucket`
- `tag`
- `sort`
- `direction`
- `limit`
- `cursor`

---

## Getting and updating one job

### Get a job

```javascript
const job = await okdb.queue.getJob(jobId);

console.log(job.status);
console.log(job.payload);
console.log(job.progress);
```

`getJob()` returns job metadata **plus payload**.

### Update a job

```javascript
await okdb.queue.updateJob(jobId, {
    payload: { fileId: 'f1', force: true },
    priority: 9,
    tags: ['edited', 'manual'],
    progress: 'manually adjusted',
    status_message: 'edited from admin ui',
    retry_delay: 500,
});
```

Editable fields currently include:

- `type`
- `payload`
- `when`
- `priority`
- `tags`
- `bucket`
- `bucket_tokens`
- `cron`
- `max_tries`
- `retry_delay`
- `backoff_multiplier`
- `progress`
- `status_message`

---

## Retrying, removing, and clearing jobs

### Retry one job

```javascript
await okdb.queue.retryJob(jobId);
```

This resets the job back to `pending` and resets `tries` to `0`.

### Remove one job

```javascript
await okdb.queue.removeJob(jobId);
```

This deletes both the job metadata and its payload.

### Bulk cleanup helpers

```javascript
await okdb.queue.clearDone();
await okdb.queue.clearFailed();
await okdb.queue.clearStuck();
await okdb.queue.retryFailed();
```

Each method also accepts optional `type` and `limit` parameters.

---

## Buckets (token-bucket rate limiting)

Create a bucket:

```javascript
await okdb.queue.addBucket('gpu-workers', {
    capacity: 100,
    tokens: 100,
    refill_amount: 10,
    refill_every: 1000,
});
```

Update a bucket:

```javascript
await okdb.queue.updateBucket('gpu-workers', {
    tokens: 50,
    refill_amount: 20,
});
```

Remove a bucket:

```javascript
await okdb.queue.removeBucket('gpu-workers');
```

Manually claim tokens:

```javascript
const ok = await okdb.queue.tryClaimTokens('gpu-workers', 2);
```

Jobs can reference a bucket via `enqueue(..., { bucket, bucket_tokens })`.

---

## Cron support

If you use `cron`, you must configure `queue.cron_next`:

```javascript
const Croner = require('croner');

const okdb = new OKDB('./db', {
    queue: {
        cron_next: (expr, fromMs) => {
            return new Croner(expr, { currentDate: new Date(fromMs) }).nextRun().getTime();
        },
    },
});
```

Then enqueue recurring work:

```javascript
await okdb.queue.enqueue(
    'daily-report',
    { reportType: 'summary' },
    {
        cron: '0 8 * * *',
    },
);
```

---

## Events

Queue lifecycle events are emitted on **`okdb.events`**, not on `okdb.queue.events`.

Current low-frequency emitted events include:

- `queue:enqueued`
- `queue:done`
- `queue:failed`
- `queue:retry`
- `queue:removed`
- `queue:reconciled`

Example:

```javascript
okdb.events.on('queue:done', ({ id, type }) => {
    console.log('done', id, type);
});

okdb.events.on('queue:failed', ({ id, type, status_code }) => {
    console.log('failed', id, type, status_code);
});
```

### About filtering by type or key

Today, filtering by job type or job id is best done in your handler:

```javascript
okdb.events.on('queue:done', (ev) => {
    if (ev.type !== 'send-email') return;
    console.log('email job done', ev.id);
});
```

If a richer queue-specific event subscription API is added later, it will likely build on top of these base events.

---

## Sync behaviour

Queue data lives in the current environment, so queue sync follows that environment's sync settings.

That means:

- a job enqueued in a synced env can replicate to peers
- a worker on another node/process can consume from the same synced data
- retry/failure state also follows the same environment data

If you want queue isolation, use a separate environment with the sync behavior you want.

---

## Queue-worker engine

For persistent, self-managing workers that use stored functions, see **queue-worker engines** in [Pipelines](./pipelines.md).

The queue-worker engine is a pipeline-compatible engine type that:

- claims jobs from the queue automatically
- invokes a stored function for each claimed payload
- handles completion/failure using normal queue semantics

Worker `concurrency` controls job throughput. The function runner pool scales automatically to support it — no manual pool sizing is needed.

---

## Admin UI

The admin queue UI supports:

- filtering by status and type
- retry / cancel / delete
- bulk clear / bulk retry actions
- viewing progress
- editing payload and queue settings for a job

Expanded rows show key metadata, and the edit dialog allows modifying payload and most queue settings without editing raw `~queue_jobs` records directly.
