---
summary: Design note for a hibernation tier for worker environments — checkpoint-based warm start via Crabbox, with dormant leases and serverless backends as gated follow-ups.
title: Worker hibernation plan
read_when:
  - Designing or reviewing cloud worker idle cost, warm start, or snapshot lifecycle
  - Changing the crabbox provider's provision/teardown flow or the WorkerProvider capability surface
---

## Status

Product update (2026-08-28, maintainer decision): warm images are now **on by
default**, answering the open default-on question below. The capture boundary
that gates it is the profile's own declared secret input: `setupEnv` is the
channel that forwards host environment into the setup command, so profiles
declaring it keep requiring an explicit `settings.warmImage: true`, while
profiles without it capture by default. An explicit value always wins in both
directions. Operators who do not want provider-side image storage, or whose
images would retain repository content through Git seeds, set
`settings.warmImage: false`.

Implementation update (2026-08-28): warm images support `machine0` through
`--strategy image`, refresh automatically at the next eligible teardown after
24 hours, and delete superseded checkpoints after replacement. Scrubbing now
also removes SSH-transport session workspaces under
`~/.openclaw-worker/workspaces`. Core node-tunnel sync adds per-repository Git
seeds: pristine post-sync repository copies stored outside the scrub root at
`~/.openclaw-worker/git-seeds`, alongside retained machine-level caches. These
extend warm starts, not phase-2 per-session hibernation, which remains gated;
the historical phase-1 scrub description below predates this cache boundary.

Accepted 2026-08-26 (maintainer: "build live test document PR land"). Phase 1
is implemented in `extensions/crabbox/` and live-proven; phases 2 and 3 remain
gated as designed. Written from direct source inspection of this repo and the
sibling `openclaw/crabbox` checkout (`../crabbox` @ 19bd2f51).

Product update (2026-08-26): profile-level `suspendAfter` is the accepted knob
for automatically suspending idle workers; the next message provisions a
replacement. Crabbox warm images remain opt-in (`settings.warmImage: true`,
paired with `suspendAfter` for warm wakes): review found that default-on
capture would retain whatever `setup` wrote outside the scrubbed worker root
(setup-created credentials included) in provider images for profiles that
never chose it. Default-on returns only with a proven capture boundary.
(Superseded by the 2026-08-28 product update above, which supplies that
boundary.) Phases 2 and 3 remain gated.

Live proof (2026-08-26, isolated dev gateway, Crabbox dev build with the
fixed-ID fork contract):

- Local-container backend, full loop: cold dispatch → `active` in 101s;
  reclaim ran scrub + `checkpoint create` (docker-commit) in a 37s teardown;
  warm dispatch forked the checkpoint to `active` in **40s (2.5x faster)**.
  Fork proven by container image ancestry (worker container ran the exact
  checkpoint-commit image), preinstalled CLI reuse, and zero fallback
  warnings. A second reclaim skipped capture because a fresh image existed
  (single-flight/dedupe path). Scrub boundary verified: the captured image
  contains no `~/.openclaw/cloud-workers` (no node identity, device token,
  bundles, or workspace bytes).
- AWS via managed Crabbox coordinator: two cold dispatches to `active` (221s,
  311s) through real EC2 + node enrollment; capture correctly degraded to
  cold-only with one warning and teardown proceeded — coordinator-brokered
  native checkpoint creation is admin-token-gated in Crabbox today (see
  follow-ups). All leases and checkpoints were cleaned; zero orphans.

Live findings folded back into the code: capture phases need a 180s budget
(60s starves a real `crabbox run`/snapshot round trip under coordinator
latency). Rig traps recorded for operators: unreleased Gateway builds need the
exact locally packed version installed by `setup` (npm >= 11.16 quarantines
install scripts for URL tarballs — install with `--ignore-scripts` to a
prefix; stock Ubuntu AMIs with root umask 077 need `chmod -R a+rX` after
`sudo npm install`).

Follow-ups (upstream `openclaw/crabbox`): allow owner-scoped native checkpoint
create/fork on coordinator-brokered leases (today the image API requires
`broker.adminToken`, so warm images on managed-coordinator profiles degrade to
cold-only); investigate the fresh-AWS-lease early-reboot window that kills the
first `crabbox run` shortly after warmup reports ready.

## Problem

Cloud worker placements are all-or-nothing. While a session is placed, the
Gateway heartbeats the lease (`extensions/crabbox/src/crabbox-worker-heartbeat.ts`)
and the machine bills continuously. When the session stops — explicit
**Stop cloud worker**, archive, move, or failure — the placement teardown
destroys the environment (`src/gateway/worker-environments/placement-dispatch.ts:539`)
and `crabbox stop` releases the machine (AWS: `TerminateInstances`,
`../crabbox/internal/cli/aws.go:876`). The next turn on a reclaimed placement
always redispatches a brand-new machine
(`src/gateway/worker-environments/reclaimed-placement-redispatch.ts`); there is
no warm-lease reuse anywhere — environment ids are derived from the dispatch
idempotency key, which includes the placement generation
(`src/gateway/worker-environments/placement-dispatch.ts:266-268`), so every
dispatch maps to a fresh environment by construction.

The cold path that fresh machine pays, per dispatch:

1. `crabbox warmup` allocation (minutes; 5m + 20m envelope per attempt,
   `extensions/crabbox/src/crabbox-worker-timeouts.ts:7`).
2. Profile `setup` — typically a Node.js install (15m budget,
   `crabbox-worker-timeouts.ts:31`).
3. Node enrollment, whose script `npx`-fetches the full `openclaw` package from
   the registry (`extensions/crabbox/src/crabbox-worker-node-enrollment.ts:94-100`)
   — the dominant fresh-box cost — plus, for Codex, the platform-native
   `@openclaw/codex` verification (`:41-77`).
4. Gateway bundle push (content-addressed; ~240 MB scale envelope,
   `src/gateway/worker-environments/bootstrap.ts:55`).
5. Workspace sync.

Hosted competitors (Hermes Agent's Modal/Daytona terminal backends, Codex
cloud's container-state cache, Claude Code on the web's post-setup snapshot)
park a sandbox at near-zero idle cost and resume it warm.
`docs/plan/cloud-workers.md:210` already names "image snapshot after bootstrap"
as the v2 fast-start path; this note designs it. (The task brief cited
`docs/start/why-openclaw.md` "Sleeping rented compute" — that page does not
exist in-tree; the gap statement here stands on code evidence only.)

## Current behavior — evidence map

| Fact                                                                                                                                                                              | Where                                                                                                                                                                                                           |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Provider contract is provision/inspect/destroy + optional `renew` (never called by any production code)                                                                           | `src/plugins/capability-provider.types.ts:192-232`; no `provider.renew` call site repo-wide                                                                                                                     |
| `WorkerLeaseStatus` already has `dormant`; lifecycle holds dormant records without orphaning                                                                                      | `src/plugins/capability-provider.types.ts:136-144`; `src/gateway/worker-environments/provider-lifecycle.ts:489-496`                                                                                             |
| Only the device provider returns `dormant` (paired-but-offline, 14-day ceiling)                                                                                                   | `src/gateway/worker-environments/device-provider.ts:153-164,21`                                                                                                                                                 |
| Crabbox maps `stopped`/`stopped_with_code` to `destroyed` — a stopped-with-disk lease would be reaped as proven teardown today                                                    | `extensions/crabbox/src/crabbox-worker-provider.ts:67-76,693-712`                                                                                                                                               |
| A new environment state is a SQLite CHECK-constraint rebuild + schema bump (v10→v11) + protocol + UI enum changes                                                                 | `src/state/openclaw-state-schema.sql:2056-2070`; `src/state/openclaw-state-db-contract.ts:10`; `packages/gateway-protocol/src/schema/environments.ts:22-34`; `src/gateway/server-methods/environments.ts:46-58` |
| No idle/TTL policy exists in core; lifetime is provider-owned (`settings.ttl`/`idleTimeout` → Crabbox flags + heartbeat)                                                          | `src/config/zod-schema.cloud-workers.ts:60-73`; `extensions/crabbox/src/crabbox-worker-profile.ts:131-135,305-308`                                                                                              |
| Docker sandboxes already do stop→start resume, and prune (24h idle / 7d age) removes the container but keeps the bind-mounted workspace                                           | `src/agents/sandbox/docker.ts:688`; `src/agents/sandbox/constants.ts:18-19`; `src/agents/sandbox/docker-backend.ts:253-258`                                                                                     |
| SSH bootstrap has a receipt fast path; enrollment script short-circuits on a live pid — a _running_ box already resumes near-free. The gap is exclusively the stopped/parked case | `src/gateway/worker-environments/bootstrap.ts:266-274`; `crabbox-worker-node-enrollment.ts:85`                                                                                                                  |

## Dependency contract — Crabbox (inspected at `../crabbox`)

Crabbox already ships the hibernation primitives; OpenClaw just doesn't use
them:

- **Checkpoints**: `crabbox checkpoint create|list|inspect|restore|fork|delete|prune`
  (`internal/cli/checkpoint.go`). Native kinds: `aws-ebs-snapshot`, `aws-ami`,
  `hetzner-snapshot`, Azure/GCP disk/image, `machine0-image`,
  `parallels-snapshot`, `docker-commit`; plus portable `workspace-archive`.
  `checkpoint fork <chk> --class …` boots a fresh lease from a native snapshot.
  Sources are cleaned with `cloud-init clean --logs` before native capture
  (fresh SSH host keys per fork). Records are two-halved: local metadata +
  provider resource; `inspect --verify --json` audits both (direct AWS/Hetzner
  or via coordinator).
- **Pause/resume**: `crabbox pause|resume` behind a `PausableBackend`
  capability (`internal/cli/provider_backend.go:297-305`) — implemented **only
  by the islo backend** today. The AWS backend has no EC2 stop path at all
  (`DeleteServer` terminates).
- **Serverless backends**: `internal/providers/daytona` (snapshot-based
  sandboxes) and `internal/providers/modal` already exist behind the same
  lease CLI the OpenClaw crabbox plugin drives.
- **Promoted base images**: `crabbox image promote` (coordinator-admin only) —
  account-level default images, complementary but not per-profile.

Gaps to close upstream (small, both repos are ours):

1. **Fixed-ID fork.** Only `warmup`/`run` accept `--lease-id`
   (`internal/cli/run.go:52`); `checkpoint fork` does not. OpenClaw's
   replay-safe provisioning derives a deterministic `cbx_…` lease id from the
   durable operation id and requires the provider to adopt exactly that lease
   on replay (`extensions/crabbox/src/crabbox-worker-profile.ts:407`;
   `docs/gateway/cloud-workers.md:135`). Fork needs the same contract
   (`checkpoint fork --lease-id`, or `warmup --from-checkpoint <chk>`).
2. **`--json` on `checkpoint create`/`fork`** (list/inspect already have it) so
   the plugin parses results structurally instead of scraping lines.

## Directions evaluated

### (a) Dormant cloud lease (stop-with-disk) — defer, with a named trigger

Park the machine stopped, keep the disk, wake on next dispatch. Rejected for
now on three grounds:

- **No provider support where it matters.** Crabbox's `PausableBackend` covers
  only islo; AWS stop-with-disk doesn't exist in Crabbox, and Hetzner bills
  stopped servers at full price (resources stay allocated), so the flagship
  backends yield either no path or no savings. AWS EC2-stop economics
  (EBS-only) are almost identical to an EBS snapshot — which needs no parked
  instance, no reserved capacity, and no held addresses.
- **Largest core surface of the three.** A parked-but-owned lease needs either
  a new environment state (CHECK rebuild, schema v10→v11, protocol + UI enums
  — the maximal persistence change) or delicate reuse of existing states,
  plus park/wake policy that core deliberately doesn't have today (no idle
  machinery, provider-owned lifetime), plus wake-time re-enrollment and
  credential re-arm.
- **Upgrade staleness erodes the value.** A stale Gateway build retires the
  environment and reprovisions rather than downgrading
  (`docs/gateway/cloud-workers.md:154`); any parked box that sleeps across a
  Gateway upgrade is destroyed on wake anyway.

**Trigger to revisit:** a proven backend with genuinely cheap sleep —
Crabbox `PausableBackend` on a big-cloud backend, or a validated
Daytona/Modal profile whose platform auto-stop we want to honor. The
`dormant` lease-status seam and the lifecycle's hold-don't-orphan branch
already exist for it; phase 3 below reserves the slot.

### (b) Checkpoint on teardown + warm-start fork — **chosen**

Maps exactly onto existing Crabbox capability, preserves every identity and
authority invariant by construction (each dispatch still provisions a fresh
lease, fresh enrollment, fresh environment id — nothing about worker
admission, owner epochs, or credential fencing changes), needs **zero core
changes in its first phase**, and its idle cost is snapshot storage
(order €0.01–0.05/GB-month) instead of a live machine.

### (c) Daytona/Modal serverless backend — validation track, not new code

Crabbox already has both backends. The OpenClaw crabbox provider passes
`settings.provider` through; the work is proving the enrolled-node flow on
those platforms (Node.js present or installable, outbound WebSocket egress to
the Gateway, SSH-shaped `crabbox run` for setup), then documenting the
profile. No new `WorkerProvider` implementation is warranted — that would
duplicate the plugin's provider policy in core's back yard. Their native
auto-stop/resume becomes interesting under direction (a)'s trigger.

## Design

### Phase 1 — profile warm images, plugin-only (no core, SDK, schema, or protocol changes)

Owner: the crabbox plugin. Everything below lives in `extensions/crabbox/`.

**Capture (at teardown).** When a profile opts in and no fresh image exists
for the profile's image key, `destroy` — after fencing the heartbeat, before
`crabbox stop` — runs one scrub-and-snapshot step:

1. Scrub via `crabbox run --script-stdin`: stop the node child and remove the
   per-lease state root (`$HOME/.openclaw/cloud-workers/`), which holds the
   node identity, durable device token, worker bundles, and **all workspaces**
   (`docs/gateway/cloud-workers.md:133`). After the scrub the box contains
   only generic profile state: OS + setup results (Node.js), npm/npx caches
   with the exact `openclaw` package, and the Codex platform binary cache.
   No OpenClaw identity, no workspace bytes — safe to share across sessions
   and operators of the same profile. (Server-side, the environment-owned
   pairing is removed at destroy regardless; the scrub is secret minimization,
   not the security boundary.)
2. `crabbox checkpoint create --id <lease> --mode native --wait=false` and
   record the checkpoint id as _pending_. EBS snapshots complete after the
   source terminates, so teardown latency gains only the scrub + initiation,
   not the snapshot wait. Backends where native capture needs the source
   alive (per Crabbox's per-provider rules) either wait bounded or skip
   capture — capture failure never fails teardown.

**Restore (at provision).** `provision` looks up the profile's image record;
if present, verifies availability (`checkpoint inspect --verify --json`), then
forks with the fixed lease id instead of `warmup`. Setup still runs (it is
contractually idempotent and now hits its `command -v` guards); enrollment
still runs fresh (new lease id → new state dir) but resolves the `openclaw`
package from the warm npm cache instead of the registry. Any fork failure
falls back to the cold `warmup` path — never dead-end a dispatch on a
missing, stale, or still-pending snapshot. (Fallback contract: product
behavior per the doctrine, not a compat shim.)

**Image key and staleness.** Record key = hash of (backend provider, profile
`setup`, sorted `setupEnv` names, `desktop`, exact machine class). Exact-class
matching is the accepted conservative decision; cross-class and cross-region
reuse are future work. A key change (edited setup command, backend switch, or
machine class) simply misses the record and cold-provisions; the stale image
ages out.

**Storage.** Plugin KV in the shared state DB (the sanctioned plugin-scratch
store — no new tables, no sidecar files): one record per image key holding
checkpoint id, provider resource id, state (pending/available), created/last
used timestamps. Crabbox's own local checkpoint metadata (its state dir on the
Gateway host) remains the fork-time authority; the KV record is the index.
Single-flight capture per key via compare-and-set so concurrent teardowns
produce at most one snapshot.

**Retention/GC.** On provision and destroy, opportunistically delete
superseded images and images unused for 14 days (matching the device dormancy
ceiling) via `crabbox checkpoint delete`; verify-orphaned records (local half
or provider half missing) are dropped. Provider-side leaks are bounded by
Crabbox's own `checkpoint prune` and crabbox-owned resource tagging.

**Config.** One provider-owned key in the existing `settings` bag —
`settings.warmImage: true` — validated in `crabbox-worker-profile.ts` like
`ttl`/`idleTimeout`. No core config schema change (the bar for new
`openclaw.json` surface stays unmet: provider-owned settings already solve
it). Off by default: warm images create provider-account storage costs and
capture provider-account state, which is an operator spend/trust decision.
Named enablement path: the Cloud workers docs page + config example, and a
`crabbox` doctor/log hint on the first eligible teardown ("enable
settings.warmImage to make the next dispatch warm").

**Cost/latency claim to prove in the PR.** Warm dispatch skips warmup-from-
base-image + Node install + registry fetch; expected minutes → tens of
seconds for the provision-through-enrollment segment on AWS. Live A/B
(cold vs. warm dispatch on the same profile) is the mandatory evidence, per
the live-verify default.

### Phase 2 — per-session hibernation (needs maintainer acceptance; SDK + additive store)

Phase 1 makes the _box_ warm; the reconciled workspace still re-syncs and the
box-local dependency state (installed `node_modules`, build caches — outside
the size-bounded reconcile) is rebuilt by the agent. Phase 2 captures the
machine at reclaim _without_ the workspace scrub (identity still scrubbed),
keyed to the session, so a reclaimed placement's redispatch forks its own
prior state:

- `WorkerProvider` gains an optional capability pair —
  `snapshot?(lease, opts)` / provision `options.restoreFrom` — mirroring how
  `renew?` and `listMachineOptions?` are already optional. Plugin SDK surface:
  additive, registration-validated, no behavior for providers that omit it.
- One additive table (`worker_environment_snapshots`: id, provider, profile,
  session id, snapshot ref JSON, state, created/last-used) in the shared state
  DB — additive-at-same-version by the documented criteria, but flagged here
  because it is persistent-store surface and per AGENTS.md needs explicit
  acceptance before implementation.
- `reclaimed-placement-redispatch` passes the session's snapshot ref into
  dispatch; teardown decides capture per policy. GC by session deletion,
  retention window, and superseded-capture pruning.
- UI: the reclaimed badge can say "resumes warm"; no new placement states.

Phase 2 only starts if phase 1's telemetry shows session redispatch is
frequent enough to justify per-session storage; it is deliberately separable.

### Phase 3 — dormant leases / platform sleep (reserved, gated on the trigger above)

Reuse the `dormant` lease status + lifecycle hold branch; map a genuinely
parked lease (Crabbox pause on a supporting backend, Daytona auto-stop) to
`dormant` instead of `destroyed`; add the wake path. Requires the environment
state-machine and schema discussion — not designed further here until a
backend earns it.

## Validation plan (phase 1)

- Unit: image-key hashing, KV single-flight, fork-vs-warmup selection,
  fallback on fork failure, GC eligibility (crabbox extension tests).
- Live (mandatory): cold dispatch → stop → warm dispatch on a real AWS
  profile; assert the fork path was taken, enrollment succeeded, timings
  recorded; then a stale-image dispatch proving cold fallback. Same-flow proof
  on Hetzner if coordinator checkpoint support verifies.
- Upstream: Crabbox PR for fixed-ID fork + `--json`, with its own e2e.

## Open questions for the maintainer

1. Accept phase 1 as scoped (plugin-only, opt-in `settings.warmImage`)?
2. Default-off confirmed? (Doctrine says defaults are the product; storage
   spend in the operator's provider account argues opt-in. Recommendation:
   opt-in now, revisit default-on once cost surfacing exists.)
3. Retention default 14 days — match device dormancy, or shorter (7d, matching
   terminal-environment row retention)?
4. Is phase 2 wanted at all before demand evidence, given the Gateway already
   owns transcript + reconciled workspace?
5. Green-light the two upstream Crabbox changes?
