# rmfmail / mailx — Target Architecture & Migration Plan

**Date:** 2026-06-12
**Goal:** Thunderbird/Outlook-grade reliability without a green-field rewrite. Re-architect the **core data + sync layer**; keep the existing UI and feature surface and migrate it onto the new core in shippable, reversible phases.
**Companion:** `architecture-review.md` (the diagnosis this design answers).

---

## 1. The two principles that make mail clients reliable

Every reliable mail client (Thunderbird's global DB / "Panorama", Outlook's OST, Apple Mail's Envelope Index) obeys these:

1. **The display layer never blocks on the network.** A local store (a database) is the single source of truth for *everything the user sees and does*. Opening a folder, reading a message, searching, deleting — all are local operations that complete instantly. The network is a **separate, fully asynchronous background engine** that reconciles the store with the server. A dead/slow server makes mail *stale*, never makes the UI *freeze*.

2. **One canonical store, one stable identity, one path per operation.** A message has one durable local identity that survives folder moves and server UID renumbering. Each operation (delete, move, flag) has exactly one implementation that mutates the store and queues a server action. There are no parallel code paths to drift.

rmfmail violates both today (see review §3, §6). This design restores both.

---

## 2. Three layers, strictly separated

```
┌─────────────────────────────────────────────────────────────┐
│  UI (client/)                                                │
│  • reads: pure store queries (instant, never await network)  │
│  • writes: call Store mutation → optimistic local change     │
│            + enqueue server action; return immediately       │
│  • re-renders from Store change events                       │
└───────────────▲───────────────────────────┬─────────────────┘
                │ store events               │ read / mutate (local only)
┌───────────────┴───────────────────────────▼─────────────────┐
│  STORE  (the source of truth for display)                    │
│  • SQLite. Stable message identity = local UUID.             │
│  • Pure, synchronous, network-free read API.                 │
│  • Mutation funnel: trash/move/flag/copy/deletePermanent —   │
│    one function each, keyed by UUID. Each = local change +    │
│    durable queued action + change event.                     │
│  • Durable action queue (idempotent, UUID-keyed).            │
└───────────────▲───────────────────────────┬─────────────────┘
                │ "here is new/changed data" │ "drain these actions"
┌───────────────┴───────────────────────────▼─────────────────┐
│  SYNC ENGINE  (the ONLY thing that touches the network)      │
│  • Owns all IMAP/Gmail connections, fully isolated.          │
│  • PULL: incremental fetch (QRESYNC/CONDSTORE → set-diff).   │
│  • PUSH: drain the action queue to the server, idempotently. │
│  • RECONCILE: re-bind identity across UID changes by         │
│    Message-ID / UUID. Maintain (folder,uid,uidvalidity) map. │
│  • Bounded connections, per-op timeout + cancellation,       │
│    backoff, poison-action cap. A hung op affects ONLY here.  │
└──────────────────────────────────────────────────────────────┘
```

The hard rule: **an arrow never points "up-and-blocking."** The UI calls into the Store and returns; it never awaits the Sync Engine. The Sync Engine pushes results down into the Store and the Store emits events up to the UI. There is no path where a network stall can sit inside a UI read.

---

## 3. Identity (fixes review §5 — the reappear/ghost/wipe class)

- Every message gets a **stable local `id` (UUID)** at first sight.
- The identity key is **Message-ID** when present; otherwise a synthesized stable hash of `(account, internaldate, from, subject, size)`. This guarantees an identity even for malformed / Message-ID-less mail (the gap that breaks re-binding today).
- `(folder, server_uid, uidvalidity)` is **transient routing metadata**, owned and maintained only by the Sync Engine, in a `message_locations` table. **Nothing in the UI or Store mutations is ever keyed on a bare `uid`.**
- Multi-folder membership (Gmail labels, or a message genuinely in two folders) is N rows in `message_locations` pointing at one message UUID.
- Server-side UID renumber (move, UIDVALIDITY bump) re-binds the location row to the same UUID by Message-ID match — never creates a new identity, never orphans the body.

**Consequence:** the 70k-row wipe (a `WHERE uid=?` that hit every folder) becomes structurally impossible — there is no UID-only delete. Cross-folder collisions, stars-on-wrong-rows, ghosts, wrong-row-on-auto-advance all dissolve because UID is no longer identity.

---

## 4. Mutation funnel (fixes review §6 — "every fix breaks another path")

The Store exposes exactly these mutators, each taking message UUIDs:

```
store.trash(uuids)            // move to Trash, or permanent-expunge if already in Trash
store.move(uuids, toFolder)
store.copy(uuids, toFolder)
store.flag(uuids, flag, on)
store.deletePermanent(uuids)
store.markRead(uuids, on)
```

Each does, atomically: **(1)** update local rows + folder counts, **(2)** record an undo entry, **(3)** enqueue a durable, idempotent server action keyed by UUID, **(4)** emit a change event.

- **Every** UI trigger (Delete key, toolbar, drag, right-drag menu, context menu, pop-out, undo) calls these and nothing else. No optimistic-removal logic scattered in the UI; the UI just re-renders from the change event.
- `emptyFolder` becomes `store.deletePermanent(all-uuids-in-folder)` — same funnel, not a separate IMAP loop.
- **Delete `packages/mailx-core`** (dead duplicate).
- **Android shares this exact Store + queue.** The only platform difference is the `Transport` the Sync Engine injects (Node TCP vs bridge). No second `trashMessage`.

---

## 5. Read path & dispatch isolation (fixes review §1/§2 — the timeouts you're hitting now)

- All reads — `listFolder`, `unifiedInbox`, `getMessage`, `search` — are **pure synchronous SQLite** behind a single read facade. They **cannot** call the network. `getMessage` returns the cached body if present, else `{cached:false}` and the UI shows a placeholder while the Sync Engine fetches and emits `bodyReady`. A read **can never time out.**
- The IPC dispatch (`bin/mailx.ts`) must not serialize a read behind a network write. Two concrete options (pick after measuring):
  - **(A)** Run the Sync Engine's network work off the dispatch thread (worker), so `dispatch()` only ever does fast local work.
  - **(B)** Split dispatch into a read lane (own SQLite read handle, always-fast) and a write/command lane; reads are answered immediately regardless of what the write lane is doing.
- Network handlers get a hard wall-clock timeout + cancellation; a hung op is abandoned and its connection discarded, never occupying a shared slot. (The `withTimeout`/dual-lane machinery already exists in `mailx-imap` — it just needs to be *above* nothing that a read depends on.)

**Consequence:** `getUnifiedInbox` (and every list/read) returns instantly even while the server is hung. The "nothing in the summary" / `mailxapi timeout` failure class is gone by construction.

---

## 6. Sync Engine robustness (the part that earns "Thunderbird-grade")

- **Connection discipline:** per-account budget with a **dedicated interactive connection** that background sync/prefetch can never starve. Per-host semaphore stays as the server-cap guard.
- **Every op:** wall-clock timeout + cancellation; on timeout, discard the connection and back off. A single un-fetchable message (today's 4966060) is recorded with backoff and never re-attempted in a tight loop.
- **Push queue:** durable, idempotent, UUID-keyed actions; retry with backoff; a **poison cap** (give up after N, surface to the UI) so one bad action can't loop forever.
- **Pull:** capability-gated incremental sync — QRESYNC/CONDSTORE when the server advertises them, UID set-diff fallback otherwise; persist per-folder `UIDVALIDITY` + `HIGHESTMODSEQ`. (Memory note: capability-gate, don't hostname-branch.)
- **Reconcile:** the only place that re-binds UID↔UUID; tombstones with retained UUID suppress re-fetch of locally-deleted mail; UIDVALIDITY bump triggers a clean re-map, not a wipe.

---

## 7. Migration plan — phased, shippable, reversible (NOT big-bang)

Each phase ships on its own, is independently verifiable, and can be reverted. We do **not** rewrite the UI or the features.

**Phase 0 — Read isolation (do first; ends the timeouts).**
- Guarantee every read RPC is network-free and cannot be head-of-line-blocked. Implement §5 option A or B in `bin/mailx.ts` + the service read methods. Add hard timeouts/cancellation to network handlers.
- *Exit test:* with the IMAP server artificially hung, the list, folder switch, search, and cached-message open all still respond instantly. No `mailxapi timeout` on any read.

**Phase 1 — Identity + mutation funnel.**
- Introduce/standardize the UUID identity and `message_locations`; backfill UUIDs (rebuild-from-server is acceptable per the "no migrations, redownload" rule).
- Route all deletes/moves/flags through the §4 funnel; delete `mailx-core`; converge `emptyFolder`.
- *Exit test:* a scripted scenario with the same numeric UID in INBOX + Trash + Sent — delete/flag/move one and confirm the others are untouched. No ghost rows after a move+delete cycle.

**Phase 2 — Sync Engine extraction.**
- Make the Sync Engine the sole network owner with §6 discipline; move cross-account move and outbox under it. Dedicated interactive lane.
- *Exit test:* hung-folder fault injection — background sync stalls, interactive open/delete stay responsive; poison action gives up instead of looping.

**Phase 3 — Search unification + Android convergence.**
- One query parser feeding the (local-FTS / server-IMAP) executors. Android uses the shared Store + queue + a `Transport`.
- *Exit test:* identical query semantics across scopes; Android delete sets the same pending/tombstone state as desktop.

Order rationale: Phase 0 stops the bleeding you're seeing *right now*; Phase 1 removes the data-integrity class (the scary one — the wipe); Phases 2–3 are robustness + cleanup.

---

## 8. What we explicitly do NOT do

- No green-field rewrite. The UI, compose/editor, providers, OAuth, calendar/tasks, Android shell, and the feature set stay.
- No big-bang cutover. Each phase coexists with the rest.
- No hot-patching the live daemon during this work — changes land, get built, and are verified deliberately (the firefighting pattern is over).

---

## 9. How we'll verify reliability (the Thunderbird bar)

A short fault-injection harness, run after each phase:
1. **Server hung** (stall `BODY[]` / SELECT): UI reads stay instant; no timeouts.
2. **Server slow** (5s/op): actions queue and drain; UI never blocks.
3. **Offline → online:** queued actions drain in order, idempotently; no dupes, no loss.
4. **UID collision** across folders: per-message ops never hit the wrong message.
5. **UIDVALIDITY bump:** re-map, no wipe.
6. **Crash mid-action:** durable queue resumes; nothing lost, nothing double-applied.

When all six pass under injection, we're at the reliability bar you're asking for.

---

*This is the target. Phase 0 is the next concrete step and is self-contained.*
