# Wall-E Architecture

This document describes Wall-E's runtime architecture: what runs, when, on
what cadence, how it talks to itself, and — equally important — what it
**deliberately does not** become.

Wall-E is a **personal-daemon AI agent**: one user, the user's data, the
user's machine. The architecture optimizes for that shape. It is not a
multi-tenant agent platform.

## Top-level shape

```
                            ┌──────────────────────────┐
                            │      User's CTM tab      │
                            │   (chat.js / browser)    │
                            └────────────┬─────────────┘
                                         │
              ┌──────────────────────────┴────────────────────────────┐
              │                  Inbound channels                     │
              │  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌────────┐ │
              │  │ iMessage │  │ Slack DM │  │ Telegram │  │Webhook │ │
              │  └─────┬────┘  └─────┬────┘  └─────┬────┘  └───┬────┘ │
              └────────┼─────────────┼─────────────┼───────────┼──────┘
                       └──────┬──────┴─────────────┘           │
                              │                                │
                              ▼                                ▼
                      ┌──────────────┐              ┌────────────────────┐
                      │  Event Bus   │              │  HTTP API trigger  │
                      │ (in-process) │              │ (api-walle.js)     │
                      └──────┬───────┘              └────────┬───────────┘
                             │                               │
                             └────────────┬──────────────────┘
                                          ▼
                             ┌────────────────────────┐
                             │    Source Registry     │  ← Item E
                             │ (dispatch_source enum) │
                             └────────────┬───────────┘
                                          │
                                          ▼
                             ┌────────────────────────┐
                             │       Scheduler        │
                             │   (lib/scheduler.js)   │
                             └────────────┬───────────┘
                                          │
                ┌─────────────┬───────────┼───────────┬─────────────┐
                ▼             ▼           ▼           ▼             ▼
          ┌─────────┐   ┌─────────┐ ┌─────────┐ ┌─────────┐  ┌──────────┐
          │ ingest  │   │  think  │ │ reflect │ │ skills  │  │ tasks +  │
          │ (60s)   │   │ (2min)  │ │ (1hr)   │ │ (5min)  │  │ replay + │
          │         │   │         │ │         │ │         │  │ backfill │
          └────┬────┘   └────┬────┘ └────┬────┘ └────┬────┘  └────┬─────┘
               │             │           │           │            │
               └─────────────┴─────┬─────┴───────────┴────────────┘
                                   ▼
                         ┌──────────────────┐
                         │   Brain (SQLite) │  47 tables
                         │  ~/.walle/data/  │
                         └──────────────────┘
```

## Loops (data-driven coordination)

Each loop is a narrow, single-purpose process. **No loop coordinates
another.** Loops share state only through the brain DB.

| Loop | Cadence | Purpose | Entry |
|---|---|---|---|
| **boot** | once at start | Validates BOOT.md, sanity-checks env | `loops/boot.js` |
| **ingest** | 60s | Pulls from external sources (Slack, Gmail, Calendar, Coda, etc.) → memories | `loops/ingest.js` |
| **think** | 2min | Knowledge extraction + entity learning + contradiction detection over pending memories | `loops/think.js` |
| **reflect** | 1hr | Long-running synthesis, briefing generation | `loops/reflect.js` |
| **skills** | 5min | `runDueSkills` walks all enabled skills, dispatches each (see Skills section) | `skills/skill-planner.js` |
| **initiative** | event-driven (woken by channels) | Decides whether to respond to inbound messages | `loops/initiative.js` |
| **tasks** | 30s | Executes scheduled tasks created by other loops or the user | `loops/tasks.js` |
| **replay** | 5min | Re-runs failed extractions / dispatches | `loops/backfill.js` |
| **backfill** | 30s | Classifies + embeds old un-tagged memories | `loops/backfill.js` |

Each loop's "is it time to run" + "should it run" decision goes through the
**centralized loop precheck** (`loops/loop-precheck.js`) — provider availability
+ fresh-signal gate. Same observability discipline as skill dispatch.

## Inbound channels

Wall-E listens on multiple inbound surfaces. All eventually reach the
scheduler via the event bus (`events/event-bus.js`):

| Channel | Module | Trigger |
|---|---|---|
| **CTM Wall-E tab** | `chat.js` (HTTP) | User typing in the browser |
| **iMessage** | `channels/imessage-channel.js` | macOS Messages.app polling |
| **Slack DM** | `channels/slack-channel.js` | Slack RTM polling |
| **Telegram** | `channels/telegram-channel.js` | Telegram getUpdates polling |
| **Webhook** | `api-walle.js` POST `/api/wall-e/webhook` | External HTTP event ingestion |
| **Manual API** | `api-walle.js` POST `/api/wall-e/skills/:id/run` | UI button or admin call |
| **Scheduled cron** | scheduler tick → `runDueSkills` | Time-based |
| **Boot / restart-catchup** | `agent.js` startup + `scheduler.runMissedJobs` | Process start |

Today, every channel emits to the event bus and the agent wakes the
**single** initiative loop. Item E (source registry) makes this routing
**observable** and **configurable** without falling into the
manager-of-managers anti-pattern (see Non-goals).

## Brain (data layer)

SQLite via `better-sqlite3`, located at `${WALL_E_DATA_DIR:-~/.walle/data}/wall-e-brain.db`.

```
PRAGMA user_version = 5  (current as of 2026-04-25)

Migrations:
  1: column consolidation across tasks/skills/chat/sessions/memories/etc.
  2: skills.execution = 'agent' | 'script'
  3: scheduler_job_state (persistent restart catchup)
  4: skill_executions.model + decision_reason
  5: skills.auto_disabled_at + auto_disabled_reason
  6: auth_profiles + skills.auth_profile_id  ← Item C this branch
```

Key tables relevant to this architecture (47 total):

- `skills` — bundled + user skills
- `skill_executions` — per-run audit trail (with `model` + `decision_reason`)
- `scheduler_job_state` — persistent per-job state for restart catchup
- `memories` — raw + derived knowledge
- `kv_store` — feature flags, last-decision cache, alerts, runtime config
- `chat_sessions` + `chat_messages` — conversational state per source
- `service_alerts` (kv-backed) — auth failures, version updates, etc.

## Skills (the unit of agency)

Skills are Wall-E's "agents" — narrow, single-purpose units with their own
prompts, tools, and metadata. They live in three precedence layers:

```
workspace > user > bundled > registry
  .wall-e/    ~/.wall-e/   wall-e/    ~/.wall-e/
  skills/     skills/      _bundled/  registry-skills/
```

Each skill is a directory with `SKILL.md`:

```yaml
---
name: slack-mentions
description: Pull recent @mentions from Slack
execution: script         # 'agent' (Claude tool-use) or 'script' (bundled JS)
trigger:
  type: interval          # 'interval' | 'manual' | 'webhook' | 'event'
  interval_ms: 300000
tags: [slack, communication]
lane: production          # Item B this branch
model_policy:             # Item B this branch
  primary: claude-haiku-4-5-20251001
  fallback: claude-sonnet-4-6
auth_profile: slack-default  # Item C this branch
requires:
  bins: [slack-mcp]
  env: [SLACK_BOT_TOKEN]
---
[markdown system prompt for agent skills]
```

### Skill dispatch lifecycle

```
1. scheduler ticks → runDueSkills(opts)
2. snapshot = getCurrentSkillSnapshot({brain})       (skills/skill-snapshot.js)
3. for each skill in snapshot.skills:
   a. decideSkillDispatch(skill, ctx)                (skills/skill-dispatch-decision.js)
      → { decision, reason, details }
      reasons: due, not_due, disabled, auto_disabled,
               just_auto_disabled, backed_off, no_trigger
   b. brain.recordSkillLastDecision(skill.name, decision)
   c. if decision.reason === 'just_auto_disabled':
        flip enabled=0, stamp auto_disabled_at, alert
        (continue)
   d. if decision.decision !== 'run': continue
   e. preflight requires (bins/env/mcp)
   f. if execution === 'script':
        resolveInternalSkill(name)                   (skills/internal-skill-registry.js)
        run factory({brain, log, authProfile})       ← Item C this branch
   g. if execution === 'agent':
        runSkillWithFallback(skill, opts)            (skills/skill-fallback.js)
          ↓
        selectSkillHarness(skill, opts)              (skills/skill-harness-registry.js)
          → claude-tool-use harness (priority 0, default)
          → registered plugin harnesses (priority N)
          ↓
        harness.runAttempt(skill, opts)
          ↓
        _runSkillInternal(skill, opts)               (skills/skill-executor.js)
          → tool-executor.js dispatches each tool call
          → optional sandbox.runner (skills/sandbox-registry.js — currently stub)
   h. log execution + memories + telemetry
```

## Scheduler (cron-style)

`lib/scheduler.js` — pool-based, persistent, observable.

- **Pool semaphores**: `llm:2`, `ollama:1`, `embedding:1`, `io:3`
- **MIN_REFIRE_GAP_MS = 2000** (spin-loop guard)
- **Persistent state** in `scheduler_job_state`: hydrate on register, run
  missed jobs on boot via `runMissedJobs()`
- **Failure alerts**: `recordJobError` triggers `service_alerts` after 2
  consecutive failures, 1hr cooldown
- **Immediate-tick on wake** for low-latency response to `eventBus.emit*`

## Observability discipline

Every dispatch decision is recorded with structured reason values. This
matches OpenClaw's `selectedReason` (harness) and `matchedBy` (route)
patterns.

| Decision point | Recorded as | Reason values |
|---|---|---|
| Skill should-run gate | `decideSkillDispatch` → kv `skill_dispatch_last_decision` | `due` / `not_due` / `disabled` / `auto_disabled` / `just_auto_disabled` / `backed_off` / `no_trigger` |
| Skill execution attempt | `skill_executions.decision_reason` | `due` / `manual_run` / `webhook` / `boot` / `restart_catchup` |
| Skill execution model | `skill_executions.model` | concrete model id |
| Skill harness selection | `telemetry.skill_harness_selected` | `pinned` / `auto` / `fallback_default` |
| Skill fallback chain | `telemetry.skill_fallback_attempt` | `success` / `failure` per candidate |
| Inbound dispatch source | `dispatch_source` (Item E) | `chat` / `webhook` / `manual_api` / `scheduled` / `boot` / `restart_catchup` / `imessage` / `slack_dm` / `telegram` |

Every channel-driven dispatch carries a `dispatch_source` so operators can
answer "what woke this loop, and why?" without reading source.

## Non-goals (architectural guardrails)

Wall-E **deliberately does NOT** use:

### 1. Manager-of-managers / nested planner trees

Coordination between loops (ingest, think, reflect, skills, initiative) is
**data-driven via the brain DB and the scheduler**. No LLM-driven
"coordinator" agent decides which loop runs.

This matches OpenClaw's `VISION.md` guidance verbatim:

> Agent-hierarchy frameworks (manager-of-managers / nested planner trees)
> as a default architecture

### 2. Heavy orchestration layers wrapping existing infrastructure

If a feature needs orchestration, it goes **inside** the existing
skill/loop/scheduler layer — not in a new wrapper above them. The skill
harness registry, internal-skill registry, trigger registry, sandbox
registry, source registry, and auth-profile resolver are all flat lookups
that PLUG INTO existing dispatch — they don't sit above it as a meta-layer.

### 3. Multi-agent role-play frameworks

We do not build "the planner agent calls the executor agent calls the
critic agent." Skills are the unit of agency. Each skill is its own
narrow agent with its own prompt and tools.

If two skills need to coordinate, they coordinate via the brain DB
(memory ingestion, knowledge extraction, briefing items) — not via inline
agent-to-agent calls.

### 4. Inline RAG-style "let the agent decide which tool to call from 200 options"

Tool definitions are scoped to the skill (`SKILL.md` `tools:` field) or
defaulted to a small set (`getToolDefinitions()`). The skill is the
filter, not a vector-search-over-tools-at-runtime.

---

These are guardrails, not laws of physics. Strong user demand + strong
technical rationale can revisit. But the default answer to "should we add
a coordinator agent?" is **no** — coordinate via data, not via LLM
orchestration.

## Per-loop directives (Item D this branch)

Most loops have their system prompt inlined in JS. For loops where the
prompt is the primary thing the user might want to tune (initiative,
skill-executor default, chat), Wall-E now supports an opt-in
**directives file** at `wall-e/loops/<name>.directives.md`. If present,
its contents override the inline default after `{{ ownerName }}`-style
template substitution.

This matches OpenClaw's scoped `AGENTS.md` pattern (`src/agents/AGENTS.md`,
`test/helpers/AGENTS.md`). The directives file lets users tune
per-loop behavior without forking the codebase.

## Per-skill auth profiles (Item C this branch)

A new `auth_profiles` table records named credential sets. Each row:

```sql
{
  id          TEXT PRIMARY KEY,
  name        TEXT NOT NULL UNIQUE,    -- 'default' | 'work-slack' | 'personal-slack'
  type        TEXT NOT NULL,           -- 'slack' | 'gmail' | 'anthropic' | ...
  credentials_kv_key TEXT NOT NULL,    -- pointer to kv_store key holding the actual token
  metadata    TEXT,                    -- JSON for type-specific config
  created_at  TEXT
}
```

Skills can declare `auth_profile: <name>` in their SKILL.md frontmatter.
At dispatch time, the resolver looks up the profile and passes the
matched credentials to the skill factory or harness.

Default behavior: a single `default` profile is auto-created at boot
that wraps the existing global env-var/kv tokens. Existing skills are
unaffected. Forward infrastructure for users who want to ingest from
multiple Slack workspaces / multiple Gmail accounts.

## Conversation collapse policy (Item F this branch)

A new kv config `wall_e_conversation_collapse` controls how Wall-E groups
inbound messages from different channels:

- `'main'` (default): all inbound from all channels feeds one chat
  history. Backwards-compatible with current behavior.
- `'per-source'`: each channel (`chat` / `slack_dm` / `imessage` /
  `telegram` / `webhook`) gets its own brain conversation. Useful when
  the user wants Wall-E to NOT mix work-Slack context into iMessage replies.

Set via Setup page or `brain.setKv('wall_e_conversation_collapse', 'per-source')`.

## Source registry (Item E this branch)

Every dispatch (chat / webhook / cron / boot / etc.) is tagged with a
**dispatch_source** enum value. The source registry
(`wall-e/dispatch/source-registry.js`) provides:

- `registerSource(id, descriptor)` — id matches `dispatch_source`
- `resolveSource(id)` — returns descriptor with `label`, `kind` (`channel` |
  `internal` | `external`), `default_collapse_policy`
- Default-registered sources: `chat`, `webhook`, `manual_api`,
  `scheduled`, `boot`, `restart_catchup`, plus the four channels.

This is the shape OpenClaw's `resolveAgentRoute` uses, adapted for
Wall-E's single-tenant model: instead of mapping
`(channel, peer, guild, role)` → `agentId`, we map `dispatch_source`
to operating-rules + collapse-policy + telemetry tag.

We do NOT build `(channel, peer, guild, role)` → `agentId` because Wall-E
has one user. The simplification is intentional, not an oversight.

## File map

```
wall-e/
├── agent.js                  daemon entry, scheduler wiring, channel listeners
├── brain.js                  SQLite layer (47 tables, 6 migrations)
├── chat.js                   CTM Wall-E tab handler
├── api-walle.js              HTTP API surface (skills, telemetry, webhook)
├── lib/
│   └── scheduler.js          pool-based job scheduler with persistent state
├── loops/
│   ├── boot.js               BOOT_OK preflight
│   ├── loop-precheck.js      provider/freshness gate
│   ├── loop-prompt-discipline.js  shared prompt scaffolding
│   ├── ingest.js             external-data → memory pipeline
│   ├── think.js              extraction / contradiction / entity
│   ├── reflect.js            long-form synthesis
│   ├── tasks.js              scheduled-task executor
│   ├── backfill.js           re-run failed extractions
│   ├── initiative.js         event-driven response decisions
│   └── *.directives.md       (Item D) opt-in per-loop prompt overrides
├── skills/
│   ├── skill-planner.js      runDueSkills orchestrator
│   ├── skill-snapshot.js     versioned per-tick snapshot freeze
│   ├── skill-dispatch-decision.js  pure run/skip decision function
│   ├── skill-loader.js       SKILL.md parser + cache
│   ├── skill-validator.js    requires-bins/env/mcp preflight
│   ├── skill-executor.js     thin wrapper around harness selection
│   ├── skill-fallback.js     model_candidates retry chain
│   ├── skill-harness-registry.js     pluggable agent runtimes
│   ├── internal-skill-registry.js    pluggable script-skill factories
│   ├── trigger-registry.js   interval / manual / webhook / event
│   ├── sandbox-registry.js   stub for future Docker/Podman plugins
│   ├── tool-executor.js      direct tool dispatch
│   └── _bundled/             53+ bundled skills
├── channels/
│   ├── channel-base.js
│   ├── imessage-channel.js
│   ├── slack-channel.js
│   ├── telegram-channel.js
│   └── agent-api.js
├── events/
│   └── event-bus.js          in-process pub/sub for channels → loops
├── dispatch/                 (Item E) source registry + collapse policy
│   └── source-registry.js
├── auth/                     (Item C) auth profile resolver
│   └── profiles.js
└── tests/                    1700+ tests
```

## OpenClaw lineage

This architecture intentionally borrows from OpenClaw's open-source
agent platform while staying true to the personal-daemon shape. Patterns
adopted:

| OpenClaw pattern | Wall-E adaptation | Branch |
|---|---|---|
| `selectAgentHarness` candidate-list logging | `decideSkillDispatch` reason enum | skill-dispatch deep-dive |
| Skills snapshot freeze (`resolveCronSkillsSnapshot`) | `getCurrentSkillSnapshot` | skill-dispatch deep-dive |
| Sandbox backend `Map<id, factory>` registry | Same shape, used 5 places | skill-dispatch deep-dive |
| `runAgentHarnessAttemptWithFallback` | `runSkillWithFallback` over `model_candidates` | skill-dispatch deep-dive |
| Persistent cron job state + restart catchup | `scheduler_job_state` + `runMissedJobs` | agent-loop deep-dive |
| `MIN_REFIRE_GAP_MS = 2000` spin-loop guard | Same constant, configurable | agent-loop deep-dive |
| `DEFAULT_FAILURE_ALERT_AFTER = 2` + 1hr cooldown | `recordJobError` → service_alerts | agent-loop deep-dive |
| Heartbeat content classifier | `loops/heartbeat-content.js` ATX/list/fence regexes | agent-loop deep-dive |
| Boot phase + workspace contract | `loops/boot.js` + BOOT.md | agent-loop deep-dive |
| Loop precheck (provider + fresh-signal) | `loops/loop-precheck.js` shared gate | agent-loop deep-dive |
| Scoped `AGENTS.md` per directory | `loops/<name>.directives.md` (Item D) | this branch |
| Auth profile per-cron override | Per-skill `auth_profile` (Item C) | this branch |
| `lastRoutePolicy: main` vs `session` | `wall_e_conversation_collapse` (Item F) | this branch |
| `matchedBy` enum on resolved route | `dispatch_source` enum (Item E) | this branch |
| `VISION.md` non-goals | Non-goals section above | this branch |

Patterns explicitly NOT adopted (and why):

| OpenClaw pattern | Why not |
|---|---|
| `(channel, peer, guild, role)` → `agentId` mapping | Wall-E has one user; routing is by `dispatch_source` only |
| Multi-tenant auth profiles with row-level isolation | Single-tenant; auth profiles are user-managed, not RBAC |
| Full Docker/Podman/SSH sandbox subsystem | Personal daemon, threat model doesn't justify ~40h plumbing |
| Manager-of-managers / nested planner trees | Anti-pattern, see Non-goals above |

## Versioning

`SCHEMA_VERSION` lives in `brain.js`. Every migration is forward-only,
idempotent, and exercised by `tests/brain.test.js`. To add a migration:

1. Bump `SCHEMA_VERSION`
2. Add `MIGRATIONS[N] = (d) => { ... }`
3. Update `CREATE TABLE` statements for fresh installs
4. Add a test that exercises the migration on a v(N-1)-shaped DB

Schema version is independent of Wall-E semver — the npm package version
is in `package.json`, the schema version is in the database. The two are
correlated but not identical: a wall-e@1.5.0 install can come up against
a v3 DB and migrate it to v6.

## Testing

`npm test` from `wall-e/` runs the full suite. Every architectural
component has its own test file. Tests use:

- Node.js built-in test runner (`node --test`)
- `--test-force-exit` to prevent handle leaks
- Per-test `WALL_E_DATA_DIR=/tmp/walle-test-<X>` isolation
- Real `better-sqlite3` (no mocks) — migrations are tested against actual SQLite

Smoke verification protocol (used by `/ctm-learn`):

1. Copy production brain DB to `/tmp/walle-smoke-<topic>/`
2. Set `WALL_E_DATA_DIR` and run new code against it
3. Verify migration ran cleanly + new endpoints behave correctly
4. Read out: `user_version`, new column presence, row counts, kv keys

This catches migration bugs that pass unit tests but break against real
data shape (e.g. the ALTER TABLE DEFAULT bug in agent-area item #6).
