# Wall-E Telemetry Lifecycle

## Goal

Wall-E telemetry should stay useful for product and reliability work without letting raw event volume or sensitive diagnostic text grow without bound. The lifecycle has four layers:

1. Recent raw events for drill-down debugging.
2. Archived daily rollups for long-term aggregate trends.
3. Feedback metadata after diagnostic text ages out.
4. Cleanup run records so every deletion has an audit trail.

## Data Classes

`telemetry_events` stores raw client events. These rows can contain event metadata, client timing, install identifiers, and transient network information. They are valuable for recent debugging but are the main database growth source.

`telemetry_installs` stores durable install state. It keeps install id, machine bucket, version, OS, and aggregate event count. Direct IP and owner display names are temporary convenience fields and should be minimized after they stop being operationally useful.

`feedback_reports` stores user-submitted diagnostic reports. The report id, type, status, severity, version, project metadata, and triage score remain useful after the free-form text and evidence payloads expire.

`telemetry_event_daily_rollups` stores aggregate archived event counts by day, event, version, and OS. This table is append-only through cleanup batches and is safe to use for summary endpoints after raw rows are deleted.

`telemetry_feedback_daily_rollups` stores aggregate archived feedback counts by day, type, status, and severity.

`telemetry_cleanup_runs` records dry-run and live cleanup executions, including the policy, row counts, checkpoint result, and errors.

## Retention Policy

Defaults are intentionally conservative and can be overridden by environment variables:

| Class | Default | Reason |
| --- | ---: | --- |
| Noisy raw events | 14 days | High-volume scheduler/skill events are useful only for recent incidents. |
| Default raw events | 45 days | Preserves recent product behavior while bounding table growth. |
| Diagnostic raw events | 90 days | Errors, provider cooldowns, compatibility, and upgrade lifecycle data need a longer debugging window. |
| Feedback full text/evidence | 180 days | Maintainer triage needs time, but raw text should not live forever. |
| Install IP | 30 days | IP is operational only for abuse/debugging. |
| Owner display name | 7 days | Hashes and machine buckets are enough for fleet analysis. |

Diagnostic event names include `error`, `skill_fallback_attempt`, `initiative_provider_cooldown`, `compat_usage`, `upgrade`, `upgrade_prompt`, `funnel`, `session_integrity_issue`, `session_integrity_issue_summary`, and the `ctm_update_` prefix.

Noisy event names include `skill_dispatch_decision`, `skill_exec`, `skill_run`, `skills_run`, `task_run`, `think`, `initiative`, `reflect`, and `ingest`.

## Cleanup Behavior

Cleanup has two modes:

`dry_run`: computes the same policy and candidate counts without changing the database. This is the default for the API and for the scheduler unless live cleanup is explicitly enabled.

`live`: archives matching raw telemetry rows into daily rollups in batches, deletes only those archived rows, redacts expired feedback payloads, clears expired install IP/name fields, records the cleanup run, and attempts a WAL checkpoint. A live API call must pass an explicit confirmation.

Raw event cleanup is batch based:

1. Select candidate row ids under one retention bucket.
2. Aggregate the selected rows by day, event, version, and OS.
3. In one transaction, add the aggregate counts to `telemetry_event_daily_rollups` and delete the selected ids.
4. Repeat until no candidates remain.

This makes archived event counts exact for deleted rows and lets summary endpoints combine raw rows with archived rollups without double counting.

Feedback cleanup does not delete the report row. It writes daily feedback rollups first, then replaces title, description, triage text, evidence, context previews, and attachment metadata with redacted placeholders.

The cleanup process does not run `VACUUM` automatically. SQLite file compaction can be expensive and should be a separate maintenance action after checking disk pressure and service load. WAL checkpointing defaults to `PASSIVE` to avoid aggressive truncate behavior on storage backends that can return short reads; operators can set `WALLE_TELEMETRY_CLEANUP_CHECKPOINT_MODE=FULL`, `RESTART`, or `TRUNCATE` only when they explicitly need stronger WAL draining.

Session integrity telemetry is intentionally aggregate-first. CTM emits `session_integrity_issue_summary` for fleet-level counts and only emits per-session `session_integrity_issue` details when `CTM_SESSION_INTEGRITY_DETAIL_TELEMETRY=1` is set for a focused debugging run.

Upgrade telemetry distinguishes in-product updates from externally observed version changes. Summary fields use `completed`/`completed_after_apply` for updates with a matching apply-start signal, and `external_completed` for version changes detected without that signal.

Compatibility telemetry reports both current usage and removal blockers. `safe_to_remove` may stay empty even when a compatibility feature has low usage if that feature is not deprecated or still has active installs.

## Operations

Status endpoint:

`GET /api/wall-e/telemetry/storage?key=...`

Returns database file sizes, raw and archived row counts, oldest timestamps, retention policy, dry-run candidate counts, and the latest cleanup run.

Cleanup endpoint:

`POST /api/wall-e/telemetry/cleanup?key=...`

Defaults to dry-run. To execute live cleanup, send `dry_run=false` and `confirm=delete`.

Scheduled cleanup:

The Wall-E scheduler registers `telemetry:cleanup` daily when telemetry server mode is enabled. It runs dry-run by default. Set `WALLE_TELEMETRY_CLEANUP_ENABLED=1` to allow scheduled live cleanup.

## Safety Rules

- Do not delete raw telemetry without writing rollups in the same committed batch.
- Do not expose storage or cleanup APIs without `WALLE_TELEMETRY_SECRET` when the service is public.
- Do not make scheduled destructive cleanup the default.
- Do not store owner display names long-term; use `owner_name_hash` and `machine_bucket` for analysis.
- Do not treat cleanup as a replacement for endpoint query bounds. Drill-down APIs must still stay bounded.
