# Large JSONL Transcript Ingestion

## Problem

Claude, Codex, and Wall-E all write provider-owned JSONL transcripts. Those
files are append-only most of the time, but they can become very large and they
can rotate or compact. CTM currently has several partial readers:

- metadata readers that inspect the head of a file;
- hot stream readers that tail a small byte window;
- background importers that update `session_conversations`;
- review routes that sometimes parse JSONL directly when cache is stale.

The repeated failures come from those readers making independent freshness
decisions. A large session can have a fresh terminal tail, a stale conversation
cache, and a review route that refuses to parse a multi-GB JSONL. The user sees
that as lost history after restart, slow resume, stale summaries, or terminal
and Conversation tabs disagreeing.

## Design Rule

Provider JSONL is the append-only source log. CTM's DB row store is the query
surface. UI requests must never depend on parsing an entire provider JSONL file.

## Access Modes

| Mode | Use for | Never use for |
| --- | --- | --- |
| Head | session identity, cwd, model, first prompt, title seed | latest prompt, current status, review history |
| Tail | recent turns, current task, stale-cache catch-up, first page of a cold large review | complete history, search indexing |
| Incremental | active sessions, live updates, restart catch-up from a saved offset | cold import without a cursor |
| Full | background backfill, export, integrity repair, one-time migration | resume, initial Review open, terminal input readiness |

## Target Architecture

### Transcript Files

`transcript_files` records one durable cursor per provider file.

- `agent_session_id`: provider-native session/thread id.
- `ctm_session_id`: owning CTM tab id when known.
- `jsonl_path`: canonical provider JSONL path.
- `provider`: `claude`, `codex`, `walle`, or another adapter name.
- `dev`, `ino`: file identity when the filesystem exposes it.
- `size`, `mtime_ms`: last observed file state.
- `offset`: byte offset after the last complete parsed line.
- `partial_line`: incomplete trailing UTF-8 line, if any.
- `parser_version`: allows safe re-index after parser changes.
- `status`, `last_error`, `updated_at`: operational state.

### Transcript Events

`transcript_events` stores one queryable row per visible transcript message.

- `ctm_session_id`, `agent_session_id`, `provider`, `jsonl_path`.
- `seq`: monotonic insertion order per agent session.
- `role`: `user`, `assistant`, or `system`.
- `timestamp`: provider timestamp when available.
- `content`: normalized visible text.
- `byte_start`, `byte_end`: provenance inside the source JSONL.
- `event_hash`: idempotency key for retries, compacts, and duplicate tails.
- `metadata_json`: structured adapter-specific details.

`session_conversations` remains as a compatibility and summary table. For large
sessions it should not be the primary transcript storage and should not rewrite a
giant JSON array on every append.

## Startup And Review Flow

1. Resolve CTM tab id to one or more agent session ids.
2. Resolve agent session ids to provider JSONL paths.
3. For each file, compare `transcript_files` cursor with current stat.
4. If unchanged, read nothing.
5. If grown, read only from saved `offset`.
6. If unknown and large, read a bounded tail and enqueue full background import.
7. If shrunk or identity changed, treat as rotate/compact and reconcile by hash.
8. Serve Review/Conversation from `transcript_events` plus hot stream tail.
9. Resume and terminal input must not wait for any full transcript hydration.
10. Serve session search from DB indexes only: `session_messages_fts`,
    `transcript_events_fts`, metadata rows, and optional semantic vectors.

## Correctness Requirements

- Watcher events are hints, not proof. Use `fs.watch` for low-latency invalidation
  and periodic stat reconciliation for correctness.
- Every parser must handle partial lines and invalid rows without losing the
  cursor for already-complete lines.
- Every ingest run must be idempotent.
- A stale large-session cache must attempt bounded on-demand catch-up before it
  is returned to the UI.
- Background backfill must be chunked and budgeted so it cannot burn CPU or block
  CTM input.
- Telemetry should distinguish driver/read failures, parser failures, stale
  cache fallback, rotate detection, and successful tail catch-up.

## Test Matrix

- A large Codex rollout with stale `session_conversations` catches up the final
  user/assistant turn without full parsing the file.
- A restart with unchanged cursor performs no JSONL read before Review renders.
- A grown file reads only appended bytes and preserves partial trailing lines.
- A compacted Claude pair preserves pre-compact and post-compact messages.
- A file shrink or inode change resets safely and logs telemetry.
- A cold large file shows a bounded recent tail and queues background backfill.
- Message API pagination reads DB rows and hot tail, not the whole JSONL.
- `/api/sessions/search` never scans provider JSONL files; keyword hits over
  freshly tailed provider output come from `transcript_events_fts`.

## Implemented Verification

Automated coverage:

- `tests/transcript-store.test.js` covers append-only cursor reads, partial-line
  carry, bounded cold-tail reads, Codex task-complete rows, shrink/reset
  recovery, transcript-event FTS indexing, and stale-cache merge behavior.
- `tests/codex-conversation-import.test.js` asserts that Codex JSONL imports now
  populate both legacy `session_conversations`/`session_messages` and the new
  `transcript_files`/`transcript_events` cursor path.

Manual isolated API check:

- Created a temporary CTM data dir and a 73MB Codex rollout with stale cached
  history plus new final tail rows.
- Started CTM on an isolated localhost port with Wall-E disabled.
- Requested `/api/session/messages?id=<ctm>&offset=0&limit=20`.
- Verified the response source was `conversation-cache+transcript-events-tail`,
  the latest user/assistant rows came from `transcript-events`, and only the
  configured 10MB tail window was read.
