# Conversation Import Freshness

CTM has two transcript paths:

- Hot path: terminal/session-stream events used for the active terminal view and
  recent live deltas.
- Warm path: `session_conversations` and `session_messages`, imported from
  provider JSONL files for Conversation, Review, prompts, and search.

The warm path is the durable source for historical conversation views. It must
catch up asynchronously; HTTP request handlers must not parse multi-GB provider
logs synchronously.

## Large Codex Sessions

Codex rollout files can grow past 1GB because they contain tool payloads and
runtime events. `/api/session/messages` therefore has a stale-cache fast path:
when a cached conversation exists but the backing files are too large for a
safe inline parse, the API returns the stale page with `stale: true` and wakes
the conversation importer.

That wake is only a freshness signal. The importer is responsible for reading
the append range from the prior cached file size and refreshing both:

- `session_conversations.messages`
- `session_messages` / `session_messages_fts`

The request path may do bounded tail repair only when it is strictly cheaper
than a full parse and only to improve the latest visible answer. For Codex, this
means reading the latest rollout tail, normalizing it through the same Codex
message parser used by the importer, and overlaying those semantic messages on
top of the stale cache. It must not invent UI-only transcript rows.

Do not "fix" stale large sessions by raising the synchronous parse threshold or
by parsing the entire Codex JSONL inside the request path.

## Provider Event Semantics

Codex rollouts are provider-owned replay logs. A `response_item` with
`role:"user"` can be a real prompt, provider context, or lifecycle control
state that Codex wants in its model-visible resume history. CTM's normalized
conversation corpus must keep only human prompts:

- `event_msg.user_message` and human `response_item` messages become CTM user
  messages.
- Provider context envelopes such as environment, permissions, skills, plugins,
  and AGENTS instructions are excluded.
- Control envelopes such as `<turn_aborted>...</turn_aborted>` are excluded from
  Conversation, Review, prompt navigation, search, and phone timelines.

This filter belongs at the parser and `/api/session/messages` boundaries. UI
surfaces can keep a defensive check for already-cached old rows, but desktop and
phone should both converge on the normalized API result.

## Codex User Echo Dedupe

Codex can record the same human prompt twice for one turn: once as a
`response_item` user message and again as an `event_msg.user_message`. CTM must
collapse that provider echo so Conversation, Review, search, and phone timelines
do not show duplicate user turns.

The dedupe key is **not** globally unique text. Long-lived Codex sessions often
repeat the same prompt days later, for example rerunning the same research skill
after follow-up design work. Those later prompts are distinct human turns and
must remain visible. The importer therefore dedupes only same-text user events
inside a short timestamp window. If the timestamp is missing, CTM falls back to
conservative same-text dedupe because it cannot prove the events are separate.

The Codex conversation importer records a parser version on
`session_conversations`. When this semantic contract changes, same-size Codex
caches with an older parser version must be eligible for full re-import; file
size alone is not enough to prove a cache is fresh. This avoids one-off DB edits
for historical sessions whose raw rollout still contains the correct prompt.

## Non-Starving Scheduling

Background DB work can defer briefly while the user is actively interacting
with terminals. Deferral must not become starvation.

The `conversation-import` job returns `retry_after_ms` when it skips because of
recent interactive activity or when it pauses after a bounded batch. The
scheduler honors that retry hint instead of waiting the normal 10 minute
interval. This keeps active sessions fresh without making request handlers do
heavy JSONL work.

## Candidate Priority

Each importer run scans known session files and builds a candidate queue. The
queue is prioritized as:

1. Existing conversation cache is behind the source file size.
2. Linked agent session has no durable conversation cache yet.
3. Compact-pair files changed and need merge repair.
4. Other files changed since the last completed scan.
5. Existing cache is missing metadata such as model provider.

The global `conversation_import_last_scan_at` is an optimization cursor, not
the source of truth. A cache whose stored file size is behind its source file
must be eligible even if the global scan cursor has advanced.

## Operational Checks

For a stale Conversation tab, compare:

- provider JSONL `mtime` and size
- `session_conversations.file_size` and `imported_at`
- `session_messages` max timestamp
- `/api/session/messages` `source`, `stale`, and `cacheFresh`

If the provider JSONL is current but the DB cache is stale, the fix belongs in
the importer/scheduler path, not in xterm rendering or tab switching.
