# Session Search Corpus Architecture

Date: 2026-05-18
Status: Implemented

## Goal

CTM session search must be fast enough for keystroke-driven desktop and phone
UX while still searching the complete durable session corpus, especially full
user prompts.

The important distinction is storage contract:

- `session_conversations.messages` is a compact conversation cache for review
  and title/status projections.
- `session_messages.content` is the canonical row-level searchable text store.
- `session_messages_fts` is an index over `session_messages`, not a second text
  store.

This lets the UI stay responsive without throwing away prompt text needed for
search, later export, or training workflows.

## Current Problem

Claude import capped each imported user/assistant message with
`text.slice(0, 5000)`. Message FTS then capped indexed content again to the
first 2000 characters. That made sense when the data path was a lightweight UI
cache, but it breaks corpus-level search:

- long pasted prompts are truncated before they reach the DB;
- FTS cannot match terms after the first 2000 characters;
- phone and desktop search share `/api/sessions/search`, so both inherit the
  same missing matches;
- removing caps blindly would risk large review payloads and event-loop stalls.

## Design

### One Canonical Text Store

Do not add a separate prompt corpus table yet. `session_messages.content`
already stores one row per user/assistant message and is refreshed by the same
import path that updates `session_conversations`.

The fix is to make that existing table canonical:

- store full extracted text in `session_messages.content`;
- keep `session_conversations.messages` compact for UI compatibility;
- pass full `search_messages` from import parsers into `importSessionConversation`;
- preserve existing rows during append-only incremental imports.

This avoids duplicating full prompt text in a second table.

### Index Without Full-Text Duplication

`session_messages_fts` remains an FTS5 virtual table with `content=''` and
`contentless_delete=1`. It stores the inverted index, not another copy of
message text. The FTS rowid maps to `session_messages.id`.

Index the full message text instead of the first 2000 characters. Result
snippets are generated from the canonical `session_messages.content` after FTS
returns a small ranked row set.

This follows SQLite FTS5's separation between external/contentless indexes and
the application-owned content table.

### Responsive Search

The unified search endpoint should stay cheap:

1. Normalize the query and run local DB-backed lexical sources first.
2. Use `session_messages_fts` for full-message matching.
3. Search metadata from DB rows, not by scanning/parsing every JSONL file in
   the HTTP request path.
4. Wake the background conversation importer when search runs so stale files are
   picked up incrementally.
5. Only use semantic search when lexical/id/metadata results are insufficient.

The request path should never parse a large JSONL corpus just because a user
typed into a search box.

### Incremental Updates

For append-only JSONL imports:

- read only the newly appended bytes;
- parse new full `search_messages` alongside compact UI messages;
- combine those with existing `session_messages` rows before refreshing the
  index;
- retain full historical content that was already indexed.

For full reimports or compact pairs:

- rebuild that session's `session_messages` rows from the complete imported
  transcript;
- keep the conversation cache compact.

### Search Query Classes

The same architecture supports:

- exact keywords: `prompt dropdown`
- descriptive recovery: `the session where the prompt dropdown was out of date`
- long prompt tail terms that appear after 5000 characters
- ids and branch/path metadata
- phone home/search and desktop recent-session search through the same API

## Non-Goals

- Do not add vector embeddings for every message in this phase.
- Do not store a duplicate full prompt table unless we later need branch-aware
  prompt versioning or training-export metadata that `session_messages` cannot
  represent cleanly.
- Do not remove compact UI projections; large transcripts still need compact
  render paths.

## References

- SQLite FTS5 external/contentless tables:
  https://www.sqlite.org/fts5.html
- SQLite write-ahead logging:
  https://www.sqlite.org/wal.html
