# Smart Session Search Design

Date: 2026-05-15
Project: `claude-task-manager`
Status: Implemented

## Goal

Make the Recent Sessions search box find the session the user means without
requiring the user to choose between keyword search and "AI search".

The default search should combine fast local matching, full-corpus lexical
search, message search, identifier matching, metadata recovery, and semantic
search when an embedding index is available. The UI should explain why a result
matched instead of exposing backend implementation modes.

## Problem

The old search model exposed an `AI` button next to the Recent Sessions search
box. That path was not actually a reliable retrieval system:

- The frontend told users it was "Searching with Claude".
- `/api/sessions/ai-search` spawned `claude -p` from the web request path.
- The endpoint passed only the first 200 session summaries to the model.
- It blocked the Node process while the child process ran.
- It could create provider-session history pollution.
- It returned unproven session objects without first-class match provenance.

This made search slow, opaque, provider-specific, and hard to trust.

## Product Decision

Remove the visible AI-search mode from the main sidebar. Search is smart by
default.

Users should see one search box and one ranked result list. CTM can still use
multiple retrieval strategies under the hood, but the UI should surface:

- result count
- match source, such as `Title`, `Message`, `Semantic`, `Session ID`, or
  `Project`
- snippets when the result came from a summary or message
- optional status text when a slower source is unavailable or still indexing

Advanced controls for source toggles or index rebuilds belong in setup/debug
surfaces, not beside the primary search input.

## Search Contract

### Default Behavior

1. On each input event, run immediate local filtering over the loaded sidebar
   list and open active tabs.
2. After a debounce, call `/api/sessions/search`.
3. The server searches the full corpus and returns ranked candidates.
4. The client merges server candidates with local active-tab candidates so live
   tabs do not vanish when the server response arrives.
5. Results are deduped by CTM session id and linked provider session ids.

### Sources

The unified server search uses these sources:

- `session_analyses_fts`: title, summary, topics, category, skills, first
  message, project.
- `session_messages_fts`: extracted user/assistant message content.
- `transcript_events_fts`: provider append-tail rows indexed by the transcript
  importer. This covers Codex/Claude/Wall-E rows before a full conversation cache
  refresh completes.
- Direct id matches: CTM ids and provider agent session ids.
- Metadata recovery: title, first message, project, branch, and provider
  metadata.
- Semantic vectors: optional embeddings over session analysis rows for
  descriptive queries, when the embedding subsystem is available.

### Ranking

The merge layer should use reciprocal rank fusion style scoring across ranked
sources. Exact identifiers and exact titles receive explicit boosts so they
outrank incidental content matches.

Semantic matches must not replace lexical search. They are additive and must
carry `_matchField: "semantic"` plus a snippet/reason so users can understand
why the row appeared.

### Descriptive Queries

The hard case is a user description rather than a keyword, for example:

- "the session where rendering broke after switching tabs"
- "tax return planning from a couple months ago"
- "the prompt dropdown was out of date"
- "the work about phone escape key"

Keyword FTS should still contribute, but semantic vectors and message-level
search should make these queries work even when the exact title does not contain
the same words.

### LLM Reranking

LLM reranking is optional and must not be the default dependency for search.
If added later:

- call the configured background provider, not a provider CLI
- rerank only the top 20-30 retrieval candidates
- include session ids, source snippets, and timestamps
- timeout quickly and fall back to fused retrieval
- never create a visible coding-agent session

## API Contract

`GET /api/sessions/search?q=...&limit=...` is the primary endpoint.

The response may include:

```json
{
  "results": [],
  "total": 0,
  "mode": "smart",
  "sources": {
    "analysisFts": 0,
    "messageFts": 0,
    "ids": 0,
    "metadata": 0,
    "semantic": 0
  },
  "semantic": {
    "attempted": false,
    "available": false,
    "model": null,
    "error": null
  }
}
```

`POST /api/sessions/ai-search` remains only as a compatibility shim. It should
delegate to the same search implementation and must not spawn a model CLI.

## UI Contract

The Recent Sessions sidebar should show:

- one search input
- no `AI` checkbox/button
- clear count badge
- match context rows for full-corpus results
- no provider-specific loading copy

Recommended loading/status copy:

- "Searching sessions..."
- "Semantic search unavailable: configure embeddings" only in setup/debug
  context, not as an intrusive empty state.

## Privacy And Reliability

Search must stay local-first:

- FTS, metadata, id, and message search run against CTM's local DB and local
  transcript indexes. Provider JSONL files are never scanned by
  `/api/sessions/search`; import/watch/repair jobs are the only components that
  read them.
- Embeddings use the configured embedding provider and existing Wall-E/CTM
  embedding storage.
- No search query should create a Claude/Codex/Wall-E coding session.
- No search result should depend on an external model call unless optional
  reranking is explicitly enabled.

## Test Plan

Unit tests should cover:

- `POST /api/sessions/ai-search` compatibility no longer shells out.
- Descriptive search query expansion finds sessions whose title does not contain
  every user word.
- Result source counts and semantic status are present.
- Existing id/title/message search rankings still hold.

Rendering tests should cover:

- the Recent Sessions sidebar has no visible AI button
- typing a description calls the unified search endpoint
- result rows show source/provenance labels

Real-life validation should cover:

- a keyword query such as `tax`
- a descriptive query such as `session where rendering broke after switching tabs`
- an id prefix
- an active renamed tab
- no new `You are searching through Claude Code session history` rows appear
  after using search

## Migration Phases

1. Document the product/search contract and remove obsolete user-facing filters.
2. Replace `/api/sessions/ai-search` with a compatibility shim over unified
   search.
3. Enable smart search by default in the Recent Sessions sidebar and remove the
   AI toggle.
4. Add tests and run live validation against realistic descriptive queries.

## Implementation Notes

- The main Recent Sessions sidebar has one search input. There is no visible
  AI toggle for session search.
- `/api/sessions/search` is the source of truth for full-corpus smart search.
- `/api/sessions/ai-search` remains as a compatibility endpoint and delegates
  to the same implementation.
- The legacy provider-CLI request path was removed; session search no longer
  shells out to `claude -p`.
- Descriptive queries use additive FTS expansion and semantic vector search
  when the local embedding index is available.
- Semantic search is skipped for keyword/id searches, used only as a fallback
  for descriptive searches with too few cheaper matches, concurrency-gated, and
  briefly cooled down after provider timeout so normal keyword/id/message search
  stays responsive.
- Search request handling is DB-only. `session_messages_fts` is the primary
  imported-message index; `transcript_events_fts` is the append-tail safety net
  for provider rows that have been ingested but not yet folded into
  `session_conversations`/`session_messages`.
