# Memory System Upgrade Plan

**Goal**: Upgrade OpenCodeKit memory from file-based markdown to hybrid SQLite + FTS5 with progressive disclosure.

**Key Benefits**:

1. **10x token savings** via progressive disclosure (compact index → timeline → full details)
2. **Fast full-text search** via SQLite FTS5 (vs O(n) regex scan)
3. **Richer schema** with facts, narrative, files_read/modified distinction
4. **Backward compatible** - existing tools continue working, markdown remains human-readable

---

## Phase 1: Database Foundation

### 1.1 Create SQLite Module

**File**: `.opencode/plugin/lib/memory-db.ts`

```typescript
// Core database class using bun:sqlite
import { Database } from "bun:sqlite";

const DB_PATH = path.join(process.cwd(), ".opencode/memory.db");

export function getMemoryDB(): Database {
  const db = new Database(DB_PATH, { create: true });
  db.run("PRAGMA journal_mode = WAL"); // Better concurrency
  db.run("PRAGMA foreign_keys = ON");
  initializeSchema(db);
  return db;
}
```

### 1.2 Database Schema

**File**: `.opencode/plugin/lib/memory-schema.ts`

**Tables**:

- `schema_versions` - Migration tracking
- `observations` - Enhanced observation records
- `observations_fts` - FTS5 virtual table
- `memory_files` - Non-observation memory files

**Schema**:

```sql
CREATE TABLE IF NOT EXISTS observations (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  type TEXT NOT NULL CHECK(type IN ('decision','bugfix','feature','pattern','discovery','learning','warning')),
  title TEXT NOT NULL,
  subtitle TEXT,
  facts TEXT,           -- JSON array
  narrative TEXT,       -- Long-form content
  concepts TEXT,        -- JSON array
  files_read TEXT,      -- JSON array
  files_modified TEXT,  -- JSON array
  confidence TEXT CHECK(confidence IN ('high','medium','low')) DEFAULT 'high',
  bead_id TEXT,
  supersedes INTEGER,
  superseded_by INTEGER,
  valid_until TEXT,
  markdown_file TEXT,   -- Reference to backup markdown file
  created_at TEXT NOT NULL,
  created_at_epoch INTEGER NOT NULL,
  updated_at TEXT
);

-- FTS5 for full-text search
CREATE VIRTUAL TABLE IF NOT EXISTS observations_fts USING fts5(
  title, subtitle, narrative, facts, concepts,
  content='observations', content_rowid='id'
);

-- Auto-sync triggers for FTS5
CREATE TRIGGER observations_fts_ai AFTER INSERT ON observations BEGIN
  INSERT INTO observations_fts(rowid, title, subtitle, narrative, facts, concepts)
  VALUES (new.id, new.title, new.subtitle, new.narrative, new.facts, new.concepts);
END;
-- (plus UPDATE and DELETE triggers)
```

---

## Phase 2: Progressive Disclosure Tools

### 2.1 Update `memory-search.ts` - Compact Index

**File**: `.opencode/tool/memory-search.ts`

**Change**: Return compact index instead of full content.

```typescript
// NEW: Returns compact index only (~50-100 tokens per result)
interface SearchIndexResult {
  id: number;
  type: string;
  title: string;
  snippet: string; // First 100 chars of narrative
  created_at: string;
  relevance_score: number;
}

execute: async (args) => {
  const db = getMemoryDB();

  // Check if FTS5 available (bun:sqlite supports it)
  const useFTS = checkFTS5Available(db);

  if (useFTS && args.query) {
    // Use FTS5 for text search with ranking
    const results = db
      .query(
        `
      SELECT o.id, o.type, o.title, 
             substr(o.narrative, 1, 100) as snippet,
             o.created_at,
             bm25(observations_fts) as relevance_score
      FROM observations o
      JOIN observations_fts fts ON fts.rowid = o.id
      WHERE observations_fts MATCH ?
      ORDER BY relevance_score
      LIMIT ?
    `,
      )
      .all(args.query, args.limit || 10);
    return formatCompactIndex(results);
  }

  // Fallback to regex search (backward compat)
  return fallbackKeywordSearch(args);
};
```

### 2.2 Create `memory-timeline.ts` - Chronological Context

**File**: `.opencode/tool/memory-timeline.ts` (NEW)

```typescript
// Get observations around an anchor point
execute: async (args) => {
  const db = getMemoryDB();
  const anchor = db
    .query(
      `
    SELECT created_at_epoch FROM observations WHERE id = ?
  `,
    )
    .get(args.anchor_id);

  const before = db
    .query(
      `
    SELECT id, type, title, substr(narrative, 1, 100) as snippet
    FROM observations
    WHERE created_at_epoch < ?
    ORDER BY created_at_epoch DESC
    LIMIT ?
  `,
    )
    .all(anchor.created_at_epoch, args.depth_before || 5);

  const after = db
    .query(
      `
    SELECT id, type, title, substr(narrative, 1, 100) as snippet
    FROM observations
    WHERE created_at_epoch > ?
    ORDER BY created_at_epoch ASC
    LIMIT ?
  `,
    )
    .all(anchor.created_at_epoch, args.depth_after || 5);

  return { anchor: args.anchor_id, before: before.reverse(), after };
};
```

### 2.3 Create `memory-get.ts` - Full Details

**File**: `.opencode/tool/memory-get.ts` (NEW)

```typescript
// Get full observation details by ID(s)
execute: async (args) => {
  const db = getMemoryDB();
  const ids = args.ids.split(",").map(Number);

  const observations = db
    .query(
      `
    SELECT * FROM observations WHERE id IN (${ids.join(",")})
  `,
    )
    .all();

  return observations.map(formatFullObservation);
};
```

---

## Phase 3: Enhanced Observation Tool

### 3.1 Update `observation.ts` - Dual Write + Enhanced Schema

**File**: `.opencode/tool/observation.ts`

**New Args**:

```typescript
args: {
  type: string;
  title: string;
  subtitle?: string;        // NEW
  facts?: string;           // NEW: Comma-separated structured facts
  narrative?: string;       // NEW: Renamed from 'content'
  content?: string;         // DEPRECATED: Alias for narrative (backward compat)
  concepts?: string;
  files_read?: string;      // NEW: Files read during task
  files_modified?: string;  // NEW: Files changed during task
  files?: string;           // DEPRECATED: Alias for files_modified
  bead_id?: string;
  confidence?: string;
  supersedes?: string;
}
```

**Execute**:

```typescript
execute: async (args) => {
  const normalized = normalizeArgs(args); // Handle deprecated fields

  // 1. Write to SQLite
  const db = getMemoryDB();
  const result = db
    .query(
      `
    INSERT INTO observations (type, title, subtitle, facts, narrative, concepts, 
                              files_read, files_modified, confidence, bead_id,
                              markdown_file, created_at, created_at_epoch)
    VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
  `,
    )
    .run(/* ... */);

  // 2. Write to markdown (backup)
  const markdownPath = generateMarkdownPath(normalized);
  await fs.writeFile(markdownPath, renderToMarkdown(normalized, result.lastInsertRowid));

  return `✓ Observation #${result.lastInsertRowid} saved (SQLite + markdown backup)`;
};
```

---

## Phase 4: Migration Script

### 4.1 Create Migration Tool

**File**: `.opencode/tool/memory-migrate.ts` (NEW)

```typescript
execute: async () => {
  const obsDir = path.join(process.cwd(), ".opencode/memory/observations");
  const db = getMemoryDB();

  // Get all markdown files
  const files = await fs.readdir(obsDir);
  const mdFiles = files.filter((f) => f.endsWith(".md") && !f.startsWith("."));

  let migrated = 0,
    errors = [];

  for (const file of mdFiles) {
    try {
      const content = await fs.readFile(path.join(obsDir, file), "utf-8");
      const parsed = parseMarkdownObservation(content, file);

      // Insert into SQLite
      db.query(`INSERT INTO observations ...`).run(parsed);
      migrated++;
    } catch (e) {
      errors.push({ file, error: e.message });
    }
  }

  // Write migration marker
  await fs.writeFile(
    path.join(obsDir, ".migrated"),
    `Migrated ${migrated} observations on ${new Date().toISOString()}`,
  );

  return `✓ Migration complete: ${migrated} observations, ${errors.length} errors`;
};
```

---

## Phase 5: Update Existing Tools

### 5.1 Update `memory-read.ts`

Add SQLite lookup before file fallback:

```typescript
execute: async (args) => {
  // NEW: Check for observation by ID
  if (args.file?.match(/^#?\d+$/)) {
    const db = getMemoryDB();
    const id = parseInt(args.file.replace("#", ""));
    const obs = db.query("SELECT * FROM observations WHERE id = ?").get(id);
    if (obs) return formatFullObservation(obs);
  }

  // Existing file-based lookup (unchanged)
  return readMarkdownFile(args.file);
};
```

### 5.2 Update `memory-update.ts`

Add dual-write to SQLite:

```typescript
execute: async (args) => {
  const db = getMemoryDB();

  // Write to SQLite memory_files table
  db.query(
    `
    INSERT INTO memory_files (file_path, content, mode, created_at, created_at_epoch)
    VALUES (?, ?, ?, ?, ?)
    ON CONFLICT(file_path) DO UPDATE SET content = ?, updated_at = ?
  `,
  ).run(/* ... */);

  // Also write to markdown (unchanged)
  await writeMarkdownFile(args.file, args.content, args.mode);

  return `Updated ${args.file} (SQLite + markdown)`;
};
```

---

## Implementation Order

| Order | Task                      | Files                                   | Effort | Depends On |
| ----- | ------------------------- | --------------------------------------- | ------ | ---------- |
| 1     | Create memory-db.ts       | `.opencode/plugin/lib/memory-db.ts`     | 3h     | -          |
| 2     | Create memory-schema.ts   | `.opencode/plugin/lib/memory-schema.ts` | 2h     | 1          |
| 3     | Create memory-migrate.ts  | `.opencode/tool/memory-migrate.ts`      | 3h     | 1, 2       |
| 4     | Update observation.ts     | `.opencode/tool/observation.ts`         | 3h     | 1, 2       |
| 5     | Update memory-search.ts   | `.opencode/tool/memory-search.ts`       | 3h     | 1, 2       |
| 6     | Create memory-timeline.ts | `.opencode/tool/memory-timeline.ts`     | 2h     | 1, 2       |
| 7     | Create memory-get.ts      | `.opencode/tool/memory-get.ts`          | 2h     | 1, 2       |
| 8     | Update memory-read.ts     | `.opencode/tool/memory-read.ts`         | 1h     | 1, 2       |
| 9     | Update memory-update.ts   | `.opencode/tool/memory-update.ts`       | 1h     | 1, 2       |
| 10    | Update memory.ts plugin   | `.opencode/plugin/memory.ts`            | 2h     | 4, 5       |

**Total**: ~22 hours of implementation

---

## Files to Modify/Create

**NEW FILES**:

- `.opencode/plugin/lib/memory-db.ts` - SQLite database manager
- `.opencode/plugin/lib/memory-schema.ts` - Schema definitions + migrations
- `.opencode/tool/memory-migrate.ts` - Migration tool
- `.opencode/tool/memory-timeline.ts` - Timeline context tool
- `.opencode/tool/memory-get.ts` - Full observation retrieval

**MODIFY**:

- `.opencode/tool/observation.ts` - Dual write + enhanced schema
- `.opencode/tool/memory-search.ts` - FTS5 + compact index
- `.opencode/tool/memory-read.ts` - SQLite lookup + fallback
- `.opencode/tool/memory-update.ts` - Dual write
- `.opencode/plugin/memory.ts` - Add FTS5 optimization hook

---

## Verification

### Manual Testing

1. Run `memory-migrate` to import existing observations
2. Create new observation with `observation` tool
3. Search with `memory-search` and verify FTS5 is used
4. Use `memory-timeline` around a result
5. Use `memory-get` to fetch full details
6. Verify markdown backups are created
7. Verify existing tools still work (backward compat)

### Check Database

```bash
# Verify SQLite file created
ls -la .opencode/memory.db

# Verify FTS5 table
sqlite3 .opencode/memory.db "SELECT * FROM observations_fts LIMIT 5;"

# Verify observation count matches markdown files
sqlite3 .opencode/memory.db "SELECT COUNT(*) FROM observations;"
ls .opencode/memory/observations/*.md | wc -l
```

### Backward Compatibility

1. `memory-read({ file: "handoffs/2024-01-01" })` still works
2. `observation({ type: "decision", title: "X", content: "Y" })` still works (deprecated `content` field)
3. `memory-search({ query: "auth" })` returns results (may be faster with FTS5)
