# NexusCLI Database Schema - SQLite

**Version**: 0.1.0
**Created**: 2025-11-17
**Database**: SQLite (better-sqlite3)
**Philosophy**: Minimal, portable, chat-friendly

---

## 🎯 Why SQLite?

### Alignment with Three Pillars

**1. Lightweight** ✅
- Single dependency: `better-sqlite3` (~500KB native)
- No server process (embedded)
- Single file database (< 10MB for 1000s of messages)

**2. Portable (Linux + Termux)** ✅
- Works on x86_64 and aarch64 (Termux)
- Pre-compiled binaries for Android
- Single `.db` file (easy backup)

**3. ChatGPT UI** ✅
- Perfect for conversation history
- Fast full-text search (`FTS5`)
- Efficient pagination

### Alternatives Considered

| Option | Pros | Cons | Verdict |
|--------|------|------|---------|
| **SQLite** | Lightweight, portable, SQL | Native dependency | ✅ **CHOSEN** |
| JSON files | Zero deps, simple | Slow search, no relations | ❌ Not scalable |
| MongoDB | Powerful | Heavy, not Termux-friendly | ❌ Too heavy |
| LevelDB | Fast K-V store | No SQL, harder queries | ❌ Not chat-friendly |

---

## 📦 Installation

### Dependency

```json
// backend/package.json
{
  "dependencies": {
    "express": "^4.18.2",
    "cors": "^2.8.5",
    "uuid": "^9.0.0",
    "better-sqlite3": "^9.2.2"  // ← Only new dependency
  }
}
```

**Total dependencies**: 4 packages (still < 10 ✅)

### Setup (Linux)

```bash
cd backend
npm install better-sqlite3
# Native compilation (uses node-gyp)
```

### Setup (Termux)

```bash
# Termux requires build tools
pkg install clang make python

# Install better-sqlite3 (compiles for aarch64)
npm install better-sqlite3
```

**Note**: Pre-built binaries usually available, compilation is fallback.

---

## 🗄️ Schema Design

### Database File Location

**Linux**:
```
/var/lib/nexuscli/nexuscli.db
```

**Termux**:
```
/data/data/com.termux/files/home/.nexuscli/nexuscli.db
```

**Code**:
```javascript
// config.js
const isTermux = process.env.PREFIX?.includes('com.termux');

const DB_PATH = isTermux
  ? `${process.env.HOME}/.nexuscli/nexuscli.db`
  : '/var/lib/nexuscli/nexuscli.db';
```

---

## 📊 Tables

### 1. `conversations`

Tracks chat sessions (like ChatGPT conversations).

```sql
CREATE TABLE conversations (
  id TEXT PRIMARY KEY,              -- UUID
  title TEXT NOT NULL,              -- "Deploy to Production"
  created_at INTEGER NOT NULL,      -- Unix timestamp (ms)
  updated_at INTEGER NOT NULL,      -- Unix timestamp (ms)
  metadata TEXT,                    -- JSON: {"tags": ["prod"], "archived": false}

  INDEX idx_conversations_updated_at ON conversations(updated_at DESC)
);
```

**Example Row**:
```json
{
  "id": "conv-abc123",
  "title": "Deploy NexusCLI to Production",
  "created_at": 1700000000000,
  "updated_at": 1700001000000,
  "metadata": "{\"tags\":[\"prod\",\"deploy\"],\"archived\":false}"
}
```

---

### 2. `messages`

Individual messages in conversations (user prompts + assistant responses).

```sql
CREATE TABLE messages (
  id TEXT PRIMARY KEY,              -- UUID
  conversation_id TEXT NOT NULL,    -- FK to conversations.id
  role TEXT NOT NULL,               -- 'user' | 'assistant' | 'system'
  content TEXT NOT NULL,            -- Markdown text
  created_at INTEGER NOT NULL,      -- Unix timestamp (ms)
  metadata TEXT,                    -- JSON: {"streaming": true, "tokens": 123}

  FOREIGN KEY (conversation_id) REFERENCES conversations(id) ON DELETE CASCADE,
  INDEX idx_messages_conversation_id ON messages(conversation_id),
  INDEX idx_messages_created_at ON messages(created_at ASC)
);
```

**Example Rows**:
```json
// User message
{
  "id": "msg-001",
  "conversation_id": "conv-abc123",
  "role": "user",
  "content": "Deploy to all production nodes",
  "created_at": 1700000000000,
  "metadata": null
}

// Assistant message
{
  "id": "msg-002",
  "conversation_id": "conv-abc123",
  "role": "assistant",
  "content": "Deploying to 3 nodes:\n- prod-001 ✅\n- prod-002 ✅\n- prod-003 ⚠️",
  "created_at": 1700000001000,
  "metadata": "{\"tokens\":45,\"duration\":1234}"
}
```

---

### 3. `jobs`

Job execution records (linked to messages).

```sql
CREATE TABLE jobs (
  id TEXT PRIMARY KEY,              -- Job ID (UUID)
  conversation_id TEXT,             -- FK to conversations.id (optional)
  message_id TEXT,                  -- FK to messages.id (optional)
  node_id TEXT NOT NULL,            -- Target node
  tool TEXT NOT NULL,               -- 'bash' | 'git' | 'docker'
  command TEXT NOT NULL,            -- Command executed
  status TEXT NOT NULL,             -- 'queued' | 'executing' | 'completed' | 'failed'
  exit_code INTEGER,                -- Exit code (NULL if not completed)
  stdout TEXT,                      -- Command stdout
  stderr TEXT,                      -- Command stderr
  duration INTEGER,                 -- Execution time (ms)
  created_at INTEGER NOT NULL,      -- Unix timestamp (ms)
  started_at INTEGER,               -- Unix timestamp (ms)
  completed_at INTEGER,             -- Unix timestamp (ms)

  FOREIGN KEY (conversation_id) REFERENCES conversations(id) ON DELETE SET NULL,
  FOREIGN KEY (message_id) REFERENCES messages(id) ON DELETE SET NULL,
  INDEX idx_jobs_conversation_id ON jobs(conversation_id),
  INDEX idx_jobs_status ON jobs(status),
  INDEX idx_jobs_created_at ON jobs(created_at DESC)
);
```

**Example Row**:
```json
{
  "id": "job-xyz",
  "conversation_id": "conv-abc123",
  "message_id": "msg-002",
  "node_id": "prod-001",
  "tool": "bash",
  "command": "systemctl restart nginx",
  "status": "completed",
  "exit_code": 0,
  "stdout": "nginx.service restarted",
  "stderr": "",
  "duration": 1234,
  "created_at": 1700000000000,
  "started_at": 1700000000100,
  "completed_at": 1700000001334
}
```

---

### 4. `nodes` (Optional - for multi-node setups)

Registered nodes.

```sql
CREATE TABLE nodes (
  id TEXT PRIMARY KEY,              -- Node ID
  hostname TEXT NOT NULL,           -- Node hostname
  ip_address TEXT,                  -- Node IP
  status TEXT NOT NULL,             -- 'online' | 'offline'
  capabilities TEXT,                -- JSON: {"tools": ["bash", "git"]}
  last_heartbeat INTEGER,           -- Unix timestamp (ms)
  created_at INTEGER NOT NULL,      -- Unix timestamp (ms)

  INDEX idx_nodes_status ON nodes(status)
);
```

---

## 🔍 Full-Text Search (Optional Enhancement)

For searching chat history.

```sql
-- Virtual FTS5 table for messages
CREATE VIRTUAL TABLE messages_fts USING fts5(
  content,
  conversation_id UNINDEXED,
  content='messages',
  content_rowid='rowid'
);

-- Triggers to keep FTS index synced
CREATE TRIGGER messages_fts_insert AFTER INSERT ON messages BEGIN
  INSERT INTO messages_fts(rowid, content, conversation_id)
  VALUES (new.rowid, new.content, new.conversation_id);
END;

CREATE TRIGGER messages_fts_delete AFTER DELETE ON messages BEGIN
  DELETE FROM messages_fts WHERE rowid = old.rowid;
END;

CREATE TRIGGER messages_fts_update AFTER UPDATE ON messages BEGIN
  UPDATE messages_fts SET content = new.content WHERE rowid = new.rowid;
END;
```

**Usage**:
```sql
-- Search messages for "nginx"
SELECT m.* FROM messages m
JOIN messages_fts fts ON fts.rowid = m.rowid
WHERE messages_fts MATCH 'nginx'
ORDER BY m.created_at DESC
LIMIT 20;
```

---

## 💾 Database Wrapper

### Initialization

```javascript
// backend/db.js
const Database = require('better-sqlite3');
const path = require('path');
const fs = require('fs');

const isTermux = process.env.PREFIX?.includes('com.termux');
const dbDir = isTermux
  ? path.join(process.env.HOME, '.nexuscli')
  : '/var/lib/nexuscli';

const dbPath = path.join(dbDir, 'nexuscli.db');

// Ensure directory exists
if (!fs.existsSync(dbDir)) {
  fs.mkdirSync(dbDir, { recursive: true });
}

// Open database (creates if doesn't exist)
const db = new Database(dbPath);

// Enable WAL mode for better concurrency
db.pragma('journal_mode = WAL');

// Enable foreign keys
db.pragma('foreign_keys = ON');

// Initialize schema
function initSchema() {
  db.exec(`
    CREATE TABLE IF NOT EXISTS conversations (
      id TEXT PRIMARY KEY,
      title TEXT NOT NULL,
      created_at INTEGER NOT NULL,
      updated_at INTEGER NOT NULL,
      metadata TEXT
    );

    CREATE INDEX IF NOT EXISTS idx_conversations_updated_at
    ON conversations(updated_at DESC);

    CREATE TABLE IF NOT EXISTS messages (
      id TEXT PRIMARY KEY,
      conversation_id TEXT NOT NULL,
      role TEXT NOT NULL,
      content TEXT NOT NULL,
      created_at INTEGER NOT NULL,
      metadata TEXT,
      FOREIGN KEY (conversation_id) REFERENCES conversations(id) ON DELETE CASCADE
    );

    CREATE INDEX IF NOT EXISTS idx_messages_conversation_id
    ON messages(conversation_id);

    CREATE INDEX IF NOT EXISTS idx_messages_created_at
    ON messages(created_at ASC);

    CREATE TABLE IF NOT EXISTS jobs (
      id TEXT PRIMARY KEY,
      conversation_id TEXT,
      message_id TEXT,
      node_id TEXT NOT NULL,
      tool TEXT NOT NULL,
      command TEXT NOT NULL,
      status TEXT NOT NULL,
      exit_code INTEGER,
      stdout TEXT,
      stderr TEXT,
      duration INTEGER,
      created_at INTEGER NOT NULL,
      started_at INTEGER,
      completed_at INTEGER,
      FOREIGN KEY (conversation_id) REFERENCES conversations(id) ON DELETE SET NULL,
      FOREIGN KEY (message_id) REFERENCES messages(id) ON DELETE SET NULL
    );

    CREATE INDEX IF NOT EXISTS idx_jobs_conversation_id
    ON jobs(conversation_id);

    CREATE INDEX IF NOT EXISTS idx_jobs_status
    ON jobs(status);

    CREATE INDEX IF NOT EXISTS idx_jobs_created_at
    ON jobs(created_at DESC);
  `);

  console.log(`✅ Database initialized: ${dbPath}`);
}

initSchema();

module.exports = db;
```

---

### CRUD Operations

```javascript
// backend/models/conversation.js
const db = require('../db');
const { v4: uuidv4 } = require('uuid');

class Conversation {
  // Create new conversation
  static create(title) {
    const id = uuidv4();
    const now = Date.now();

    const stmt = db.prepare(`
      INSERT INTO conversations (id, title, created_at, updated_at)
      VALUES (?, ?, ?, ?)
    `);

    stmt.run(id, title, now, now);

    return { id, title, created_at: now, updated_at: now };
  }

  // Get conversation by ID
  static getById(id) {
    const stmt = db.prepare('SELECT * FROM conversations WHERE id = ?');
    return stmt.get(id);
  }

  // List recent conversations
  static listRecent(limit = 20) {
    const stmt = db.prepare(`
      SELECT * FROM conversations
      ORDER BY updated_at DESC
      LIMIT ?
    `);
    return stmt.all(limit);
  }

  // Update conversation title
  static updateTitle(id, title) {
    const stmt = db.prepare(`
      UPDATE conversations
      SET title = ?, updated_at = ?
      WHERE id = ?
    `);
    stmt.run(title, Date.now(), id);
  }

  // Delete conversation (cascade deletes messages)
  static delete(id) {
    const stmt = db.prepare('DELETE FROM conversations WHERE id = ?');
    stmt.run(id);
  }
}

module.exports = Conversation;
```

```javascript
// backend/models/message.js
const db = require('../db');
const { v4: uuidv4 } = require('uuid');

class Message {
  // Add message to conversation
  static create(conversationId, role, content, metadata = null) {
    const id = uuidv4();
    const now = Date.now();

    const stmt = db.prepare(`
      INSERT INTO messages (id, conversation_id, role, content, created_at, metadata)
      VALUES (?, ?, ?, ?, ?, ?)
    `);

    stmt.run(id, conversationId, role, content, now,
             metadata ? JSON.stringify(metadata) : null);

    // Update conversation updated_at
    const updateConv = db.prepare(`
      UPDATE conversations SET updated_at = ? WHERE id = ?
    `);
    updateConv.run(now, conversationId);

    return { id, conversation_id: conversationId, role, content, created_at: now };
  }

  // Get messages for conversation
  static getByConversation(conversationId, limit = 100) {
    const stmt = db.prepare(`
      SELECT * FROM messages
      WHERE conversation_id = ?
      ORDER BY created_at ASC
      LIMIT ?
    `);
    return stmt.all(conversationId, limit);
  }

  // Search messages (if FTS enabled)
  static search(query, limit = 20) {
    const stmt = db.prepare(`
      SELECT m.* FROM messages m
      JOIN messages_fts fts ON fts.rowid = m.rowid
      WHERE messages_fts MATCH ?
      ORDER BY m.created_at DESC
      LIMIT ?
    `);
    return stmt.all(query, limit);
  }
}

module.exports = Message;
```

---

## 🔄 Migration Strategy

### Initial Setup (v0.1.0)

```javascript
// backend/migrations/001-initial.js
module.exports = {
  up(db) {
    db.exec(`
      CREATE TABLE conversations (...);
      CREATE TABLE messages (...);
      CREATE TABLE jobs (...);
    `);
  },

  down(db) {
    db.exec(`
      DROP TABLE IF EXISTS jobs;
      DROP TABLE IF EXISTS messages;
      DROP TABLE IF EXISTS conversations;
    `);
  }
};
```

### Migration Runner

```javascript
// backend/migrate.js
const db = require('./db');
const fs = require('fs');
const path = require('path');

function getCurrentVersion() {
  try {
    const row = db.prepare('SELECT value FROM schema_version WHERE key = "version"').get();
    return parseInt(row.value);
  } catch {
    // First run
    db.exec('CREATE TABLE IF NOT EXISTS schema_version (key TEXT PRIMARY KEY, value TEXT)');
    db.prepare('INSERT INTO schema_version (key, value) VALUES ("version", "0")').run();
    return 0;
  }
}

function runMigrations() {
  const currentVersion = getCurrentVersion();
  const migrationsDir = path.join(__dirname, 'migrations');

  const migrations = fs.readdirSync(migrationsDir)
    .filter(f => f.endsWith('.js'))
    .sort();

  for (const file of migrations) {
    const migration = require(path.join(migrationsDir, file));
    const version = parseInt(file.split('-')[0]);

    if (version > currentVersion) {
      console.log(`Running migration ${file}...`);
      migration.up(db);
      db.prepare('UPDATE schema_version SET value = ? WHERE key = "version"').run(version);
    }
  }

  console.log('✅ Migrations complete');
}

runMigrations();
```

---

## 📈 Performance Optimization

### WAL Mode (Write-Ahead Logging)

```javascript
// Already enabled in db.js
db.pragma('journal_mode = WAL');
```

**Benefits**:
- Concurrent reads while writing
- Faster writes
- Better crash recovery

### Prepared Statements

```javascript
// ✅ GOOD: Reuse prepared statement
const stmt = db.prepare('SELECT * FROM messages WHERE conversation_id = ?');
const messages = stmt.all(conversationId);

// ❌ BAD: Parse SQL every time
db.prepare('SELECT * FROM messages WHERE conversation_id = ?').all(conversationId);
```

### Indexes

All critical queries have indexes:
- `conversations.updated_at` (recent list)
- `messages.conversation_id` (get messages)
- `messages.created_at` (chronological order)
- `jobs.status` (filter by status)

---

## 💾 Backup & Recovery

### Backup Script

```bash
#!/bin/bash
# backup-db.sh

DB_PATH="/var/lib/nexuscli/nexuscli.db"
BACKUP_DIR="/var/backups/nexuscli"
TIMESTAMP=$(date +%Y%m%d_%H%M%S)

mkdir -p "$BACKUP_DIR"

# SQLite backup (safe even with active connections)
sqlite3 "$DB_PATH" ".backup $BACKUP_DIR/nexuscli-$TIMESTAMP.db"

# Keep last 30 days
find "$BACKUP_DIR" -name "nexuscli-*.db" -mtime +30 -delete

echo "✅ Backup complete: $BACKUP_DIR/nexuscli-$TIMESTAMP.db"
```

### Restore

```bash
# Stop NexusCLI
systemctl stop nexuscli

# Restore backup
cp /var/backups/nexuscli/nexuscli-20251117.db /var/lib/nexuscli/nexuscli.db

# Start NexusCLI
systemctl start nexuscli
```

---

## 🎯 API Integration

### REST Endpoints

```javascript
// backend/routes/conversations.js
const express = require('express');
const Conversation = require('../models/conversation');
const Message = require('../models/message');

const router = express.Router();

// GET /api/v1/conversations - List recent
router.get('/', (req, res) => {
  const conversations = Conversation.listRecent(20);
  res.json({ conversations });
});

// POST /api/v1/conversations - Create new
router.post('/', (req, res) => {
  const { title } = req.body;
  const conversation = Conversation.create(title || 'New Conversation');
  res.status(201).json(conversation);
});

// GET /api/v1/conversations/:id - Get conversation with messages
router.get('/:id', (req, res) => {
  const conversation = Conversation.getById(req.params.id);
  if (!conversation) {
    return res.status(404).json({ error: 'Conversation not found' });
  }

  const messages = Message.getByConversation(req.params.id);
  res.json({ ...conversation, messages });
});

// POST /api/v1/conversations/:id/messages - Add message
router.post('/:id/messages', (req, res) => {
  const { role, content } = req.body;
  const message = Message.create(req.params.id, role, content);
  res.status(201).json(message);
});

// DELETE /api/v1/conversations/:id - Delete conversation
router.delete('/:id', (req, res) => {
  Conversation.delete(req.params.id);
  res.json({ success: true });
});

module.exports = router;
```

---

## 🧪 Testing

### Unit Tests (SQLite in-memory)

```javascript
// backend/tests/conversation.test.js
const Database = require('better-sqlite3');
const Conversation = require('../models/conversation');

// Use in-memory DB for tests
const testDb = new Database(':memory:');

beforeEach(() => {
  // Initialize schema
  testDb.exec(`CREATE TABLE conversations (...)`);
});

test('create conversation', () => {
  const conv = Conversation.create('Test Conversation');
  expect(conv.title).toBe('Test Conversation');
  expect(conv.id).toBeTruthy();
});

test('list recent conversations', () => {
  Conversation.create('Conv 1');
  Conversation.create('Conv 2');

  const list = Conversation.listRecent();
  expect(list.length).toBe(2);
  expect(list[0].title).toBe('Conv 2'); // Most recent first
});
```

---

## 📊 Size Estimates

| Data | Rows | Size per Row | Total Size |
|------|------|--------------|------------|
| Conversations | 100 | ~200 bytes | ~20 KB |
| Messages | 10,000 | ~500 bytes | ~5 MB |
| Jobs | 5,000 | ~1 KB | ~5 MB |
| **Total** | - | - | **~10 MB** |

**Conclusion**: Even with heavy usage, DB stays < 50MB (lightweight ✅)

---

## 🎯 Alignment with Three Pillars

### 1. Lightweight ✅
- Single dependency: `better-sqlite3`
- Embedded DB (no server)
- WAL mode (< 10MB typical size)

### 2. Portable ✅
- Works on Linux x86_64
- Works on Termux aarch64
- Single `.db` file (easy sync)

### 3. ChatGPT UI ✅
- Perfect for conversation history
- Fast message retrieval
- Full-text search support

---

## 📚 Related Documents

- [Design Principles](./DESIGN_PRINCIPLES.md) - Three pillars
- [Architecture Overview](./ARCHITECTURE.md) - System design
- [API Contract](./API_WRAPPER_CONTRACT.md) - REST API spec

---

_Database design for NexusCLI - Lightweight, Portable, Chat-Friendly_
_Generated by Claude Code (Sonnet 4.5) - 2025-11-17_
