================================================================================
TASK 1: DATA FLOW INVESTIGATION — Usage Monitoring System
Date: 2026-05-30
================================================================================

## EXECUTIVE SUMMARY

The usage monitoring system has TWO separate tables serving different purposes:
- `usageHistory` — legacy ops monitoring (provider-level, connection-based)
- `usage_records` — new billing system (user-level, API-key-based)

ROOT CAUSE: The write paths and read paths are split across these two tables,
causing the dashboard to show empty data depending on which tab/view is used.

================================================================================
## TABLE 1: `usageHistory` (Legacy Ops Monitoring)
================================================================================

### Schema (src/lib/db/schema.js:180-201)
  - id: INTEGER PRIMARY KEY AUTOINCREMENT
  - timestamp, provider, model, connectionId, apiKey, endpoint
  - promptTokens, completionTokens, cost, status
  - tokens (JSON), meta (JSON)

### Purpose
  Provider-level request logging for ops monitoring. Tracks every proxied
  LLM request with provider, model, connection, and token counts.

### WRITE PATH
  Written by: `saveRequestUsage()` in `src/lib/db/repos/usageRepo.js:243-287`

  Callers:
  1. `open-sse/utils/usageTracking.js:345` — `logUsage()` function
     → Called from `open-sse/utils/stream.js:289,359` (SSE proxy handler)
     → This is the MAIN write path for all proxied LLM requests

  2. `open-sse/handlers/chatCore/requestDetail.js:93` — `saveUsageStats()` function
     → Called from chat core request handler

### READ PATH
  Read by functions in `src/lib/db/repos/usageRepo.js`:
  - `getUsageStats()` (line 319) — reads usageHistory + usageDaily
  - `getChartData()` (line 620) — reads usageHistory + usageDaily
  - `getRecentLogs()` (line 701) — reads usageHistory
  - `getUsageHistory()` (line 289) — reads usageHistory

  Also read by `src/lib/db/repos/rateLimitRepo.js`:
  - `getMonthlyTokens()` (line 54) — reads usageHistory for rate limit checks
  - `getMonthlyCost()` (line 68) — reads usageHistory for credit checks
  - Rate limit RTPM/RTPD checks in `rateLimitMiddleware.js:78,97`

================================================================================
## TABLE 2: `usage_records` (New Billing System)
================================================================================

### Schema (migration 004 + schema.js:226-246)
  - id: TEXT PRIMARY KEY (UUID)
  - userId, apiKeyId, model
  - inputTokens, outputTokens, totalTokens
  - cost, creditUsed, status, errorMessage, endpoint, timestamp

### Purpose
  User-level billing records. Tracks usage per user/API-key for billing,
  rate limiting, and per-user usage dashboards.

### WRITE PATH
  Written by: `logUsage()` in `src/lib/db/repos/usageRecordsRepo.js:23-48`

  Callers:
  1. `src/lib/auth/rateLimitMiddleware.js:58-68` — `logUsageAsync()` function
     → Called from `enforceRateLimits()` (lines 215, 230)
     → Triggered when API key users make requests through the LLM API gate
     → Called from `src/dashboardGuard.js:239-240`

### READ PATH
  Read by functions in `src/lib/db/repos/usageRecordsRepo.js`:
  - `getUsageByUser()` (line 50) — reads usage_records by userId
  - `getUsageSummary()` (line 97) — aggregates usage_records by userId
  - `getUsageByModel()` (line 137) — groups usage_records by model
  - `getUsageDaily()` (line 173) — groups usage_records by date
  - `getAllUsersUsage()` (line 211) — aggregates across all users

================================================================================
## THE DISCONNECT — Root Cause Analysis
================================================================================

### Problem 1: Dashboard Overview Tab reads from WRONG table for SSE proxy data

  Dashboard "Overview" tab:
  → `UsageStats` component fetches `/api/usage/stats` + `/api/usage/stream`
  → These call `getUsageStats()` from `usageRepo.js`
  → Reads from `usageHistory` table
  → ✅ CORRECT — SSE proxy writes to `usageHistory` via `saveRequestUsage()`

### Problem 2: Dashboard Table Tab reads from WRONG table for SSE proxy data

  Dashboard "Table" tab:
  → `UsageTableView` component fetches `/api/usage`
  → This calls `getUsageByUser()` + `getUsageSummary()` from `usageRecordsRepo.js`
  → Reads from `usage_records` table
  → ❌ WRONG — SSE proxy NEVER writes to `usage_records`

  The SSE proxy path writes ONLY to `usageHistory` via `saveRequestUsage()`.
  The `usage_records` table is ONLY written by `rateLimitMiddleware.js`.
  So the Table tab shows EMPTY for all SSE proxy requests.

### Problem 3: Export reads from `usage_records` (same issue as Table tab)

  Export:
  → `/api/usage/export` calls `getUsageByUser()` + `getUsageSummary()` from `usageRecordsRepo.js`
  → Reads from `usage_records` table
  → ❌ WRONG — Same disconnect as Table tab

### Problem 4: Rate limit checks read from `usageHistory` but middleware writes to `usage_records`

  Rate limit middleware (`rateLimitMiddleware.js`):
  → `enforceRateLimits()` writes to `usage_records` via `logUsageAsync()` (line 215)
  → But `checkRTPM()` reads from `usageHistory` (line 78)
  → And `checkRTPD()` reads from `usageHistory` (line 97)
  → And `getMonthlyTokens()` reads from `usageHistory` (rateLimitRepo.js:59)
  → And `getMonthlyCost()` reads from `usageHistory` (rateLimitRepo.js:73)

  This means rate limiting is based on `usageHistory` data, but the middleware
  logs to `usage_records`. The rate limiting works ONLY because the SSE proxy
  also writes to `usageHistory`. If a request only goes through the rate limit
  middleware (without SSE proxy), its usage is logged to `usage_records` but
  NOT counted for rate limiting.

### Problem 5: Table name mismatch between schema.js and migration

  - `schema.js` defines table key as `usageRecords` (camelCase)
  - Migration 004 creates table as `usage_records` (snake_case)
  - `syncSchemaFromTables()` in migrate.js creates `usageRecords` (from schema key)
  - `usageRecordsRepo.js` queries `usage_records` (from migration)
  - This creates TWO tables: `usageRecords` (empty, unused) and `usage_records` (used)
  - This is a secondary bug but doesn't directly cause the data display issue

================================================================================
## COMPLETE DATA FLOW DIAGRAM
================================================================================

  SSE Proxy Request Flow:
  ┌─────────────────────┐
  │  Client → SSE Proxy │
  │  (open-sse/stream.js)│
  └──────────┬──────────┘
             │
             ▼
  ┌─────────────────────┐
  │  logUsage()          │
  │  usageTracking.js    │
  └──────────┬──────────┘
             │
             ▼
  ┌─────────────────────────┐
  │  saveRequestUsage()      │
  │  usageRepo.js:243        │
  └──────────┬──────────────┘
             │
             ▼
  ┌─────────────────────────┐
  │  INSERT INTO usageHistory│  ← WRITE PATH
  │  + usageDaily upsert     │
  │  + _meta counter         │
  └─────────────────────────┘

  Rate Limit Middleware Flow:
  ┌─────────────────────────┐
  │  API Key Request         │
  │  (dashboardGuard.js)     │
  └──────────┬──────────────┘
             │
             ▼
  ┌─────────────────────────┐
  │  enforceRateLimits()     │
  │  rateLimitMiddleware.js  │
  └──────────┬──────────────┘
             │
             ▼
  ┌─────────────────────────┐
  │  logUsageAsync()         │
  │  → logUsage() from       │
  │    usageRecordsRepo.js   │
  └──────────┬──────────────┘
             │
             ▼
  ┌──────────────────────────┐
  │  INSERT INTO usage_records│  ← WRITE PATH
  └──────────────────────────┘

  Dashboard Read Flows:
  ┌──────────────────────────────────────────────────────────────┐
  │  Overview Tab                                                  │
  │  → /api/usage/stats → getUsageStats() → usageHistory ✅       │
  │  → /api/usage/stream → getUsageStats() → usageHistory ✅      │
  │  → /api/usage/chart → getChartData() → usageHistory ✅        │
  ├──────────────────────────────────────────────────────────────┤
  │  Table Tab                                                     │
  │  → /api/usage → getUsageByUser() → usage_records ❌ EMPTY     │
  │  → /api/usage → getUsageSummary() → usage_records ❌ EMPTY    │
  ├──────────────────────────────────────────────────────────────┤
  │  Export                                                        │
  │  → /api/usage/export → getUsageByUser() → usage_records ❌    │
  │  → /api/usage/export → getUsageSummary() → usage_records ❌   │
  ├──────────────────────────────────────────────────────────────┤
  │  Logs Tab                                                      │
  │  → /api/usage/logs → getRecentLogs() → usageHistory ✅        │
  ├──────────────────────────────────────────────────────────────┤
  │  Request Details Tab                                           │
  │  → /api/usage/request-details → getRequestDetails()            │
  │    → requestDetails table (separate, not part of disconnect)   │
  └──────────────────────────────────────────────────────────────┘

================================================================================
## AFFECTED FEATURES (All 5 reported broken)
================================================================================

1. DATA NOT SHOWING — Table tab reads from `usage_records` which is empty
   for SSE proxy requests. Overview tab works (reads from `usageHistory`).

2. CHARTS BROKEN — Charts in Table tab use `records` from `/api/usage`
   which returns empty from `usage_records`. Overview charts work.

3. CALCULATIONS BROKEN — Summary cards in Table tab use `summary` from
   `getUsageSummary()` which reads from `usage_records` (empty).

4. ADMIN ACCESS BROKEN — Admin queries (`getAllUsersUsage()`) read from
   `usage_records` which has no SSE proxy data.

5. EXPORT BROKEN — Export reads from `usage_records` via `getUsageByUser()`
   and `getUsageSummary()` — both return empty for SSE proxy data.

================================================================================
## RECOMMENDED FIX APPROACH (for Task 2)
================================================================================

Option A (Quick fix): Make SSE proxy also write to `usage_records`
  - Add `logUsage()` call in `usageTracking.js` alongside `saveRequestUsage()`
  - Requires mapping SSE proxy fields to `usage_records` schema
  - Requires userId resolution (SSE proxy doesn't have userId context)

Option B (Better fix): Unify read paths to use `usageHistory`
  - Change `/api/usage` route to read from `usageHistory` via `usageRepo.js`
  - Change `/api/usage/export` to read from `usageHistory`
  - Add userId filtering by joining with apiKeys table
  - This matches the existing data in `usageHistory`

Option C (Best fix): Dual-write from SSE proxy
  - Keep both tables serving their purposes
  - Add `logUsage()` call in SSE proxy path for `usage_records`
  - Ensure both tables get populated for all request types
  - Most resilient but most complex

================================================================================
