# Query Optimization & Performance Features

LibreDB Studio includes enterprise-grade query optimization features to prevent system freezes and provide performance insights for DBAs, data engineers, and developers.

## Table of Contents

- [Query Pagination System](#query-pagination-system)
- [Silent Auto-Limiting](#silent-auto-limiting)
- [Load More Functionality](#load-more-functionality)
- [Query EXPLAIN Integration](#query-explain-integration)
- [Performance Insights](#performance-insights)
- [Architecture](#architecture)

---

## Query Pagination System

### Overview

All SELECT queries are automatically paginated to prevent browser freezes when dealing with large datasets. This is handled transparently without interrupting the user workflow.

### Key Constants

| Constant | Value | Description |
|----------|-------|-------------|
| `DEFAULT_QUERY_LIMIT` | 500 | Default rows per page |
| `MAX_UNLIMITED_ROWS` | 100,000 | Maximum rows for "Load All" |

### How It Works

1. User executes a SELECT query
2. System automatically adds `LIMIT 500 OFFSET 0` if no LIMIT exists
3. If user already specified a LIMIT, it's preserved (no override)
4. Results display with pagination metadata

---

## Silent Auto-Limiting

### Philosophy

Instead of showing warning popups for large datasets, LibreDB Studio silently limits results to 500 rows. This provides:

- **Uninterrupted workflow** - No confirmation dialogs
- **Safe defaults** - System never freezes
- **User control** - Load More when needed

### Visual Indicators

When auto-limiting is applied:

```
┌─────────────────────────────────────────────────────┐
│ Results   500 rows  │  AUTO-LIMITED  │  Load More  │
└─────────────────────────────────────────────────────┘
```

- **AUTO-LIMITED badge** - Shows when the system added a LIMIT
- **Row count** - Displays actual returned rows
- **Load More button** - Appears when more data is available

### Query Limiter Utility

Location: `src/lib/db/utils/query-limiter.ts`

```typescript
import { analyzeQuery, applyQueryLimit } from '@/lib/db/utils/query-limiter';

// Analyze a query
const info = analyzeQuery('SELECT * FROM users WHERE active = true');
// Returns: { type: 'SELECT', hasLimit: false, hasOffset: false, ... }

// Apply limit to query
const result = applyQueryLimit('SELECT * FROM users', 500, 0);
// Returns: { sql: 'SELECT * FROM users LIMIT 500 OFFSET 0', wasLimited: true, ... }
```

### Supported Query Types

| Query Type | Auto-Limit Applied |
|------------|-------------------|
| SELECT | Yes |
| SELECT with LIMIT | No (preserved) |
| SELECT with UNION | Yes (wrapped) |
| SELECT with CTE | Yes |
| INSERT/UPDATE/DELETE | No |
| DDL (CREATE, ALTER) | No |

---

## Load More Functionality

### User Flow

1. Execute query → 500 rows displayed
2. Click "Load More" → Next 500 rows appended
3. Repeat until all data loaded or satisfied

### API Request

```typescript
// Initial query
POST /api/db/query
{
  "connection": {...},
  "sql": "SELECT * FROM orders",
  "options": { "limit": 500, "offset": 0 }
}

// Load More
POST /api/db/query
{
  "connection": {...},
  "sql": "SELECT * FROM orders",
  "options": { "limit": 500, "offset": 500 }
}
```

### Response Format

```typescript
{
  "rows": [...],
  "fields": ["id", "name", ...],
  "rowCount": 500,
  "executionTime": 45,
  "pagination": {
    "limit": 500,
    "offset": 0,
    "hasMore": true,        // More rows available
    "totalReturned": 500,
    "wasLimited": true      // System added LIMIT
  }
}
```

### Load All Option

For advanced users, a "Load All" button triggers an unlimited query (max 100K rows) with a confirmation dialog:

```
┌──────────────────────────────────────┐
│  Load all results?                   │
│                                      │
│  This may slow down your browser.    │
│  Max 100K rows will be loaded.       │
│                                      │
│  [Cancel]          [Load All]        │
└──────────────────────────────────────┘
```

---

## Query EXPLAIN Integration

### Automatic EXPLAIN

Every SELECT query automatically runs EXPLAIN in the background (parallel execution). This provides instant performance insights without user action.

### Supported Databases

| Database | EXPLAIN Format |
|----------|---------------|
| PostgreSQL | `EXPLAIN (ANALYZE, BUFFERS, FORMAT JSON)` |
| MySQL | `EXPLAIN FORMAT=JSON` |
| SQLite | Not supported (no JSON format) |

### How It Works

```
User executes: SELECT * FROM orders WHERE status = 'pending'

┌─────────────────────────────────────────────────────────────┐
│                    Parallel Execution                        │
│                                                              │
│  ┌──────────────────┐      ┌──────────────────────────────┐ │
│  │   Main Query     │      │   Background EXPLAIN          │ │
│  │   (with LIMIT)   │      │   (no LIMIT, ANALYZE)         │ │
│  └────────┬─────────┘      └────────────┬─────────────────┘ │
│           │                              │                   │
│           ▼                              ▼                   │
│     Results Tab                    Explain Tab               │
└─────────────────────────────────────────────────────────────┘
```

### Accessing EXPLAIN Data

Click the "Explain" tab in the results panel to view:
- Performance Insights
- Execution Plan Tree
- Raw JSON

---

## Performance Insights

### VisualExplain Component

Location: `src/components/VisualExplain.tsx`

The VisualExplain component analyzes execution plans and provides actionable insights.

### Automatic Warning Detection

| Warning | Trigger | Severity |
|---------|---------|----------|
| Sequential Scan | Seq Scan on >10K rows | Warning |
| Estimate Mismatch | Actual/Planned ratio >10x | Info |
| Expensive Sort | Sort operation >100ms | Warning |
| High Loop Count | Nested Loop >1000 iterations | Critical |

### Warning Examples

**Sequential Scan Warning:**
```
⚠️ Sequential Scan
Full table scan on "orders" (15.2K rows). Consider adding an index.
```

**N+1 Problem Detection:**
```
🔴 High Loop Count
Nested loop executed 5.2K times. This could indicate an N+1 problem.
```

**Estimate Mismatch:**
```
ℹ️ Estimate Mismatch
Expected 100 rows, got 15.2K. Statistics may be outdated.
```

### Metrics Grid

| Metric | Description |
|--------|-------------|
| Cache Hit Rate | Buffer cache efficiency (>95% is good) |
| Operations | Number of plan nodes |
| Execution Time | Total query time |

### Plan Tree View

Interactive, collapsible execution plan with:
- Node type icons (Seq Scan, Index Scan, Join, Sort, etc.)
- Time bars showing relative cost
- Row counts and costs
- Filter conditions
- Index usage

```
▼ Limit (0.12ms, 500 rows)
  └─▼ Sort (45.2ms, 500 rows)
      └─▼ Seq Scan on orders (120.5ms, 15.2K rows)
          Filter: status = 'pending'
```

---

## Architecture

### File Structure

```
src/
├── lib/db/utils/
│   └── query-limiter.ts      # Query parsing and LIMIT injection
├── app/api/db/
│   └── query/route.ts        # Query API with pagination
├── components/
│   ├── Studio.tsx            # Query execution orchestration
│   ├── ResultsGrid.tsx       # Results display with Load More
│   └── VisualExplain.tsx     # EXPLAIN visualization
└── lib/types.ts              # QueryPagination interface
```

### Data Flow

```
┌─────────────┐     ┌──────────────────┐     ┌─────────────────┐
│  Dashboard  │────▶│  /api/db/query   │────▶│  DB Provider    │
│             │     │                  │     │                 │
│ executeQuery│     │ - Parse query    │     │ - Execute SQL   │
│             │     │ - Apply LIMIT    │     │ - Return rows   │
└─────────────┘     │ - Add pagination │     └─────────────────┘
       │            └──────────────────┘
       │
       ▼
┌─────────────┐     ┌──────────────────┐
│ ResultsGrid │     │  VisualExplain   │
│             │     │                  │
│ - Show rows │     │ - Parse plan     │
│ - Load More │     │ - Show warnings  │
│ - Stats bar │     │ - Render tree    │
└─────────────┘     └──────────────────┘
```

### Key Interfaces

```typescript
// Query pagination metadata
interface QueryPagination {
  limit: number;
  offset: number;
  hasMore: boolean;
  totalReturned: number;
  wasLimited: boolean;
}

// Query result with pagination
interface QueryResult {
  rows: any[];
  fields: string[];
  rowCount: number;
  executionTime: number;
  explainPlan?: any;
  pagination?: QueryPagination;
}

// Query tab state
interface QueryTab {
  id: string;
  name: string;
  query: string;
  result: QueryResult | null;
  explainPlan?: any;
  currentOffset?: number;
  isLoadingMore?: boolean;
  allRows?: any[];
}
```

---

## Best Practices

### For Users

1. **Use WHERE clauses** - Filter data at the database level
2. **Add LIMIT when known** - If you only need 10 rows, add `LIMIT 10`
3. **Check Explain tab** - Review performance before running in production
4. **Use indexes** - Add indexes for frequently filtered columns

### For Developers

1. **Never bypass the limiter** - Always use the query API
2. **Handle pagination** - Support `hasMore` in custom implementations
3. **Parse EXPLAIN** - Use the analyzePlan function for custom analysis

---

## Configuration

Currently, limits are hardcoded. Future versions may support configuration:

```typescript
// Future: .env configuration
QUERY_DEFAULT_LIMIT=500
QUERY_MAX_UNLIMITED=100000
EXPLAIN_AUTO_RUN=true
```

---

## Changelog

| Version | Changes |
|---------|---------|
| 0.7.0 | Initial query optimization system |
| 0.7.1 | Removed Large Dataset popup (silent limiting) |
| 0.7.1 | Added automatic background EXPLAIN |
| 0.7.1 | Added VisualExplain with Performance Insights |
