# Context Statistics Tracking

**Status**: ✅ Implemented (v6.4.0)
**Part of**: Phase 2 - Context Editing Enhancement
**Related**: Memory Tool Integration, Context Management

---

## Overview

Context Statistics Tracking provides comprehensive monitoring of token usage, context clearing events, and memory operations to measure the effectiveness of context management.

## Features

### Tracking Capabilities

1. **Context Clear Events**
   - Timestamp of each clear
   - Input tokens at clear
   - Tool uses cleared
   - Tokens saved
   - Trigger type (auto at 100k or manual)
   - Agent ID

2. **Memory Operations**
   - Every view, create, str_replace, insert, delete, rename
   - Success/failure status
   - Agent performing operation
   - Estimated tokens used
   - Timestamp for trend analysis

3. **Session Metrics**
   - Total input/output tokens
   - Number of clear events
   - Total tokens saved
   - Memory operations count
   - Peak token usage
   - Agent-specific performance

### Storage

```
~/.versatil/stats/
├── clear-events.json      # Last 1,000 context clear events
├── memory-ops.json        # Last 5,000 memory operations
└── sessions.jsonl         # All session metrics (append-only)
```

**Size**: ~50KB per 1,000 operations (negligible disk usage)

## Usage

### Quick Check

```bash
npm run context:stats
```

Shows:
- Total tokens processed
- Clear events count
- Tokens saved
- Memory operations by type
- Clear events by agent
- Last clear event details

### Detailed Report

```bash
npm run context:report
```

Generates markdown report with:
- Summary statistics
- Token savings rate calculation
- Memory ops per clear ratio
- Recent clear events (last 5)
- Efficiency metrics

### Cleanup Old Data

```bash
npm run context:cleanup
```

Removes stats older than 30 days while keeping recent data for trends.

### Test with Sample Data

```bash
node scripts/test-context-stats.cjs
```

Demonstrates tracking with sample clear events and memory operations.

## Dashboard Output Example

```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  📊 VERSATIL Context Statistics
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

📈 Summary Statistics

  Total Tokens Processed: 450,000
  Total Clear Events: 4
  Total Tokens Saved: 14,000
  Avg Tokens per Clear: 3,500
  Total Memory Operations: 67
  Uptime: 1.25 hours

🔧 Memory Operations by Type

  view        : █████████████████████ 42
  create      : ███████ 15
  str_replace : ████ 8
  insert      : █ 1
  delete      : █ 1

📊 Clear Events by Agent

  maria-qa            : ███ 2
  marcus-backend      : █ 1
  james-frontend      : █ 1

⏱️  Last Clear Event

  Timestamp: 2025-10-18T14:32:15.000Z
  Input Tokens: 105,000
  Tool Uses Cleared: 15
  Tokens Saved: 3,500
  Agent: maria-qa
```

## Integration

### Automatic Tracking

Context statistics are tracked automatically:

1. **Memory Operations**: All operations through `MemoryToolHandler` are automatically tracked
2. **Context Clears**: Track via `ContextStatsTracker.trackClearEvent()`
3. **Session Metrics**: Start/end sessions for comprehensive tracking

### Programmatic Usage

```typescript
import { getGlobalContextTracker } from '@versatil/sdlc-framework/memory';

// Get tracker instance
const tracker = getGlobalContextTracker();
await tracker.initialize();

// Start a session
const sessionId = tracker.startSession('maria-qa');

// Track context clear
await tracker.trackClearEvent({
  inputTokens: 105000,
  toolUsesCleared: 15,
  tokensSaved: 3500,
  triggerType: 'input_tokens',
  triggerValue: 100000,
  agentId: 'maria-qa'
});

// Track memory operation (automatic via MemoryToolHandler)
await tracker.trackMemoryOperation({
  operation: 'view',
  path: 'maria-qa/test-patterns.md',
  success: true,
  agentId: 'maria-qa',
  tokensUsed: 120
});

// Update token usage
tracker.updateTokenUsage(50000, 12000);

// Get statistics
const stats = tracker.getStatistics();
console.log(`Total tokens saved: ${stats.totalTokensSaved}`);

// Generate report
const report = await tracker.generateReport();
console.log(report);

// End session
const metrics = await tracker.endSession();
```

### Visualization

```typescript
import {
  ContextStatsVisualizer,
  createDashboard,
  createMarkdownReport
} from '@versatil/sdlc-framework/memory';

const visualizer = new ContextStatsVisualizer();

// Generate dashboard
const stats = tracker.getStatistics();
const dashboard = visualizer.generateDashboard(stats);
console.log(dashboard.join('\n'));

// Generate markdown report
const events = tracker.getClearEvents();
const report = visualizer.generateMarkdownReport(stats, events);
console.log(report);
```

## Use Cases

### Debugging Context Issues

```bash
# User reports: "Agent forgot our discussion"

npm run context:stats

# Check last clear event:
# - When did it happen?
# - Was it expected (>100k tokens)?
# - What was cleared?
# - Did agent store patterns to memory before clear?
```

### Performance Optimization

```bash
npm run context:report

# Identify:
# - Which agents use context most efficiently
# - Average tokens saved per clear
# - Memory operation patterns
# - Opportunities for optimization
```

### Team Reports

```bash
npm run context:report > weekly-stats.md

# Share with team:
# - Total tokens processed this week
# - Context clearing efficiency
# - Memory operation trends
# - Agent utilization patterns
```

## Metrics

### Performance Impact
- **Tracking Overhead**: <1ms per operation
- **Storage Growth**: ~50KB per 1,000 operations
- **Query Performance**: O(n) for time-range queries
- **Memory Impact**: Negligible (async tracking)

### Efficiency Metrics Calculated
- **Token Savings Rate**: (Total Saved / Total Processed) × 100
- **Memory Ops per Clear**: Total Operations / Total Clears
- **Avg Time Between Clears**: Uptime / Clear Events
- **Avg Tokens per Clear**: Total Saved / Clear Events

## Files

### Implementation
- **Tracker**: `src/memory/context-stats-tracker.ts` (~460 lines)
- **Visualizer**: `src/memory/context-stats-visualizer.ts` (~380 lines)
- **CLI Script**: `scripts/context-stats.cjs` (~167 lines)
- **Test**: `scripts/test-context-stats.cjs` (~120 lines)

### Exports
All functionality exported from `src/memory/index.ts`:
- `getGlobalContextTracker()` - Singleton tracker instance
- `ContextStatsTracker` - Main tracker class
- `ContextStatsVisualizer` - Visualization utilities
- Types: `ContextStatistics`, `SessionMetrics`, `ChartData`, etc.

## Integration with Memory Tool

Context statistics automatically integrate with Memory Tool operations:

1. Every `view`, `create`, `str_replace`, etc. is tracked
2. Agent attribution tracked
3. Token usage estimated (1 token ≈ 4 characters)
4. Zero performance impact (<1ms overhead)

**Example Workflow**:
```yaml
1. Maria-QA creates test pattern:
   → Tracked: create operation, maria-qa agent, 125 tokens

2. Context reaches 100k tokens:
   → Tracked: Clear event, 15 tools cleared, 3,500 tokens saved

3. Marcus reads security pattern:
   → Tracked: view operation, marcus-backend agent, 87 tokens

4. End of day:
   → Run: npm run context:stats
   → See: Complete picture of context usage
```

## Future Enhancements

### Planned (v7.0+)
- [ ] Real-time dashboard (live updates)
- [ ] Trend analysis (velocity over time)
- [ ] Anomaly detection (unusual patterns)
- [ ] Export to external monitoring (Prometheus, Grafana)
- [ ] Per-agent efficiency scoring
- [ ] Predictive context clearing (ML-based)

### Experimental
- [ ] Context "heat map" visualization
- [ ] Memory operation impact analysis
- [ ] Token usage forecasting
- [ ] Automated optimization suggestions

## References

- **CLAUDE.md**: Lines 513-672 (Context Statistics section)
- **Phase 2 Documentation**: `docs/enhancements/CONTEXT_EDITING_PHASE2.md`
- **Memory Tool Integration**: `docs/enhancements/MEMORY_TOOL_INTEGRATION.md`
- **Claude Context Editing**: https://docs.claude.com/en/docs/build-with-claude/context-editing

---

**Last Updated**: 2025-10-19
**Status**: Production-ready
**Version**: 6.4.0
