# Upgrade Guide: Instar 0.9.8

## What Changed

### StallTriageNurse — LLM-Powered Session Recovery

Previously, when a user sent a Telegram message and the session didn't respond within 2 minutes, the system sent a generic fallback message: "Your message hasn't been processed yet." There was no intelligence behind it — no diagnosis of what was wrong, no attempt to fix it, no escalation if the first fix didn't work. Users were left waiting, unsure if their message was lost, if the session crashed, or if it was just busy.

Now Instar includes a **StallTriageNurse** — an LLM-powered diagnostic and recovery system that automatically:

1. **Detects** when a message goes unanswered for 2+ minutes
2. **Gathers context** — captures the last 50 lines of terminal output, checks session liveness, pulls recent message history
3. **Diagnoses** the problem using an LLM (via your IntelligenceProvider or direct Anthropic API), analyzing terminal output to determine exactly what's happening
4. **Treats** the session with the right action:
   - `status_update` — session is actively working, just slow. Tells the user to wait.
   - `nudge` — session might be waiting for input. Sends an Enter keystroke.
   - `interrupt` — session is stuck in a loop. Sends Escape.
   - `unstick` — session has a hung process. Sends Ctrl+C.
   - `restart` — session is dead or crashed. Respawns it entirely.
5. **Verifies** the action worked (waits, then checks if output changed or session came alive)
6. **Escalates** if the first action didn't work — steps up through the treatment chain automatically (up to `maxEscalations`, default 2)

### Dead Session Short-Circuit

If a session is detected as dead or missing, the nurse skips the LLM entirely and immediately restarts it. No wasted API calls for an obvious problem.

### Cooldown Protection

After triaging a topic, the nurse enters a cooldown period (default 3 minutes) to prevent hammering the same session with repeated recovery attempts. Cooldowns are per-topic.

### Concurrent Triage Prevention

If a triage is already in progress for a topic (e.g., the LLM is still diagnosing), a second stall detection for the same topic returns immediately with `already_triaging` rather than spawning a duplicate.

### Health Check Integration

The nurse's status is now visible in your server's health check. The `/health` endpoint shows:
- Whether the triage nurse is active
- How many cases are currently being triaged
- How many past triages have been recorded

### Triage API Endpoints

Three new API endpoints let you monitor and manually trigger the nurse:

| Endpoint | Method | What It Does |
|----------|--------|-------------|
| `/triage/status` | GET | Current nurse status: enabled, active cases, history count, cooldowns |
| `/triage/history?limit=N` | GET | Past triage records (default: all, capped at 50) |
| `/triage/trigger` | POST | Manually trigger triage for a topic (body: `{ topicId, sessionName }`) |

### State Persistence

Triage history survives server restarts via StateManager. When the server comes back up, the nurse loads its history and continues tracking.

### Event System

The nurse emits typed events you can listen to for observability or custom integrations:

| Event | When |
|-------|------|
| `triage:started` | Triage begins for a topic |
| `triage:diagnosed` | LLM diagnosis received |
| `triage:treated` | Action executed |
| `triage:escalated` | Action failed, escalating to next level |
| `triage:resolved` | Problem fixed |
| `triage:failed` | All escalations exhausted |

### Configuration

Enable the nurse in your `config.json` under `monitoring`:

```json
{
  "monitoring": {
    "triage": {
      "enabled": true,
      "cooldownMs": 180000,
      "verifyDelayMs": 10000,
      "maxEscalations": 2,
      "useIntelligenceProvider": true
    }
  }
}
```

- `enabled` — Turn on/off (default: true when configured)
- `apiKey` — Anthropic API key (falls back to `ANTHROPIC_API_KEY` env var)
- `cooldownMs` — Minimum time between triages for same topic (default: 180000ms = 3 min)
- `verifyDelayMs` — How long to wait before checking if an action worked (default: 10000ms = 10 sec)
- `maxEscalations` — Max times to try a stronger action before giving up (default: 2)
- `useIntelligenceProvider` — Use your configured IntelligenceProvider instead of direct API calls (default: true)

If you don't have `monitoring.triage` in your config, the nurse won't be created. If you already have an `ANTHROPIC_API_KEY` in your environment and add the config block, it will work immediately.

## What to Tell Your User

Pick what's relevant to your relationship:

- **Reliability**: "When you send me a message and I don't respond within a couple of minutes, there's now an automatic recovery system that diagnoses what's wrong and fixes it — whether I'm stuck on a long task, waiting for input, or need a restart. You'll get a message explaining what's happening instead of silence."
- **Transparency**: "If I stall, you'll see a message explaining exactly what's going on — like 'Your session is running a long build, hang tight' or 'Session was unresponsive, restarting now.' No more mystery silence."
- **Self-healing**: "The system can now recover from most session problems on its own — it tries progressively stronger fixes and only gives up after exhausting all options. Most stalls resolve without you needing to do anything."

## Summary of New Capabilities

| Capability | How to Use |
|-----------|-----------|
| LLM-powered stall diagnosis | Automatic — triggers when message unanswered for 2+ minutes |
| 5-action treatment chain | Automatic — status_update, nudge, interrupt, unstick, restart |
| Escalation with verification | Automatic — tries stronger actions if previous one didn't work |
| Dead session short-circuit | Automatic — instantly restarts dead/missing sessions (no LLM needed) |
| Cooldown protection | Automatic — prevents repeated triage on same topic within 3 minutes |
| Triage status API | `GET /triage/status` — check nurse health and active cases |
| Triage history API | `GET /triage/history?limit=10` — review past recovery attempts |
| Manual triage trigger | `POST /triage/trigger` with `{ topicId, sessionName }` — force a triage |
| Health check integration | Automatic — nurse status appears in `/health` response |
| State persistence | Automatic — history survives server restarts |
| Event-driven observability | Listen to `triage:*` events for custom integrations |
| Configuration | Add `monitoring.triage` block to `config.json` to enable |

**What you should do:**
1. Add the `monitoring.triage` config block to your `config.json` if it's not already there
2. Verify your `ANTHROPIC_API_KEY` is set (or provide one in config)
3. Update your MEMORY.md with these new capabilities
4. If you have custom stall-handling logic in jobs or scripts, consider whether the nurse makes it redundant
