<!-- diátaxis: how-to -->

# Run the Ecosystem Daily Report

> Step-by-step operator guide for verifying, triggering, and troubleshooting the ecosystem daily report.

**Time budget:** ≤ 5 minutes per troubleshooting section.

## Table of Contents

1. [Verify the schedule is running](#verify-the-schedule-is-running)
2. [Trigger manually](#trigger-manually)
3. [Check collector output](#check-collector-output)
4. [Read the golden-signals report](#read-the-golden-signals-report)
5. [Handle failures](#handle-failures)

## Verify the schedule is running

**Command:**
```bash
vds-cli scheduler schedules show ecosystem-daily-report
```

**Expected output:**
```
Name:       ecosystem-daily-report
Cron:       0 6 * * *
Queue:      ops-default
Status:     active
Next run:   2026-04-28 06:00:00+00:00
```

**If output is unexpected:**

| Output | Meaning | Next step |
|--------|---------|-----------|
| `Status: paused` | Schedule is paused | Run `vds-cli scheduler schedules resume ecosystem-daily-report` |
| `Status: missing` | Schedule not registered | Check `VDS_SCHEDULER_ENABLED=true` in shell/env, then re-register the schedule |
| Empty / not found | Scheduler service may be down | Check Docker: `docker ps | grep scheduler` |

## Trigger manually

**Command:**
```bash
vds-cli scheduler schedules trigger-now ecosystem-daily-report
```

**Expected output:**
```
Triggered ecosystem-daily-report
Workflow ID: wf-<uuid>
Status:      PENDING
```

**If output is unexpected:**

| Output | Meaning | Next step |
|--------|---------|-----------|
| `Scheduler not found` | Scheduler service not reachable | Verify scheduler container: `docker ps | grep scheduler` |
| `Authentication error` | DBOS or scheduler credentials missing | Check `VDS_SCHEDULER_ENABLED` and state DSN env vars |
| No response | Network or queue congestion | Retry once; if still failing, check `ops-default` queue depth |

## Check collector output

**Command (diagnose a single collector):**
```bash
./scripts/worktree_uv.sh run --project scheduler_orchestrator \
  python scheduler_orchestrator/scripts/collector-diagnostics.py "Prometheus"
```

**Expected output:**
```json
{
  "collector": "Prometheus",
  "status": "ok",
  "metrics": { ... },
  "golden_signals": { ... }
}
```

**Command (all collectors, JSON):**
```bash
./scripts/worktree_uv.sh run --project scheduler_orchestrator \
  python scheduler_orchestrator/scripts/collector-diagnostics.py --all --json
```

**If output is unexpected:**

| Output | Meaning | Next step |
|--------|---------|-----------|
| `Collector unavailable — timed out after 10s` | Collector exceeded timeout | Bump `VDS_ECOSYSTEM_DAILY__COLLECTOR_TIMEOUT_S=30`; restart; if persistent, check downstream dependency (e.g., Prometheus endpoint down) |
| `connection refused` | Target port/service not reachable | Verify target is up: `curl http://localhost:8005/metrics` or `docker ps | grep <service>` |
| `subprocess error` | Git or Docker command failed | Check `git worktree list` or `docker ps` directly |

## Read the golden-signals report

**Command (read the latest report):**
```bash
cat reports/ecosystem/$(ls -t reports/ecosystem/ | head -n 1)
```

**Expected output (exec mode, healthy):**
```
📊 Ecosystem Report — 2026-04-28 | 🟢 Health: 94/100
ERRORS:    2 workflow failures (↓3 vs yesterday)
LATENCY:   p95 workflow start: 0.8s | p95 event dispatch: 120ms
SATURATION: PG pool: 23/500 (5%) | Redis: 42MB/256MB (16%) | Queue: 3 pending
TRAFFIC:   142 workflows completed | 47 events dispatched | 72+70 commits
```

**If the file is missing:**
1. Check DB: `SELECT report_date, health_score FROM sched.ecosystem_report_runs ORDER BY report_date DESC LIMIT 3;`
2. If no row for today → report never ran: see [Report not generated by 06:30 UTC](#report-not-generated-by-0630-utc).
3. If row exists with `report_path` → check path or filesystem permissions.

## Handle failures

### Report not generated by 06:30 UTC

**Step 1 (30 s) — Check workflow errors:**
```bash
vds-cli scheduler workflows list --status ERROR
```

**Step 2 (30 s) — Check if schedule is paused:**
```bash
vds-cli scheduler schedules show ecosystem-daily-report
```

**Step 3 (30 s) — Verify scheduler enabled:**
```bash
echo $VDS_SCHEDULER_ENABLED
# Expected: true
```

**Step 4 (1 min) — Trigger manually and watch:**
```bash
vds-cli scheduler schedules trigger-now ecosystem-daily-report
# Then tail events:
vds-cli scheduler events tail --topic ecosystem.daily.failed
```

**Step 5 (2 min) — Programmatic check:**
```bash
./scripts/worktree_uv.sh run --project scheduler_orchestrator \
  python -c "from scheduler_orchestrator.tasks.ecosystem_report import check_missed_run; check_missed_run()"
```

### Health score is 🔴 (<60)

**Step 1 (1 min) — Read the full report for triage hints:**
```bash
./scripts/worktree_uv.sh run --project scheduler_orchestrator \
  python scheduler_orchestrator/scripts/generate-report.py --mode full --output /tmp/triage.md
cat /tmp/triage.md
```

**Step 2 (1 min) — Identify which collectors are failing:**
Look for `[unavailable]` placeholders in the report output.

**Step 3 (2 min) — Diagnose the failing collector:**
```bash
./scripts/worktree_uv.sh run --project scheduler_orchestrator \
  python scheduler_orchestrator/scripts/collector-diagnostics.py <CollectorName>
```

**Step 4 (1 min) — Adjust thresholds if anomaly is a false positive:**
```bash
export VDS_ECOSYSTEM_DAILY__ERROR_THRESHOLD=20
export VDS_ECOSYSTEM_DAILY__PENDING_THRESHOLD=100
export VDS_ECOSYSTEM_DAILY__COMMIT_THRESHOLD=75
```

### Specific collector timeout

**Symptom:** Section shows `⚠️ Collector unavailable — <Name> timed out after 10s`

**Quick command:**
```bash
# Confirm the collector works with longer timeout
VDS_ECOSYSTEM_DAILY__COLLECTOR_TIMEOUT_S=30 \
./scripts/worktree_uv.sh run --project scheduler_orchestrator \
  python scheduler_orchestrator/scripts/collector-diagnostics.py "<Name>"
```

**Expected:** Returns metrics instead of timeout.

**Next steps:**
1. Make timeout change permanent in `~/.vds/.env` or the scheduler's env.
2. If still timing out, check downstream dependency (endpoint/process/container).

### DBOS migration drift

**Symptom:** `sched.ecosystem_report_runs` row exists but `health_score` is NULL; or `golden_signals` column is missing.

**Step 1 (1 min) — Check migration status:**
```bash
./scripts/worktree_uv.sh run --project scheduler_orchestrator \
  alembic -c scheduler_orchestrator/alembic.ini current
```

**Expected:** Current revision is `20260428_a1b2c3d4e5f8_add_phase2_report_columns` (or newer).

**Step 2 (3 min) — Apply migration:**
```bash
./scripts/worktree_uv.sh run --project scheduler_orchestrator \
  alembic -c scheduler_orchestrator/alembic.ini upgrade head
```

**Step 3 (1 min) — Re-run report:**
```bash
vds-cli scheduler schedules trigger-now ecosystem-daily-report
```
