# Knowledge Base Sweep

How the `knowledge/` directory stays curated, deduplicated, and small enough to keep injecting into agent prompts.

## What the Sweep Does

The KB sweep is a 3-pass cycle implemented in [`engine/kb-sweep.js`](../engine/kb-sweep.js) that prunes duplicate, stale, and oversized entries from `knowledge/`. It runs asynchronously, archives (rather than deletes) what it removes, and writes a `_swept` frontmatter flag so the expensive rewrite pass is idempotent.

```
Entries in knowledge/<category>/*.md
  → Pass 1: Hash dedup       (cheap, no LLM)
  → Pass 2: LLM batch sweep  (Haiku, batches of 30)
  → Pass 3: Per-entry rewrite (Haiku, concurrency 5)
  → Prune _swept/ archive    (>30 days)
  → Invalidate KB cache, write engine/kb-swept.json
```

## The 3-Pass Cycle

### Pass 1 — Hash Dedup

Cheap, deterministic. No LLM call.

For every entry, compute `sha256(normalize(content).slice(0,500) + ':' + content.length)` (source: [`engine/kb-sweep.js:29`](../engine/kb-sweep.js#L29)). Group by hash. When two or more entries collide, keep the most recent (by `date:` frontmatter, then mtime) and archive the rest into `knowledge/_swept/` with a `<!-- swept: ... | reason: hash-duplicate of ... -->` header (source: [`engine/kb-sweep.js:84-105`](../engine/kb-sweep.js#L84)).

This catches near-identical re-postings across batches that the LLM pass might miss because they land in different batches.

### Pass 2 — LLM Batch Sweep

The remaining survivors are sent to Claude Haiku in batches of `LLM_BATCH_SIZE = 30` (source: [`engine/kb-sweep.js:25`](../engine/kb-sweep.js#L25)). Each batch prompt asks for three lists in JSON:

| Field | What the LLM looks for |
|-------|------------------------|
| `duplicates` | Entries with substantially the same content. Returned as `{keep, remove[], reason}`. |
| `reclassify` | Entries in the wrong category. Returned as `{index, from, to, reason}`. |
| `remove`     | Stale or empty entries (boilerplate, "no changes needed", bail-out notes). Returned as `{index, reason}`. |

Each action archives the file via the same `_archiveKbFile()` helper used by Pass 1; reclassification rewrites the `category:` frontmatter line and moves the file into the new category directory (source: [`engine/kb-sweep.js:243-279`](../engine/kb-sweep.js#L243)).

Reclassification targets are validated against `shared.KB_CATEGORIES` (`architecture`, `conventions`, `project-notes`, `build-reports`, `reviews` — source: [`engine/shared.js`](../engine/shared.js) `KB_CATEGORIES`); unknown categories are silently dropped.

If a batch returns invalid JSON or the runtime is unavailable, that batch is skipped with a warning and the rest of the sweep continues (source: [`engine/kb-sweep.js:139-151`](../engine/kb-sweep.js#L139)).

### Pass 3 — Per-Entry Rewrite

Survivors of Passes 1 and 2 that are still on disk get rewritten one entry at a time, with up to `NORMALIZE_CONCURRENCY = 5` parallel workers (source: [`engine/kb-sweep.js:26`](../engine/kb-sweep.js#L26)). Each entry is sent to Haiku with a fixed template prompt that:

- Caps output at ~800 words.
- Forces the structure `## Summary` → `## Key Findings` → `## Action Items` (omit if none) → `## References` (omit if none).
- Preserves all `file:line` references and code snippets.
- Drops boilerplate (full dates, agent IDs in the body, narrative scaffolding).

After a successful rewrite, the entry's frontmatter gains a `_swept: <ISO timestamp>` key (source: [`engine/kb-sweep.js:229`](../engine/kb-sweep.js#L229)).

## The `_swept` Frontmatter Flag

`_swept` makes Pass 3 idempotent across runs. Semantics:

- **Set** to `new Date().toISOString()` whenever the rewrite pass successfully rewrites the body (source: [`engine/kb-sweep.js:229`](../engine/kb-sweep.js#L229)).
- **Skipped** by Pass 3 on subsequent sweeps as long as the file's `mtimeMs` is `<= sweptAt + 1000` ms (1 s grace for FS timestamp jitter — source: [`engine/kb-sweep.js:196-202`](../engine/kb-sweep.js#L196)).
- **Re-processed** if the file is edited after the sweep flag was set (mtime exceeds the grace window).

Hash dedup (Pass 1) and the LLM sweep (Pass 2) ignore `_swept` — those passes act on metadata and similarity, not on whether the entry was previously rewritten.

The flag key is exported as `SWEPT_FLAG_KEY` for tests (source: [`engine/kb-sweep.js:27,429`](../engine/kb-sweep.js#L27)).

## 30-Day Archive Retention

Archived entries land in `knowledge/_swept/` with their original filename (deduped via `shared.uniquePath()` if it collides). They're not deleted immediately — `_pruneOldSwept()` runs at the end of every sweep and removes any file whose `mtimeMs` is older than `SWEPT_RETENTION_MS = 30 * 24 * 60 * 60 * 1000` (30 days — source: [`engine/kb-sweep.js:23,69-81`](../engine/kb-sweep.js#L23)).

This gives humans a recovery window: if a sweep archives something useful, you have ~30 days to copy it back out of `_swept/` before it's permanently pruned.

## Manual Trigger

The dashboard exposes two endpoints:

| Method | Path | Purpose |
|--------|------|---------|
| `POST` | `/api/knowledge/sweep` | Start a sweep in the background. Returns `202 { ok: true, started: true }` (source: [`dashboard.js:7351`](../dashboard.js#L7351), [`dashboard.js:4285`](../dashboard.js#L4285)). |
| `GET`  | `/api/knowledge/sweep/status` | Poll `{ inFlight, startedAt, lastResult, lastCompletedAt }` (source: [`dashboard.js:7352`](../dashboard.js#L7352), [`dashboard.js:4335`](../dashboard.js#L4335)). |

POST body is optional. Pass `{ "pinnedKeys": ["knowledge/conventions/foo.md", ...] }` to add request-level pins on top of `pinned.md`. Pinned entries are excluded from every pass (source: [`engine/kb-sweep.js:345-356`](../engine/kb-sweep.js#L345)).

```bash
# Trigger a sweep
curl -X POST http://localhost:7331/api/knowledge/sweep \
  -H 'Content-Type: application/json' \
  -d '{}'

# Poll status
curl http://localhost:7331/api/knowledge/sweep/status
```

### Concurrency Guard

Only one sweep runs at a time. The POST handler refuses to start a second sweep with `{ ok: true, alreadyRunning: true, startedAt }` while one is in flight (source: [`dashboard.js:4306-4311`](../dashboard.js#L4306)).

The guard auto-releases when the sweep has been running longer than `staleGuardMs(entryCount)` — `max(30 min, 1 s × entryCount)` — which protects against a dashboard process that died mid-sweep leaving the flag stuck (source: [`engine/kb-sweep.js:413-417`](../engine/kb-sweep.js#L413), [`dashboard.js:4286-4305`](../dashboard.js#L4286)).

### Persistent State

Sweep state is mirrored to `engine/kb-sweep-state.json` so the dashboard can recover the in-flight indicator across restarts:

```jsonc
// While running
{ "status": "in-flight", "startedAt": 1715472000000, "startedAtIso": "2026-05-12T..." }

// On success
{ "status": "completed", "startedAt": ..., "completedAt": ..., "durationMs": ..., "summary": "...", "lastResult": { ... } }

// On failure
{ "status": "failed", "startedAt": ..., "completedAt": ..., "error": "..." }
```

Memory still wins when present; the disk file is a fallback (source: [`engine/kb-sweep.js:298-320`](../engine/kb-sweep.js#L298), [`dashboard.js:4296-4354`](../dashboard.js#L4296)). A separate `engine/kb-swept.json` is written after each successful sweep with the human-readable summary line shown by the dashboard's "swept N days ago" badge (source: [`engine/kb-sweep.js:407`](../engine/kb-sweep.js#L407)).

## Automatic Periodic Sweep (opt-in)

The engine tick loop can also auto-spawn the KB sweep without dashboard interaction. Gated by `engine.autoConsolidateMemory` (default `false` — source: [`engine/shared.js`](../engine/shared.js) `ENGINE_DEFAULTS.autoConsolidateMemory`):

- When `engine.autoConsolidateMemory: true`, every tick the engine consults `shouldAutoSweep()` from [`engine/kb-sweep.js`](../engine/kb-sweep.js) and, when the 4-hour cadence has elapsed since the last completion, calls `spawnSweepRunnerDetached()` to fire-and-forget a fresh `engine/kb-sweep-runner.js` process (source: [`engine.js`](../engine.js) tick step 2.1).
- The inbox→`notes.md` consolidation runs every tick *regardless* of this flag via `consolidateInbox()`; `autoConsolidateMemory` controls **only** the heavier `knowledge/` sweep.
- The detached runner survives `minions restart` (same pattern used by the manual trigger), and the in-flight guard above prevents overlap with manual sweeps.

## Dashboard UI

The KB page surfaces sweep state in two places:

1. **Trigger button** (`kbSweep()` in [`dashboard/js/render-kb.js:166`](../dashboard/js/render-kb.js#L166)) posts to `/api/knowledge/sweep` with the user's pinned keys and shows an immediate "queued" toast. If the response says `alreadyRunning`, the toast switches to "already running (Xm elapsed) — let it finish first".
2. **`#kb-swept-time` indicator** renders `swept N days ago · now sweeping (Xm)` (source: [`dashboard/js/render-kb.js:72-82`](../dashboard/js/render-kb.js#L72)). The `sweepInFlight` and `sweepStartedAt` fields come from `GET /api/knowledge` (source: [`dashboard.js:4262-4267`](../dashboard.js#L4262)).

### Polling

The KB page is refreshed by the dashboard's main refresh loop **every third tick (~12 s)** in [`dashboard/js/refresh.js:121`](../dashboard/js/refresh.js#L121). There is no separate sweep-status polling timer and no auto-stop — the in-flight badge keeps refreshing for as long as the user has the dashboard open and the sweep is running. Sweeps can take many minutes for a large KB; the persistent `engine/kb-sweep-state.json` flag plus the indefinite refresh loop are what let the indicator survive dashboard restarts and arbitrarily long sweeps.

## Result Summary

`runKbSweep()` returns (and persists in `engine/kb-swept.json`):

```js
{
  ok: true,
  entriesBefore, entriesAfter,
  bytesBefore,  bytesAfter,
  hashDuplicatesArchived,  // Pass 1
  llmDuplicatesArchived,   // Pass 2 (within-batch dupes)
  staleRemoved,            // Pass 2 (LLM-flagged removals)
  reclassified,            // Pass 2 (category moves)
  rewritten,               // Pass 3
  rewriteBytesBefore, rewriteBytesAfter,
  sweptArchivePruned,      // Files purged from _swept/ (>30 days)
  durationMs,
  summary: '<n> hash-dup, <n> llm-dup, <n> stale, <n> reclassified, <n> rewritten (<bytes> saved)',
}
```

After a non-dry-run sweep, `queries.invalidateKnowledgeBaseCache()` is called so the next `/api/knowledge` read sees the new state (source: [`engine/kb-sweep.js:408`](../engine/kb-sweep.js#L408)).

## What NOT to Do

> **Never write to `knowledge/` directly.** The sweep is the only writer. Every other code path — agents, dashboard handlers, scripts — should write findings to `notes/inbox/<agent>-<id>-<date>.md` and let the consolidation engine classify them into `knowledge/<category>/`.

Why this matters:

- Direct writes bypass the hash-dedup, classification, and `_swept` rewrite passes, so future sweeps see a "fresh" entry and may immediately rewrite or archive it.
- Editing an entry resets its `mtime`, which makes the `_swept` flag stale (re-rewrite on next sweep) and silently changes the entry the next consolidation pass diffs against.
- Manual edits collide with the LLM rewrite pass — your edit can be replaced wholesale by the next sweep.
- Per the team-wide rule injected into every dispatched playbook: **"Never delete, move, or overwrite files in `knowledge/`."**

If you spot an incorrect KB entry, write a note to `notes/inbox/` describing the issue. The next sweep will reclassify, merge, or remove it; humans can also pin entries (via `pinned.md` or the dashboard) to keep them out of the sweep entirely.

The archive directory `knowledge/_swept/` is also off-limits to manual edits — touching it interferes with the 30-day retention timestamp.

## Related Files

- [`engine/kb-sweep.js`](../engine/kb-sweep.js) — implementation
- [`engine/queries.js`](../engine/queries.js) — `getKnowledgeBaseEntries()`, `invalidateKnowledgeBaseCache()`
- [`dashboard.js`](../dashboard.js) — `handleKnowledgeSweep`, `handleKnowledgeSweepStatus`, `handleKnowledgeList`
- [`dashboard/js/render-kb.js`](../dashboard/js/render-kb.js) — UI trigger and indicator
- [`dashboard/js/refresh.js`](../dashboard/js/refresh.js) — refresh cadence (every 3rd tick)
- [`engine/shared.js`](../engine/shared.js) — `KB_CATEGORIES`, `getPinnedItems()`, `uniquePath()`
