---
name: substack-import
description: Ingest an extracted Substack "Export your data" archive into the graph as one `:KnowledgeDocument {kind:'substack-post'}` per essay plus one `:KnowledgeDocument {kind:'substack-subscriber-roster'}` carrying `:MENTIONS` edges to each subscriber `:Person`. Triggers when the operator drops a Substack archive directory or `<publication>.zip` path into chat. Owner + publication name confirm against the importing `:AdminUser` before any read. Idempotent on `substackPostId` and `(accountId, email)`.
---

# Substack Import

**Invoked from `specialists:archive-ingest-operator`** — the admin agent dispatches the archive-ingest-operator when the operator hands over a Substack archive, and the specialist loads this skill. The admin agent never calls this skill directly.

One Substack archive becomes:

- **N `:KnowledgeDocument` nodes** (`source='substack'`, `kind='substack-post'`) — one per essay, MERGEd on the synthetic stable key `attachmentId = "substack-post-${substackPostId}"`. Substack post IDs survive edits, so re-importing after an edit replaces the prior KD's sections under the same `sourceDocumentId`. `document-ingest`'s in-turn typed-section classification chunks the body into `:Section` children.
- **One `:KnowledgeDocument`** (`source='substack'`, `kind='substack-subscriber-roster'`) per import run, MERGEd on `(accountId, kind, publicationName, importId)`, carrying the subscriber count and one `:MENTIONS` edge per subscriber.
- **N `:Person` nodes** — one per subscriber row, MERGEd on `(accountId, email)`. Existing `:Person` nodes are property-update no-ops on re-import.
- **N `:DigitalDocument` nodes** (`kind='substack-image'`) — one per image referenced from any post body, MERGEd on `(accountId, sha256)`, copied to `{accountDir}/archive/substack/<importId>/`.

## Anchor model

A Substack archive has one operator-side identity (the publication's author / owner) and N subscribers. Provenance anchors on the importing `:AdminUser` via `[:IMPORTED]` to every node this skill writes. Subscribers are not "owned by" the importer — they are `:Person` nodes scoped by `accountId`, with the subscription expressed as a `:MENTIONS` edge from the roster KD to each `:Person`. The `publicationName` short string lives as a property on every post KD and the roster KD; there is no `:SubstackPublication` anchor node.

Every node this skill writes (or that document-ingest writes on this skill's dispatch) carries `importId` (UUID per skill run) so a future dream-cycle pass can detect orphans or selectively re-ingest.

## Archive shape

See [archive-shape.md](references/archive-shape.md) for the directory layout, the `posts.csv` columns this skill consumes, the subscriber CSV columns, and the engagement-events filename detection list. Substack export filenames vary by export version — the skill probes a known list and continues with engagement properties absent if none match.

## Pipeline (deterministic, four phases)

### Phase 0 — Provenance confirmation

Before any file body is read:

1. **Identify the importing AdminUser.** Default to the live operator (`$USER_ID`). Surface to the operator: `"Substack archive: importing as :AdminUser <name> ({userId}). Confirm (yes/no)."`. If the operator is importing on behalf of someone else, accept either an existing AdminUser userId or a new `:Person` natural-key (`givenName` + `familyName` + at least one of `email` / `telephone`). Persist as `$importerNodeId`.

2. **Confirm the publication label.** Ask: `"What's the publication's display name? (short string, e.g. 'Stories from the Field')"`. Persist as `$publicationName` — used as a property on every post KD and the roster KD. **No `:SubstackPublication` node is MERGEd** — the label is a property, not an anchor.

3. **Persist run identity.** Generate `$importId = randomUUID()` and `$sessionId` (the writer-session for this skill run). Both flow into every write.

Refuse to read the archive until both `$importerNodeId` and `$publicationName` are echo-confirmed.

### Phase 1 — Walk + engagement detection

1. **Walk the archive directory.** Classify entries per [archive-shape.md](references/archive-shape.md):
   - `posts.csv` — post metadata index.
   - `posts/<slug>.html` — per-post body files.
   - `subscribers.csv` — subscriber roster.
   - Engagement events file — try in order: `email_activity.csv`, `subscriber_activity.csv`, `emails.csv`. Persist the matched filename as `$engagementFile` or `null`.
   - `images/` (or any directory referenced from post-body `<img src="…">`) — image enclosures.

2. **Count candidates.** `postCount = posts.csv row count`, `subscriberCount = subscribers.csv row count`, `imageCount = distinct image src refs across all posts/*.html`.

3. **Emit walk-complete log line.** `[substack-import] start importId=<id> publication=<name> exportPath=<path> posts=<n> subscribers=<n> images=<n> engagementFile=<name|none>`. This line is the operator's grep-able run-shape signal.

### Phase 2 — Bulk-import gate

If `postCount > 200` OR `subscriberCount > 2000`:

1. Emit `[substack-import:gate] threshold=posts>200|subscribers>2000 reason=bulk awaiting=filter`.
2. Surface to the operator: `"This archive has <n> posts and <n> subscribers. Filter before write? Available axes: publishedAt date range (e.g. 'since 2024-01-01'), post tag (e.g. 'only tag X'), subscriber tier ('free' | 'paid' | 'comp'). Reply with one or more, or 'all' to bypass the gate."`.
3. Apply the operator's filter to the candidate set before any write begins.
4. Re-emit a fresh `start` line with the reduced counts.

Below threshold, skip this phase entirely.

### Phase 3 — Writes (three sub-passes, in this order)

**3a. Per-post `:KnowledgeDocument` via librarian/document-ingest.**

For each post (after gate filter), dispatch `librarian` via the Task tool. The dispatch brief carries:

- **Anchor:** the confirmed `$importerNodeId` (`:AdminUser` if the importer was the operator, else the named `:Person`).
- **`attachmentId`:** `"substack-post-${substackPostId}"` — synthetic stable key. Substack post IDs survive edits; re-imports of an edited post replace prior sections under the same `sourceDocumentId`.
- **`storagePath`:** `{accountDir}/archive/substack/<importId>/posts/<slug>.html` (copy the source file there before dispatch so the storage path is stable).
- **`mimeType`:** `text/html`.
- **`sourceUrl`:** `https://<publication-domain>/p/<slug>` if the export carries it, else absent.
- **`sourceType`:** `'substack'` (carried into `:KnowledgeDocument.source`).
- **`scope`:** `'admin'` (the post is the operator's authored content).
- **Additional KD properties** (set by document-ingest from the brief's `extraProperties` block): `kind: 'substack-post'`, `substackPostId`, `slug`, `publishedAt`, `isPaid`, `publicationName`, `bodyTruncated` (true if the body is a paywall stub — detected by length < 200 chars on `isPaid=true` posts), `importId`, `createdByAgent: 'substack-import'`, `createdBySession: $sessionId`.

Per-post outcomes (created / updated / skipped) are returned by document-ingest and accumulated into the per-pass counter — no per-post log line is emitted.

See [posts.md](references/posts.md) for HTML-to-markdown body conversion rules and inline-mention handling.

**3b. Roster KD + per-subscriber `:Person` + `:MENTIONS` edges.**

`archive-ingest-operator` runs these writes directly using its `memory-write` tool — no dispatch needed.

1. **Read engagement aggregates** (once, before the per-subscriber loop). If `$engagementFile` is non-null, parse it per [engagement.md](references/engagement.md) and build an in-memory map `emailToAggregates: { email -> { totalOpens, totalClicks, lastOpenedAt, lastClickedAt, engagementWindowDays } }`. If `$engagementFile` is null, emit `[substack-import:warn] no-engagement-file` once and use an empty map.

2. **MERGE the roster KD** via `memory-write`:
   ```cypher
   MERGE (r:KnowledgeDocument {accountId: $accountId, kind: 'substack-subscriber-roster', publicationName: $publicationName, importId: $importId})
   ON CREATE SET r.source = 'substack', r.subscriberCount = $subscriberCount,
                 r.createdByAgent = 'substack-import', r.createdBySession = $sessionId,
                 r.createdAt = datetime()
   ```
   Persist the returned elementId as `$rosterNodeId`.

3. **Write the `:IMPORTED` edge** from `$importerNodeId` to `$rosterNodeId` carrying `importId` and `createdAt`.

4. **Per-subscriber loop.** For each row in `subscribers.csv` (after gate filter), parse per [subscribers.md](references/subscribers.md), look up `emailToAggregates[row.email]`, then one `memory-write` MERGE:
   ```cypher
   MERGE (p:Person {accountId: $accountId, email: $email})
   ON CREATE SET p.givenName = $givenName, p.familyName = $familyName,
                 p.source = 'substack', p.createdByAgent = 'substack-import',
                 p.createdBySession = $sessionId, p.createdAt = datetime(),
                 p.importId = $importId
   WITH p
   MATCH (r:KnowledgeDocument) WHERE elementId(r) = $rosterNodeId
   MERGE (r)-[m:MENTIONS {mentionContext: 'substack-subscription'}]->(p)
   ON CREATE SET m.tier = $tier, m.subscribedAt = $subscribedAt,
                 m.createdByAgent = 'substack-import', m.createdAt = datetime()
   SET m.unsubscribedAt = $unsubscribedAt,
       m.totalOpens = $totalOpens, m.totalClicks = $totalClicks,
       m.lastOpenedAt = $lastOpenedAt, m.lastClickedAt = $lastClickedAt,
       m.engagementWindowDays = $engagementWindowDays
   ```
   Engagement properties OVERWRITE on re-import (the export carries totals-to-date, not deltas). Absent engagement (no row in the events file) sets the properties to `null`, which Neo4j removes — so missing aggregates are absent, not zero.

Per-row outcomes accumulate into the per-pass counter — no per-row log line is emitted.

**3c. Image enclosures.**

For each distinct image referenced from any imported post body:

1. Copy file to `{accountDir}/archive/substack/<importId>/<sha256><ext>` (perms 0o600).
2. `memory-write` MERGE:
   ```cypher
   MERGE (d:DigitalDocument {accountId: $accountId, sha256: $sha256})
   ON CREATE SET d.kind = 'substack-image', d.source = 'substack',
                 d.storagePath = $storagePath, d.mimeType = $mimeType,
                 d.createdByAgent = 'substack-import', d.createdBySession = $sessionId,
                 d.createdAt = datetime(), d.importId = $importId
   ```
3. For each post KD that references this image, attach the canonical enclosure edge:
   ```cypher
   MATCH (k:KnowledgeDocument), (d:DigitalDocument)
   WHERE k.attachmentId = $postAttachmentId AND elementId(d) = $imageElementId
   MERGE (k)-[e:HAS_ENCLOSURE]->(d)
   ON CREATE SET e.createdByAgent = 'substack-import', e.createdAt = datetime()
   ```
   `:HAS_ENCLOSURE` is the canonical attachment edge for `:KnowledgeDocument → :DigitalDocument` declared in `schema-base.md`. If at skill-load the canonical edge is absent from the schema, fall back to `:MENTIONS {mentionContext: 'substack-image-enclosure'}` — the documented closed-edge fallback.

See [attachments.md](references/attachments.md) for the image-reference discovery walk and storage-path conventions.

### Phase 4 — Completion log

Emit one grep-able summary line:

```
[substack-import] importId=<id> posts=<created>/<updated>/<skipped> subscribers=<created>/<updated>/<skipped> images=<copied>/<skipped> engagementSubscribers=<n> ms=<elapsed>
```

`engagementSubscribers` counts subscribers whose `:MENTIONS` edge received engagement properties (non-null). A `start` line that named an engagement file followed by `engagementSubscribers=0` at completion is a parse defect — investigate the events-file shape.

## Idempotency

- Re-importing identical archive bytes is a no-op: post KDs match on the synthetic `attachmentId = "substack-post-${substackPostId}"`; `document-ingest` skips identical content. Subscribers match on `(accountId, email)`; the `:Person` MERGE is a property-update no-op. Images match on `(accountId, sha256)`.
- Re-importing an archive with edited posts: the synthetic `attachmentId` is stable, so the same KD is updated in place (`document-ingest` regenerates sections under the same `sourceDocumentId`).
- Re-importing an archive with newer engagement data: engagement properties OVERWRITE via `ON MATCH SET`. The export carries totals-to-date, not deltas; the latest export's totals are authoritative.
- Re-importing with an additive subscriber list: existing subscribers' `:MENTIONS` edges keep their `subscribedAt`; new subscribers get a fresh edge. Unsubscribes set `unsubscribedAt` (and do not delete the edge).
- The roster KD is one-per-import-run (keyed on `importId`), so each import leaves an auditable trail of which run produced which `:MENTIONS` set.

## Closed-edge defence

The skill never invents an edge type. Every edge written:

- `:IMPORTED` (AdminUser → roster KD) — provenance, schema-declared.
- `:MENTIONS {mentionContext:'substack-subscription', ...engagement}` (roster KD → Person) — documented catch-all for any relation not in the closed enumeration.
- `:HAS_ENCLOSURE` (post KD → DigitalDocument) — canonical attachment edge if schema-base declares it; `:MENTIONS {mentionContext:'substack-image-enclosure'}` fallback.

Audit query: `MATCH ()-[r]->() WHERE r.createdByAgent='substack-import' RETURN DISTINCT type(r)` must return only the above. Any other edge type is a defect.

## What this skill does not do

- **New labels.** No `:SubstackPublication`, `:Subscriber`, `:Subscription`, `:Post`. Everything maps to existing `:KnowledgeDocument` + `:Person` + `:DigitalDocument`.
- **New edge types.** No `:SUBSCRIBED_TO`, `:OPENED`, `:CLICKED`. Subscription is `:MENTIONS`; engagement is properties on that edge.
- **Per-event engagement detail.** Per-subscriber-per-post events are aggregated to per-edge totals; the individual open/click events are not preserved.
- **Comments.** Substack export does not include reader comments.
- **Inbound newsletter ingestion.** A Substack post landing in the operator's Gmail goes through `conversation-archive --source email` (Task 426), not this plugin.
- **Live API or RSS poll.** Operator-dropped export only.
- **Single-URL ingestion via `WebFetch`.** A Substack URL passed to `document-ingest` via `memory-ingest-web` will NOT recover the article's verbatim prose for gated / paywalled / subscriber-only posts — `WebFetch` retrieves only the public, server-rendered surface (meta-description, preview blurb, third-person publisher summary). Even for free public posts, JS-rendering on Substack means the WebFetch return is unreliable for verbatim capture. **If the operator hands over a Substack URL, advise:** (a) for the operator's own publication — run the export-archive flow this skill owns (Substack admin → "Export your data" → drop the ZIP); (b) for a third-party publication — paste the article text into chat, or save the page as PDF / `.txt` and attach the file. Then re-dispatch the librarian on the pasted text or attached file. Do not attempt URL-based ingestion as a fallback — the prose will be lost. (Recurring failure surfaced on session `d725df52` 2026-05-27; see `.tasks/465-memory-ingest-body-server-sliced-from-offsets.md` and the WebFetch-limitation note in `document-ingest/SKILL.md`.)
- **Audio / video enclosures.** Substack podcast / video posts use additional file shapes the v1 plugin does not handle; only images referenced from post-body `<img>` tags are enclosed.
