# Document Import Runbook

This runbook covers the GhostCrab document import path exposed through
`gcp brain document`. It is the operator-facing guide for getting documents into
MindBrain SQLite, with and without an LLM provider.

For the lower-level engine reference and all flags, see
`vendor/mindbrain/docs/document-profile.md`.

For **tabular structured data** (CSV, JSON, YAML), see
[structured-import.md](./structured-import.md).

## What This Imports

The document flow can write several layers:

- normalized `.txt` / `.md` files and sidecar metadata;
- raw documents in `documents_raw`;
- raw chunks in `chunks_raw`;
- source-derived facets and LLM qualification rows in `facet_assignments_raw`;
- optional contextual retrieval text and embeddings for search.

`gcp brain document` launches the bundled `ghostcrab-document` engine. For
database-backed commands, the wrapper injects `--db` automatically from
`GHOSTCRAB_SQLITE_PATH`, the user default SQLite file, or an explicit
wrapper-level `--db <path>`.

## Prerequisites

Install GhostCrab and authorize native binaries:

```bash
npm install
npm run build
node bin/gcp.mjs authorize
```

For a packaged install, use `gcp authorize`.

Set the database path when you do not want the default `~/.ghostcrab/databases/ghostcrab.sqlite`:

```bash
export GHOSTCRAB_SQLITE_PATH="$PWD/data/ghostcrab.sqlite"
```

If you built the document engine locally, point the wrapper at it:

```bash
export GHOSTCRAB_DOCUMENT_ENGINE="$PWD/cmd/backend/zig-out/bin/ghostcrab-document"
```

Database-backed import commands should run while MCP / `ghostcrab-backend` is
stopped. The wrapper probes backend health and refuses to run when the backend
is alive unless you pass `--force`.

Optional extraction tools:

- `pdftotext` for PDF text layers;
- `ocrmypdf` plus Tesseract language packs for OCR;
- `pandoc` for HTML to Markdown;
- `--html-backend builtin-strip` when Pandoc is not installed;
- `--deepseek-command <template>` for a custom OCR wrapper.

## Choose A Mode

Use the no-LLM path when you only need deterministic document/chunk storage or
when no API key is configured. This can normalize files and ingest raw content,
but it cannot infer a document profile or controlled taxonomy assignment unless
you supply mock JSON.

Use the live LLM path when you want corpus-aware profiling, legal/document
splitter choices, contextual retrieval, or taxonomy/facet qualification.

Common provider flags can be passed explicitly:

```bash
--base-url https://api.openai.com/v1 --api-key "$MB_DOCUMENTS_LLM_API_KEY" --model gpt-4.1-mini
```

For `gcp brain document`, the wrapper also reads local `.env` defaults before
spawning the native engine. Explicit CLI flags keep priority over `.env`.

```bash
MB_DOCUMENTS_LLM_MODE=live
MB_DOCUMENTS_LLM_BASE_URL=https://api.openai.com/v1
MB_DOCUMENTS_LLM_MODEL=gpt-4.1-mini
MB_DOCUMENTS_LLM_API_KEY=...
MB_DOCUMENTS_LLM_TIMEOUT_MS=30000
```

Use `MB_DOCUMENTS_LLM_MODE=mock` for deterministic mock-profile or
mock-qualification runs. Do not commit API keys.

## Workflow 1: Normalize Only

Normalize a PDF, HTML, or text source into files that can be inspected before
import:

```bash
gcp brain document document-normalize \
  --input ./source.pdf \
  --output-dir ./out \
  --languages fr,nl \
  --split-by-language
```

Useful fallbacks:

```bash
gcp brain document document-normalize \
  --input ./page.html \
  --output-dir ./out \
  --html-backend builtin-strip
```

```bash
gcp brain document document-normalize \
  --input ./scan.pdf \
  --output-dir ./out \
  --pdf-backend none
```

Normalization does not require an LLM unless you explicitly use a custom OCR
command that calls one.

## Workflow 2: Deterministic Import Without LLM

Create or reuse a workspace and collection, then ingest a text file directly:

```bash
gcp brain document --force document-ingest \
  --workspace-id my_ws \
  --collection-id my_ws::docs \
  --doc-id 1 \
  --source-ref ./out/source.md \
  --language english \
  --strategy paragraph \
  --content-file ./out/source.md
```

This writes `documents_raw`, `chunks_raw`, and deterministic `source.*` facets.
It does not create a semantic profile or taxonomy qualification.

Inspect the inserted document:

```bash
gcp brain document --force document-by-nanoid --nanoid <doc_nanoid>
```

Export a collection backup for review or restore:

```bash
gcp brain backup --force \
  --workspace-id my_ws \
  --scope collection \
  --collection-id my_ws::docs \
  --output ./my_ws_docs.backup.json
```

Export only the workspace taxonomies:

```bash
gcp brain backup --force \
  --workspace-id my_ws \
  --scope taxonomies \
  --output ./my_ws_taxonomies.backup.json
```

Import a normalized OWL2/RDF N-Triples ontology source:

```bash
gcp brain ontology import --force \
  --workspace-id my_ws \
  --ontology-id my_ws::owl \
  --input ./ontology.nt \
  --materialize-graph
```

Export preserved triples or a taxonomies bundle:

```bash
gcp brain ontology export --force \
  --ontology-id my_ws::owl \
  --format ntriples \
  --output ./ontology.export.nt

gcp brain ontology export --force \
  --workspace-id my_ws \
  --ontology-id my_ws::owl \
  --format bundle \
  --output ./ontology.taxonomies.json
```

Validate or restore a backup bundle:

```bash
gcp brain load ./my_ws_docs.backup.json --dry-run
gcp brain load ./my_ws_docs.backup.json --force
```

## Workflow 3: Profile And Persist

One-shot live profile:

```bash
gcp brain document document-profile \
  --content-file ./out/source.md \
  --base-url "$MB_DOCUMENTS_LLM_BASE_URL" \
  --api-key "$MB_DOCUMENTS_LLM_API_KEY" \
  --model "$MB_DOCUMENTS_LLM_MODEL"
```

No-LLM profile inspection:

```bash
gcp brain document document-profile \
  --content-file ./out/source.md \
  --dry-run
```

No-LLM mock profile:

```bash
gcp brain document document-profile \
  --content-file ./out/source.md \
  --mock-profile-json ./fixtures/profile.json
```

Queue a directory for profiling and persistence:

```bash
gcp brain document --force document-profile-enqueue \
  --content-dir ./out \
  --include-ext md,txt \
  --workspace-id my_ws \
  --collection-id my_ws::docs \
  --doc-id-start 1
```

Process queued jobs with a live LLM:

```bash
gcp brain document --force document-profile-worker \
  --limit 4
```

Process queued jobs without an LLM by supplying a validated mock profile:

```bash
gcp brain document --force document-profile-worker \
  --mock-profile-json ./fixtures/profile.json \
  --limit 4
```

Optional contextual retrieval needs both an LLM and embeddings:

```bash
gcp brain document --force document-profile-worker \
  --contextual-retrieval \
  --contextual-search-table-id 1 \
  --embedding-model text-embedding-3-small
```

## Workflow 4: List Qualification Vocabulary

After documents and chunks exist, list the controlled vocabulary. This command
returns the taxonomy IDs to pass as `--taxonomies` and facet IDs to pass as
`--facets`:

```bash
gcp brain document --force qualification-vocab-list \
  --workspace-id my_ws \
  --collection-id my_ws::docs
```

The output shape is JSON:

```json
{
  "workspace_id": "my_ws",
  "collection_id": "my_ws::docs",
  "taxonomies": [
    {
      "id": "my_ws::core",
      "name": "Core taxonomy",
      "version": "1.0.0",
      "source_kind": "constructed",
      "role": "primary"
    }
  ],
  "facets": [
    {
      "id": "topic.category",
      "taxonomy_id": "my_ws::core",
      "namespace": "topic",
      "dimension": "category",
      "value_type": "text",
      "is_multi": false,
      "hierarchy_kind": "flat",
      "values_count": 3,
      "values_preview": ["governance", "risk", "compliance"]
    }
  ]
}
```

You can narrow the listing:

```bash
gcp brain document --force qualification-vocab-list \
  --workspace-id my_ws \
  --collection-id my_ws::docs \
  --taxonomies my_ws::core \
  --facets topic.category,source.filename
```

Use returned taxonomy IDs and facet IDs with a native engine that exposes
`document-qualify`:

```bash
gcp brain document --force document-qualify \
  --workspace-id my_ws \
  --collection-id my_ws::docs \
  --taxonomies my_ws::core \
  --facets topic.category
```

No-LLM qualification fallback when `document-qualify` is available:

```bash
gcp brain document --force document-qualify \
  --workspace-id my_ws \
  --collection-id my_ws::docs \
  --taxonomies my_ws::core \
  --facets topic.category \
  --mock-qualification-json ./fixtures/qualification.json
```

Dry-run qualification checks prompt construction and target selection without
calling an LLM or writing assignments when the native engine provides the verb:

```bash
gcp brain document --force document-qualify \
  --workspace-id my_ws \
  --collection-id my_ws::docs \
  --taxonomies my_ws::core \
  --facets topic.category \
  --dry-run
```

Accepted qualification rows are written to `facet_assignments_raw`. Chunk
assignments are persisted directly and aggregated to document-level assignments.

`document-qualify` writes accepted rows to `facet_assignments_raw`. Use dry-run
first to inspect the prompt envelope, then a mock JSON fixture, then live mode.

## Fallbacks And Troubleshooting

| Situation | Fallback |
|-----------|----------|
| No LLM key | Use `document-ingest`, `document-profile --dry-run`, `--mock-profile-json`, or `--mock-qualification-json`. |
| No Pandoc | Use `--html-backend builtin-strip`. |
| No OCR tooling | Use text-layer PDFs with `pdftotext`, pass already extracted text, or use `--pdf-backend none`. |
| Backend is running | Stop MCP / `ghostcrab-backend`, or pass `--force` only when you accept SQLite lock risk. |
| Wrong database | Set `GHOSTCRAB_SQLITE_PATH` or pass wrapper-level `--db <path>` before the subcommand. |
| Need to inspect what happened | Run `collection-export`, `document-by-nanoid`, or `qualification-vocab-list`. |
| Need IDs for `--taxonomies` / `--facets` | Run `qualification-vocab-list --workspace-id <id> [--collection-id <id>]`. |
| `document-qualify` returns `InvalidArguments` | The installed native engine may not expose full controlled qualification yet; use `qualification-vocab-list` to inspect vocabulary and verify `gcp brain document --help`. |

If a DB-backed command refuses to run, inspect holders:

```bash
gcp brain db-who --path "$GHOSTCRAB_SQLITE_PATH"
```

If the document engine is missing, build it:

```bash
ZIG=zig-0.16 npm run backend:build:debug
```

Then set:

```bash
export GHOSTCRAB_DOCUMENT_ENGINE="$PWD/cmd/backend/zig-out/bin/ghostcrab-document"
```
