# Process / Workflow Engine

Deep reference for the agentic process engine that lives under `lib/process/` and `app/workflows/`.

## What it is

A stage-gated business process engine. Each process is a named, versioned definition with an ordered list of stages. A **case** is a live instance — it has a current stage, an intake payload, accumulated stage outputs, and a status. Agents work inside stages but cannot mutate case state directly. The engine is the single source of truth for "where is case X right now."

The first shipped process is **ADGM Entity Setup** — a 5-stage onboarding flow that takes an applicant intake form and drives it through verification, compliance approval, and a final document package.

## Load-bearing invariant

`lib/process/engine.js` is the only writer of `process_cases.current_stage_id`, `process_cases.current_stage_slug`, `process_cases.status`, and `process_cases.context_json`. Every other module is a reader.

This is enforced by the engine's API surface:
- Stage handlers receive a read-only case snapshot. They cannot reach the DB to mutate the case.
- The single mutation primitive `transitionTo({ case_id, from_stage_id, next_stage|terminal_status })` is wrapped in a transaction and asserts the case is currently in the expected `from_stage_id` before writing — stale transitions throw `PROCESS_TRANSITION_CONFLICT`.
- Stage execution rows are append-only once terminal (`pass`/`fail`/`skipped`/`cancelled`). Retries create new rows with `attempt = previous + 1`.

Tests in `tests/process-engine.test.js` assert these invariants.

## Artifact-first presentation invariant

Every workflow **touchpoint that a user or colleague sees is an HTML artifact**, not a bare dashboard form or JSON panel. Artifacts are the canonical presentation + audit medium of the engine; the engine state machine is the source of truth, and artifacts are *presentation evidence* generated as a side-effect of state transitions.

Three artifact kinds are produced by the engine on a happy path:

| Artifact | Generated at | Persistence link |
|---|---|---|
| **Intake form artifact** (HTML) | `startCase` (first stage = `intake_form`) | `process_cases.context_json.intake_form_artifact_id` |
| **Intake submission artifact** (HTML, PII-redacted) | `submitIntake` | Intake `process_stage_executions.artifact_ids_json` (alongside the form artifact id) |
| **Analysis artifact** (HTML) | Each `agent_run` stage completion (pass or fail) | That stage's `process_stage_executions.artifact_ids_json` |

Plus the existing **case package** markdown artifact written by the `package` stage handler at the end of the flow.

Implementation: `lib/process/artifacts.js`. All three flow through `lib/runtime.js::createArtifact` so they appear in the project library automatically and respect existing ACL / encryption rules.

**Why this is safe vs. the engine-only-writes invariant:** the artifact module performs disk + `artifacts` table writes only. It never writes `process_cases` columns. Artifact-write failures are silently swallowed so a disk error cannot abort a transition — engine state stays correct even if presentation is missing.

**PII redaction:** `redactSensitiveId()` masks the middle of passport / national-ID / SSN / tax-ID values before they land in any artifact HTML on disk. Full values stay only inside `process_cases.intake_payload_json` (protected by `MADARX_ARTIFACT_ENCRYPTION` + project ACLs). The package stage uses the same masking via its own `redactId()` helper.

## Stage kinds (5)

| Kind | Driver | Output |
|---|---|---|
| `intake_form` | Human submitting the form via UI or `submitIntake` API | Writes `intake_payload` onto the case |
| `agent_run` | Agent engine (`runAgentEngine`) with the configured template | Structured JSON output |
| `policy_gate` | Engine — evaluates declarative JSON predicate rules over `intake` + `context` | `{ trace, all_pass }` |
| `human_approval` | Approval request → `/approvals` UI or API | Approved/denied decision |
| `package` | Engine — renders a markdown cover sheet + supporting docs into artifacts | `case_package` artifact id(s) |

### `intake_form` — branded interactive artifact (only intake path as of v2.14.0)

At `startCase`, the engine renders the first `intake_form` stage as a **branded interactive HTML artifact** (`createIntakeFormArtifact` → `renderIntakeFormHtml`): a self-contained document with real form controls that POSTs to `/api/processes/<id>/stages/<slug>/submit`. The artifact id is stored on `context_json.intake_form_artifact_id`.

The Start button on `/workflows` routes through `POST /api/processes/start-and-route`, which creates the case and redirects to `/workflows/new/<case_id>` (the branded intake page) for intake-first definitions, or to `/workflows/<case_id>` for non-intake first stages. Anonymous share links (`/workflows/share/<case_id>?token=…`) render the same artifact with a one-time submit token.

The legacy read-only "preview" renderer, the bare-React intake form (`app/workflows/new/page.js` + `new-case-flow.jsx`), and the `MADARX_BRANDED_INTAKE` env flag were all removed in v2.14.0 — there is exactly one intake surface. Artifact write failure is observation-only: engine state stays the source of truth and the case still advances.

### `policy_gate` — JSON predicate rules

`config.rules[]` is an array of rules, ALL of which must pass (implicit AND) for the gate to pass. Each rule is a **JSON predicate** (the canonical form):

```json
{
  "op": "and",
  "rules": [
    { "op": "eq",  "path": "intake.country", "value": "US" },
    { "op": "gte", "path": "prior_stage.kyc-risk.risk_score", "value": 0.3 },
    { "op": "in",  "path": "intake.type", "value": ["MSA", "SOW", "NDA"] }
  ]
}
```

- **Logical ops:** `and` / `or` (with `rules[]`), `not` (with `rule`).
- **Compare ops:** `eq` / `neq` (loose `==`/`!=`, so `neq null` is the "field must be provided" idiom), `lt` / `lte` / `gt` / `gte` (numeric coercion), `in` / `nin` (array `value`).
- **Path resolution:** `intake.<field…>`, `prior_stage.<slug>.<field…>`, `context.<field…>`. Missing paths resolve to `undefined`. Max nesting depth 20.
- An optional `label` on any rule is shown in the gate trace.

Evaluator: `lib/process/policy-predicates.js`. Validator: `lib/process/spec-validator.js::validatePredicate` checks predicate shape at authoring time. There is no string parsing — no tokenizer, no escaping bugs.

**Legacy string DSL (deprecated, one-release back-compat).** The old string-expression grammar (`intake.x == 'y' && prior_stage('kyc').score < 0.5`) still evaluates via `lib/process/stages/policy-dsl-legacy.js`, routed per-rule by the handler. Seeds, docs, and new authoring use predicate JSON only; the string path is on a removal track and should not be used in new workflows.

Stage handlers live at `lib/process/stages/<kind>.js`. Each exports `executeStage({ case, stage, execution, heartbeat_task_id })` and returns an outcome envelope:

```js
{
  kind: 'pass' | 'fail',
  output: { /* structured JSON */ },
  artifact_ids?: ['art-...'],
  agent_run_id?: 'agentrun-...',
  approval_required?: { approval_request_id, summary },
  terminal_status?: 'rejected' | 'failed',
  error?: 'string',
}
```

## Allowed transitions

```
intake --submitIntake--> running --stage pass--> running
                                  --stage fail w/ on_fail--> running (failure route)
                                  --stage fail w/o on_fail--> rejected
                                  --human_approval pending--> awaiting_approval
                                                              --approve--> running
                                                              --deny----> rejected
                                  --final stage pass--> complete
                                  --user cancel--> cancelled
```

Cancellation is allowed from any non-terminal state. Re-cancelling a cancelled case is a no-op. Cancelling a complete/rejected/failed case throws.

## Database

Four tables in `lib/schema.js`:

- `process_definitions` — slug, name, version, intake_form_schema_json, status
- `process_stages` — definition_id, position, slug, name, kind, config_json, on_pass_stage_slug, on_fail_stage_slug
- `process_cases` — definition_id, project_id, title, status, current_stage_id, current_stage_slug, intake_payload_json, context_json
- `process_stage_executions` — case_id, stage_id, attempt, status, outcome, heartbeat_task_id, approval_request_id, output_json, artifact_ids_json

Audit trail: every transition writes a row to `audit_logs` with `action_target_type='process_case'` via `appendProcessAudit` (engine-internal). Live-event types `process.case.created`, `process.stage.entered`, `process.stage.completed`, `process.case.complete` fan out through `publishLiveEvent` → `activity_log` → SSE clients.

## Heartbeat integration

Engine-driven stages (`agent_run`, `policy_gate`, `human_approval`, `package`) execute as heartbeat tasks of type `process_stage`. The worker at `lib/heartbeat-worker.js` dispatches by type and calls `runStageExecution({ case_id, stage_id, execution_id, heartbeat_task_id })`.

- `human_approval` stages return `approval_required` on first run, which pauses the heartbeat task via the existing `pauseHeartbeatTaskForApproval` machinery.
- `decideApprovalRequest` in `lib/approvals.js` fires a synchronous handler registered by the engine at module init. Approval → heartbeat queue auto-resumes the task → engine sees the approval consumed → advances. Denial → engine routes via `on_fail_stage_slug` or terminal `rejected`.

## Authoring a new process definition

1. Create a seed file at `lib/process/seeds/<slug>.js` exporting a `seed<Name>()` function that calls `upsertDefinition(spec)` from `lib/process/definitions.js`.
2. Spec shape:
   ```js
   {
     slug: 'my-process',
     name: 'My Process',
     version: 1,
     description: '...',
     category: 'onboarding' | 'compliance' | ...,
     stages: [
       { slug, name, kind, config, on_pass_stage_slug?, on_fail_stage_slug? },
       ...
     ],
   }
   ```
3. For `intake_form` stages, `config.form_schema` must hold the JSON schema (see `adgm-entity-setup.js` for the supported shape).
4. For `agent_run` stages, `config.agent_template_slug` names a `kind='agent'` marketplace template installed in the case's project. Add the template to `lib/process/seeds/agent-templates.js` if it doesn't exist.
5. Call your seed from `lib/seeds.js::seedDefaults()` after `seedProcessAgentTemplates`.

Definitions are versioned (`process_definitions.version`). Bump the version when you change the stage list; in-flight cases keep their original `definition_id`.

## Demo mode

The agent_run stage handler short-circuits to synthetic outputs when:
- `process.env.MADARX_PROCESS_FAKE_AGENT === '1'`, OR
- `process.env.NODE_ENV === 'test'`, OR
- `process.env.VILLA515_DATA_DIR` is set (tests run in this branch)

Synthetic outputs are deterministic functions of the intake payload — they make the demo path runnable end-to-end without an LLM round-trip and let tests assert on stable behavior.

## Out of scope (deliberate)

- **No XOR/AND splits.** Flow is linear with `on_pass` / `on_fail` routing only. Anything more complex belongs in a custom orchestrator agent, not in the engine.
- **No BPMN compatibility layer.** Definitions are plain JS objects → SQLite rows. No XML, no namespaces.
- **No parallel sub-flows in v1.** A stage owns one execution at a time.
- **No external workflow engine integration.** Camunda / Temporal / similar are not in scope.

## Pointers

- Engine: `lib/process/engine.js`
- Stage handlers: `lib/process/stages/*.js`
- Artifact renderers: `lib/process/artifacts.js`
- Definitions catalog: `lib/process/definitions.js`
- Validator: `lib/process/spec-validator.js` (strict; single source of truth for spec shape)
- ADGM seed: `lib/process/seeds/adgm-entity-setup.js`
- Agent templates seed: `lib/process/seeds/agent-templates.js`
- Marketplace workflow templates: `lib/process/seeds/marketplace-workflows.js` (`kind:'workflow'` — vendor-onboarding, contract-review, support-escalation)
- Agent tools: `lib/tools/adapters/workflow-definitions.js` (`create_workflow_definition`, `update_workflow_definition`; risk='medium', approval_required)
- API: `app/api/processes/**/route.js`
- UI: `app/workflows/page.js`, `app/workflows/new/page.js`, `app/workflows/[caseId]/page.js`
- Marketplace WorkflowCard: `app/marketplace/workflow-card.jsx`
- Unit tests: `tests/process-engine.test.js`, `tests/process-spec-validator.test.js`
- Phase 3 tests: `tests/marketplace-workflow-kind.test.js`, `tests/agent-create-workflow-tool.test.js`
- Artifact tests: `tests/process-artifacts.test.js`
- End-to-end test: `tests/process-adgm-demo.test.js`
