# Connectors

**Connectors** expose external services (Mantis, GitLab, Teams, …) as
tools that the Forge chat agent and the browser extension can call.
Each connector is a declarative YAML manifest — schema + the extraction
scripts that implement each tool — pulled from the remote
[`forge-connectors`](https://github.com/aiwatching/forge-connectors)
registry.

> **Architecture**: connectors are an independent subsystem (`lib/connectors/`).
> They are NOT plugins. Pipeline-node plugins (`lib/plugins/`) and connectors
> share no schema, no registry, no config file.

There are **no built-in connectors**. A fresh install starts with an empty
connector list; the user picks what they want from the marketplace.

## Marketplace flow

1. On boot, Forge syncs every configured **marketplace source**:
   - **Public** — default `https://raw.githubusercontent.com/aiwatching/forge-connectors/main` (override via `connectorsRepoUrl`). Fetched anonymously.
   - **Enterprise repos** — any keys registered via Settings → Marketplace Providers (or `--enterprise-key=`). Each gets pulled via the GitHub Contents API with its bearer PAT. Sources are ordered by configured priority; the first one added wins on conflict.
   Each source's `registry.json` is cached at `<dataDir>/connectors/sources/<source-id>/registry-cache.json`.
2. The Settings → Connectors panel merges all source caches into a single marketplace view:
   - Same id present in multiple sources → highest-priority source is "in use", lower sources land in `hidden_sources` (shown in the detail pane).
   - Every row carries a source badge: `🌐 public` / `🔒 <tenant>` / `📦 local` (install-local upload).
   - Install state: **Available** / **Installed v0.5.0** / **Update 0.5 → 0.6**.
3. Clicking **Install** fetches `<id>/manifest.yaml` from the winning source, writes it to `<dataDir>/connectors/<id>/manifest.yaml`, and records `installed_source_id` in `<dataDir>/connector-configs.json`. The badge in the marketplace reflects that installed_source — not the current registry winner — so users can see "I installed mantis from fortinet" even after the public version bumps.
4. The user fills in settings (host URL, PAT, etc.) via the existing settings UI in the extension or `/api/connectors/<id>/settings`.
5. Chat tools become available the instant the manifest lands on disk — no Forge restart.

Sync re-runs on manual **Refresh** in the Settings panel and after adding a new enterprise key. Network failures per-source are isolated — the offline cache from each source's last successful sync keeps the marketplace usable.

See `01-settings.md` § Marketplace Providers for how to register enterprise keys and what the priority rules mean in practice.

## Repository layout

The `forge-connectors` repo (forkable via `connectorsRepoUrl`) is:

```
registry.json
<id>/
  manifest.yaml      ← what Forge installs
  README.md          ← optional, surfaced in the marketplace
```

`registry.json` lists every connector with id/name/version/icon/description
+ `min_forge_version`. Forge only consults the registry for the install
flow; once installed, `<dataDir>/connectors/<id>/manifest.yaml` is the
source of truth for that user.

## Local data layout

```
<dataDir>/
├── connectors/
│   ├── sources/
│   │   ├── public/
│   │   │   ├── registry-cache.json    last fetch from public source
│   │   │   └── wizard-template.json   optional wizard template cache
│   │   └── enterprise-fortinet/       one dir per enterprise key
│   │       ├── registry-cache.json
│   │       └── wizard-template.json
│   ├── mantis/manifest.yaml           installed manifest copy (one per id, regardless of source)
│   ├── gitlab/manifest.yaml
│   └── …
├── connector-configs.json             { "<id>": { config, installed_version, enabled, installed_source_id } }
└── .enterprise-keys.json              encrypted list of registered enterprise keys
```

Secret fields (`type: secret`) in `config` are encrypted at rest with
AES-256-GCM via the existing key at `<dataDir>/.encrypt-key`. The on-wire
GET response masks them with bullets so plaintext never leaves the server.

## Execution model: DOM extraction, not REST

**Browser-side connectors do NOT call REST APIs.** They navigate the
user's tabs and parse rendered HTML via `chrome.scripting.executeScript`.
This is the whole point of routing through a browser extension — reuse
the user's already-authenticated UI session, with zero token management.

| System | REST API needs | Browser DOM needs |
|---|---|---|
| MantisBT | Per-user API token | Already-logged-in PHPSESSID cookie ✅ |
| GitLab UI | Personal Access Token | Session cookie on `/-/issues/` etc. ✅ |
| JIRA Server | PAT or basic auth | Session cookie ✅ |
| Teams web | MSAL bearer token | Session cookie ✅ |

This means:

- **No PAT / API tokens to manage** for browser connectors — they reuse
  the user's logged-in session.
- **Data never flows through Forge** — Forge only ships the manifest +
  extraction script. The LLM in the chat backend calls the tool, the
  extension scrapes in the user's tab, the LLM sees the result.
- Trade-off: site redesigns break scripts. Bump the manifest `version`
  in the registry and users pull the fix on next sync — no Forge or
  extension release needed.

## Manifest format

```yaml
id: mantis
name: MantisBT
icon: "🐞"
version: "0.5.0"
author: forge
description: |
  Read + comment on MantisBT bugs from the user's logged-in browser session.

min_forge_version: "0.8.0"

# Default extension execution context.
#   main     — permissive-CSP sites (Mantis, GitLab self-hosted)
#   isolated — strict-CSP sites that block eval (Teams, github.com)
runner: main

# Per-user settings (rendered as a form in the extension).
settings:
  base_url:
    type: string
    label: Mantis base URL
    required: true
  token:
    type: secret             # encrypted at rest (AES-256-GCM)
    label: API token (optional fallback)

# Chrome match pattern; {settings.*} expanded server-side.
host_match: "{base_url}/*"

# Substring detected after navigation → "user not logged in".
login_redirect: "/login_page.php"

tools:
  list_my_bugs:
    description: "List bugs assigned to me."
    parameters:
      status: { type: select, options: ["open", "closed", "all"], default: "open" }
      limit:  { type: number, default: 50 }

    # Default protocol is 'browser' — runs in the user's tab.
    page:
      url: "{base_url}/view_all_bug_page.php"
      on_target: "/view_all_bug_page.php"
    script: |
      const { status, limit } = args;
      const list = document.querySelector('#bug_list');
      if (!list) return { bugs: [], total: 0, _error: '#bug_list not found' };
      // …extract rows, filter, return
      return { bugs, total };

  add_comment:
    destructive: true        # extension prompts before running
    description: "Add a note to a bug."
    parameters:
      bug_id: { type: number, required: true }
      text:   { type: string, required: true }
    page:
      url: "{base_url}/bug_view_page.php?bug_id={args.bug_id}"
    script: |
      // Form-submit or fetch() with same-origin cookies
      …
```

**Template variables**:
- `{base_url}` / `{settings.<name>}` → expanded server-side from saved settings
- `{args.<name>}` → expanded by the runtime from the LLM's tool input

**Forge-file args (`scratch://` connector-ref protocol)**: any tool arg
whose whole value is a `scratch://<path>` reference is read from
`<dataDir>/<path>` and replaced with the file's **base64** content,
server-side, before the tool runs. The `scratch://` prefix is the
ref protocol name (historical — not tied to the `scratch/` subdir); the
path after it is **dataDir-relative**, so `scratch://tmp/foo.md`,
`scratch://scratch/report.md`, `scratch://flows/x.yaml` all work.
Sensitive items (encryption keys, sqlite DBs, server log, `*-tokens.json`
at root) are refused. Bare-form references (`scratch/<file>` and
`/api/scratch/<file>`) are accepted for back-compat — same dataDir-rooted
resolution. This lets the chat agent hand a saved file straight to a
connector (e.g. `onedrive.upload_file` `content_base64:
"scratch://tmp/report.md"`) without hand-encoding it — hand-encoding
large or non-ASCII content is unreliable. A missing file fails the call
with a clear error (no retry). The companion builtin `read_forge_file`
pulls Forge-managed file content back into chat context (same dataDir-
relative paths, same sensitive blacklist). For *uploading*, prefer the
`scratch://` ref so the bytes never round-trip through the model.

**`script` contract**:
- Receives `args` (the LLM's parameters)
- Returns a JSON-serializable value (no DOM nodes, no functions)
- Has access to `document`, `fetch`, `URL`, etc. (page context)
- Same-origin `fetch()` — cookies auto-attached
- Self-contained — no closures over the manifest's surroundings
- Uncaught throws become tool errors

## Server-side protocols (`http`, `shell`)

When a service has a clean REST API and you don't need a browser tab at
all, declare `protocol: http` on the tool. Forge issues the request
server-side and returns the body to the LLM. Same for `protocol: shell`
— Forge spawns a process with an explicit arg array (no `shell:true`,
so templated values cannot inject metacharacters).

```yaml
tools:
  get_repo:
    protocol: http
    parameters:
      repo: { type: string, required: true }
    request:
      method: GET
      url: 'https://api.github.com/repos/{args.repo}'
      headers:
        Accept: 'application/vnd.github+json'
        Authorization: 'Bearer {settings.token}'

  git_log:
    protocol: shell
    parameters:
      repo: { type: string, required: true }
      n: { type: number }
    command: ['git', '-C', '{args.repo}', 'log', '-n', '{args.n}', '--oneline']
    timeout_ms: 5000
```

Response body / stdout is truncated to ~8 KB for the LLM context;
default timeout 30 s, max 5 min. `protocol: browser` (the default) runs
in the extension; `http` and `shell` run entirely on Forge (chat-standalone,
port 8408).

**Safety:** `protocol: shell` lets a manifest invoke any binary on PATH.
Always read the script before installing.

## 1:1 vs 1:N suites

The default is **one connector = one tool surface** (Mantis, GitLab,
Teams, …). For same-vendor suites with shared auth (Atlassian, Google
Workspace, M365), a manifest can declare multiple `connectors[]`
entries that share user settings:

```yaml
id: atlassian-suite
connectors:
  - id: jira
    host_match: "*://*.atlassian.net/*"
    tools: { … }
  - id: confluence
    host_match: "*://*.atlassian.net/wiki/*"
    tools: { … }
```

Most manifests should stay 1:1. Use 1:N only for genuine shared-auth suites.

## HTTP API

All endpoints require `X-Forge-Token` (obtain via `POST /api/auth/verify`).

### `GET /api/connectors`
List installed connectors. `page.url` / `host_match` etc. are expanded
from the user's settings; `{args.*}` stays literal for the runtime.

```json
{
  "connectors": [
    {
      "plugin_id": "mantis",         /* legacy wire-name; equals id */
      "name": "MantisBT",
      "icon": "🐞",
      "version": "0.5.0",
      "installed": true,
      "host_match": "https://mantis.acme.com/*",
      "login_redirect": "/login_page.php",
      "runner": "main",
      "entries": [
        {
          "id": "mantis",
          "tools": { … },
          "settings": { "base_url": { … } }
        }
      ]
    }
  ]
}
```

### `GET /api/connectors?id=<id>`
Single connector detail, wrapped in `{ connector: … }`.

### `GET /api/connectors/<id>/settings`
Read the user's saved settings. Secret fields come back as bullets.

```json
{
  "settings": { "base_url": "https://mantis.acme.com", "token": "••••••••" },
  "schema":   { "base_url": { … },                       "token": { "type": "secret", … } },
  "installed": true
}
```

### `POST /api/connectors/<id>/settings`
Save settings. If the client sends `••••••••` for a secret, the stored
value is preserved (so editing one field doesn't wipe the PAT).

### `GET /api/connectors/marketplace`
List registry entries merged with installed state.

```json
{
  "fetched_at": "2026-05-19T16:31:02.000Z",
  "base_url":   "https://raw.githubusercontent.com/aiwatching/forge-connectors/main",
  "entries": [
    { "id": "mantis", "version": "0.5.0", "installed_version": "0.5.0", "update_available": false, … },
    { "id": "gitlab", "version": "1.1.0", "installed_version": "1.0.0", "update_available": true,  … },
    { "id": "teams",  "version": "0.8.1",                                /* available */            … }
  ]
}
```

### `POST /api/connectors/marketplace`
Marketplace actions.

| `action` | Effect |
|---|---|
| `sync` | Re-fetch `registry.json` (and refresh installed manifests with `refreshInstalled: true`) |
| `install` | Fetch the manifest for `id` and write it to `<dataDir>/connectors/<id>/` |
| `update` | Alias for `install` — pulls the latest manifest |
| `uninstall` | Drop the manifest. `keepConfig: true` (default) preserves the user's settings + PAT |
| `status` | Probe a single `id`: returns `{ installed, installed_version, enabled }` |

## Authoring a new connector

Three ways to land a custom connector:

1. **Local-only (fastest)** — write a `manifest.yaml`, upload it via
   the "+ Upload" button in Settings → Connectors (or drag-drop). For
   multi-file connectors, zip it with `manifest.yaml` at the root.
   See `21-build-connector.md` for the manifest schema and an
   interview-style template.
2. **Forge Help AI** — ask Forge chat "build me a connector for X"; the
   AI walks through requirements, writes the manifest to
   `<dataDir>/connectors/<id>/manifest.yaml`, and registers it via
   `POST /api/connectors/install-local`.
3. **Share via forge-connectors** — fork
   [`forge-connectors`](https://github.com/aiwatching/forge-connectors),
   add `<id>/manifest.yaml`, add an entry to `registry.json`, open a PR.
   Maintainers review the script bodies before merge.

Iterating on a private registry: point `settings.connectorsRepoUrl` at
a private fork.

### POST /api/connectors/install-local

Used by both the Upload button and the Help AI:

```http
POST /api/connectors/install-local
Content-Type: application/json
X-Forge-Token: …

{ "yaml": "id: my-jira\nname: MyJira\nversion: \"0.1.0\"\n..." }
```

or `multipart/form-data` with a `file` field accepting `.yaml`,
`.yml`, or `.zip`. Zip must have `manifest.yaml` at the root.

### POST /api/connectors/import-config-template

Bulk-fill `connector-configs.json` from a JSON template. The Settings →
Connectors panel exposes this as the **↥ Import Template** button.

The template format:

```json
{
  "_README": "...",
  "_prompts": {
    "gitlab_pat": {
      "label": "GitLab PAT",
      "hint": "where to grab it",
      "secret": true,
      "required": true
    }
  },
  "gitlab": {
    "config": { "base_url": "https://...", "token": "${gitlab_pat}" },
    "enabled": true
  }
}
```

Behavior:

- Two-phase: `multipart/form-data` (analyze) returns deduplicated list of
  `${key}` placeholders the user still needs to fill. `application/json
  {template, values}` (apply) substitutes and merges.
- Same placeholder key referenced from multiple connectors (e.g.
  `gitlab_pat` used by `gitlab.token` AND `jenkins.instances[0].gitlab_pat`)
  is asked **once** and applied to every target.
- Static values (non-placeholder) are applied as-is.
- **Existing non-empty fields are preserved** — the template never
  overwrites a real token a user already configured (a `TODO_*` carry-over
  is treated as empty and gets overwritten).
- Connectors whose manifest is not installed yet are skipped and reported
  in the result. Sync the marketplace first, then re-import.
- Keys starting with `_` (`_README`, `_prompts`, etc.) are metadata.

Forge ships a default template at `templates/connector-config-template.json`,
bundled into the build. The Import button always works out of the box.
Users can override the bundled one by dropping their own at
`<dataDir>/config-template.json` — the GET endpoint picks the override
first, else falls back to the bundle.

Each prompt may include `url` + `url_label`, surfaced in the modal as
a "↗ Get token" link next to the field so the user can jump straight
to the page that issues the value.

## Migration from pre-v0.9

Pre-v0.9 Forge stored connectors as built-in plugins under
`lib/builtin-plugins/<id>.yaml` with `category: connector`. On upgrade:

1. `migrateConnectorConfigs()` runs once per boot. For each known id
   (mantis, gitlab, teams, pmdb, github-api), if there's a row in
   `plugin-configs.json`, the encrypted config blob is moved to
   `connector-configs.json` and the old row is removed.
2. The next `syncRegistry({ refreshInstalled: true })` pulls the
   matching manifest from `forge-connectors` and writes it to
   `<dataDir>/connectors/<id>/manifest.yaml`.
3. Chat tools come back online ~1 s after migration completes. PAT
   and base URL are preserved across the upgrade.

If the user is offline at upgrade time, the marketplace will be empty
until the network returns; config rows remain, so the next sync
restores everything without re-entering credentials.

## Troubleshooting

| Symptom | Cause + fix |
|---|---|
| Marketplace is empty | Initial sync failed (offline / bad URL). Click **Refresh** in Settings → Connectors. |
| `Sync error: … self-signed certificate in certificate chain` | Corporate TLS-interception proxy (Zscaler, Palo Alto, etc.). Node doesn't trust the proxy's root CA by default. Export `NODE_EXTRA_CA_CERTS=/path/to/corporate-ca.pem` before starting Forge, or run `forge server restart` with that env var set. macOS users can extract the bundle from Keychain Access → System Roots. |
| `Sync error: … ENOTFOUND` | DNS can't resolve `raw.githubusercontent.com` — VPN / firewall blocking. Point `connectorsRepoUrl` at a private mirror, or sync manifests manually into `<dataDir>/connectors/<id>/manifest.yaml`. |
| Installed connector missing from chat | Manifest deleted but config row remains. Re-install from the marketplace. |
| "Update available" badge stuck | Network failure on the refresh. Re-sync via **Refresh**. |
| Tool not appearing for the LLM | Connector installed but disabled. Toggle it on in the Settings panel. |
| Tool returns "login required" | `login_redirect` substring matched the tab URL after nav — user needs to log in to that site. |
| Strict-CSP site refuses `new Function` | Page CSP blocks dynamic eval. Set `runner: isolated` in the manifest, or rewrite the tool as `protocol: http`. |
| `POST /api/connectors/<id>/settings` returns 404 | Manifest not on disk (`<dataDir>/connectors/<id>/manifest.yaml`). Install the connector first. |
