# Connectors

**Connectors** expose external services (Mantis, GitLab, Teams, …) as
tools that the Forge chat agent and the browser extension can call.
Each connector is a declarative YAML manifest — schema + the extraction
scripts that implement each tool — pulled from the remote
[`forge-connectors`](https://github.com/aiwatching/forge-connectors)
registry.

> **Architecture**: connectors are an independent subsystem (`lib/connectors/`).
> They are NOT plugins. Pipeline-node plugins (`lib/plugins/`) and connectors
> share no schema, no registry, no config file.

There are **no built-in connectors**. A fresh install starts with an empty
connector list; the user picks what they want from the marketplace.

## Marketplace flow

1. On boot, Forge fetches `registry.json` from `connectorsRepoUrl`
   (default `https://raw.githubusercontent.com/aiwatching/forge-connectors/main`)
   and caches it at `<dataDir>/connectors/registry-cache.json`.
2. The Settings → Connectors panel reads the cache and lists every
   available connector with its install state:
   - **Available** — in registry, not installed
   - **Installed v0.5.0** — manifest on disk, config row present
   - **Update 0.5 → 0.6** — registry has newer version
3. Clicking **Install** fetches `<id>/manifest.yaml` from the registry,
   writes it to `<dataDir>/connectors/<id>/manifest.yaml`, and adds a
   row to `<dataDir>/connector-configs.json`.
4. The user fills in settings (host URL, PAT, etc.) via the existing
   settings UI in the extension or `/api/connectors/<id>/settings`.
5. Chat tools become available the instant the manifest lands on disk —
   no Forge restart.

Sync re-runs every hour and on manual **Refresh** in the Settings panel.
Network failures are silent — the cache from the last successful sync
keeps the marketplace usable offline.

## Repository layout

The `forge-connectors` repo (forkable via `connectorsRepoUrl`) is:

```
registry.json
<id>/
  manifest.yaml      ← what Forge installs
  README.md          ← optional, surfaced in the marketplace
```

`registry.json` lists every connector with id/name/version/icon/description
+ `min_forge_version`. Forge only consults the registry for the install
flow; once installed, `<dataDir>/connectors/<id>/manifest.yaml` is the
source of truth for that user.

## Local data layout

```
<dataDir>/
├── connectors/
│   ├── registry-cache.json       last fetch from forge-connectors
│   ├── mantis/manifest.yaml      installed manifest copy
│   ├── gitlab/manifest.yaml
│   └── …
└── connector-configs.json        { "<id>": { config, installed_version, enabled } }
```

Secret fields (`type: secret`) in `config` are encrypted at rest with
AES-256-GCM via the existing key at `<dataDir>/.encrypt-key`. The on-wire
GET response masks them with bullets so plaintext never leaves the server.

## Execution model: DOM extraction, not REST

**Browser-side connectors do NOT call REST APIs.** They navigate the
user's tabs and parse rendered HTML via `chrome.scripting.executeScript`.
This is the whole point of routing through a browser extension — reuse
the user's already-authenticated UI session, with zero token management.

| System | REST API needs | Browser DOM needs |
|---|---|---|
| MantisBT | Per-user API token | Already-logged-in PHPSESSID cookie ✅ |
| GitLab UI | Personal Access Token | Session cookie on `/-/issues/` etc. ✅ |
| JIRA Server | PAT or basic auth | Session cookie ✅ |
| Teams web | MSAL bearer token | Session cookie ✅ |

This means:

- **No PAT / API tokens to manage** for browser connectors — they reuse
  the user's logged-in session.
- **Data never flows through Forge** — Forge only ships the manifest +
  extraction script. The LLM in the chat backend calls the tool, the
  extension scrapes in the user's tab, the LLM sees the result.
- Trade-off: site redesigns break scripts. Bump the manifest `version`
  in the registry and users pull the fix on next sync — no Forge or
  extension release needed.

## Manifest format

```yaml
id: mantis
name: MantisBT
icon: "🐞"
version: "0.5.0"
author: forge
description: |
  Read + comment on MantisBT bugs from the user's logged-in browser session.

min_forge_version: "0.8.0"

# Default extension execution context.
#   main     — permissive-CSP sites (Mantis, GitLab self-hosted)
#   isolated — strict-CSP sites that block eval (Teams, github.com)
runner: main

# Per-user settings (rendered as a form in the extension).
settings:
  base_url:
    type: string
    label: Mantis base URL
    required: true
  token:
    type: secret             # encrypted at rest (AES-256-GCM)
    label: API token (optional fallback)

# Chrome match pattern; {settings.*} expanded server-side.
host_match: "{base_url}/*"

# Substring detected after navigation → "user not logged in".
login_redirect: "/login_page.php"

tools:
  list_my_bugs:
    description: "List bugs assigned to me."
    parameters:
      status: { type: select, options: ["open", "closed", "all"], default: "open" }
      limit:  { type: number, default: 50 }

    # Default protocol is 'browser' — runs in the user's tab.
    page:
      url: "{base_url}/view_all_bug_page.php"
      on_target: "/view_all_bug_page.php"
    script: |
      const { status, limit } = args;
      const list = document.querySelector('#bug_list');
      if (!list) return { bugs: [], total: 0, _error: '#bug_list not found' };
      // …extract rows, filter, return
      return { bugs, total };

  add_comment:
    destructive: true        # extension prompts before running
    description: "Add a note to a bug."
    parameters:
      bug_id: { type: number, required: true }
      text:   { type: string, required: true }
    page:
      url: "{base_url}/bug_view_page.php?bug_id={args.bug_id}"
    script: |
      // Form-submit or fetch() with same-origin cookies
      …
```

**Template variables**:
- `{base_url}` / `{settings.<name>}` → expanded server-side from saved settings
- `{args.<name>}` → expanded by the runtime from the LLM's tool input

**`script` contract**:
- Receives `args` (the LLM's parameters)
- Returns a JSON-serializable value (no DOM nodes, no functions)
- Has access to `document`, `fetch`, `URL`, etc. (page context)
- Same-origin `fetch()` — cookies auto-attached
- Self-contained — no closures over the manifest's surroundings
- Uncaught throws become tool errors

## Server-side protocols (`http`, `shell`)

When a service has a clean REST API and you don't need a browser tab at
all, declare `protocol: http` on the tool. Forge issues the request
server-side and returns the body to the LLM. Same for `protocol: shell`
— Forge spawns a process with an explicit arg array (no `shell:true`,
so templated values cannot inject metacharacters).

```yaml
tools:
  get_repo:
    protocol: http
    parameters:
      repo: { type: string, required: true }
    request:
      method: GET
      url: 'https://api.github.com/repos/{args.repo}'
      headers:
        Accept: 'application/vnd.github+json'
        Authorization: 'Bearer {settings.token}'

  git_log:
    protocol: shell
    parameters:
      repo: { type: string, required: true }
      n: { type: number }
    command: ['git', '-C', '{args.repo}', 'log', '-n', '{args.n}', '--oneline']
    timeout_ms: 5000
```

Response body / stdout is truncated to ~8 KB for the LLM context;
default timeout 30 s, max 5 min. `protocol: browser` (the default) runs
in the extension; `http` and `shell` run entirely on Forge (chat-standalone,
port 8408).

**Safety:** `protocol: shell` lets a manifest invoke any binary on PATH.
Always read the script before installing.

## 1:1 vs 1:N suites

The default is **one connector = one tool surface** (Mantis, GitLab,
Teams, …). For same-vendor suites with shared auth (Atlassian, Google
Workspace, M365), a manifest can declare multiple `connectors[]`
entries that share user settings:

```yaml
id: atlassian-suite
connectors:
  - id: jira
    host_match: "*://*.atlassian.net/*"
    tools: { … }
  - id: confluence
    host_match: "*://*.atlassian.net/wiki/*"
    tools: { … }
```

Most manifests should stay 1:1. Use 1:N only for genuine shared-auth suites.

## HTTP API

All endpoints require `X-Forge-Token` (obtain via `POST /api/auth/verify`).

### `GET /api/connectors`
List installed connectors. `page.url` / `host_match` etc. are expanded
from the user's settings; `{args.*}` stays literal for the runtime.

```json
{
  "connectors": [
    {
      "plugin_id": "mantis",         /* legacy wire-name; equals id */
      "name": "MantisBT",
      "icon": "🐞",
      "version": "0.5.0",
      "installed": true,
      "host_match": "https://mantis.acme.com/*",
      "login_redirect": "/login_page.php",
      "runner": "main",
      "entries": [
        {
          "id": "mantis",
          "tools": { … },
          "settings": { "base_url": { … } }
        }
      ]
    }
  ]
}
```

### `GET /api/connectors?id=<id>`
Single connector detail, wrapped in `{ connector: … }`.

### `GET /api/connectors/<id>/settings`
Read the user's saved settings. Secret fields come back as bullets.

```json
{
  "settings": { "base_url": "https://mantis.acme.com", "token": "••••••••" },
  "schema":   { "base_url": { … },                       "token": { "type": "secret", … } },
  "installed": true
}
```

### `POST /api/connectors/<id>/settings`
Save settings. If the client sends `••••••••` for a secret, the stored
value is preserved (so editing one field doesn't wipe the PAT).

### `GET /api/connectors/marketplace`
List registry entries merged with installed state.

```json
{
  "fetched_at": "2026-05-19T16:31:02.000Z",
  "base_url":   "https://raw.githubusercontent.com/aiwatching/forge-connectors/main",
  "entries": [
    { "id": "mantis", "version": "0.5.0", "installed_version": "0.5.0", "update_available": false, … },
    { "id": "gitlab", "version": "1.1.0", "installed_version": "1.0.0", "update_available": true,  … },
    { "id": "teams",  "version": "0.8.1",                                /* available */            … }
  ]
}
```

### `POST /api/connectors/marketplace`
Marketplace actions.

| `action` | Effect |
|---|---|
| `sync` | Re-fetch `registry.json` (and refresh installed manifests with `refreshInstalled: true`) |
| `install` | Fetch the manifest for `id` and write it to `<dataDir>/connectors/<id>/` |
| `update` | Alias for `install` — pulls the latest manifest |
| `uninstall` | Drop the manifest. `keepConfig: true` (default) preserves the user's settings + PAT |
| `status` | Probe a single `id`: returns `{ installed, installed_version, enabled }` |

## Authoring a new connector

Three ways to land a custom connector:

1. **Local-only (fastest)** — write a `manifest.yaml`, upload it via
   the "+ Upload" button in Settings → Connectors (or drag-drop). For
   multi-file connectors, zip it with `manifest.yaml` at the root.
   See `21-build-connector.md` for the manifest schema and an
   interview-style template.
2. **Forge Help AI** — ask Forge chat "build me a connector for X"; the
   AI walks through requirements, writes the manifest to
   `<dataDir>/connectors/<id>/manifest.yaml`, and registers it via
   `POST /api/connectors/install-local`.
3. **Share via forge-connectors** — fork
   [`forge-connectors`](https://github.com/aiwatching/forge-connectors),
   add `<id>/manifest.yaml`, add an entry to `registry.json`, open a PR.
   Maintainers review the script bodies before merge.

Iterating on a private registry: point `settings.connectorsRepoUrl` at
a private fork.

### POST /api/connectors/install-local

Used by both the Upload button and the Help AI:

```http
POST /api/connectors/install-local
Content-Type: application/json
X-Forge-Token: …

{ "yaml": "id: my-jira\nname: MyJira\nversion: \"0.1.0\"\n..." }
```

or `multipart/form-data` with a `file` field accepting `.yaml`,
`.yml`, or `.zip`. Zip must have `manifest.yaml` at the root.

## Migration from pre-v0.9

Pre-v0.9 Forge stored connectors as built-in plugins under
`lib/builtin-plugins/<id>.yaml` with `category: connector`. On upgrade:

1. `migrateConnectorConfigs()` runs once per boot. For each known id
   (mantis, gitlab, teams, pmdb, github-api), if there's a row in
   `plugin-configs.json`, the encrypted config blob is moved to
   `connector-configs.json` and the old row is removed.
2. The next `syncRegistry({ refreshInstalled: true })` pulls the
   matching manifest from `forge-connectors` and writes it to
   `<dataDir>/connectors/<id>/manifest.yaml`.
3. Chat tools come back online ~1 s after migration completes. PAT
   and base URL are preserved across the upgrade.

If the user is offline at upgrade time, the marketplace will be empty
until the network returns; config rows remain, so the next sync
restores everything without re-entering credentials.

## Troubleshooting

| Symptom | Cause + fix |
|---|---|
| Marketplace is empty | Initial sync failed (offline / bad URL). Click **Refresh** in Settings → Connectors. |
| `Sync error: … self-signed certificate in certificate chain` | Corporate TLS-interception proxy (Zscaler, Palo Alto, etc.). Node doesn't trust the proxy's root CA by default. Export `NODE_EXTRA_CA_CERTS=/path/to/corporate-ca.pem` before starting Forge, or run `forge server restart` with that env var set. macOS users can extract the bundle from Keychain Access → System Roots. |
| `Sync error: … ENOTFOUND` | DNS can't resolve `raw.githubusercontent.com` — VPN / firewall blocking. Point `connectorsRepoUrl` at a private mirror, or sync manifests manually into `<dataDir>/connectors/<id>/manifest.yaml`. |
| Installed connector missing from chat | Manifest deleted but config row remains. Re-install from the marketplace. |
| "Update available" badge stuck | Network failure on the refresh. Re-sync via **Refresh**. |
| Tool not appearing for the LLM | Connector installed but disabled. Toggle it on in the Settings panel. |
| Tool returns "login required" | `login_redirect` substring matched the tab URL after nav — user needs to log in to that site. |
| Strict-CSP site refuses `new Function` | Page CSP blocks dynamic eval. Set `runner: isolated` in the manifest, or rewrite the tool as `protocol: http`. |
| `POST /api/connectors/<id>/settings` returns 404 | Manifest not on disk (`<dataDir>/connectors/<id>/manifest.yaml`). Install the connector first. |
