# Connector — Declarative Extraction Spec

> **Goal**: Adding a new browser-side connector should be a Forge-only change.
> No `pnpm build && release` of the browser extension required.
>
> **Today**: Forge ships the manifest (schema), Extension ships a hand-written
> handler (`mantis.ts`, eventually `gitlab.ts`, …). Adding a new connector
> means coordinated changes in two repos.
>
> **Target**: Manifest carries the extraction script. Extension is a generic
> runner that knows nothing about specific connectors.

---

## 1. Architecture before / after

### Today
```
Forge                            Extension
─────                            ─────────
mantis.yaml      ──────────────▶ list of tool schemas
(schema only)                    │
                                 │ LLM calls mantis.list_my_bugs
                                 ▼
                                 mantis.ts (hand-written, ~300 LOC)
                                  ├─ findMantisTab()         ◀── Mantis-specific
                                  ├─ ensureOnListPage()      ◀── Mantis URLs hardcoded
                                  ├─ extractBugList()        ◀── #bug_list selector hardcoded
                                  └─ Forge column quirks  ◀── per-instance hardcoded
                                 │
                                 ▼
                                 chrome.scripting.executeScript → result
```
**Adding GitLab**: write `gitlab.ts` in extension, ship new extension version.

### Target
```
Forge                                    Extension
─────                                    ─────────
mantis.yaml                              src/lib/connectors/runner.ts
  tools:                                 (≤ 100 LOC, generic)
    list_my_bugs:
      page: { url, on_target, ... }     │ LLM calls mantis.list_my_bugs
      script: |  ──────────────────────▶│
        const list = document.qs(...)   │
        return { bugs, ... }            ▼
                                        runConnectorTool(plugin, tool, args)
                                         ├─ acquireTab(host_match, url)
                                         ├─ waitForLoad
                                         ├─ detectLoginRedirect
                                         └─ executeScript(script, args) → result
```
**Adding GitLab**: drop `gitlab.yaml` in Forge with `page` + `script` per tool. Done.

---

## 2. Schema change to plugin YAML

A connector plugin's `tools` map gets two new keys per tool:
`page` (where to extract from) and `script` (the in-page function body).

```yaml
id: mantis
name: MantisBT
icon: "🐞"
version: "0.2.0"     # bump on schema change
category: connector
mode: browser-side

settings:
  base_url:
    type: string
    label: Mantis base URL
    required: true

# plugin-level: how to find / open an authenticated tab on this host
host_match: "{base_url}/*"      # Chrome match pattern, used to find existing tabs
login_redirect: "/login_page.php"   # substring; if URL contains this after nav, abort

tools:
  list_my_bugs:
    description: List bugs assigned to me.
    parameters:
      status:
        type: select
        options: ["open", "closed", "all"]
        default: "open"
      limit:
        type: number
        default: 50

    page:
      url: "{base_url}/view_all_bug_page.php"
      on_target: "/view_all_bug_page.php"   # substring match; if current URL has this, don't navigate

    # Function body. Receives `args` (the LLM's parsed parameters).
    # Returns a JSON-serializable value. Top-level Web APIs only.
    # NO closures over Forge / extension globals — this runs in the user's tab.
    script: |
      const { status, limit } = args;
      const list = document.querySelector('#bug_list');
      if (!list) return { bugs: [], total: 0, _error: '#bug_list not found' };
      // ... ports the body of mantis.ts/extractBugList() ...
      return { bugs, total, _page: location.href };
```

### Field reference

| Field | Level | Required | Notes |
|---|---|---|---|
| `host_match` | plugin | yes (browser-side) | Used by extension to find/open tabs. Chrome match pattern with `{settings.*}` substitution. |
| `login_redirect` | plugin | optional | Substring. If the tab URL contains this after navigation, treat as "user not logged in". |
| `page.url` | tool | yes | Template URL to navigate to. `{base_url}` etc. expanded from settings. |
| `page.on_target` | tool | optional | If current tab URL contains this substring, skip the navigation step (current page is fine). |
| `script` | tool | yes | Function body. Implicit param: `args`. Must return a JSON-serializable value. |

### Multi-step tools (defer)

`add_comment`-style tools that POST need form interaction (fill input + click submit + wait for redirect). For v1 we only spec read-only extraction. Destructive tools stay marked `destructive: true` but **`script` is allowed to call `fetch()` from within the tab context** (same-origin, cookies auto-attached) for cases where the rendered HTML form is too brittle. That's a per-connector decision the script author makes.

---

## 3. HTTP API change — `GET /api/connectors`

Today's response (per `app/api/connectors/route.ts`):
```json
{
  "connectors": [
    {
      "plugin_id": "mantis",
      "name": "MantisBT",
      "mode": "browser-side",
      "installed": true,
      "entries": [
        {
          "id": "mantis",
          "host_permissions": ["..."],
          "tools": {
            "list_my_bugs": {
              "description": "...",
              "parameters": {...},
              "returns": "..."
            }
          },
          "settings": {...}
        }
      ]
    }
  ]
}
```

After this change, **`tools[name]` must additionally include `page` and `script`** (and the plugin level needs `host_match` + `login_redirect`):

```json
{
  "connectors": [
    {
      "plugin_id": "mantis",
      "name": "MantisBT",
      "mode": "browser-side",
      "installed": true,
      "host_match": "https://mantis.acme.com/*",
      "login_redirect": "/login_page.php",
      "entries": [
        {
          "id": "mantis",
          "tools": {
            "list_my_bugs": {
              "description": "...",
              "parameters": {...},
              "page": {
                "url": "https://mantis.acme.com/view_all_bug_page.php",
                "on_target": "/view_all_bug_page.php"
              },
              "script": "const { status, limit } = args; ... return { bugs, total };"
            }
          },
          "settings": {...}
        }
      ]
    }
  ]
}
```

Note `{base_url}` / `{settings.base_url}` is **expanded server-side** using the user's saved settings before the response goes out — extension shouldn't need to know template syntax. (If the user hasn't saved settings yet, omit the tool from the listing or return `null` for `page.url`.)

---

## 4. Template substitution rules

Anywhere `page.url`, `host_match`, `login_redirect` appear, expand `{<key>}` tokens:

- `{base_url}` → `settings.base_url` (with trailing slashes stripped)
- `{settings.<name>}` → `settings.<name>` (verbose form)
- Unknown tokens → leave as literal `{foo}` (don't error — gives author a way to ship a static URL with curly braces)

Do this in Forge before returning the manifest. Avoids re-implementing on the extension side.

---

## 5. Script execution contract (what the extension will do)

The extension will:
1. Take `tool.script` as a string
2. Find or open a tab matching `host_match`, navigating to `page.url` unless `page.on_target` says current URL is fine
3. Detect `login_redirect` substring in tab URL → return `{ error: 'login required', loginRequired: true }`
4. Run via `chrome.scripting.executeScript`:
   ```js
   // executes IN THE USER'S TAB (Mantis page context)
   func: function(scriptText, args) {
     try {
       const fn = new Function('args', scriptText);
       return { ok: true, value: fn(args) };
     } catch (err) {
       return { ok: false, error: err.message };
     }
   }
   ```
5. Return `result.value` to the LLM as the tool result

**Script authors should assume**:
- `document`, `fetch`, `URL`, etc. are available
- Same-origin cookies are attached to `fetch()` calls
- No access to `chrome.*` APIs (running in page context)
- No closures over Forge/extension code — script is a self-contained function body
- Return value must be JSON-serializable (no DOM nodes, no functions)
- Errors should be caught and returned as `{ _error: '...' }` shape; thrown errors are also caught by the runner

---

## 6. Security considerations

This change moves trust: extension now executes arbitrary JS shipped by Forge. Constraints:

| Risk | Mitigation |
|---|---|
| Compromised Forge serves malicious script | Forge is already a trusted authenticated peer (`X-Forge-Token`). User explicitly chose to connect. Same trust level as `npm install`. |
| Script reads from / writes to sites it shouldn't | Run only in the tab matching `host_match`; never inject into arbitrary tabs. |
| `new Function` blocked by page CSP | Most internal corp tools have permissive CSPs. Strict-CSP sites (github.com, banks) will fail; document as a known limitation. For those, write a server-side connector instead (REST with token). |
| Script exfiltrates page content | Already inherent to the "scrape user's page" design — script could always do this. Extension UI should log connector tool calls so user can audit. |

**Recommendation for v1**: log every script run to extension console with `{plugin_id, tool, tab_url, script_length}`. Add a "Connector audit log" panel in Settings later.

---

## 7. Concrete: full migration of `mantis.yaml`

The current `mantis.ts` in extension has these page-script blocks:

- `extractBugList(args)` — for `list_my_bugs`, runs on `/view_all_bug_page.php`
- `extractBugDetail(args)` — for `get_bug`, runs on `/bug_view_page.php?bug_id=N`

These two functions get **copy-pasted into the YAML's `script` fields** (minus type annotations — script is plain JS). Their function bodies are already self-contained, so the port is mechanical:

```yaml
id: mantis
name: MantisBT
version: "0.2.0"
category: connector
mode: browser-side

settings:
  base_url: { type: string, required: true, label: "Mantis base URL" }
  default_project: { type: string, label: "Default project" }

host_match: "{base_url}/*"
login_redirect: "/login_page.php"

tools:
  list_my_bugs:
    description: |
      List bugs assigned to the current user.
    parameters:
      status:
        type: select
        options: ["open", "closed", "all"]
        default: "open"
      limit:
        type: number
        default: 50
    page:
      url: "{base_url}/view_all_bug_page.php"
      on_target: "/view_all_bug_page.php"
    script: |
      // Body of current extractBugList in src/lib/connectors/mantis.ts
      // (paste lines 69–148 of mantis.ts here, drop the function signature)
      const list = document.querySelector('#bug_list');
      if (!list) return { bugs: [], total: 0, _error: '#bug_list not found' };
      // ... etc
      return { bugs: filtered.slice(0, args.limit), total: filtered.length };

  get_bug:
    description: |
      Get full details of a single bug.
    parameters:
      bug_id:
        type: number
        required: true
    page:
      # Note: bug_id is in args, so we use it in the URL via template
      # extension expands `{args.bug_id}` at run-time (this is the one case
      # where args, not settings, drives the URL — special-case in spec).
      url: "{base_url}/bug_view_page.php?bug_id={args.bug_id}"
      on_target: "/bug_view_page.php"
    script: |
      // Body of current extractBugDetail
      ...
```

> ⚠ **`{args.*}` substitution in URLs is the one extra template feature** —
> needed for tools that target a specific record by ID. Forge needs to do
> two passes: settings-time substitution at API response time, then args-time
> substitution inside the extension when actually executing the tool. (Or
> Forge can leave `{args.*}` literals unexpanded; the extension finishes
> them.) **Recommendation**: leave `{args.*}` for the extension to expand
> from the LLM's input; Forge only expands `{settings.*}` / `{base_url}`.

The `mantis.probe.js` file stays as a dev tool; it's not shipped to extension at runtime.

---

## 8. Migration order

To avoid breaking the extension during the change:

1. **Forge: add the schema fields, no behavior change**
   - Extend manifest parser to accept `page` + `script` + `host_match` + `login_redirect`. Tools missing them are still valid (old shape).
   - Update `/api/connectors` to emit the new fields when present, omit when absent.
   - Mantis manifest gains `page` + `script` for both tools. Bump version `0.1.1` → `0.2.0`.
   - Verify with curl: `/api/connectors/mantis` returns the new shape, fields populated.

2. **Extension: write the generic runner alongside the existing per-connector handler**
   - New file `src/lib/connectors/runner.ts` implementing `runConnectorTool(connector, toolName, input)`.
   - Update `build-tools.ts` so a tool is "executable" if its manifest has `script` (not "if `hasHandler(plugin_id)`").
   - Routing: if tool has `script` → runner; else (fallback) → old `dispatchConnector(plugin_id, …)` for backward compat.

3. **Cut over**
   - Once the runner works for `mantis.list_my_bugs` end-to-end, delete `src/lib/connectors/mantis.ts` and the `mantis` entry in `registry.ts`.
   - Remove the fallback branch in routing.

4. **Add second connector to prove the loop**
   - Pick the simplest GitLab page (e.g. "my issues") and write the YAML. Confirm extension picks it up with zero extension-side changes.

---

## 9. Acceptance criteria

A connector is "shippable via Forge-only changes" iff:

- ✅ Adding a new `<name>.yaml` in `lib/builtin-plugins/` makes the tool visible in extension on next Connectors-tab refresh.
- ✅ Saving connector settings (`base_url`) makes the tool callable by the LLM.
- ✅ LLM calls succeed end-to-end without rebuilding/reloading the extension binary.
- ✅ Existing `mantis` connector continues to work throughout the migration (no flag day).
- ✅ Bumping a connector's `version` and changing only the YAML's `script` body propagates to all extension users on next refresh — no extension release needed.

---

## 10. Open questions for Forge side

- [ ] Where does the YAML's `script` body actually live in a multi-line YAML? Confirm Forge's YAML parser preserves it verbatim (it should — block scalar `|` is the right syntax).
- [ ] For users on `installed=false`, omit `page.url` / `script` from the response, or include them so the marketplace UI can preview? (Recommendation: include, but with `{settings.base_url}` left literal, so consumer can preview.)
- [ ] Probe scripts (`mantis.probe.js`) — keep as devtool only, or also serve via `/api/connectors/<id>/probe` so the extension can re-run probe on a "selectors broken" report? (Recommendation: defer — solve when selectors actually break.)
