# Forge change request — declarative connector extraction

> **Read this first**: full design rationale lives in
> [`Connector-DeclarativeExtract-Spec.md`](./Connector-DeclarativeExtract-Spec.md).
> This file is the **change list** for the Forge code + a ready-to-commit
> `mantis.yaml` so you can ship without further design discussion.

## TL;DR

Today: adding a connector means coordinated changes in Forge (manifest) **and** the browser extension (handler code).
Goal: adding a connector should be a **Forge-only YAML change**. Extension is now a generic runner; it has no Mantis/GitLab-specific code.

Status:
- Extension side is done. `src/lib/connectors/runner.ts` is a generic executor; `mantis.ts` / `registry.ts` / `types.ts` are deleted. The extension can run ANY connector whose manifest contains the new fields.
- **Forge side is what this doc is about.** Until Forge ships the new fields, the LLM can't see any connector tools (the extension's gate filters out tools without `script`).

## What Forge needs to change

### 1. Plugin manifest parser — accept four new fields

Allowed YAML at the plugin level:

| Field | Required | Notes |
|---|---|---|
| `host_match` | yes (for browser-side mode) | Chrome match pattern. `{settings.*}` template variables allowed. Used by extension to find or open an authenticated tab. |
| `login_redirect` | optional | Substring. If the tab URL contains this after navigation, the runner returns `loginRequired: true`. |

Allowed at the **tool** level (under `tools.<name>`):

| Field | Required | Notes |
|---|---|---|
| `page.url` | yes (executable tool) | URL the runner navigates to before running the script. `{settings.*}` AND `{args.*}` template variables allowed. |
| `page.on_target` | optional | Substring. If current tab URL already contains this, skip the navigation step. |
| `script` | yes (executable tool) | Function body. Receives `args` (LLM's parsed parameters), returns JSON-serializable. Runs in the user's tab. |

Tools missing `page` or `script` are still valid manifests — they just won't be callable by the LLM. (Useful for tools that you want to surface in the Connectors UI but haven't yet implemented.)

### 2. `GET /api/connectors[?id=<id>]` response — include the new fields

The response body must include the new fields on every connector and every tool. Settings-template substitution happens server-side; args-template substitution happens client-side.

```jsonc
// GET /api/connectors  →
{
  "connectors": [
    {
      "plugin_id": "mantis",
      "name": "MantisBT",
      "icon": "🐞",
      "version": "0.2.0",
      "mode": "browser-side",
      "installed": true,

      // NEW — expanded from settings.base_url on the server side
      "host_match": "https://mantis.acme.com/*",
      "login_redirect": "/login_page.php",

      "entries": [
        {
          "id": "mantis",
          "tools": {
            "list_my_bugs": {
              "description": "...",
              "parameters": {...},
              // NEW — also expanded server-side
              "page": {
                "url": "https://mantis.acme.com/view_all_bug_page.php",
                "on_target": "/view_all_bug_page.php"
              },
              // NEW — verbatim string (no template substitution; runs as JS)
              "script": "const list = document.querySelector('#bug_list');\n..."
            }
          },
          "settings": {...}
        }
      ]
    }
  ]
}
```

### 3. Template substitution

Forge expands at API response time:
- `{base_url}` → `settings.base_url`
- `{settings.<name>}` → `settings.<name>`
- Strip trailing slashes from `base_url` before substituting (so `{base_url}/foo` doesn't double-slash)

Forge does **not** expand `{args.*}`. The extension does that at tool-call time. (It also URL-encodes `{args.*}` substitutions inside URLs to be safe.) Forge can ship `{args.bug_id}` literally inside `page.url`.

If a tool's `page.url` contains an unexpanded `{settings.foo}` because the user hasn't saved that setting yet, return the tool with `page.url` unchanged (literal `{settings.foo}`). The extension will surface a clear error when called.

### 4. `installed=false` connectors

For uninstalled connectors, the user hasn't saved `base_url` etc. Return `page.url` / `host_match` with `{settings.*}` left literal so the Connectors marketplace UI can still preview the manifest.

### 5. Plugin id, version, mode

Bump version on every manifest content change. The extension's Connectors tab uses `version` to flag stale cached manifests on refresh. Mode stays `browser-side` for these.

## Acceptance criteria

✅ Adding `lib/builtin-plugins/foo.yaml` with `host_match` + `tools.bar.page` + `tools.bar.script` makes `foo.bar` callable by the LLM after extension refresh, with no extension code changes.

✅ Bumping a `script` body in a YAML and restarting Forge propagates to all extension users on next Connectors tab refresh — no extension release.

✅ Old manifests (no `page` / `script` / `host_match`) parse without error; tools just don't show up as callable.

✅ `mantis.list_my_bugs` works end-to-end:
   1. User asks "list my Mantis bugs" in chat
   2. LLM calls `mantis.list_my_bugs`
   3. Extension runner: finds existing Mantis tab OR opens background tab on `host_match`
   4. Runner navigates to `page.url`
   5. Runner executes the `script` body in MAIN world via `chrome.scripting.executeScript`
   6. LLM gets `{bugs: [...]}` back, summarizes for the user

## Full `mantis.yaml` — ready to commit

Drop this in `lib/builtin-plugins/mantis.yaml`. It contains the same five tools as before, with the extension's old `extractBugList` / `extractBugDetail` / `extractProjects` / `addBugnoteInPage` bodies inlined as `script`. Tested against Mantis 1.x classic (Forge self-hosted theme).

```yaml
id: mantis
name: MantisBT
icon: "🐞"
version: "0.2.0"
author: forge
category: connector
mode: browser-side
description: |
  Read + comment on MantisBT bugs from the user's logged-in browser session.
  Reuses the existing PHPSESSID cookie — no API token required.

  All tools run as DOM extraction scripts inside the user's Mantis tab,
  shipped by Forge in this manifest. The extension is generic.

settings:
  base_url:
    type: string
    label: Mantis base URL
    description: "e.g. https://mantis.mycompany.com (no trailing slash)"
    required: true
  default_project:
    type: string
    label: Default project name
    description: "Optional — used when no project specified"

host_match: "{base_url}/*"
login_redirect: "/login_page.php"

tools:
  list_my_bugs:
    description: |
      List bugs assigned to the current user. Returns id, summary, status,
      severity, priority, project, last_updated.
    parameters:
      status:
        type: select
        options: ["open", "feedback", "acknowledged", "confirmed", "assigned", "resolved", "closed", "all"]
        default: "open"
      limit:
        type: number
        default: 50
    page:
      url: "{base_url}/view_all_bug_page.php"
      on_target: "/view_all_bug_page.php"
    script: |
      const list = document.querySelector('#bug_list');
      if (!list) {
        return { bugs: [], total: 0, _error: '#bug_list not found on ' + location.pathname };
      }
      const allRows = Array.from(list.querySelectorAll('tr'));
      const headerRow = allRows.find(r => r.className === 'row-category');
      const dataRows = allRows.filter(r => r.querySelector('input[name*="bug"]'));
      const headerIdx = {};
      if (headerRow) {
        Array.from(headerRow.querySelectorAll('td, th')).forEach((c, i) => {
          const name = (c.textContent || '').trim().toLowerCase();
          if (name) headerIdx[name] = i;
        });
      }
      const col = (cells, header) => {
        const idx = headerIdx[header.toLowerCase()];
        if (idx === undefined) return '';
        return ((cells[idx] && cells[idx].innerText) || '').trim();
      };
      const STATUS_RE = /^(new|feedback|acknowledged|confirmed|assigned|open|resolved|closed|reopened)$/i;
      const FILTERS = {
        open:   /^(new|feedback|acknowledged|confirmed|assigned|open|reopened)$/i,
        closed: /^(resolved|closed)$/i,
        all:    /./,
      };
      const all = dataRows.map(r => {
        const cells = Array.from(r.querySelectorAll('td'));
        const checkbox = r.querySelector('input[type="checkbox"][name*="bug"]');
        const link = r.querySelector('a[href*="bug_view_page.php?bug_id="]');
        const id = checkbox && checkbox.value
          ? Number(checkbox.value)
          : Number((link && link.href.match(/bug_id=(\d+)/) || [])[1] || 0);
        if (!id) return null;
        // Forge quirk: "Status" column shows handler; real status is in "Resolution"
        let status = col(cells, 'Status');
        if (!STATUS_RE.test(status)) {
          const resolution = col(cells, 'Resolution');
          if (STATUS_RE.test(resolution)) status = resolution;
        }
        return {
          id,
          summary: (link && link.getAttribute('title') && link.getAttribute('title').trim()) || col(cells, 'Summary'),
          status,
          priority: col(cells, 'Priority'),
          severity: col(cells, 'Severity') || col(cells, 'S'),
          keywords: col(cells, 'Keyword') || col(cells, 'Category'),
          reporter: col(cells, 'Reporter'),
          assignee: (col(cells, 'Status') || col(cells, 'Handler') || col(cells, 'Assigned To')).replace(/^\(|\)$/g, '').trim(),
          source: col(cells, 'Source'),
          product_version: col(cells, 'Reported Version') || col(cells, 'Product Version'),
          fix_schedule: col(cells, 'Fix Schedule'),
          last_updated: col(cells, 'Updated') || col(cells, 'Last Updated'),
          created_at: col(cells, 'Date Submitted') || col(cells, 'Date'),
          url: (link && link.href) || '',
        };
      }).filter(b => b !== null);
      const key = (args.status || 'all').toLowerCase();
      const filter = FILTERS[key] || FILTERS.all;
      const filtered = all.filter(b => filter.test(b.status));
      return {
        bugs: filtered.slice(0, args.limit || 50),
        total: filtered.length,
        rawCount: all.length,
        _page: location.href,
      };

  get_bug:
    description: |
      Get full details of a single bug — description, all notes, history.
    parameters:
      bug_id:
        type: number
        required: true
    page:
      url: "{base_url}/bug_view_page.php?bug_id={args.bug_id}"
      on_target: "/bug_view_page.php"
    script: |
      const detailsTitle = Array.from(document.querySelectorAll('.form-title'))
        .find(e => e.textContent && e.textContent.trim().startsWith('Viewing Bug'));
      const detailsTable = detailsTitle ? detailsTitle.closest('table') : null;
      if (!detailsTable) return { _error: 'bug detail table not found on ' + location.pathname };
      const rows = Array.from(detailsTable.querySelectorAll('tr'));
      const fields = {};
      if (rows[1] && rows[2]) {
        const labelCells = Array.from(rows[1].querySelectorAll('td'));
        const valueCells = Array.from(rows[2].querySelectorAll('td'));
        labelCells.forEach((l, i) => {
          const label = l.innerText.trim();
          const val = valueCells[i] ? valueCells[i].innerText.trim() : '';
          if (label && !fields[label]) fields[label] = val;
        });
      }
      Array.from(document.querySelectorAll('td.category')).forEach(label => {
        const labelText = (label.textContent || '').trim();
        if (labelText.includes('\n') && labelText.length > 100) return;
        if (labelText.startsWith('Select File')) return;
        if (labelText === 'Bugnote') return;
        const valueEl = label.nextElementSibling;
        if (!valueEl || valueEl.tagName !== 'TD' || valueEl.classList.contains('category')) return;
        const val = valueEl.innerText.trim();
        if (!fields[labelText]) fields[labelText] = val;
      });
      const notesTitle = Array.from(document.querySelectorAll('.form-title'))
        .find(e => e.textContent && e.textContent.trim().startsWith('Bug Notes'));
      const notesTable = notesTitle && notesTitle.closest('table');
      const notes = Array.from((notesTable && notesTable.querySelectorAll('tr.bugnote')) || []).map((r, i) => {
        const cells = r.querySelectorAll('td');
        const header = (cells[0] && cells[0].innerText.trim()) || '';
        const body   = (cells[1] && cells[1].innerText.trim()) || '';
        const m = header.match(/^(.*?)\n(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})/);
        return {
          idx: i,
          author: (m && m[1]) ? m[1].trim() : (header.split('\n')[0] || '').trim(),
          date:   (m && m[2]) || '',
          body,
        };
      });
      const historyTitle = Array.from(document.querySelectorAll('.form-title'))
        .find(e => e.textContent && e.textContent.trim().startsWith('Bug History'));
      const historyTable = historyTitle && historyTitle.closest('table');
      const history = Array.from((historyTable && historyTable.querySelectorAll('tr')) || [])
        .filter(r => r.querySelectorAll('td').length === 4)
        .slice(1)
        .map(r => {
          const c = Array.from(r.querySelectorAll('td')).map(x => x.innerText.trim());
          return { date: c[0], user: c[1], field: c[2], change: c[3] };
        });
      const id = Number(fields.ID || args.bug_id);
      return {
        id,
        summary: fields.Summary || '',
        description: fields.Description || '',
        additional_information: fields['Additional Information'] || '',
        status: fields.Status || '',
        resolution: fields.Resolution || '',
        priority: fields.Priority || '',
        severity: fields.Severity || '',
        reproducibility: fields.Reproducibility || '',
        category: fields.Category || '',
        keywords: fields.Keyword || '',
        reporter: fields.Reporter || '',
        assignee: fields['Assigned To'] || '',
        qa_assignee: fields['QA Assignee'] || '',
        view_status: fields['View Status'] || '',
        eta: fields.ETA || '',
        date_submitted: fields['Date Submitted'] || '',
        last_update: fields['Last Update'] || '',
        reported_version: fields['Reported Version'] || '',
        fix_schedule: fields['Fix Schedule'] || '',
        source: fields.Source || '',
        tag: fields.Tag || '',
        notes,
        history,
        url: location.href,
      };

  search_bugs:
    description: |
      Search bugs by text query. Mantis 1.x classic does this via
      view_all_set.php which sets a temporary filter and redirects to
      view_all_bug_page.php with results.
    parameters:
      query:
        type: string
        required: true
      status:
        type: string
        description: "Optional Mantis status keyword: new|feedback|acknowledged|confirmed|assigned|resolved|closed"
      limit:
        type: number
        default: 25
    page:
      # No on_target — we always want to re-set the filter on each call.
      # {args.query} URL-encoded by the extension at call time.
      url: "{base_url}/view_all_set.php?type=1&temporary=y&search={args.query}"
    script: |
      // After view_all_set.php's filter is set, Mantis redirects to
      // view_all_bug_page.php. The script body is identical to list_my_bugs
      // but with `status` forced to 'all' so the in-page filter governs.
      const list = document.querySelector('#bug_list');
      if (!list) return { bugs: [], total: 0, _error: '#bug_list not found' };
      const allRows = Array.from(list.querySelectorAll('tr'));
      const headerRow = allRows.find(r => r.className === 'row-category');
      const dataRows = allRows.filter(r => r.querySelector('input[name*="bug"]'));
      const headerIdx = {};
      if (headerRow) {
        Array.from(headerRow.querySelectorAll('td, th')).forEach((c, i) => {
          const name = (c.textContent || '').trim().toLowerCase();
          if (name) headerIdx[name] = i;
        });
      }
      const col = (cells, header) => {
        const idx = headerIdx[header.toLowerCase()];
        if (idx === undefined) return '';
        return ((cells[idx] && cells[idx].innerText) || '').trim();
      };
      const STATUS_RE = /^(new|feedback|acknowledged|confirmed|assigned|open|resolved|closed|reopened)$/i;
      const all = dataRows.map(r => {
        const cells = Array.from(r.querySelectorAll('td'));
        const checkbox = r.querySelector('input[type="checkbox"][name*="bug"]');
        const link = r.querySelector('a[href*="bug_view_page.php?bug_id="]');
        const id = checkbox && checkbox.value
          ? Number(checkbox.value)
          : Number((link && link.href.match(/bug_id=(\d+)/) || [])[1] || 0);
        if (!id) return null;
        let status = col(cells, 'Status');
        if (!STATUS_RE.test(status)) {
          const resolution = col(cells, 'Resolution');
          if (STATUS_RE.test(resolution)) status = resolution;
        }
        return {
          id,
          summary: (link && link.getAttribute('title') && link.getAttribute('title').trim()) || col(cells, 'Summary'),
          status,
          severity: col(cells, 'Severity') || col(cells, 'S'),
          project: col(cells, 'Project') || col(cells, 'Category'),
          url: (link && link.href) || '',
        };
      }).filter(b => b !== null);
      const limit = args.limit || 25;
      return {
        query: args.query,
        bugs: all.slice(0, limit),
        total: all.length,
      };

  list_projects:
    description: List projects visible to the current user (from the project_id dropdown on the list page).
    page:
      url: "{base_url}/view_all_bug_page.php"
      on_target: "/view_all_bug_page.php"
    script: |
      const sel = document.querySelector('select[name="project_id"]');
      if (!sel) return { _error: 'project_id select not found on page' };
      const projects = Array.from(sel.options)
        .map(o => ({
          id: Number(o.value),
          name: (o.text || '').trim(),
          depth: ((o.text || '').match(/^[\s ]+/) || ['']).length,
        }))
        .filter(p => p.id > 0);
      return { projects, total: projects.length };

  add_comment:
    description: |
      Add a note (comment) to a bug. Posts to bugnote_add.php with the
      session cookie. Destructive — extension prompts the user first.
    destructive: true
    parameters:
      bug_id:
        type: number
        required: true
      text:
        type: string
        required: true
    page:
      url: "{base_url}/bug_view_page.php?bug_id={args.bug_id}"
      on_target: "/bug_view_page.php"
    script: |
      const body = new URLSearchParams({
        bug_id: String(args.bug_id),
        bugnote_text: args.text,
      });
      try {
        const r = await fetch('bugnote_add.php', {
          method: 'POST',
          headers: { 'Content-Type': 'application/x-www-form-urlencoded' },
          body: body.toString(),
          credentials: 'include',
          redirect: 'follow',
        });
        return {
          ok: r.ok,
          status: r.status,
          finalUrl: r.url,
          success: r.ok && r.url.indexOf('bug_id=' + args.bug_id) >= 0,
        };
      } catch (e) {
        return { ok: false, error: e.message };
      }
```

## Notes on the YAML

- **Block scalar `|`** preserves newlines verbatim. The `script` content is plain JS, no TypeScript syntax, no imports — runs as the body of `new Function('args', script)`.
- **No closures**: scripts can't reference variables outside `args` and the page's global Web APIs (`document`, `fetch`, etc.).
- **`async/await` allowed**: the runner uses `new Function('args', script)` which returns a value; if that value is a Promise, the runner awaits it. (See `runner.ts` — the wrapper handles both sync and async return values.) The `add_comment` script uses `await fetch(...)` — works fine.
- **`{args.foo}` substitution and URL encoding**: the extension encodes args values before substituting into URLs. So `{args.query}` with `q="hello world"` becomes `search=hello%20world`, not `search=hello world`.
- **Forge-specific column quirks** (Status vs Resolution fallback) are baked into the `list_my_bugs` script. If you have a non-Forge Mantis instance, you may need a separate `mantis-vanilla.yaml`.

## Open questions / nice-to-haves (not blockers)

- **Probe script delivery** — `mantis.probe.js` stays as dev-time tool. If we want auto-recovery from selector drift, expose `GET /api/connectors/<id>/probe` later.
- **Server-side `script` linting** — Forge could `new Function(scriptStr)` at load time to catch parse errors before the manifest ever ships to the extension. Optional.
- **Schema versioning** — when we change the manifest schema again in the future, bump a top-level `schema_version` and let the extension reject incompatible older shapes.

## What happens when Forge ships this

1. User refreshes the Connectors tab in the extension.
2. Extension fetches `/api/connectors`, sees `script` + `page` populated on each Mantis tool.
3. `buildConnectorTools()` un-filters those tools (they now satisfy the `tool.script` gate).
4. On the next chat turn, the LLM's tool list includes `mantis.list_my_bugs`, `mantis.get_bug`, `mantis.search_bugs`, `mantis.list_projects`, `mantis.add_comment`.
5. User asks "list my Mantis bugs" → LLM calls `mantis.list_my_bugs` → runner opens background Mantis tab (or reuses existing) → executes the YAML's `script` body in MAIN world → returns JSON to LLM → LLM summarizes.

No extension code changes required at any point in this flow.
