{"_id":"@agent-ctrl/cli","_rev":"3-bd2ff0ba9ec0cc637f0d1b5dbab8b7b0","name":"@agent-ctrl/cli","dist-tags":{"latest":"0.1.4"},"versions":{"0.1.1":{"name":"@agent-ctrl/cli","version":"0.1.1","keywords":["agent","automation","accessibility","uia","cli","windows","computer-use"],"license":"Apache-2.0","_id":"@agent-ctrl/cli@0.1.1","maintainers":[{"name":"k4cper-g","email":"k4cpergadomski@gmail.com"}],"homepage":"https://github.com/k4cper-g/agent-ctrl#readme","bugs":{"url":"https://github.com/k4cper-g/agent-ctrl/issues"},"bin":{"agent-ctrl":"bin/agent-ctrl.js"},"dist":{"shasum":"034f2c4134f11c882595817ad3e04830b83b6829","tarball":"https://registry.npmjs.org/@agent-ctrl/cli/-/cli-0.1.1.tgz","fileCount":9,"integrity":"sha512-E2Rsm7W5U34SJXuzGxAlCGaSD6ISYh6FrOeYECQqYq03+txVk26eBQPlLNYq8ezcYd35QVRQPTnXWx/V42Otfw==","signatures":[{"sig":"MEUCID2gYN+o2or9DQigByFjM/Z1H5EOmkKV2eWUwSxTxZdUAiEAyzN66eRq7U5FfdyFe/siJhfjFMFbnkEMADpGi3tEjUw=","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":4566826},"type":"module","engines":{"node":">=20"},"gitHead":"0e55198014ff4b4b65b3fabf71279353c78b551d","scripts":{"test":"npm run test --workspaces --if-present","build":"npm run build --workspaces --if-present","release":"npm run version:sync && npm run build:windows && npm publish --access=public","version":"npm run version:sync && git add Cargo.toml Cargo.lock","typecheck":"npm run typecheck --workspaces --if-present","postinstall":"node scripts/postinstall.js","build:native":"npm run version:sync && cargo build --release -p agent-ctrl-cli && node scripts/copy-native.js","version:sync":"node scripts/sync-version.js","build:windows":"npm run build:native","release:dryrun":"npm run version:sync && npm run build:windows && npm publish --dry-run --access=public"},"_npmUser":{"name":"k4cper-g","email":"k4cpergadomski@gmail.com"},"repository":{"url":"git+https://github.com/k4cper-g/agent-ctrl.git","type":"git"},"workspaces":["packages/*"],"_npmVersion":"11.11.0","description":"Cross-platform computer-use framework for AI agents.","directories":{},"_nodeVersion":"24.14.1","_hasShrinkwrap":false,"_npmOperationalInternal":{"tmp":"tmp/cli_0.1.1_1778350177740_0.892723683742598","host":"s3://npm-registry-packages-npm-production"}},"0.1.3":{"name":"@agent-ctrl/cli","version":"0.1.3","keywords":["agent","automation","accessibility","uia","cli","windows","computer-use"],"license":"Apache-2.0","_id":"@agent-ctrl/cli@0.1.3","maintainers":[{"name":"k4cper-g","email":"k4cpergadomski@gmail.com"}],"homepage":"https://github.com/k4cper-g/agent-ctrl#readme","bugs":{"url":"https://github.com/k4cper-g/agent-ctrl/issues"},"bin":{"agent-ctrl":"bin/agent-ctrl.js"},"dist":{"shasum":"2fd37fd9d9a1857072e14b55a78889949b597361","tarball":"https://registry.npmjs.org/@agent-ctrl/cli/-/cli-0.1.3.tgz","fileCount":9,"integrity":"sha512-pR3bhQu84BlcsO7rf4/GRaW6proihLQrQRitTzSKfHLscPVizCuBE9D4fe+LOegHYxAsvpaqZjOw4KYhEtf88Q==","signatures":[{"sig":"MEYCIQDBclsFJijvECTgI33G5KK1GtlS4Q7I38pX3w/QRnaTsgIhAP1py9HQ9CLUUU9lW0y7xycg5Azk1yK9cYVOR1eOjqjm","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":53697},"type":"module","engines":{"node":">=20"},"gitHead":"72b58cf31da04d052f60e566639f2780c7512660","scripts":{"test":"npm run test --workspaces --if-present","build":"npm run build --workspaces --if-present","release":"npm run version:sync && npm run build:windows && npm publish --access=public","version":"npm run version:sync && git add Cargo.toml Cargo.lock","typecheck":"npm run typecheck --workspaces --if-present","postinstall":"node scripts/postinstall.js","build:native":"npm run version:sync && cargo build --release -p agent-ctrl-cli && node scripts/copy-native.js","version:sync":"node scripts/sync-version.js","build:windows":"npm run build:native","release:dryrun":"npm run version:sync && npm run build:windows && npm publish --dry-run --access=public"},"_npmUser":{"name":"k4cper-g","email":"k4cpergadomski@gmail.com"},"repository":{"url":"git+https://github.com/k4cper-g/agent-ctrl.git","type":"git"},"workspaces":["packages/*"],"_npmVersion":"11.11.0","description":"Cross-platform computer-use framework for AI agents.","directories":{},"_nodeVersion":"24.14.1","_hasShrinkwrap":false,"_npmOperationalInternal":{"tmp":"tmp/cli_0.1.3_1778360695016_0.755024926256189","host":"s3://npm-registry-packages-npm-production"}},"0.1.4":{"name":"@agent-ctrl/cli","version":"0.1.4","description":"Cross-platform computer-use framework for AI agents.","type":"module","bin":{"agent-ctrl":"bin/agent-ctrl.js"},"scripts":{"version:sync":"node scripts/sync-version.js","version":"npm run version:sync && git add Cargo.toml Cargo.lock","build:native":"npm run version:sync && cargo build --release -p agent-ctrl-cli && node scripts/copy-native.js","build:windows":"npm run build:native","release:dryrun":"npm run version:sync && npm run build:windows && npm publish --dry-run --access=public","release":"npm run version:sync && npm run build:windows && npm publish --access=public","postinstall":"node scripts/postinstall.js","build":"npm run build --workspaces --if-present","test":"npm run test --workspaces --if-present","typecheck":"npm run typecheck --workspaces --if-present"},"keywords":["agent","automation","accessibility","uia","ax","cli","windows","macos","computer-use"],"license":"Apache-2.0","repository":{"type":"git","url":"git+https://github.com/k4cper-g/agent-ctrl.git"},"homepage":"https://github.com/k4cper-g/agent-ctrl#readme","bugs":{"url":"https://github.com/k4cper-g/agent-ctrl/issues"},"workspaces":["packages/*"],"engines":{"node":">=20"},"gitHead":"7cc6b3d6d25ae6064aa1cf56d32df3af6e24a212","_id":"@agent-ctrl/cli@0.1.4","_nodeVersion":"24.18.1","_npmVersion":"11.16.0","dist":{"integrity":"sha512-ZJyzuGXSztZyxiQ+CBkjhFWvcpZWo8PTxYGW5ilcl1DWmSxUq2WaCjmQlzBuYevD2q6CJlOI9dvjrNtb/JQ8nQ==","shasum":"6ec02cb5d0b0f3f2d8445aca9a79a88a023938ca","tarball":"https://registry.npmjs.org/@agent-ctrl/cli/-/cli-0.1.4.tgz","fileCount":9,"unpackedSize":59818,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEYCIQCX3Rdyw8y+w8RdcXFn8lAIEjKvzCDFkb2FwtO2QMpVLgIhAMrdOQDX0pcvAvHOoMFfPKU/Am8A6wyJ0EU8vOjYqe1X"}]},"_npmUser":{"name":"k4cper-g","email":"k4cpergadomski@gmail.com"},"directories":{},"maintainers":[{"name":"k4cper-g","email":"k4cpergadomski@gmail.com"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/cli_0.1.4_1785627736826_0.669662728372318"},"_hasShrinkwrap":false}},"time":{"created":"2026-05-09T18:09:37.682Z","modified":"2026-08-01T23:42:17.125Z","0.1.1":"2026-05-09T18:09:37.958Z","0.1.3":"2026-05-09T21:04:55.153Z","0.1.4":"2026-08-01T23:42:16.967Z"},"bugs":{"url":"https://github.com/k4cper-g/agent-ctrl/issues"},"license":"Apache-2.0","homepage":"https://github.com/k4cper-g/agent-ctrl#readme","keywords":["agent","automation","accessibility","uia","ax","cli","windows","macos","computer-use"],"repository":{"type":"git","url":"git+https://github.com/k4cper-g/agent-ctrl.git"},"description":"Cross-platform computer-use framework for AI agents.","maintainers":[{"name":"k4cper-g","email":"k4cpergadomski@gmail.com"}],"readme":"

\n \"agent-ctrl\"\n

\n\n

\n Desktop automation CLI for AI agents. Fast native Rust CLI.\n

\n\n> **Browser automation is out of scope.** agent-ctrl drives native UI; for Chromium-via-CDP use the sibling [agent-browser](https://github.com/vercel-labs/agent-browser) project. The two are designed to compose in the same agent loop.\n\n## Installation\n\n### npm (recommended)\n\n```bash\nnpm install -g @agent-ctrl/cli\n```\n\nThe package ships a Node launcher; the postinstall step downloads the\nmatching native binary for your platform from the corresponding GitHub\nRelease and verifies its release-published SHA-256 checksum before installing\nit. Supported in v0.1.x: Windows x64, macOS arm64, macOS x64. Linux is on the\nroadmap for prebuilt distribution.\n\n### Windows binary\n\nFor tagged releases, download the Windows zip from GitHub Releases or run:\n\n```powershell\npowershell -ExecutionPolicy Bypass -File .\\scripts\\install-windows.ps1\n```\n\nThe installer downloads the latest `agent-ctrl.exe`, installs it under\n`%LOCALAPPDATA%\\agent-ctrl\\bin`, and adds that directory to the user PATH\nunless `-NoPath` is passed.\n\n### macOS binary\n\nFor tagged releases, download the macOS tarball from GitHub Releases (one\nasset per arch: `aarch64-apple-darwin` for Apple Silicon, `x86_64-apple-darwin`\nfor Intel) or run:\n\n```bash\ncurl -fsSL https://raw.githubusercontent.com/k4cper-g/agent-ctrl/main/scripts/install-macos.sh | bash\n```\n\nThe installer detects the host arch, downloads the matching tarball, places\n`agent-ctrl` at `~/.local/bin/agent-ctrl`, and runs `agent-ctrl info` to\nverify. Pass `--install-dir ` to install elsewhere or `--no-path` to\nskip the PATH-update reminder.\n\nAfter install, grant Accessibility permission in *System Settings >\nPrivacy & Security > Accessibility* (and Screen Recording in the same pane\nif you'll use `screenshot`). Run `agent-ctrl doctor` to verify.\n\n### From source (recommended for v0.1)\n\n```bash\ngit clone https://github.com/k4cper-g/agent-ctrl\ncd agent-ctrl\ncargo build --release -p agent-ctrl-cli\n# put target/release/agent-ctrl on your PATH\n```\n\nThe Rust workspace crates are not published to crates.io in v0.1. The public\ndistribution paths are `npm install -g @agent-ctrl/cli`, the GitHub release\nbinaries (`agent-ctrl-*`), and source builds.\n\n### TypeScript client\n\n```bash\nnpm install @agent-ctrl/client\n# expects `agent-ctrl` on PATH for the daemon transport\n```\n\n### Requirements\n\n- **Windows 10/11** for UIA, **macOS 12+** for AX. Both surfaces ship the full action vocabulary. **Linux** (AT-SPI) ships the snapshot-read path - `snapshot`, `find`, inspect, and `window-list`; its action vocabulary, plus Android / iOS, are not implemented yet. Other OSes build cleanly with stub surfaces.\n- **Rust 1.85+** (workspace MSRV; rustup will install it from `rust-toolchain.toml`).\n- **Node.js 20+** only when using the TypeScript client.\n\n## Quick start\n\n```bash\nagent-ctrl info # OS, available surfaces, active sessions\nagent-ctrl open uia # spawn a daemon (background)\nagent-ctrl snapshot --target-process # tree of refs (@e0, @e1, ...)\nagent-ctrl click @e4 # click by ref\nagent-ctrl get name @e4 # inspect cached snapshot fields\nagent-ctrl is enabled @e4 # boolean state checks\nagent-ctrl fill @e0 \"hello from agent-ctrl\" # set value via UIA ValuePattern\nagent-ctrl press \"Ctrl+S\" # key chord via SendInput\nagent-ctrl screenshot result.png # PNG of the pinned window\nagent-ctrl close # stop the daemon\n```\n\nEvery action follows the same pattern: `snapshot` once to learn what's on screen,\nthen issue actions by ref. Actionable elements use `@eN`; structural containers\nsuch as windows and dialogs use scope-only `@sN` refs. Both are valid only for\nthe snapshot that produced them. Use scope refs with `find --in`, `get`, and\n`is`; actions reject them. Re-snapshot before acting on a tree that has changed.\n\nSurfaces advertise capabilities when a session opens, and the daemon enforces\nthem before dispatch. Native OS handles stay inside the surface process and are\nnever included in JSON snapshots or TypeScript values.\n\n## Commands\n\n### Core\n\n```bash\nagent-ctrl open # spawn a daemon (uia, mock, ...)\nagent-ctrl close # stop the daemon\nagent-ctrl list [--json] # active sessions\nagent-ctrl info [--json] # static facts about this binary\nagent-ctrl doctor [--json] [--fix] [--quick] # diagnose the install + live probe\nagent-ctrl launch [--json] [--wait MS] # spawn an app detached from this shell\n```\n\n### Snapshot\n\n```bash\nagent-ctrl snapshot # capture pinned window's a11y tree\nagent-ctrl snapshot --target-process # pin by process executable name\nagent-ctrl snapshot --target-pid # pin by PID\nagent-ctrl snapshot --target-title # pin by window title (locale-dependent)\nagent-ctrl snapshot --settle # re-snapshot until the tree stabilizes\nagent-ctrl snapshot --json # full JSON for programmatic consumption\nagent-ctrl snapshot --compact false # disable compact-tree filtering\n```\n\nThe first `snapshot` after `open` pins the session to a target window. Subsequent actions on the session target that window until a `focus-window` re-pins it.\n\n`--settle` re-snapshots (every 200ms, ~8s cap) until the tree's structural signature holds steady, then prints it. Use it right after `launch` / `switch-app` against a Chromium/Electron app (Slack, Teams, VS Code, ...), whose accessibility tree is populated lazily on the first query - so the first plain `snapshot` often shows only the window frame.\n\n### Pointer / focus\n\n```bash\nagent-ctrl click @eN # primary-button click on a ref\nagent-ctrl double-click @eN # double-click\nagent-ctrl right-click @eN # secondary-button click\nagent-ctrl hover @eN # cursor over element, no buttons\nagent-ctrl focus @eN # UIA SetFocus\nagent-ctrl highlight @eN # move cursor to element for human debugging\n```\n\n### Keyboard\n\n```bash\nagent-ctrl type \"hello\" # synthetic Unicode keystrokes\nagent-ctrl fill @eN \"value\" # native value setting where supported\nagent-ctrl clear @eN # clear an editable field\nagent-ctrl press \"Ctrl+S\" # key chord - Enter, Tab, Ctrl+A, Cmd+A, etc.\nagent-ctrl key-down \"Shift\" # hold a modifier\nagent-ctrl key-up \"Shift\" # release it\nagent-ctrl clipboard read # read clipboard text\nagent-ctrl clipboard write \"text\" # replace clipboard text\nagent-ctrl clipboard copy # send Ctrl+C\nagent-ctrl clipboard paste # send Ctrl+V\n```\n\n### Selection / scroll\n\n```bash\nagent-ctrl select @eN \"Option name\" # pick an item in a select / combo / list\nagent-ctrl select-all [@eN] # select all in field; without ref, sends Ctrl+A to focus\nagent-ctrl check @eN # set a TogglePattern control on\nagent-ctrl uncheck @eN # set a TogglePattern control off\nagent-ctrl toggle @eN # toggle a TogglePattern control\nagent-ctrl scroll [--ref @eN] # wheel scroll (positive DY = down)\nagent-ctrl scroll-into-view @eN # UIA ScrollItemPattern\nagent-ctrl drag @eFROM @eTO # source-to-destination drag\nagent-ctrl mouse move X Y # raw mouse move\nagent-ctrl mouse down X Y --button left # raw button down\nagent-ctrl mouse up X Y --button left # raw button up\nagent-ctrl mouse wheel X Y --dy -120 # raw wheel\n```\n\n### Find\n\n```bash\nagent-ctrl find \"Save\" # case-insensitive substring on name\nagent-ctrl find \"Save\" --role button # narrow by role (kebab-case)\nagent-ctrl find \"Save\" --exact # case-sensitive equality\nagent-ctrl find --role menu-item # all nodes of a role; no name filter\nagent-ctrl find --role dialog --first # structural container as a scope ref\nagent-ctrl find \"OK\" --in @s0 # restrict to subtree under a scope\nagent-ctrl find \"Save\" --first # bare ref for shell substitution\nagent-ctrl find --limit 5 # cap result count\n```\n\n`find` queries the *cached* snapshot - it does not re-walk the OS tree. With no\nmatch, it writes `no match` to stderr and exits non-zero. `--first` prints a\nbare ref, so the canonical \"find then act\" pattern composes:\n\n```bash\nagent-ctrl click \"$(agent-ctrl find \"Save\" --role button --first)\"\n```\n\nActionable matches are `@eN`. An explicit structural role query can return an\n`@sN` scope ref, which is useful for narrowing a later query without making a\nwindow, dialog, toolbar, or other container accidentally actionable:\n\n```bash\nDIALOG=\"$(agent-ctrl find --role dialog --first)\"\nagent-ctrl click \"$(agent-ctrl find \"OK\" --role button --in \"$DIALOG\" --first)\"\n```\n\n### Inspect\n\n```bash\nagent-ctrl get text @eN # value if present, otherwise accessible name\nagent-ctrl get value @eN # editable/value-bearing field value\nagent-ctrl get name @eN # accessible name\nagent-ctrl get role @eN # canonical role\nagent-ctrl get state @eN # full state object\nagent-ctrl get bounds @eN # logical screen bounds\nagent-ctrl get window # cached window context\n\nagent-ctrl is visible @eN\nagent-ctrl is enabled @eN\nagent-ctrl is focused @eN\nagent-ctrl is selected @eN\nagent-ctrl is checked @eN\nagent-ctrl is expanded @eN\n```\n\nInspect commands read the cached snapshot and accept both `@eN` and `@sN` refs.\nThey are fast and deterministic, but require a prior `snapshot`.\n\n### Wait\n\n```bash\nagent-ctrl wait # dumb sleep on the daemon worker\nagent-ctrl wait-for \"Save\" --role button # wait for a node to appear\nagent-ctrl wait-for \"Loading...\" --gone # wait for a node to disappear\nagent-ctrl wait-for \"Agree\" --state checked # wait for a boolean state\nagent-ctrl wait-for --role text-field --value-contains ready\nagent-ctrl wait-for --window-appears \"Dialog title\" # wait for a sibling window title\nagent-ctrl wait-for --stable [--idle-ms 500] # wait for the tree signature to settle\nagent-ctrl wait-for ... --timeout 10000 --poll 250 # tune the poll loop\n```\n\nThree reliability tiers. Use `--stable` after a click to let the UI settle before\nthe next action. Polling observations do not replace the session's committed\naction refs. When the wait matches or times out, its terminal observation is\npromoted once as the new cached snapshot. Exit codes: 0 satisfied, 1 bad args,\n2 timeout - branch on those in shell pipelines instead of parsing strings.\n\n### Windows\n\n```bash\nagent-ctrl window-list # all top-level windows owned by the pinned process\nagent-ctrl window-list --first-other # bare hex id of the first non-pinned window\nagent-ctrl focus-window # bring a window to the foreground; re-pins the session\nagent-ctrl switch-app # foreground an app by id (path or bare exe name); re-pins\n```\n\nWhen a file dialog, confirmation dialog, or popup appears as a sibling top-level window, `window-list` is how you find it. `focus-window` re-pins so subsequent `snapshot` / `find` / actions target the dialog. Mirrors agent-browser's `tab_list` / `tab_switch`.\n\n`switch-app` and `focus-window` un-minimize their target (a window only minimized to the taskbar) before bringing it forward. They do **not** un-hide a window the app has hidden to the system tray (Slack, Teams, Discord and friends \"close to tray\") - tray apps re-hide a window shown out from under them. If `snapshot --target-process X` reports the process is running but its window is hidden, bring it forward through the app's own channels: click its tray icon, or `agent-ctrl launch ` (launching a packaged/Store app's exe still routes through the activation broker, which resumes and shows it correctly).\n\n```bash\nagent-ctrl press \"Ctrl+S\" # may open a sibling dialog HWND\nagent-ctrl focus-window \"$(agent-ctrl window-list --first-other)\"\nagent-ctrl snapshot # now sees the dialog\nagent-ctrl click \"$(agent-ctrl find \"OK\" --role button --first)\"\n```\n\nFor detailed Windows guidance on dialogs, elevation, stale refs, foreground focus, IME, screenshots, and app framework quirks, see [`docs/windows-reliability.md`](docs/windows-reliability.md).\n\n### Output\n\n```bash\nagent-ctrl screenshot # PNG of the pinned window to a temp path\nagent-ctrl screenshot result.png # to a specific path\nagent-ctrl screenshot --region X,Y,W,H # crop in physical screen pixels\nagent-ctrl screenshot --target desktop # virtual desktop\nagent-ctrl screenshot --target window # pinned window\nagent-ctrl screenshot --target ref --ref @eN # element bounds\nagent-ctrl screenshot --annotated # draw @eN labels from cached snapshot bounds\n```\n\n`--annotated` draws cached snapshot refs onto the PNG. Run `snapshot` first so the screenshot has a current ref map and bounds.\n\n### Batch\n\n```bash\nagent-ctrl batch --file steps.json\nGet-Content steps.json | agent-ctrl batch --stdin # PowerShell-friendly\nagent-ctrl batch '[{\"op\":\"find\",\"query\":{\"name\":\"Save\",\"limit\":1}}]' # Unix-shell-friendly\n```\n\nBatch steps run in order on one daemon session and return structured per-step results. Supported step ops: `act`, `find`, `get`, `is`, `wait`, and `list_windows`.\n\n### JSON mode\n\n```bash\nagent-ctrl list --json\nagent-ctrl find \"Save\" --role button --json\nagent-ctrl get state @eN --json\nagent-ctrl is enabled @eN --json\nagent-ctrl click @eN --json\nagent-ctrl wait-for --stable --json\nagent-ctrl window-list --json\nagent-ctrl screenshot out.png --json\n```\n\nMost runtime commands accept `--json` for machine-readable output. `snapshot --json` returns the full snapshot; `get --json`, `is --json`, action commands, `wait-for --json`, and `window-list --json` return structured protocol results. `batch` output is always JSON, so `batch --json` is accepted as a compatibility no-op.\n\nSession commands redact the TCP auth token from JSON output. `screenshot --json` writes the PNG to disk and prints file metadata (`path`, `width`, `height`, `bytes`, `annotated`) instead of echoing the base64 image payload.\n\nWhen `--json` is present, parse and runtime failures are emitted as one structured object with `ok: false`, `error.code`, `error.message`, and, when available, `error.hint`. Exit codes still matter: 0 means success, 1 means command/request failure, and `wait-for --json` keeps exit 2 for timeouts while printing the structured wait outcome.\n\n## Sessions\n\nRun multiple isolated UIA sessions side by side:\n\n```bash\nagent-ctrl open uia --session app1\nagent-ctrl open uia --session app2\n\nagent-ctrl snapshot --session app1 --target-process \nagent-ctrl snapshot --session app2 --target-process \n\nagent-ctrl list\n# SESSION SURFACE PID ENDPOINT\n# app1 uia 12345 127.0.0.1:54001\n# app2 uia 12346 127.0.0.1:54002\n\nagent-ctrl close --session app1\nagent-ctrl close --session app2\n```\n\nThe default session is `default`, so most commands need no flag. Each session has its own daemon process, pinned target window, cached snapshot, and refs. Session metadata lives at `~/.agent-ctrl/.json` while the daemon is running. TCP session files include a random per-session auth token, and every CLI TCP request sends it automatically. Stdio daemon clients, including the TypeScript client, do not need a token.\n\n## Mock surface\n\nThe `mock` surface returns a fixed two-button window - handy for testing the protocol without UIA permissions or a target app:\n\n```bash\nagent-ctrl open mock\nagent-ctrl snapshot\nagent-ctrl click @e0\nagent-ctrl close\n```\n\nAvailable on every OS, no setup required. Used by the integration tests under `packages/client/tests/`.\n\n## TypeScript client\n\n```typescript\nimport { AgentCtrl } from \"@agent-ctrl/client\";\n\nconst ctrl = new AgentCtrl(); // spawns `agent-ctrl daemon` over stdio\nconst session = await ctrl.openSession(\"uia\");\n\nawait ctrl.snapshot(session, {\n target: { by: \"process-name\", name: \"target-app\" },\n});\n\nconst matches = await ctrl.find(session, {\n name: \"Save\",\n role: \"button\",\n});\n\nawait ctrl.act(session, { kind: \"click\", ref_id: matches[0]!.ref_id });\n\nconst outcome = await ctrl.waitFor(session, {\n predicate: { kind: \"stable\", idle_ms: 500 },\n timeout_ms: 5000,\n poll_ms: 250,\n});\n\nawait ctrl.closeSession(session);\nawait ctrl.close();\n```\n\nMethod surface: `openSession`, `snapshot`, `act`, `find`, `waitFor`, `listWindows`, `closeSession`, `close`. Both transports (shell CLI and stdio TypeScript) talk the same wire protocol; agents can mix and match.\n\nSee [packages/client/README.md](packages/client/README.md) for the full API.\n\n## Architecture\n\nagent-ctrl uses a client-daemon architecture mirroring agent-browser:\n\n1. **Rust CLI** (`crates/cli`) - parses commands, dials the daemon, prints results.\n2. **Rust daemon** (`crates/daemon`) - long-running process that owns surface sessions and dispatches snapshot / action / find / wait / list-windows requests.\n3. **Surface trait** (`crates/core`) - cross-platform contract every backend implements. Per-platform crates (`crates/surface-uia`, `surface-ax`, `surface-atspi`) provide the implementations, gated by `target_os`.\n\nThe daemon starts via `agent-ctrl open ` and persists across CLI invocations for fast subsequent operations. Each session has its own daemon process and writes an atomic discovery file at `~/.agent-ctrl/.json`. On Unix, the directory and files are restricted to the current user. Read-only discovery never removes stale files; `agent-ctrl doctor --fix` performs explicit cleanup.\n\n## Workspace layout\n\nThe repository is a **dual workspace** - a Cargo workspace for the Rust engine and an npm workspace for the TypeScript client.\n\n| Crate / package | Purpose |\n|---|---|\n| [`crates/core`](crates/core) | Shared types and the `Surface` trait. Schema, role taxonomy, action vocabulary, errors. |\n| [`crates/daemon`](crates/daemon) | Long-running process that owns surface sessions and dispatches actions. |\n| [`crates/cli`](crates/cli) | The `agent-ctrl` binary - user-facing entrypoint. |\n| [`crates/surface-uia`](crates/surface-uia) | Windows UI Automation surface (Windows-only). |\n| [`crates/uia-fixture`](crates/uia-fixture) | Deterministic native Win32 fixture app for UIA reliability tests. |\n| [`crates/surface-ax`](crates/surface-ax) | macOS Accessibility surface (full action vocabulary; macOS-only). |\n| [`crates/ax-fixture`](crates/ax-fixture) | Deterministic native Cocoa fixture app for AX reliability tests. |\n| [`crates/surface-atspi`](crates/surface-atspi) | Linux AT-SPI surface - snapshot-read path (Linux-only). |\n| [`crates/atspi-fixture`](crates/atspi-fixture) | Deterministic GTK4 fixture app (Python) for AT-SPI tests. |\n| [`packages/client`](packages/client) | `@agent-ctrl/client` - typed TypeScript wrapper over stdio JSON-RPC. |\n\nSurfaces gated by `target_os` compile to empty crates on other platforms, so the workspace builds on any host.\n\n## Platforms\n\nA **surface** is one accessibility protocol - UIA, AX, AT-SPI, etc. A **platform** is an operating system. They aren't 1-to-1: most platforms can be driven by more than one surface.\n\n| Platform | Native surface | Status |\n|---|---|---|\n| Windows | [`surface-uia`](crates/surface-uia) - UI Automation | **ready** |\n| macOS | [`surface-ax`](crates/surface-ax) - Accessibility / AX | **ready** |\n| Linux | [`surface-atspi`](crates/surface-atspi) - AT-SPI / D-Bus | **snapshot-read** - `snapshot`, `find`, inspect, and `window-list` work; the action vocabulary is a follow-up. Mapping in [`docs/atspi-mapping.md`](docs/atspi-mapping.md); headless dev/CI stack in [`docker/linux-dev/`](docker/linux-dev) |\n| Android | _planned_ `surface-accessibility-service` (JNI) | not started |\n| iOS | _planned_ `surface-xcuitest` (WebDriverAgent) | not started |\n\nFor browsers, run agent-ctrl alongside [agent-browser](https://github.com/vercel-labs/agent-browser); the two are complementary, not competing.\n\nAcronyms in one line: **UIA** = Microsoft UI Automation, **AX** = macOS Accessibility, **AT-SPI** = the Linux GNOME accessibility bus, **XCUITest** = Apple's UI test framework.\n\nAX feature coverage is in [docs/macos-ax.md](docs/macos-ax.md); production\nguidance for macOS lives in [docs/macos-ax-reliability.md](docs/macos-ax-reliability.md).\nWindows production guidance lives in [docs/windows-reliability.md](docs/windows-reliability.md).\n\n## Build\n\n```bash\ncargo check --workspace # fast type-check\ncargo build --release -p agent-ctrl-cli # the binary\ncargo test --workspace # all unit + integration tests\ncargo clippy --workspace --all-targets -- -D warnings # lint, fail on warnings\ncargo fmt --all -- --check # format check\n```\n\nWindows UIA fixture:\n\n```powershell\ncargo build -p agent-ctrl-cli -p agent-ctrl-uia-fixture\n.\\target\\debug\\agent-ctrl-uia-fixture.exe --ready-file \"$env:TEMP\\agent-ctrl-fixture.ready\"\n.\\target\\debug\\agent-ctrl.exe open uia --session fixture\n.\\target\\debug\\agent-ctrl.exe snapshot --session fixture --target-process agent-ctrl-uia-fixture\n```\n\nThe fixture is the preferred real-UIA test target. It exposes common native controls through stable Win32/UIA patterns so tests do not depend on Notepad, Calculator, localized strings, or Windows-version-specific app redesigns.\n\nOpt-in fixture integration test:\n\n```powershell\ncargo build -p agent-ctrl-uia-fixture\n$env:RUN_UIA_TESTS = \"1\"\ncargo test -p agent-ctrl-cli --test windows_uia_fixture\n```\n\nSuccessful UIA actions may print a method diagnostic such as `ok method=keyboard-space`, `ok method=selection-item-pattern`, or `ok method=toggle-pattern`. These are intended for agents and humans debugging cross-app behavior.\n\nmacOS AX fixture:\n\n```bash\ncargo build -p agent-ctrl-cli -p agent-ctrl-ax-fixture\ntarget/debug/agent-ctrl-ax-fixture --ready-file /tmp/agent-ctrl-ax-fixture.ready &\ntarget/debug/agent-ctrl open ax --session fixture\ntarget/debug/agent-ctrl snapshot --session fixture --target-process agent-ctrl-ax-fixture\n```\n\nOpt-in AX fixture integration test:\n\n```bash\ncargo build -p agent-ctrl-ax-fixture\nRUN_AX_TESTS=1 cargo test -p agent-ctrl-cli --test macos_ax_fixture\n```\n\nThe AX fixture exercises the full macOS action loop against real Cocoa\ncontrols, including Unicode keyboard input, popup selection, screenshots,\nduplicate accessibility identifiers, and scope-ref navigation through an\nattached sheet and popover. It also verifies that check-state actions reject\nordinary buttons without mutating them.\n\nLinux AT-SPI fixture:\n\nAT-SPI does not exist on Windows or macOS, so the `surface-atspi` crate is\ndeveloped and tested inside the headless container in [`docker/linux-dev/`](docker/linux-dev)\n(Xvfb + a private session D-Bus + the AT-SPI registry + GTK4). The fixture is a\ndeterministic GTK4 app, [`crates/atspi-fixture/main.py`](crates/atspi-fixture/main.py).\n\n```bash\ndocker build -t agent-ctrl-linux-dev docker/linux-dev/\ndocker run --rm -v \"$PWD:/work\" -w /work agent-ctrl-linux-dev \\\n bash -c 'RUN_ATSPI_TESTS=1 cargo test -p agent-ctrl-cli --test linux_atspi_fixture'\n```\n\nThe `linux_atspi_fixture` test is gated by `RUN_ATSPI_TESTS=1` for local runs\nand runs automatically in CI's headless `atspi-smoke` job. It opens an `atspi`\nsession, launches the GTK fixture, and exercises `snapshot`, `find`, `get`,\n`is`, `wait-for`, and `window-list`. See\n[`docker/linux-dev/README.md`](docker/linux-dev/README.md) for the full\ncontainer invocation.\n\nTypeScript client:\n\n```bash\nnpm install\nnpm run build --workspace=@agent-ctrl/client\nnpm run test --workspace=@agent-ctrl/client # spawns the Rust daemon under cargo run\n```\n\nThe TS test suite spawns the Rust daemon under `cargo run` and exercises the full protocol against the mock surface - including `find`, `waitFor`, and `listWindows`.\n\n## Usage with AI agents\n\n### Just ask the agent\n\nThe simplest approach - tell your agent it can use it:\n\n```\nUse agent-ctrl to drive Windows apps. Run `agent-ctrl --help` to see the command list,\nand `agent-ctrl info` to check what's available on this machine.\n```\n\nThe `--help` output is comprehensive and most modern agents can figure out the rest from there.\n\n### AGENTS.md / CLAUDE.md\n\nFor consistent results, add to your project or global instructions:\n\n```markdown\n## OS automation\n\nUse `agent-ctrl` for native UI automation on Windows and macOS. Core workflow:\n\n1. `agent-ctrl open uia` (Windows) or `agent-ctrl open ax` (macOS) - spawn a daemon\n2. Bring the app forward: `agent-ctrl switch-app ` (un-minimizes it too) if it has a visible\n window; `agent-ctrl launch ` if it isn't running or is parked in the system tray\n3. `agent-ctrl snapshot --target-process ` - pin to the app and capture refs.\n Add `--settle` for Chromium/Electron apps (Slack, Teams, VS Code) - their tree\n is built lazily, so the first plain snapshot is often just the window frame\n4. `agent-ctrl find \"Save\" --role button --first` - discover refs by name/role\n5. `agent-ctrl click @eN` / `fill @eN \"text\"` / `type \"text\"` / `press \"Ctrl+S\"` (or `Cmd+S` on macOS) - interact.\n Prefer `type` over `fill` for web-based / Chromium text boxes that need real key events\n6. `agent-ctrl wait-for --stable` - let the UI settle before the next action\n7. Before anything hard to undo (send a message, hit OK, delete): re-`snapshot`,\n confirm the field shows your input and the submit control is enabled, then `press \"Enter\"`\n8. `agent-ctrl window-list` + `focus-window ` - switch to dialogs / popups\n9. Re-`snapshot` after the tree changes\n```\n\n### Example flows\n\nThe recommended pattern is app-agnostic: bring the target forward, snapshot\n(with `--settle` for Chromium-based apps), find by role/name, act, wait for\nstability, **re-snapshot to verify before any irreversible step**, then commit.\nConcrete walkthroughs:\n\n- [examples/notepad-tour.sh](examples/notepad-tour.sh) - a simple Win32 app\n- [examples/chat-dm.sh](examples/chat-dm.sh) - the quick-switcher -> compose -> verify -> send flow for a Slack/Teams-style chat app\n\nProduction agents should prefer the generic loop over app-specific assumptions.\n\n## Known limitations\n\nThese are real today - the goal is to fix or document them as the project matures.\n\n- **Windows and macOS are the action-ready surfaces.** Linux (AT-SPI) is snapshot-read only - it captures trees, resolves `find`/inspect refs, and lists windows, but cannot yet click, type, or focus; Android / iOS / browser flows are not implemented in this project yet. macOS additionally requires Screen Recording permission for `screenshot` and may require Automation permission for some Apple system apps (Notes, Calendar, Music) - see [docs/macos-ax-reliability.md](docs/macos-ax-reliability.md).\n- **Linux apps must have accessibility enabled to be visible.** GTK and Qt only build their AT-SPI tree when `org.a11y.Status.IsEnabled` is set; `agent-ctrl open atspi` flips it, but an app already running may take a moment to register its tree (use `snapshot --settle`). Headless geometry is approximate - GTK under Xvfb reports no screen coordinates, so `bounds` may be absent.\n- **Local TCP daemon auth is developer-machine scoped.** TCP session files include a random bearer token and the daemon rejects missing or incorrect tokens, but anyone who can read `~/.agent-ctrl/.json` can still use that session. Treat sessions as a local developer-machine boundary, not a multi-user security sandbox.\n- **Refs are valid only against the snapshot that produced them.** `wait-for`\n keeps intermediate polling observations isolated, then promotes its terminal\n capture once. After a wait returns, use its returned ref or run `find` again\n before acting.\n- **Modern Win11 file dialogs and popup menus open as sibling top-level windows**, not as children of the app's main window. Use `window-list` + `focus-window` to discover and switch to them.\n- **`type` bypasses IME.** Synthetic Unicode keystrokes via `SendInput` are reliable for ASCII; CJK with IME composition is not supported yet. `fill` (UIA `ValuePattern`) is the right escape hatch for non-ASCII text input.\n- **HWND recycling.** Windows reassigns numeric HWNDs after a window closes; `window-list` shows whatever currently holds an id, with no UIA-runtime-id verification. Theoretical, never observed in practice.\n- **An unresponsive target wedges the UIA session.** UIA calls are cross-process COM calls; if the target app stops pumping messages, a snapshot or action can't return. After ~45s the call times out, the session is marked wedged, and every subsequent call on it fails fast - run `agent-ctrl close` then `agent-ctrl open uia` to start a fresh one. (The stuck worker thread is abandoned, so the daemon and other sessions keep working.)\n\n## License\n\nApache-2.0. See [LICENSE](LICENSE).\n","readmeFilename":"README.md"}