---
description: Use this skill for desktop automation -- choosing between Playwright browser tools and computer_use tools, clipboard operations, app switching, screenshots, file operations, and system commands. Provides a tool selection guide for different automation scenarios on both macOS and Windows.
---

# Desktop Automation Patterns

## Tool Selection Guide

| Task | Preferred Tool | Notes |
|------|---------------|-------|
| Web browsing / forms | Playwright MCP (`browser_*`) | Fast, uses accessibility tree |
| Desktop GUI apps | `computer_use_*` tools | Direct pixel coordinate input from screenshots |
| WeChat | `computer_use_*` ONLY | AppleScript/native automation unreliable |
| Feishu | `computer_use_*` ONLY | Must use desktop app, not browser |
| File operations | Bash/PowerShell (`cp`, `mv`, etc.) | Direct and fast |
| Screenshots | `computer_use_screenshot` | Returns base64 image |
| Clipboard | macOS: `pbcopy`/`pbpaste`, Windows: `Set-Clipboard`/`Get-Clipboard` | Great for pasting long text into apps |

## Playwright Browser Workflow

```python
# 1. Navigate
browser_navigate(url="https://example.com")

# 2. Get accessibility snapshot (gives element refs)
browser_snapshot()  # returns YAML with ref IDs

# 3. Interact using refs
browser_click(ref="e123", element="submit button")
browser_type(ref="e456", text="hello world")
browser_fill_form(fields=[{"name": "email", "type": "textbox", "ref": "e789", "value": "test@example.com"}])

# 4. Wait for content
browser_wait_for(text="Success")

# 5. Screenshot for visual confirmation
browser_take_screenshot(type="png", filename="result.png")
```

## computer_use Tools

```python
# 1. Take a screenshot first to see the screen and get coordinate space
computer_use_screenshot()  # returns image + pixel dimensions

# 2. Click using pixel coordinates from the screenshot
computer_use_click(point={"x": 500, "y": 300}, button="left")

# Type text (into focused field)
computer_use_type(text="Hello world", press_enter=True)

# Scroll at a position
computer_use_scroll(point={"x": 500, "y": 400}, direction="down", amount=3)

# Key press (cmd/meta maps to Ctrl on Windows automatically)
computer_use_key(key="cmd+v")     # paste (Ctrl+V on Windows)
computer_use_key(key="escape")
computer_use_key(key="return")
computer_use_key(key="cmd+tab")   # switch apps (Alt+Tab on Windows — use alt+tab explicitly)

# Drag from one point to another
computer_use_drag(startPoint={"x": 100, "y": 200}, endPoint={"x": 400, "y": 200})
```

## Pasting Long Text Into Desktop Apps

### macOS
```bash
echo -n "your long text here" | pbcopy
# Then in the app:
computer_use_key("cmd+v")
```

### Windows
```powershell
Set-Clipboard -Value "your long text here"
# Then in the app:
computer_use_key("ctrl+v")
```

For multiline text (macOS):
```bash
python3 -c "print('multiline\ntext\nhere', end='')" | pbcopy
```

## App Switching

### macOS
```bash
computer_use_key("cmd+tab")  # cycles through recent apps
computer_use_click(point={"x": 960, "y": 1060})  # click WeChat icon in dock (coordinates from screenshot)
```

### Windows
```bash
computer_use_key("alt+tab")  # cycles through recent apps
computer_use_click(point={"x": 960, "y": 1060})  # click WeChat icon in taskbar (coordinates from screenshot)
```

## Taking Screenshots

```python
# Via VisionClaw tool
computer_use_screenshot()  # returns image + file path

# Via Playwright
browser_take_screenshot(type="png", filename="~/screenshots/result.png")
# Playwright saves to ~/screenshots/ by default
```

## Opening Chrome with CDP (for Playwright)

If Playwright browser isn't running:
```python
browser(headless=False, port=9222)  # opens Chrome with CDP enabled
```

## File Operations Shortcuts

```bash
# macOS / Linux
/tmp/          # temporary files
~/screenshots/ # Playwright screenshots

# Clipboard (macOS)
pbcopy < /tmp/transcript.txt

# Clipboard (Windows PowerShell)
Get-Content /tmp/transcript.txt | Set-Clipboard

# Check if a process is running
# macOS: pgrep -x "FutuOpenD"
# Windows: Get-Process -Name FutuOpenD -ErrorAction SilentlyContinue

# View running processes
pm2 list
```

## System Commands

### macOS
```bash
sw_vers                    # macOS version
uname -m                   # chip architecture (arm64 = Apple Silicon)
df -h /                    # disk space
curl -s https://api.ipify.org  # public IP
open -a "WeChat"           # open applications
```

### Windows
```powershell
[System.Environment]::OSVersion   # Windows version
$env:PROCESSOR_ARCHITECTURE       # CPU architecture
Get-PSDrive C | Select-Object Used,Free  # disk space
Invoke-RestMethod https://api.ipify.org   # public IP
Start-Process "WeChat"                     # open applications
```

## Notifications / Dialogs

When a system dialog appears (permissions, alerts):
```python
browser_handle_dialog(accept=True)   # for browser dialogs
# For system dialogs: use computer_use_click to click the button
computer_use_click(point={"x": 800, "y": 500})  # click Allow button (coordinates from screenshot)
```
