---
description: Use this skill for controlling a connected Android phone via ADB -- taking screenshots, navigating apps, tapping, swiping, typing, and other phone automation tasks. Covers WeChat, WeCom, and other installed Chinese apps. Includes coordinate scaling, Chinese input workarounds, and security keyboard handling.
---

# Android Phone ADB Automation

Complete guide for remotely controlling an Android phone connected via USB ADB. Covers screenshot workflow, touch interactions, app launching, text input limitations, and common troubleshooting.

## Helper Scripts (Ready to Use)

This skill includes ready-to-use scripts that agents can call directly. All scripts are installed alongside this SKILL.md file. Reference them using the skill directory path:

| Script | Purpose | Key Options |
|--------|---------|-------------|
| `phone_setup.sh` | **Run first!** Install ADBKeyboard, verify device | `--status`, `--uninstall`, `--skip-kbd` |
| `phone_screenshot.sh` | Take, pull & scale screenshot | `-s SCALE%`, `-o PATH`, `-r` (raw) |
| `phone_tap.sh` | Tap with auto coordinate scaling | `-S SCALE%`, `-l` (long press) |
| `phone_swipe.sh` | Swipe with presets (up/down/left/right) | preset or `x1 y1 x2 y2`, `-S SCALE%` |
| `phone_wake_unlock.sh` | Wake screen & swipe to unlock | `-d DEVICE` |
| `phone_launch.sh` | Launch app by alias or package | `-l` (list aliases) |
| `phone_input.sh` | Type text with method options | `-m direct\|adbkbd\|clipboard`, `-e` (enter), `-c` (clear) |
| `phone_ui_parse.py` | Dump & parse UI hierarchy | `-c` (clickable), `-s SEARCH`, `-j` (JSON) |
| `phone_security_kbd.py` | Type on randomized security keyboard | `--dry-run`, `--delay MS` |

All shell scripts support `-d DEVICE_ID` for multi-device setups and `-q` for quiet mode.

### First-Time Setup

Before using this skill for the first time, run the setup script to install **ADBKeyboard** (required for Chinese text input) and verify the device connection:

```bash
SKILL_DIR="<path-to-installed-skill>"
bash "$SKILL_DIR/phone_setup.sh"            # Full setup: checks device, installs ADBKeyboard
bash "$SKILL_DIR/phone_setup.sh" --status   # Check if everything is configured
```

> **Note**: If you skip setup, `phone_input.sh -m adbkbd` will auto-detect the missing ADBKeyboard and run setup automatically on first use.

### Quick Example: Screenshot → Analyze → Tap

```bash
SKILL_DIR="$(dirname "$0")"  # or use the installed skill path

# Take scaled screenshot
bash "$SKILL_DIR/phone_screenshot.sh" -s 50 -o /tmp/phone_screen.png

# (Agent analyzes /tmp/phone_screen.png, identifies target at scaled coords 305,664)

# Tap with auto-scaling from 50% screenshot coords to native
bash "$SKILL_DIR/phone_tap.sh" 305 664 -S 50
```

## Pre-flight Check

Every time before operating the phone, verify the device connection:

```bash
adb devices
# Expected: <device_id>  device
# If "unauthorized": user must tap "Allow USB debugging" on phone
# If "offline" or missing: check USB cable connection
```

If adb is quarantined by macOS:
```bash
sudo xattr -rd com.apple.quarantine $(which adb)
```

Get device info (resolution, model, Android version):
```bash
adb shell wm size          # e.g. Physical size: 1220x2656
adb shell getprop ro.product.model
adb shell getprop ro.build.version.sdk
```

## Screenshot Workflow (Core Loop)

Every phone operation follows this cycle:

```
Screenshot → Analyze UI → Calculate coordinates → Execute action → Wait → Screenshot to confirm
```

### Taking Screenshots

**Preferred: Use the helper script** which handles capture, pull, scale, and cleanup:
```bash
bash phone_screenshot.sh                      # Default: 50% scale to /tmp/phone_screen.png
bash phone_screenshot.sh -o /tmp/my.png -s 40 # 40% scale, custom output path
bash phone_screenshot.sh -r                   # Raw native resolution (no scaling)
```

Manual approach (if needed):
```bash
# 1. Capture on device
adb shell screencap -p /sdcard/screen.png

# 2. Pull to local
adb pull /sdcard/screen.png /tmp/phone_raw.png

# 3. Scale down (REQUIRED — native resolution often exceeds image processing limits)
# For 1220x2656 device, scale to 50%:
sips --resampleHeightWidth 1328 610 /tmp/phone_raw.png --out /tmp/phone_screen.png

# 4. Clean up device temp file
adb shell rm /sdcard/screen.png
```

> **IMPORTANT**: Always scale screenshots before analysis. Native resolutions (2656px+) exceed the 2000px processing limit. Use 50% scaling as default.

### Coordinate Scaling (Critical!)

When the screenshot is scaled, you must convert coordinates back to native resolution before sending ADB commands:

```
Native ADB coordinate = Scaled screenshot coordinate × (Native resolution / Scaled resolution)
```

**Example** (50% scale): Screenshot shows target at (305, 664) → ADB tap coordinate = (610, 1328)

> Always double-check: **read coordinates from scaled image, execute with native coordinates.**

## Basic Operations

### Wake & Unlock Screen

```bash
# Check screen state
adb shell dumpsys power | grep "mWakefulness"
# Awake = screen on, Asleep = screen off

# Wake up
adb shell input keyevent KEYCODE_WAKEUP

# Swipe up to unlock (from bottom to middle)
adb shell input swipe 500 2000 500 800 300
```

### Tap

**Preferred: Use the helper script** with auto coordinate scaling:
```bash
bash phone_tap.sh 610 1328              # Tap at native coords
bash phone_tap.sh 305 664 -S 50        # Auto-scale from 50% screenshot coords
bash phone_tap.sh 305 664 -S 50 -l     # Long press at scaled coords
```

Manual approach:
```bash
# Use NATIVE coordinates (not scaled!)
adb shell input tap X Y

# Example: tap screen center on 1220x2656 device
adb shell input tap 610 1328
```

### Swipe

**Preferred: Use the helper script** with directional presets:
```bash
bash phone_swipe.sh up                  # Scroll up (auto-detects resolution)
bash phone_swipe.sh down                # Scroll down
bash phone_swipe.sh left                # Swipe left
bash phone_swipe.sh right               # Swipe right
bash phone_swipe.sh 100 2000 100 800    # Custom swipe (native coords)
bash phone_swipe.sh 50 1000 50 400 -S 50  # Custom with auto-scaling
```

Manual approach:
```bash
# adb shell input swipe startX startY endX endY duration_ms
adb shell input swipe X1 Y1 X2 Y2 300

# Scroll up (view more content)
adb shell input swipe 610 2000 610 800 300

# Scroll down
adb shell input swipe 610 800 610 2000 300

# Swipe left (next page)
adb shell input swipe 1000 1328 200 1328 300

# Swipe right (previous page)
adb shell input swipe 200 1328 1000 1328 300
```

### Key Events

```bash
adb shell input keyevent KEYCODE_BACK        # Back
adb shell input keyevent KEYCODE_HOME        # Home
adb shell input keyevent KEYCODE_WAKEUP      # Wake screen
adb shell input keyevent KEYCODE_POWER       # Power toggle
adb shell input keyevent KEYCODE_ENTER       # Enter/Confirm
adb shell input keyevent KEYCODE_DEL         # Backspace/Delete
adb shell input keyevent KEYCODE_VOLUME_UP   # Volume up
adb shell input keyevent KEYCODE_VOLUME_DOWN # Volume down
adb shell input keyevent 187                 # Recent apps / Multitask
```

### Launch Apps

**Preferred: Use the helper script** with app name aliases:
```bash
bash phone_launch.sh wechat             # Launch by alias
bash phone_launch.sh com.tencent.mm     # Launch by package name
bash phone_launch.sh -l                 # List all known aliases
```

Manual approach:
```bash
# WeChat
adb shell monkey -p com.tencent.mm -c android.intent.category.LAUNCHER 1

# WeCom (企业微信)
adb shell monkey -p com.tencent.wework -c android.intent.category.LAUNCHER 1

# Meituan (美团)
adb shell monkey -p com.sankuai.meituan -c android.intent.category.LAUNCHER 1

# Ctrip (携程)
adb shell monkey -p ctrip.android.view -c android.intent.category.LAUNCHER 1

# Xiaohongshu (小红书)
adb shell monkey -p com.xingin.xhs -c android.intent.category.LAUNCHER 1

# Bilibili (哔哩哔哩)
adb shell monkey -p tv.danmaku.bili -c android.intent.category.LAUNCHER 1

# Baidu (百度)
adb shell monkey -p com.baidu.searchbox -c android.intent.category.LAUNCHER 1

# BOSS Zhipin (BOSS直聘)
adb shell monkey -p com.hpbr.bosszhipin -c android.intent.category.LAUNCHER 1

# Doubao (豆包)
adb shell monkey -p com.larus.nova -c android.intent.category.LAUNCHER 1
```

Find any app's package name:
```bash
adb shell pm list packages | grep -i keyword
```

Check which app is in foreground:
```bash
adb shell dumpsys activity activities | grep "mResumedActivity"
```

## Text Input

### English & Numbers (Works directly)

```bash
adb shell input text "hello123"
```

### Chinese Input (Known Limitations)

Direct Chinese input via ADB is **not supported** on Android. All of the following methods have known issues:

1. **`adb shell input text '你好'`** → NullPointerException (Android `input text` does not support CJK characters)
2. **Pinyin via `input text`** → Sogou/other IME nine-grid layouts misparse pinyin segmentation
3. **Clipboard** → `adb shell cmd clipboard set` returns "No shell command implementation" on many devices
4. **ADB broadcast** → Requires ADBKeyboard app installed

### Chinese Input Workarounds

- **WeChat/WeCom messages**: Use the agent's messaging API (webhook/notify_user) to send directly, bypassing phone input entirely
- **Search by contact**: Try English names or pinyin initials
- **ADBKeyboard**: If available, install the ADBKeyboard APK to enable `am broadcast` based text input:
  ```bash
  adb install ADBKeyboard.apk
  adb shell ime set com.android.adbkeyboard/.AdbIME
  adb shell am broadcast -a ADB_INPUT_TEXT --es msg '你好'
  ```

## Reading UI State (Alternative to Screenshots)

**Preferred: Use the helper script** for easy UI parsing:
```bash
python3 phone_ui_parse.py                   # Dump & show all elements with bounds
python3 phone_ui_parse.py -c                # Only clickable elements
python3 phone_ui_parse.py -s "send"         # Filter by text/description
python3 phone_ui_parse.py -c -s "确认" -j   # Clickable "confirm" as JSON
python3 phone_ui_parse.py -f /tmp/ui.xml    # Parse existing XML file
```

Manual approach — dump and parse:
```bash
# Dump UI accessibility tree
adb shell uiautomator dump /sdcard/ui.xml 2>/dev/null
adb pull /sdcard/ui.xml /tmp/ui.xml
```

Parse with Python to find interactive elements:
```python
import xml.etree.ElementTree as ET, re

tree = ET.parse('/tmp/ui.xml')
root = tree.getroot()

for node in root.iter():
    text = node.get('text', '')
    desc = node.get('content-desc', '')
    bounds = node.get('bounds', '')
    clickable = node.get('clickable', '')
    if bounds and (text or desc):
        m = re.findall(r'\d+', bounds)
        if len(m) == 4:
            cx = (int(m[0]) + int(m[2])) // 2
            cy = (int(m[1]) + int(m[3])) // 2
            print(f'"{text}" desc="{desc}" center=({cx},{cy}) clickable={clickable}')
```

### Security Keyboard Handling

Financial apps use randomized security keyboards where digit positions change each time.

**Preferred: Use the helper script** which auto-detects and types on security keyboards:
```bash
python3 phone_security_kbd.py "123456"              # Type PIN
python3 phone_security_kbd.py "888888" --dry-run    # Show digit positions without tapping
python3 phone_security_kbd.py "1234" --delay 300    # Slower tapping (300ms between digits)
```

Manual approach — read `content-desc` from UI dump to map digits:
```python
def get_security_keyboard_map(root):
    """Returns dict of digit/char -> (center_x, center_y)"""
    kbd = {}
    for node in root.iter():
        desc = node.get('content-desc', '')
        bounds = node.get('bounds', '')
        if desc and bounds and node.get('clickable') == 'true':
            m = re.findall(r'\d+', bounds)
            if len(m) == 4:
                cx = (int(m[0]) + int(m[2])) // 2
                cy = (int(m[1]) + int(m[3])) // 2
                kbd[desc] = (cx, cy)
    return kbd
```

## Standard Operating Procedures

### Open WeChat and Browse

**Using helper scripts (recommended):**
```bash
bash phone_wake_unlock.sh               # Wake & unlock
sleep 1
bash phone_launch.sh wechat             # Launch WeChat
sleep 2
bash phone_screenshot.sh -s 50          # Take scaled screenshot
# Now read /tmp/phone_screen.png to analyze the interface
```

**Manual approach:**
```bash
# Wake phone
adb shell input keyevent KEYCODE_WAKEUP
sleep 0.5
adb shell input swipe 500 2000 500 800 300   # Unlock
sleep 1

# Launch WeChat
adb shell monkey -p com.tencent.mm -c android.intent.category.LAUNCHER 1
sleep 2

# Take screenshot to analyze current view
adb shell screencap -p /sdcard/screen.png
adb pull /sdcard/screen.png /tmp/phone_raw.png
sips --resampleHeightWidth 1328 610 /tmp/phone_raw.png --out /tmp/phone_screen.png
adb shell rm /sdcard/screen.png
# Now read /tmp/phone_screen.png to analyze the interface
```

### Install an APK

```bash
# Install from local file
adb install -r /path/to/app.apk

# Or push then install (more reliable for large APKs)
adb push /path/to/app.apk /data/local/tmp/app.apk
adb shell pm install -r /data/local/tmp/app.apk
```

## Troubleshooting

| Problem | Cause | Solution |
|---------|-------|----------|
| Screenshot is all black | Phone screen is off / asleep | `adb shell input keyevent KEYCODE_WAKEUP` + swipe to unlock |
| Screenshot too large to analyze | Native resolution too high | Scale to 50% with `sips` before reading |
| Device shows "unauthorized" | USB debugging not approved | Notify user to tap "Allow" on phone screen |
| adb quarantined on macOS | Gatekeeper blocked binary | `sudo xattr -rd com.apple.quarantine $(which adb)` |
| Chinese text can't be typed | Android `input text` CJK limitation | Use webhook/API, or install ADBKeyboard |
| Tap lands on wrong position | Coordinates not scaled back | Multiply scaled coordinates by scale factor for native coords |
| Pinyin input garbled | Sogou nine-grid layout | Avoid pinyin via `input text`; use ADBKeyboard or API |
| INJECT_EVENTS SecurityException | Xiaomi USB debug security setting | Enable "USB调试（安全设置）" in Developer Options |
| FLAG_SECURE blocks screenshot | App security policy | Use `uiautomator dump` to read UI tree instead |
| Screen locks during operation | Auto-lock timeout | `adb shell settings put system screen_off_timeout 1800000` |

## Best Practices

1. **Always screenshot before acting** — never tap blindly; confirm the current UI state first
2. **Wait after interactions** — `sleep 1-2` seconds after tap/swipe before taking the next screenshot
3. **Always scale coordinates** — read from scaled image, execute with native coordinates
4. **Clean up temp files** — delete screenshots from both phone (`/sdcard/`) and local (`/tmp/`) after use
5. **Avoid rapid screenshots** — each involves device storage I/O; don't use in tight loops
6. **Re-wake if idle** — phone may auto-lock after inactivity; always check screen state first
7. **Use UI dump for secure apps** — FLAG_SECURE apps return black screenshots; `uiautomator dump` still works
