---
description: Use this skill to transcribe audio files (voice messages, recordings) to text, extract audio from Bilibili videos, or handle Telegram voice messages. Supports OGG, MP3, WAV, M4A, and other common audio formats. Uses whisper.cpp with auto language detection -- supports Chinese, English, and all other languages.
---

# Transcribe Audio & Media Extraction

Transcribe audio files to text using whisper.cpp (local, free, private, native macOS binary).

## Prerequisites

whisper-cpp, ffmpeg, and the whisper model are checked during `visionclaw setup`. If they're missing, run setup again or install manually:

```bash
brew install whisper-cpp ffmpeg
mkdir -p ~/.local/share/whisper-cpp && curl -L -o ~/.local/share/whisper-cpp/ggml-large-v3-turbo.bin 'https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-large-v3-turbo.bin?download=true'
```

Binary is `whisper-cli` (not `whisper`). If not on PATH: `/opt/homebrew/bin/whisper-cli`

## Steps

1. **Download the audio file** if it's a URL:
   ```bash
   curl -s -o /tmp/audio_input.ogg "THE_AUDIO_URL"
   ```

2. **Convert to 16kHz WAV** (required by whisper.cpp):
   ```bash
   ffmpeg -y -i /tmp/audio_input.ogg -ar 16000 -ac 1 /tmp/audio_input.wav 2>/dev/null
   ```

3. **Transcribe**:
   ```bash
   whisper-cli -m ~/.local/share/whisper-cpp/ggml-large-v3-turbo.bin -f /tmp/audio_input.wav --no-timestamps -l auto 2>/dev/null
   ```

4. **Use the transcription** as needed -- respond to the user, take action, etc.

> **CRITICAL**: Always use `-l auto` for language detection. The default is `-l en` which BREAKS Chinese transcription.

## Notes

- Uses `large-v3-turbo` (~1.5GB) -- near-large accuracy at half the size of full large-v3.
- whisper.cpp is optimized for Apple Silicon -- runs efficiently on any M1/M2/M3/M4 MacBook.
- Supports: OGG, MP3, WAV, M4A, FLAC, WEBM, and most common audio formats (via ffmpeg conversion).
- Runs entirely locally -- no API calls, no costs, fully private.
- If whisper-cli is not found, verify with `which whisper-cli`. The Homebrew package is `whisper-cpp` but the binary is `whisper-cli`.

## Telegram Voice Messages

Telegram voice messages arrive as OGG files:

```bash
curl -s -o /tmp/voice.ogg "{file_url}"
ffmpeg -y -i /tmp/voice.ogg -ar 16000 -ac 1 /tmp/voice.wav 2>/dev/null
whisper-cli -m ~/.local/share/whisper-cpp/ggml-large-v3-turbo.bin -f /tmp/voice.wav --no-timestamps -l auto 2>/dev/null
```

## Bilibili Video Audio Extraction

### Method: Browser Network Interception

When the standard yt-dlp approach fails (Bilibili returns HTTP 412), use Playwright to intercept the .m4s audio stream URLs directly from the browser's network requests.

```python
# 1. Navigate to Bilibili video in Playwright browser
await page.goto(f'https://www.bilibili.com/video/{bv_id}')

# 2. Wait for video to load, then check network requests
# The audio stream URL contains "30216" in the filename (30216 = audio codec ID)
# Example: [CID]-1-30216.m4s

# 3. Download the audio .m4s with Referer header
curl -s -L \
  -H "Referer: https://www.bilibili.com/video/[BV_ID]" \
  -H "User-Agent: Mozilla/5.0 ..." \
  -o /tmp/audio.m4a \
  "{audio_url}"
```

### Finding Bilibili Video for a WeChat 视频号 Video

1. Note the creator name (shown in WeChat video player) and video title keywords
2. Search Bilibili: `https://search.bilibili.com/all?keyword={creator}+{keywords}`
3. Extract BV IDs from search results (look for `BV[A-Za-z0-9]+` patterns)
4. Get video metadata from initial page state:
   ```javascript
   window.__INITIAL_STATE__.videoData.cid  // CID for subtitle API
   ```
5. Check for subtitles via API: `GET https://api.bilibili.com/x/player/v2?bvid={bvid}&cid={cid}`
   - If `subtitle.subtitles` is empty → use audio extraction + Whisper

### M4S File ID Numbers (Bilibili)
- `30032` — Video stream
- `30216` — Audio stream (64kbps AAC)
- `30280` — Higher quality audio

### yt-dlp (when it works)
```bash
yt-dlp --cookies-from-browser chrome \
  --write-auto-subs --sub-lang zh-CN \
  --skip-download -o /tmp/subtitle \
  https://www.bilibili.com/video/{BV_ID}
```
Often fails with HTTP 412 on Bilibili even with cookies. Prefer the network interception method.

## Generating Voice Replies (TTS)

To send a voice message back to the user, use macOS text-to-speech:

```bash
# For Chinese text, use the Tingting voice:
say -v "Tingting (Chinese (China mainland))" -o /tmp/reply.aiff "你好！这是语音回复"
# For English text:
say -o /tmp/reply.aiff "Your reply text here"
# Convert to OGG:
ffmpeg -y -i /tmp/reply.aiff /tmp/reply_voice.ogg 2>/dev/null
```

Then send via notify_user with `file_path: "/tmp/reply_voice.ogg"`.

Most users prefer TEXT messages. Use voice only when specifically requested.
