# SPAvatarKit SDK

Real-time virtual avatar rendering SDK based on 3D Gaussian Splatting, supporting audio-driven animation rendering and high-quality 3D rendering.

## 🚀 Features

- **3D Gaussian Splatting Rendering** - Based on the latest point cloud rendering technology, providing high-quality 3D virtual avatars
- **Audio-Driven Real-Time Animation Rendering** - Users provide audio data, SDK handles receiving animation data and rendering
- **Multi-Character Support** - Support multiple avatar instances simultaneously, each with independent state and rendering
- **WebGPU/WebGL Dual Rendering Backend** - Automatically selects the best rendering backend for compatibility
- **WASM High-Performance Computing** - Uses C++ compiled WebAssembly modules for geometric calculations
- **TypeScript Support** - Complete type definitions and IntelliSense
- **Modular Architecture** - Clear component separation, easy to integrate and extend

## 📦 Installation

```bash
npm install @spatialwalk/avatarkit
```

## 🎯 Quick Start

### Basic Usage

```typescript
import { 
  AvatarKit, 
  AvatarManager, 
  AvatarView,
  Configuration,
  Environment,
  DrivingServiceMode
} from '@spatialwalk/avatarkit'

// 1. Initialize SDK

const configuration: Configuration = {
  environment: Environment.test,
  drivingServiceMode: DrivingServiceMode.sdk, // Optional, 'sdk' is default
  // - DrivingServiceMode.sdk: SDK mode - SDK handles WebSocket communication
  // - DrivingServiceMode.host: Host mode - Host app provides audio and animation data
}

await AvatarKit.initialize('your-app-id', configuration)

// Set sessionToken (if needed, call separately)
// AvatarKit.setSessionToken('your-session-token')

// 2. Load character
const avatarManager = AvatarManager.shared
const avatar = await avatarManager.load('character-id', (progress) => {
  console.log(`Loading progress: ${progress.progress}%`)
})

// 3. Create view (automatically creates Canvas and AvatarController)
// The playback mode is determined by drivingServiceMode in AvatarKit configuration
// - DrivingServiceMode.sdk: SDK mode - SDK handles WebSocket communication
// - DrivingServiceMode.host: Host mode - Host app provides audio and animation data
const container = document.getElementById('avatar-container')
const avatarView = new AvatarView(avatar, container)

// 4. Start real-time communication (SDK mode only)
await avatarView.avatarController.start()

// 5. Send audio data (SDK mode, must be 16kHz mono PCM16 format)
const audioData = new ArrayBuffer(1024) // Example: 16kHz PCM16 audio data
avatarView.avatarController.send(audioData, false) // Send audio data
avatarView.avatarController.send(audioData, true) // end=true marks the end of current conversation round
```

### Host Mode Example

```typescript

// 1-3. Same as SDK mode (initialize SDK, load character)

// 3. Create view with Host mode
const container = document.getElementById('avatar-container')
const avatarView = new AvatarView(avatar, container)

// 4. Host Mode Workflow:
// Send audio data first to get conversationId, then use it to send animation data
const conversationId = avatarView.avatarController.yieldAudioData(audioData, false)
avatarView.avatarController.yieldFramesData(animationData, conversationId)
```

### Complete Examples

Check the example code in the GitHub repository for complete usage flows for both modes.

**Example Project:** [AvatarKit-Web-Demo](https://github.com/spatialwalk/AvatarKit-Web-Demo)

This repository contains complete examples for Vanilla JS, Vue 3, and React, demonstrating:
- SDK mode: Real-time audio input with automatic animation data reception
- Host mode: Custom data sources with manual audio/animation data management

## 🏗️ Architecture Overview

### Three-Layer Architecture

The SDK uses a three-layer architecture for clear separation of concerns:

1. **Rendering Layer (AvatarView)** - Responsible for 3D rendering only
2. **Playback Layer (AvatarController)** - Manages audio/animation synchronization and playback
3. **Network Layer** - Handles WebSocket communication (only in SDK mode, internal implementation)

### Core Components

- **AvatarKit** - SDK initialization and management
- **AvatarManager** - Character resource loading and management
- **AvatarView** - 3D rendering view (rendering layer)
- **AvatarController** - Audio/animation playback controller (playback layer)

### Playback Modes

The SDK supports two playback modes, configured in `AvatarKit.initialize()`:

#### 1. SDK Mode (Default)
- Configured via `drivingServiceMode: DrivingServiceMode.sdk` in `AvatarKit.initialize()`
- SDK handles WebSocket communication automatically
- Send audio data via `AvatarController.send()`
- SDK receives animation data from backend and synchronizes playback
- Best for: Real-time audio input scenarios

#### 2. Host Mode
- Configured via `drivingServiceMode: DrivingServiceMode.host` in `AvatarKit.initialize()`
- Host application manages its own network/data fetching
- Host application provides both audio and animation data
- SDK only handles synchronized playback
- Best for: Custom data sources, pre-recorded content, or custom network implementations

**Note:** The playback mode is determined by `drivingServiceMode` in `AvatarKit.initialize()` configuration.

### Fallback Mechanism

The SDK includes a fallback mechanism to ensure audio playback continues even when animation data is unavailable:

- **SDK Mode Connection Failure**: If WebSocket connection fails to establish within 15 seconds, the SDK automatically enters fallback mode. In this mode, audio data can still be sent and will play normally, even though no animation data will be received from the server. This ensures that audio playback is not interrupted even when the service connection fails.
- **SDK Mode Server Error**: If the server returns an error after connection is established, the SDK automatically enters audio-only mode for that session and continues playing audio independently.
- **Host Mode**: If empty animation data is provided (empty array or undefined), the SDK automatically enters audio-only mode.
- Once in audio-only mode, any subsequent animation data for that session will be ignored, and only audio will continue playing.
- The fallback mode is interruptible, just like normal playback mode.
- Connection state callbacks (`onConnectionState`) will notify you when connection fails or times out, allowing you to handle the fallback state appropriately.

### Data Flow

#### SDK Mode Flow

```
User audio input (16kHz mono PCM16) 
    ↓
AvatarController.send() 
    ↓
WebSocket → Backend processing
    ↓
Backend returns animation data (FLAME keyframes) 
    ↓
AvatarController → AnimationPlayer
    ↓
FLAME parameters → AvatarCore.computeFrameFlatFromParams() → Splat data
    ↓
AvatarController (playback loop) → AvatarView.renderRealtimeFrame()
    ↓
RenderSystem → WebGPU/WebGL → Canvas rendering
```

#### Host Mode Flow

```
External data source (audio + animation)
    ↓
AvatarController.yieldAudioData(audioChunk) // Returns conversationId
    ↓
AvatarController.yieldFramesData(keyframes, conversationId)
    ↓
AvatarController → AnimationPlayer (synchronized playback)
    ↓
FLAME parameters → AvatarCore.computeFrameFlatFromParams() → Splat data
    ↓
AvatarController (playback loop) → AvatarView.renderRealtimeFrame()
    ↓
RenderSystem → WebGPU/WebGL → Canvas rendering
```

### Audio Format Requirements

**⚠️ Important:** The SDK requires audio data to be in **16kHz mono PCM16** format:

- **Sample Rate**: 16kHz (16000 Hz) - This is a backend requirement
- **Channels**: Mono (single channel)
- **Format**: PCM16 (16-bit signed integer, little-endian)
- **Byte Order**: Little-endian

**Audio Data Format:**
- Each sample is 2 bytes (16-bit)
- Audio data should be provided as `ArrayBuffer` or `Uint8Array`
- For example: 1 second of audio = 16000 samples × 2 bytes = 32000 bytes

**Resampling:**
- If your audio source is at a different sample rate (e.g., 24kHz, 48kHz), you must resample it to 16kHz before sending to the SDK
- For high-quality resampling, we recommend using Web Audio API's `OfflineAudioContext` with anti-aliasing filtering
- See example projects for resampling implementation

## 📚 API Reference

### AvatarKit

The core management class of the SDK, responsible for initialization and global configuration.

```typescript
// Initialize SDK
await AvatarKit.initialize(appId: string, configuration: Configuration)

// Check initialization status
const isInitialized = AvatarKit.isInitialized

// Get initialized app ID
const appId = AvatarKit.appId

// Get configuration
const config = AvatarKit.configuration

// Set sessionToken (if needed, call separately)
AvatarKit.setSessionToken('your-session-token')

// Set userId (optional, for telemetry)
AvatarKit.setUserId('user-id')

// Get sessionToken
const sessionToken = AvatarKit.sessionToken

// Get userId
const userId = AvatarKit.userId

// Get SDK version
const version = AvatarKit.version

// Cleanup resources (must be called when no longer in use)
AvatarKit.cleanup()
```

### AvatarManager

Character resource manager, responsible for downloading, caching, and loading character data. Use the singleton instance via `AvatarManager.shared`.

```typescript
// Get singleton instance
const manager = AvatarManager.shared

// Load character
const avatar = await manager.load(
  characterId: string, 
  onProgress?: (progress: LoadProgressInfo) => void
)

// Clear cache
manager.clearCache()
```

### AvatarView

3D rendering view (rendering layer), responsible for 3D rendering only. Internally automatically creates and manages `AvatarController`.

```typescript
constructor(avatar: Avatar, container: HTMLElement)
```

**Parameters:**
- `avatar`: Avatar 实例
- `container`: Canvas 容器元素（必选）
  - Canvas 自动使用容器的完整尺寸（宽度和高度）
  - Canvas 宽高比适应容器尺寸 - 设置容器尺寸以控制宽高比
  - Canvas 会自动添加到容器中
  - SDK automatically handles resize events via ResizeObserver

**Playback Mode:**
- The playback mode is determined by `drivingServiceMode` in `AvatarKit.initialize()` configuration
- The playback mode is fixed when creating `AvatarView` and persists throughout its lifecycle
- Cannot be changed after creation

```typescript
// Create view (Canvas is automatically added to container)
const container = document.getElementById('avatar-container')
const avatarView = new AvatarView(avatar, container)

// Wait for first frame to render
await avatarView.ready // Promise that resolves when the first frame is rendered

// Cleanup resources (must be called before switching characters)
avatarView.dispose()
```

**Character Switching Example:**

```typescript
// To switch characters, simply dispose the old view and create a new one
if (currentAvatarView) {
  currentAvatarView.dispose()
}

// Load new character
const newAvatar = await avatarManager.load('new-character-id')

// Create new AvatarView
currentAvatarView = new AvatarView(newAvatar, container)

// SDK mode: start connection (will throw error if not in SDK mode)
  await currentAvatarView.controller.start()
```

### AvatarController

Audio/animation playback controller (playback layer), manages synchronized playback of audio and animation. Automatically handles WebSocket communication in SDK mode.

**Two Usage Patterns:**

#### SDK Mode Methods

```typescript
// Start WebSocket service
await avatarView.avatarController.start()

// Send audio data (must be 16kHz mono PCM16 format)
const conversationId = avatarView.avatarController.send(audioData: ArrayBuffer, end: boolean)
// Returns: conversationId - Conversation ID for this conversation session
// end: false (default) - Continue sending audio data for current conversation
// end: true - Mark the end of current conversation round. After end=true, sending new audio data will interrupt any ongoing playback from the previous conversation round

// Close WebSocket service
avatarView.avatarController.close()
```

#### Host Mode Methods

```typescript
// Stream audio chunks (must be 16kHz mono PCM16 format)
const conversationId = avatarView.avatarController.yieldAudioData(
  data: Uint8Array,               // Audio chunk data
  isLast: boolean = false         // Whether this is the last chunk
)
// Returns: conversationId - Conversation ID for this audio session

// Stream animation keyframes (requires conversationId from audio data)
avatarView.avatarController.yieldFramesData(
  keyframes: any[],                // Animation keyframes (obtained from your service)
  conversationId: string          // Conversation ID (required)
)
```

**⚠️ Important: Conversation ID (conversationId) Management**

**SDK Mode:**
- `send()` returns a conversationId to distinguish each conversation round
- `end=true` marks the end of a conversation round

**Host Mode:**
- `yieldAudioData()` returns a conversationId (automatically generates if starting new session)
   - `yieldFramesData()` requires a valid conversationId parameter
   - Animation data with mismatched conversationId will be **discarded**
   - Use `getCurrentConversationId()` to retrieve the current active conversationId

#### Common Methods (Both Modes)

```typescript
// Pause playback (can be resumed later)
avatarView.avatarController.pause()

// Resume playback (from paused state)
await avatarView.avatarController.resume()

// Interrupt current playback (stops and clears data)
avatarView.avatarController.interrupt()

// Clear all data and resources
avatarView.avatarController.clear()

// Get current conversation ID (for Host mode)
const conversationId = avatarView.avatarController.getCurrentConversationId()
// Returns: Current conversationId for the active audio session, or null if no active session

// Volume control (affects only avatar audio player, not system volume)
avatarView.avatarController.setVolume(0.5)  // Set volume to 50% (0.0 to 1.0)
const currentVolume = avatarView.avatarController.getVolume()  // Get current volume (0.0 to 1.0)

// Set event callbacks
avatarView.avatarController.onConnectionState = (state: ConnectionState) => {} // SDK mode only
avatarView.avatarController.onConversationState = (state: ConversationState) => {}
avatarView.avatarController.onError = (error: Error) => {}
```

**Important Notes:**
- `start()` and `close()` are only available in SDK mode
- `yieldAudioData()` and `yieldFramesData()` are only available in Host mode
- `pause()`, `resume()`, `interrupt()`, `clear()`, `getCurrentConversationId()`, `setVolume()`, and `getVolume()` are available in both modes
- The playback mode is determined when creating `AvatarView` and cannot be changed

## 🔧 Configuration

### Configuration

```typescript
interface Configuration {
  environment: Environment
  drivingServiceMode?: DrivingServiceMode  // Optional, default is 'sdk' (SDK mode)
}
```

**Description:**
- `environment`: Specifies the environment (cn/intl/test), SDK will automatically use the corresponding API address and WebSocket address based on the environment
- `drivingServiceMode`: Specifies the driving service mode
  - `DrivingServiceMode.sdk` (default): SDK mode - SDK handles WebSocket communication automatically
  - `DrivingServiceMode.host`: Host mode - Host application provides audio and animation data
- `sessionToken`: Set separately via `AvatarKit.setSessionToken()`, not in Configuration

```typescript
enum Environment {
  cn = 'cn',    // China region
  intl = 'intl',    // International region
  test = 'test' // Test environment
}
```

### CameraConfig

```typescript
interface CameraConfig {
  position: [number, number, number]  // Camera position
  target: [number, number, number]    // Camera target
  fov: number                         // Field of view angle
  near: number                        // Near clipping plane
  far: number                         // Far clipping plane
  up?: [number, number, number]       // Up direction
  aspect?: number                     // Aspect ratio
}
```

## 📊 State Management

### ConnectionState

```typescript
enum ConnectionState {
  disconnected = 'disconnected',
  connecting = 'connecting',
  connected = 'connected',
  failed = 'failed'
}
```

### ConversationState

```typescript
enum ConversationState {
  idle = 'idle',      // 呼吸态
  playing = 'playing' // 播放态
}
```

**状态说明：**
- `idle`: 数字人处于呼吸态，等待对话开始
- `playing`: 数字人正在播放对话内容（包括过渡动画期间）

**注意：** 过渡动画期间会提前通知目标状态：
- 从 `idle` 过渡到 `playing` 时，立即通知 `playing` 状态
- 从 `playing` 过渡到 `idle` 时，立即通知 `idle` 状态

## 🎨 Rendering System

The SDK supports two rendering backends:

- **WebGPU** - High-performance rendering for modern browsers
- **WebGL** - Better compatibility traditional rendering

The rendering system automatically selects the best backend, no manual configuration needed.

## 🚨 Error Handling

### SPAvatarError

The SDK uses custom error types, providing more detailed error information:

```typescript
import { SPAvatarError } from '@spatialwalk/avatarkit'

try {
  await avatarView.avatarController.start()
} catch (error) {
  if (error instanceof SPAvatarError) {
    console.error('SDK Error:', error.message, error.code)
  } else {
    console.error('Unknown error:', error)
  }
}
```

### Error Callbacks

```typescript
avatarView.avatarController.onError = (error: Error) => {
  console.error('AvatarController error:', error)
  // Handle error, such as reconnection, user notification, etc.
}
```

## 🔄 Resource Management

### Lifecycle Management

#### SDK Mode Lifecycle

```typescript
// Initialize
const container = document.getElementById('avatar-container')
const avatarView = new AvatarView(avatar, container)
await avatarView.avatarController.start()

// Use
avatarView.avatarController.send(audioData, false)

// Cleanup
avatarView.avatarController.close()
avatarView.dispose() // Automatically cleans up all resources
```

#### Host Mode Lifecycle

```typescript
// Initialize
const container = document.getElementById('avatar-container')
const avatarView = new AvatarView(avatar, container)

// Use
const conversationId = avatarView.avatarController.yieldAudioData(audioChunk, false)
avatarView.avatarController.yieldFramesData(keyframes, conversationId)

// Cleanup
avatarView.avatarController.clear() // Clear all data and resources
avatarView.dispose() // Automatically cleans up all resources
```

**⚠️ Important Notes:** 
- When disposing AvatarView instances, must call `dispose()` to properly clean up resources
- Not properly cleaning up may cause resource leaks and rendering errors
- In SDK mode, call `close()` before `dispose()` to properly close WebSocket connections
- In Host mode, call `clear()` before `dispose()` to clear all playback data

### Memory Optimization

- SDK automatically manages WASM memory allocation
- Supports dynamic loading/unloading of character and animation resources
- Provides memory usage monitoring interface

## 🌐 Browser Compatibility

- **Chrome/Edge** 90+ (WebGPU recommended)
- **Firefox** 90+ (WebGL)
- **Safari** 14+ (WebGL)
- **Mobile** iOS 14+, Android 8+

## 📝 License

MIT License

## 🤝 Contributing

Issues and Pull Requests are welcome!

## 📞 Support

For questions, please contact:
- Email: support@spavatar.com
- Documentation: https://docs.spatialreal.ai
- GitHub: https://github.com/spavatar/sdk
