### [Aphrodite Engine](https://github.com/PygmalionAI/aphrodite-engine)

> Handle: `aphrodite`
> URL: [http://localhost:33921](http://localhost:33921)

![aphrodite](https://raw.githubusercontent.com/PygmalionAI/aphrodite-engine/main/assets/aphrodite.png)

PygmalionAI's large-scale inference engine

### Starting

```bash
# [Optional] pre-pull the image, ~5GB
harbor pull aphrodite

# Start the service
harbor up aphrodite

# [Optional] When loading closed/gated models
# provision the token
harbor hf token <your-token>
```

### Models

- [Official Quantization docs](https://aphrodite.pygmalion.chat/pages/quantization/support-matrix.html)
- [Supported Model Architectures](https://aphrodite.pygmalion.chat/pages/usage/models.html)

```bash
# Open HF Search to find the models
harbor find gptq awq

# Download model repo to the global HF cache
# user/repo format
harbor hf download infly/INF-34B-Chat-AWQ

# Get/set the model to run
# in the aphrodite engine
harbor aphrodite model infly/INF-34B-Chat-AWQ
```

### Configuration

- [Official Engine Options docs](https://aphrodite.pygmalion.chat/usage/openai/#command-line-arguments-for-the-server)

```bash
# See available options
harbor run aphrodite --help

# Get/Set the extra arguments for
# the aphrodite engine
harbor aphrodite args
```

#### Set specific version

You can adjust used version (docker image tag) of the engine:

```bash
# Get the current version - "latest" by default
harbor config get aphrodite.version

# Set the version
harbor config set aphrodite.version latest
```
