### [llama-swap](https://github.com/mostlygeek/llama-swap)

> Handle: `llamaswap`<br/>
> URL: [http://localhost:34401](http://localhost:34401)

llama-swap is a lightweight, transparent proxy server that provides automatic model swapping to llama.cpp's server.

### Starting

```bash
# [Optional] pre-pull the image
harbor pull llamaswap

# Run the service
harbor up llamaswap
```

- `llamaswap` image in Harbor will run its own llama.cpp server, that is different from the one running in the [`llamacpp`](./2.2.2-Backend&colon-llama.cpp) service
- Harbor will connect `llamaswap` to [Open WebUI](./2.1.1-Frontend&colon-Open-WebUI) when run together
- Harbor will mount following local caches to be available within llama-swap container:
  - Ollama - `/root/.ollama`
  - Hugging Face - `/root/.cache/huggingface`
  - llama.cpp - `/root/.cache/llama.cpp`
  - vLLM - `/root/.cache/vllm`

### Configuration

Expected way to configure llama-swap is by editing the `config.yaml` file:

```bash
# Open in your default editor
open $(harbor home)/llamaswap/config.yaml
```

See [official configuration example](https://github.com/mostlygeek/llama-swap/blob/main/config.example.yaml) for reference.