### [TabbyAPI](https://github.com/theroyallab/tabbyAPI)

> Handle: `tabbyapi`<br/>
> URL: [http://localhost:33931](http://localhost:33931)

<p align="left">
    <img src="https://img.shields.io/badge/Python-3.10%20|%203.11%20|%203.12-blue" alt="Python 3.10, 3.11, and 3.12">
    <a href="/LICENSE">
        <img src="https://img.shields.io/badge/License-AGPLv3-blue.svg" alt="License: AGPL v3"/>
    </a>
    <a href="https://discord.gg/sYQxnuD7Fj">
        <img src="https://img.shields.io/discord/545740643247456267.svg?logo=discord&color=blue" alt="Discord Server"/>
    </a>
</p>

<p align="left">
    <a href="https://theroyallab.github.io/tabbyAPI">
        <img src="https://img.shields.io/badge/Documentation-API-orange" alt="Developer facing API documentation">
    </a>
</p>

<p align="left">
    <a href="https://ko-fi.com/I2I3BDTSW">
        <img src="https://img.shields.io/badge/Support_on_Ko--fi-FF5E5B?logo=ko-fi&style=for-the-badge&logoColor=white" alt="Support on Ko-Fi">
    </a>
</p>

An OAI compatible exllamav2 API that's both lightweight and fast

#### Models

- Supports same set of models as exllamav2

##### HuggingFaceDownloader

Harbor integrates with the [HuggingFaceDownloader CLI](./3.-Harbor-CLI-Reference#harbor-hf-dl) which can be used to download models for the TabbyAPI service.

```bash
# [Optional] lookup models on the HF Hub
harbor hf find exl2

# [Optional] If pulling from the closed or gated repo
# Pre-configure the HF access token
harbor hf token <your-token>

# 1. Download the desired model, use "user/repo" specifier
# Note the "./hf" directory set as the download location - this is
# where the HuggingFace cache is mounted for downloader CLI
harbor hf dl -m Annuvin/gemma-2-2b-it-abliterated-4.0bpw-exl2 -s ./hf
harbor hf dl -m bartowski/Phi-3.1-mini-4k-instruct-exl2 -s ./hf -b 8_0

# 2. Set the model to run
# Use the same specifier as for the downloader
harbor tabbyapi model Annuvin/gemma-2-2b-it-abliterated-4.0bpw-exl2
harbor tabbyapi model bartowski/Phi-3.1-mini-4k-instruct-exl2

# 3. Start the service
harbor up tabbyapi
```

##### Native HF CLI

```bash
# Download with a model specifier
harbor hf download ChenMnZ/Mistral-Large-Instruct-2407-EfficientQAT-w2g64-GPTQ
# With a specific revision
harbor hf download turboderp/Llama-3.1-8B-Instruct-exl2 --revision 6.0bpw

# Grab actual name for the folder
harbor find ChenMnZ

# Set the model to run
harbor config set tabbyapi.model.specifier /hub/models--ChenMnZ--Mistral-Large-Instruct-2407-EfficientQAT-w2g64-GPTQ/snapshots/f46105941fa36d2663f77f11840c2f49a69d6681/
```

#### Starting

TabbyAPI exposes an OpenAI-compatible API and can be used with related services directly.

```bash
# [Optional] Pull the tabbyapi images
harbor pull tabbyapi

# Start the service
harbor up tabbyapi

# [Optional] Set additional arguments
harbor tabbyapi args --log-prompt true

# See TabbyAPI docs
harbor tabbyapi docs
```

#### Configuration

- [Server options](https://github.com/theroyallab/tabbyAPI/wiki/02.-Server-options)

Harbor will mount a few volumes for the TabbyAPI container:
- Host HuggingFace cache - `/models/hf`
- [llama.cpp](./2.2.2-Backend:-llama.cpp) cache - `/models/llama.cpp`