### [mistral.rs](https://github.com/EricLBuehler/mistral.rs)

> Handle: `mistralrs`
> URL: [http://localhost:33951](http://localhost:33951)

Blazingly fast LLM inference.

<p align="center">
| <a href="https://ericlbuehler.github.io/mistral.rs/mistralrs/"><b>Rust Documentation</b></a> | <a href="https://github.com/EricLBuehler/mistral.rs/blob/master/mistralrs-pyo3/API.md"><b>Python Documentation</b></a> | <a href="https://discord.gg/SZrecqK8qw"><b>Discord</b></a> | <a href="https://matrix.to/#/#mistral.rs:matrix.org"><b>Matrix</b></a> |
</p>

Mistral.rs is a fast LLM inference platform supporting inference on a variety of devices, quantization, and easy-to-use application with an Open-AI API compatible HTTP server and Python bindings.

#### Starting

```bash
# [Optional] Pull the mistralrs images
harbor pull mistralrs

# Start the service
harbor up mistralrs

# Verify service health and logs
harbor mistralrs health
harbor logs mistralrs -n 200
```

#### Models

- [Official compatibility matrix](https://github.com/EricLBuehler/mistral.rs?tab=readme-ov-file#supported-models)
- [Supported architectures in the CLI](https://github.com/EricLBuehler/mistral.rs?tab=readme-ov-file#run-with-the-cli)

```bash
# Open HF Search to find compatible models
harbor hf find gemma2
```

##### Plain models

```bash
# For "plain" models:
# Download the model to the global HF cache
harbor hf download IlyaGusev/gemma-2-2b-it-abliterated
# Set model/type/arch
harbor mistralrs model IlyaGusev/gemma-2-2b-it-abliterated
harbor mistralrs type plain
harbor mistralrs arch gemma2
# Gemma 2 doesn't support paged attention
harbor mistralrs args --no-paged-attn
# Launch, mistralrs
# Running model will be available in the webui
harbor up mistralrs
```

##### ISQ

mistral.rs supports an interesting technique of [in situ quantization](https://github.com/EricLBuehler/mistral.rs/blob/master/docs/ISQ.md)

```bash
# Gemma2 from the previous example
# IlyaGusev/gemma-2-2b-it-abliterated
# nvidia-smi > 5584MiB

# Enable ISQ
harbor mistralrs isq Q2K
# Restart the service
harbor restart mistralrs
# nvidia-smi > 2094MiB

# Disable ISQ if not needed
harbor mistralrs isq ""
```

The difference will increase for models with larger `Linear` layers. Note that ISQ will affect the performance of the model.

##### GGUF models

Harbor mounts global `llama.cpp` cache to the `mistralrs` service as a `gguf` folder. You can download models in the same way as for `llama.cpp`.

```bash
# Set the model type to GGUF
harbor mistralrs type gguf

# - Unset ISQ off, as it's not supported
# for GGUF models
# - For GGUFs, architecture is inferred from the file
harbor mistralrs isq ""
harbor mistralrs arch ""

# Example 1: llama.cpp cache

# [Optional] See which GGUFs were already downloaded for the llama.cpp
# `config get llamacpp.cache` is also a folder Harbor will mount for Mistral.rs
ls $(eval echo "$(harbor config get llamacpp.cache)")

# Use "folder" specifier to point to the model
# "gguf"          - mounted llama.cpp cache
# "-f Model.gguf" - the model file
harbor mistralrs model "gguf -f Phi-3-mini-4k-instruct-q4.gguf"

# Example 2: HF cache

# [Optional] Grab the folder where the model is located
harbor hf scan-cache

# Use "folder" specifier to point to the model
# "hf/full/path"  - mounted HF cache. Note that you need
#                   a full path to the folder with .gguf
# "-f Model.gguf" - the model file
harbor mistralrs model "hf/hub/models--microsoft--Phi-3-mini-4k-instruct-gguf/snapshots/999f761fe19e26cf1a339a5ec5f9f201301cbb83/ -f Phi-3-mini-4k-instruct-q4.gguf"

# When configured, launch
harbor up mistralrs
```

#### Configuration

Specify extra args via the Harbor CLI:

```bash
# See available options
harbor run mistralrs --help

# Get/Set the extra arguments
harbor mistralrs args --no-paged-attn
```
