### [vLLM](https://github.com/vllm-project/vllm)

> Handle: `vllm`
> URL: [http://localhost:33911](http://localhost:33911)

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/vllm-project/vllm/main/docs/source/assets/logos/vllm-logo-text-dark.png">
    <img alt="vLLM" src="https://raw.githubusercontent.com/vllm-project/vllm/main/docs/source/assets/logos/vllm-logo-text-light.png" width=55%>
  </picture>
</p>

A high-throughput and memory-efficient inference and serving engine for LLMs

#### Models

- [Official vLLM models](https://docs.vllm.ai/en/latest/models/supported_models.html)
- [Quantization kernels](https://docs.vllm.ai/en/latest/quantization/supported_hardware.html)

Once you've found a model you want to run, you can configure it with Harbor:

```bash
# Quickly lookup some of the compatible quants
harbor hf find awq
harbor hf find gptq

# This propagates the settings
# to the relevant configuration files
harbor vllm model google/gemma-2-2b-it

# To run a gated model, ensure that you've
# also set your Huggingface API Token
harbor hf token <your-token>
```

#### Starting

```bash
harbor up vllm
```

Models served by vLLM should be available in the Open WebUI by default.

#### Configuration

- [Engine CLI Arguments](https://docs.vllm.ai/en/stable/serving/engine_args.html)
- [Environment Variables](https://docs.vllm.ai/en/stable/serving/env_vars.html)

You can configure specific portions of vllm via Harbor CLI:

```bash
# See original CLI help
harbor run vllm --help

# Get/Set the extra arguments
harbor vllm args
harbor vllm args '--dtype bfloat16 --code-revision 3.5'

# Select attention backend
harbor vllm attention ROCM_FLASH

harbor config set vllm.host.port 4090

# Get/set desired vLLM version
harbor vllm version # v0.5.3
# Command accepts a docker tag
harbor vllm version latest
```

You can specify more options directly via the `.env` file.

#### VRAM

Below are some steps to take if running out of VRAM (no magic, though).

##### Limit Context Length

You can limit the context length to reduce the memory footprint. This can be done via the `--max-model-len` flag.

```bash
harbor vllm args --max-model-len 2048
```

##### Quantization

vLLM supports many different quantization formats. You would typically configure this via `--load-format` and `--quantization` flags. For example:

```bash
harbor vllm args --load-format bitsandbytes --quantization bitsandbytes
```

##### Offloading

vLLM supports partial offloading to the CPU, similar to [llama.cpp](./2.2.2-Backend:-llama.cpp) and some other backends. This can be configured via the `--cpu-offload-gb` flag.

```bash
harbor vllm args --cpu-offload-gb 4
```

##### Disable CUDA Graphs

When loading the model, VRAM usage can spike when computing the CUDA graphs. This can be disabled via `--enforce-eager` flag.

```bash
harbor vllm args --enforce-eager
```

##### GPU Memory Utilization

Reduce the amount of VRAM allocated for the model executor. Can be ranged from 0 to 1.0, `0.9` by default.

```bash
harbor vllm args --gpu-memory-utilization 0
```

##### Run on CPU

You can move to CPU by setting the `--device cpu` flag.

```bash
harbor vllm args --device cpu
```