### [SGLang](https://github.com/sgl-project/sglang)

> Handle: `sglang`
> URL: [http://localhost:34091](http://localhost:34091)

<div align="center">
<img src="https://raw.githubusercontent.com/sgl-project/sglang/main/assets/logo.png" alt="logo" width="400"></img>

[![PyPI](https://img.shields.io/pypi/v/sglang)](https://pypi.org/project/sglang)
![PyPI - Downloads](https://img.shields.io/pypi/dm/sglang)
[![license](https://img.shields.io/github/license/sgl-project/sglang.svg)](https://github.com/sgl-project/sglang/tree/main/LICENSE)
[![issue resolution](https://img.shields.io/github/issues-closed-raw/sgl-project/sglang)](https://github.com/sgl-project/sglang/issues)
[![open issues](https://img.shields.io/github/issues-raw/sgl-project/sglang)](https://github.com/sgl-project/sglang/issues)
</div>

SGLang is a fast serving framework for large language models and vision language models.

#### Starting

```bash
# [Optional] Pre-pull the image
harbor pull sglang
```

#### Configuration

SGLang is similar to vLLM in the models it can run, so the configuration is similar.

```bash
# Quickly lookup some of the compatible quants
harbor hf find awq
harbor hf find gptq

# Download with HF CLI
harbor hf download bartowski/Meta-Llama-3.1-70B-Instruct-GGUF

# Set the model to run using HF specifier
harbor sglang model google/gemma-2-2b-it

# To run a gated model, ensure that you've
# also set your Huggingface API Token
harbor hf token <your-token>
```

You can specify additional args via `harbor sglang args`:

```bash
# See original CLI help for available options
harbor run sglang --help

# Set the extra arguments via "harbor args"
harbor sglang args --context-length 2048 --disable-cuda-graph
```

- [Official SGLang Documentation](https://sglang.readthedocs.io/en/latest/index.html)
- [Supported Model architectures](https://sglang.readthedocs.io/en/latest/backend.html#supported-models)
