### [🚅 LiteLLM](https://docs.litellm.ai/docs/)

> Handle: `litellm`
> URL: [http://localhost:33841/](http://localhost:33841/)

LLM API Proxy/Gateway.

LiteLLM is very useful for setups where the target LLM backend is either:
- Not supported by Harbor directly
- Doesn't have an OpenAI-compatible API that can be plugged into Open WebUI directly (for example, `text-generation-inference`)

`litellm` is also a way to use API-based LLM providers with Harbor.

#### Starting

```bash
# [Optional] Pull the litellm images
# ahead of starting the service
harbor pull litellm

# Start the service
harbor up litellm
```

You'll likely want to start it with at least one of the compatible LLM backends, or pointing to an external API ahead of time, see below.

Harbor's LiteLLM service is configured with a DB and an API key, so that you can access LiteLLM UI. The UI is available on the `/ui` endpoint of the service.

```bash
# Open LiteLLM API docs
harbor open litellm
# Open LiteLLM UI directly
harbor litellm ui
```

You can login with the default credentials: `admin` / `admin`. To adjust either modify the `.env` file or use the `harbor` CLI:
```bash
# Set the new credentials
harbor litellm username paole
harbor litellm password $(tr -dc 'A-Za-z0-9!?%=' < /dev/urandom | head -c 10)
```

#### Configuration

Harbor runs LiteLLM in the [proxy mode](https://docs.litellm.ai/docs/proxy/configs). In order to configure it, you'll need to edit [./litellm/config.yaml](./litellm/config.yaml) file according to the documentation.

For example:

```yaml
model_list:
  # What LiteLLM client will see
  - model_name: sllama
    litellm_params:
      # What LiteLLM will send to downstream API
      model: huggingface/repo/model
      # This can be pointed to one of the compatible Harbor
      # backends or to the external API compatible with the LiteLLM
      api_base: http://tgi:80
  - model_name: llamaster
    litellm_params:
      model: bedrock/meta.llama3-1-405b-instruct-v1:0
      aws_region_name: us-west-2
```

Please see [official LiteLLM documentation](https://docs.litellm.ai/docs/) for plenty of additional examples.
