# picoLLM Inference Engine Web Binding

Made in Vancouver, Canada by [Picovoice](https://picovoice.ai)

## picoLLM Inference Engine

picoLLM Inference Engine is a highly accurate and cross-platform SDK optimized for running compressed large language
models. picoLLM Inference Engine is:

- Accurate; picoLLM Compression improves GPTQ by [significant margins](https://picovoice.ai/blog/picollm-towards-optimal-llm-quantization/)
- Private; LLM inference runs 100% locally.
- Cross-Platform
- Runs on CPU and GPU
- Free for open-weight models

## Compatibility

- Chrome / Edge
- Firefox
- Safari

## Requirements

PicoLLM Web Binding uses [SharedArrayBuffer](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer).

Include the following headers in the response to enable the use of `SharedArrayBuffers`:

```
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp
```

Refer to our [Web demo](../../demo/web) for an example on creating a server with the corresponding response headers.

**NOTE**: IndexedDB, SIMD and SharedArrayBuffers are required to use `picoLLM`. `PicoLLM` only supports running in a worker thread.

## Installation

Using `Yarn`:

```console
yarn add @picovoice/picollm-web
```

or using `npm`:

```console
npm install --save @picovoice/picollm-web
```

## Models

picoLLM Inference Engine on Web supports the following open-weight models. The models are on
[Picovoice Console](https://console.picovoice.ai/).

- DeepSeek-OCR-2
  - `deepseek-ocr-2`
- EmbeddingGemma
  - `embeddinggemma-300m`
- Gemma
  - `gemma-2b`
  - `gemma-2b-it`
- Gemma3
  - `gemma-3-270m`
  - `gemma-3-270m-it`
- Llama-2
  - `llama-2-7b`
  - `llama-2-7b-chat`
- Llama-3
  - `llama-3-8b`
  - `llama-3-8b-instruct`
- Llama-3.2
  - `llama3.2-1b-instruct`
  - `llama3.2-3b-instruct`
- Mistral
  - `mistral-7b-v0.1`
  - `mistral-7b-instruct-v0.1`
  - `mistral-7b-instruct-v0.2`
- Phi-2
  - `phi2`
- Phi-3
  - `phi3`
- Phi-3.5
  - `phi3.5`
- Qwen3-VL
  - `qwen3-vl-2b-it`

**NOTE**: Only Gemma, Phi-2, and Phi-3 models have been tested on multiple browsers across different platforms.
The rest of the models depend on the user's system in order to run properly.

## AccessKey

AccessKey is your authentication and authorization token for deploying Picovoice SDKs, including picoLLM. Anyone who is
using Picovoice needs to have a valid AccessKey. You must keep your AccessKey secret. You would need internet
connectivity to validate your AccessKey with Picovoice license servers even though the LLM inference is running 100%
offline and completely free for open-weight models. Everyone who signs up for
[Picovoice Console](https://console.picovoice.ai/) receives a unique AccessKey.

## Usage

### picoLLM Model File Types

`picoLLM` accepts model files in three different types:

#### File URL(s):

```typescript
const modelFile = `${SERVER_URL}/${PATH_TO_MODEL_FILE}`;
```

or if the model file is too big (2GB or larger) consider using chunks:

```typescript
const modelFile = [
  `${SERVER_URL}/${PATH_TO_MODEL_FILE_PART1}`,
  `${SERVER_URL}/${PATH_TO_MODEL_FILE_PART2}`,
  `...` // add more parts if needed
];
```

#### File Object(s):

```typescript
const modelFile = new File([/* file contents */]);
```

or if the model file is too big (2GB or larger) consider using chunks:

```typescript
const modelFile = [
  new File([/* file contents part 1 */]),
  new File([/* file contents part 2 */]),
  ... // add more parts if needed
];
```

File objects are usually used with HTML's input tag:

```html
<input id="modelFile" type="file" accept="pllm" />

<script>
  const modelFile = document.getElementById("modelFile").files;
</script>
```

#### Blob Object(s):

```typescript
const modelFile = new Blob([new Uint8Array(/* model bytes */)]);
```

or if the model file is too big (2GB or larger) consider using chunks:

```typescript
const modelFile = [
  new Blob([new Uint8Array(/* model bytes part 1 */)]),
  new Blob([new Uint8Array(/* model bytes part 2 */)]),
  ... // add more parts if needed
];
```

#### picoLLM Model

`picoLLM` saves and caches your parameter model file (`.pllm`) in IndexedDB to be
used by Web Assembly. Use a different `cacheFilePath` variable to hold and cache
multiple model values and set the `cacheFileOverwrite` value to true to force
re-save the model file. If the model file changes, `cacheFileVersion` should be
incremented to force the cached models to be updated. Use `numFetchRetries` to
change the number of fetch retry attempts for the model file.

```typescript
const picoLLMModel = {
  modelFile: modelFile, // Based on the sections before,

  // Optional
  cacheFilePath: 'custom_model',
  cacheFileOverwrite: true,
  cacheFileVersion: 1,
  numFetchRetries: 0,
}
```

### Initialize picoLLM

Initialize an instance of `picoLLM` in a worker thread:

```typescript
const picoLLM = await PicoLLMWorker.create(
  ${ACCESS_KEY},
  picoLLMModel,
);
```

Replace `${ACCESS_KEY}` with yours obtained from Picovoice Console.

### Generate Completion (Text Models)

```typescript
const res = await picoLLM.generate(`${PROMPT}`);
console.log(res.completion);
```

Replace `${PROMPT}` with a prompt string.

### Instruction-tuned models

Instruction-tuned models (e.g., `llama-2-7b-chat`, and `gemma-2b-it`) have a specific chat
template. You can either directly format the prompt or use a dialog helper:

```typescript
const dialog = picoLLM.getDialog()
dialog.addHumanRequest(prompt)

const res = await picoLLM.generate(dialog.prompt())
dialog.addLLMResponse(res.completion)
print(res.completion)
```

### Interrupt Text Generation

```typescript
picoLLM.interrupt();
```

This will stop text generation and if it was properly interrupted, it will set `res.completion.endpoint`
as an interrupted state.

### Clean Up

Clean up used resources by `picoLLM` or `picoLLMWorker`:

```typescript
await picoLLM.release()
```

### Vision models

To run a VLM such as `qwen3-vl-2b-it`:

```typescript
const image: PicoLLMImage = {
  width: ${IMAGE_NUM_PIXELS_WIDTH},
  height: ${IMAGE_NUM_PIXELS_HEIGHT},
  data: ${IMAGE_DATA},
};

const res = await picoLLM.generateWithImage(`${PROMPT}`, image);
console.log(res.completion);
```

Replace `${PROMPT}` with a text prompt. For the image, you will need to get image height and width in number of pixels and the raw pixel values of the image in 8-bit, RGB format.

### OCR models

To run an OCR model such as `deepseek-ocr-2`:

```typescript
const image: PicoLLMImage = {
  width: ${IMAGE_NUM_PIXELS_WIDTH},
  height: ${IMAGE_NUM_PIXELS_HEIGHT},
  data: ${IMAGE_DATA},
};

const res = await picoLLM.generateOCR(image);
console.log(res.completion);
```

For the image, you will need to get image height and width in number of pixels and the raw pixel values of the image in 8-bit, RGB format.

### Embedding models

To run an embedding model such as `embeddinggemma-300m`:

```typescript
const res = await picoLLM.generateEmbeddings(`${PROMPT}`, image);
res.embeddings.forEach(embedding => {
  console.log(embedding);
});
```

Replace `${PROMPT}` with a text prompt that you want to generate embeddings for.

## Demos

Refer to our [Web demos](https://github.com/Picovoice/picollm/tree/master/demo/web) for examples using LLM completion
and chat using `picoLLM`.
