# Qwen3.5 llama.cpp Deployment Guide

How to run Qwen3.5 35B A3B with the official Qwen3 chat template, LoRA adapters, and structured tool call output via llama.cpp.

## Prerequisites

- [llama.cpp](https://github.com/ggml-org/llama.cpp) built with CUDA/Metal support
- Qwen3.5 35B A3B GGUF model (e.g. `qwen3.5-a3b-iq4xs.gguf`)
- (Optional) Draft model for speculative decoding: `Qwen3.5-0.8B-Q8_0.gguf`
- (Optional) LoRA adapter GGUF for improved tool call reliability

## Quick Start

```bash
llama-server \
  --model /path/to/qwen3.5-a3b-iq4xs.gguf \
  --chat-template-file chat_template.jinja \
  --n-predict 16384 \
  --temp 0.6 --top-p 0.9 --top-k 20 --min-p 0.05 \
  --repeat-penalty 1.0 \
  --threads 8 --ctx-size 131072 --batch-size 8 \
  --gpu-layers 35 --mlock --flash-attn
```

## Configuration Files

| File                                        | Purpose                                                             |
| ------------------------------------------- | ------------------------------------------------------------------- |
| `chat_template.jinja`                       | Official Qwen3 chat template with native tool descriptions          |
| `tools/agents/config/tool-call.gbnf`        | GBNF grammar for per-request use (do NOT use with `--grammar-file`) |
| `tools/agents/config/tool-call-schema.json` | JSON Schema for the tool call payload                               |
| `config/qwen35-settings.json`               | Full model settings, optimization config                            |
| `config/lora-finetune.yaml`                 | LoRA training configuration (axolotl/unsloth compatible)            |

## Important: Do NOT Use `--grammar-file`

The `--grammar-file` flag applies a GBNF grammar **globally to every completion**. This breaks normal chat because the grammar forces `<tool_call>` output even when no tools are provided.

llama.cpp's **differential autoparser** handles tool calls automatically:

1. It analyzes the Jinja template to discover `<tool_call>`/`</tool_call>` markers
2. It generates PEG grammar rules with **lazy activation** (`grammar_lazy = true`)
3. When `tool_choice == "auto"`, the model generates freely until it emits `<tool_call>`, at which point the grammar activates to constrain the JSON payload
4. After `</tool_call>`, the grammar allows another `<tool_call>` for parallel calls
5. Plain chat (no tools) is unconstrained

The GBNF file is kept in the repo for per-request use via the `grammar` field in API payloads, but should never be a server startup flag.

## Server Configurations

### Basic (no LoRA, no speculative decoding)

```bash
llama-server \
  --model /path/to/qwen3.5-a3b-iq4xs.gguf \
  --chat-template-file chat_template.jinja \
  --n-predict 16384 \
  --temp 0.6 --top-p 0.9 --top-k 20 --min-p 0.05 \
  --repeat-penalty 1.0 \
  --threads 8 --ctx-size 131072 --batch-size 8 \
  --gpu-layers 35 --mlock --flash-attn
```

### With LoRA Adapter

```bash
llama-server \
  --model /path/to/qwen3.5-a3b-iq4xs.gguf \
  --lora /path/to/qwen35-tool-call-lora/adapter.gguf \
  --lora-scale 1.0 \
  --chat-template-file chat_template.jinja \
  --n-predict 16384 \
  --temp 0.6 --top-p 0.9 --top-k 20 --min-p 0.05 \
  --repeat-penalty 1.0 \
  --threads 8 --ctx-size 131072 --batch-size 8 \
  --gpu-layers 35 --mlock --flash-attn
```

### Full Setup (LoRA + Speculative Decoding)

```bash
llama-server \
  --model /path/to/qwen3.5-a3b-iq4xs.gguf \
  --lora /path/to/qwen35-tool-call-lora/adapter.gguf \
  --lora-scale 1.0 \
  --chat-template-file chat_template.jinja \
  --draft-model /path/to/Qwen3.5-0.8B-Q8_0.gguf \
  --draft-max 16 --draft-p-min 0.75 \
  --n-predict 16384 \
  --temp 0.6 --top-p 0.9 --top-k 20 --min-p 0.05 \
  --repeat-penalty 1.0 \
  --threads 8 --ctx-size 131072 --batch-size 8 \
  --gpu-layers 35 --mlock --flash-attn
```

## Key Parameters

### Chat Template & Tool Calls

| Flag                   | Value                 | Purpose                                                                                                                                                        |
| ---------------------- | --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--chat-template-file` | `chat_template.jinja` | Official Qwen3 template with native `tools` block. llama.cpp's autoparser discovers `<tool_call>` markers and generates lazy grammar + triggers automatically. |

### LoRA

| Flag           | Value                  | Purpose                                                                                                |
| -------------- | ---------------------- | ------------------------------------------------------------------------------------------------------ |
| `--lora`       | Path to `adapter.gguf` | Loads LoRA adapter at runtime (no model merge needed). Improves tool call format adherence by ~15-20%. |
| `--lora-scale` | `0.0` - `1.0`          | Adapter strength. Use `1.0` for full effect, `0.5`-`0.8` to blend with base model behavior.            |

### Speculative Decoding

| Flag            | Value                            | Purpose                                                                 |
| --------------- | -------------------------------- | ----------------------------------------------------------------------- |
| `--draft-model` | Path to `Qwen3.5-0.8B-Q8_0.gguf` | Small draft model proposes tokens verified by the main model.           |
| `--draft-max`   | `16`                             | Max tokens to draft per iteration. Higher = more throughput, more VRAM. |
| `--draft-p-min` | `0.75`                           | Minimum acceptance probability. Lower = more aggressive drafting.       |

## Extension Options for Speculative Decoding

### Option 1: Adaptive Runtime Tuning (implemented)

Use acceptance and rollback rates to auto-adjust `draft-max`, `draft-min`, and `draft-p-min` over time.

- Best for immediate gains without kernel changes
- Reduces bad bursts when acceptance drops
- Increases burst length automatically during high-acceptance windows

Commands:

```bash
# Tune once from observed metrics
llama-optimize spec-autotune --acceptance 0.71 --rollback 0.14 --profile throughput

# Compare static defaults vs adaptive tuning using deterministic simulation
llama-optimize spec-benchmark --profile throughput --trace mixed --steps 180

# Live benchmark active server and get tuned flag recommendation
llama-optimize spec-benchmark-live \
  --endpoint http://127.0.0.1:8080/v1 \
  --model qwen3.5-a3b-iq4xs \
  --runs 5 --max-tokens 256 --profile throughput
```

Recommended workflow:

1. Run `spec-benchmark-live` with your current startup flags and note `Throughput`.
2. Restart `llama-server` with the `Suggested params` flags.
3. Re-run `spec-benchmark-live` with the same settings to measure actual gain.

### Option 2: GPU Residency + Overlap

- Keep draft model and draft KV fully on GPU
- Preallocate buffers and overlap draft + verify passes with CUDA streams
- Improves p95 latency consistency on long runs

### Option 3: GPU Checkpoint/Rollback

- Move speculative checkpoint snapshots from CPU RAM to GPU buffers
- Remove host-device copy overhead from rollback paths
- Highest upside, but requires deeper runtime changes

### Sampling

| Flag               | Value  | Purpose                                           |
| ------------------ | ------ | ------------------------------------------------- |
| `--temp`           | `0.6`  | Low temperature for deterministic tool calls.     |
| `--top-p`          | `0.9`  | Nucleus sampling threshold.                       |
| `--top-k`          | `20`   | Limits token candidates per step.                 |
| `--min-p`          | `0.05` | Filters tokens below 5% of top token probability. |
| `--repeat-penalty` | `1.0`  | No repetition penalty — code naturally repeats patterns. |

### Performance

| Flag           | Value    | Purpose                                           |
| -------------- | -------- | ------------------------------------------------- |
| `--flash-attn` | (flag)   | 1.5-2x speed on long context.                     |
| `--gpu-layers` | `35`     | Layers offloaded to GPU. Increase if VRAM allows. |
| `--ctx-size`   | `131072` | Full 128K context window.                         |
| `--mlock`      | (flag)   | Prevents OS from swapping model to disk.          |

## VRAM Estimates

| Component           | VRAM       |
| ------------------- | ---------- |
| Main model (IQ4_XS) | ~17 GB     |
| Draft model (Q8_0)  | ~0.8 GB    |
| KV cache (128K ctx) | ~2-3 GB    |
| LoRA adapter        | ~50 MB     |
| **Total**           | **~20 GB** |

## Anthropic API Proxy (for Claude Code / Forge Code)

Claude Code and Forge Code speak the Anthropic Messages API, but llama.cpp exposes an OpenAI-compatible API. The UAP Anthropic Proxy bridges this gap by translating between the two protocols in real time, including full streaming and tool calling support.

### Architecture

```
Claude Code  --(Anthropic API :4000)-->  UAP Proxy  --(OpenAI API :8080)-->  llama.cpp
```

### Quick Start

```bash
# Install Python dependencies
pip install -r tools/agents/scripts/requirements-proxy.txt

# Start the proxy (default: listen on :4000, forward to llama.cpp on :8080)
python tools/agents/scripts/anthropic_proxy.py
```

### Configuration

All settings are via environment variables:

| Variable                | Default                              | Description                              |
| ----------------------- | ------------------------------------ | ---------------------------------------- |
| `LLAMA_CPP_BASE`        | `http://192.168.1.165:8080/v1`       | OpenAI-compatible upstream server URL    |
| `PROXY_PORT`            | `4000`                               | Port for the proxy to listen on          |
| `PROXY_HOST`            | `0.0.0.0`                            | Host/IP to bind to                       |
| `PROXY_LOG_LEVEL`       | `INFO`                               | Logging level (DEBUG/INFO/WARNING/ERROR) |
| `PROXY_READ_TIMEOUT`    | `600`                                | Read timeout (seconds) for LLM streaming |
| `PROXY_MAX_CONNECTIONS` | `20`                                 | Max concurrent upstream connections      |
| `PROXY_MAX_TOKENS_FLOOR` | `16384`                             | Minimum floor applied to incoming `max_tokens` (`0` disables floor) |
| `PROXY_CONTEXT_PRUNE_TARGET_FRACTION` | `0.65`            | Target context utilization after pruning (`0.0 < value < 1.0`) |
| `PROXY_STREAM_REASONING_FALLBACK` | `off`                      | Streaming behavior for reasoning-only empty turns (`off`, `sanitized`, `visible`) |
| `PROXY_STREAM_REASONING_MAX_CHARS` | `240`                      | Max fallback length when `PROXY_STREAM_REASONING_FALLBACK=sanitized` |
| `PROXY_TOOL_NARROWING` | `off`                                | Narrow large tool lists to top relevant tools per turn |
| `PROXY_TOOL_NARROWING_KEEP` | `8`                             | Number of tools to keep when narrowing is enabled |
| `PROXY_TOOL_NARROWING_MIN_TOOLS` | `12`                       | Minimum tool count before narrowing activates |
| `PROXY_DISABLE_THINKING_ON_TOOL_TURNS` | `off`                 | Sends `enable_thinking=false` when tools are present |
| `PROXY_MALFORMED_TOOL_GUARDRAIL` | `on`                      | Detects malformed pseudo tool payloads and retries with strict settings |
| `PROXY_MALFORMED_TOOL_RETRY_MAX` | `1`                        | Number of malformed-tool retries |
| `PROXY_MALFORMED_TOOL_RETRY_MAX_TOKENS` | `2048`             | Retry cap for `max_tokens` during malformed-tool recovery |
| `PROXY_MALFORMED_TOOL_RETRY_TEMPERATURE` | `0`                | Retry temperature for malformed-tool recovery |
| `PROXY_MALFORMED_TOOL_STREAM_STRICT` | `off`                 | For stream+tools requests, use guarded non-stream upstream path then replay SSE |
| `PROXY_SESSION_CONTAMINATION_BREAKER` | `on`                 | Resets long-running malformed sessions to recent context |
| `PROXY_SESSION_CONTAMINATION_THRESHOLD` | `3`                | Consecutive malformed turns before reset |
| `PROXY_SESSION_CONTAMINATION_KEEP_LAST` | `8`                 | Number of latest messages to preserve during contamination reset |
| `PROXY_AGENTIC_SUPPLEMENT_MODE` | `clean`                   | Agentic system supplement variant (`clean`, `legacy`) |

For agentic coding workloads, keep `PROXY_STREAM_REASONING_FALLBACK=off` (default) to avoid leaking malformed internal reasoning as user-visible output. Use `sanitized` only for debugging.

For Claude Code + Qwen malformed-tool loops, recommended starting profile:

```bash
PROXY_STREAM_REASONING_FALLBACK=off
PROXY_MAX_TOKENS_FLOOR=4096
PROXY_MALFORMED_TOOL_GUARDRAIL=on
PROXY_TOOL_NARROWING=on
PROXY_DISABLE_THINKING_ON_TOOL_TURNS=on
PROXY_SESSION_CONTAMINATION_BREAKER=on
PROXY_AGENTIC_SUPPLEMENT_MODE=clean
```

### Example: Custom upstream

```bash
LLAMA_CPP_BASE=http://localhost:8080/v1 PROXY_PORT=5000 python tools/agents/scripts/anthropic_proxy.py
```

### Claude Code Configuration

Point Claude Code at the proxy by setting the API base URL:

```bash
export ANTHROPIC_BASE_URL=http://localhost:4000
```

### Endpoints

The proxy speaks **Anthropic Messages API as its canonical interface** and
keeps an **OpenAI Chat Completions passthrough** for clients that require the
OpenAI shape. Both paths run through the same guarded pipeline (loop
detection, tool narrowing, malformed-payload retry, context pruning, etc.) —
the OpenAI route converts the request to Anthropic, runs the pipeline, and
re-shapes the final response back to OpenAI.

| Path                     | Method | Shape     | Description                                                     |
| ------------------------ | ------ | --------- | --------------------------------------------------------------- |
| `/v1/messages`           | POST   | Anthropic | Anthropic Messages API — default/canonical (streaming + sync)   |
| `/anthropic/v1/messages` | POST   | Anthropic | Alias for `/v1/messages` (some Claude Code configs use this)    |
| `/v1/chat/completions`   | POST   | OpenAI    | OpenAI Chat Completions passthrough (e.g. Forge, OpenCode)      |
| `/v1/models`             | GET    | Anthropic | Lists spoofed Anthropic model IDs                               |
| `/health`                | GET    | —         | Health check (verifies upstream reachability)                   |
| `/v1/context`            | GET    | —         | Current session context usage and pruning state                 |

### Running as a Service (systemd)

```ini
[Unit]
Description=UAP Anthropic Proxy
After=network.target

[Service]
Type=simple
User=cogtek
Environment=LLAMA_CPP_BASE=http://192.168.1.165:8080/v1
Environment=PROXY_PORT=4000
ExecStart=/usr/bin/python3 /path/to/tools/agents/scripts/anthropic_proxy.py
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
```

## Tool Call Format

The model emits tool calls in the official Qwen3 format:

```
<tool_call>
{"name": "read_file", "arguments": {"path": "/etc/hosts"}}
</tool_call>
```

Multiple tool calls in a single turn:

```
<tool_call>
{"name": "read_file", "arguments": {"path": "/etc/hosts"}}
</tool_call>
<tool_call>
{"name": "list_dir", "arguments": {"path": "/tmp"}}
</tool_call>
```

llama.cpp's autoparser handles stop behavior structurally via PEG grammar rules, not stop sequences. No explicit `</tool_call>` stop sequence is needed at the server level.

## LoRA Training Pipeline

### 1. Generate Training Data

```bash
python3 tools/agents/scripts/generate_lora_training_data.py -n 500
```

Produces `tool_call_training_data.jsonl` with ChatML-formatted examples using the official `<tool_call>` format.

### 2. Fine-Tune

Using axolotl:

```bash
accelerate launch -m axolotl.cli.train config/lora-finetune.yaml
```

Using unsloth (faster, less VRAM):

```bash
unsloth train --config config/lora-finetune.yaml
```

Training config highlights (`config/lora-finetune.yaml`):

- LoRA rank 16, alpha 32
- Targets all linear layers (q/k/v/o/gate/up/down projections)
- 3 epochs, cosine LR schedule, 2e-4 learning rate
- BF16 + gradient checkpointing + flash attention

### 3. Convert to GGUF

```bash
python3 convert_lora_to_gguf.py \
  --base Qwen/Qwen3.5-35B-A3B \
  --lora output/qwen35-tool-call-lora \
  --output adapter.gguf
```

### 4. Load at Runtime

```bash
llama-server --model base.gguf --lora adapter.gguf --lora-scale 1.0
```

## Quantization Options

| Quant  | VRAM  | Accuracy | Tool Call Reliability |
| ------ | ----- | -------- | --------------------- |
| IQ4_XS | 17 GB | 96%      | 94%                   |
| Q4_K_M | 20 GB | 95%      | 95%                   |
| Q5_K_M | 24 GB | 97%      | 97%                   |
| Q6_K   | 28 GB | 98%      | 98%                   |

## Troubleshooting

### "Template supports tool calls but does not natively describe tools"

This warning means llama.cpp detected `tool_calls` handling but no `tools` variable access in the template. The `chat_template.jinja` in this repo resolves this by including a `{%- if tools %}` block that renders tool descriptions in `<tools></tools>` XML tags.

Verify the template is loaded:

```bash
llama-server --chat-template-file chat_template.jinja --verbose
```

### LoRA not taking effect

- Ensure the adapter was converted to GGUF format (not safetensors/PyTorch)
- Check `--lora-scale` is not `0.0`
- Verify the adapter was trained against the same base model architecture

### Grammar rejecting valid output

If using the GBNF grammar via per-request `grammar` field and it's too restrictive, the model may produce truncated output. Check `tools/agents/config/tool-call.gbnf` allows the argument types your tools use (strings, numbers, objects, arrays, booleans, null are all supported).

### Model only outputs tool calls, never plain text

You are likely using `--grammar-file` on the server command line. This forces ALL output into `<tool_call>` format. Remove `--grammar-file` from the startup command and let the autoparser handle tool call detection lazily.

### Multi-tool calls truncated to single call

Two possible causes:

1. `--grammar-file` is set globally and the stop sequence `</tool_call>` terminates after the first call. Remove `--grammar-file`.
2. The client is not passing `parallel_tool_calls: true` in the request. Add it to enable multiple tool calls per turn.

## Related Files

- `tools/agents/scripts/qwen_tool_call_wrapper.py` - Python wrapper with retry logic and format validation
- `tools/agents/scripts/fix_qwen_chat_template.py` - Template verifier/fixer (detects format, validates Jinja2)
- `tools/agents/scripts/qwen_tool_call_test.py` - Test suite using OpenAI-compatible API
- `src/cli/tool-calls.ts` - CLI command for template management
- `src/bin/llama-server-optimize.ts` - llama-server startup optimizer
- `docs/deployment/UAP_LLAMA_ANTHROPIC_PROXY_BOOTSTRAP.md` - service bootstrap + ngram-cache A/B benchmarking
