<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: implementing-llm-input-sanitization
description: >-
  Implement robust input sanitization for LLM applications including
  encoding normalization, token budget enforcement, structured delimiter
  protection, and multi-layer filtering to prevent injection, exfiltration,
  and denial-of-service through crafted inputs.
domain: cybersecurity
subdomain: ai-llm-security
tags:
  - input-sanitization
  - encoding-normalization
  - token-limits
  - llm-firewall
  - owasp-llm-top10
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1059", "T1027"]
  owasp-llm: ["LLM01", "LLM04"]
  mitre-atlas: ["AML.T0051"]
  tools: ["python3", "tiktoken", "unicodedata", "re"]
---

# Implementing LLM Input Sanitization

## Overview

LLM input sanitization operates at a different layer than traditional web
input validation. Attacks exploit Unicode normalization gaps, invisible
characters, token-level obfuscation, and structured delimiter confusion.
Effective sanitization must normalize encodings, enforce token budgets,
strip hidden characters, and protect system/user message boundaries —
all without breaking legitimate multilingual input.

## Prerequisites

| Tool / Requirement | Details |
|---|---|
| `python3` | Security tooling |
| `tiktoken` | Security tooling |
| `unicodedata` | Security tooling |
| `re` | Security tooling |
| Understanding of target model's tokenizer behavior | Environment requirement |
| Baseline of legitimate input character distributions | Environment requirement |
| Test corpus of Unicode-based evasion payloads | Environment requirement |

## Key Concepts

- **Unicode normalization**: NFC/NFKC normalization to collapse homoglyphs and confusables
- **Invisible character stripping**: Remove zero-width spaces, joiners, and bidirectional overrides
- **Token budget enforcement**: Hard limits on input token count to prevent DoS
- **Delimiter protection**: Prevent user input from escaping message boundaries
- **Encoding detection**: Identify Base64, hex, and URL-encoded payloads in input

## Workflow

### Step 1: Unicode Normalization and Invisible Character Removal

```python
import re
import unicodedata
from typing import Any

INVISIBLE_CHARS = re.compile(
    r"[\u200b-\u200f\u2028-\u202f\u2060-\u2069\ufeff\u00ad"
    r"\u034f\u061c\u115f\u1160\u17b4\u17b5\u180e]"
)

BIDI_OVERRIDES = re.compile(r"[\u202a-\u202e\u2066-\u2069]")


def sanitize_unicode(text: str) -> dict[str, Any]:
    original_len = len(text)
    normalized = unicodedata.normalize("NFKC", text)
    invisible_found = len(INVISIBLE_CHARS.findall(normalized))
    bidi_found = len(BIDI_OVERRIDES.findall(normalized))
    cleaned = INVISIBLE_CHARS.sub("", normalized)
    cleaned = BIDI_OVERRIDES.sub("", cleaned)
    cleaned = re.sub(r"\s+", " ", cleaned).strip()
    return {
        "original_length": original_len,
        "cleaned_length": len(cleaned),
        "invisible_chars_removed": invisible_found,
        "bidi_overrides_removed": bidi_found,
        "normalized": cleaned,
    }
```

### Step 2: Token Budget Enforcement

```python
import tiktoken


def enforce_token_budget(
    text: str, max_tokens: int = 4096, model: str = "cl100k_base",
) -> dict[str, Any]:
    enc = tiktoken.get_encoding(model)
    tokens = enc.encode(text)
    exceeded = len(tokens) > max_tokens
    if exceeded:
        truncated_tokens = tokens[:max_tokens]
        text = enc.decode(truncated_tokens)
    return {
        "original_tokens": len(tokens),
        "max_tokens": max_tokens,
        "exceeded": exceeded,
        "final_tokens": min(len(tokens), max_tokens),
        "text": text,
    }
```

### Step 3: Encoding Detection and Neutralization

```python
import base64
import binascii


def detect_encoded_payloads(text: str) -> dict[str, Any]:
    findings: list[dict[str, str]] = []
    b64_pattern = re.compile(r"[A-Za-z0-9+/]{20,}={0,2}")
    for match in b64_pattern.finditer(text):
        candidate = match.group()
        try:
            decoded = base64.b64decode(candidate).decode("utf-8", errors="ignore")
            if any(c.isprintable() for c in decoded):
                findings.append({
                    "type": "base64",
                    "encoded": candidate[:50],
                    "decoded_preview": decoded[:100],
                })
        except (binascii.Error, ValueError):
            pass
    hex_pattern = re.compile(r"(?:0x)?[0-9a-fA-F]{20,}")
    for match in hex_pattern.finditer(text):
        findings.append({"type": "hex", "value": match.group()[:50]})
    url_encoded = re.compile(r"(?:%[0-9a-fA-F]{2}){5,}")
    for match in url_encoded.finditer(text):
        findings.append({"type": "url_encoded", "value": match.group()[:50]})
    return {
        "encoded_payloads_found": len(findings),
        "findings": findings,
        "risk": "HIGH" if findings else "NONE",
    }
```

### Step 4: Delimiter Injection Protection

```python
DELIMITER_PATTERNS = [
    re.compile(r"<\|?(system|assistant|user|im_start|im_end)\|?>", re.IGNORECASE),
    re.compile(r"\[INST\]|\[/INST\]", re.IGNORECASE),
    re.compile(r"###\s*(System|Human|Assistant)\s*:", re.IGNORECASE),
    re.compile(r"<\|begin_of_text\|>|<\|end_of_text\|>", re.IGNORECASE),
]


def sanitize_delimiters(text: str) -> dict[str, Any]:
    violations: list[str] = []
    cleaned = text
    for pattern in DELIMITER_PATTERNS:
        matches = pattern.findall(cleaned)
        if matches:
            violations.extend(matches)
            cleaned = pattern.sub("[FILTERED]", cleaned)
    return {
        "delimiter_violations": len(violations),
        "violations": violations,
        "sanitized": cleaned,
        "risk": "CRITICAL" if violations else "NONE",
    }
```

## Detection

```yaml
title: Llm Input Sanitization Detection
id: f0e8d9c1-d154-4ed5-a0f7-fe69f5ed91cc
status: experimental
description: Detects suspicious activity related to implementing llm input sanitization techniques in ai llm security context
logsource:
  category: application
  product: webserver
detection:
  selection:
    EventType: error
  condition: selection
level: medium
tags:
  - attack.t1059
  - attack.t1027
  - attack.initial_access
falsepositives:
  - Automated AI model training and inference pipeline operations
```


**Detection Opportunities**

| Indicator | Source | Detection Logic |
|---|---|---|
| Llm Input Sanitization Detection | webserver/application | Sigma rule (medium) |
| ATT&CK Coverage | MITRE ATT&CK | T1059, T1027 |

## Verification

- [ ] Unicode normalization collapses known homoglyph attacks
- [ ] Invisible character stripping removes all zero-width and bidi chars
- [ ] Token budget enforced with clean truncation
- [ ] Base64 and hex payloads detected in user input
- [ ] Delimiter injection patterns blocked across model formats
- [ ] Legitimate multilingual input passes without corruption

## References

- [OWASP LLM01 — Prompt Injection](https://genai.owasp.org/llmrisk/llm01-prompt-injection/)
- [Unicode Security Considerations — TR36](https://unicode.org/reports/tr36/)
- [NIST AI 100-2 — Adversarial Machine Learning](https://csrc.nist.gov/pubs/ai/100/2/e2023/final)
- [Token-Level Prompt Injection Research](https://arxiv.org/abs/2302.12173)
