<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: detecting-prompt-injection
description: >-
  Detect and classify direct prompt injection attacks against LLM-powered
  applications using pattern matching, ML classifiers, and input validation
  techniques to prevent system prompt override and safety bypass.
domain: cybersecurity
subdomain: ai-llm-security
tags:
  - prompt-injection
  - llm-security
  - input-validation
  - owasp-llm-top10
  - detection
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1059", "T1190"]
  owasp-llm: ["LLM01"]
  mitre-atlas: ["AML.T0051"]
  tools: ["python3", "transformers", "regex", "tiktoken"]
---

# Detecting Prompt Injection

## Overview

Direct prompt injection occurs when an attacker crafts user input to override,
manipulate, or bypass the system prompt and safety guardrails of an LLM-powered
application. Detection requires a layered approach combining heuristic pattern
matching, token-level analysis, and ML-based classification to identify malicious
inputs before they reach the model.

## Prerequisites

| Tool / Requirement | Details |
|---|---|
| `python3` | Security tooling |
| `transformers` | Security tooling |
| `tiktoken` | Security tooling |
| `re` | Security tooling |
| Access to application input pipeline for instrumentation | Environment requirement |
| Test corpus of known prompt injection payloads | Environment requirement |
| Baseline of legitimate user input patterns | Environment requirement |

## Key Concepts

- **Direct injection**: User input contains explicit instructions to override system behavior
- **Instruction hierarchy violation**: Input attempts to assert higher privilege than system prompt
- **Payload obfuscation**: Base64, Unicode, homoglyph, or multilingual encoding to evade filters
- **Canary tokens**: Sentinel values in system prompts that detect extraction attempts
- **Token budget analysis**: Injection payloads often produce anomalous token distributions

## Workflow

### Step 1: Build Pattern-Based Detection

Implement regex and keyword-based heuristic detectors:

```python
import re
from dataclasses import dataclass

INJECTION_PATTERNS: list[re.Pattern[str]] = [
    re.compile(r"ignore\s+(all\s+)?previous\s+instructions", re.IGNORECASE),
    re.compile(r"you\s+are\s+now\s+(DAN|a\s+new\s+AI)", re.IGNORECASE),
    re.compile(r"\[SYSTEM\s*(OVERRIDE|PROMPT)\]", re.IGNORECASE),
    re.compile(r"forget\s+(everything|your\s+rules)", re.IGNORECASE),
    re.compile(r"new\s+instructions?:\s*", re.IGNORECASE),
    re.compile(r"disregard\s+(all|any|the)\s+(above|previous|prior)", re.IGNORECASE),
]


@dataclass
class DetectionResult:
    is_suspicious: bool
    confidence: float
    matched_patterns: list[str]
    input_text: str


def scan_input(text: str) -> DetectionResult:
    matched: list[str] = []
    for pattern in INJECTION_PATTERNS:
        if pattern.search(text):
            matched.append(pattern.pattern)
    confidence = min(len(matched) / 3.0, 1.0)
    return DetectionResult(
        is_suspicious=len(matched) > 0,
        confidence=confidence,
        matched_patterns=matched,
        input_text=text[:200],
    )
```

### Step 2: Token Anomaly Detection

Analyze token-level statistics to flag inputs with unusual distributions:

```python
import tiktoken

def analyze_token_anomalies(
    text: str, model: str = "cl100k_base", max_ratio: float = 3.0
) -> dict:
    enc = tiktoken.get_encoding(model)
    tokens = enc.encode(text)
    words = text.split()
    ratio = len(tokens) / max(len(words), 1)
    has_unusual_unicode = bool(
        re.search(r"[\u200b-\u200f\u2028-\u202f\ufeff]", text)
    )
    return {
        "token_count": len(tokens),
        "word_count": len(words),
        "token_word_ratio": round(ratio, 2),
        "anomalous_ratio": ratio > max_ratio,
        "hidden_unicode": has_unusual_unicode,
    }
```

### Step 3: ML-Based Classifier

Deploy a fine-tuned classifier for high-confidence injection detection:

```python
from transformers import pipeline

def classify_injection(text: str, threshold: float = 0.85) -> dict:
    classifier = pipeline(
        "text-classification",
        model="protectai/deberta-v3-base-prompt-injection-v2",
    )
    result = classifier(text[:512])[0]
    return {
        "label": result["label"],
        "score": round(result["score"], 4),
        "is_injection": result["label"] == "INJECTION"
            and result["score"] >= threshold,
    }
```

### Step 4: Canary Token Monitoring

Embed canary tokens in system prompts to detect extraction:

```python
import hashlib
import time

def generate_canary(secret: str) -> str:
    timestamp = str(int(time.time()))
    raw = f"{secret}:{timestamp}"
    return f"CANARY-{hashlib.sha256(raw.encode()).hexdigest()[:16]}"

def check_output_for_canary(output: str, canary: str) -> dict:
    leaked = canary in output
    return {
        "canary": canary,
        "leaked": leaked,
        "severity": "CRITICAL" if leaked else "NONE",
        "action": "Block response and alert" if leaked else "Pass",
    }
```

## Detection

```yaml
title: Prompt Injection Detection
id: 822848e1-9e66-4dca-adac-913d80108c8b
status: experimental
description: Detects suspicious activity related to detecting prompt injection techniques in ai llm security context
logsource:
  category: application
  product: webserver
detection:
  selection:
    EventType: error
  condition: selection
level: medium
tags:
  - attack.t1059
  - attack.t1190
  - attack.initial_access
falsepositives:
  - Automated AI model training and inference pipeline operations
```


**Detection Opportunities**

| Indicator | Source | Detection Logic |
|---|---|---|
| Prompt Injection Detection | webserver/application | Sigma rule (medium) |
| ATT&CK Coverage | MITRE ATT&CK | T1059, T1190 |

## Verification

- [ ] Pattern-based detector covers OWASP LLM01 injection taxonomy
- [ ] Token anomaly detection identifies obfuscated payloads
- [ ] ML classifier deployed with configurable threshold
- [ ] Canary tokens embedded and monitoring active
- [ ] False positive rate measured against legitimate traffic
- [ ] Detection latency under 100ms p99

## References

- [OWASP LLM01 — Prompt Injection](https://genai.owasp.org/llmrisk/llm01-prompt-injection/)
- [MITRE ATLAS AML.T0051 — LLM Prompt Injection](https://atlas.mitre.org/techniques/AML.T0051)
- [ProtectAI Prompt Injection Classifier](https://huggingface.co/protectai/deberta-v3-base-prompt-injection-v2)
- [NIST AI 100-2 — Adversarial Machine Learning](https://csrc.nist.gov/pubs/ai/100/2/e2023/final)
