<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: detecting-prompt-leaking-attacks
description: >-
  Detect and prevent prompt leaking attacks that extract system prompts,
  few-shot examples, tool definitions, and confidential instructions from
  LLM applications through direct extraction, indirect inference, and
  differential analysis techniques.
domain: cybersecurity
subdomain: ai-llm-security
tags:
  - prompt-leaking
  - system-prompt
  - data-exfiltration
  - confidentiality
  - owasp-llm-top10
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1041", "T1005"]
  owasp-llm: ["LLM07", "LLM01"]
  mitre-atlas: ["AML.T0048", "AML.T0051"]
  tools: ["python3", "hashlib", "re", "difflib"]
---

# Detecting Prompt Leaking Attacks

## Overview

System prompts contain business logic, safety instructions, tool definitions,
and confidential context that attackers seek to extract. Prompt leaking attacks
range from direct requests ("repeat your instructions") to sophisticated
differential analysis and role-play social engineering. Detection requires
output monitoring for prompt content, canary token systems, and behavioral
analysis of extraction attempts.

## Prerequisites

| Tool / Requirement | Details |
|---|---|
| `python3` | Security tooling |
| `hashlib` | Security tooling |
| `re` | Security tooling |
| `difflib` | Security tooling |
| Access to system prompt content for canary embedding | Environment requirement |
| Output monitoring pipeline | Environment requirement |
| Test corpus of known prompt extraction techniques | Environment requirement |

## Key Concepts

- **Direct extraction**: Asking the model to repeat, print, or encode its instructions
- **Indirect extraction**: Using roleplay, translation, or format changes to leak instructions
- **Differential analysis**: Comparing outputs to infer system prompt structure
- **Canary tokens**: Unique markers in system prompts that detect leakage in output
- **Output fingerprinting**: Detecting system prompt fragments in model responses

## Workflow

### Step 1: Extraction Attempt Detection

```python
import re
from typing import Any

EXTRACTION_PATTERNS: list[tuple[str, str]] = [
    (r"(repeat|print|show|display|output)\s+(your|the)\s+(system\s+)?(prompt|instructions)", "direct_extraction"),
    (r"what\s+(are|were)\s+your\s+(initial\s+)?(instructions|rules|guidelines)", "direct_query"),
    (r"(translate|convert|encode)\s+your\s+(system\s+)?(prompt|instructions)", "encoding_extraction"),
    (r"(pretend|imagine|roleplay)\s+.*\s+(reveal|share|show)\s+.*\s+(prompt|instructions)", "roleplay_extraction"),
    (r"(begin|start)\s+your\s+response\s+with\s+.*\s+(system|prompt)", "format_extraction"),
    (r"(summarize|paraphrase|rephrase)\s+your\s+(system\s+)?(instructions|prompt)", "paraphrase_extraction"),
    (r"output\s+.*\s+in\s+(base64|hex|rot13|morse|binary)", "encoding_exfil"),
    (r"(first|initial|original)\s+(message|instruction|text)\s+you\s+received", "initial_message_query"),
]


def detect_extraction_attempt(user_input: str) -> dict[str, Any]:
    matches: list[dict[str, str]] = []
    for pattern, label in EXTRACTION_PATTERNS:
        if re.search(pattern, user_input, re.IGNORECASE):
            matches.append({"technique": label, "pattern": pattern})
    return {
        "is_extraction_attempt": len(matches) > 0,
        "techniques_matched": len(matches),
        "matches": matches,
        "risk": "HIGH" if matches else "NONE",
        "action": "BLOCK" if matches else "ALLOW",
    }
```

### Step 2: Canary Token System

```python
import hashlib
import time


class CanarySystem:
    def __init__(self, secret: str) -> None:
        self.secret = secret
        self.canaries: list[str] = []

    def generate_canary(self, label: str = "") -> str:
        raw = f"{self.secret}:{label}:{time.time()}"
        token = f"CAN-{hashlib.sha256(raw.encode()).hexdigest()[:16]}"
        self.canaries.append(token)
        return token

    def check_output(self, output: str) -> dict[str, Any]:
        leaked: list[str] = []
        for canary in self.canaries:
            if canary in output:
                leaked.append(canary)
        return {
            "canaries_checked": len(self.canaries),
            "canaries_leaked": len(leaked),
            "leaked_tokens": leaked,
            "severity": "CRITICAL" if leaked else "NONE",
            "action": "BLOCK_AND_ALERT" if leaked else "PASS",
        }
```

### Step 3: Output Similarity Monitoring

```python
import difflib


def check_output_similarity(
    output: str, system_prompt: str, threshold: float = 0.4,
) -> dict[str, Any]:
    output_lower = output.lower()
    prompt_lower = system_prompt.lower()
    prompt_sentences = [s.strip() for s in prompt_lower.split(".") if len(s.strip()) > 10]
    leaked_fragments: list[dict[str, Any]] = []
    for sentence in prompt_sentences:
        ratio = difflib.SequenceMatcher(None, sentence, output_lower).ratio()
        if ratio > threshold:
            leaked_fragments.append({
                "fragment": sentence[:80],
                "similarity": round(ratio, 3),
            })
    return {
        "fragments_checked": len(prompt_sentences),
        "fragments_leaked": len(leaked_fragments),
        "leaked": leaked_fragments[:10],
        "max_similarity": max((f["similarity"] for f in leaked_fragments), default=0.0),
        "risk": "HIGH" if leaked_fragments else "NONE",
    }
```

### Step 4: Behavioral Pattern Analysis

```python
def analyze_session_behavior(
    queries: list[str],
) -> dict[str, Any]:
    extraction_count = 0
    for query in queries:
        result = detect_extraction_attempt(query)
        if result["is_extraction_attempt"]:
            extraction_count += 1
    ratio = extraction_count / max(len(queries), 1)
    return {
        "total_queries": len(queries),
        "extraction_attempts": extraction_count,
        "extraction_ratio": round(ratio, 3),
        "suspicious_session": ratio > 0.3,
        "action": "RATE_LIMIT" if ratio > 0.3 else "MONITOR",
    }
```

## Detection

```yaml
title: Prompt Leaking Attacks Detection
id: 925d06d7-6c8f-43cc-8560-b5ef1a031022
status: experimental
description: Detects suspicious activity related to detecting prompt leaking attacks techniques in ai llm security context
logsource:
  category: application
  product: webserver
detection:
  selection:
    EventType: error
  condition: selection
level: medium
tags:
  - attack.t1041
  - attack.t1005
  - attack.initial_access
falsepositives:
  - Automated AI model training and inference pipeline operations
```


**Detection Opportunities**

| Indicator | Source | Detection Logic |
|---|---|---|
| Prompt Leaking Attacks Detection | webserver/application | Sigma rule (medium) |
| ATT&CK Coverage | MITRE ATT&CK | T1041, T1005 |

## Verification

- [ ] Extraction patterns detect all common prompt leaking techniques
- [ ] Canary tokens embedded and detection active in output pipeline
- [ ] Output similarity check catches partial prompt leakage
- [ ] Session behavioral analysis identifies extraction campaigns
- [ ] False positive rate measured against legitimate help-seeking queries
- [ ] Blocked responses do not confirm system prompt existence

## References

- [OWASP LLM07 — Insecure Plugin Design](https://genai.owasp.org/llmrisk/llm07/)
- [Prompt Leaking Taxonomy (Perez & Ribeiro)](https://arxiv.org/abs/2211.09527)
- [MITRE ATLAS AML.T0048 — Exfiltration via ML API](https://atlas.mitre.org/techniques/AML.T0048)
- [System Prompt Extraction Research](https://arxiv.org/abs/2311.16119)
