<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: detecting-model-hallucination-exploitation
description: >-
  Detect and prevent exploitation of LLM hallucinations including fabricated
  citations, invented API endpoints, fake package names, and confabulated
  facts that attackers weaponize for social engineering, supply chain attacks,
  and misinformation campaigns.
domain: cybersecurity
subdomain: ai-llm-security
tags:
  - hallucination
  - fact-checking
  - supply-chain
  - misinformation
  - groundedness
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1195", "T1566"]
  owasp-llm: ["LLM09"]
  mitre-atlas: ["AML.T0048"]
  tools: ["python3", "requests", "json", "difflib"]
---

# Detecting Model Hallucination Exploitation

## Overview

LLMs hallucinate — they generate plausible but fabricated outputs including
nonexistent packages, fake CVEs, invented API endpoints, and spurious
citations. Attackers exploit this by registering hallucinated package names
(slopsquatting), creating domains matching fabricated URLs, or leveraging
confabulated authority claims. Detection requires fact-checking pipelines,
groundedness scoring, and supply chain verification.

## Prerequisites

| Tool / Requirement | Details |
|---|---|
| `python3` | Security tooling |
| `requests` | Security tooling |
| `difflib` | Security tooling |
| `json` | Security tooling |
| Access to package registries (PyPI, npm) for supply chain checks | Environment requirement |
| Reference corpus for fact verification | Environment requirement |
| Groundedness evaluation framework | Environment requirement |

## Key Concepts

- **Slopsquatting**: Registering package names that LLMs hallucinate in code suggestions
- **Citation fabrication**: Invented references, DOIs, or URLs that appear authoritative
- **Confabulated endpoints**: Fake API paths that could be weaponized if registered
- **Groundedness score**: Measure of how well output is supported by source material
- **Fact anchoring**: Requiring outputs to cite verifiable sources

## Workflow

### Step 1: Package Hallucination Detection

Verify that recommended packages actually exist:

```python
import requests
from typing import Any


def check_pypi_package(package_name: str) -> dict[str, Any]:
    url = f"https://pypi.org/pypi/{package_name}/json"
    try:
        resp = requests.get(url, timeout=10)
        if resp.status_code == 200:
            data = resp.json()
            return {
                "package": package_name,
                "exists": True,
                "version": data["info"]["version"],
                "author": data["info"]["author"],
            }
        return {"package": package_name, "exists": False, "risk": "slopsquatting"}
    except requests.RequestException as e:
        return {"package": package_name, "error": str(e)}


def check_npm_package(package_name: str) -> dict[str, Any]:
    url = f"https://registry.npmjs.org/{package_name}"
    try:
        resp = requests.get(url, timeout=10)
        if resp.status_code == 200:
            data = resp.json()
            return {
                "package": package_name,
                "exists": True,
                "version": data.get("dist-tags", {}).get("latest", "unknown"),
            }
        return {"package": package_name, "exists": False, "risk": "slopsquatting"}
    except requests.RequestException as e:
        return {"package": package_name, "error": str(e)}
```

### Step 2: URL and Citation Verification

Check whether cited URLs and references actually exist:

```python
def verify_url(url: str) -> dict[str, Any]:
    try:
        resp = requests.head(url, timeout=10, allow_redirects=True)
        return {
            "url": url,
            "reachable": resp.status_code < 400,
            "status_code": resp.status_code,
            "final_url": resp.url,
        }
    except requests.RequestException:
        return {"url": url, "reachable": False, "status_code": None}


def verify_cve(cve_id: str) -> dict[str, Any]:
    url = f"https://cveawg.mitre.org/api/cve/{cve_id}"
    try:
        resp = requests.get(url, timeout=10)
        if resp.status_code == 200:
            data = resp.json()
            return {
                "cve": cve_id,
                "exists": True,
                "state": data.get("cveMetadata", {}).get("state", "unknown"),
            }
        return {"cve": cve_id, "exists": False, "risk": "fabricated_cve"}
    except requests.RequestException as e:
        return {"cve": cve_id, "error": str(e)}
```

### Step 3: Groundedness Scoring

Measure how well LLM output is grounded in provided context:

```python
import difflib


def compute_groundedness(
    output: str, reference: str, threshold: float = 0.3,
) -> dict[str, Any]:
    output_sentences = [s.strip() for s in output.split(".") if s.strip()]
    grounded_count = 0
    ungrounded: list[str] = []
    for sentence in output_sentences:
        ratio = difflib.SequenceMatcher(None, sentence.lower(), reference.lower()).ratio()
        if ratio >= threshold:
            grounded_count += 1
        else:
            ungrounded.append(sentence)
    total = max(len(output_sentences), 1)
    score = grounded_count / total
    return {
        "groundedness_score": round(score, 3),
        "total_claims": total,
        "grounded": grounded_count,
        "ungrounded_count": len(ungrounded),
        "ungrounded_samples": ungrounded[:5],
        "risk": "HIGH" if score < 0.5 else "LOW",
    }
```

## Detection

```yaml
title: Model Hallucination Exploitation Detection
id: 3434d67d-c583-4135-be7c-c59869342f8b
status: experimental
description: Detects suspicious activity related to detecting model hallucination exploitation techniques in ai llm security context
logsource:
  category: application
  product: webserver
detection:
  selection:
    EventType: error
  condition: selection
level: medium
tags:
  - attack.t1195
  - attack.t1566
  - attack.initial_access
falsepositives:
  - Automated AI model training and inference pipeline operations
```


**Detection Opportunities**

| Indicator | Source | Detection Logic |
|---|---|---|
| Model Hallucination Exploitation Detection | webserver/application | Sigma rule (medium) |
| ATT&CK Coverage | MITRE ATT&CK | T1195, T1566 |

## Verification

- [ ] Package verification catches known hallucinated package names
- [ ] URL checker identifies fabricated citations
- [ ] CVE verification flags non-existent CVE IDs
- [ ] Groundedness scorer differentiates grounded vs hallucinated outputs
- [ ] Pipeline latency acceptable for inline checking

## References

- [OWASP LLM09 — Misinformation](https://genai.owasp.org/llmrisk/llm09-misinformation/)
- [Slopsquatting — AI Package Hallucination Attacks](https://blog.socket.dev/slopsquatting-how-ai-hallucinations-are-fueling-a-new-class-of-supply-chain-attacks)
- [NIST AI 100-2 — Adversarial Machine Learning](https://csrc.nist.gov/pubs/ai/100/2/e2023/final)
- [MITRE ATLAS — LLM Data Leakage](https://atlas.mitre.org/techniques/AML.T0048)
