<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: detecting-training-data-poisoning
description: >-
  Detect and prevent training data poisoning attacks including backdoor injection,
  label flipping, data corruption, and adversarial samples that compromise model
  integrity during fine-tuning or training.
domain: cybersecurity
subdomain: ai-llm-security
tags:
  - data-poisoning
  - backdoor
  - model-integrity
  - training-security
  - supply-chain
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1195.002", "T1565"]
  owasp-llm: ["LLM03"]
  mitre-atlas: ["AML.T0020", "AML.T0043"]
  tools: ["python3", "numpy", "scikit-learn", "hashlib"]
---

# Detecting Training Data Poisoning

## Overview

Training data poisoning compromises model integrity by manipulating data used during
training or fine-tuning. Attacks include backdoor injection, label flipping, data
corruption, and supply chain poisoning. Detection requires data provenance
verification, statistical anomaly detection, and behavioral testing.

## Prerequisites

| Tool / Requirement | Details |
|---|---|
| `python3` | Security tooling |
| `numpy` | Security tooling |
| `scikit-learn` | Security tooling |
| `hashlib` | Security tooling |
| Access to training data pipeline and data sources | Environment requirement |
| Baseline model performance metrics | Environment requirement |
| Understanding of the model's training process | Environment requirement |

## Key Concepts

- **Backdoor injection**: Planting trigger patterns that activate specific model behaviors
- **Label flipping**: Changing correct labels to corrupt or bias model performance
- **Data provenance**: Tracking origin and integrity of every training data sample
- **Clean-label attack**: Poisoning without changing labels using adversarial perturbations
- **Model behavior testing**: Probing for hidden backdoors through input variation

## Workflow

### Step 1: Data Integrity Verification

```python
import hashlib
import json
from pathlib import Path

def verify_dataset_integrity(dataset_path: str, manifest_path: str) -> dict:
    with open(manifest_path) as f:
        manifest = json.load(f)
    known_files = {e["path"]: e["sha256"] for e in manifest["files"]}
    results = {"verified": 0, "modified": 0, "missing": 0, "new": 0}
    actual_files = set()
    for fpath in Path(dataset_path).rglob("*"):
        if fpath.is_dir():
            continue
        rel = str(fpath.relative_to(dataset_path))
        actual_files.add(rel)
        sha = hashlib.sha256(fpath.read_bytes()).hexdigest()
        if rel not in known_files:
            results["new"] += 1
        elif sha != known_files[rel]:
            results["modified"] += 1
        else:
            results["verified"] += 1
    for known in known_files:
        if known not in actual_files:
            results["missing"] += 1
    results["integrity_ok"] = results["modified"] == 0 and results["missing"] == 0
    return results
```

### Step 2: Statistical Anomaly Detection

```python
from collections import Counter

def detect_label_anomalies(
    labels: list[str], expected_distribution: dict[str, float] | None = None,
    threshold: float = 0.1,
) -> dict:
    counts = Counter(labels)
    total = len(labels)
    actual_dist = {k: v / total for k, v in counts.items()}
    anomalies = []
    if expected_distribution:
        for label, expected_pct in expected_distribution.items():
            actual_pct = actual_dist.get(label, 0.0)
            if abs(actual_pct - expected_pct) > threshold:
                anomalies.append({
                    "label": label,
                    "expected": round(expected_pct, 4),
                    "actual": round(actual_pct, 4),
                })
    return {"label_distribution": actual_dist, "anomalies": anomalies, "suspicious": len(anomalies) > 0}
```

### Step 3: Backdoor Trigger Testing

```python
def test_backdoor_triggers(model_fn, test_inputs: list[str], triggers: list[str]) -> dict:
    results = []
    for base_input in test_inputs:
        base_output = model_fn(base_input)
        for trigger in triggers:
            triggered_input = f"{trigger} {base_input}"
            triggered_output = model_fn(triggered_input)
            results.append({
                "trigger": trigger,
                "output_changed": base_output != triggered_output,
                "base_output": str(base_output)[:100],
                "triggered_output": str(triggered_output)[:100],
            })
    changed = sum(1 for r in results if r["output_changed"])
    return {"test": "backdoor_trigger_scan", "outputs_changed": changed, "results": results[:20]}
```

## Detection

```yaml
title: Training Data Poisoning Detection
id: e02a5d89-66e1-43ce-af74-18ba817ee29f
status: experimental
description: Detects suspicious activity related to detecting training data poisoning techniques in ai llm security context
logsource:
  category: application
  product: webserver
detection:
  selection:
    EventType: error
  condition: selection
level: medium
tags:
  - attack.t1195.002
  - attack.t1565
  - attack.initial_access
falsepositives:
  - Automated AI model training and inference pipeline operations
```


**Detection Opportunities**

| Indicator | Source | Detection Logic |
|---|---|---|
| Training Data Poisoning Detection | webserver/application | Sigma rule (medium) |
| ATT&CK Coverage | MITRE ATT&CK | T1195.002, T1565 |

## Verification

- [ ] Dataset integrity verified against known-good manifest
- [ ] Label distribution anomalies checked
- [ ] Duplicate and near-duplicate samples analyzed
- [ ] Backdoor trigger candidates tested against model
- [ ] Data provenance chain documented
- [ ] Findings mapped to OWASP LLM03 and MITRE ATLAS

## References

- [OWASP LLM03 — Training Data Poisoning](https://genai.owasp.org/llmrisk/llm03-training-data-poisoning/)
- [MITRE ATLAS AML.T0020 — Poison Training Data](https://atlas.mitre.org/techniques/AML.T0020)
- [Poisoning Language Models During Instruction Tuning — Wan et al.](https://arxiv.org/abs/2305.00944)
- [NIST AI 100-2 — Data Poisoning Taxonomy](https://csrc.nist.gov/pubs/ai/100/2/e2023/final)
