<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: detecting-pii-in-data-flows
description: >-
  Detect and classify Personally Identifiable Information (PII) across data
  pipelines, APIs, logs, and storage using pattern matching, NLP-based entity
  recognition, and data flow analysis.
domain: cybersecurity
subdomain: privacy-engineering
tags:
  - pii
  - data-classification
  - data-flows
  - gdpr
  - hipaa
  - dlp
  - ner
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1005", "T1039"]
---

# Detecting PII in Data Flows

## Overview

PII detection identifies personal data as it moves through pipelines, APIs, logs,
databases, and message queues. Undetected PII in logs, analytics, or third-party
transfers creates regulatory exposure (GDPR Art. 5, HIPAA §164.502). This skill
covers building PII detection using regex patterns, Named Entity Recognition (NER),
and data flow scanning with structured classification output.

Mode: `[MODE: PRIVACY]` — Data discovery and classification.

## Prerequisites

- Python 3.10+ with `re` module (regex patterns)
- Optional: spaCy or Presidio for NLP-based entity recognition
- Access to data sources: log files, API responses, database exports, message queues
- Data flow diagram identifying all processing stages

## Key Concepts

### PII Classification Taxonomy

| Category | Examples | Sensitivity |
|----------|----------|-------------|
| Direct Identifiers | SSN, passport, driver's license | Critical |
| Contact | Email, phone, address | High |
| Financial | Credit card, bank account, IBAN | Critical |
| Health | Diagnosis, prescription, insurance ID | Critical (HIPAA) |
| Biometric | Fingerprint, facial geometry, voice | Critical (GDPR Art. 9) |
| Online Identifiers | IP address, cookie ID, device ID | Medium-High |
| Demographic | DOB, gender, ethnicity | Medium |

### Detection Patterns

```python
PII_PATTERNS = {
    "ssn": r"\b\d{3}-\d{2}-\d{4}\b",
    "email": r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b",
    "phone_us": r"\b(?:\+1[-.]?)?\(?\d{3}\)?[-.\s]?\d{3}[-.\s]?\d{4}\b",
    "credit_card": r"\b(?:4\d{3}|5[1-5]\d{2}|3[47]\d{2}|6(?:011|5\d{2}))\d{8,12}\b",
    "ip_v4": r"\b(?:\d{1,3}\.){3}\d{1,3}\b",
    "date_of_birth": r"\b(?:0[1-9]|1[0-2])/(?:0[1-9]|[12]\d|3[01])/(?:19|20)\d{2}\b",
    "iban": r"\b[A-Z]{2}\d{2}[A-Z0-9]{4}\d{7}(?:[A-Z0-9]?){0,16}\b",
    "passport_us": r"\b[A-Z]\d{8}\b",
}
```

### Scanning CLI

```bash
# Scan a file for PII
node scripts/agent.js --action scan --input /var/log/app.log \
  --output /tmp/pii_findings.json

# Scan with specific pattern categories
node scripts/agent.js --action scan --input data_export.csv \
  --categories "ssn,email,credit_card" --output /tmp/pii_report.json

# Scan a directory recursively
node scripts/agent.js --action scan-dir --input /var/log/ \
  --extensions ".log,.csv,.json" --output /tmp/pii_scan.json

# Classify detected PII by sensitivity
node scripts/agent.js --action classify \
  --findings /tmp/pii_findings.json --output /tmp/classified.json
```

## Workflow

### Step 1: Map Data Flows

Identify all locations where personal data may exist:
log files, databases, API payloads, message queues, caches, backups.

### Step 2: Scan for PII

```bash
node scripts/agent.js --action scan --input /var/log/app.log \
  --output /tmp/pii_findings.json
```

### Step 3: Classify Findings

```bash
node scripts/agent.js --action classify \
  --findings /tmp/pii_findings.json
```

### Step 4: Remediate

Apply masking, tokenization, or removal based on classification:
- **Critical PII in logs**: Implement log scrubbing pipeline
- **PII in API responses**: Add response filtering middleware
- **PII in analytics**: Apply anonymization before collection

## Verification

- [ ] All data flow stages scanned for PII
- [ ] Detection patterns cover all relevant PII categories
- [ ] False positive rate assessed and tuning applied
- [ ] Critical PII findings escalated for remediation
- [ ] Log scrubbing implemented for detected PII in logs
- [ ] API response filtering applied where PII detected
- [ ] Recurring scan schedule configured

## References

- [GDPR Article 4 — Definition of Personal Data](https://gdpr-info.eu/art-4-gdpr/)
- [NIST SP 800-122 — Guide to Protecting PII](https://csrc.nist.gov/publications/detail/sp/800-122/final)
- [Microsoft Presidio — PII Detection](https://microsoft.github.io/presidio/)
- [HIPAA Safe Harbor De-Identification](https://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/)
- [GCP DLP InfoTypes Reference](https://cloud.google.com/dlp/docs/infotypes-reference)
