<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
<!-- NERF is a trademark of defconxt. -->
---
name: implementing-security-data-retention-policies
description: >-
  Design and enforce security data retention policies that balance compliance
  requirements, forensic investigation needs, and storage cost optimization
  across heterogeneous log sources and storage tiers.
domain: cybersecurity
subdomain: security-data-engineering
tags:
  - data-retention
  - compliance
  - lifecycle-management
  - gdpr
  - storage-tiering
  - policy-enforcement
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1485", "T1565"]
  frameworks: ["NIST 800-92", "GDPR Art. 5", "PCI DSS 10.7"]
  tools: ["python3", "aws-cli", "elasticsearch", "curator"]
---

# Implementing Security Data Retention Policies

## Overview

Security data retention policies define how long log data is stored, when it
transitions between storage tiers, and when it is purged. Effective policies
balance regulatory mandates (PCI DSS 1 year, HIPAA 6 years) with forensic
value and storage costs.

## Prerequisites

| Requirement | Purpose |
|---|---|
| Regulatory mapping | Know applicable retention mandates |
| Storage tier inventory | Hot, warm, cold, archive storage costs |
| Data classification scheme | Categorize log sources by sensitivity |
| Lifecycle automation tools | Curator, S3 lifecycle, or equivalent |

## Key Concepts

- **Regulatory floor**: Minimum retention required by law or standard
- **Forensic ceiling**: Maximum useful retention for investigation
- **Tiered lifecycle**: Automatic migration from hot to cold storage
- **Immutability window**: Period where logs cannot be modified or deleted
- **Legal hold**: Override that suspends deletion for litigation
- **Purge verification**: Confirm data is cryptographically destroyed

## Workflow

### Step 1: Map Regulatory Requirements

```python
RETENTION_REQUIREMENTS = {
    "pci_dss": {"min_days": 365, "source": "PCI DSS 10.7"},
    "hipaa": {"min_days": 2190, "source": "45 CFR 164.530(j)"},
    "gdpr": {"min_days": None, "source": "Art. 5(1)(e) — minimize"},
    "sox": {"min_days": 2555, "source": "SOX Section 802"},
    "nist_800_92": {"min_days": 90, "source": "NIST SP 800-92 Sec. 4.2"},
}
```

### Step 2: Define Retention Tiers

```python
from dataclasses import dataclass

@dataclass
class RetentionTier:
    """Storage tier with retention parameters."""
    name: str
    storage_class: str
    max_age_days: int
    cost_per_gb_month: float
    query_latency_sla: str
    immutable: bool = True

TIERS = [
    RetentionTier("hot", "SSD", 30, 0.10, "< 1s"),
    RetentionTier("warm", "S3_STANDARD", 90, 0.023, "< 30s"),
    RetentionTier("cold", "S3_IA", 365, 0.0125, "< 5m"),
    RetentionTier("archive", "GLACIER", 2190, 0.004, "< 12h"),
]
```

### Step 3: Implement Lifecycle Automation

```bash
# Elasticsearch ILM policy
curl -X PUT "localhost:9200/_ilm/policy/security_retention" \
  -H 'Content-Type: application/json' -d '{
  "policy": {
    "phases": {
      "hot":    {"min_age": "0d", "actions": {"rollover": {"max_size": "50gb", "max_age": "1d"}}},
      "warm":   {"min_age": "30d", "actions": {"shrink": {"number_of_shards": 1}}},
      "cold":   {"min_age": "90d", "actions": {"searchable_snapshot": {"snapshot_repository": "cold_repo"}}},
      "delete": {"min_age": "365d", "actions": {"delete": {}}}
    }
  }
}'
```

### Step 4: Validate and Audit

```bash
node agent.js audit --policy security_retention --check-compliance
node agent.js report --sources all --format json
```

## Detection

```yaml
title: Security Data Retention Policies Detection
id: 41ee2ea4-e01c-43b1-92e1-4d5ea9508984
status: experimental
description: Detects suspicious activity related to implementing security data retention policies techniques in security data engineering context
logsource:
  category: process_creation
  product: linux
detection:
  selection:
    CommandLine: "*implementing*security*"
  condition: selection
level: medium
tags:
  - attack.t1485
  - attack.t1565
  - attack.collection
falsepositives:
  - Data pipeline ETL processes during scheduled batch windows
```

## Verification

| Check | Method |
|---|---|
| Regulatory coverage | All mandates mapped to retention tiers |
| Lifecycle automation | Policies applied and transitions verified |
| Purge verification | Deleted data confirmed unrecoverable |
| Legal hold tested | Hold overrides deletion correctly |

## References

- [NIST SP 800-92 — Log Management](https://csrc.nist.gov/publications/detail/sp/800-92/final)
- [PCI DSS v4.0 Requirement 10.7](https://www.pcisecuritystandards.org/)
- [GDPR Article 5 — Data Minimization](https://gdpr-info.eu/art-5-gdpr/)
