<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: automating-log-retention-management
description: >-
  Automate log retention policy enforcement including lifecycle management,
  tiered storage migration, compliance-driven retention schedules, integrity
  verification, and automated purge with audit trails.
domain: cybersecurity
subdomain: security-operations-automation
tags:
  - log-retention
  - compliance
  - lifecycle
  - storage-tiering
  - audit
  - automation
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1070"]
---

# Automating Log Retention Management

## Overview

Log retention policies must balance compliance requirements (PCI-DSS 1 year,
HIPAA 6 years, SOX 7 years) against storage costs. This technique automates
lifecycle management: classifying log sources by compliance requirement,
enforcing retention periods, migrating aged data to cold storage, verifying
integrity of archived logs, and purging expired data with audit trails.

Mode: `[MODE: BLUE]` — Compliance-driven log management.

## Prerequisites

| Requirement | Details |
|---|---|
| Python 3.10+ with `hashlib`, `pyyaml` | Required |
| Object storage with lifecycle policies (S3, GCS) | Required |
| Log classification mapping by compliance framework | Required |
| Integrity verification database (hash store) | Required |

## Key Concepts

### Retention Policy Definition

```python
from dataclasses import dataclass
from typing import Any


@dataclass
class RetentionPolicy:
    """Log retention policy per source and compliance requirement."""

    source: str
    hot_days: int
    warm_days: int
    cold_days: int
    total_retention_days: int
    compliance: str
    integrity_check: bool = True


POLICIES: list[RetentionPolicy] = [
    RetentionPolicy("firewall", 30, 90, 365, 365, "PCI-DSS"),
    RetentionPolicy("auth_logs", 90, 365, 2190, 2190, "HIPAA"),
    RetentionPolicy("application", 14, 60, 180, 180, "internal"),
    RetentionPolicy("audit_trail", 90, 365, 2555, 2555, "SOX"),
    RetentionPolicy("dns_queries", 7, 30, 90, 90, "internal"),
]
```

### Storage Tier Migration

```python
from datetime import datetime, timezone


def classify_log_tier(
    log_date: str,
    policy: RetentionPolicy,
) -> str:
    """Determine storage tier based on log age and policy."""
    log_dt = datetime.fromisoformat(log_date)
    age_days = (datetime.now(timezone.utc) - log_dt).days

    if age_days <= policy.hot_days:
        return "hot"
    if age_days <= policy.warm_days:
        return "warm"
    if age_days <= policy.cold_days:
        return "cold"
    return "expired"
```

### Integrity Verification

```python
import hashlib


def compute_log_hash(content: bytes) -> str:
    """Compute SHA-256 hash for log integrity verification."""
    return hashlib.sha256(content).hexdigest()


def verify_archive_integrity(
    archive_path: str,
    expected_hash: str,
) -> dict[str, Any]:
    """Verify archived log file integrity against stored hash."""
    from pathlib import Path

    content = Path(archive_path).read_bytes()
    actual_hash = compute_log_hash(content)
    return {
        "path": archive_path,
        "expected_hash": expected_hash[:16] + "...",
        "actual_hash": actual_hash[:16] + "...",
        "integrity_valid": actual_hash == expected_hash,
    }
```

## Workflow

### Step 1: Review Retention Policies

```bash
node scripts/agent.js --action list-policies
```

### Step 2: Run Lifecycle Assessment

```bash
node scripts/agent.js --action assess --source firewall
```

### Step 3: Execute Purge

```bash
node scripts/agent.js --action purge --dry-run
```

## Detection

```yaml
title: Automating Log Retention Management Detection
id: 83009993-6f54-4e4b-80a1-7cdfe1927de6
status: experimental
description: Detects suspicious activity related to automating log retention management techniques in security operations automation context
logsource:
  category: process_creation
  product: linux
detection:
  selection:
    CommandLine: "*automating*log*"
  condition: selection
level: medium
tags:
  - attack.t1070
  - attack.discovery
falsepositives:
  - Orchestration platform executing approved automation workflows
```


**Detection Opportunities**

| Indicator | Source | Detection Logic |
|---|---|---|
| Automating Log Retention Management Detection | linux/process_creation | Sigma rule (medium) |
| ATT&CK Coverage | MITRE ATT&CK | T1070 |

## Verification

- [ ] Retention policies match compliance requirements
- [ ] Log tier classification correct based on age
- [ ] Archive integrity verified before purge
- [ ] Purge operations logged with full audit trail

## References

- [PCI-DSS Requirement 10.7 — Log Retention](https://www.pcisecuritystandards.org/)
- [NIST SP 800-92 — Guide to Computer Security Log Management](https://csrc.nist.gov/publications/detail/sp/800-92/final)
- [AWS S3 Lifecycle Policies](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html)
