<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: building-security-data-lakes
description: >-
  Design and implement security data lakes using open table formats (Iceberg,
  Delta Lake) with hot/warm/cold tiering, partitioning strategies, and access
  controls optimized for security analytics workloads.
domain: cybersecurity
subdomain: security-data-engineering
tags:
  - data-lake
  - iceberg
  - delta-lake
  - s3
  - partitioning
  - tiering
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1530"]
  frameworks: ["NIST 800-92", "CIS Controls v8"]
  tools: ["python3", "duckdb", "apache-spark", "aws-cli"]
---

# Building Security Data Lakes

## Overview

A security data lake centralizes heterogeneous telemetry — endpoint, network,
cloud, identity — into a queryable store optimized for threat hunting, detection
analytics, and compliance retention. This skill covers architecture decisions
including table format selection, partitioning by time and source, tiered storage
for cost optimization, and access control models.

## Prerequisites

| Requirement | Details |
|---|---|
| Object storage (S3, GCS, MinIO) with IAM policies | Required |
| Familiarity with Apache Iceberg or Delta Lake table formats | Required |
| Understanding of security log volumes and retention requirements | Required |
| Query engine (Spark, Trino, DuckDB) for analytics | Required |

## Key Concepts

- **Open table format**: Iceberg/Delta Lake for ACID transactions on object storage
- **Time-based partitioning**: Partition by day/hour for efficient range scans
- **Tiered storage**: Hot (SSD/local), warm (S3 Standard), cold (Glacier/Archive)
- **Schema evolution**: Add fields without rewriting historical data
- **Compaction**: Merge small files to optimize query performance

## Workflow

### Step 1: Define Lake Architecture

```python
from dataclasses import dataclass, field

@dataclass
class LakeTier:
    """Storage tier configuration."""
    name: str  # "hot", "warm", "cold"
    retention_days: int
    storage_class: str  # "SSD", "STANDARD", "GLACIER"
    query_latency_sla: str  # "< 1s", "< 30s", "< 5m"

@dataclass
class SecurityDataLake:
    """Security data lake architecture specification."""
    tiers: list[LakeTier] = field(default_factory=list)
    partition_keys: list[str] = field(default_factory=lambda: ["event_date", "source_type"])
    table_format: str = "iceberg"
    catalog: str = "glue"  # or "hive", "nessie"
```

### Step 2: Create Iceberg Tables

```sql
CREATE TABLE security.events (
    event_id       STRING,
    time_dt        TIMESTAMP,
    source_type    STRING,
    severity_id    INT,
    category_uid   INT,
    raw_event      STRING,
    enrichments    MAP<STRING, STRING>
)
USING iceberg
PARTITIONED BY (days(time_dt), source_type)
TBLPROPERTIES (
    'write.target-file-size-bytes' = '536870912',
    'history.expire.max-snapshot-age-ms' = '259200000'
);
```

### Step 3: Implement Lifecycle Policies

```bash
# Transition warm data to Glacier after 90 days
aws s3api put-bucket-lifecycle-configuration \
  --bucket security-data-lake \
  --lifecycle-configuration '{
    "Rules": [{
      "ID": "SecurityTiering",
      "Status": "Enabled",
      "Transitions": [
        {"Days": 30, "StorageClass": "STANDARD_IA"},
        {"Days": 90, "StorageClass": "GLACIER"}
      ],
      "Expiration": {"Days": 365}
    }]
  }'
```

### Step 4: Validate Lake Health

```bash
node agent.js audit --lake s3://security-data-lake/ --format iceberg
node agent.js partitions --table security.events --check-skew
```

## Detection

```yaml
title: Security Data Lakes Detection
id: 36bbd2fe-a6ed-4fcc-8d00-16c23dc843bc
status: experimental
description: Detects suspicious activity related to building security data lakes techniques in security data engineering context
logsource:
  category: process_creation
  product: linux
detection:
  selection:
    CommandLine: "*building*security*"
  condition: selection
level: medium
tags:
  - attack.t1530
  - attack.collection
falsepositives:
  - Data pipeline ETL processes during scheduled batch windows
```


**Detection Opportunities**

| Indicator | Source | Detection Logic |
|---|---|---|
| Security Data Lakes Detection | linux/process_creation | Sigma rule (medium) |
| ATT&CK Coverage | MITRE ATT&CK | T1530 |

## References

- [Apache Iceberg Spec](https://iceberg.apache.org/spec/)
- [NIST SP 800-92](https://csrc.nist.gov/publications/detail/sp/800-92/final)
- [CIS Controls v8 — Control 8](https://www.cisecurity.org/controls/audit-log-management)
