<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: implementing-security-data-lake-pipelines
description: >-
  Build automated pipelines to ingest, normalize, and route security telemetry
  into a data lake architecture using schema-on-read, partitioning strategies,
  and retention policies for cost-effective long-term security analytics.
domain: cybersecurity
subdomain: security-operations-automation
tags:
  - data-lake
  - pipeline
  - log-ingestion
  - normalization
  - ocsf
  - parquet
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: []
---

# Implementing Security Data Lake Pipelines

## Overview

Security data lakes decouple storage from compute, enabling cost-effective
retention of high-volume telemetry (DNS, NetFlow, EDR, cloud audit logs)
while supporting ad-hoc threat hunting. This technique covers pipeline
architecture: ingestion, normalization to OCSF/ECS, partitioning, and
query optimization.

Mode: `[MODE: BLUE]` — Detection infrastructure and analytics.

## Prerequisites

| Requirement | Details |
|---|---|
| Python 3.10+ with `pyarrow`, `pyyaml`, `requests` | Required |
| Object storage (S3, GCS, MinIO) | Required |
| Schema registry or OCSF mapping definitions | Required |
| Query engine (Athena, Trino, DuckDB) | Required |

## Key Concepts

### Event Normalization to OCSF

```python
from datetime import datetime, timezone
from typing import Any


OCSF_CLASS_MAP: dict[str, int] = {
    "authentication": 3002,
    "network_activity": 4001,
    "process_activity": 1007,
    "dns_activity": 4003,
    "file_activity": 1001,
}


def normalize_to_ocsf(raw_event: dict[str, Any], event_class: str) -> dict[str, Any]:
    """Normalize a raw log event to OCSF schema."""
    return {
        "class_uid": OCSF_CLASS_MAP.get(event_class, 0),
        "class_name": event_class,
        "time": raw_event.get("timestamp", datetime.now(timezone.utc).isoformat()),
        "severity_id": raw_event.get("severity_id", 1),
        "metadata": {
            "version": "1.1.0",
            "product": {"name": raw_event.get("source", "unknown")},
            "log_name": raw_event.get("log_type", "generic"),
        },
        "observables": _extract_observables(raw_event),
        "unmapped": {
            k: v for k, v in raw_event.items()
            if k not in ("timestamp", "severity_id", "source", "log_type")
        },
    }


def _extract_observables(event: dict) -> list[dict[str, str]]:
    """Pull IOC-relevant fields into observables array."""
    obs: list[dict[str, str]] = []
    if "src_ip" in event:
        obs.append({"name": "src_ip", "type": "ip_address", "value": event["src_ip"]})
    if "dst_ip" in event:
        obs.append({"name": "dst_ip", "type": "ip_address", "value": event["dst_ip"]})
    if "domain" in event:
        obs.append({"name": "domain", "type": "hostname", "value": event["domain"]})
    return obs
```

### Partitioned Storage Writer

```python
from pathlib import Path


def write_partitioned(
    events: list[dict],
    base_path: str,
    partition_keys: list[str] | None = None,
) -> dict[str, int]:
    """Write events to date-partitioned JSON files."""
    partition_keys = partition_keys or ["year", "month", "day"]
    now = datetime.now(timezone.utc)
    partition = f"year={now.year}/month={now.month:02d}/day={now.day:02d}"
    output_dir = Path(base_path) / partition
    output_dir.mkdir(parents=True, exist_ok=True)

    output_file = output_dir / f"batch_{now.strftime('%H%M%S')}.json"
    import json

    output_file.write_text(json.dumps(events, indent=2))
    return {"records_written": len(events), "path": str(output_file)}
```

## Workflow

### Step 1: Ingest Raw Events

```bash
node scripts/agent.js --action ingest --source firewall --count 100
```

### Step 2: Normalize to OCSF

```bash
node scripts/agent.js --action normalize --schema ocsf
```

### Step 3: Query Partitioned Data

```bash
node scripts/agent.js --action query --days 7
```

## Verification

- [ ] Raw events normalized to OCSF with class_uid mapping
- [ ] Observables extracted for IOC-relevant fields
- [ ] Data partitioned by date for efficient querying
- [ ] Unmapped fields preserved for forensic analysis

## References

- [OCSF Schema — Open Cybersecurity Schema Framework](https://schema.ocsf.io/)
- [AWS Security Lake](https://docs.aws.amazon.com/security-lake/)
- [Apache Parquet for Security Data](https://parquet.apache.org/)
