<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: security-data-engineering
description: >-
  Security data engineering covering data lake architecture, log normalization
  pipelines, OCSF schema implementation, SIEM data architecture, detection data
  modeling, security metrics pipelines, data quality assessment, enrichment
  workflows, log volume optimization, data governance, and log source reliability
  analysis. Enables scalable, high-fidelity security analytics through structured
  data engineering practices.
domain: cybersecurity
subdomain: security-data-engineering
tags:
  - data-lake
  - log-normalization
  - ocsf
  - siem-architecture
  - detection-engineering
  - data-quality
  - enrichment
  - metrics-pipeline
  - data-governance
  - log-optimization
  - data-modeling
  - log-reliability
version: "1.0"
author: defconxt
license: AGPL-3.0
compatibility: Designed for Claude Code, GitHub Copilot, OpenAI Codex, Cursor, Gemini CLI, and any agentskills.io-compatible agent.
metadata:
  mitre-attack: ["T1005", "T1530", "T1119"]
  frameworks: ["NIST 800-92", "OCSF 1.1", "CIS Controls v8", "NIST CSF 2.0"]
  tools: ["python3", "apache-spark", "apache-kafka", "elasticsearch", "duckdb", "sigma-cli"]
---

# Security Data Engineering

## When to Use

Activate when the operator asks about building security data lakes, normalizing
log data, implementing OCSF schemas, designing SIEM data architectures, creating
detection analytics pipelines, enriching security telemetry, optimizing log
volumes, building metrics dashboards, assessing data quality, or establishing
security data governance programs.

Mode: `[MODE: BLUE]` for pipeline design and data quality; `[MODE: ARCHITECT]` for data lake and SIEM architecture; `[MODE: PURPLE]` for detection data modeling.

## Prerequisites

- Understanding of security log sources (EDR, firewall, proxy, cloud audit)
- Familiarity with data engineering tools (Spark, Kafka, SQL, Python)
- Knowledge of OCSF, ECS, or CEF normalization schemas
- Access to log ingestion infrastructure and SIEM platforms

## Quick Reference

| Capability | Tools / Standards | Use Case |
|------------|-------------------|----------|
| Data lake architecture | S3, Delta Lake, Iceberg, DuckDB | Centralized security telemetry storage |
| Log normalization | OCSF, ECS, CEF, custom parsers | Schema-consistent event processing |
| Data quality assessment | Great Expectations, dbt tests | Validate completeness, freshness, accuracy |
| Threat detection analytics | Sigma, YARA-L, KQL, SPL | Rule-based and statistical detection |
| Data enrichment | MISP, MaxMind, Shodan, VirusTotal | Context augmentation for triage |
| Volume optimization | Filtering, sampling, aggregation | Cost reduction without fidelity loss |
| Metrics pipelines | Prometheus, Grafana, custom ETL | Security KPI tracking and reporting |
| OCSF normalization | OCSF 1.1 schema, mapping tools | Vendor-neutral event taxonomy |
| Detection data modeling | Entity graphs, session stitching | Behavioral analytics foundation |
| SIEM architecture | Hot/warm/cold tiers, index design | Scalable query and retention |
| Data governance | Classification, lineage, retention | Regulatory compliance and data hygiene |
| Source reliability | Coverage scoring, gap detection | Ensure detection coverage completeness |

## Key Concepts

- **Schema-on-read vs schema-on-write**: trade-offs for security analytics
- **Hot/warm/cold tiering**: balance query speed against storage cost
- **OCSF (Open Cybersecurity Schema Framework)**: vendor-neutral event taxonomy
- **Detection-as-code**: version-controlled analytics with CI/CD pipelines
- **Data lineage**: tracking transformations from source to detection
- **Log fidelity**: ensuring telemetry supports detection hypothesis testing

## Workflow

### Step 1: Assess Current Data Landscape

```bash
# Inventory log sources and daily volumes
node agent.js inventory --sources /etc/logstash/conf.d/
# Assess schema coverage against OCSF categories
node agent.js assess --schema ocsf --output coverage-report.json
```

### Step 2: Design Normalization Pipeline

```python
from dataclasses import dataclass

@dataclass
class NormalizationMapping:
    """Map vendor fields to OCSF schema."""
    source_field: str
    ocsf_field: str
    transform: str  # "direct", "regex", "lookup", "computed"
    category_uid: int  # OCSF category UID
```

### Step 3: Implement Quality Gates

```yaml
# Great Expectations checkpoint for security logs
expectations:
  - expect_column_values_to_not_be_null:
      column: timestamp
  - expect_column_values_to_not_be_null:
      column: source_ip
  - expect_column_values_to_be_between:
      column: severity_id
      min_value: 0
      max_value: 5
```

### Step 4: Build Detection Analytics Layer

```sql
-- Detect anomalous authentication volume per source
SELECT
    src_endpoint.ip AS source_ip,
    COUNT(*) AS auth_count,
    COUNT(DISTINCT dst_endpoint.ip) AS unique_targets
FROM authentication_events
WHERE time_dt >= CURRENT_TIMESTAMP - INTERVAL '1 hour'
GROUP BY src_endpoint.ip
HAVING auth_count > 100 OR unique_targets > 20;
```

### Step 5: Validate and Monitor Pipeline Health

```bash
# Run data quality checks
node agent.js quality --pipeline auth-normalization --window 24h
# Generate metrics dashboard data
node agent.js metrics --output pipeline-health.json
```

## References

- [OCSF Schema 1.1](https://schema.ocsf.io/) — Open Cybersecurity Schema Framework
- [NIST SP 800-92](https://csrc.nist.gov/publications/detail/sp/800-92/final) — Guide to Computer Security Log Management
- [CIS Controls v8 — Control 8](https://www.cisecurity.org/controls/audit-log-management) — Audit Log Management
- [NIST CSF 2.0 DE.CM](https://www.nist.gov/cyberframework) — Continuous Monitoring
- [Apache Iceberg](https://iceberg.apache.org/) — Table format for data lakes
- [Sigma HQ](https://github.com/SigmaHQ/sigma) — Detection-as-code standard
