<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: implementing-log-normalization-pipelines
description: >-
  Build log normalization pipelines that transform heterogeneous vendor formats
  into consistent schemas using field mapping, regex extraction, lookup tables,
  and streaming processors for real-time security analytics.
domain: cybersecurity
subdomain: security-data-engineering
tags:
  - log-normalization
  - etl
  - parsing
  - schema-mapping
  - streaming
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1005", "T1119"]
  frameworks: ["NIST 800-92", "OCSF 1.1"]
  tools: ["python3", "logstash", "apache-kafka", "vector"]
---

# Implementing Log Normalization Pipelines

## Overview

Log normalization converts vendor-specific event formats into a common schema,
enabling cross-source correlation, consistent detection rules, and unified
analytics. This skill covers parser development, field mapping strategies,
streaming pipeline architecture, and validation of normalized output.

## Prerequisites

| Requirement | Details |
|---|---|
| Sample logs from target sources (syslog, JSON, CEF, XML) | Required |
| Target normalization schema (OCSF, ECS, or custom) | Required |
| Streaming infrastructure (Kafka, Kinesis) or batch ETL tooling | Required |
| Parser development environment with Python 3.10+ | Required |

## Key Concepts

- **Field mapping**: Direct, computed, and lookup-based transformations
- **Parser chain**: Sequential extraction stages (timestamp → source → fields)
- **Schema validation**: Enforce required fields and type constraints post-parse
- **Dead letter queue**: Route unparseable events for investigation

## Workflow

### Step 1: Analyze Source Format

```python
import re
from typing import Any

def analyze_log_format(sample: str) -> dict[str, Any]:
    """Detect log format from sample line."""
    if sample.startswith("{"):
        return {"format": "json", "parser": "json_decode"}
    if re.match(r"CEF:\d\|", sample):
        return {"format": "cef", "parser": "cef_decode"}
    if re.match(r"<\d+>", sample):
        return {"format": "syslog", "parser": "syslog_rfc5424"}
    return {"format": "unknown", "parser": "regex_custom"}
```

### Step 2: Build Field Mapping

```yaml
# Mapping: CrowdStrike Falcon -> OCSF Authentication
mappings:
  - source_field: "timestamp"
    target_field: "time_dt"
    transform: "iso8601_parse"
  - source_field: "UserName"
    target_field: "actor.user.name"
    transform: "direct"
  - source_field: "SourceIP"
    target_field: "src_endpoint.ip"
    transform: "direct"
  - source_field: "EventType"
    target_field: "activity_id"
    transform: "lookup"
    lookup_table: "crowdstrike_activity_map"
```

### Step 3: Implement Streaming Normalizer

```python
def normalize_event(raw: dict, mapping: list[dict]) -> dict:
    """Apply field mappings to normalize a raw event."""
    normalized: dict[str, Any] = {}
    for m in mapping:
        value = raw.get(m["source_field"])
        if value is None:
            continue
        if m["transform"] == "direct":
            normalized[m["target_field"]] = value
        elif m["transform"] == "iso8601_parse":
            normalized[m["target_field"]] = parse_timestamp(value)
        elif m["transform"] == "lookup":
            normalized[m["target_field"]] = lookup(m["lookup_table"], value)
    return normalized
```

### Step 4: Validate Normalized Output

```bash
node agent.js validate --schema ocsf --input normalized_events.jsonl
node agent.js stats --input normalized_events.jsonl --report coverage
```

## Detection

```yaml
title: Log Normalization Pipelines Detection
id: b3e3ba57-62a7-40da-86c1-a32cb2a9124d
status: experimental
description: Detects suspicious activity related to implementing log normalization pipelines techniques in security data engineering context
logsource:
  category: process_creation
  product: linux
detection:
  selection:
    CommandLine: "*implementing*log*"
  condition: selection
level: medium
tags:
  - attack.t1005
  - attack.t1119
  - attack.collection
falsepositives:
  - Data pipeline ETL processes during scheduled batch windows
```


**Detection Opportunities**

| Indicator | Source | Detection Logic |
|---|---|---|
| Log Normalization Pipelines Detection | linux/process_creation | Sigma rule (medium) |
| ATT&CK Coverage | MITRE ATT&CK | T1005, T1119 |

## References

- [OCSF Schema 1.1](https://schema.ocsf.io/)
- [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/index.html)
- [Vector by Datadog](https://vector.dev/) — Log processing pipeline
