<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
<!-- NERF is a trademark of defconxt. -->
---
name: performing-log-correlation-engineering
description: >-
  Engineer cross-source log correlation rules that link events from disparate
  security systems to identify attack chains, reduce alert noise, and surface
  high-fidelity composite detections.
domain: cybersecurity
subdomain: security-data-engineering
tags:
  - log-correlation
  - siem
  - detection-engineering
  - event-linking
  - alert-enrichment
  - attack-chain
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1078", "T1071", "T1059"]
  frameworks: ["NIST 800-92", "MITRE ATT&CK", "Sigma"]
  tools: ["python3", "elasticsearch", "splunk", "sigma"]
---

# Performing Log Correlation Engineering

## Overview

Log correlation engineering transforms isolated events from multiple security
sources into linked attack narratives. By joining events across time windows,
shared entities (IPs, users, hostnames), and kill-chain phases, correlation
rules produce high-confidence composite alerts.

## Prerequisites

| Requirement | Purpose |
|---|---|
| Normalized log sources | Common schema (OCSF/ECS) across sources |
| SIEM or correlation engine | Splunk, Elastic, or custom pipeline |
| Entity resolution | Map IPs/users/hosts across log sources |
| ATT&CK knowledge | Map events to tactics and techniques |

## Key Concepts

- **Correlation window**: Time bound for linking related events
- **Entity pivot**: Shared field (IP, user, hostname) connecting events
- **Attack chain**: Ordered sequence of correlated events across kill chain
- **Suppression**: Deduplicate redundant alerts from same root cause
- **Enrichment join**: Attach threat intel or asset context to correlated events

## Workflow

### Step 1: Identify Correlation Candidates

```python
CORRELATION_PAIRS = [
    {
        "name": "phishing_to_execution",
        "source_a": "email_gateway",
        "source_b": "endpoint_edr",
        "join_key": "recipient_email -> user_principal",
        "window_minutes": 30,
        "attack_chain": ["T1566.001", "T1204.002", "T1059.001"],
    },
    {
        "name": "brute_force_to_lateral",
        "source_a": "auth_logs",
        "source_b": "network_flows",
        "join_key": "source_ip",
        "window_minutes": 15,
        "attack_chain": ["T1110", "T1021"],
    },
]
```

### Step 2: Build Correlation Rules

```python
def correlate_events(
    events_a: list[dict],
    events_b: list[dict],
    join_key: str,
    window_seconds: int = 900,
) -> list[dict]:
    """Correlate events from two sources within a time window."""
    correlated = []
    for ea in events_a:
        for eb in events_b:
            if ea.get(join_key) == eb.get(join_key):
                delta = abs(ea["timestamp"] - eb["timestamp"])
                if delta <= window_seconds:
                    correlated.append({
                        "chain_id": f"{ea['event_id']}-{eb['event_id']}",
                        "source_a_event": ea["event_id"],
                        "source_b_event": eb["event_id"],
                        "join_value": ea[join_key],
                        "time_delta_seconds": delta,
                    })
    return correlated
```

### Step 3: Implement in SIEM

```spl
| tstats count FROM datamodel=Authentication WHERE Authentication.action=failure
  BY Authentication.src, Authentication.user, _time span=5m
| where count > 10
| rename Authentication.src AS src_ip, Authentication.user AS target_user
| join src_ip type=inner
  [search index=network sourcetype=zeek:conn dest_port!=443 dest_port!=80
   | stats dc(dest_ip) AS unique_dests BY src_ip
   | where unique_dests > 5]
| eval severity="critical",
       attack_chain="T1110->T1021",
       description="Brute force followed by lateral movement from ".src_ip
```

### Step 4: Validate Correlation Quality

```bash
node agent.js validate --rule brute_force_to_lateral --days 30
node agent.js tune --rule brute_force_to_lateral --fp-threshold 5
```

## Detection

```yaml
title: Log Correlation Engineering Detection
id: 3efad556-db42-4644-b23e-bbdd2e1b3331
status: experimental
description: Detects suspicious activity related to performing log correlation engineering techniques in security data engineering context
logsource:
  category: process_creation
  product: linux
detection:
  selection:
    CommandLine: "*performing*log*"
  condition: selection
level: medium
tags:
  - attack.t1078
  - attack.t1071
  - attack.t1059
  - attack.collection
falsepositives:
  - Data pipeline ETL processes during scheduled batch windows
```

## Verification

| Check | Method |
|---|---|
| Correlation fires on test data | Replay known attack scenario |
| False positive rate acceptable | < 5% over 30-day evaluation |
| Time window appropriate | No missed links, minimal noise |
| Entity resolution accurate | Join keys map correctly across sources |

## References

- [Sigma Correlation Rules](https://sigmahq.io/)
- [MITRE ATT&CK — Technique Chaining](https://attack.mitre.org/)
- [NIST SP 800-92 — Log Management](https://csrc.nist.gov/publications/detail/sp/800-92/final)
