<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: building-log-parsing-pipelines
description: >-
  Build automated log parsing pipelines for security analysis using Python regex,
  structured extraction, anomaly detection, and timeline reconstruction from
  syslog, auth.log, Apache/Nginx, and cloud audit logs.
domain: cybersecurity
subdomain: automation-scripting
tags:
  - log-analysis
  - siem
  - parsing
  - regex
  - timeline
  - forensics
  - anomaly-detection
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1070.002"]
---

# Building Log Parsing Pipelines

## Overview

Security log parsing transforms unstructured log data into actionable intelligence.
This skill covers building Python pipelines that ingest, normalize, and analyze
logs from multiple sources — extracting IOCs, building timelines, detecting anomalies,
and feeding structured output into SIEM or alerting systems.

Mode: `[MODE: BLUE]` — Defensive log analysis and threat detection.

## Prerequisites

| Requirement | Details |
|---|---|
| Python 3.10+ with `re`, `json`, `csv` standard library | Required |
| Log sources | syslog, auth.log, Apache/Nginx access logs, cloud audit trails |
| Optional | `pandas` for dataframe analysis, `python-dateutil` for timestamp parsing |

## Key Concepts

### Syslog Parser

```python
import re
from datetime import datetime

SYSLOG_PATTERN = re.compile(
    r"(?P<timestamp>\w{3}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2})\s+"
    r"(?P<hostname>\S+)\s+"
    r"(?P<process>\S+?)(?:\[(?P<pid>\d+)\])?\s*:\s*"
    r"(?P<message>.*)"
)

def parse_syslog(line: str) -> dict | None:
    match = SYSLOG_PATTERN.match(line)
    if not match:
        return None
    return match.groupdict()
```

### Failed SSH Login Detector

```python
FAILED_SSH = re.compile(
    r"Failed (?:password|publickey) for (?:invalid user )?(?P<user>\S+) "
    r"from (?P<ip>\d+\.\d+\.\d+\.\d+) port (?P<port>\d+)"
)

def extract_failed_logins(log_lines: list[str]) -> list[dict]:
    failures = []
    for line in log_lines:
        match = FAILED_SSH.search(line)
        if match:
            failures.append(match.groupdict())
    return failures
```

### Apache Access Log Parser

```python
APACHE_COMBINED = re.compile(
    r'(?P<ip>\S+) \S+ \S+ \[(?P<timestamp>[^\]]+)\] '
    r'"(?P<method>\S+) (?P<path>\S+) \S+" (?P<status>\d+) (?P<size>\S+) '
    r'"(?P<referer>[^"]*)" "(?P<useragent>[^"]*)"'
)

def parse_apache_log(line: str) -> dict | None:
    match = APACHE_COMBINED.match(line)
    if not match:
        return None
    result = match.groupdict()
    result["status"] = int(result["status"])
    return result
```

## Workflow

### Step 1: Parse Auth Logs

```bash
node scripts/agent.js --action parse-auth --log-file /var/log/auth.log --output /tmp/auth-parsed.json
```

### Step 2: Extract Failed Logins

```bash
node scripts/agent.js --action failed-logins --log-file /var/log/auth.log --output /tmp/failed-logins.json
```

### Step 3: Analyze Web Access Logs

```bash
node scripts/agent.js --action parse-web --log-file /var/log/apache2/access.log --output /tmp/web-parsed.json
```

## Verification

- [ ] Syslog parser handles multi-format timestamps
- [ ] Failed login extractor identifies brute-force patterns (>10 failures from same IP)
- [ ] Apache parser processes combined log format without errors
- [ ] IOC extraction captures IPs, domains, and file hashes
- [ ] Timeline reconstruction orders events chronologically

## References

- [Syslog RFC 5424](https://tools.ietf.org/html/rfc5424)
- [Apache Log Formats](https://httpd.apache.org/docs/current/logs.html)
- [MITRE ATT&CK T1078 — Valid Accounts](https://attack.mitre.org/techniques/T1078/)
- [Sigma Log Sources](https://github.com/SigmaHQ/sigma-specification)
