<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: building-custom-log-collectors
description: >-
  Build custom log collection agents that tail, parse, normalize, and forward
  security-relevant logs from heterogeneous sources (syslog, Windows Event Log,
  application logs, cloud audit trails) into SIEM or data lake backends.
  Covers structured parsing with regex and CEF/LEEF formatting.
domain: cybersecurity
subdomain: automation-scripting
tags:
  - log-collection
  - syslog
  - siem-integration
  - log-parsing
  - cef
  - event-normalization
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1005"]
---

# Building Custom Log Collectors

## Overview

Commercial log collectors miss edge cases. This skill covers building custom
agents that tail log files, parse unstructured entries into normalized fields,
convert to CEF/JSON, and forward to SIEM infrastructure. Handles multi-line
entries, rotation, and backpressure.

Mode: `[MODE: BLUE]` — Defensive log engineering for detection coverage.

## Prerequisites

| Requirement | Details |
|---|---|
| Python 3.10+ with `watchdog`, `regex` | Required |
| Syslog receiver or SIEM endpoint (Splunk HEC, Elastic, etc.) | Required |
| Read access to target log files or journal | Required |

## Key Concepts

### File Tail with Rotation Handling

```python
import time
from pathlib import Path

def tail_file(path: str, callback, poll_interval: float = 0.5) -> None:
    """Tail a log file, handling rotation via inode tracking."""
    fpath = Path(path)
    inode = fpath.stat().st_ino
    fh = open(fpath)
    fh.seek(0, 2)  # seek to end
    while True:
        line = fh.readline()
        if line:
            callback(line.strip())
        else:
            try:
                if fpath.stat().st_ino != inode:
                    fh.close()
                    fh = open(fpath)
                    inode = fpath.stat().st_ino
            except FileNotFoundError:
                pass
            time.sleep(poll_interval)
```

### Syslog Parser to CEF

```python
import re
from datetime import datetime

SYSLOG_RE = re.compile(
    r"^<(\d+)>(\w{3}\s+\d+\s+[\d:]+)\s+(\S+)\s+(\S+?):\s*(.*)"
)

def parse_syslog(line: str) -> dict | None:
    """Parse RFC 3164 syslog into structured dict."""
    match = SYSLOG_RE.match(line)
    if not match:
        return None
    priority, timestamp, host, program, message = match.groups()
    facility = int(priority) >> 3
    severity = int(priority) & 0x07
    return {
        "facility": facility,
        "severity": severity,
        "timestamp": timestamp,
        "host": host,
        "program": program,
        "message": message,
    }

def to_cef(event: dict) -> str:
    """Convert parsed event to CEF format."""
    return (
        f"CEF:0|Custom|LogCollector|1.0|{event['program']}|"
        f"{event['message'][:80]}|{event['severity']}|"
        f"src={event['host']} msg={event['message']}"
    )
```

### HTTP Forwarder (Splunk HEC)

```python
import requests

def forward_to_splunk(event: dict, hec_url: str, token: str) -> bool:
    """Forward normalized event to Splunk HTTP Event Collector."""
    payload = {"event": event, "sourcetype": "custom:security"}
    resp = requests.post(
        hec_url,
        json=payload,
        headers={"Authorization": f"Splunk {token}"},
        timeout=10,
    )
    return resp.status_code == 200
```

## Workflow

### Step 1: Tail and Parse Log File

```bash
node scripts/agent.js --action tail --log-path /var/log/auth.log --format syslog
```

### Step 2: Parse and Normalize a Log Batch

```bash
node scripts/agent.js --action parse --log-path /var/log/auth.log --lines 100
```

### Step 3: Forward to SIEM

```bash
node scripts/agent.js --action forward --log-path /var/log/auth.log --hec-url https://splunk:8088/services/collector
```

## Verification

- [ ] Syslog parser handles RFC 3164 and RFC 5424 formats
- [ ] File tail detects log rotation and reopens correctly
- [ ] CEF output conforms to ArcSight CEF specification
- [ ] Forwarder handles connection failures with retry logic
- [ ] Exit code 0 on success, 1 on parse errors, 2 on fatal error

## References

- [RFC 3164 — BSD Syslog Protocol](https://www.rfc-editor.org/rfc/rfc3164)
- [CEF Format Specification](https://www.microfocus.com/documentation/arcsight/arcsight-smartconnectors/)
- [Splunk HEC Documentation](https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector)
- [MITRE ATT&CK T1070 — Indicator Removal](https://attack.mitre.org/techniques/T1070/)
