<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: building-custom-log-parsers
description: >-
  Develop custom log parsers for non-standard log sources including
  regex extraction, structured parsing, field normalization, and
  SIEM-compatible output formatting.
domain: cybersecurity
subdomain: detection-engineering
tags:
  - log-parsing
  - regex
  - field-extraction
  - normalization
  - siem-integration
  - custom-parsers
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1070"]
---

# Building Custom Log Parsers

## Overview

Custom log parsers extend detection coverage to non-standard sources.
Covers regex development, structured parsing, field normalization to
ECS/CIM schemas, and SIEM ingestion pipeline integration.

## Prerequisites

| Requirement | Purpose |
|---|---|
| Sample logs | Representative samples from target source |
| Schema reference | ECS or CIM field mapping docs |
| Python 3.10+ | Parser development and testing |

## Key Concepts

### Log Parser Architecture

```python
import re

AUTH_PATTERN = re.compile(
    r'(?P<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2})\s+'
    r'(?P<level>\w+)\s+auth:\s+(?P<action>\w+)\s+'
    r'user=(?P<user>\S+)\s+src=(?P<src_ip>[\d.]+)\s+result=(?P<result>\w+)'
)

def parse_auth_log(line: str) -> dict | None:
    match = AUTH_PATTERN.match(line)
    if match:
        return {
            "timestamp": match.group("timestamp"),
            "event.category": "authentication",
            "user.name": match.group("user"),
            "source.ip": match.group("src_ip"),
        }
    return None
```

## Workflow

1. **Analyze log format** — Study samples for structure and fields
2. **Design regex** — Build patterns with named capture groups
3. **Map to schema** — Normalize to ECS or CIM standard
4. **Implement parser** — Python module with error handling
5. **Test thoroughly** — Diverse samples including edge cases
6. **Integrate with SIEM** — Configure ingestion pipeline
7. **Monitor parsing** — Track success rate and accuracy

## Detection

```yaml
title: Custom Log Parsers Detection
id: 0e973597-dd1d-46da-a3af-c75e4c637cf9
status: experimental
description: Detects suspicious activity related to building custom log parsers techniques in detection engineering context
logsource:
  category: process_creation
  product: windows
detection:
  selection:
    CommandLine: "*building*custom*"
  condition: selection
level: medium
tags:
  - attack.t1070
  - attack.defense_evasion
falsepositives:
  - Detection rule testing in staging SIEM environment
```

## Verification

| Check | Method |
|---|---|
| Workflow completed | All steps executed and documented |
| Output validated | Results match expected format and content |
| Quality assured | Peer review or automated validation passed |

## References

- [NIST Cybersecurity Framework](https://www.nist.gov/cyberframework)
- [MITRE ATT&CK](https://attack.mitre.org/)
