<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: performing-log-source-onboarding
description: >-
  Onboard new log sources into SIEM platforms with proper parsing, field mapping,
  normalization, and validation. Covers CIM/ECS compliance, custom sourcetypes,
  ingestion pipelines, and data quality checks.
domain: cybersecurity
subdomain: blue-team
tags:
  - log-management
  - siem
  - data-onboarding
  - splunk
  - elastic
  - normalization
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1070"]
---

# Performing Log Source Onboarding

## Overview

Log source onboarding is the process of ingesting, parsing, normalizing, and
validating new data sources in a SIEM. Poorly onboarded logs create detection
blind spots and false positives. This skill covers structured onboarding from
intake through CIM/ECS field mapping and validation.

## Prerequisites

| Requirement | Install |
|---|---|
| Splunk 9+ or Elastic 8+ | SIEM platform |
| Splunk Add-on Builder | `splunkbase.splunk.com` for custom TAs |
| Elastic Ingest Pipelines | Built-in with Elastic Agent |
| Python 3.10+ | For agent tooling |
| Sample log data | Representative samples from the source |

## Key Concepts

### Onboarding Phases

1. **Discovery** — Identify log source, format, transport, and volume
2. **Collection** — Configure forwarder/agent to ship logs
3. **Parsing** — Extract fields from raw events
4. **Normalization** — Map fields to CIM/ECS data model
5. **Validation** — Confirm field accuracy and completeness
6. **Documentation** — Record source metadata and field mappings

### Splunk Sourcetype Configuration

```conf
# props.conf — parsing configuration
[custom:firewall:paloalto]
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
TIME_PREFIX = receive_time=
MAX_TIMESTAMP_LOOKAHEAD = 30
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TRUNCATE = 10000
category = Network & Security
description = Palo Alto Networks Firewall Logs

# Field extractions
EXTRACT-src_ip = src=(?<src_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})
EXTRACT-dest_ip = dst=(?<dest_ip>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})
EXTRACT-action = action=(?<action>\w+)

# CIM field aliases
FIELDALIAS-src = src_ip AS src
FIELDALIAS-dest = dest_ip AS dest
```

```conf
# transforms.conf — field lookups and transforms
[paloalto_action_lookup]
filename = paloalto_actions.csv
match_type = WILDCARD(action)
```

### Elastic Ingest Pipeline

```json
{
  "description": "Parse Palo Alto firewall logs",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": ["%{TIMESTAMP_ISO8601:@timestamp} %{IP:source.ip} %{IP:destination.ip} %{WORD:event.action}"]
      }
    },
    {
      "geoip": {
        "field": "source.ip",
        "target_field": "source.geo"
      }
    },
    {
      "set": {
        "field": "event.kind",
        "value": "event"
      }
    },
    {
      "set": {
        "field": "event.category",
        "value": ["network"]
      }
    }
  ]
}
```

### CIM Data Model Mapping

| CIM Field | Source Field | Transform |
|---|---|---|
| `src` | `src_ip` | Direct alias |
| `dest` | `dst_ip` | Direct alias |
| `action` | `action` | Lookup normalization (allow/deny) |
| `bytes_in` | `bytes_received` | Type cast to integer |
| `bytes_out` | `bytes_sent` | Type cast to integer |
| `transport` | `proto` | Lowercase normalization |

### Data Quality Checks

```spl
| tstats count WHERE index=firewall by sourcetype, _time span=1h
| timechart span=1h count by sourcetype

| tstats count WHERE index=firewall sourcetype="custom:firewall:paloalto"
  BY src, dest, action
| stats dc(src) AS unique_src, dc(dest) AS unique_dest, dc(action) AS unique_actions
```

## Workflow

1. **Collect samples** — Gather 1000+ representative events
2. **Identify format** — JSON, CSV, syslog, CEF, LEEF, or custom
3. **Configure collection** — Set up forwarder with correct sourcetype
   ```bash
   # Splunk: inputs.conf
   [monitor:///var/log/firewall/*.log]
   sourcetype = custom:firewall:paloalto
   index = firewall
   ```
4. **Build parsing** — Create props.conf/transforms.conf or ingest pipeline
5. **Map to data model** — Align extracted fields with CIM/ECS
6. **Validate** — Run data quality queries to confirm parsing accuracy
   ```spl
   index=firewall sourcetype="custom:firewall:paloalto"
   | stats count(src) AS src_parsed, count(dest) AS dest_parsed, count(action) AS action_parsed
   | eval parse_rate = round(src_parsed / dest_parsed * 100, 2)
   ```
7. **Enable accelerated data model** — Confirm CIM compliance
8. **Document** — Record source metadata in runbook

## Detection

```yaml
title: Log Source Onboarding Detection
id: e0f8b23c-3de4-4252-80b5-0cbff5f131b9
status: experimental
description: Detects suspicious activity related to performing log source onboarding techniques in blue team context
logsource:
  category: process_creation
  product: windows
detection:
  selection:
    CommandLine: "*performing*log*"
  condition: selection
level: medium
tags:
  - attack.t1070
  - attack.defense_evasion
falsepositives:
  - Security team running authorized detection validation tools
```

## Verification

| Check | Method |
|---|---|
| Events ingested | `index=<idx> sourcetype=<st> earliest=-1h` returns data |
| Fields extracted | Key fields (src, dest, action) populated >95% |
| CIM compliant | Data model acceleration shows source in Splunk CIM |
| No parsing errors | `index=_internal "WARN" sourcetype=<st>` returns zero |
| Volume expected | EPS matches estimated volume from source |

## References

- [Splunk CIM Documentation](https://docs.splunk.com/Documentation/CIM)
- [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current)
- [Splunk Add-on Builder](https://docs.splunk.com/Documentation/AddonBuilder)
- [OCSF Schema](https://schema.ocsf.io/)
