<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: performing-log-source-onboarding
description: >-
  Onboard new log sources into SIEM platforms including data ingestion
  configuration, field extraction, CIM normalization, index strategy,
  parsing validation, and data quality assurance across Splunk, Elastic,
  and Microsoft Sentinel.
domain: cybersecurity
subdomain: soc-operations
tags:
  - log-onboarding
  - siem
  - splunk
  - elastic
  - sentinel
  - data-ingestion
  - cim
  - parsing
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1070"]
---

# Performing Log Source Onboarding

## Overview

Log source onboarding brings new data into the SIEM for detection and
investigation. Proper onboarding ensures reliable parsing, CIM-compliant
field mapping, appropriate retention, and validation that detections fire
against the new data. Poor onboarding causes blind spots and broken rules.

## Prerequisites

| Requirement | Purpose |
|---|---|
| SIEM admin access | Index and sourcetype configuration |
| Network access to log source | Data transport (syslog, API, agent) |
| Log source documentation | Field definitions and formats |
| CIM mapping reference | Field normalization targets |

## Key Concepts

### Onboarding Checklist

```
LOG SOURCE ONBOARDING:
├── 1. Discovery
│   ├── What does the source generate? (auth, network, endpoint)
│   ├── Volume estimate (GB/day, EPS)
│   ├── Format (JSON, syslog, CEF, CSV, XML)
│   └── Transport (syslog, HEC, Filebeat, agent)
├── 2. Ingestion
│   ├── Configure collection (forwarder, API poller, syslog receiver)
│   ├── Set index/data stream
│   ├── Apply sourcetype/pipeline
│   └── Validate data arrival
├── 3. Parsing
│   ├── Field extraction (props.conf / ingest pipeline)
│   ├── Timestamp recognition
│   ├── Event breaking (LINE_BREAKER / multiline)
│   └── Validation: fields populate correctly
├── 4. Normalization
│   ├── Map to CIM/ECS fields
│   ├── Tag for data model acceleration
│   └── Verify tstats/data model queries work
├── 5. Validation
│   ├── Sample queries return expected results
│   ├── Existing detections fire against new data
│   ├── Dashboards populate
│   └── Data quality checks pass
└── 6. Documentation
    ├── Update CMDB with log source entry
    ├── Document field mappings
    └── Note retention and volume
```

### Splunk Onboarding

```conf
# inputs.conf — syslog collection
[udp://514]
sourcetype = palo_alto_firewall
index = network
connection_host = dns

# props.conf — field extraction and timestamp
[palo_alto_firewall]
TIME_FORMAT = %Y/%m/%d %H:%M:%S
TIME_PREFIX = ^
MAX_TIMESTAMP_LOOKAHEAD = 26
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TRANSFORMS-sourcetype = pan_sourcetype_lookup

# transforms.conf — field extraction
[pan_sourcetype_lookup]
REGEX = ^[^,]+,\d+/\d+/\d+ \d+:\d+:\d+,\d+,(\w+),
FORMAT = sourcetype::pan:$1
DEST_KEY = MetaData:Sourcetype

# tags.conf — CIM mapping
[sourcetype=pan:traffic]
network = enabled
communicate = enabled
```

```spl
-- Validate data ingestion
index=network sourcetype="palo_alto_firewall" earliest=-1h
| stats count, earliest(_time) as first, latest(_time) as last
| eval lag_seconds = now() - last

-- Validate CIM field mapping
| datamodel Network_Traffic All_Traffic search
| where sourcetype="pan:traffic"
| stats count by src, dest, dest_port, action, bytes_in, bytes_out
| head 10

-- Check for parsing errors
index=_internal sourcetype=splunkd component=LineBreakingProcessor
| where log_level="WARN" OR log_level="ERROR"
| stats count by data_source
```

### Elastic Onboarding

```json
{
  "description": "Palo Alto firewall log pipeline",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "%{TIMESTAMP_ISO8601:timestamp},%{DATA:serial},%{WORD:type}"
        ]
      }
    },
    {
      "date": {
        "field": "timestamp",
        "formats": ["yyyy/MM/dd HH:mm:ss"]
      }
    },
    {
      "rename": {
        "field": "src_ip",
        "target_field": "source.ip"
      }
    }
  ]
}
```

### Sentinel KQL Validation

```kql
// Verify data ingestion
CommonSecurityLog
| where DeviceVendor == "Palo Alto Networks"
| where TimeGenerated > ago(1h)
| summarize count(), min(TimeGenerated), max(TimeGenerated)

// Check field population
CommonSecurityLog
| where DeviceVendor == "Palo Alto Networks"
| summarize
    SourceIPPct = countif(isnotempty(SourceIP)) * 100.0 / count(),
    DestIPPct = countif(isnotempty(DestinationIP)) * 100.0 / count(),
    ActionPct = countif(isnotempty(DeviceAction)) * 100.0 / count()
```

## Workflow

1. **Discover** — Identify log source, format, volume, and transport
2. **Configure** — Set up collection, index, and sourcetype
3. **Parse** — Build field extraction and timestamp recognition
4. **Normalize** — Map fields to CIM/ECS schema
5. **Validate** — Run sample queries, check data model population
6. **Test** — Verify existing detections work with new data
7. **Document** — Record source metadata and field mappings
8. **Monitor** — Set up ingestion health alerts

## Detection

```yaml
title: Log Source Onboarding Detection
id: b93be52a-cee2-4f82-88e4-ca6b17de9ad1
status: experimental
description: Detects suspicious activity related to performing log source onboarding techniques in soc operations context
logsource:
  category: process_creation
  product: windows
detection:
  selection:
    CommandLine: "*performing*log*"
  condition: selection
level: medium
tags:
  - attack.t1070
  - attack.defense_evasion
falsepositives:
  - SOAR platform executing automated response playbooks
```

## Verification

| Check | Method |
|---|---|
| Data arriving in SIEM | Count query last 1h |
| Timestamp parsed correctly | Compare event time vs. ingestion time |
| Fields extracted | Sample search with key fields |
| CIM/ECS mapped | Data model or index pattern query |
| Volume as expected | Compare actual vs. estimated GB/day |
