<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: performing-log-source-health-monitoring
description: >-
  Continuous monitoring of SIEM log source health including ingestion rate
  tracking, data gap detection, parser error identification, latency
  measurement, source onboarding validation, and capacity planning across
  Splunk, Elastic, and Microsoft Sentinel.
domain: cybersecurity
subdomain: soc-operations
tags:
  - log-source
  - soc
  - siem
  - ingestion
  - health-monitoring
  - data-quality
  - splunk
  - elastic
  - sentinel
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1562"]
---

# Performing Log Source Health Monitoring

## Overview

Detection capability is only as good as the data feeding the SIEM. Silent
log source failures create blind spots that attackers exploit. This skill
covers proactive monitoring of log ingestion health, gap detection, latency
tracking, and capacity planning to ensure continuous detection coverage.

## Prerequisites

| Requirement | Purpose |
|---|---|
| SIEM admin access | Ingestion metrics and source management |
| Infrastructure monitoring (Grafana/Datadog) | Complementary health data |
| Log source inventory | Baseline of expected sources |
| Network monitoring | Transport layer health |

## Key Concepts

### Health Monitoring Framework

```
LOG SOURCE HEALTH DIMENSIONS:
├── Availability — Is the source sending data?
│   Check: event count > 0 in expected interval
├── Volume — Is the volume within normal range?
│   Check: count within 2 standard deviations of baseline
├── Latency — How fresh is the data?
│   Check: max(_time) - now() < acceptable threshold
├── Parsing — Are events being parsed correctly?
│   Check: parse error rate < 1%
├── Completeness — Are all expected event types present?
│   Check: all expected sourcetypes/indices populated
└── Quality — Do field extractions work?
    Check: key fields (src_ip, user, action) populated > 95%
```

### Splunk Log Source Health Queries

```spl
-- Log source availability dashboard (last 24h)
| tstats count where index=* earliest=-24h by index, sourcetype, _time span=1h
| xyseries _time, sourcetype, count
| foreach * [eval "<<FIELD>>"=if('<<FIELD>>'="", 0, '<<FIELD>>')]

-- Sources with zero events in last 4 hours (gap detection)
| tstats latest(_time) as last_event count where index=* by sourcetype
| eval hours_since=round((now()-last_event)/3600,1)
| where hours_since > 4
| sort -hours_since
| table sourcetype, last_event, hours_since, count

-- Ingestion latency per sourcetype
| tstats avg(_indextime-_time) as avg_latency,
    max(_indextime-_time) as max_latency
    where index=* earliest=-1h by sourcetype
| eval avg_latency_sec=round(avg_latency,0),
    max_latency_sec=round(max_latency,0)
| where avg_latency_sec > 300
| sort -avg_latency_sec

-- Volume anomaly detection (vs 7-day baseline)
| tstats count where index=* earliest=-1d by sourcetype
| appendcols [
    | tstats count where index=* earliest=-8d latest=-1d by sourcetype
    | eval baseline_daily=count/7
    | fields sourcetype, baseline_daily]
| eval deviation=round(abs(count-baseline_daily)/baseline_daily*100,1)
| where deviation > 50
| sort -deviation
```

### Elastic Log Source Health

```
// Index health overview
GET _cat/indices?v&s=docs.count:desc&h=index,health,docs.count,store.size

// Data gap detection (indices with no data in 4h)
GET _cat/indices?v&h=index,docs.count
// Cross-reference with expected index list

// Ingestion rate per data stream
GET _data_stream/*/_stats
```

### Sentinel KQL Log Source Health

```kql
// Log source heartbeat (last 24h by table)
Usage
| where TimeGenerated > ago(24h)
| summarize LastEvent=max(TimeGenerated), EventCount=sum(Quantity)
    by DataType
| extend HoursSinceLastEvent=datetime_diff("hour", now(), LastEvent)
| where HoursSinceLastEvent > 4
| sort by HoursSinceLastEvent desc

// Ingestion volume anomaly
Usage
| where TimeGenerated > ago(8d)
| summarize DailyGB=sum(Quantity)/1024 by DataType, Day=startofday(TimeGenerated)
| summarize AvgDaily=avg(DailyGB), StdDev=stdev(DailyGB), Latest=max(DailyGB)
    by DataType
| extend Anomaly=iff(abs(Latest-AvgDaily) > 2*StdDev, "YES", "no")
| where Anomaly == "YES"
```

### Alert Thresholds

```
MONITORING THRESHOLDS:
  Source silent > 4 hours     → WARNING alert
  Source silent > 8 hours     → CRITICAL alert
  Volume deviation > 50%      → WARNING alert
  Volume deviation > 80%      → CRITICAL alert
  Ingestion latency > 5 min   → WARNING alert
  Ingestion latency > 30 min  → CRITICAL alert
  Parse error rate > 1%       → WARNING alert
  Parse error rate > 5%       → CRITICAL alert
```

## Workflow

1. **Inventory** — Catalog all expected log sources with normal behavior
2. **Baseline** — Establish volume and latency baselines per source
3. **Monitor** — Run health checks on schedule (hourly critical, daily full)
4. **Alert** — Trigger notifications for threshold breaches
5. **Triage** — Investigate gaps: source issue, network, parser, or capacity
6. **Remediate** — Fix source, update parser, or expand capacity
7. **Report** — Weekly health report to SOC and infrastructure teams

## Detection

```yaml
title: Log Source Health Monitoring Detection
id: 2d8ca0b2-7790-4c31-9f66-76275c1b4f27
status: experimental
description: Detects suspicious activity related to performing log source health monitoring techniques in soc operations context
logsource:
  category: process_creation
  product: windows
detection:
  selection:
    CommandLine: "*performing*log*"
  condition: selection
level: medium
tags:
  - attack.t1562
  - attack.defense_evasion
falsepositives:
  - SOAR platform executing automated response playbooks
```

## Verification

| Check | Method |
|---|---|
| All sources in inventory monitored | Source count matches expected list |
| Gap detection functional | Simulate source silence, alert fires |
| Latency alerts working | Verify threshold breach triggers notification |
| Volume baselines current | Baselines refreshed within last 30 days |
| Health dashboard accessible | SOC team can view real-time status |
