<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: validating-siem-detection-rules
description: >-
  Test and validate SIEM detection rules against known-bad and known-good data.
  Covers Sigma rule validation, SPL/KQL/EQL testing, false positive analysis,
  and automated regression testing for detection-as-code pipelines.
domain: cybersecurity
subdomain: purple-team
tags:
  - siem
  - sigma
  - detection-validation
  - splunk
  - elastic
  - sentinel
  - detection-as-code
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: []
---

# Validating SIEM Detection Rules

## Overview

Detection rules are only effective if they fire on real attacks and stay silent on
benign activity. This skill covers the full validation lifecycle: unit testing rules
against sample data, integration testing against live SIEM instances, measuring
false positive rates, and building CI/CD pipelines for detection-as-code.

## Prerequisites

| Requirement | Install |
|---|---|
| sigma-cli | `pip install sigma-cli` |
| pySigma backends | `pip install pysigma-backend-splunk pysigma-backend-elasticsearch` |
| Splunk SDK | `pip install splunk-sdk` |
| Elasticsearch client | `pip install elasticsearch` |
| Python 3.10+ | For agent tooling |

## Key Concepts

### Validation Levels

| Level | Description | Automation |
|---|---|---|
| Syntax check | Rule parses without errors | `sigma check rule.yml` |
| Conversion test | Rule converts to target query | `sigma convert -t splunk rule.yml` |
| Unit test | Query matches known-bad sample events | Custom test harness |
| Integration test | Rule fires in live SIEM against replayed data | SIEM API |
| Regression test | Rule still fires after SIEM/schema updates | CI pipeline |

### Test Data Format

```json
{
  "test_name": "T1003.001 - LSASS Access via Mimikatz",
  "rule_file": "rules/credential_access/lsass_access.yml",
  "should_match": [
    {
      "TargetImage": "C:\\Windows\\System32\\lsass.exe",
      "GrantedAccess": "0x1010",
      "SourceImage": "C:\\temp\\mimikatz.exe"
    }
  ],
  "should_not_match": [
    {
      "TargetImage": "C:\\Windows\\System32\\lsass.exe",
      "GrantedAccess": "0x1010",
      "SourceImage": "C:\\Program Files\\Windows Defender\\MsMpEng.exe"
    }
  ]
}
```

## Workflow

### 1. Syntax Validation

```bash
# Validate single rule
sigma check rules/credential_access/lsass_access.yml

# Batch validate directory
find rules/ -name '*.yml' -exec sigma check {} \; 2>&1 | grep -E 'ERROR|WARNING'
```

### 2. Conversion Testing

```bash
# Convert and verify output is non-empty
sigma convert -t splunk -p splunk_cim rules/credential_access/lsass_access.yml

# Test all backends
for backend in splunk elasticsearch microsoft365defender; do
  echo "=== $backend ==="
  sigma convert -t "$backend" rules/credential_access/lsass_access.yml
done
```

### 3. Unit Testing Against Sample Data

```bash
# Run unit tests
node scripts/agent.js test-rule \
  --rule rules/credential_access/lsass_access.yml \
  --test-data tests/lsass_access_test.json

# Batch test all rules with test data
node scripts/agent.js batch-test --rules-dir rules/ --tests-dir tests/
```

### 4. Integration Testing

```bash
# Test rule against live Splunk
node scripts/agent.js integration-test \
  --rule rules/credential_access/lsass_access.yml \
  --target splunk \
  --host https://splunk:8089 \
  --index test_events

# Test against Elasticsearch
node scripts/agent.js integration-test \
  --rule rules/credential_access/lsass_access.yml \
  --target elasticsearch \
  --host http://elastic:9200 \
  --index test-events
```

## Verification

| Check | Method |
|---|---|
| Syntax valid | `sigma check` exits 0 on all rules |
| Converts cleanly | All target backends produce non-empty output |
| True positives | Rule matches all known-bad samples |
| True negatives | Rule does not match known-good samples |
| FP rate | < 5% false positive rate over 7-day window |

## References

- [Sigma Specification](https://sigmahq.io/docs/basics/rules.html)
- [Splunk REST API](https://docs.splunk.com/Documentation/Splunk/latest/RESTREF)
- [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html)
