<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: building-threat-detection-analytics
description: >-
  Build detection analytics pipelines combining rule-based (Sigma, KQL, SPL),
  statistical, and behavioral approaches to identify threats across normalized
  security telemetry at scale.
domain: cybersecurity
subdomain: security-data-engineering
tags:
  - detection-engineering
  - sigma
  - kql
  - spl
  - analytics
  - behavioral
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1059", "T1078", "T1190"]
  frameworks: ["NIST CSF 2.0", "MITRE ATT&CK"]
  tools: ["python3", "sigma-cli", "duckdb"]
---

# Building Threat Detection Analytics

## Overview

Detection analytics transform normalized security data into actionable alerts.
This skill covers building rule-based detections (Sigma, KQL, SPL), statistical
baselines for anomaly detection, entity-based behavioral analytics, and
detection pipeline CI/CD for continuous validation.

## Prerequisites

| Requirement | Details |
|---|---|
| Normalized security event data (OCSF or ECS schema) | Required |
| Detection rule authoring tools (sigma-cli, KQL editor) | Required |
| Analytics query engine (Spark SQL, DuckDB, SIEM) | Required |
| MITRE ATT&CK knowledge for coverage mapping | Required |

## Key Concepts

- **Detection-as-code**: Version-controlled rules with automated testing
- **Threshold tuning**: Balance detection rate against false positive volume
- **Correlation rules**: Multi-event patterns across time windows
- **Behavioral baseline**: Statistical profiles for anomaly scoring

## Workflow

### Step 1: Define Detection Hypothesis

```python
from dataclasses import dataclass

@dataclass
class DetectionHypothesis:
    """Structured detection hypothesis."""
    technique_id: str  # MITRE ATT&CK ID
    data_source: str   # Required telemetry
    observable: str     # What behavior to detect
    logic: str          # Detection approach
    threshold: int | None = None
```

### Step 2: Author Sigma Rule

```yaml
title: Anomalous Authentication Volume from Single Source
id: a1b2c3d4-e5f6-7890-abcd-ef1234567890
status: experimental
description: Detects high-volume authentication attempts from single IP
logsource:
  category: authentication
  product: generic
detection:
  selection:
    src_endpoint.ip|count: ">100"
  timeframe: 1h
  condition: selection
falsepositives:
  - Vulnerability scanners during authorized testing
  - Batch service accounts with legitimate high-volume auth
level: high
tags:
  - attack.t1110
  - attack.credential_access
```

### Step 3: Build Statistical Baseline

```sql
-- Compute 7-day baseline for auth volume per source
WITH baseline AS (
    SELECT
        src_endpoint.ip AS source_ip,
        DATE_TRUNC('hour', time_dt) AS hour,
        COUNT(*) AS event_count
    FROM authentication_events
    WHERE time_dt >= CURRENT_TIMESTAMP - INTERVAL '7 days'
    GROUP BY source_ip, hour
)
SELECT
    source_ip,
    AVG(event_count) AS mean_count,
    STDDEV(event_count) AS stddev_count
FROM baseline
GROUP BY source_ip;
```

### Step 4: Validate Detections

```bash
node agent.js test --rule auth_anomaly.yml --dataset test_events.jsonl
node agent.js coverage --rules-dir ./rules/ --attack-layer enterprise
```

## References

- [Sigma HQ](https://github.com/SigmaHQ/sigma)
- [MITRE ATT&CK](https://attack.mitre.org/)
- [Palantir ADS Framework](https://github.com/palantir/alerting-detection-strategy-framework)
