<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
<!-- NERF is a trademark of defconxt. -->

---
technique: performing-detection-signal-analysis
subdomain: security-data-engineering
title: "Performing Detection Signal Analysis"
description: "Techniques and methodologies for performing detection signal analysis"
tags:
  - detection-engineering
  - signal-analysis
  - alert-tuning
  - noise-reduction
  - SOC-optimization
metadata:
  mitre-attack: []
---

# Performing Detection Signal Analysis

> Analyze detection signal quality, fidelity, and coverage to optimize alert pipelines and reduce noise in security monitoring environments.

## Overview

Detection signal analysis evaluates the quality and effectiveness of security alerts by measuring true positive rates, false positive ratios, detection latency, and coverage gaps. This technique provides methods for systematic signal evaluation and optimization.

## Core Concepts

- **Signal-to-noise ratio (SNR)**: Measure useful alerts vs. total alert volume
- **Detection fidelity scoring**: Rate each detection rule on accuracy, coverage, and timeliness
- **Coverage mapping**: Map detections to MITRE ATT&CK matrix for gap identification
- **Alert fatigue analysis**: Identify rules contributing most to analyst burnout
- **Bayesian updating**: Refine detection confidence based on analyst feedback loops

## Workflow

1. Export detection rule inventory and alert telemetry
2. Calculate per-rule true positive, false positive, and false negative rates
3. Score detection fidelity using weighted multi-factor model
4. Map detection coverage against ATT&CK techniques
5. Identify high-noise, low-value rules for tuning or retirement
6. Analyze detection latency distribution per rule category
7. Correlate alert volume with analyst investigation outcomes
8. Generate signal quality report with prioritized recommendations
9. Implement tuning changes and measure delta
10. Establish continuous signal quality monitoring dashboard

## Implementation

```python
from dataclasses import dataclass
from typing import Optional


@dataclass
class DetectionSignal:
    rule_id: str
    rule_name: str
    true_positives: int = 0
    false_positives: int = 0
    false_negatives: int = 0
    avg_latency_ms: float = 0.0
    attack_techniques: list[str] | None = None

    @property
    def precision(self) -> float:
        total = self.true_positives + self.false_positives
        return self.true_positives / total if total > 0 else 0.0

    @property
    def recall(self) -> float:
        total = self.true_positives + self.false_negatives
        return self.true_positives / total if total > 0 else 0.0

    @property
    def f1_score(self) -> float:
        p, r = self.precision, self.recall
        return 2 * p * r / (p + r) if (p + r) > 0 else 0.0

    @property
    def fidelity_score(self) -> float:
        return round(self.f1_score * 0.6 + (1 - min(self.avg_latency_ms / 10000, 1.0)) * 0.4, 3)


class SignalAnalyzer:
    """Analyze detection signal quality across rule inventory."""

    def __init__(self) -> None:
        self.signals: list[DetectionSignal] = []

    def add_signal(self, signal: DetectionSignal) -> None:
        self.signals.append(signal)

    def get_low_fidelity(self, threshold: float = 0.3) -> list[DetectionSignal]:
        return [s for s in self.signals if s.fidelity_score < threshold]

    def coverage_gaps(self, attack_matrix: list[str]) -> list[str]:
        covered = set()
        for s in self.signals:
            if s.attack_techniques:
                covered.update(s.attack_techniques)
        return [t for t in attack_matrix if t not in covered]

    def noise_report(self) -> list[dict]:
        return sorted(
            [{"rule": s.rule_name, "fp_rate": 1 - s.precision, "volume": s.true_positives + s.false_positives} for s in self.signals],
            key=lambda x: x["fp_rate"],
            reverse=True,
        )
```

## Best Practices

- Measure signal quality continuously, not just during tuning sprints
- Track analyst feedback on alert usefulness to build ground truth datasets
- Retire rules with sustained precision below 10%—they actively harm SOC performance
- Use ATT&CK coverage heatmaps to prioritize new detection development
- Account for detection latency in fidelity scoring—late detections have reduced value
- Separate signal analysis by environment (prod vs. dev) to avoid skewed metrics

## Signal Quality Metrics

| Metric | Formula | Target | Action Below Target |
|---|---|---|---|
| Precision (TP rate) | TP / (TP + FP) | >0.80 | Tune or retire rule |
| Recall (coverage) | TP / (TP + FN) | >0.70 | Improve detection logic |
| F1 Score | 2 × (P × R) / (P + R) | >0.75 | Balanced tuning needed |
| Alert latency (p95) | Time detection fires vs. event | <5 min for critical | Optimize pipeline |
| Analyst closure rate | Closed alerts / total alerts | >0.90 | Rule retirement candidates |
| ATT&CK coverage | Techniques with detections / total | Baseline → track improvement | Detection engineering backlog |

## Fidelity Classification

| Classification | Precision | Action |
|---|---|---|
| High fidelity | >0.90 | Maintain; use as training data |
| Medium fidelity | 0.60–0.90 | Tune FP conditions, add context |
| Low fidelity | 0.30–0.60 | Aggressive tuning or downgrade to informational |
| Noise | <0.30 | Retire — actively harms SOC |

## Related Techniques

- [Building Security Event Correlation Engine](../building-security-event-correlation-engine/SKILL.md)
- [Building Security Data API Layer](../building-security-data-api-layer/SKILL.md)
