<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
<!-- NERF is a trademark of defconxt. -->

# Testing & Optimization Reference

Deep reference for detection validation, false positive tuning, and SIEM query performance.
Parent skill: [../SKILL.md](../SKILL.md)

---

## Atomic Red Team Integration

### Overview

Atomic Red Team provides small, portable tests mapped to MITRE ATT&CK techniques. Use them to validate that detection rules fire on real adversary behavior.

### Validation Workflow

```
1. Identify technique under test (e.g., T1003.001 — LSASS Memory)
2. Select atomic test(s) for that technique
3. Clear SIEM alert queue for clean baseline
4. Execute atomic on test endpoint
5. Wait for log ingestion (observe latency)
6. Verify detection rule triggered
7. Record result: TP / FN / partial
8. If FN → analyze why (missing data source, rule logic, timing)
```

### Running Atomics

```powershell
# PowerShell — Invoke-AtomicRedTeam
Install-Module -Name invoke-atomicredteam -Scope CurrentUser
Import-Module invoke-atomicredteam

# List tests for a technique
Invoke-AtomicTest T1003.001 -ShowDetailsBrief

# Execute specific test
Invoke-AtomicTest T1003.001 -TestNumbers 1 -GetPrereqs
Invoke-AtomicTest T1003.001 -TestNumbers 1

# Cleanup after test
Invoke-AtomicTest T1003.001 -TestNumbers 1 -Cleanup
```

```bash
# Linux — using atomic-operator (Python)
pip install atomic-operator
atomic-operator run --technique T1059.004 --test-number 1

# Or manually via YAML
curl -sL https://raw.githubusercontent.com/redcanaryco/atomic-red-team/master/atomics/T1059.004/T1059.004.yaml
```

### Test Result Tracking

| Technique | Test | Expected Alert | Result | Notes |
|-----------|------|---------------|--------|-------|
| T1003.001 | Mimikatz LSASS dump | sigma-lsass-001 | TP ✓ | Fired in 45s |
| T1059.001 | Encoded PowerShell | sigma-ps-encoded-001 | FN ✗ | Rule checks -enc, test used -e |
| T1053.005 | Scheduled task via schtasks | sigma-schtasks-001 | TP ✓ | Fired in 12s |

---

## False Positive Investigation Workflow

### Triage Process

```
Alert fires → Is this a true positive?
    ├── YES → Escalate, investigate
    └── NO → False Positive
              ├── Why did it fire?
              │   ├── Legitimate tool matches signature
              │   ├── Rule logic too broad
              │   └── Environmental noise (backup jobs, admin scripts)
              ├── How often does this FP occur?
              │   ├── Rare (< 1/week) → Add exclusion comment, keep rule
              │   ├── Moderate (daily) → Tune rule with specific filter
              │   └── Excessive (> 10/day) → Redesign detection logic
              └── Document the FP pattern for future tuning
```

### Tuning Strategies

**1. Allowlist by process path (most common):**
```yaml
detection:
  selection:
    Image|endswith: '\certutil.exe'
    CommandLine|contains: '-urlcache'
  filter_legitimate:
    ParentImage:
      - 'C:\Program Files\SCCM\ccmexec.exe'
      - 'C:\ProgramData\ManageEngine\*'
  condition: selection and not filter_legitimate
```

**2. Threshold adjustment:**
```yaml
# Before: any failed login
detection:
  selection:
    EventID: 4625
  condition: selection

# After: 5+ failures from same source in 10 minutes
detection:
  selection:
    EventID: 4625
  timeframe: 10m
  condition: selection | count(TargetUserName) by SourceIP > 5
```

**3. Time-based suppression:**
```yaml
# Suppress during known maintenance windows
detection:
  selection:
    Image|endswith: '\psexec.exe'
  filter_maintenance:
    # Saturday 02:00-06:00 UTC — patching window
    UtcTime|re: '.*T0[2-5]:.*'
  condition: selection and not filter_maintenance
```

### FP Tracking Table

| Rule ID | FP Source | Frequency | Action | Status |
|---------|-----------|-----------|--------|--------|
| sigma-certutil-001 | SCCM downloads | 12/day | Added ParentImage filter | Resolved |
| sigma-ps-encoded-001 | Monitoring agent | 3/day | Added CommandLine exclusion | Resolved |
| sigma-schtasks-001 | Backup software | 1/week | Documented, no change | Accepted |

---

## Query Performance Optimization

### Performance Profiling

Before deploying a detection rule to production, profile its query cost:

| SIEM | Profiling Method |
|------|-----------------|
| Splunk | `| rest /services/search/jobs/{sid}` → check `scanCount`, `resultCount`, `runDuration` |
| Elastic | `_search?profile=true` → analyze shard-level timing |
| Sentinel | `workspace("").Usage` → check query execution stats in Log Analytics |
| Chronicle | Detection dashboard → rule execution time metrics |

### Optimization Techniques

**1. Filter early (most impactful):**
```spl
# Bad — scans all events, then filters
index=* sourcetype=WinEventLog EventCode=4688
| where Image LIKE "%mimikatz%"

# Good — narrow index and sourcetype first
index=wineventlog sourcetype=WinEventLog:Security EventCode=4688 Image="*mimikatz*"
```

**2. Avoid unbounded wildcards:**
```spl
# Bad — leading wildcard forces full scan
index=wineventlog CommandLine="*-encodedcommand*"

# Better — use specific field + endswith pattern
index=wineventlog EventCode=4688 Image="*\\powershell.exe" CommandLine="*-enc*"
```

**3. Use indexed fields for filtering:**
```yaml
# Sigma — prefer fields the SIEM indexes natively
detection:
  selection:
    EventID: 1          # Indexed field — fast
    Image|endswith: '\rundll32.exe'  # Indexed (Sysmon)
  filter:
    CommandLine|contains: 'javascript:'  # Full-text — slower, but narrowed by selection
  condition: selection and not filter
```

**4. Limit time range:**
```spl
# Schedule searches with explicit earliest/latest
earliest=-15m latest=now
index=wineventlog EventCode=4688 Image="*\\certutil.exe"
```

### Performance Budget

| Metric | Threshold | Action if exceeded |
|--------|-----------|-------------------|
| Query execution time | < 30s (scheduled), < 5s (real-time) | Optimize or split rule |
| Events scanned | < 10M per execution | Narrow index/sourcetype |
| Memory usage | < 500 MB per search | Remove unnecessary field extractions |
| Concurrent searches | < 80% of search head capacity | Stagger schedules |

---

## Detection Efficacy Measurement

### Key Metrics

| Metric | Formula | Target |
|--------|---------|--------|
| True Positive Rate | TP / (TP + FN) | > 90% for critical techniques |
| False Positive Rate | FP / (FP + TN) | < 5% per rule |
| Mean Time to Detect (MTTD) | Alert timestamp − event timestamp | < 5 min |
| Alert-to-Triage ratio | Alerts reviewed / Total alerts | > 80% |
| Detection coverage | Techniques with score ≥ 3 / Total techniques | > 60% |

### Efficacy Testing Cadence

| Frequency | Activity |
|-----------|----------|
| Per rule deployment | Atomic Red Team validation |
| Weekly | FP rate review, top noisy rules |
| Monthly | Coverage heatmap update, new technique mapping |
| Quarterly | Full adversary emulation (Caldera/Vectr), gap analysis |
| Annually | Purple team engagement with external partner |
