<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
<!-- NERF is a trademark of defconxt. -->
---
name: performing-security-telemetry-optimization
description: >-
  Optimize security telemetry collection by identifying redundant data,
  right-sizing log volumes, tuning collection agents, and implementing
  intelligent filtering to maximize detection value while minimizing
  storage and processing costs.
domain: cybersecurity
subdomain: security-data-engineering
tags:
  - telemetry
  - optimization
  - log-volume
  - cost-reduction
  - agent-tuning
  - data-quality
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1562.001", "T1070"]
  frameworks: ["NIST 800-92", "CIS Controls v8"]
  tools: ["python3", "elasticsearch", "splunk", "cribl"]
---

# Performing Security Telemetry Optimization

## Overview

Security telemetry optimization ensures that every byte of collected log data
contributes to detection, investigation, or compliance value. Unoptimized
telemetry leads to excessive storage costs, slow queries, and analyst fatigue
from noise.

## Prerequisites

| Requirement | Purpose |
|---|---|
| Telemetry volume metrics | Current ingest rates by source |
| Detection rule inventory | Map sources to detection coverage |
| Cost baseline | Current storage and processing costs |
| Agent management platform | Centralized agent configuration |

## Key Concepts

- **Signal-to-noise ratio**: Proportion of events contributing to detections
- **Detection-linked telemetry**: Data directly referenced by active detection rules
- **Noise sources**: High-volume, low-value event types consuming resources
- **Edge filtering**: Drop or summarize events at the agent before transmission
- **Routing**: Direct events to appropriate tiers based on value

## Workflow

### Step 1: Analyze Telemetry Volume

```python
from dataclasses import dataclass

@dataclass
class TelemetrySource:
    """Telemetry source with volume and value metrics."""
    name: str
    daily_gb: float
    event_types: int
    detection_linked: bool
    compliance_required: bool
    cost_monthly: float

def analyze_volumes(sources: list[TelemetrySource]) -> dict:
    """Analyze telemetry volumes and identify optimization targets."""
    total_gb = sum(s.daily_gb for s in sources)
    unlinked = [s for s in sources if not s.detection_linked and not s.compliance_required]
    return {
        "total_daily_gb": total_gb,
        "optimization_candidates": [s.name for s in unlinked],
        "potential_savings_gb": sum(s.daily_gb for s in unlinked),
    }
```

### Step 2: Score Telemetry Value

```python
def score_telemetry_value(source: dict) -> float:
    """Score telemetry source value 0.0-1.0."""
    score = 0.0
    if source.get("detection_rules_count", 0) > 0:
        score += 0.4
    if source.get("compliance_required"):
        score += 0.3
    if source.get("investigation_usage_30d", 0) > 0:
        score += 0.2
    if source.get("unique_fields_used", 0) > 5:
        score += 0.1
    return min(1.0, score)
```

### Step 3: Implement Filtering Rules

```yaml
# Cribl Stream pipeline — filter noisy Windows events
routes:
  - id: windows_noise_reduction
    filter: sourcetype=='WinEventLog:Security'
    pipeline:
      - id: drop_noise
        filter: EventCode IN [4688, 5156, 5158] AND NOT _raw MATCH 'cmd.exe|powershell'
        action: drop
      - id: summarize_logon
        filter: EventCode == 4624 AND LogonType IN [3, 10]
        action: aggregate
        group_by: [src_ip, user]
        interval: 300
```

### Step 4: Validate Optimization Impact

```bash
node agent.js analyze --sources all --period 30d
node agent.js recommend --target-reduction 30 --protect-detections
node agent.js validate --before-snapshot baseline.json --after-snapshot optimized.json
```

## Detection

```yaml
title: Security Telemetry Optimization Detection
id: 9fd9dce6-baa5-4568-8da0-7626f60d8375
status: experimental
description: Detects suspicious activity related to performing security telemetry optimization techniques in security data engineering context
logsource:
  category: process_creation
  product: linux
detection:
  selection:
    CommandLine: "*performing*security*"
  condition: selection
level: medium
tags:
  - attack.t1562.001
  - attack.t1070
  - attack.collection
falsepositives:
  - Data pipeline ETL processes during scheduled batch windows
```

## Verification

| Check | Method |
|---|---|
| No detection coverage lost | All detection rules still fire on test data |
| Volume reduced to target | Ingest rate decreased by expected percentage |
| Compliance data intact | Required log sources unaffected |
| Cost savings realized | Monthly bill decreased as projected |

## References

- [Cribl Stream Documentation](https://docs.cribl.io/)
- [NIST SP 800-92 — Log Management](https://csrc.nist.gov/publications/detail/sp/800-92/final)
- [CIS Controls v8 — Control 8](https://www.cisecurity.org/controls/audit-log-management)
