<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
<!-- NERF is a trademark of defconxt. -->
---
name: implementing-security-observability
description: >-
  Implement security observability pipelines that combine metrics, logs, and
  traces to detect threats, measure security posture, and enable rapid
  incident investigation across distributed systems.
domain: cybersecurity
subdomain: devsecops
tags:
  - security-observability
  - opentelemetry
  - siem-integration
  - security-metrics
  - distributed-tracing
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1562", "T1070"]
  cwe: ["CWE-778", "CWE-223"]
  tools: ["opentelemetry", "prometheus", "grafana", "loki", "jaeger"]
---

## Overview

Security observability extends traditional monitoring by correlating metrics,
logs, and distributed traces to detect attacks in real time, measure security
control effectiveness, and accelerate incident investigation. Unlike passive
logging, observability provides queryable, structured telemetry that enables
proactive threat hunting and automated response.

## Prerequisites

- Tools: `opentelemetry-collector`, `prometheus`, `grafana`, `loki`, `jaeger`
- Kubernetes cluster or application deployment infrastructure
- Log aggregation pipeline (Loki, Elasticsearch, or Splunk)
- Alerting infrastructure (Alertmanager, PagerDuty)

## Key Concepts

- **Three pillars**: Metrics (quantitative), Logs (events), Traces (request flow)
- **Security telemetry**: Auth events, access patterns, error rates, latency anomalies
- **Correlation**: Linking trace IDs across services to reconstruct attack paths
- **SLI/SLO for security**: Measurable security service level indicators
- **Cardinality management**: Controlling label dimensions to prevent metric explosion

## Workflow

### Step 1: Deploy OpenTelemetry Collector

```yaml
# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  filelog:
    include: [/var/log/auth.log, /var/log/syslog]
    operators:
      - type: regex_parser
        regex: '(?P<timestamp>\w+ \d+ [\d:]+) (?P<host>\S+) (?P<service>\S+): (?P<message>.*)'

processors:
  batch:
    timeout: 5s
  attributes:
    actions:
      - key: environment
        value: production
        action: upsert

exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch, attributes]
      exporters: [prometheus]
    logs:
      receivers: [otlp, filelog]
      processors: [batch]
      exporters: [loki]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/jaeger]
```

### Step 2: Instrument Security-Relevant Metrics

```python
from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider

meter = metrics.get_meter("security.metrics")

auth_failures = meter.create_counter(
    name="security.auth.failures",
    description="Authentication failure count",
    unit="1",
)

auth_latency = meter.create_histogram(
    name="security.auth.latency",
    description="Authentication request latency",
    unit="ms",
)

active_sessions = meter.create_up_down_counter(
    name="security.sessions.active",
    description="Currently active user sessions",
    unit="1",
)

# Record security events
def on_auth_failure(user: str, reason: str, ip: str) -> None:
    auth_failures.add(1, {
        "user": user,
        "reason": reason,
        "source_ip": ip,
    })
```

### Step 3: Configure Security Alerting Rules

```yaml
# Prometheus alerting rules
groups:
  - name: security_alerts
    rules:
      - alert: HighAuthFailureRate
        expr: rate(security_auth_failures_total[5m]) > 10
        for: 2m
        labels:
          severity: warning
          team: security
        annotations:
          summary: "High authentication failure rate"
          description: "{{ $value }} auth failures/sec over 5m"

      - alert: SuspiciousLatencySpike
        expr: histogram_quantile(0.99, rate(security_auth_latency_bucket[5m])) > 5000
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Auth latency p99 > 5s — possible brute force"

      - alert: AuditLogGap
        expr: absent(up{job="auditd"}) or changes(up{job="auditd"}[1h]) > 2
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Audit logging gap detected — possible T1562"
```

### Step 4: Build Security Dashboards

```bash
# Grafana dashboard provisioning
cat > /etc/grafana/provisioning/dashboards/security.json << 'EOF'
{
  "dashboard": {
    "title": "Security Observability",
    "panels": [
      {
        "title": "Auth Failures by Source IP",
        "type": "timeseries",
        "targets": [{"expr": "sum by (source_ip)(rate(security_auth_failures_total[5m]))"}]
      },
      {
        "title": "Active Sessions",
        "type": "stat",
        "targets": [{"expr": "security_sessions_active"}]
      },
      {
        "title": "Security Events by Type",
        "type": "piechart",
        "targets": [{"expr": "sum by (event_type)(security_events_total)"}]
      }
    ]
  }
}
EOF
```

### Step 5: Implement Distributed Trace Security Analysis

```python
from opentelemetry import trace
from opentelemetry.trace import StatusCode

tracer = trace.get_tracer("security.tracer")

def trace_auth_request(user: str, ip: str) -> None:
    with tracer.start_as_current_span("auth.verify") as span:
        span.set_attribute("user.id", user)
        span.set_attribute("net.peer.ip", ip)
        span.set_attribute("security.event_type", "authentication")

        # Auth logic here
        authenticated = verify_credentials(user)

        if not authenticated:
            span.set_status(StatusCode.ERROR, "Authentication failed")
            span.set_attribute("security.outcome", "failure")
        else:
            span.set_attribute("security.outcome", "success")
```

## Detection Opportunities

| Signal | Source | Description |
|--------|--------|-------------|
| Telemetry pipeline gaps | Prometheus | Missing metrics indicate log tampering (T1070) |
| Auth failure spikes | Metrics | Brute force or credential stuffing |
| Latency anomalies | Traces | Timing attacks or resource exhaustion |
| Trace ID missing | Application logs | Request not properly instrumented |

```yaml
title: Security Telemetry Pipeline Disruption
id: a7b8c9d0-e1f2-3456-0123-456789abcdef
status: experimental
description: Detects gaps in security observability pipeline indicating tampering
logsource:
  category: application
  product: prometheus
detection:
  selection:
    alertname: "AuditLogGap"
  condition: selection
falsepositives:
  - Planned maintenance window for logging infrastructure
level: critical
tags:
  - attack.t1562
  - attack.defense_evasion
```

## Verification

- [ ] OpenTelemetry collector deployed and receiving all three signal types
- [ ] Security-specific metrics instrumented and scraped
- [ ] Alerting rules configured for auth failures and log gaps
- [ ] Dashboards deployed with security posture visibility
- [ ] Distributed traces correlate across service boundaries
- [ ] Alert routing configured to security team channels

## References

- [OpenTelemetry Documentation](https://opentelemetry.io/docs/)
- [Prometheus Alerting Rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/)
- [MITRE ATT&CK T1562 — Impair Defenses](https://attack.mitre.org/techniques/T1562/)
