<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
<!-- NERF is a trademark of defconxt. -->
---
name: building-real-time-streaming-analytics
description: >-
  Design and deploy real-time streaming analytics pipelines for security event
  processing using Apache Kafka, Flink, and Spark Structured Streaming to detect
  threats with sub-second latency on high-volume telemetry feeds.
domain: cybersecurity
subdomain: security-data-engineering
tags:
  - streaming
  - kafka
  - flink
  - spark-streaming
  - real-time-detection
  - cep
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1040", "T1071"]
  frameworks: ["NIST 800-92", "CIS Controls v8"]
  tools: ["python3", "kafka", "apache-flink", "spark"]
---

# Building Real-Time Streaming Analytics

## Overview

Real-time streaming analytics enables security teams to detect threats as they
unfold rather than after the fact. This technique covers building pipelines that
ingest, enrich, correlate, and alert on security events within seconds of
occurrence using modern stream-processing frameworks.

## Prerequisites

| Requirement | Purpose |
|---|---|
| Apache Kafka cluster | Event ingestion and buffering |
| Stream processor (Flink/Spark) | Stateful event processing |
| Schema registry | Event schema management and evolution |
| Alerting infrastructure | Downstream notification delivery |

## Key Concepts

- **Event time vs. processing time**: Use event timestamps for accurate windowing
- **Watermarks**: Handle late-arriving events gracefully
- **Stateful processing**: Maintain session and pattern state across events
- **Complex event processing (CEP)**: Detect multi-step attack patterns
- **Backpressure**: Prevent pipeline overload during traffic spikes
- **Exactly-once semantics**: Guarantee no duplicate or lost alerts

## Workflow

### Step 1: Define Streaming Topology

```python
from dataclasses import dataclass, field

@dataclass
class StreamSource:
    """Kafka topic source configuration."""
    topic: str
    schema: str
    partitions: int = 12
    retention_hours: int = 168

@dataclass
class StreamingPipeline:
    """Real-time security analytics pipeline specification."""
    sources: list[StreamSource] = field(default_factory=list)
    window_size_seconds: int = 60
    watermark_delay_seconds: int = 10
    checkpoint_interval_ms: int = 30000
    parallelism: int = 4
```

### Step 2: Implement Stream Processor

```python
from pyflink.datastream import StreamExecutionEnvironment
from pyflink.table import StreamTableEnvironment

def build_flink_pipeline(env: StreamExecutionEnvironment) -> None:
    """Build Flink streaming pipeline for security event correlation."""
    t_env = StreamTableEnvironment.create(env)
    t_env.execute_sql("""
        CREATE TABLE security_events (
            event_id STRING,
            event_time TIMESTAMP(3),
            source_ip STRING,
            dest_ip STRING,
            event_type STRING,
            severity INT,
            WATERMARK FOR event_time AS event_time - INTERVAL '10' SECOND
        ) WITH (
            'connector' = 'kafka',
            'topic' = 'security-events',
            'properties.bootstrap.servers' = 'kafka:9092',
            'format' = 'json'
        )
    """)
```

### Step 3: Build Detection Windows

```sql
-- Detect brute force: >10 failed logins from same IP in 5 minutes
SELECT source_ip, COUNT(*) AS attempt_count,
       TUMBLE_START(event_time, INTERVAL '5' MINUTE) AS window_start
FROM security_events
WHERE event_type = 'auth_failure'
GROUP BY source_ip, TUMBLE(event_time, INTERVAL '5' MINUTE)
HAVING COUNT(*) > 10;

-- Detect lateral movement: same source hitting >5 unique destinations
SELECT source_ip, COUNT(DISTINCT dest_ip) AS unique_targets,
       HOP_START(event_time, INTERVAL '1' MINUTE, INTERVAL '10' MINUTE) AS window_start
FROM security_events
WHERE event_type = 'connection'
GROUP BY source_ip, HOP(event_time, INTERVAL '1' MINUTE, INTERVAL '10' MINUTE)
HAVING COUNT(DISTINCT dest_ip) > 5;
```

### Step 4: Deploy and Monitor Pipeline

```bash
# Submit Flink job
flink run -d -p 4 streaming_analytics.jar

# Monitor consumer lag
kafka-consumer-groups.sh --bootstrap-server kafka:9092 \
  --describe --group security-analytics

# Validate end-to-end latency
node agent.js measure-latency --topic security-events --samples 100
```

## Detection

```yaml
title: Real Time Streaming Analytics Detection
id: a764af17-43f6-44f3-ab2d-bd7e9d3a5f37
status: experimental
description: Detects suspicious activity related to building real time streaming analytics techniques in security data engineering context
logsource:
  category: process_creation
  product: linux
detection:
  selection:
    CommandLine: "*building*real*"
  condition: selection
level: medium
tags:
  - attack.t1040
  - attack.t1071
  - attack.collection
falsepositives:
  - Data pipeline ETL processes during scheduled batch windows
```

## Verification

| Check | Method |
|---|---|
| Pipeline running | Flink dashboard shows active job |
| Consumer lag < threshold | Kafka consumer group lag under 1000 |
| Alert latency | End-to-end < 5 seconds |
| No data loss | Input count matches processed count |

## References

- [Apache Flink Documentation](https://flink.apache.org/docs/)
- [Kafka Streams DSL](https://kafka.apache.org/documentation/streams/)
- [NIST SP 800-92 — Log Management](https://csrc.nist.gov/publications/detail/sp/800-92/final)
