<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: detecting-network-anomalies-with-ml
description: >-
  Apply machine learning techniques to network traffic analysis for anomaly detection
  including baseline modeling, unsupervised clustering, time-series analysis, and
  feature engineering from Zeek logs and NetFlow data using scikit-learn and statistical methods.
domain: cybersecurity
subdomain: network-security
tags:
  - machine-learning
  - anomaly-detection
  - zeek
  - netflow
  - scikit-learn
  - statistics
version: "1.0"
author: defconxt
license: AGPL-3.0
compatibility: Designed for Claude Code, GitHub Copilot, OpenAI Codex, Cursor, Gemini CLI, and any agentskills.io-compatible agent.
metadata:
  mitre-attack: ["T1071", "T1048", "T1572"]
  nist-csf: ["DE.AE-1", "DE.AE-3", "DE.CM-1"]
  uuid: a1b2c3d4-1011-4a00-b011-f00d00000011
---

# Detecting Network Anomalies with ML

## Overview

Machine learning enables detection of novel threats that signature-based systems miss.
By establishing statistical baselines of normal network behavior and flagging deviations,
ML-based detection identifies zero-day attacks, insider threats, and advanced persistent
threats. This skill covers feature engineering from network logs, unsupervised anomaly
detection with Isolation Forest and DBSCAN, time-series anomaly detection, and
integration with existing detection pipelines.

Mode: `[MODE: BLUE]` for detection deployment; `[MODE: PURPLE]` for coverage analysis.

## Prerequisites

| Requirement | Details |
|---|---|
| Zeek 6.0+ with conn.log, dns.log, and http.log | Required |
| Python 3.10+ with scikit-learn, pandas, numpy | Required |
| Historical network logs for training (minimum 7 days baseline) | Required |
| NetFlow/IPFIX data for volumetric analysis | Required |
| Jupyter or similar for exploratory analysis | Required |

## Key Concepts

### Feature Engineering from Zeek Logs

```bash
# Extract connection features from Zeek conn.log
zeek-cut -d ts id.orig_h id.resp_h id.resp_p proto duration orig_bytes resp_bytes \
  conn_state < conn.log > features.tsv

# Aggregate per-source features
zeek-cut id.orig_h id.resp_p proto < conn.log | \
  sort | uniq -c | sort -rn > source_port_profile.txt
```

### Isolation Forest Anomaly Detection

```python
#!/usr/bin/env python3
"""Network anomaly detection using Isolation Forest on Zeek conn.log features."""

import pandas as pd
import numpy as np
from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler

def load_zeek_conn(path: str) -> pd.DataFrame:
    """Load and parse Zeek conn.log into DataFrame."""
    cols = ["ts", "uid", "orig_h", "orig_p", "resp_h", "resp_p",
            "proto", "service", "duration", "orig_bytes", "resp_bytes",
            "conn_state", "missed_bytes", "orig_pkts", "resp_pkts"]
    df = pd.read_csv(path, sep="\t", comment="#", names=cols,
                     na_values="-", low_memory=False)
    df["duration"] = pd.to_numeric(df["duration"], errors="coerce").fillna(0)
    df["orig_bytes"] = pd.to_numeric(df["orig_bytes"], errors="coerce").fillna(0)
    df["resp_bytes"] = pd.to_numeric(df["resp_bytes"], errors="coerce").fillna(0)
    return df

def extract_features(df: pd.DataFrame) -> pd.DataFrame:
    """Engineer features for anomaly detection."""
    features = df.groupby("orig_h").agg(
        conn_count=("uid", "count"),
        unique_dsts=("resp_h", "nunique"),
        unique_ports=("resp_p", "nunique"),
        avg_duration=("duration", "mean"),
        total_bytes_sent=("orig_bytes", "sum"),
        total_bytes_recv=("resp_bytes", "sum"),
        avg_bytes_sent=("orig_bytes", "mean"),
    ).reset_index()
    features["bytes_ratio"] = (
        features["total_bytes_sent"] /
        features["total_bytes_recv"].replace(0, 1)
    )
    return features

def detect_anomalies(features: pd.DataFrame, contamination: float = 0.05):
    """Run Isolation Forest on extracted features."""
    numeric_cols = features.select_dtypes(include=[np.number]).columns
    scaler = StandardScaler()
    scaled = scaler.fit_transform(features[numeric_cols])
    model = IsolationForest(contamination=contamination, random_state=42, n_jobs=-1)
    features["anomaly_score"] = model.fit_predict(scaled)
    features["score"] = model.decision_function(scaled)
    anomalies = features[features["anomaly_score"] == -1].sort_values("score")
    return anomalies
```

### Time-Series Anomaly Detection

```python
"""Detect anomalous traffic volume using rolling statistics."""

def detect_volume_anomalies(df: pd.DataFrame, window: str = "5min", threshold: float = 3.0):
    """Flag time windows with traffic volume beyond N standard deviations."""
    df["ts"] = pd.to_datetime(df["ts"], unit="s")
    volume = df.set_index("ts").resample(window).agg(
        conn_count=("uid", "count"),
        bytes_total=("orig_bytes", "sum"),
    )
    for col in ["conn_count", "bytes_total"]:
        rolling_mean = volume[col].rolling(window=12, min_periods=3).mean()
        rolling_std = volume[col].rolling(window=12, min_periods=3).std()
        volume[f"{col}_zscore"] = (volume[col] - rolling_mean) / rolling_std.replace(0, 1)
        volume[f"{col}_anomaly"] = volume[f"{col}_zscore"].abs() > threshold
    return volume[volume.filter(like="_anomaly").any(axis=1)]
```

### Zeek Integration Script

```zeek
# ml-feature-export.zeek — export connection features for ML pipeline
@load base/protocols/conn

module MLExport;

event connection_state_remove(c: connection) {
    if ( c$conn?$duration && c$conn?$orig_bytes ) {
        Log::write(Log::LOG, [$ts=c$conn$ts,
                              $orig_h=c$id$orig_h,
                              $resp_h=c$id$resp_h,
                              $resp_p=c$id$resp_p,
                              $duration=c$conn$duration,
                              $orig_bytes=c$conn$orig_bytes,
                              $resp_bytes=c$conn$resp_bytes]);
    }
}
```

### Suricata Alert Correlation

```bash
# Feed ML anomaly results back into Suricata threshold tuning
# If ML flags an IP as anomalous, lower detection threshold for that source
cat anomalies.json | jq -r '.orig_h' | while read ip; do
  echo "suppress gen_id 1, sig_id 0, track by_src, ip $ip" >> /etc/suricata/threshold.config
done
```

## Workflow

### Step 1: Collect Training Data

Gather minimum 7 days of Zeek conn.log during normal operations.

### Step 2: Feature Engineering

Extract per-host aggregated features from connection logs.

### Step 3: Train Baseline Model

Fit Isolation Forest on clean training data with 1-5% contamination.

### Step 4: Deploy Detection

Run model on live Zeek output, flag anomalies, route to SIEM.

### Step 5: Tune and Retrain

Review false positives, adjust features, retrain on updated baseline.

## Detection

```yaml
title: Network Anomalies With Ml Detection
id: 213016f7-4585-4963-8a18-09c5e2275877
status: experimental
description: Detects suspicious activity related to detecting network anomalies with ml techniques in network security context
logsource:
  category: firewall
  product: linux
detection:
  selection:
    Action: blocked
  condition: selection
level: medium
tags:
  - attack.t1071
  - attack.t1048
  - attack.t1572
  - attack.command_and_control
falsepositives:
  - Network monitoring tools performing scheduled connectivity checks
```


**Detection Opportunities**

| Indicator | Source | Detection Logic |
|---|---|---|
| Network Anomalies With Ml Detection | linux/firewall | Sigma rule (medium) |
| ATT&CK Coverage | MITRE ATT&CK | T1071, T1048, T1572 |

## Verification

- [ ] Feature extraction runs on production Zeek logs without errors
- [ ] Isolation Forest detects known attack patterns in test data
- [ ] Time-series detector flags volumetric anomalies
- [ ] False positive rate under 5%
- [ ] Anomaly alerts include actionable context (host, features, score)
- [ ] Model retrained on weekly schedule

## References

- [scikit-learn Isolation Forest](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.IsolationForest.html)
- [Zeek Log Formats](https://docs.zeek.org/en/current/log-formats.html)
- NIST AI 100-2: Adversarial Machine Learning
- MITRE ATLAS — Adversarial Threat Landscape for AI Systems
