<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: building-siem-data-architecture
description: >-
  Design SIEM data architectures including index strategies, hot/warm/cold
  tiering, search optimization, data routing, and retention policies for
  scalable security operations at enterprise scale.
domain: cybersecurity
subdomain: security-data-engineering
tags:
  - siem
  - architecture
  - indexing
  - tiering
  - retention
  - elasticsearch
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  frameworks: ["NIST 800-92", "CIS Controls v8"]
  tools: ["python3", "elasticsearch", "opensearch", "splunk"]
  mitre-attack: ["T1070"]
---

# Building SIEM Data Architecture

## Overview

SIEM data architecture determines query performance, storage cost, and
operational effectiveness. This skill covers index design for security event
types, hot/warm/cold tier configuration, search-time optimization, data
routing rules, retention policies, and capacity planning for enterprise
security operations.

## Prerequisites

| Requirement | Details |
|---|---|
| SIEM platform (Elasticsearch, Splunk, Sentinel, Chronicle) | Required |
| Understanding of daily log volumes by source type | Required |
| Retention requirements (regulatory and operational) | Required |
| Capacity planning for peak ingestion rates | Required |

## Key Concepts

- **Index strategy**: Separate indices by source type and time for performance
- **ILM (Index Lifecycle Management)**: Automate tier transitions and deletion
- **Search optimization**: Field mappings, runtime fields, saved searches
- **Data routing**: Direct sources to appropriate storage tiers on ingest
- **Capacity planning**: Size nodes for peak ingestion and concurrent queries

## Workflow

### Step 1: Design Index Strategy

```python
from dataclasses import dataclass

@dataclass
class IndexDesign:
    """SIEM index configuration."""
    name_pattern: str  # "security-{source_type}-{date}"
    shards: int
    replicas: int
    refresh_interval: str  # "1s", "30s", "60s"
    tier: str  # "hot", "warm", "cold"
    retention_days: int
    rollover_size_gb: int = 50
```

### Step 2: Configure ILM Policy

```json
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {"max_size": "50gb", "max_age": "1d"},
          "set_priority": {"priority": 100}
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": {"number_of_shards": 1},
          "forcemerge": {"max_num_segments": 1},
          "set_priority": {"priority": 50}
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "searchable_snapshot": {"snapshot_repository": "security-archive"},
          "set_priority": {"priority": 0}
        }
      },
      "delete": {"min_age": "365d", "actions": {"delete": {}}}
    }
  }
}
```

### Step 3: Plan Capacity

```sql
-- Estimate storage needs per source type
SELECT
    source_type,
    COUNT(*) AS daily_events,
    SUM(LENGTH(raw_event)) / 1073741824.0 AS daily_gb_raw,
    SUM(LENGTH(raw_event)) * 1.1 / 1073741824.0 AS daily_gb_indexed
FROM security.events
WHERE time_dt >= CURRENT_DATE - INTERVAL '1 day'
GROUP BY source_type
ORDER BY daily_gb_raw DESC;
```

### Step 4: Validate Architecture

```bash
node agent.js design --sources endpoint,network,cloud --retention 365 --output arch.json
node agent.js capacity --daily-gb 500 --retention 365 --replicas 1
```

## Detection

```yaml
title: Siem Data Architecture Detection
id: f3dc2ec5-50cf-4c30-ab44-f7dc518f7a0d
status: experimental
description: Detects suspicious activity related to building siem data architecture techniques in security data engineering context
logsource:
  category: process_creation
  product: linux
detection:
  selection:
    CommandLine: "*building*siem*"
  condition: selection
level: medium
tags:
  - attack.t1070
  - attack.collection
falsepositives:
  - Data pipeline ETL processes during scheduled batch windows
```


**Detection Opportunities**

| Indicator | Source | Detection Logic |
|---|---|---|
| Siem Data Architecture Detection | linux/process_creation | Sigma rule (medium) |
| ATT&CK Coverage | MITRE ATT&CK | T1070 |

## References

- [Elasticsearch ILM](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html)
- [Splunk Index Architecture](https://docs.splunk.com/Documentation/Splunk/latest/Indexer/Aboutindexesandindexers)
- [NIST SP 800-92](https://csrc.nist.gov/publications/detail/sp/800-92/final)
