<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
<!-- NERF is a trademark of defconxt. -->
---
name: implementing-threat-intelligence-data-pipelines
description: >-
  Build automated threat intelligence data pipelines that ingest, normalize,
  enrich, score, and distribute indicators of compromise (IOCs) and threat
  reports from STIX/TAXII feeds, OSINT sources, and commercial providers.
domain: cybersecurity
subdomain: security-data-engineering
tags:
  - threat-intelligence
  - stix
  - taxii
  - ioc
  - enrichment
  - misp
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1588", "T1608"]
  frameworks: ["MITRE ATT&CK", "STIX 2.1", "TAXII 2.1"]
  tools: ["python3", "misp", "opencti", "stix2"]
---

# Implementing Threat Intelligence Data Pipelines

## Overview

Threat intelligence pipelines automate the lifecycle of IOC and threat report
management — from ingestion through STIX/TAXII feeds and OSINT sources to
enrichment, scoring, deduplication, and distribution to downstream security
controls (SIEM, firewall, EDR).

## Prerequisites

| Requirement | Purpose |
|---|---|
| TI platform (MISP/OpenCTI) | Central intelligence management |
| STIX/TAXII feed access | Structured threat intelligence feeds |
| Enrichment APIs | VirusTotal, Shodan, abuse.ch |
| Distribution targets | SIEM, EDR, firewall rule feeds |

## Key Concepts

- **STIX 2.1**: Structured language for threat intelligence objects
- **TAXII 2.1**: Transport protocol for exchanging STIX bundles
- **IOC lifecycle**: Ingest → validate → enrich → score → distribute → expire
- **Confidence scoring**: Weight indicators by source reliability and age
- **Deduplication**: Merge duplicate IOCs from multiple feeds
- **Expiration policy**: Age out stale indicators automatically

## Workflow

### Step 1: Configure Feed Ingestion

```python
from dataclasses import dataclass

@dataclass
class ThreatFeed:
    """Threat intelligence feed configuration."""
    name: str
    feed_type: str  # "taxii", "csv", "api", "misp"
    url: str
    poll_interval_minutes: int = 60
    confidence_weight: float = 0.8
    enabled: bool = True

FEEDS = [
    ThreatFeed("alienvault_otx", "api", "https://otx.alienvault.com/api/v1/"),
    ThreatFeed("abuse_ch", "csv", "https://urlhaus.abuse.ch/downloads/csv_recent/"),
    ThreatFeed("cisa_kev", "csv", "https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json"),
]
```

### Step 2: Normalize to STIX Format

```python
from stix2 import Indicator, Bundle

def normalize_ioc(raw_ioc: dict) -> Indicator:
    """Normalize raw IOC to STIX 2.1 Indicator."""
    return Indicator(
        name=raw_ioc["value"],
        pattern=f"[ipv4-addr:value = '{raw_ioc['value']}']",
        pattern_type="stix",
        valid_from=raw_ioc.get("first_seen"),
        confidence=raw_ioc.get("confidence", 50),
        labels=raw_ioc.get("tags", ["malicious-activity"]),
    )
```

### Step 3: Enrich and Score

```python
def enrich_indicator(indicator: dict) -> dict:
    """Enrich indicator with external intelligence sources."""
    enrichments = {}
    enrichments["virustotal"] = {"malicious": 45, "total": 70}
    enrichments["shodan"] = {"ports": [80, 443, 8080], "org": "Example ISP"}
    enrichments["abuse_ch"] = {"threat_type": "botnet_cc"}
    score = calculate_confidence(indicator, enrichments)
    indicator["enrichments"] = enrichments
    indicator["confidence_score"] = score
    return indicator

def calculate_confidence(indicator: dict, enrichments: dict) -> float:
    """Calculate composite confidence score 0.0-1.0."""
    base = indicator.get("source_confidence", 0.5)
    vt = enrichments.get("virustotal", {})
    vt_ratio = vt.get("malicious", 0) / max(vt.get("total", 1), 1)
    return min(1.0, base * 0.4 + vt_ratio * 0.6)
```

### Step 4: Distribute to Controls

```bash
node agent.js ingest --feed alienvault_otx --since 24h
node agent.js enrich --ioc-file new_indicators.json --output enriched.json
node agent.js distribute --targets siem,edr,firewall --min-confidence 0.7
```

## Detection

```yaml
title: Threat Intelligence Data Pipelines Detection
id: 9720b4ac-528c-4c0f-aa86-5d0dfb70b477
status: experimental
description: Detects suspicious activity related to implementing threat intelligence data pipelines techniques in security data engineering context
logsource:
  category: process_creation
  product: linux
detection:
  selection:
    CommandLine: "*implementing*threat*"
  condition: selection
level: medium
tags:
  - attack.t1588
  - attack.t1608
  - attack.collection
falsepositives:
  - Data pipeline ETL processes during scheduled batch windows
```

## Verification

| Check | Method |
|---|---|
| Feed ingestion working | Poll returns new indicators |
| Normalization correct | STIX validation passes |
| Enrichment APIs responding | All enrichment sources return data |
| Distribution confirmed | IOCs visible in downstream controls |

## References

- [STIX 2.1 Specification](https://oasis-open.github.io/cti-documentation/)
- [MISP Project](https://www.misp-project.org/)
- [OpenCTI Platform](https://www.opencti.io/)
