<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: automating-threat-feed-processing
description: >-
  Automate threat intelligence feed ingestion, normalization, deduplication,
  and enrichment from STIX/TAXII, MISP, OTX, and custom CSV/JSON feeds.
  Covers IOC extraction, confidence scoring, SIEM integration, and
  automated blocklist generation for proactive defense.
domain: cybersecurity
subdomain: automation-scripting
tags:
  - threat-intelligence
  - stix-taxii
  - misp
  - otx
  - ioc-feeds
  - blocklist
  - threat-feeds
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1608"]
---

# Automating Threat Feed Processing

## Overview

Threat intelligence is only valuable when operationalized. This skill covers
building automated pipelines that ingest feeds from multiple sources
(STIX/TAXII, MISP, OTX, CSV), normalize IOCs into a common schema,
deduplicate, score by confidence, and push to SIEM or generate blocklists
for firewalls and proxies.

Mode: `[MODE: BLUE]` — Defensive threat intelligence operationalization.

## Prerequisites

| Requirement | Details |
|---|---|
| Python 3.10+ with `stix2`, `taxii2-client`, `requests` | Required |
| TAXII server endpoint or MISP instance for feed ingestion | Required |
| SIEM API access for IOC push (Splunk, Elastic, Sentinel) | Required |

## Key Concepts

### STIX/TAXII Feed Ingestion

```python
from taxii2client.v20 import Collection
import stix2

def ingest_taxii_feed(collection_url: str) -> list[dict]:
    """Pull IOCs from a TAXII 2.0 collection."""
    collection = Collection(collection_url)
    bundle = collection.get_objects()
    indicators = []
    for obj in bundle.get("objects", []):
        if obj.get("type") == "indicator":
            indicators.append({
                "id": obj["id"],
                "pattern": obj.get("pattern", ""),
                "created": obj.get("created", ""),
                "labels": obj.get("labels", []),
                "confidence": obj.get("confidence", 50),
            })
    return indicators
```

### IOC Normalization

```python
import re

IOC_PATTERNS = {
    "ipv4": re.compile(r"\b(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b"),
    "domain": re.compile(r"\b([a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\.[a-zA-Z]{2,})+)\b"),
    "md5": re.compile(r"\b([a-fA-F0-9]{32})\b"),
    "sha256": re.compile(r"\b([a-fA-F0-9]{64})\b"),
    "url": re.compile(r"(https?://[^\s<>\"']+)"),
}

def normalize_iocs(raw_text: str) -> dict[str, list[str]]:
    """Extract and categorize IOCs from unstructured text."""
    results: dict[str, list[str]] = {}
    for ioc_type, pattern in IOC_PATTERNS.items():
        matches = list(set(pattern.findall(raw_text)))
        if matches:
            results[ioc_type] = sorted(matches)
    return results

def deduplicate_iocs(feeds: list[dict[str, list[str]]]) -> dict[str, list[str]]:
    """Merge and deduplicate IOCs from multiple feeds."""
    merged: dict[str, set[str]] = {}
    for feed in feeds:
        for ioc_type, values in feed.items():
            if ioc_type not in merged:
                merged[ioc_type] = set()
            merged[ioc_type].update(values)
    return {k: sorted(v) for k, v in merged.items()}
```

### Blocklist Generation

```python
def generate_blocklist(iocs: dict[str, list[str]], format: str = "plain") -> str:
    """Generate firewall/proxy blocklist from IOCs."""
    if format == "plain":
        lines = []
        for ip in iocs.get("ipv4", []):
            lines.append(ip)
        for domain in iocs.get("domain", []):
            lines.append(domain)
        return "\n".join(lines)
    elif format == "snort":
        rules = []
        for i, ip in enumerate(iocs.get("ipv4", []), 1):
            rules.append(
                f'alert ip {ip} any -> $HOME_NET any '
                f'(msg:"TI: Malicious IP {ip}"; sid:{1000000+i}; rev:1;)'
            )
        return "\n".join(rules)
    return ""
```

## Workflow

### Step 1: Ingest and Normalize Feed

```bash
node scripts/agent.js --action ingest --feed-file /tmp/threat_feed.json
```

### Step 2: Extract IOCs from Text Report

```bash
node scripts/agent.js --action extract --feed-file /tmp/threat_report.txt
```

### Step 3: Generate Blocklist

```bash
node scripts/agent.js --action blocklist --feed-file /tmp/threat_feed.json --format snort
```

## Verification

- [ ] TAXII client pulls indicators from collection endpoints
- [ ] IOC extractor identifies IPs, domains, hashes, and URLs
- [ ] Deduplication merges multiple feeds without duplicates
- [ ] Blocklist generator produces valid Snort rules and plain lists
- [ ] Exit code 0 on success, 1 on empty feed, 2 on error

## References

- [STIX 2.1 Specification](https://docs.oasis-open.org/cti/stix/v2.1/stix-v2.1.html)
- [TAXII 2.1 Specification](https://docs.oasis-open.org/cti/taxii/v2.1/taxii-v2.1.html)
- [MISP API Documentation](https://www.misp-project.org/openapi/)
- [AlienVault OTX API](https://otx.alienvault.com/api)
- [MITRE ATT&CK T1588 — Obtain Capabilities](https://attack.mitre.org/techniques/T1588/)
