<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: implementing-threat-feed-aggregation
description: >-
  Aggregate, normalize, and deduplicate threat intelligence feeds from multiple
  sources including commercial, open-source, and government feeds. Covers feed
  quality scoring, confidence calibration, and automated IOC lifecycle management.
domain: cybersecurity
subdomain: threat-intelligence
tags:
  - threat-feeds
  - ioc-aggregation
  - feed-scoring
  - automation
  - otx
  - abuse-ch
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: []
---
# Implementing Threat Feed Aggregation

## Overview

Threat feed aggregation consolidates IOCs from multiple intelligence sources into
a unified, deduplicated, and scored dataset. This skill covers configuring feed
ingestion from open-source (abuse.ch, OTX, PhishTank), commercial (Recorded Future,
Mandiant), and government (CISA, US-CERT) sources, normalizing to STIX 2.1,
scoring feed quality, and managing IOC lifecycle with automated expiry.

## Prerequisites

- Python 3.10+ with `requests`, `stix2`, `pymisp`, `OTXv2` libraries
- MISP instance for feed aggregation and deduplication
- API keys for: AlienVault OTX, abuse.ch, VirusTotal
- Cron or task scheduler for periodic feed polling

```bash
pip install requests stix2 pymisp OTXv2
```

## Key Concepts

### Feed Taxonomy

| Category | Examples | Refresh Rate | Typical FP Rate |
|----------|----------|-------------|-----------------|
| Open-source | abuse.ch, PhishTank, OTX | Minutes-Hours | 15-40% |
| Commercial | Recorded Future, Mandiant | Hours | 5-15% |
| Government | CISA AIS, US-CERT | Hours-Days | <5% |
| Community | MISP sharing groups, ISACs | Hours | 10-25% |
| Internal | Sandbox output, IR findings | Real-time | <5% |

### Feed Quality Metrics

- **Timeliness**: latency from first observation to feed publication
- **Accuracy**: false positive rate measured against sinkhole/known-good lists
- **Coverage**: percentage of threats relevant to your sector detected
- **Context**: metadata richness (malware family, actor, campaign, TLP)
- **Actionability**: percentage of IOCs directly usable in detection rules

### IOC Deduplication Strategy

```
Hash normalization → lowercase → trim whitespace → type validation →
deduplicate by (type, value) → merge metadata from multiple sources →
highest confidence wins → earliest first_seen preserved
```

## Workflow

### Step 1: Poll Open-Source Feeds

```python
import requests
from datetime import datetime

def poll_abuse_ch_urlhaus() -> list[dict]:
    """Pull recent malware URLs from URLhaus."""
    resp = requests.get(
        "https://urlhaus-api.abuse.ch/v1/urls/recent/",
        timeout=30,
    )
    data = resp.json()
    return [
        {
            "type": "url",
            "value": entry["url"],
            "threat": entry.get("threat", ""),
            "tags": entry.get("tags", []),
            "source": "urlhaus",
            "first_seen": entry.get("date_added", ""),
            "confidence": 65,
        }
        for entry in data.get("urls", [])[:100]
    ]

def poll_otx_pulses(api_key: str, days: int = 7) -> list[dict]:
    """Pull recent OTX pulses and extract IOCs."""
    from OTXv2 import OTXv2
    otx = OTXv2(api_key)
    pulses = otx.getall(modified_since=f"{days}d")
    iocs = []
    for pulse in pulses:
        for indicator in pulse.get("indicators", []):
            iocs.append({
                "type": indicator["type"],
                "value": indicator["indicator"],
                "pulse": pulse["name"],
                "source": "otx",
                "confidence": 55,
            })
    return iocs
```

### Step 2: Normalize and Deduplicate

```python
def normalize_ioc(ioc: dict) -> dict:
    """Normalize IOC value and type for deduplication."""
    value = ioc["value"].strip().lower()
    ioc_type = ioc["type"].lower()

    type_map = {
        "ipv4": "ip-dst", "ip": "ip-dst", "ipv4-addr": "ip-dst",
        "domain": "domain", "hostname": "domain",
        "url": "url", "uri": "url",
        "filehash-sha256": "sha256", "sha256": "sha256",
        "filehash-md5": "md5", "md5": "md5",
    }
    normalized_type = type_map.get(ioc_type, ioc_type)
    return {**ioc, "value": value, "type": normalized_type}

def deduplicate_iocs(iocs: list[dict]) -> list[dict]:
    """Deduplicate IOCs, merging metadata from multiple sources."""
    seen: dict[tuple, dict] = {}
    for ioc in iocs:
        key = (ioc["type"], ioc["value"])
        if key in seen:
            existing = seen[key]
            existing["sources"] = list(set(existing.get("sources", []) + [ioc.get("source", "")]))
            existing["confidence"] = max(existing.get("confidence", 0), ioc.get("confidence", 0))
        else:
            ioc["sources"] = [ioc.get("source", "")]
            seen[key] = ioc
    return list(seen.values())
```

### Step 3: Score and Classify

```python
def score_feed_quality(iocs: list[dict], known_good: set[str]) -> dict:
    """Score feed quality based on false positive rate."""
    total = len(iocs)
    fps = sum(1 for i in iocs if i["value"] in known_good)
    tp = total - fps
    return {
        "total_iocs": total,
        "true_positives": tp,
        "false_positives": fps,
        "fp_rate": round(fps / max(total, 1) * 100, 1),
        "quality_grade": "A" if fps / max(total, 1) < 0.05 else
                         "B" if fps / max(total, 1) < 0.15 else
                         "C" if fps / max(total, 1) < 0.30 else "D",
    }
```

### Step 4: Push to MISP

```python
from pymisp import PyMISP

def push_to_misp(misp_url: str, misp_key: str, iocs: list[dict], event_info: str) -> dict:
    """Push aggregated IOCs to MISP as a new event."""
    misp = PyMISP(misp_url, misp_key, ssl=False)
    event = misp.new_event(
        info=event_info,
        distribution=1,
        threat_level_id=2,
        analysis=2,
    )
    added = 0
    for ioc in iocs:
        if ioc.get("confidence", 0) >= 50:
            misp.add_attribute(
                event,
                {"type": ioc["type"], "value": ioc["value"], "to_ids": ioc.get("confidence", 0) >= 70},
            )
            added += 1
    return {"event_id": event.get("Event", {}).get("id"), "attributes_added": added}
```

## Verification

- [ ] Open-source feeds polled and IOCs extracted with source attribution
- [ ] IOC normalization handles type aliases and value formatting
- [ ] Deduplication merges metadata while preserving highest confidence
- [ ] Feed quality scoring identifies high-FP feeds for tuning
- [ ] MISP integration pushes scored IOCs with correct to_ids flags
- [ ] Automated expiry removes stale IOCs past confidence threshold

## References

- [abuse.ch URLhaus API](https://urlhaus-api.abuse.ch/)
- [AlienVault OTX](https://otx.alienvault.com/)
- [MISP Feeds](https://www.misp-project.org/feeds/)
- [ThreatFox API](https://threatfox-api.abuse.ch/)
- [STIX 2.1 Indicator SDO](https://docs.oasis-open.org/cti/stix/v2.1/os/stix-v2.1-os.html#_muftrcpnf89v)
