<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: building-ioc-enrichment-pipeline
description: >-
  Build automated IOC enrichment pipelines that normalize, deduplicate, score,
  and contextualize indicators using VirusTotal, Shodan, GreyNoise, and AbuseIPDB.
domain: cybersecurity
subdomain: threat-intelligence
tags:
  - ioc-enrichment
  - virustotal
  - shodan
  - greynoise
  - abuseipdb
  - automation
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: []
---
# Building IOC Enrichment Pipeline

## Overview

IOC enrichment transforms raw indicators (IPs, domains, hashes, URLs) into
contextualized intelligence by querying multiple reputation sources, correlating
results, and assigning confidence scores. This skill covers building automated
pipelines that normalize IOC types, deduplicate, enrich via API, score confidence,
tag with context (malware family, campaign, actor), and export to detection platforms.

## Prerequisites

- Python 3.10+ with `requests`, `vt-py`, `shodan`
- API keys: VirusTotal, Shodan, GreyNoise, AbuseIPDB
- MISP or OpenCTI instance for IOC storage
- Understanding of IOC types and confidence decay rates

```bash
pip install vt-py shodan requests
```

## Key Concepts

### IOC Type Normalization

Normalize all indicators before enrichment:
- **IP addresses**: validate IPv4/IPv6, strip port numbers, resolve CIDR notation
- **Domains**: lowercase, strip protocol/path, extract FQDN from URLs
- **Hashes**: identify hash type by length (MD5=32, SHA1=40, SHA256=64)
- **URLs**: normalize encoding, extract domain + path components

### Confidence Scoring Model

```
Score 90-100: Sandbox-confirmed C2, multiple vendor detections >10
Score 70-89:  Multiple sources corroborate, 3-10 vendor detections
Score 50-69:  Single credible source, 1-2 vendor detections
Score 30-49:  Automated feed only, limited context
Score <30:    Stale or low-confidence — retire from active blocking
```

### Enrichment Source Priority

| IOC Type | Primary Sources | Secondary Sources |
|----------|----------------|-------------------|
| IP | VirusTotal, Shodan, GreyNoise | AbuseIPDB, Censys, RIPE |
| Domain | VirusTotal, urlscan.io, PassiveDNS | WHOIS, SecurityTrails |
| Hash | VirusTotal, MalwareBazaar | Hybrid Analysis, Any.Run |
| URL | VirusTotal, urlscan.io | Google Safe Browsing |

### Pipeline Architecture

```
Input → Normalize → Deduplicate → Enrich (parallel API calls)
→ Score → Tag (malware family, campaign) → Store (MISP/OpenCTI)
→ Export (SIEM rules, EDR blocklist) → Schedule expiry
```

## Workflow

### Step 1: Normalize and Deduplicate

```python
import re
import hashlib
from urllib.parse import urlparse

def classify_ioc(value: str) -> str:
    """Classify IOC type from raw value."""
    value = value.strip()
    if re.match(r'^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$', value):
        return "ip"
    if re.match(r'^[a-fA-F0-9]{64}$', value):
        return "sha256"
    if re.match(r'^[a-fA-F0-9]{40}$', value):
        return "sha1"
    if re.match(r'^[a-fA-F0-9]{32}$', value):
        return "md5"
    if re.match(r'^https?://', value):
        return "url"
    if re.match(r'^[a-zA-Z0-9]([a-zA-Z0-9-]*\.)+[a-zA-Z]{2,}$', value):
        return "domain"
    return "unknown"

def normalize_ioc(value: str, ioc_type: str) -> str:
    """Normalize IOC value by type."""
    if ioc_type == "domain":
        return value.lower().strip(".")
    if ioc_type == "url":
        return value.lower().rstrip("/")
    if ioc_type in ("md5", "sha1", "sha256"):
        return value.lower()
    return value.strip()
```

### Step 2: Enrich via APIs

```python
import requests

def enrich_ip_virustotal(ip: str, api_key: str) -> dict:
    """Enrich IP via VirusTotal API v3."""
    resp = requests.get(
        f"https://www.virustotal.com/api/v3/ip_addresses/{ip}",
        headers={"x-apikey": api_key},
    )
    if resp.status_code == 200:
        data = resp.json()["data"]["attributes"]
        stats = data.get("last_analysis_stats", {})
        return {
            "source": "virustotal",
            "malicious": stats.get("malicious", 0),
            "suspicious": stats.get("suspicious", 0),
            "country": data.get("country", ""),
            "as_owner": data.get("as_owner", ""),
        }
    return {"source": "virustotal", "error": resp.status_code}

def enrich_ip_greynoise(ip: str, api_key: str) -> dict:
    """Check if IP is known scanner via GreyNoise."""
    resp = requests.get(
        f"https://api.greynoise.io/v3/community/{ip}",
        headers={"key": api_key},
    )
    if resp.status_code == 200:
        data = resp.json()
        return {
            "source": "greynoise",
            "noise": data.get("noise", False),
            "riot": data.get("riot", False),
            "classification": data.get("classification", ""),
        }
    return {"source": "greynoise", "error": resp.status_code}

def enrich_ip_shodan(ip: str, api_key: str) -> dict:
    """Enrich IP with Shodan host data."""
    resp = requests.get(
        f"https://api.shodan.io/shodan/host/{ip}?key={api_key}"
    )
    if resp.status_code == 200:
        data = resp.json()
        return {
            "source": "shodan",
            "ports": data.get("ports", []),
            "os": data.get("os"),
            "hostnames": data.get("hostnames", []),
            "vulns": data.get("vulns", []),
        }
    return {"source": "shodan", "error": resp.status_code}
```

### Step 3: Score and Tag

```python
def score_ioc(enrichment_results: list[dict]) -> int:
    """Calculate confidence score from enrichment results."""
    score = 30  # baseline
    for result in enrichment_results:
        if result.get("source") == "virustotal":
            malicious = result.get("malicious", 0)
            if malicious > 10:
                score += 40
            elif malicious > 3:
                score += 25
            elif malicious > 0:
                score += 10
        if result.get("source") == "greynoise":
            if result.get("classification") == "malicious":
                score += 20
            elif result.get("riot"):
                score -= 20  # known benign service
        if result.get("source") == "shodan":
            if result.get("vulns"):
                score += 10
    return min(max(score, 0), 100)
```

### Step 4: Export to MISP

```python
def push_to_misp(ioc: str, ioc_type: str, score: int, misp_url: str, misp_key: str) -> dict:
    """Push enriched IOC to MISP as an attribute."""
    type_map = {"ip": "ip-dst", "domain": "domain", "sha256": "sha256", "url": "url"}
    payload = {
        "value": ioc,
        "type": type_map.get(ioc_type, "text"),
        "to_ids": score >= 70,
        "comment": f"Auto-enriched. Confidence: {score}/100",
        "Tag": [{"name": f"confidence:{score}"}],
    }
    resp = requests.post(
        f"{misp_url}/attributes/add/0",
        headers={"Authorization": misp_key, "Content-Type": "application/json"},
        json=payload,
    )
    return resp.json()
```

## Verification

- [ ] IOC normalization handles all types (IP, domain, hash, URL) correctly
- [ ] Enrichment queries return valid data from at least 2 sources
- [ ] Confidence scoring produces consistent results across IOC types
- [ ] High-confidence IOCs (>70) pushed to MISP with `to_ids=true`
- [ ] Pipeline handles API rate limits and errors gracefully
- [ ] Stale IOCs auto-expire based on type-specific decay rates

## References

- [VirusTotal API v3](https://docs.virustotal.com/reference/overview)
- [Shodan API](https://developer.shodan.io/api)
- [GreyNoise API](https://docs.greynoise.io/)
- [AbuseIPDB API](https://docs.abuseipdb.com/)
- [MISP Automation API](https://www.misp-project.org/openapi/)
