<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: automating-osint-collection
description: >-
  Automate OSINT collection from DNS, WHOIS, certificate transparency, Shodan,
  and web scraping sources. Build structured intelligence pipelines with confidence
  tagging, source tracking, and passive-only collection constraints.
domain: cybersecurity
subdomain: automation-scripting
tags:
  - osint
  - reconnaissance
  - dns
  - whois
  - shodan
  - certificate-transparency
  - intelligence
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1593", "T1594"]
---

# Automating OSINT Collection

## Overview

OSINT automation accelerates reconnaissance by programmatically querying public
sources — DNS records, WHOIS databases, certificate transparency logs, Shodan,
and web archives. This skill builds structured collection pipelines that tag
source confidence, distinguish passive from active collection, and output
normalized intelligence for further analysis.

Mode: `[MODE: RECON]` — Passive intelligence gathering with source attribution.

## Prerequisites

| Requirement | Details |
|---|---|
| Python 3.10+ with `requests`, `dnspython` (`pip install requests dnspython`) | Required |
| Optional | Shodan API key, VirusTotal API key, SecurityTrails API key |
| `whois` CLI tool installed | Required |
| Network access to public DNS and CT log APIs | Required |

## Key Concepts

### DNS Enumeration

```python
import dns.resolver

def enumerate_dns(domain: str) -> dict:
    """Query common DNS record types for a domain."""
    records = {}
    for rtype in ["A", "AAAA", "MX", "NS", "TXT", "CNAME", "SOA"]:
        try:
            answers = dns.resolver.resolve(domain, rtype)
            records[rtype] = [str(r) for r in answers]
        except (dns.resolver.NoAnswer, dns.resolver.NXDOMAIN, dns.resolver.NoNameservers):
            pass
    return records
```

### Certificate Transparency Search

```python
import requests

def search_crt_sh(domain: str) -> list[dict]:
    """Search crt.sh for certificates issued to a domain."""
    url = f"https://crt.sh/?q=%.{domain}&output=json"
    resp = requests.get(url, timeout=30)
    if resp.status_code == 200:
        entries = resp.json()
        subdomains = set()
        for entry in entries:
            name = entry.get("name_value", "")
            for sub in name.split("\n"):
                sub = sub.strip().lstrip("*.")
                if sub.endswith(domain):
                    subdomains.add(sub)
        return sorted(subdomains)
    return []
```

### WHOIS Lookup

```python
import subprocess

def whois_lookup(target: str) -> dict:
    """Run WHOIS lookup and extract key fields."""
    result = subprocess.run(["whois", target], capture_output=True, text=True, timeout=15)
    output = result.stdout
    fields = {}
    for line in output.splitlines():
        if ":" in line:
            key, _, val = line.partition(":")
            key = key.strip().lower()
            if key in ("registrant", "registrar", "creation date", "expiration date", "name server"):
                fields.setdefault(key, []).append(val.strip())
    return fields
```

## Workflow

### Step 1: DNS Enumeration

```bash
node scripts/agent.js --action dns --target example.com --output /tmp/dns-enum.json
```

### Step 2: Subdomain Discovery via CT Logs

```bash
node scripts/agent.js --action subdomains --target example.com --output /tmp/subdomains.json
```

### Step 3: WHOIS Intelligence

```bash
node scripts/agent.js --action whois --target example.com --output /tmp/whois.json
```

## Verification

- [ ] DNS enumeration returns A, MX, NS, TXT records
- [ ] CT log search discovers subdomains not in public DNS
- [ ] WHOIS extracts registrant, registrar, and dates
- [ ] All collection is passive (no direct target interaction)
- [ ] Output includes source attribution and confidence tags

## References

- [crt.sh Certificate Search](https://crt.sh/)
- [DNSPython Documentation](https://dnspython.readthedocs.io/)
- [Shodan API Documentation](https://developer.shodan.io/)
- [MITRE ATT&CK T1596 — Search Open Technical Databases](https://attack.mitre.org/techniques/T1596/)
