<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: performing-osint-gathering
description: >-
  Perform open-source intelligence gathering using passive reconnaissance
  techniques, social media analysis, and public data sources for threat research.
domain: cybersecurity
subdomain: threat-intelligence
tags:
  - osint
  - reconnaissance
  - passive-recon
  - social-media
  - domain-recon
  - shodan
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1593", "T1594"]
---
# Performing OSINT Gathering

## Overview

Open-source intelligence (OSINT) gathering collects publicly available information
to support threat intelligence, investigations, and security assessments. This skill
covers domain/IP reconnaissance, social media analysis, document metadata extraction,
breach database queries, and structured OSINT workflows with source documentation
and confidence scoring.

## Prerequisites

- Python 3.10+ with `requests`, `shodan`, `dnspython`, `whois`
- API access: Shodan, VirusTotal, SecurityTrails, Hunter.io
- Tools: theHarvester, Maltego, SpiderFoot (optional)
- Understanding of passive vs. active reconnaissance boundaries
- Legal authorization for target investigation

```bash
pip install requests shodan dnspython python-whois beautifulsoup4
```

## Key Concepts

### OSINT Collection Framework

```
Intelligence Requirement → Source Identification → Collection Plan →
Data Gathering → Processing/Normalization → Analysis → Reporting
```

### Passive vs. Active Reconnaissance

| Type | Examples | Detection Risk |
|------|---------|---------------|
| Passive | WHOIS, DNS records, Google dorking, social media | None |
| Semi-passive | Shodan queries, CT logs, archive.org | Minimal |
| Active | Port scanning, vulnerability probing, web crawling | HIGH |

**Rule:** Stay passive unless explicitly authorized for active recon.

### Source Reliability (Admiralty Scale)

```
A — Completely reliable (government records, registrars)
B — Usually reliable (established security vendors)
C — Fairly reliable (community reports, researcher blogs)
D — Not usually reliable (anonymous sources, forums)
E — Unreliable (single unverified source)
F — Cannot be judged (new source, no track record)
```

### Information Credibility

```
1 — Confirmed by other independent sources
2 — Probably true (consistent with known patterns)
3 — Possibly true (plausible but unconfirmed)
4 — Doubtful (inconsistent with other evidence)
5 — Improbable (contradicted by reliable sources)
6 — Cannot be judged
```

## Workflow

### Step 1: Domain Reconnaissance

```python
import requests
import dns.resolver

def domain_recon(domain: str) -> dict:
    """Gather domain intelligence from public sources."""
    results = {"domain": domain, "dns": {}, "whois": {}, "subdomains": []}

    # DNS records
    for rtype in ["A", "AAAA", "MX", "NS", "TXT", "SOA"]:
        try:
            answers = dns.resolver.resolve(domain, rtype)
            results["dns"][rtype] = [str(r) for r in answers]
        except Exception:
            pass

    # Certificate Transparency subdomains
    try:
        resp = requests.get(f"https://crt.sh/?q=%.{domain}&output=json", timeout=30)
        if resp.status_code == 200:
            names = set()
            for cert in resp.json():
                for name in cert.get("name_value", "").split("\n"):
                    name = name.strip().lstrip("*.")
                    if name.endswith(domain):
                        names.add(name)
            results["subdomains"] = sorted(names)
    except Exception:
        pass

    return results

def google_dorks(domain: str) -> list[dict]:
    """Generate Google dork queries for a domain."""
    return [
        {"dork": f'site:{domain} filetype:pdf', "purpose": "Find PDF documents"},
        {"dork": f'site:{domain} filetype:xlsx OR filetype:csv', "purpose": "Find spreadsheets"},
        {"dork": f'site:{domain} inurl:admin OR inurl:login', "purpose": "Find admin panels"},
        {"dork": f'site:{domain} "password" OR "username" OR "credentials"', "purpose": "Find credential leaks"},
        {"dork": f'site:{domain} ext:conf OR ext:cfg OR ext:env', "purpose": "Find config files"},
        {"dork": f'"{domain}" site:pastebin.com OR site:github.com', "purpose": "External mentions"},
    ]
```

### Step 2: Email and Employee Discovery

```python
def discover_emails(domain: str, hunter_key: str) -> dict:
    """Discover email addresses associated with domain."""
    resp = requests.get(
        "https://api.hunter.io/v2/domain-search",
        params={"domain": domain, "api_key": hunter_key}, timeout=30,
    )
    if resp.status_code == 200:
        data = resp.json().get("data", {})
        return {
            "domain": domain,
            "pattern": data.get("pattern", ""),
            "emails": [
                {"email": e["value"], "type": e.get("type", ""),
                 "confidence": e.get("confidence", 0),
                 "first_name": e.get("first_name", ""),
                 "last_name": e.get("last_name", "")}
                for e in data.get("emails", [])
            ],
        }
    return {"domain": domain, "error": resp.status_code}
```

### Step 3: IP and Infrastructure Recon

```python
def ip_recon(ip: str, shodan_key: str) -> dict:
    """Gather intelligence on an IP address."""
    results = {"ip": ip}

    # Shodan host info
    resp = requests.get(f"https://api.shodan.io/shodan/host/{ip}?key={shodan_key}", timeout=30)
    if resp.status_code == 200:
        data = resp.json()
        results["shodan"] = {
            "org": data.get("org", ""),
            "asn": data.get("asn", ""),
            "ports": data.get("ports", []),
            "hostnames": data.get("hostnames", []),
            "os": data.get("os"),
            "country": data.get("country_code", ""),
        }

    # VirusTotal reputation
    vt_key = ""  # Set from env
    if vt_key:
        resp = requests.get(
            f"https://www.virustotal.com/api/v3/ip_addresses/{ip}",
            headers={"x-apikey": vt_key}, timeout=30,
        )
        if resp.status_code == 200:
            stats = resp.json()["data"]["attributes"].get("last_analysis_stats", {})
            results["virustotal"] = {"malicious": stats.get("malicious", 0)}

    return results
```

### Step 4: Social Media and Public Data

```python
def check_username(username: str) -> dict:
    """Check username availability across platforms."""
    platforms = {
        "github": f"https://api.github.com/users/{username}",
        "twitter": f"https://api.twitter.com/2/users/by/username/{username}",
    }
    results = {}
    for platform, url in platforms.items():
        try:
            resp = requests.get(url, timeout=10)
            results[platform] = {"exists": resp.status_code == 200}
        except Exception:
            results[platform] = {"exists": False, "error": "timeout"}
    return {"username": username, "platforms": results}
```

### Step 5: Structured OSINT Report

```python
def create_osint_report(target: str, findings: dict) -> dict:
    """Generate structured OSINT report with sourcing."""
    return {
        "report_type": "osint_collection",
        "target": target,
        "classification": "TLP:GREEN",
        "collection_methods": ["passive_dns", "certificate_transparency",
                                "whois", "shodan", "social_media"],
        "findings": findings,
        "source_reliability": "B2",  # Admiralty scale
        "confidence": "MODERATE",
        "limitations": [
            "Passive collection only — no active scanning performed",
            "Social media data may be outdated or fabricated",
            "WHOIS privacy may obscure registrant information",
        ],
    }
```

## Detection

```yaml
title: Osint Gathering Detection
id: 618a93f8-1c17-42ec-ac5e-be694d5c2b0b
status: experimental
description: Detects suspicious activity related to performing osint gathering techniques in threat intelligence context
logsource:
  category: network_connection
  product: firewall
detection:
  selection:
    DestinationPort: 443
    Initiated: "true"
  condition: selection
level: medium
tags:
  - attack.t1593
  - attack.t1594
  - attack.reconnaissance
falsepositives:
  - Threat intelligence platform aggregating feeds from approved sources
```

## Verification

- [ ] DNS enumeration returns A, MX, NS, TXT records
- [ ] Subdomain discovery via certificate transparency logs
- [ ] Email patterns identified with confidence scores
- [ ] IP intelligence includes ASN, ports, and reputation
- [ ] Google dorks generated for target-specific searches
- [ ] Report includes source reliability ratings and caveats

## References

- [Shodan API](https://developer.shodan.io/api)
- [Hunter.io API](https://hunter.io/api-documentation)
- [crt.sh Certificate Search](https://crt.sh/)
- [Google Hacking Database](https://www.exploit-db.com/google-hacking-database)
- [OSINT Framework](https://osintframework.com/)
- [theHarvester](https://github.com/laramies/theHarvester)
