<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
<!-- NERF is a trademark of defconxt. -->
---
name: reverse-engineering-custom-encryption
description: >-
  Analyze and break custom or proprietary encryption implementations used in malware and commercial software.
domain: cybersecurity
subdomain: reverse-engineering
tags:
  - custom-crypto
  - encryption-analysis
  - cryptanalysis
  - proprietary
  - cipher-breaking
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1573"]
  cwe: ["CWE-327"]
  tools: ["ghidra", "ida-pro", "z3-solver", "python-crypto"]
---

# Reverse Engineering Custom Encryption

## Overview

Analyze and break custom or proprietary encryption implementations used in malware and commercial software.

## Prerequisites

| Tool / Requirement | Details |
|---|---|
| `ghidra` | Security tooling |
| `ida-pro` | Security tooling |
| `z3-solver` | Security tooling |
| `python-crypto` | Security tooling |
| Isolated lab environment for testing | Environment requirement |
| Authorization and signed Rules of Engagement (RoE) | Environment requirement |
| Relevant target samples or systems acquired through authorized channels | Environment requirement |

## Quick Reference

```bash
# Initialize working environment
mkdir -p /tmp/reverse-engineering-custom-encryption/{output,logs,artifacts}

# Execute primary analysis with ghidra
echo "[*] Running Reverse Engineering Custom Encryption with ghidra..."

# Validate results with ida-pro
echo "[*] Cross-validating with ida-pro..."

# Generate summary report
cat /tmp/reverse-engineering-custom-encryption/output/report.json
```

## Workflow

### Step 1: Preparation and Reconnaissance

```bash
# Identify target and gather initial intelligence
file ./target_sample
ghidra --version 2>/dev/null || echo "Install ghidra"

# Set up working directory
mkdir -p /tmp/reverse-engineering-custom-encryption/{output,logs,artifacts}
```

### Step 2: Primary Analysis

```bash
# Execute primary analysis with ghidra
# Refer to Quick Reference above for detailed commands
echo "[*] Running Reverse Engineering Custom Encryption with ghidra..."

# Log all operations
script -q /tmp/reverse-engineering-custom-encryption/logs/session.log
```

### Step 3: Deep Investigation

```bash
# Apply ida-pro for secondary analysis
echo "[*] Deep investigation with ida-pro..."

# Cross-reference findings
diff /tmp/reverse-engineering-custom-encryption/output/primary.json /tmp/reverse-engineering-custom-encryption/output/secondary.json
```

### Step 4: Documentation and Reporting

```bash
# Generate structured findings report
cat <<'EOF' > /tmp/reverse-engineering-custom-encryption/output/report.json
{
  "technique": "reverse-engineering-custom-encryption",
  "domain": "reverse-engineering",
  "tools_used": ["ghidra", "ida-pro"],
  "findings": [],
  "recommendations": []
}
EOF
```

## Detection

```yaml
title: Reverse Engineering Custom Encryption Detection
id: 3cc8032e-1ea5-473f-96fe-3be30edd8a2b
status: experimental
description: Detects suspicious activity related to reverse engineering custom encryption techniques in reverse engineering context
logsource:
  category: process_creation
  product: windows
detection:
  selection:
    CommandLine: "*reverse*engineering*"
  condition: selection
level: medium
tags:
  - attack.t1573
  - attack.defense_evasion
falsepositives:
  - Malware analysis sandbox performing automated binary inspection
```


**Detection Opportunities**

| Indicator | Source | Detection Logic |
|---|---|---|
| Reverse Engineering Custom Encryption Detection | windows/process_creation | Sigma rule (medium) |
| ATT&CK Coverage | MITRE ATT&CK | T1573 |

## Verification

- [ ] Environment and tools verified and operational
- [ ] Target samples acquired through authorized channels
- [ ] Primary analysis completed with findings documented
- [ ] Secondary validation performed with independent tooling
- [ ] All artifacts preserved in structured output directory
- [ ] Detection opportunities documented for blue team

## References

- [MITRE ATT&CK T1573](https://attack.mitre.org/techniques/T1573) — Related technique
- [ghidra Documentation](https://ghidra.org/) — Primary tooling
- [ida-pro Reference](https://ida-pro.org/) — Secondary tooling
