<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 -->
<!-- NERF is a trademark of defconxt. -->
---
name: performing-container-incident-response
description: >-
  Performs incident response in containerized environments including Docker and
  Kubernetes, covering runtime threat detection, container forensics, pod isolation,
  image analysis, and orchestrator-level investigation.
domain: cybersecurity
subdomain: incident-response
tags:
  - container-security
  - kubernetes
  - docker
  - container-forensics
  - runtime-security
  - pod-isolation
version: '1.0'
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1610"]
---

# Performing Container Incident Response

## Overview

Container incident response requires different techniques than traditional IR.
Containers are ephemeral — a compromised pod may be terminated before evidence
is collected. This skill covers Docker container forensics, Kubernetes pod
investigation and isolation, container image analysis, runtime security event
triage with Falco, and evidence preservation from ephemeral workloads.

Key challenge: container environments operate at a pace where traditional
evidence collection is too slow. Pre-instrumentation with runtime security
tools (Falco, Tetragon, Sysdig) is essential.

## Prerequisites

| Tool | Purpose | Install |
|------|---------|---------|
| kubectl | Kubernetes cluster access | `curl -LO https://dl.k8s.io/release/stable.txt` |
| crictl | Container runtime CLI | `apt install cri-tools` |
| Falco | Runtime threat detection | `helm install falco falcosecurity/falco` |
| Trivy | Image vulnerability scanning | `apt install trivy` |
| docker | Container management | Pre-installed |
| cosign | Image signature verification | `go install github.com/sigstore/cosign/v2/cmd/cosign@latest` |

## Key Concepts

### Container Attack Surface

| Vector | Description |
|--------|-------------|
| Vulnerable image | Known CVEs in base image or dependencies |
| Misconfigured pod | Privileged mode, hostPID, hostNetwork |
| Exposed API server | Unauthenticated kubelet or API access |
| Supply chain | Compromised base image or CI/CD pipeline |
| Container escape | Kernel exploit or misconfigured mounts |
| Lateral movement | Service mesh abuse, pod-to-pod network access |

### Evidence Volatility in Containers

```
Most Volatile (collect first):
1. Running process list and network connections (inside container)
2. Container filesystem diff (changes from base image)
3. Runtime security events (Falco/Tetragon logs)
4. Pod logs and events
5. Container image layers

Least Volatile:
6. Kubernetes audit logs
7. Node-level logs and journal
8. Container registry history
```

## Workflow

### Step 1 — Identify Compromised Containers

```bash
# List pods with recent restarts or anomalous status
kubectl get pods -A --sort-by='.status.containerStatuses[0].restartCount' | tail -20

# Check Falco alerts for runtime threats
kubectl logs -n falco -l app.kubernetes.io/name=falco --tail=100 | \
  grep -E "Warning|Error|Critical"

# Find pods running as root
kubectl get pods -A -o json | jq '.items[] | select(.spec.containers[].securityContext.runAsUser == 0) | .metadata.name'

# Check for privileged containers
kubectl get pods -A -o json | jq '.items[] | select(.spec.containers[].securityContext.privileged == true) | {ns: .metadata.namespace, name: .metadata.name}'
```

### Step 2 — Preserve Container Evidence

```bash
# Capture running processes inside container
kubectl exec -n $NS $POD -- ps auxf > /evidence/pod_processes.txt

# Capture network connections
kubectl exec -n $NS $POD -- cat /proc/net/tcp > /evidence/pod_connections.txt
kubectl exec -n $NS $POD -- ss -tulnp > /evidence/pod_sockets.txt 2>/dev/null

# Export container filesystem diff
CONTAINER_ID=$(crictl ps --name $CONTAINER_NAME -q)
docker diff "$CONTAINER_ID" > /evidence/container_diff.txt

# Export full container filesystem
docker export "$CONTAINER_ID" > /evidence/container_fs.tar

# Save pod logs
kubectl logs -n $NS $POD --all-containers --timestamps > /evidence/pod_logs.txt

# Save pod description and events
kubectl describe pod -n $NS $POD > /evidence/pod_describe.txt
kubectl get events -n $NS --field-selector involvedObject.name=$POD > /evidence/pod_events.txt
```

### Step 3 — Isolate Compromised Pod

```bash
# Apply network policy to isolate pod (deny all ingress/egress)
cat <<'EOF' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ir-isolate-pod
  namespace: $NS
spec:
  podSelector:
    matchLabels:
      app: $COMPROMISED_APP
  policyTypes:
  - Ingress
  - Egress
EOF

# Cordon node to prevent new scheduling
kubectl cordon $NODE_NAME

# Label pod for IR tracking
kubectl label pod -n $NS $POD incident=INC-2026-0342 --overwrite
```

### Step 4 — Analyze Container Image

```bash
# Scan image for vulnerabilities
trivy image --severity HIGH,CRITICAL $IMAGE:$TAG

# Inspect image layers for suspicious additions
docker history --no-trunc $IMAGE:$TAG

# Verify image signature
cosign verify --key cosign.pub $IMAGE:$TAG

# Compare running image digest to registry
RUNNING_DIGEST=$(kubectl get pod -n $NS $POD -o jsonpath='{.status.containerStatuses[0].imageID}')
REGISTRY_DIGEST=$(cosign triangulate $IMAGE:$TAG | head -1)
echo "Running: $RUNNING_DIGEST"
echo "Registry: $REGISTRY_DIGEST"
```

### Step 5 — Investigate Kubernetes Audit Logs

```bash
# Search for suspicious API calls
cat /var/log/kubernetes/audit.log | jq 'select(.verb == "create" and .objectRef.resource == "pods") | {user: .user.username, ns: .objectRef.namespace, time: .requestReceivedTimestamp}'

# Detect secret access
cat /var/log/kubernetes/audit.log | jq 'select(.objectRef.resource == "secrets" and .verb == "get") | {user: .user.username, secret: .objectRef.name, time: .requestReceivedTimestamp}'

# Find exec into pods (attacker accessing containers)
cat /var/log/kubernetes/audit.log | jq 'select(.objectRef.subresource == "exec") | {user: .user.username, pod: .objectRef.name, time: .requestReceivedTimestamp}'
```

## Detection

```yaml
title: Container Incident Response Detection
id: b931fd90-9bcd-4d2a-b08b-a2395a727643
status: experimental
description: Detects suspicious activity related to performing container incident response techniques in incident response context
logsource:
  category: process_creation
  product: windows
detection:
  selection:
    CommandLine: "*performing*container*"
  condition: selection
level: high
tags:
  - attack.t1610
  - attack.impact
falsepositives:
  - Incident response team running authorized forensic collection tools
```

## Verification

- [ ] Compromised containers identified via runtime alerts
- [ ] Container processes and network state captured before termination
- [ ] Container filesystem exported for offline analysis
- [ ] Pod logs and events preserved
- [ ] Network policy applied to isolate compromised workloads
- [ ] Container image scanned for vulnerabilities and verified for tampering
- [ ] Kubernetes audit logs reviewed for unauthorized API access
- [ ] All evidence hashed and chain of custody documented

## References

- Kubernetes Security Documentation: https://kubernetes.io/docs/concepts/security/
- Falco Runtime Security: https://falco.org/docs/
- MITRE ATT&CK Containers Matrix: https://attack.mitre.org/matrices/enterprise/containers/
- NIST SP 800-190: Application Container Security Guide
- CIS Kubernetes Benchmark
