<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
<!-- NERF is a trademark of defconxt. -->

---
technique: performing-security-data-migration
subdomain: security-data-engineering
title: "Performing Security Data Migration"
description: "Techniques and methodologies for performing security data migration"
tags:
  - data-migration
  - ETL
  - platform-migration
  - data-integrity
  - compliance
metadata:
  mitre-attack: []
---

# Performing Security Data Migration

> Plan and execute migrations of security data between platforms, formats, and environments while maintaining data integrity, chain of custody, and compliance.

## Overview

Security data migration involves moving logs, alerts, detection rules, and forensic artifacts between SIEM platforms, cloud environments, or storage tiers. This technique covers migration planning, data validation, rollback strategies, and chain-of-custody preservation.

## Core Concepts

- **Schema mapping**: Translate field names and types between source and target platforms
- **Data validation**: Verify record counts, checksums, and content integrity post-migration
- **Incremental migration**: Migrate data in batches to minimize downtime and risk
- **Chain of custody**: Maintain cryptographic proof that data was not altered during migration
- **Rollback strategy**: Define clear rollback procedures for each migration phase

## Workflow

1. Inventory source data: volume, format, schema, and dependencies
2. Map source schema to target platform schema
3. Build and test migration pipeline in staging environment
4. Execute pilot migration with representative data subset
5. Validate pilot results against integrity checksums
6. Execute incremental production migration in phases
7. Run parallel validation during cutover window
8. Verify complete data transfer with reconciliation reports
9. Decommission source after validation hold period
10. Document migration for compliance and audit trail

## Implementation

```python
import hashlib
import json
from dataclasses import dataclass, field
from datetime import datetime
from typing import Any, Optional


@dataclass
class MigrationRecord:
    record_id: str
    source_hash: str
    target_hash: str | None = None
    status: str = "pending"
    migrated_at: datetime | None = None


class SecurityDataMigrator:
    """Manage security data migration with integrity verification."""

    def __init__(self, source_name: str, target_name: str) -> None:
        self.source_name = source_name
        self.target_name = target_name
        self.records: list[MigrationRecord] = []

    def prepare_record(self, record_id: str, data: dict) -> MigrationRecord:
        source_hash = hashlib.sha256(json.dumps(data, sort_keys=True).encode()).hexdigest()
        record = MigrationRecord(record_id=record_id, source_hash=source_hash)
        self.records.append(record)
        return record

    def migrate_record(self, record_id: str, transformed_data: dict) -> bool:
        record = next((r for r in self.records if r.record_id == record_id), None)
        if not record:
            return False
        record.target_hash = hashlib.sha256(
            json.dumps(transformed_data, sort_keys=True).encode()
        ).hexdigest()
        record.status = "migrated"
        record.migrated_at = datetime.utcnow()
        return True

    def validate(self) -> dict:
        total = len(self.records)
        migrated = sum(1 for r in self.records if r.status == "migrated")
        failed = sum(1 for r in self.records if r.status == "failed")
        return {"total": total, "migrated": migrated, "failed": failed, "pending": total - migrated - failed}

    def reconciliation_report(self) -> list[dict]:
        return [
            {"id": r.record_id, "status": r.status, "integrity": r.source_hash == r.target_hash if r.target_hash else None}
            for r in self.records
        ]
```

## Best Practices

- Never migrate without a validated rollback plan—test rollback before production
- Use checksums at record level, not just batch level, to catch partial corruption
- Maintain parallel read access during migration to avoid detection gaps
- Document schema mappings for audit—regulators want proof of data integrity
- Plan for schema evolution—target platform may require field transformations
- Run migration during low-alert periods to minimize SOC disruption

## Migration Phase Checklist

| Phase | Key Actions | Validation Criteria |
|---|---|---|
| Planning | Schema mapping, volume estimate, rollback plan | Schema delta documented, rollback tested |
| Staging | Pipeline build, pilot migration, integrity check | Pilot checksums match, zero record loss |
| Cutover preparation | Parallel read enabled, window scheduled | SIEM coverage confirmed during migration |
| Production migration | Incremental batches, real-time monitoring | Per-batch reconciliation reports passing |
| Validation hold | Compare source vs target counts and hashes | Reconciliation report: 0 failed records |
| Decommission | Source access revoked, documentation filed | Audit trail complete |

## Data Integrity Controls

| Control | Scope | Implementation |
|---|---|---|
| Record-level hash | Every migrated record | SHA-256 of normalized record content |
| Batch reconciliation | Per migration batch | Count + hash comparison source vs target |
| Schema validation | Post-migration sample | Spot-check 0.1% of records against schema |
| Retention verification | Compliance-sensitive data | Confirm retention policy applied in target |
| Chain of custody log | Full migration | Immutable audit log with operator identity |

## Related Techniques

- [Implementing Security Data Backup Strategy](../implementing-security-data-backup-strategy/SKILL.md)
- [Implementing Security Data Classification](../implementing-security-data-classification/SKILL.md)
