<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: performing-security-data-quality-assessment
description: >-
  Assess security data quality across completeness, freshness, accuracy,
  consistency, and schema conformance to ensure detection pipelines receive
  trustworthy telemetry for threat identification.
domain: cybersecurity
subdomain: security-data-engineering
tags:
  - data-quality
  - completeness
  - freshness
  - accuracy
  - validation
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  frameworks: ["NIST 800-92", "CIS Controls v8"]
  tools: ["python3", "great-expectations", "dbt", "duckdb"]
  mitre-attack: []
---

# Performing Security Data Quality Assessment

## Overview

Security detection is only as reliable as the underlying data. Data quality
assessment validates that log sources deliver complete, timely, accurate, and
schema-conformant events. This skill covers quality dimension measurement,
automated expectation suites, freshness monitoring, and gap alerting.

## Prerequisites

| Requirement | Details |
|---|---|
| Access to security event storage (SIEM, data lake, database) | Required |
| Baseline understanding of expected log volumes per source | Required |
| Python 3.10+ with data quality libraries | Required |

## Key Concepts

- **Completeness**: Are all expected fields populated?
- **Freshness**: Is data arriving within acceptable latency?
- **Accuracy**: Do field values match expected formats and ranges?
- **Consistency**: Are events uniform across similar sources?
- **Volume anomaly**: Unexpected drops or spikes in event count

## Workflow

### Step 1: Define Quality Expectations

```python
from dataclasses import dataclass

@dataclass
class QualityExpectation:
    """Single data quality check definition."""
    dimension: str  # completeness, freshness, accuracy, consistency
    field: str
    check: str  # not_null, regex_match, range, freshness_sla
    threshold: float  # minimum pass rate (0.0 - 1.0)
    parameters: dict | None = None
```

### Step 2: Measure Quality Dimensions

```python
def measure_completeness(events: list[dict], required_fields: list[str]) -> dict:
    """Calculate field completeness rates."""
    total = len(events)
    if total == 0:
        return {"completeness": {}, "total": 0}
    scores = {}
    for field in required_fields:
        present = sum(1 for e in events if e.get(field) is not None)
        scores[field] = round(present / total, 4)
    return {"completeness": scores, "total": total}
```

### Step 3: Monitor Freshness SLAs

```sql
-- Detect stale log sources (no events in last 15 minutes)
SELECT
    source_type,
    MAX(time_dt) AS last_event,
    CURRENT_TIMESTAMP - MAX(time_dt) AS staleness
FROM security.events
GROUP BY source_type
HAVING staleness > INTERVAL '15 minutes';
```

### Step 4: Generate Quality Report

```bash
node agent.js assess --source auth_events --window 24h --output quality.json
node agent.js freshness --sla 15m --alert-on-breach
```

## References

- [Great Expectations](https://greatexpectations.io/) — Data quality framework
- [NIST SP 800-92](https://csrc.nist.gov/publications/detail/sp/800-92/final)
- [Monte Carlo Data Observability](https://www.montecarlodata.com/)
