<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
<!-- NERF is a trademark of defconxt. -->
# Security Data Lake Pipelines — API Reference

## AWS Security Lake Integration

```python
import boto3

client = boto3.client("securitylake")

# Create custom log source
client.create_custom_log_source(
    sourceName="custom-firewall-logs",
    eventClasses=["NETWORK_ACTIVITY"],
    configuration={"crawlerConfiguration": {"roleArn": "arn:aws:iam::role/lake"}},
)

# Query via Athena
athena = boto3.client("athena")
athena.start_query_execution(
    QueryString="SELECT * FROM security_lake.cloud_trail_mgmt_1_0 LIMIT 10",
    ResultConfiguration={"OutputLocation": "s3://query-results/"},
)
```

## OCSF Schema Mapping

```python
OCSF_CLASSES = {
    1001: "File Activity",
    1007: "Process Activity",
    3002: "Authentication",
    4001: "Network Activity",
    4003: "DNS Activity",
}

# Base OCSF event envelope
ocsf_event = {
    "class_uid": 4001,
    "class_name": "Network Activity",
    "time": "2026-03-18T01:00:00Z",
    "severity_id": 1,  # 0=Unknown, 1=Info, 2=Low, 3=Medium, 4=High, 5=Critical
    "metadata": {"version": "1.1.0", "product": {"name": "firewall"}},
    "observables": [{"name": "src_ip", "type": "ip_address", "value": "10.0.1.1"}],
}
```

## Apache Parquet Writer

```python
import pyarrow as pa
import pyarrow.parquet as pq

schema = pa.schema([
    ("time", pa.string()),
    ("class_uid", pa.int32()),
    ("severity_id", pa.int32()),
    ("src_ip", pa.string()),
    ("dst_ip", pa.string()),
])

table = pa.table({"time": [...], "class_uid": [...], ...}, schema=schema)
pq.write_table(table, "output/events.parquet", compression="zstd")
```

## Exit Codes

| Code | Meaning |
|------|---------|
| 0 | Pipeline completed successfully |
| 1 | Normalization warnings (unmapped fields) |
| 2 | Ingestion or storage failure |
