<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: implementing-ai-content-filtering
description: >-
  Implement content filtering pipelines for AI applications covering harmful
  content detection, topic restriction, output format validation, and policy
  compliance enforcement using classifiers and rule engines.
domain: cybersecurity
subdomain: ai-llm-security
tags:
  - content-filtering
  - safety
  - guardrails
  - policy-enforcement
  - llm-security
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: ["T1059"]
  owasp-llm: ["LLM02", "LLM09"]
  mitre-atlas: ["AML.T0048.004"]
  tools: ["python3", "transformers", "openai", "regex"]
---

# Implementing AI Content Filtering

## Overview

AI content filtering enforces safety and compliance policies on both model inputs
and outputs. This covers harmful content detection (violence, hate speech), topic
restriction, output format validation, and regulatory compliance. A robust pipeline
combines rule-based filters, ML classifiers, and LLM-as-judge evaluation.

## Prerequisites

| Tool / Requirement | Details |
|---|---|
| `python3` | Security tooling |
| `transformers` | Security tooling |
| `openai (for LLM-as-judge)` | Security tooling |
| `re` | Security tooling |
| Defined content policy with prohibited categories | Environment requirement |
| Test corpus covering edge cases and adversarial inputs | Environment requirement |
| Baseline acceptable false positive/negative rates | Environment requirement |

## Key Concepts

- **Content taxonomy**: Structured classification of prohibited content categories
- **Multi-layer filtering**: Combining regex, ML classifiers, and LLM judges
- **LLM-as-judge**: Using a separate LLM to evaluate outputs against content policy
- **Topic restriction**: Preventing model from generating content outside approved domains
- **Format validation**: Ensuring output matches expected JSON schema or template

## Workflow

### Step 1: Content Category Taxonomy

```python
from dataclasses import dataclass
from enum import Enum

class ContentCategory(Enum):
    SAFE = "safe"
    VIOLENCE = "violence"
    HATE_SPEECH = "hate_speech"
    SELF_HARM = "self_harm"
    ILLEGAL_ACTIVITY = "illegal_activity"
    PII_DISCLOSURE = "pii_disclosure"
    PROMPT_LEAK = "prompt_leak"

@dataclass
class FilterResult:
    category: ContentCategory
    confidence: float
    action: str  # "pass", "flag", "block"
    reason: str

def determine_action(category: ContentCategory, confidence: float) -> str:
    block_categories = {
        ContentCategory.VIOLENCE, ContentCategory.SELF_HARM,
        ContentCategory.ILLEGAL_ACTIVITY,
    }
    if category in block_categories and confidence > 0.7:
        return "block"
    if category != ContentCategory.SAFE and confidence > 0.5:
        return "flag"
    return "pass"
```

### Step 2: Rule-Based Content Filter

```python
import re

CONTENT_RULES = {
    "illegal_activity": [
        re.compile(r"how\s+to\s+(?:make|build)\s+(?:a\s+)?(?:bomb|weapon)", re.IGNORECASE),
        re.compile(r"(?:synthesize|manufacture)\s+(?:drugs|meth)", re.IGNORECASE),
    ],
    "pii_disclosure": [
        re.compile(r"\b\d{3}-\d{2}-\d{4}\b"),
        re.compile(r"\b\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4}\b"),
    ],
}

def rule_based_filter(text: str) -> list[dict]:
    results = []
    for category, patterns in CONTENT_RULES.items():
        for pat in patterns:
            if pat.search(text):
                results.append({"category": category, "pattern": pat.pattern[:60], "action": "block"})
                break
    return results
```

### Step 3: LLM-as-Judge Evaluation

```python
import openai

def llm_judge_filter(text: str, policy: str) -> dict:
    client = openai.OpenAI()
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": f"You are a content policy judge. Policy:\n{policy}\n"
                "Respond with JSON: {{\"compliant\": bool, \"reason\": str}}"},
            {"role": "user", "content": f"Evaluate this content:\n{text[:1000]}"},
        ],
        response_format={"type": "json_object"},
        temperature=0.0,
    )
    import json
    judgment = json.loads(response.choices[0].message.content)
    return {
        "compliant": judgment["compliant"],
        "action": "pass" if judgment["compliant"] else "block",
        "reason": judgment.get("reason", ""),
    }
```

## Detection

```yaml
title: Ai Content Filtering Detection
id: c4f8ed59-f6c3-49c2-b618-0abd1b1c1b80
status: experimental
description: Detects suspicious activity related to implementing ai content filtering techniques in ai llm security context
logsource:
  category: application
  product: webserver
detection:
  selection:
    EventType: error
  condition: selection
level: medium
tags:
  - attack.t1059
  - attack.initial_access
falsepositives:
  - Automated AI model training and inference pipeline operations
```


**Detection Opportunities**

| Indicator | Source | Detection Logic |
|---|---|---|
| Ai Content Filtering Detection | webserver/application | Sigma rule (medium) |
| ATT&CK Coverage | MITRE ATT&CK | T1059 |

## Verification

- [ ] Content taxonomy covers all applicable policy categories
- [ ] Rule-based filter detects known prohibited patterns
- [ ] ML classifier evaluated against test corpus with measured accuracy
- [ ] LLM-as-judge tested for policy compliance evaluation
- [ ] False positive rate measured and within acceptable threshold
- [ ] Pipeline latency profiled and within SLA

## References

- [OWASP LLM02 — Insecure Output Handling](https://genai.owasp.org/llmrisk/llm02-insecure-output-handling/)
- [OpenAI Content Moderation Guide](https://platform.openai.com/docs/guides/moderation)
- [Perspective API — Toxic Content Detection](https://perspectiveapi.com/)
- [NIST AI RMF — Trustworthy AI](https://airc.nist.gov/AI_RMF_Playbook)
