<!-- Copyright (c) 2026 defconxt. All rights reserved. -->
<!-- Licensed under AGPL-3.0 — see LICENSE file for details. -->
---
name: performing-security-tool-evaluation
description: >-
  Evaluate and select security tools through structured proof-of-concept testing, scoring, and vendor assessment to support informed procurement decisions.
domain: cybersecurity
subdomain: soc-operations
tags:
  - tool-evaluation
  - vendor-assessment
  - proof-of-concept
  - procurement
  - soc-tooling
version: "1.0"
author: defconxt
license: AGPL-3.0
metadata:
  mitre-attack: []
  nist-csf: ["ID.GV-1", "PR.IP-1"]
---

# Performing Security Tool Evaluation

## Overview

Tool evaluations fail when they become vendor demo sessions instead of evidence-based assessments against real requirements. A structured POC with defined success criteria, representative data, and scored rubric produces defensible decisions and avoids shelfware.

Mode: `[MODE: BUILD]` — Requirements drive selection. Never start with a vendor.

## Evaluation Framework

| Phase | Duration | Activities | Output |
|---|---|---|---|
| Requirements | 1 week | Interview stakeholders, define use cases, set must-haves | Weighted requirements doc |
| Market scan | 1 week | Identify 5-8 candidates, eliminate by must-haves | Shortlist of 3-4 vendors |
| RFI / scripted demo | 2 weeks | Send standard scenario to all vendors, score responses | Scored RFI matrix |
| POC | 4 weeks | Deploy top 2-3 in lab with representative data | POC results per criterion |
| Reference checks | 1 week | Call 3 customer references per finalist | Reference notes |
| Decision | 1 week | Score all evidence, present to stakeholders | Vendor selection memo |

## Scoring Rubric

| Category | Weight | Criteria |
|---|---|---|
| Detection coverage | 25% | ATT&CK technique coverage, false positive rate in POC |
| Integration | 20% | SIEM/SOAR/existing stack compatibility, API quality |
| Operational overhead | 20% | Time to deploy, tuning effort, alert volume per analyst |
| Scalability | 15% | EPS capacity, cloud/on-prem flexibility, licensing model |
| Vendor stability | 10% | Financials, customer retention, roadmap credibility |
| Total cost of ownership | 10% | License + infra + FTE + training over 3 years |

## POC Test Scenarios

| Scenario | What It Tests | Pass Criteria |
|---|---|---|
| Known malware execution | Detection fidelity | Alert fires within 5 min |
| LOLBin abuse (certutil download) | Behavioral detection | Alert with correct ATT&CK tag |
| Lateral movement (SMB scan) | Network detection | Alerts for 3/5 simulation variants |
| False positive baseline | Noise level | <10 alerts per analyst per day on clean traffic |
| Deployment speed | Operational overhead | Full deploy + baseline in <2 days |
| API integration | Integration quality | Successfully ingest 1000 events/sec via API |

## Vendor Risk Assessment

| Risk Category | Questions | Red Flags |
|---|---|---|
| Financial stability | Profitable? Series? Runway? | <12 month runway, acquisition rumors |
| Data handling | Where does telemetry go? SOC 2 Type II? | No certifications, vague data policies |
| Support SLA | P1 response time? On-call coverage? | >4h P1 SLA, no 24/7 option |
| Roadmap | What's shipping in 6-12 months? | Vague answers, feature already promised for >2 years |
| Customer retention | Renewal rate? Reference willingness? | <80% renewal, <2 reachable references |

## Verification

- [ ] Requirements documented before any vendor contact
- [ ] POC environment uses representative production data (sanitized)
- [ ] Same test scenarios run against all finalists
- [ ] Scores recorded per criterion before committee discussion
- [ ] TCO calculated over 3 years including hidden costs (FTE tuning time)
- [ ] Security review of vendor's own security posture completed

## Related Techniques

- [Implementing SOC Tooling Integration](../implementing-soc-tooling-integration/SKILL.md)
- [Building SOC Metrics and KPIs](../building-soc-metrics-dashboard/SKILL.md)
