SIR-Bench: A Benchmark for Evaluating Autonomous Security Incident Response Agents

Summary

Researchers introduce SIR-Bench, a comprehensive benchmark for evaluating autonomous security incident response agents. The benchmark consists of 794 test cases derived from 129 anonymized real-world incident patterns with expert-validated ground truth. SIR-Bench distinguishes between genuine forensic investigation and simple alert parroting by measuring not just triage accuracy but also novel evidence discovery through active investigation. The framework uses Once Upon A Threat (OUAT) to replay real incidents in controlled cloud environments, producing authentic telemetry. Evaluation uses three metrics: triage accuracy, novel finding discovery, and tool usage appropriateness, assessed through an adversarial LLM-as-Judge that requires concrete forensic evidence. Initial evaluation shows 97.1% true positive detection, 73.4% false positive rejection, and 5.67 novel key findings per case.