Research Study: AI Agents vs Human Cybersecurity Professionals in Penetration Testing

Summary

This research paper presents the first comprehensive evaluation comparing AI agents to human cybersecurity professionals in real-world penetration testing. The study tested ten cybersecurity professionals alongside six existing AI agents and ARTEMIS, a new multi-agent framework developed by the researchers. In a live enterprise environment with ~8,000 hosts across 12 subnets, ARTEMIS placed second overall, discovering 9 valid vulnerabilities with an 82% valid submission rate, outperforming 9 of 10 human participants. The research found AI agents offer advantages in systematic enumeration, parallel exploitation, and cost-effectiveness ($18/hour vs $60/hour for professionals), but also identified key gaps including higher false-positive rates and struggles with GUI-based tasks.