Prompt Injection Explained as Role Confusion in LLMs

Summary

This article presents a theory that prompt injection attacks on LLMs succeed because models identify roles (like "user" vs "assistant") through writing style rather than explicit tags. The authors introduce "CoT Forgery," a new attack that injects fake chain-of-thought reasoning that models mistake for their own internal thoughts. They argue that understanding roles as perceived by LLMs is key to predicting and preventing these attacks, and propose a research agenda for a "science of roles" in AI safety.

Source

Hacker NewsPrompt Injection Explained as Role Confusion in LLMsrole-confusion.github.io