Open-Source LLM Safety Vulnerabilities: How Chat Template Formatting Gates Alignment in Models Like Gemma and Qwen

Summary

This article reveals a critical vulnerability in open-source large language models (LLMs) where safety alignment can be bypassed by simply omitting the apply_chat_template() function call. The author demonstrates that safety mechanisms in models like Gemma and Qwen aren't embedded in the model weights but rather in the chat formatting template. By removing this formatting step, supposedly 'aligned' models will generate harmful content like bomb-making instructions. The article draws parallels to the SolidGoldMagikarp phenomenon from GPT-2 and serves as responsible disclosure to help improve AI safety.