JaiLIP: A new image-based jailbreak attack method for Vision-Language Models

This paper introduces JaiLIP (Jailbreaking with Loss-guided Image Perturbation), a novel attack method targeting Vision-Language Models (VLMs). The technique generates adversarial images by minimizing a joint objective that combines mean squared error loss between clean and adversarial images with the model's harmful-output loss. The method produces highly effective and imperceptible adversarial images that outperform existing jailbreaking techniques in generating toxic outputs. The authors evaluate their approach using standard toxicity metrics (Perspective API and Detoxify) and also demonstrate its practicality in the transportation domain beyond toxic text generation. The study highlights the security vulnerabilities of VLMs and the urgent need for robust defense mechanisms.

[Submitted on 24 Sep 2025 (v1), last revised 22 Oct 2025 (this version, v2)]11d ago2 min readenInsight

technology science

Read on arxiv.org

Key quotes

In this study, we propose Jailbreaking with Loss-guided Image Perturbation (JaiLIP), a jailbreaking attack in the image space that minimizes a joint objective combining the mean squared error (MSE) loss between clean and adversarial image with the models harmful-output loss.

Experimental results demonstrate that our method generates highly effective and imperceptible adversarial images, outperforming existing methods in producing toxicity.

Our findings emphasize the practical challenges of image-based jailbreak attacks and the need for efficient defense mechanisms for VLMs.

Among various attack vectors, recent studies have demonstrated that image-based perturbations are particularly effective in generating harmful outputs.

From the article

Vision-Language Models (VLMs) have remarkable abilities in generating multimodal reasoning tasks. However, potential misuse or safety alignment concerns of VLMs have increased significantly due to different categories of attack vectors. Among various atta

Continue reading on arxiv.org

You might also wanna read

Adversarial Poetry Functions as Universal Jailbreak Technique for Large Language Models

Research demonstrates that adversarial poetry serves as an effective universal jailbreak technique for Large Language Models (LLMs). Across

arxiv.org·7mo ago

AEGIS: An Inference-Time Defense Against Visual Synonym Jailbreak Attacks on Text-to-Image Models

This paper presents AEGIS (Adaptive Evasion Guard via Identification and Steering), a defense mechanism against Visual Synonym Attacks (VSA)

arxiv.org·4h ago

MemoAttack: A Memory-Driven Framework for Automated LLM Jailbreak Attacks

This paper introduces MemoAttack, a novel memory-driven black-box jailbreak framework for large language models (LLMs). Unlike existing meth

arxiv.org·1mo ago

Research Reveals LLM Refusal Behavior Is Controlled by a Single Direction in Model Activations

This research paper investigates the internal mechanisms of refusal behavior in large language models (LLMs). The authors demonstrate that a

arXiv.org·2mo ago

Cross-Trace Verification Protocol: A Framework for Detecting Malicious Code in AI-Generated Programs

Researchers present Cross-Trace Verification Protocol (CTVP), a novel AI control framework for detecting malicious code generated by large l