New ASL Benchmark Reveals Sign Language AI Models Overlook Facial and Body Cues

Summary

This paper introduces ASL Minimal Translation Pairs (ASL-MTP), a new benchmark dataset for American Sign Language designed to evaluate how well sign language models capture linguistic phenomena. The dataset is divided into multiple types of sign language phenomena with corresponding minimal translation pairs. Using this benchmark, the authors analyze a state-of-the-art ASL-to-English translation model by ablating various input cues (manual and non-manual) during training and inference. Results indicate the model performs above chance on most phenomena but relies heavily on manual cues (hand movements) while often missing crucial non-manual cues (upper body, facial expressions), revealing significant gaps in current sign language AI models.