Visual Iconicity Challenge: A New Benchmark for Evaluating Vision-Language Models on Sign Language Understanding
Summary
This paper introduces the Visual Iconicity Challenge, a video-based benchmark that evaluates vision-language models (VLMs) on their ability to understand iconicity in sign languages—the resemblance between linguistic form and meaning. The benchmark adapts psycholinguistic measures for three tasks: phonological sign-form prediction, transparency (inferring meaning from visual form), and graded iconicity ratings. The authors assess 17 state-of-the-art VLMs in zero- and few-shot settings on Sign Language of the Netherlands, comparing them to human baselines. The work was presented at the 2026 ACL conference.
Source
bskyVisual Iconicity Challenge: A New Benchmark for Evaluating Vision-Language Models on Sign Language Understandingdoi.orgKey quotes
· 3 pulledIconicity, the resemblance between linguistic form and meaning, is pervasive in sign languages, offering a natural testbed for visual grounding in vision–language models (VLMs).
We introduce the Visual Iconicity Challenge, a video-based benchmark that adapts psycholinguistic measures to evaluate VLMs on three tasks.
We assess 17 state-of-the-art VLMs in zero- and few-shot settings on Sign Language of the Netherlands and compare them to human baselines.
You might also wanna read
BabyVision Benchmark Reveals MLLMs Fail at Basic Visual Tasks That 3-Year-Olds Can Solve
This paper introduces BabyVision, a benchmark designed to assess core visual reasoning abilities in Multimodal LLMs (MLLMs) independent of l
arxiv.org·14d ago
DatBench: A New Framework for More Faithful and Efficient Vision-Language Model Evaluation
The article introduces DatBench, a new evaluation framework for vision-language models (VLMs) that addresses critical issues in current eval
arxiv.org·6mo ago
Using Vision-Language Models to Segment Robot Demonstration Videos into Subtask Annotations
This article presents a benchmark and field report on using Vision-Language Models (VLMs) to segment robot demonstration videos and egocentr
macrodata.co·7d ago
DataComp-VLM: A Benchmark Reveals Data Mixing Beats Filtering for Vision-Language Model Training
This paper introduces DataComp for VLMs (DCVLM), a benchmark for controlled data-centric experiments to improve Vision-Language Model (VLM)
arxiv.org·3d agoDataComp-VLM: A Benchmark Reveals Data Mixing Beats Filtering for Vision-Language Model Training
This paper introduces DataComp for VLMs (DCVLM), a benchmark for controlled data-centric experiments to improve Vision-Language Model (VLM)
arxiv.org·3d ago
Efficient Vision Encoding for Vision Language Models
Vision Language Models (VLMs) combine visual understanding with textual inputs by utilizing pretrained vision encoders and Large Language Mo
machinelearning.apple.com·11mo ago
Research: LLMs Encode Human-Labeled Problem Difficulty Better Than Model-Derived Difficulty
This research paper investigates whether large language models (LLMs) internally encode problem difficulty in alignment with human judgment.
arxiv.org·8mo ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.