VLMs Fail Basic Spatial Test in New Multilingual Benchmark
Leading vision-language models, including proprietary systems, show near-zero capability in parsing simple spatial commands like "this" and "that," revealing a fundamental gap in their real-world…
Read the full articleYou might also wanna read
BabyVision Benchmark Reveals MLLMs Fail at Basic Visual Tasks That 3-Year-Olds Can Solve
While humans develop core visual skills long before acquiring language, contemporary Multimodal LLMs (MLLMs) still rely heavily on linguisti
arxiv.org·18d ago
DatBench: A New Framework for More Faithful and Efficient Vision-Language Model Evaluation
Empirical evaluation serves as the primary compass guiding research progress in foundation models. Despite a large body of work focused on t
arxiv.org·6mo ago
DataComp-VLM: A Benchmark Reveals Data Mixing Beats Filtering for Vision-Language Model Training
Building performant Vision-Language Models (VLMs) requires carefully curating large-scale training datasets, yet the community lacks systema
arxiv.org·7d agoDataComp-VLM: A Benchmark Reveals Data Mixing Beats Filtering for Vision-Language Model Training
Building performant Vision-Language Models (VLMs) requires carefully curating large-scale training datasets, yet the community lacks systema
arxiv.org·7d ago
Efficient Vision Encoding for Vision Language Models
Vision Language Models (VLMs) enable visual understanding alongside textual inputs. They are typically built by passing visual tokens from a
machinelearning.apple.com·11mo ago
SnapBench: A Spatial Reasoning Benchmark for LLMs Inspired by Pokémon Snap
📸 gotta find 'em all; spatial reasoning benchmark for LLMs - kxzk/snapbench
github.com·5mo ago
Visual Iconicity Challenge: A New Benchmark for Evaluating Vision-Language Models on Sign Language Understanding
Onur Keleş, Asli Ozyurek, Gerardo Ortega, Kadir Gökgöz, Esam Ghaleb. Proceedings of the 64th Annual Meeting of the Association for Computati
doi.org·3d ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.