SOB: A Multi-Source Structured Output Benchmark for Evaluating LLM JSON Accuracy

Summary

This article introduces SOB (Structured Output Benchmark), a new multi-source benchmark for evaluating LLMs' ability to produce structured JSON data from unstructured and semi-structured sources including text, images, and audio. Unlike existing benchmarks that only check schema compliance or evaluate value correctness within a single domain, SOB measures JSON value accuracy per field across multiple source types. The benchmark tests 20+ models using 7 metrics and provides a full leaderboard, addressing the critical need for deterministic structured output in production workflows like invoice parsing, medical record processing, and PDF conversion.