When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability
arXiv:2607.08535v1 Announce Type: new Abstract: An LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed. We treat this…
Read the full articleYou might also wanna read
Systematic evaluation of 21 LLM-as-a-Judge models reveals reliability flaws and position bias across 541,000 judgments
LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agree
arxiv.org·17d agoSystematic evaluation of 21 LLM-as-a-Judge models reveals reliability flaws and position bias across 541,000 judgments
LLM-as-a-Judge has become the dominant evaluation paradigm for language models, but judge validation in practice relies on exact-match agree
arxiv.org·17d ago
Research on LLM Output Drift in Financial Workflows: Quantifying Consistency Across Model Sizes
Financial institutions deploy Large Language Models (LLMs) for reconciliations, regulatory reporting, and client communications, but nondete
arxiv.org·7mo ago
DecompR: A Method for Reducing Weighting Noise in Multi-Stakeholder LLM Alignment
Multi-stakeholder tasks require one output to satisfy users with conflicting preferences. Holistic LLM judges conflate utility estimation an
arxiv.org·1mo ago
LLM-as-a-Verifier: A probabilistic verification framework for scaling LLM correctness assessment
Scaling pre-training, post-training, and test-time compute have become the central paradigms for improving the capabilities of LLMs. In this
arxiv.org·2d agoLLM-as-a-Verifier: A probabilistic verification framework for scaling LLM correctness assessment
Scaling pre-training, post-training, and test-time compute have become the central paradigms for improving the capabilities of LLMs. In this
arxiv.org·2d ago
LLM-as-a-Judge Simply Explained: The Complete Guide to Run LLM Evals at Scale
Complete guide to LLM-as-a-Judge: how it works, single-output vs pairwise scoring, G-Eval, DAG, prompting techniques, and how to use LLM jud
BINEVAL: A Binary Question Framework for Interpretable LLM Evaluation and Self-Improvement
Evaluating LLM outputs remains a major bottleneck in NLP: human evaluation is expensive and slow, lexical metrics correlate poorly with huma
arxiv.org·12d agoBINEVAL: A Binary Question Framework for Interpretable LLM Evaluation and Self-Improvement
Evaluating LLM outputs remains a major bottleneck in NLP: human evaluation is expensive and slow, lexical metrics correlate poorly with huma
arxiv.org·12d ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.