ClassicLogic: Testing AI's Compositional Grit
ClassicLogic challenges AI with complex logic puzzles to test its compositional generalization skills. The benchmark suite is a breakthrough for AI reasoning.
Read the full articleYou might also wanna read
Separating Problem Solving from Code Generation: Evaluating LLMs on Competitive Programming Through Natural-Language Editorials
Large Language Models (LLMs) increasingly succeed on competitive programming problems, yet existing evaluations conflate algorithmic reasoni
arxiv.org·3d ago
New Benchmark Uses Esoteric Programming Languages to Evaluate LLM Reasoning Abilities
EsoLang-Bench: A benchmark of 80 problems across 5 esoteric languages to evaluate genuine reasoning in LLMs.
AI-Powered Translation of Logical Foundations Textbook Achieves 350x Speed-Up in Verified Software Engineering
We present lf-lean, a verified translation of all 1,276 statements of the Logical Foundations textbook from Rocq to Lean, produced by fronti
theorem.dev·4mo ago
SkillsBench: A Benchmark for Evaluating AI Agent Skills Across Diverse Tasks
Agent Skills are structured packages of procedural knowledge that augment LLM agents at inference time. Despite rapid adoption, there is no
arxiv.org·4mo ago
Study Reveals Large Reasoning Models Fail at Complex Problem-Solving Despite Strong Benchmark Performance
Large language models (LLMs) have shown significant progress in reasoning tasks. However, recent studies show that transformers and LLMs fai
arxiv.org·8mo agoLean4 - How the theorem prover works and why it's the new competitive edge in AI
As large language models (LLMs) permeate high-stakes industries like finance and medicine, the tolerance for “hallucinations”—confident but
Comments
Sign in to join the conversation.
No comments yet. Be the first.