Benchmarks in Leipzig: LLMs Solve 98 of 100 Research-Level Math Problems

A group of 49 mathematicians compiled a dataset of 100 research-level mathematics questions with known answers during a workshop at the Max Planck Institute in Leipzig. They evaluated these questions against state-of-the-art LLMs in three stages. Initially, 41 questions remained unsolved by five LLMs in a single attempt. After 20 runs per model with three models, only 16 remained unsolved. Finally, with two heavy-thinking models over 3 runs, only 2 questions remained unsolved. The results demonstrate that LLMs' mathematical reasoning capabilities are becoming increasingly impressive.