Separating Problem Solving from Code Generation: Evaluating LLMs on Competitive Programming Through Natural-Language Editorials

This paper argues that evaluations of Large Language Models (LLMs) on competitive programming tasks conflate two distinct abilities: algorithmic problem-solving and code-level implementation. The authors propose using natural-language editorials as an intermediate step to separate these skills. Their experiments show that generating editorials before code improves solve rates for some LLMs, with larger gains when using expertly written "gold" editorials. However, models still struggle with implementation, and the gap between generated and gold editorials reveals a persistent problem-solving bottleneck. The study introduces a dataset of 83 ICPC-style problems with gold editorials and test suites, evaluates 19 LLMs, and validates an LLM-as-a-judge protocol for scalable evaluation. The authors recommend that future benchmarks explicitly separate problem solving from implementation.