SFT Overtraining Causes Rank Inversion in RLVR via Entropy Collapse
By
[Submitted on 16 Jun 2026 (v1), last revised 22 Jun 2026 (this version, v2)]
Summary
This research paper investigates a failure mode in Reinforcement Learning from Verifiable Rewards (RLVR) applied to code generation models. The authors show that selecting the Supervised Fine-Tuning (SFT) checkpoint with the highest pass@1 for Group Relative Policy Optimization (GRPO) can backfire when SFT overtraining compresses the rollout distribution. For binary rewards, when early GRPO drives the probability of success below a critical threshold, most groups have identical rewards and provide no relative signal. Experiments on Qwen2.5-Coder-3B and DeepSeek-Coder-6.7B show that deeper SFT training increases pre-RL pass@1 but can lead to rank inversion where GRPO performance degrades (Qwen pass@10 dropped from 0.806 to 0.481). The paper proposes a two-stage diagnostic combining pre-RL entropy triage with early GRPO entropy monitoring to flag high-risk checkpoints. Standard fixes like KL regularization and label smoothing did not resolve the issue.
Source
bskySFT Overtraining Causes Rank Inversion in RLVR via Entropy Collapsearxiv.orgKey quotes
· 4 pulledThe standard heuristic of selecting the SFT checkpoint with the highest pass@1 for GRPO can fail when SFT compresses the rollout distribution.
On Qwen, pre RL pass@1 rises with SFT depth, but peak GRPO pass@10 falls from $0.806$ to $0.481$ (3 seed mean, $n{=}20$); pre RL entropy is positively associated with the GRPO outcome ($ρ{=}{+}0.69$).
A two stage diagnostic, combining pre RL entropy triage with an early GRPO entropy monitor, flags high risk checkpoints and can stop failing runs early.
Simple KL to reference regularisation and label smoothing variants do not rescue the collapsed Qwen checkpoint in our setting, suggesting the failure is not a trivial GRPO hyperparameter artefact.
You might also wanna read
Study Finds Single Transformer Layer Can Match Full-Parameter RL Training in LLMs
This research paper challenges the common assumption that reinforcement learning (RL) post-training for large language models (LLMs) require
arxiv.org·6d agoStudy Finds Single Transformer Layer Can Match Full-Parameter RL Training in LLMs
This research paper challenges the common assumption that reinforcement learning (RL) post-training for large language models (LLMs) require
arxiv.org·6d ago
Supervised Fine-Tuning as Reinforcement Learning: Introducing Importance-Weighted SFT
The article explores the connection between supervised fine-tuning (SFT) of large language models and reinforcement learning (RL), arguing t
arxiv.org·11mo ago
Reward-Channel Addiction: How Visible Incentives Can Break AI Safety Alignment
This research paper introduces the concept of "reward-channel addiction" in reinforcement learning agents. The authors demonstrate that when
arxiv.org·15d ago
LLM-as-a-Verifier: A probabilistic verification framework for scaling LLM correctness assessment
This paper introduces LLM-as-a-Verifier, a general-purpose verification framework that uses LLMs to determine the correctness of solutions i
arxiv.org·22h agoLLM-as-a-Verifier: A probabilistic verification framework for scaling LLM correctness assessment
This paper introduces LLM-as-a-Verifier, a general-purpose verification framework that uses LLMs to determine the correctness of solutions i
arxiv.org·22h ago
Exploring RLHF on every prompt for local coding models
A Hacker News user explores the idea of using Reinforcement Learning from Human Feedback (RLHF) on every prompt with a medium-sized local mo
news.ycombinator.com·23d ago
Cross-Trace Verification Protocol: A Framework for Detecting Malicious Code in AI-Generated Programs
Researchers present Cross-Trace Verification Protocol (CTVP), a novel AI control framework for detecting malicious code generated by large l
arxiv.org·5mo ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.