14x Lower Gradient Variance: What GRADE Reveals About LLM Alignment

According to the paper 'GRADE: Replacing Policy Gradients with Backpropagation for LLM Alignment' by Lukas Abrie Nel, replacing PPO with a fully differentiable approach reduces gradient variance by 14x and improves alignment performance by 50%. This means we can finally align our LLMs with the stability of supervised learning instead of wrestling with PPO's notorious instability.