GeoSD: A More Reliable Path for Language Model Training
GeoSD offers a novel approach to language model training, addressing critical issues in on-policy distillation by introducing geometric self-distillation. This method enhances model accuracy…
Read the full articleYou might also wanna read
RLCSD: A Contrastive Self-Distillation Method to Fix Style Drift in Reasoning Models
On-policy self-distillation (OPSD) provides dense, token-level supervision for reasoning models by aligning a model's own distribution with
arxiv.org·27d ago
Feedback Distillation: A New Training Method for Improving LLM Reasoning in Theorem Proving
Post-training for reasoning models typically combines supervised fine-tuning with reinforcement learning from verifiable rewards, most commo
arxiv.org·1mo ago
Proxy-KD: A Novel Method for Knowledge Distillation from Black-Box Large Language Models
Given the exceptional performance of proprietary large language models (LLMs) like GPT-4, recent research has increasingly focused on boosti
arxiv.org·11d ago
Mathematical Foundations of High-Dimensional Concept Representation in Language Models
In a recent 3Blue1Brown video series on transformer models, Grant Sanderson posed a fascinating question: How can a relatively modest embedd
nickyoder.com·9mo ago
Bridge-Garden Theory Explains Why Mixing Hard and Soft Labels Improves Knowledge Distillation for LLMs
Knowledge distillation (KD) transfers knowledge from a large teacher model to a smaller student. In language modeling, the student is traine
arxiv.org·1mo ago
Dispersion loss counteracts embedding condensation to improve small language model generalization
Dispersion loss counteracts embedding condensation and improves generalization in small language models (ICML 2026).
chenliu-1996.github.io·6d ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.