V-Pretraining: Using Adaptive Task Construction to Improve Continued Pretraining in Machine Learning
Task construction as the control surface in continued pretraining. A construction rule (c) maps each unlabeled example (xsimmathcal D_u) into a self-supervised prediction problem ((x_c,y,m)), such as…
Read the full articleYou might also wanna read
Exploring the Limits of Rich Sutton's 'Bitter Lesson' in AI Development
Recently, “the bitter lesson” is having a moment. Coined in an essay by Rich Sutton, the bitter lesson is that, “general methods that levera
dbreunig.com·11mo ago
Study Finds AI Discourse in Pretraining Data Creates Self-Fulfilling (Mis)alignment in LLMs
Pretraining corpora contain extensive discourse about AI systems, yet the causal influence of this discourse on downstream alignment remains
arxiv.org·2mo ago
Learning More from Less: Reinforcement Learning from Hindsight
arXiv:2607.09042v1 Announce Type: new Abstract: Reinforcement learning (RL) is increasingly used to post-train vision-language-action (VLA)
Exploring the Use of Randomly Generated Data for Model Pre-Training
We investigate the use of randomly generated data for the sake of pre-training a model. We justify this approach theoretically from the pers
arxiv.org·1y ago
The Bitter Lesson of AI Product Development: Why General Methods Win
What actually matters when building AI products when the world keeps changing
hex.tech·9mo ago
Balancing Domain Knowledge and General-Purpose Methods in AI Research
There is no dichotomy between domain knowledge vs. “general purpose methods that leverage data+compute”. They are both powerful tools that c
assaf-pinhasi.medium.com·1y ago
Comments
Sign in to join the conversation.
No comments yet. Be the first.