LeJEPA: A Theoretically Grounded Self-Supervised Learning Framework for AI Representation Learning
By
nothrowaways
Right out the toaster. Reliable, with some real depth.
Summary
Researchers present LeJEPA, a theoretically grounded self-supervised learning framework that addresses limitations in Joint-Embedding Predictive Architectures (JEPAs). The approach identifies the isotropic Gaussian as the optimal embedding distribution and introduces Sketched Isotropic Gaussian Regularization (SIGReg) to constrain embeddings to this ideal. LeJEPA offers practical benefits including linear time/memory complexity, stability across architectures and domains, elimination of heuristics like stop-gradient and teacher-student setups, and a simple implementation requiring only about 50 lines of code. The method was validated across 10+ datasets and 60+ architectures, achieving 79% accuracy on ImageNet-1k with a ViT-H/14 model.
Key quotes
· 5 pulledLearning manipulable representations of the world and its dynamics is central to AI.
We present a comprehensive theory of JEPAs and instantiate it in LeJEPA, a lean, scalable, and theoretically grounded training objective.
First, we identify the isotropic Gaussian as the optimal distribution that JEPAs' embeddings should follow to minimize downstream prediction risk.
LeJEPA offers numerous theoretical and practical benefits: (i) single trade-off hyperparameter, (ii) linear time and memory complexity, (iii) stability across hyper-parameters, architectures and domains, (iv) heuristics-free, e.g., no stop-gradient, no teacher-student, no hyper-parameter schedulers.
We hope that the simplicity and theory-friendly ecosystem offered by LeJEPA will reestablish self-supervised pre-training as a core pillar of AI research.
You might also wanna read
Parametric Memory Law: A Quantitative Framework for Understanding LoRA Memory Capacity in LLMs
This research paper introduces the Parametric Memory Law, a quantitative framework for understanding how Low-Rank Adaptation (LoRA) enables
arxiv.org·1d ago
Bridge-Garden Theory Explains Why Mixing Hard and Soft Labels Improves Knowledge Distillation for LLMs
This research paper investigates knowledge distillation (KD) for language models, specifically why mixing hard labels (sampled tokens) and s
arxiv.org·3d ago
Researchers Develop Method to Predict Real-Time Progress in Reasoning Language Models
This research paper investigates whether real-time progress prediction is feasible for reasoning language models that use long latent chains
arxiv.org·3d ago
AI systems achieve 50% pass rate in standard three-party Turing test, study finds
This paper demonstrates that three current AI systems (when suitably prompted) achieve a pass rate of at least 50% in a standard three-party
pnas.org·4d ago
RICP: A Teacher-Student Framework for Retrieved In-Context Principles from Mistakes in LLMs
This paper introduces Retrieved In-Context Principles (RICP), a novel teacher-student framework for improving Large Language Models (LLMs) t
arxiv.org·4d ago
HSIR: New Method Improves Self-Improvement Training for Large Reasoning Models
This research paper identifies two key problems in self-improvement training for Large Reasoning Models (LRMs): data imbalance (too many sim
arxiv.org·5d ago