Parcae: A Stable Looped Language Model Architecture with Predictable Scaling Laws
Traditional fixed-depth architectures scale quality by increasing training FLOPs, typically through increased parameterization, at the expense of a higher memory footprint, or data. A potential…
Read the full articleYou might also wanna read
Parcae: Doing more with fewer parameters using stable looped models
Parcae is a stable looped language model that matches the quality of a Transformer twice its size — a 770M model reaching 1.3B-level perform
PEFT-Arena: A Benchmark Evaluating Parameter-Efficient Finetuning Through the Stability-Plasticity Dilemma
Parameter-efficient finetuning (PEFT) has become the standard approach for adapting large language models, yet evaluations largely emphasize
arxiv.org·1mo ago
Ouro: Looped Language Models That Build Reasoning into Pre-Training Through Latent Space Iteration
Modern LLMs are trained to "think" primarily via explicit text generation, such as chain-of-thought (CoT), which defers reasoning to post-tr
arxiv.org·6mo ago
δ-mem: A Compact Online Memory Mechanism for Efficient Long-Context LLM Processing
Large language models increasingly need to accumulate and reuse historical information in long-term assistants and agent systems. Simply exp
arxiv.org·2mo ago
Fast-dLLM: Training-Free Acceleration Method for Diffusion Language Models Using KV Cache and Parallel Decoding
Diffusion-based large language models (Diffusion LLMs) have shown promise for non-autoregressive text generation with parallel decoding capa
arxiv.org·8mo agoFlash-MSA Method Aims to Speed Up AI Training on Million-Token Sequences
Researchers have introduced Flash-MSA, a technique designed to accelerate the training of large language models on very long sequences of up
Comments
Sign in to join the conversation.
No comments yet. Be the first.