Parametric Memory Law: A Quantitative Framework for Understanding LoRA Memory Capacity in LLMs

[Submitted on 28 May 2026]

1d ago· 2 min readenInsight

75/100

Toasty

Bagelometer↗

Toasted just enough. A reliable bake, gently seasoned.

Score75TypeanalysisSentimentneutral

Summary

This research paper introduces the Parametric Memory Law, a quantitative framework for understanding how Low-Rank Adaptation (LoRA) enables memory updates in Large Language Models (LLMs). The authors systematically quantify exact parametric memory capacity in latent space, establishing a power law linking loss reduction to effective parameters and sequence length. They discover a deterministic phase transition at the token level where prediction probability > 0.5 enables verbatim recall under greedy decoding. Based on these findings, they propose MemFT, a threshold-guided optimization strategy that dynamically redistributes training budgets toward sub-threshold tokens to improve memory fidelity and efficiency.

Key quotes

· 5 pulled

We introduce the Parametric Memory Law, a robust power law linking loss reduction Delta L to effective parameters and sequence length.

Fine-grained analysis reveals a deterministic phase transition, demonstrating that a prediction probability of p > 0.5 constitutes a sufficient condition for verbatim recall under greedy decoding.

We employ LoRA as a controlled memory capacity probe within the latent space to systematically quantify exact parametric memory.

We introduce MemFT, a threshold-guided optimization strategy that dynamically redistributes the training budget toward sub-threshold tokens.

Empirical evaluations demonstrate that MemFT can enhance memory fidelity and efficiency.

Snippet from the RSS feed

Large Language Models (LLMs) must continuously learn and update knowledge to remain effective in dynamic real-world environments. While Low-Rank Adaptation (LoRA) is widely used for such memory updates, existing studies mainly rely on qualitative downstre

You might also wanna read

δ-mem: A Compact Online Memory Mechanism for Efficient Long-Context LLM Processing

The article presents δ-mem, a lightweight memory mechanism for large language models that augments frozen full-attention backbones with a co

arxiv.org·15d ago

Research Directions for Overcoming Memory and Interconnect Challenges in Large Language Model Inference Hardware

This article discusses the technical challenges of Large Language Model (LLM) inference, highlighting how the autoregressive Decode phase ma

arxiv.org·4mo ago

2025 LLM Paradigm Shifts: Key Technological Advances in Large Language Models

The article provides a comprehensive review of major paradigm shifts in Large Language Models (LLMs) throughout 2025, highlighting key techn

karpathy.bearblog.dev·5mo ago

Addressing Unwanted Information Memorization in Large Language Models with Targeted Information Forgetting Framework

Large Language Models (LLMs) tend to memorize unwanted information like private or copyrighted content, leading to privacy and legal concern

arxiv.org·1y ago

Research Study: Effectiveness of Adaptive Merging for Recycling LoRA Modules from Public Repositories

This research paper examines the effectiveness of adaptive merging methods for recycling LoRA (Low-Rank Adaptation) modules from public repo

arxiv.org·3mo ago

How New Open-Weight LLMs Are Reducing Long-Context Costs: KV Sharing, Attention Budgeting, and Compressed Attention

The article analyzes recent developments in open-weight LLM architectures, focusing on how newer models like Gemma 4 and DeepSeek V4 are imp

magazine.sebastianraschka.com·12d ago