arxiv: 2603.01097 · v2 · submitted 2026-03-01 · 💻 cs.LG

Recognition: no theorem link

Understanding LoRA as Knowledge Memory: An Empirical Analysis

Seungju Back , Dongwoo Lee , Naun Kang , Taehee Lee , S. K. Hong , Youngjune Gwon , Sungjin Ahn

Authors on Pith no claims yet

Pith reviewed 2026-05-15 18:06 UTC · model grok-4.3

classification 💻 cs.LG

keywords LoRAknowledge memoryparametric memoryadapter compositionlong-context reasoningLLM updatingempirical study

0 comments

The pith

LoRA adapters can function as modular parametric memory for LLMs, with measurable capacity and composability limits.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper investigates Low-Rank Adaptation (LoRA) as a way to store knowledge directly inside model parameters instead of relying on context windows or external retrieval. It runs experiments to quantify how much data one LoRA can reliably hold, how multiple LoRAs combine without destroying each other's knowledge, and whether this setup helps with tasks that need long reasoning chains. The study aims to give concrete operational rules for when LoRA memory is practical rather than proposing one new architecture. If the patterns hold, parametric memory could become a standard third option alongside retrieval and in-context learning for keeping models up to date.

Core claim

LoRA can serve as a modular knowledge memory whose capacity, composability, and long-context reasoning performance can be systematically characterized to provide practical operational guidance.

What carries the argument

LoRA adapters treated as parametric knowledge stores, with measurements of their storage capacity, internalization during fine-tuning, and scaling behavior when multiple adapters are merged or used together.

If this is right

Single LoRA modules exhibit finite and predictable storage capacity that increases with rank and training data volume.
Multiple LoRA modules can be merged or composed while preserving most of their individual knowledge under controlled conditions.
LoRA-based memory improves performance on long-context reasoning tasks by reducing dependence on large context windows.
Clear practical boundaries emerge for choosing LoRA memory over retrieval-augmented or in-context methods.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Dynamic knowledge bases could be built by training separate adapters on new data batches and swapping them in at inference time without retraining the base model.
Hybrid systems that combine a small set of LoRA memories with selective retrieval might achieve both efficiency and accuracy gains not available from either approach alone.
The same capacity-mapping approach could be applied to other adapter families to compare their suitability as modular memory.

Load-bearing premise

The observed capacity limits, internalization behaviors, and multi-module scaling rules generalize beyond the specific models, datasets, and training regimes tested in the experiments.

What would settle it

A single experiment showing that merging two LoRAs each trained on disjoint factual sets causes more than 20 percent drop in recall accuracy on questions drawn from either set would falsify the claimed composability scaling.

Figures

Figures reproduced from arXiv: 2603.01097 by Dongwoo Lee, Naun Kang, Seungju Back, S. K. Hong, Sungjin Ahn, Taehee Lee, Youngjune Gwon.

**Figure 1.** Figure 1: Performance trend on the CF as rank increases. thetic supervision for self-updating (Zweiger et al., 2025). However, these methods primarily emphasize end-to-end system for a single module, leaving the memory-specific questions of what LoRA can store, when it saturates, and how supervision format affects factual retrievability less pinned down. Related analyses of LoRA efficiency and optimization (Schulman… view at source ↗

**Figure 2.** Figure 2: Performance as data size increases. (Left) PB results. (Right) CF results. 2 4 16 128 1024 LoRA Rank 0 2 4 6 8 T o k e n s / P a r a m ( 1 0 4 ) Threshold = 90% Threshold = 95% Threshold = 100% 2 4 16 128 1024 LoRA Rank 0 5 10 15 T o k e n s / P a r a m ( 1 0 4 ) Threshold = 75% Threshold = 80% Threshold = 85% [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Figure 3: Efficiency as rank increases. (Left) PB results. (Right) CF results. maximizing rank the most efficient choice? Higher ranks incur substantial costs in parameters, training time, and memory, motivating a measure of parameter efficiency—the amount of knowledge stored per trainable parameter. Concretely, we estimate the largest tokenized knowledge load Tmax for which performance stays above a fixed threshol… view at source ↗

**Figure 5.** Figure 5: (Left) Performance across different Qwen3 model sizes. (Right) Performance comparison when using Llama vs. GPT for synthetic data generation. mats—QA, Summary, and Rewrite—against a raw-text baseline, scaling the amount of training data (number of generated instances) made using GPT-4.1 (Achiam et al., 2023) generation [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗

**Figure 6.** Figure 6: (Left) Multi-LoRA PoC on 64K PhoneBook. (Right) Performance gap between perfect and RAG routing. the correct module, yielding an upper bound where routing error is removed [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗

**Figure 7.** Figure 7: (Left) Merging strategies comparison. (Right) Comparison between selecting a single optimal LoRA and merging N LoRAs. Q10. Can Merging Multiple LoRAs Mitigate Routing Error? Given routing uncertainty, a natural hedge is to retrieve multiple candidate modules (top-3 in our experiments) and merge them, rather than committing to a single (possibly misrouted) LoRA. We evaluate four representative merging str… view at source ↗

**Figure 8.** Figure 8: A breakdown of total processing time for different methods. The chart dissects the latency into key stages, including ready time, adapter loading, merging, and inference. largely systems-driven, leaving clear headroom for further optimization (details in Appendix U). Implication: In scenarios requiring repeated and interactive access to a consistent knowledge base, the LoRAbased memory approach offers a … view at source ↗

**Figure 9.** Figure 9: PhoneBook layer ablation (Early vs. Late) under ranks r ∈ {8, 16, 32, 64}. 1 2 3 4 5 6 7 8 9 10 Data Size (10 3 tokens) 0 20 40 60 80 100 Accuracy (%) PhoneBook · Module Comparison (R=16) Attn Ffn Baseline 1 2 3 4 5 6 7 8 9 10 Data Size (10 3 tokens) 0 20 40 60 80 100 Accuracy (%) PhoneBook · Module Comparison (R=32) Attn Ffn Baseline 1 2 3 4 5 6 7 8 9 10 Data Size (10 3 tokens) 60 70 80 90 100 Accuracy (%… view at source ↗

**Figure 10.** Figure 10: PhoneBook module ablation (Attn vs. FFN) under ranks r ∈ {8, 16, 32, 64}. 1 2 3 4 5 6 7 8 9 10 Data Size (10 3 tokens) 66 68 70 72 74 76 78 80 Efficacy (%) CounterFact · Layer Comparison (R=2) Early Late Baseline 1 2 3 4 5 6 7 8 9 10 Data Size (10 3 tokens) 72 74 76 78 80 82 Efficacy (%) CounterFact · Layer Comparison (R=4) Early Late Baseline 1 2 3 4 5 6 7 8 9 10 Data Size (10 3 tokens) 76 78 80 82 84 Ef… view at source ↗

**Figure 11.** Figure 11: CounterFact layer ablation (Early vs. Late) under ranks r ∈ {2, 4, 8, 16, 32, 64}. 1 2 3 4 5 6 7 8 9 10 Data Size (10 3 tokens) 60 65 70 75 80 Efficacy (%) CounterFact · Module Comparison (R=2) Attn Ffn Baseline 1 2 3 4 5 6 7 8 9 10 Data Size (10 3 tokens) 62.5 65.0 67.5 70.0 72.5 75.0 77.5 80.0 Efficacy (%) CounterFact · Module Comparison (R=4) Attn Ffn Baseline 1 2 3 4 5 6 7 8 9 10 Data Size (10 3 token… view at source ↗

**Figure 12.** Figure 12 [PITH_FULL_IMAGE:figures/full_fig_p019_12.png] view at source ↗

**Figure 13.** Figure 13: LoRA variant comparison. We compare standard LoRA, DoRA, and PiSSA under the same experimental protocol, using PaperQA. (left) Llama results. (right) Qwen results [PITH_FULL_IMAGE:figures/full_fig_p021_13.png] view at source ↗

**Figure 14.** Figure 14: Full results for memory capacity experiment on Llama (top row) and Qwen (bottom row). (Left) Performance on PhoneBook for various ranks as data length increases. (Right) Performance on CounterFact for various ranks as data size increases. 2 4 16 128 1024 LoRA Rank 0 5 10 15 T o k e n s / P a r a m ( 1 0 4 ) Threshold = 75% Threshold = 80% Threshold = 85% 2 4 16 128 1024 LoRA Rank 0 2 4 6 8 T o k e n s / P… view at source ↗

**Figure 15.** Figure 15: Efficiency for the two datasets (Qwen). (Left) CounterFact results. (Right) PhoneBook results [PITH_FULL_IMAGE:figures/full_fig_p022_15.png] view at source ↗

**Figure 16.** Figure 16: Surface generated by rank, threshold and efficiency measured on Llama. (Left) CounterFact. (Right) PhoneBook. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_16.png] view at source ↗

**Figure 17.** Figure 17: Performance scaling with different synthetic data generation methods. (Top) Llama performance across LLM judge, BLEU, and ROUGE metrics.(Bottom) Qwen performance across LLM judge, BLEU, and ROUGE metrics. I. Details of Q4. How Does Synthetic Data Enhance Single LoRA’s Knowledge Memorization? Motivation Our analysis has established that a LoRA module possesses a finite memory capacity. Training such a modu… view at source ↗

**Figure 18.** Figure 18: Performance comparison of synthetic data generator. (Top) Results for Llama across LLM judge, BLEU, and ROUGE metrics. (Bottom) Results for Qwen across LLM judge, BLEU, and ROUGE metrics. Experimental Results Our results indicate a general trend where performance improves as the size of the base model increases. As depicted in [PITH_FULL_IMAGE:figures/full_fig_p027_18.png] view at source ↗

**Figure 19.** Figure 19: Performance comparison of routing strategies. Top Llama model results across LLM judge, BLEU, and ROUGE metrics. Bottom Qwen model results across LLM judge, BLEU, and ROUGE metrics. 0 2 4 6 LLM Judge Score 4.92 0.19 5.41 1.63 Linear 1.32 CAT TIES DARE-Linear DARE-TIES Llama - LLM Judge 0.00 0.05 0.10 BLEU Score 0.09 0.00 0.10 0.03 0.03 Linear CAT TIES DARE-Linear DARE-TIES Llama - BLEU 0.0 0.2 0.4 ROUGE S… view at source ↗

**Figure 20.** Figure 20: Performance comparison of different merging strategies. (Top) Results for Llama across LLM judge, BLEU, and ROUGE metrics. (Bottom) Results for Qwen across LLM judge, BLEU, and ROUGE metrics. 1 2 3 4 5 Number of Merged LoRAs 0 2 4 6 LLM Judge Score Llama - LLM Judge Full-Set Merging Perfect Routing, Top 1 1 2 3 4 5 Number of Merged LoRAs 0.05 0.10 0.15 BLEU Score Llama - BLEU Full-Set Merging Perfect Rout… view at source ↗

**Figure 21.** Figure 21: Comparison of Top-N performance. (Top) Llama results across LLM judge, BLEU, and ROUGE metrics. (Bottom) Qwen results across LLM judge, BLEU, and ROUGE metrics. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_21.png] view at source ↗

**Figure 22.** Figure 22: Comparison of method execution times. (Top) Without Flash Attention. (Bottom) With Flash Attention. Timing Measurement Protocol. To ensure accurate and reliable timing, especially for GPU operations, we implemented a standardized measurement protocol. All timed operations were wrapped in a utility function that uses time.perf counter() for high-precision timing. Crucially, before starting and after endin… view at source ↗

read the original abstract

Continuous knowledge updating for pre-trained large language models (LLMs) is increasingly necessary yet remains challenging. Although inference-time methods like In-Context Learning (ICL) and Retrieval-Augmented Generation (RAG) are popular, they face constraints in context budgets, costs, and retrieval fragmentation. Departing from these context-dependent paradigms, this work investigates a parametric approach using Low-Rank Adaptation (LoRA) as a modular knowledge memory. Although few recent works examine this concept, the fundamental mechanics governing its capacity and composability remain largely unexplored. We bridge this gap through the first systematic empirical study mapping the design space of LoRA-based memory, ranging from characterizing storage capacity and optimizing internalization to scaling multi-module systems and evaluating long-context reasoning. Rather than proposing a single architecture, we provide practical guidance on the operational boundaries of LoRA memory. Overall, our findings position LoRA as the complementary axis of memory alongside RAG and ICL, offering distinct advantages.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript conducts the first systematic empirical study of Low-Rank Adaptation (LoRA) as a modular parametric knowledge memory for LLMs. It maps storage capacity, internalization procedures, multi-module composability, and long-context reasoning performance across design choices, with the aim of deriving practical operational boundaries and positioning LoRA as a complementary axis to ICL and RAG for continuous knowledge updating.

Significance. If the reported capacity limits, internalization behaviors, and multi-module scaling rules prove robust, the work supplies concrete operational guidance that could inform deployment decisions for parametric memory, addressing a gap left by context-dependent methods. The purely empirical framing and absence of circular derivations are strengths that keep the contribution focused on falsifiable measurements.

major comments (2)

The central claim that the study yields transferable practical guidance on capacity, composability, and long-context reasoning rests on the assumption that observed patterns generalize beyond the specific base models, datasets, LoRA ranks, and training regimes tested. The manuscript provides no explicit discussion or ablation of this transferability (e.g., cross-family LLM experiments or out-of-distribution knowledge sets), which directly undermines the operational-guidance conclusion.
Results sections reporting scaling rules and performance boundaries lack error bars, confidence intervals, or statistical tests for the multi-module and long-context experiments. Without these, it is impossible to determine whether the stated boundaries reflect reliable trends or artifacts of the chosen experimental matrix.

minor comments (2)

Notation for LoRA rank, module count, and internalization loss is introduced without a consolidated table of symbols, making cross-section comparisons harder than necessary.
The abstract states that the work is 'the first systematic empirical study,' yet the related-work section does not quantify how prior LoRA memory papers differ in scope; a brief comparison table would strengthen this positioning.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback and the recommendation for major revision. We address each major comment below with clarifications and indicate the specific revisions planned for the manuscript.

read point-by-point responses

Referee: The central claim that the study yields transferable practical guidance on capacity, composability, and long-context reasoning rests on the assumption that observed patterns generalize beyond the specific base models, datasets, LoRA ranks, and training regimes tested. The manuscript provides no explicit discussion or ablation of this transferability (e.g., cross-family LLM experiments or out-of-distribution knowledge sets), which directly undermines the operational-guidance conclusion.

Authors: We acknowledge that the experiments are confined to specific base models, datasets, and regimes, as is typical for an initial systematic empirical mapping. To address this directly, we will add a dedicated subsection in the Discussion that explicitly discusses the scope of generalization. This will analyze the consistency of observed patterns (e.g., capacity scaling and composability rules) across the tested conditions and clearly delineate limitations, while recommending future cross-family and out-of-distribution validation. This textual addition strengthens the operational guidance without requiring new experiments. revision: yes
Referee: Results sections reporting scaling rules and performance boundaries lack error bars, confidence intervals, or statistical tests for the multi-module and long-context experiments. Without these, it is impossible to determine whether the stated boundaries reflect reliable trends or artifacts of the chosen experimental matrix.

Authors: We agree that the lack of statistical measures reduces the robustness assessment of the reported boundaries. In the revised manuscript, we will incorporate error bars (standard deviation across repeated runs) into the relevant scaling and long-context figures. We will also add statistical significance tests (e.g., paired t-tests with p-values) for key multi-module performance comparisons. These will be computed from our existing experimental data and integrated into the Results sections. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical mapping with no derivation chain

full rationale

The paper conducts a systematic empirical study of LoRA as modular memory, characterizing capacity, composability, and long-context performance through experiments. No mathematical derivations, equations, or predictions are present that could reduce to fitted inputs or self-definitions. Central claims rest on direct experimental observations rather than any load-bearing self-citation, uniqueness theorem, or ansatz smuggled from prior work. The provided guidance is framed as operational boundaries observed in the tested regimes, with no reduction of results to the inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on standard domain assumptions in LLM adaptation rather than new axioms or invented entities; no free parameters are introduced beyond typical training hyperparameters.

axioms (1)

domain assumption LoRA adapters can internalize and retrieve factual knowledge when trained on appropriate data
Core premise invoked throughout the abstract to justify treating LoRA as memory.

pith-pipeline@v0.9.0 · 5481 in / 1144 out tokens · 37125 ms · 2026-05-15T18:06:28.608371+00:00 · methodology

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

$\delta$-mem: Efficient Online Memory for Large Language Models
cs.AI 2026-05 unverdicted novelty 6.0

δ-mem augments frozen LLMs with an 8x8 online memory state updated by delta-rule learning to generate low-rank attention corrections, delivering 1.10x average gains over the backbone and larger improvements on memory-...

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages · cited by 1 Pith paper

[1]

What is the phone number of John Smith?

highlights the importance of data quality and structure for fine-tuning LLMs. Beyond directly training on raw data, studies show that transforming source material into diverse synthetic formats, such as QA pairs, summaries, or rewrites, can improve generalization and robustness. These findings suggest that synthetic augmentation serves as an effective str...

work page 2024
[2]

Monte Carlo Tree Diffusion for System 2 Planning https://arxiv.org/abs/2502.07202

work page arXiv
[3]

An Analysis for Reasoning Bias of Language Models with Small Initialization https://arxiv.org/abs/2502.04375

work page arXiv
[4]

Training Dynamics of In-Context Learning in Linear Attention https://arxiv.org/abs/2501.16265

work page arXiv
[5]

Inductive Moment Matching https://arxiv.org/abs/2503.07565

work page arXiv
[6]

Statistical Test for Feature Selection Pipelines by Selective Inference https://arxiv.org/abs/2406.18902 NeurIPS 2024

work page arXiv 2024
[7]

Human Expertise in Algorithmic Prediction https://arxiv.org/abs/2402.00793

work page arXiv
[8]

Enhancing Preference-based Linear Bandits via Human Response Time https://arxiv.org/abs/2409.05798

work page arXiv
[9]

Rho-1: Not All Tokens Are What You Need https://arxiv.org/abs/2404.07965

work page arXiv
[10]

Learning Action and Reasoning-Centric Image Editing from Videos and Simulations https://arxiv.org/abs/2407.03471

work page arXiv
[11]

The Value of Reward Lookahead in Reinforcement Learning https://arxiv.org/abs/2403.11637 ICLR 2025

work page arXiv 2025
[12]

Artificial Kuramoto Oscillatory Neurons https://arxiv.org/abs/2410.13821

work page arXiv
[13]

Exploring the Loss Landscape of Regularized Neural Networks via Convex Duality https://arxiv.org/abs/2411.07729

work page arXiv
[14]

In Search of Forgotten Domain Generalization https://arxiv.org/abs/2410.08258 15 Understanding LoRA as Knowledge Memory

work page arXiv
[15]

Programming Refusal with Conditional Activation Steering https://arxiv.org/abs/2409.05907

work page arXiv
[16]

What is...?

Efficient and Accurate Explanation Estimation with Distribution Compression https://arxiv.org/abs/2406.18334 C.2. Generation Prompts You are an expert academic assistant tasked with creating a high-quality question-answering dataset from a research paper’s introduction. Your goal is to generate 30 question-and-answer pairs based exclusively on the provide...

work page arXiv
[17]

**Factual Alignment **: Does the candidate answer state the same facts as the gold answer? It must not contradict the gold answer

work page
[18]

**Completeness**: Does the candidate answer include all the key information and nuances present in the gold answer?

work page
[19]

score" (an integer from 0-10) and

**Relevance**: Is the answer focused and on-topic? It must not contain irrelevant or hallucinatory information. [SCORING RUBRIC (0-10 SCALE)] - **10**: Perfect. The candidate answer is factually identical to the gold answer, complete, and contains no extraneous information. - **7-9**: Mostly Correct. The answer is factually correct but might omit a minor ...

work page arXiv 2048
[20]

By retaining only the parameters with the largest change in magnitude (e.g., the top-k percentile), this process filters out redundant or less impactful parameters

Trim:This step zeroes out parameters that underwent minimal change during the fine-tuning of each LoRA module. By retaining only the parameters with the largest change in magnitude (e.g., the top-k percentile), this process filters out redundant or less impactful parameters

work page
[21]

For a given parameter, some LoRAs may have induced a positive update while others induced a negative one

Elect Sign:This step resolves directional conflicts in parameter updates across different LoRA modules. For a given parameter, some LoRAs may have induced a positive update while others induced a negative one. TIES elects a single, dominant sign based on a majority vote, where the sign corresponding to the greatest total magnitude of updates is chosen as ...

work page
[22]

Parameters with conflicting signs are discarded from the merge for that specific weight, thus minimizing negative interference

Merge:Finally, only the parameter values that align with the elected sign are averaged. Parameters with conflicting signs are discarded from the merge for that specific weight, thus minimizing negative interference. DARE (Drop And REscale).To address the extreme redundancy in delta parameters, we employ DARE (Yu et al., 2024), which randomly drops a fract...

work page 2024