Planning vs Reasoning: Ablations to Test Capabilities of LoRA layers

Neel Redkar

REVIEW 5 major objections 6 minor 19 references

Planning vs Reasoning: Ablations to Test Capabilities of LoRA layers

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Reasoning in a language model can be added with low-rank adapters, while planning cannot.

desk verdict A reasonable question and a usable new eval, but the low-rank reasoning claim is not supported by the current evidence. read the letter →

arxiv 2412.00029 v2 pith:NNHPPQS4 submitted 2024-11-19 cs.AI

Neel Redkar This is my paper · ORCID

classification cs.AI

keywords low-rankadaptationreasoningplanningHashChaineffectiverankentropyregularizedadaptersGSM8KGPT-2

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether low-rank adapter layers can add new reasoning and planning abilities to a frozen language model, rather than merely restyle existing behavior. The author claims the answer is different for the two: reasoning can be added because it occupies a low-rank subspace of the weights, while multi-step planning cannot because it needs higher-rank structure. The evidence is a new deterministic benchmark, HashChain Reasoning, in which the model must compare several hash chains from one start token and return the shortest chain's endpoint; LoRA matrices trained on this reasoning task show effective rank about 50, two to three times lower than the roughly 150 seen on the single-chain HashHop planning task, and the adapters sharply improve four-chain accuracy. The paper also introduces ELoRA, an entropy-regularized variant that converges faster and improves GSM8K accuracy by about five percentage points over standard LoRA. If the interpretation holds, reasoning is a cheap capability to add to pretrained models, and effective rank is a practical diagnostic for whether a task is LoRA-amenable.

What carries the argument

The load-bearing instrument is the effective rank of trained LoRA matrices, measured two ways: Shannon entropy of normalized singular values and a singular-value cutoff, following the effective-rank measure. A low value is read as evidence that the learned rule fits in a low-dimensional subspace. The second piece is HashChain Reasoning, a deterministic dataset in which several hash chains share a start token and the model must output the terminal hash of the shortest chain; the task forces comparison across branches, that is, breadth-first reasoning, unlike HashHop's single-depth planning. The third piece is ELoRA, which prepends an unactivated entropy matrix to the LoRA path and adds the linear entropy loss $L_{\text{entropy}} = \log(\|Z^T Z\|_F^2)$, an affordable proxy for Rényi $\alpha = 2$ entropy, before the low-rank projection.

What would settle it

Train LoRA on a matched control task with the same hash-chain structure but no comparison operation, such as always returning the terminal hash of the first listed chain, and measure effective rank; if the control also shows rank near 50, the 2-3x rank gap reflects dataset structure rather than reasoning. Alternatively, train a full-rank adapter on HashChain Reasoning: if it substantially outperforms the low-rank adapters, the claim that reasoning is stored in a low-rank subspace is incomplete.

Watch

Extended reading notes

Core claim

The central discovery claimed is that reasoning in a language model lives in a low-rank subspace of the weights, while planning does not. In the author's experiments, LoRA matrices trained on HashChain Reasoning have effective rank around 50, versus around 150 for LoRA matrices trained on HashHop, a two to threefold difference. This gap is presented as evidence that the reasoning rule, comparing branches and picking the shortest chain, is representable in far fewer effective parameters than the planning rule, following one chain n hops ahead. Consistent with that, LoRA layers raise four-chain reasoning accuracy from 0.192 to 0.369, and ELoRA raises it to 0.451, while HashHop gains are confined to hop counts the base model already handled. ELoRA, a LoRA variant with an entropy-maximizing linear pre-matrix and the entropy loss $L_{\text{entropy}} = \log(\|Z^T Z\|_F^2)$, reaches 48% on GSM8K versus 43% for standard LoRA and 33% for the base Phi-1.5 model.

Load-bearing premise

The central claim rests on treating the effective rank of trained LoRA matrices as a faithful measure of how many parameters a capability's learned circuit really needs, rather than as an artifact of optimization, regularization, or dataset statistics.

Editorial extensions

If this is right

If reasoning is genuinely low-rank, LoRA adapters are a natural, cheap way to add or strengthen reasoning skills in deployed models.
Planning improvements require higher-rank or different mechanisms, so LoRA-style adapters should not be expected to teach new multi-step planning.
The 2-3x effective-rank gap can serve as a diagnostic for whether a new task is reasoning-like (low-rank) or planning-like (high-rank) before investing in fine-tuning.
ELoRA's entropy pre-matrix suggests representation disentanglement is compatible with post-training weight merging, offering faster convergence on math and reasoning data.
HashChain Reasoning provides a deterministic, dynamically generated benchmark for isolating reasoning ability from memorization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension is to train LoRA on a HashChain control with identical chain statistics but a memorized target instead of a shortest-chain rule; if effective rank stays near 50, the low-rank signature comes from dataset structure rather than reasoning per se.
If ELoRA's entropy matrix is a transferable prior, pretraining it on one reasoning dataset and then attaching small adapters to new tasks should reproduce the faster convergence reported in the paper.
The planning/reasoning split suggests a broader capability taxonomy by effective rank: one could profile other tasks, such as retrieval, arithmetic, and instruction following, and predict where LoRA will help before running experiments.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this paper asks a good question—where are LoRA layers actually effective—and HashChain Reasoning is a usable new eval. But the headline claim that reasoning lives in low-rank spaces does not follow from the evidence. The stress-test concern is right: the rank gap is confounded by base competence and training difficulty, and the paper's own rank measures disagree with each other.

What's new and worth keeping: HashChain Reasoning is a sensible variant of HashHop that forces the model to compare multiple chains, which separates breadth from depth better than pure chain-following. The ELoRA adapter is incremental, but the idea of an entropy-regularized linear map is simple and testable, and the reported ~5 point gain on GSM8K is something to look at. I also credit the authors for being upfront in Sec 2.1 that HashHop is artificial and may be measuring planning more than reasoning, and for disclosing in Sec 3.1 that the 15-20 hop results were cut at chance.

The soft spots are real and load-bearing. First, no control. The base model is at 0.192 on 4-chain HashChain and 0.391 on 3-chain, while HashHop long-hop accuracy is near chance. So the HashHop LoRA has to create new capability; the HashChain LoRA just extends a partially acquired skill. The effective rank of the delta matrix likely reflects the size of the required weight change under the chosen optimizer, not an intrinsic property of reasoning. There is no calibration experiment, e.g., a task with known intrinsic rank. Second, the rank metric itself is unstable: for the same HashHop LoRA, Shannon entropy gives a mean effective rank of 17.87 while the cutoff gives 158. That is an 8x disagreement, and it makes a 2-3x cross-task gap hard to trust. Third, no error bars or seed counts anywhere, and the code link is an anonymous placeholder, so the headline numbers are not checkable. Fourth, the post-hoc hop cutoff is defensible but should have been pre-specified.

The paper is not worthless. The dataset and the ELoRA variant are worth a workshop-level discussion, and the central question is timely. But the abstract's conclusion—that reasoning capabilities exist primarily in low-rank spaces—is a stretch from this evidence. I would send it to serious review only if the authors commit to major revision: add seeds and error bars, fix the rank estimator, report the two tasks at matched base competence, and release actual code. Otherwise a desk reject is defensible.

For my own work, I wouldn't cite it yet, but I'd point someone building reasoning evals at HashChain.

Referee Report

5 major / 6 minor

Summary. The paper studies whether LoRA layers can enhance reasoning and planning abilities in GPT-2. It introduces HashChain Reasoning, a multi-chain hash prediction task, and ELoRA, a LoRA variant with an entropy-based regularizer. Based on effective rank measurements of trained LoRA adapters, the authors claim that reasoning tasks require 2-3x lower rank than planning tasks and that reasoning therefore lives in low-rank subspaces. The paper also reports that ELoRA converges faster and improves GSM8K accuracy by about 5 percentage points over standard LoRA.

Significance. If the conclusions held, the paper would provide a practical guideline for when to use LoRA for reasoning tasks and a new deterministic benchmark (HashChain) for reasoning evaluation. The introduction of ELoRA with an entropy regularizer is a plausible contribution worth further study. However, the central rank claim is not currently supported by the evidence, due to inconsistent rank measures, uncontrolled task differences, and missing statistical reproducibility. The paper is honest about some limitations but does not address the main confounds between task difficulty and intrinsic rank.

major comments (5)

[Section 3.2, Figures 3 and 4] The two effective rank measures disagree by an order of magnitude for the same HashHop LoRA matrix (mean 17.87 via Shannon entropy vs 158 via cutoff), yet the paper uses 'effective rank' as a single concept and claims a 2-3x reduction for HashChain. The text also states that HashChain LoRA rank 'hovered around 50' for most layers, which is higher than the entropy-based HashHop value; the paper does not reconcile these numbers. The rank-gap claim is therefore not well-defined until the paper commits to one measure and reports both consistency and layer-wise values.
[Table 1 and Figures 2, 3, 4, 6, and 7] No error bars, seed counts, or significance tests are reported anywhere. The central quantitative claims—the 5% GSM8K improvement and the 4-chain accuracy jump from 0.192 to 0.369/0.451—are based on what appear to be single runs. The paper should report mean and standard deviation over at least three (ideally five) random seeds and state the number of runs in every figure caption.
[Section 3.1, Figure 6] The paper states that 15-20 hop results were 'cut off in the graph for being equally close to random chance.' Dropping data points after observing the outcome can bias the reported curves and the subsequent rank analysis. Please show the full 1-20 hop curves in an appendix, or define a pre-registered criterion for excluding chance-level hops.
[Sections 2.3 and 3.2] The inference that the 2-3x rank gap between HashChain and HashHop reflects a fundamental difference between reasoning and planning is confounded by task difficulty and base competence. Table 1 shows base accuracies of 0.283 (HashHop), 0.391 (3-chain), and 0.192 (4-chain), and Section 3.1 reports that LoRA only improves HashHop hops the model already partially mastered. The effective rank of a LoRA delta matrix measures the size of the parameter change needed under the chosen optimizer, not necessarily an intrinsic property of the task. To support the claim, the paper needs a calibration experiment—for example, a task with known intrinsic rank or a matched-base-competence comparison—that rules out the alternative explanation that harder/larger adaptations simply produce higher-rank deltas.
[Section 4, Table 2] ELoRA validation does not test the low-rank reasoning hypothesis. No effective-rank measurements are reported for ELoRA-trained matrices, and the experiments switch from GPT-2 to Phi-1.5 without explanation. The claim that ELoRA 'validates the prior analysis' requires showing that the entropy regularizer reduces or preserves LoRA rank, and that the GSM8K improvement is not just a hyperparameter artifact.

minor comments (6)

[Section 2.3] The statement 'All training scripts are open source' points to https://github.com/anon/opensource, which is a placeholder; a working repository or an anonymized link with the code is needed for reproducibility.
[Section 4] The loss term Lentropy = log(|Z^T Z|_2^F ) is ambiguous; please define whether the notation means the squared Frobenius norm or the operator norm, and specify the full loss combination (weight of the entropy term, batch size, etc.).
[Figure 7] The caption says 'ELoRA converges faster than LoRA on average' but no average or error bars are shown; if this is a single run, please say so or show multiple runs.
[References] References [2] and [3] are duplicates; please fix the reference list.
[Section 4 vs Table 2] The paper states 'Phi1.5b was utilized for training' in Section 4, but Table 2 uses 'Phi 1.5 base'; please be consistent and report the exact model variant and tokenizer settings.
[Figure 1] The caption is partially garbled; please clarify the relationship between the graph and the HashChain template in Section 2.2.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation chain found; the rank comparison is empirical and the ELoRA claim is externally benchmarked on GSM8K.

full rationale

I walked the paper's derivation chain: the effective-rank numbers in Section 3.2 and Figure 3 are empirical statistics computed on LoRA matrices trained on two different synthetic tasks, HashHop and HashChain. No equation in the paper defines 'reasoning' in terms of the measured rank, and no fitted parameter is renamed as a prediction. The central inference, from lower effective rank on HashChain to the claim that reasoning is intrinsically low-rank, depends on the operational assumption that HashChain captures reasoning and on the matched-difficulty assumption between the two training setups; these are threats to construct and internal validity, not circularity by construction. The separate ELoRA result is validated against GSM8K, an external benchmark independent of the HashChain training distribution, so that sub-claim does not reduce to its own input. The paper also does not rely on a load-bearing self-citation or an imported uniqueness theorem. Under the rule that circularity must be exhibited as a specific reduction, no such step is present, so the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 2 invented entities

The paper introduces no mathematical derivation; its conclusions rest on interpreting LoRA matrix ranks and task accuracies. The key quantities are not fully specified, and the central claim depends on the validity of treating hash tasks as reasoning and planning proxies.

free parameters (4)

Effective rank cutoff threshold = not reported
The paper uses a 'cutoff' measure of effective rank (Section 3.2) but does not state the singular-value threshold or normalization; the choice affects the reported rank values.
ELoRA entropy regularization weight = not reported
The ELoRA loss adds an entropy term, but the weighting coefficient on Lentropy relative to the task loss is not given; performance may depend on it.
LoRA rank and alpha = not reported
The intrinsic LoRA rank and scaling alpha are not specified in the ablation, so the effective rank measurements cannot be reproduced.
Training steps or epochs = not reported
'Iterations were done till the loss plateaued' (Section 2.3) leaves training duration unspecified.

assumptions (3)

domain assumption Effective rank of a LoRA matrix measures the intrinsic dimensionality of the task representation.
Used in Section 3.2 to conclude that HashChain reasoning has lower rank than HashHop planning; if rank instead reflects optimization or vocabulary structure, the conclusion fails.
domain assumption HashHop and HashChain tasks respectively isolate planning and reasoning capabilities.
Defined in Section 2; the paper assumes single-chain multi-hop requires depth (planning) while multi-chain shortest-path requires branch comparison (reasoning).
domain assumption GPT-2's fine-tuning to capacity before LoRA ensures LoRA gains are additive rather than relearning.
Section 2 states the model is fine-tuned to capacity first, but no verification is shown.

invented entities (2)

ELoRA (Entropy LoRA) adapter
purpose: Adds an entropy-maximizing linear layer before LoRA to improve reasoning and convergence.
The only evidence is the paper's own GSM8K and hash evaluations; no external benchmark or independent implementation is provided, and the architecture is not fully specified.
HashChain Reasoning benchmark
purpose: A deterministic multiple-chain evaluation intended to test reasoning by finding the shortest chain.
The dataset generation is described but no generation code or data is released, so others cannot independently verify or use it.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Planning vs Reasoning: Ablations to Test Capabilities of LoRA layers." pith.science (2026). https://pith.science/paper/NNHPPQS4

@misc{pith2026241200029,
  author       = {Pith},
  title        = {Pith review of: Planning vs Reasoning: Ablations to Test Capabilities of LoRA layers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NNHPPQS4}},
  note         = {Machine review of arXiv:2412.00029}
}

read the original abstract

Low-Rank Adaptation (LoRA) layers have emerged as a promising approach for efficient model fine-tuning, but their capabilities and limitations have not been fully explored. This paper: 1) Investigates the fundamental question of whether LoRA layers are effective at increasing reasoning + planning abilities 2) We introduce HashChain Reasoning, a novel evaluation dataset that deterministically tests reasoning capabilities. Through systematic ablation studies on GPT-2, we demonstrate that reasoning capabilities appear to exist primarily in low-rank spaces and can be effectively enhanced using LoRA layers. The effective rank analysis of trained LoRA matrices reveals a 2-3x lower rank requirement for reasoning tasks compared to planning tasks, giving context on where LoRA layers would be effective. This also provides evidence for reasoning fundamentally preferring low-parameter spaces for generalization.

Figures

Figures reproduced from arXiv: 2412.00029 by the authors.

**Figure 1.** HashHop examples showing (a) a basic hash chain and (b) the structure of the new proposed HashChain Reasoning eval with multiple chains. inside of the weights of the network. 1.3. Contributions The contributions that this paper brings are: • A new reasoning eval, HashChain Reasoning, which can be utilized to deterministically check reasoning ability. This is extremely useful for model ablations & deterministic tests… view at source ↗

**Figure 2.** HashChain Reasoning graph with 3 chain (left) and 4 chain (right) accuracies showing performance as individual chain lengths increase. 4 chain reasoning accuracy demonstrated a significant increase with LoRA modules suggesting better reasoning generalization, with minimal fluctuation in 3 chain accuracy metric is also dynamically generated, it is impossible to “overfit” to the eval like other tasks. This model for r… view at source ↗

**Figure 3.** Using the same metrics, Shannon entropy [13] (left) and cutoffs (right), there is about a 2x-3x decrease in effective rank for the HashChain reasoning task compared to the baseline HashHop task [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Calculating the effective rank of the LoRA matrix trained on the regular HashHop task. Shannon Entropy [13](left) gave a mean of 17.87 and a cutoff (right) gave a mean of 158 ELoRA Architecture [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: ELoRA architecture diagram. Similar to LoRA except for a small linear matrix with maximized entropy. latent space. This shows the horizontal scaling of reasoning or handling multiple threads of reasoning is low rank, and can be expanded with LoRA layers. The implicati…

**Figure 6.** Figure 6: Negative result for regular HashHop LoRA tests. LoRA layers only increase previously known planning capabilities, without any large boosts in accuracy. ELoRA Convergence [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

**Figure 7.** Figure 7: ELoRA converges faster than LoRA on average when given the same data [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 6 canonical work pages

[1]

Seq-VCR: Preventing Collapse in In- termediate Transformer Representations for Enhanced Reasoning, November 2024

Md Rifat Arefin, Gopeshh Subbaraj, Nicolas Gon- tier, Yann LeCun, Irina Rish, Ravid Shwartz-Ziv, and Christopher Pal. Seq-VCR: Preventing Collapse in In- termediate Transformer Representations for Enhanced Reasoning, November 2024. arXiv:2411.02344

arXiv 2024
[3]

The pitfalls of next-token prediction, July 2024

Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction, July 2024. arXiv:2403.06963 [cs]

arXiv 2024
[4]

Cunningham

Dan Biderman, Jacob Portes, Jose Javier Gonza- lez Ortiz, Mansheej Paul, Philip Greengard, Con- nor Jennings, Daniel King, Sam Havens, Vitaliy Chi- ley, Jonathan Frankle, Cody Blakeney, and John P. Cunningham. LoRA Learns Less and Forgets Less, September 2024. arXiv:2405.09673 [cs]

arXiv 2024
[5]

Training Verifiers to Solve Math Word Problems, November 2021

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems, November 2021. arXiv:2110.14168

arXiv 2021
[6]

DeepSeek-AI, Daya Guo, Dejian Yang, and ... Zhang. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, January 2025. arXiv:2501.12948

arXiv 2025
[7]

Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and V . S. Subrahmanian. Higher Layers Need More LoRA Experts, February

work page
[8]

RULER: What’s the Real Con- text Size of Your Long-Context Language Models?, August 2024

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the Real Con- text Size of Your Long-Context Language Models?, August 2024. arXiv:2404.06654 [cs]

arXiv 2024
[9]

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models, October 2021. arXiv:2106.09685 [cs]

arXiv 2021

Show all 19 references

[10]

Textbooks Are All You Need II: phi-1.5 technical report, Septem- ber 2023

Yuanzhi Li, S´ebastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks Are All You Need II: phi-1.5 technical report, Septem- ber 2023. arXiv:2309.05463

2023 arXiv
[11]

HashHop: Long Context Evaluation, 2024

Magic. HashHop: Long Context Evaluation, 2024

2024
[12]

Language Models are Unsupervised Multitask Learners

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. 2019. 7 Planning vs Reasoning: Ablations to Test Capabilities of LoRA layers

2019
[13]

The effective rank: A measure of effective dimensionality

Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European Signal Processing Conference, pages 606– 610, September 2007

2007
[14]

Continual Learning of Large Language Models: A Comprehensive Survey, June

Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Zifeng Wang, Sayna Ebrahimi, and Hao Wang. Continual Learning of Large Language Models: A Comprehensive Survey, June
[15]

FroSSL: Frobenius Norm Minimization for Efficient Multiview Self-Supervised Learning, August 2024

Oscar Skean, Aayush Dhakal, Nathan Jacobs, and Luis Gonzalo Sanchez Giraldo. FroSSL: Frobenius Norm Minimization for Efficient Multiview Self-Supervised Learning, August 2024. arXiv:2310.02903

2024 arXiv
[16]

arXiv:2404.16789 [cs]

arXiv
[17]

Batched Low- Rank Adaptation of Foundation Models, April 2024

Yeming Wen and Swarat Chaudhuri. Batched Low- Rank Adaptation of Foundation Models, April 2024. arXiv:2312.05677 [cs]

2024 arXiv
[18]

Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM, March 2024

Sainbayar Sukhbaatar, Olga Golovneva, Vasu Sharma, Hu Xu, Xi Victoria Lin, Baptiste Rozi `ere, Ja- cob Kahn, Daniel Li, Wen-tau Yih, Jason Weston, and Xian Li. Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM, March 2024. arXiv:2403.07816 [cs]

2024 arXiv
[20]

Continual Learning with Low Rank Adaptation, November 2023

Martin Wistuba, Prabhu Teja Sivaprasad, Lukas Balles, and Giovanni Zappella. Continual Learning with Low Rank Adaptation, November 2023. arXiv:2311.17601 [cs]. 8

2023 arXiv
[2024]

arXiv:2402.08562 [cs] version: 1

arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Seq-VCR: Preventing Collapse in In- termediate Transformer Representations for Enhanced Reasoning, November 2024

Md Rifat Arefin, Gopeshh Subbaraj, Nicolas Gon- tier, Yann LeCun, Irina Rish, Ravid Shwartz-Ziv, and Christopher Pal. Seq-VCR: Preventing Collapse in In- termediate Transformer Representations for Enhanced Reasoning, November 2024. arXiv:2411.02344

arXiv 2024

[2] [3]

The pitfalls of next-token prediction, July 2024

Gregor Bachmann and Vaishnavh Nagarajan. The pitfalls of next-token prediction, July 2024. arXiv:2403.06963 [cs]

arXiv 2024

[3] [4]

Cunningham

Dan Biderman, Jacob Portes, Jose Javier Gonza- lez Ortiz, Mansheej Paul, Philip Greengard, Con- nor Jennings, Daniel King, Sam Havens, Vitaliy Chi- ley, Jonathan Frankle, Cody Blakeney, and John P. Cunningham. LoRA Learns Less and Forgets Less, September 2024. arXiv:2405.09673 [cs]

arXiv 2024

[4] [5]

Training Verifiers to Solve Math Word Problems, November 2021

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems, November 2021. arXiv:2110.14168

arXiv 2021

[5] [6]

DeepSeek-AI, Daya Guo, Dejian Yang, and ... Zhang. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, January 2025. arXiv:2501.12948

arXiv 2025

[6] [7]

Chongyang Gao, Kezhen Chen, Jinmeng Rao, Baochen Sun, Ruibo Liu, Daiyi Peng, Yawen Zhang, Xiaoyuan Guo, Jie Yang, and V . S. Subrahmanian. Higher Layers Need More LoRA Experts, February

work page

[7] [8]

RULER: What’s the Real Con- text Size of Your Long-Context Language Models?, August 2024

Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the Real Con- text Size of Your Long-Context Language Models?, August 2024. arXiv:2404.06654 [cs]

arXiv 2024

[8] [9]

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models, October 2021. arXiv:2106.09685 [cs]

arXiv 2021