{"id":"c57aee1f-a7b7-4aa7-aa71-8dc419651b56","arxiv_id":"2412.00029","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"On GPT-2 hash tasks, LoRA layers improved a synthetic reasoning task more than a planning task, and the authors interpret this as evidence that reasoning is inherently low rank.","lead":"This paper tests whether LoRA adapters can add reasoning and planning abilities to small language models, using synthetic hash-chain tasks. It claims reasoning lives in low-rank spaces and introduces a new adapter, ELoRA, that reportedly beats standard LoRA on GSM8K.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The low-rank evidence is confounded: HashHop and HashChain tasks differ in base competence and training difficulty, so the 2-3x rank gap may reflect how much adaptation LoRA had to do, not intrinsic reasoning dimensionality.","rationale":"The reader's REJECT rests on the effective-rank proxy being unvalidated. I agree with that verdict, but I would sharpen the concern: the paper's two task families are not matched on the variables that control adapter rank. In §2.3 the model is first fine-tuned to capacity on each eval, and §3.1/Fig. 6a show that HashHop LoRA only improves hops the base model already handles, with longer hops at chance. For HashChain, Table 1 reports base 3-chain 0.391 and 4-chain 0.192, and LoRA lifts the latter to 0.369. Thus the HashHop adapter may need a high-rank update simply because it is being asked to implement a large capability gain, while the HashChain adapter implements a smaller or partially supported gain. The effective rank of a delta matrix is a statement about the parameter change under a specific optimizer, not a direct measurement of the task's intrinsic dimensionality. The paper makes no attempt to validate this proxy, e.g., with a task whose intrinsic rank is known, and it does not control for base-model competence, training duration, or optimizer state. Therefore Figure 3's 2-3x rank gap is exactly the kind of evidence that would reverse if the confound is real. The ELoRA GSM8K improvement is interesting but does not repair the inference: a better adapter can improve accuracy even if reasoning is not 'stored low-rank'; it only shows the entropy prior helps optimization. The proposed control experiment—matching base accuracy across tasks before LoRA training and re-measuring effective rank—would settle whether the rank gap is a task-difficulty artifact. Given the absence of that control and of error bars, the central claim is not supported; the REJECT verdict remains appropriate.","tokens_in":6635,"tokens_out":6591,"duration_ms":64213,"concrete_test":"Control for base competence: fine-tune the base GPT-2 to intermediate accuracy on HashHop so that its accuracy on the target hop range is comparable to the 4-chain base accuracy (≈0.19), then train the same LoRA adapter and compute the effective-rank statistic of §3.2/Fig. 3. If the resulting effective rank falls to the ~50 range, the 2-3x gap in Fig. 3 is a difficulty/base-competence artifact and does not establish that reasoning is intrinsically low-rank; if it remains near 150, this particular confound is ruled out.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the inference in §3.2 / Fig. 3 that LoRA effective ranks of ~50 on HashChain vs ~150 on HashHop show reasoning is intrinsically lower-dimensional than planning. This inference requires that the two training setups differ only in the task dimension under study. They do not. Per §2.3, the base GPT-2 is first fine-tuned to capacity on each eval; Table 1 shows base 4-chain HashChain accuracy is 0.192 and 3-chain is 0.391, while §3.1 reports HashHop accuracy quickly falls to chance for longer hops and LoRA only improves previously-mastered short hops. Thus the HashHop LoRA is being asked to create a substantially new capability, while the HashChain LoRA extends a partially acquired skill or can exploit a simple short-branch heuristic. The effective rank of a trained delta matrix reflects the size of the parameter change needed under the chosen optimizer, not necessarily any intrinsic property of the reasoning task. The paper provides no calibration experiment—e.g., training LoRA on a task with known intrinsic rank, or matching base competence across tasks—so Figure 3 cannot rule out that the 2-3x gap is a training-difficulty artifact.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether LoRA layers can enhance reasoning and planning abilities in GPT-2. It introduces HashChain Reasoning, a multi-chain hash prediction task, and ELoRA, a LoRA variant with an entropy-based regularizer. Based on effective rank measurements of trained LoRA adapters, the authors claim that reasoning tasks require 2-3x lower rank than planning tasks and that reasoning therefore lives in low-rank subspaces. The paper also reports that ELoRA converges faster and improves GSM8K accuracy by about 5 percentage points over standard LoRA.","tokens_in":6937,"tokens_out":5611,"duration_ms":47154,"significance":"If the conclusions held, the paper would provide a practical guideline for when to use LoRA for reasoning tasks and a new deterministic benchmark (HashChain) for reasoning evaluation. The introduction of ELoRA with an entropy regularizer is a plausible contribution worth further study. However, the central rank claim is not currently supported by the evidence, due to inconsistent rank measures, uncontrolled task differences, and missing statistical reproducibility. The paper is honest about some limitations but does not address the main confounds between task difficulty and intrinsic rank.","major_comments":[{"comment":"The two effective rank measures disagree by an order of magnitude for the same HashHop LoRA matrix (mean 17.87 via Shannon entropy vs 158 via cutoff), yet the paper uses 'effective rank' as a single concept and claims a 2-3x reduction for HashChain. The text also states that HashChain LoRA rank 'hovered around 50' for most layers, which is higher than the entropy-based HashHop value; the paper does not reconcile these numbers. The rank-gap claim is therefore not well-defined until the paper commits to one measure and reports both consistency and layer-wise values.","section":"Section 3.2, Figures 3 and 4"},{"comment":"No error bars, seed counts, or significance tests are reported anywhere. The central quantitative claims—the 5% GSM8K improvement and the 4-chain accuracy jump from 0.192 to 0.369/0.451—are based on what appear to be single runs. The paper should report mean and standard deviation over at least three (ideally five) random seeds and state the number of runs in every figure caption.","section":"Table 1 and Figures 2, 3, 4, 6, and 7"},{"comment":"The paper states that 15-20 hop results were 'cut off in the graph for being equally close to random chance.' Dropping data points after observing the outcome can bias the reported curves and the subsequent rank analysis. Please show the full 1-20 hop curves in an appendix, or define a pre-registered criterion for excluding chance-level hops.","section":"Section 3.1, Figure 6"},{"comment":"The inference that the 2-3x rank gap between HashChain and HashHop reflects a fundamental difference between reasoning and planning is confounded by task difficulty and base competence. Table 1 shows base accuracies of 0.283 (HashHop), 0.391 (3-chain), and 0.192 (4-chain), and Section 3.1 reports that LoRA only improves HashHop hops the model already partially mastered. The effective rank of a LoRA delta matrix measures the size of the parameter change needed under the chosen optimizer, not necessarily an intrinsic property of the task. To support the claim, the paper needs a calibration experiment—for example, a task with known intrinsic rank or a matched-base-competence comparison—that rules out the alternative explanation that harder/larger adaptations simply produce higher-rank deltas.","section":"Sections 2.3 and 3.2"},{"comment":"ELoRA validation does not test the low-rank reasoning hypothesis. No effective-rank measurements are reported for ELoRA-trained matrices, and the experiments switch from GPT-2 to Phi-1.5 without explanation. The claim that ELoRA 'validates the prior analysis' requires showing that the entropy regularizer reduces or preserves LoRA rank, and that the GSM8K improvement is not just a hyperparameter artifact.","section":"Section 4, Table 2"}],"minor_comments":[{"comment":"The statement 'All training scripts are open source' points to https://github.com/anon/opensource, which is a placeholder; a working repository or an anonymized link with the code is needed for reproducibility.","section":"Section 2.3"},{"comment":"The loss term Lentropy = log(|Z^T Z|_2^F ) is ambiguous; please define whether the notation means the squared Frobenius norm or the operator norm, and specify the full loss combination (weight of the entropy term, batch size, etc.).","section":"Section 4"},{"comment":"The caption says 'ELoRA converges faster than LoRA on average' but no average or error bars are shown; if this is a single run, please say so or show multiple runs.","section":"Figure 7"},{"comment":"References [2] and [3] are duplicates; please fix the reference list.","section":"References"},{"comment":"The paper states 'Phi1.5b was utilized for training' in Section 4, but Table 2 uses 'Phi 1.5 base'; please be consistent and report the exact model variant and tokenizer settings.","section":"Section 4 vs Table 2"},{"comment":"The caption is partially garbled; please clarify the relationship between the graph and the HashChain template in Section 2.2.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper reads like an early preprint; the claims are much broader than the evidence. The central result would need substantial new experiments to be publishable. I would only consider acceptance after a major revision that addresses the confounds and provides reproducible multi-seed results. The placeholder code link and duplicate references also suggest the manuscript was not thoroughly checked."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper asks a good question—where are LoRA layers actually effective—and HashChain Reasoning is a usable new eval. But the headline claim that reasoning lives in low-rank spaces does not follow from the evidence. The stress-test concern is right: the rank gap is confounded by base competence and training difficulty, and the paper's own rank measures disagree with each other.\n\nWhat's new and worth keeping: HashChain Reasoning is a sensible variant of HashHop that forces the model to compare multiple chains, which separates breadth from depth better than pure chain-following. The ELoRA adapter is incremental, but the idea of an entropy-regularized linear map is simple and testable, and the reported ~5 point gain on GSM8K is something to look at. I also credit the authors for being upfront in Sec 2.1 that HashHop is artificial and may be measuring planning more than reasoning, and for disclosing in Sec 3.1 that the 15-20 hop results were cut at chance.\n\nThe soft spots are real and load-bearing. First, no control. The base model is at 0.192 on 4-chain HashChain and 0.391 on 3-chain, while HashHop long-hop accuracy is near chance. So the HashHop LoRA has to create new capability; the HashChain LoRA just extends a partially acquired skill. The effective rank of the delta matrix likely reflects the size of the required weight change under the chosen optimizer, not an intrinsic property of reasoning. There is no calibration experiment, e.g., a task with known intrinsic rank. Second, the rank metric itself is unstable: for the same HashHop LoRA, Shannon entropy gives a mean effective rank of 17.87 while the cutoff gives 158. That is an 8x disagreement, and it makes a 2-3x cross-task gap hard to trust. Third, no error bars or seed counts anywhere, and the code link is an anonymous placeholder, so the headline numbers are not checkable. Fourth, the post-hoc hop cutoff is defensible but should have been pre-specified.\n\nThe paper is not worthless. The dataset and the ELoRA variant are worth a workshop-level discussion, and the central question is timely. But the abstract's conclusion—that reasoning capabilities exist primarily in low-rank spaces—is a stretch from this evidence. I would send it to serious review only if the authors commit to major revision: add seeds and error bars, fix the rank estimator, report the two tasks at matched base competence, and release actual code. Otherwise a desk reject is defensible.\n\nFor my own work, I wouldn't cite it yet, but I'd point someone building reasoning evals at HashChain.","headline":"A reasonable question and a usable new eval, but the low-rank reasoning claim is not supported by the current evidence.","tokens_in":7417,"tokens_out":3762,"would_cite":false,"duration_ms":34470,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reasoning in a language model can be added with low-rank adapters, while planning cannot.","keywords":["low-rank adaptation","reasoning","planning","HashChain Reasoning","effective rank","entropy regularized adapters","GSM8K","GPT-2"],"falsifier":"Train LoRA on a matched control task with the same hash-chain structure but no comparison operation, such as always returning the terminal hash of the first listed chain, and measure effective rank; if the control also shows rank near 50, the 2-3x rank gap reflects dataset structure rather than reasoning. Alternatively, train a full-rank adapter on HashChain Reasoning: if it substantially outperforms the low-rank adapters, the claim that reasoning is stored in a low-rank subspace is incomplete.","tokens_in":6453,"feed_emoji":"🧠","tokens_out":6311,"duration_ms":54532,"temperature":0.7,"pith_summary":"This paper tests whether low-rank adapter layers can add new reasoning and planning abilities to a frozen language model, rather than merely restyle existing behavior. The author claims the answer is different for the two: reasoning can be added because it occupies a low-rank subspace of the weights, while multi-step planning cannot because it needs higher-rank structure. The evidence is a new deterministic benchmark, HashChain Reasoning, in which the model must compare several hash chains from one start token and return the shortest chain's endpoint; LoRA matrices trained on this reasoning task show effective rank about 50, two to three times lower than the roughly 150 seen on the single-chain HashHop planning task, and the adapters sharply improve four-chain accuracy. The paper also introduces ELoRA, an entropy-regularized variant that converges faster and improves GSM8K accuracy by about five percentage points over standard LoRA. If the interpretation holds, reasoning is a cheap capability to add to pretrained models, and effective rank is a practical diagnostic for whether a task is LoRA-amenable.","feed_headline":"Reasoning fits in low-rank adapters; planning does not","feed_subtitle":"A deterministic benchmark shows reasoning needs 2-3x less rank than planning, and an entropy adapter lifts GSM8K by about 5 points.","key_machinery":"The load-bearing instrument is the effective rank of trained LoRA matrices, measured two ways: Shannon entropy of normalized singular values and a singular-value cutoff, following the effective-rank measure. A low value is read as evidence that the learned rule fits in a low-dimensional subspace. The second piece is HashChain Reasoning, a deterministic dataset in which several hash chains share a start token and the model must output the terminal hash of the shortest chain; the task forces comparison across branches, that is, breadth-first reasoning, unlike HashHop's single-depth planning. The third piece is ELoRA, which prepends an unactivated entropy matrix to the LoRA path and adds the linear entropy loss $L_{\\text{entropy}} = \\log(\\|Z^T Z\\|_F^2)$, an affordable proxy for Rényi $\\alpha = 2$ entropy, before the low-rank projection.","core_discovery":"The central discovery claimed is that reasoning in a language model lives in a low-rank subspace of the weights, while planning does not. In the author's experiments, LoRA matrices trained on HashChain Reasoning have effective rank around 50, versus around 150 for LoRA matrices trained on HashHop, a two to threefold difference. This gap is presented as evidence that the reasoning rule, comparing branches and picking the shortest chain, is representable in far fewer effective parameters than the planning rule, following one chain n hops ahead. Consistent with that, LoRA layers raise four-chain reasoning accuracy from 0.192 to 0.369, and ELoRA raises it to 0.451, while HashHop gains are confined to hop counts the base model already handled. ELoRA, a LoRA variant with an entropy-maximizing linear pre-matrix and the entropy loss $L_{\\text{entropy}} = \\log(\\|Z^T Z\\|_F^2)$, reaches 48% on GSM8K versus 43% for standard LoRA and 33% for the base Phi-1.5 model.","pith_inferences":["A testable extension is to train LoRA on a HashChain control with identical chain statistics but a memorized target instead of a shortest-chain rule; if effective rank stays near 50, the low-rank signature comes from dataset structure rather than reasoning per se.","If ELoRA's entropy matrix is a transferable prior, pretraining it on one reasoning dataset and then attaching small adapters to new tasks should reproduce the faster convergence reported in the paper.","The planning/reasoning split suggests a broader capability taxonomy by effective rank: one could profile other tasks, such as retrieval, arithmetic, and instruction following, and predict where LoRA will help before running experiments."],"forward_implications":["If reasoning is genuinely low-rank, LoRA adapters are a natural, cheap way to add or strengthen reasoning skills in deployed models.","Planning improvements require higher-rank or different mechanisms, so LoRA-style adapters should not be expected to teach new multi-step planning.","The 2-3x effective-rank gap can serve as a diagnostic for whether a new task is reasoning-like (low-rank) or planning-like (high-rank) before investing in fine-tuning.","ELoRA's entropy pre-matrix suggests representation disentanglement is compatible with post-training weight merging, offering faster convergence on math and reasoning data.","HashChain Reasoning provides a deterministic, dynamically generated benchmark for isolating reasoning ability from memorization."],"supporting_citations":[{"why":"Defines the LoRA low-rank adaptation method whose capabilities the paper ablates.","marker":"[9]"},{"why":"Supplies the HashHop planning task used as the high-rank comparison baseline.","marker":"[11]"},{"why":"Introduced the HashHop-style long-context evaluation that the planning baseline is drawn from.","marker":"[8]"},{"why":"GPT-2 is the base model for the main LoRA ablation experiments.","marker":"[12]"},{"why":"Provides the Shannon-entropy and singular-value-cutoff measures used to compute effective rank.","marker":"[13]"},{"why":"Supplies the linear entropy loss used by ELoRA, equivalent to Rényi alpha=2 entropy.","marker":"[15]"},{"why":"Phi-1.5 is the base model for the ELoRA GSM8K validation.","marker":"[10]"},{"why":"GSM8K is the math word-problem benchmark used to validate ELoRA's reasoning improvement.","marker":"[5]"}],"fun_headline_variants":["LoRA layers boost reasoning, not planning","Reasoning needs low-rank adapters; planning doesn't","HashChain proves reasoning prefers low-rank space","ELoRA lifts reasoning over planning in low-rank tests","Reasoning needs 2-3x lower rank than planning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on treating the effective rank of trained LoRA matrices as a faithful measure of how many parameters a capability's learned circuit really needs, rather than as an artifact of optimization, regularization, or dataset statistics.","fun_headline_variants_meta":{"raw":{"variants":["LoRA layers boost reasoning, not planning","Reasoning needs low-rank adapters; planning doesn't","HashChain proves reasoning prefers low-rank space","ELoRA lifts reasoning over planning in low-rank tests","Reasoning needs 2-3x lower rank than planning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000477,"raw_usage":{"total_tokens":2348,"prompt_tokens":914,"completion_tokens":1434,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":1358}},"tokens_in":530,"tokens_out":1434,"duration_ms":10810,"temperature":1.0,"reasoning_tokens":1358,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:33:14.895279+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train LoRA on a matched control task with the same hash-chain structure but no comparison operation, such as always returning the terminal hash of the first listed chain, and measure effective rank; if the control also shows rank near 50, the 2-3x rank gap reflects dataset structure rather than reasoning. Alternatively, train a full-rank adapter on HashChain Reasoning: if it substantially outperforms the low-rank adapters, the claim that reasoning is stored in a low-rank subspace is incomplete.","supporting_citations":[{"cited_title":"HashHop: Long Context Evaluation, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the HashHop planning task used as the high-rank comparison baseline."},{"cited_title":"Language Models are Unsupervised Multitask Learners","cited_arxiv_id":null,"evidence_quote":"GPT-2 is the base model for the main LoRA ablation experiments."},{"cited_title":"The effective rank: A measure of effective dimensionality","cited_arxiv_id":null,"evidence_quote":"Provides the Shannon-entropy and singular-value-cutoff measures used to compute effective rank."},{"cited_title":"FroSSL: Frobenius Norm Minimization for Efficient Multiview Self-Supervised Learning","cited_arxiv_id":"2310.02903","evidence_quote":"Supplies the linear entropy loss used by ELoRA, equivalent to Rényi alpha=2 entropy."}],"review_version":1}