REVIEW 15 cited by
On Memorization of Large Language Models in Logical Reasoning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) achieve good performance on challenging reasoning benchmarks, yet could also make basic reasoning mistakes. This contrasting behavior is puzzling when it comes to understanding the mechanisms behind LLMs' reasoning capabilities. One hypothesis is that the increasingly high and nearly saturated performance on common reasoning benchmarks could be due to the memorization of similar problems. In this paper, we systematically investigate this hypothesis with a quantitative measurement of memorization in reasoning tasks, using a dynamically generated logical reasoning benchmark based on Knights and Knaves (K&K) puzzles. We find that LLMs could interpolate and memorize the training puzzles (achieving near-perfect accuracy) after fine-tuning, yet they struggle with slight variations of these puzzles. On the other hand, we show that while fine-tuning leads to heavy memorization, it also consistently improves generalization performance. Through in-depth analyses with perturbation tests, cross difficulty-level transferability, probing model internals, and fine-tuning with wrong answers, we establish that LLMs develop reasoning skills on K&K puzzles alongside memorization. Finally, our analysis based on a per-sample memorization score sheds light on how LLMs switch between reasoning and memorization when solving logical puzzles. Our code and data are available at https://memkklogic.github.io.
Forward citations
Cited by 15 Pith papers
-
SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyond
A synthetic logic dataset with verifiable answers, trained via reinforcement learning, improves LLM logical reasoning and generalizes to math and coding.
-
Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs
RL-trained LLMs keep most of their skills after weight merging, while SFT-trained LLMs drop about 19% on average, because RL keeps parameter updates smaller and more task-compatible.
-
RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments
Training a 1.5B language model on 400 adaptively-difficult, procedurally generated, verifiable environments improved average reasoning-benchmark performance by 3.37% absolute, compared with 0.49% from continuing origi...
-
Understanding the Ability of LLMs to Handle Character-Level Perturbation
LLMs remain surprisingly accurate on math and coding when invisible Unicode noise is inserted after every character, with robustness driven by implicit internal denoising and, for some models, explicit rewriting in ch...
-
Throttling Web Agents Using Reasoning Gates
Rebus-based reasoning gates, puzzles built from random word/domain clue sets, impose token costs on LM web agents that are up to 9.2x the generator's cost.
-
Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning
Training a Qwen2.5-7B model with GRPO on math and puzzle data improves both domains, code transfer depends on the starting model, and template or reward mismatches sharply hurt performance.
-
ABench-Physics: Benchmarking Physical Reasoning in LLMs via High-Difficulty and Dynamic Physics Problems
A new physics benchmark with static and dynamically varied numeric problems shows top LLMs solve at most 43 percent of the static set and drop sharply when problem constants change.
-
OIBench: Benchmarking Strong Reasoning Models with Olympiad in Informatics
A private, contamination-resistant benchmark of 250 olympiad-level programming problems shows top reasoning models reaching about 36% solve rates, far above conventional models.
-
SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis
Large language models perform poorly on a new C-code vulnerability benchmark, indicating they rely on pattern matching rather than genuine reasoning.
-
Towards Revealing the Effectiveness of Small-Scale Fine-tuning in R1-style Reinforcement Learning
Re-distilling a model's own RL-trained policy into 1K SFT samples reproduces RL accuracy at a fraction of the compute.
-
Adaptive Multi-Agent Reasoning via Automated Workflow Generation
Automated workflow generation and iterative prompt refinement let a standard GPT-4.1 model outperform state-of-the-art reasoning models on a revised riddle benchmark.
-
Answer-Centric or Reasoning-Driven? Uncovering the Latent Memory Anchor in LLMs
Across ten LLMs, masking the final answer inside a complete reasoning chain causes a 26.9-point accuracy drop, evidence that models anchor to answers, not reasoning templates.
-
The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants
Clustering-based routing plus self-consistency voting among ten 7B open models reportedly outranks GPT-4.1 and GPT-4.5 on average over 15 diverse benchmarks.
-
Infinite Sampling: Efficient and Stable Grouped RL Training for Large Language Models
A GRPO decoding framework that cuts memory via micro-batched KV-cache reuse and improves decoding-round efficiency with predicted-length scheduling, at the cost of serialization.
-
Beyond Frequency: The Role of Redundancy in Large Language Model Memorization
LLM memorization is concentrated in low-redundancy (high-entropy) samples, which are roughly twice as fragile under prefix perturbation as high-redundancy samples, according to the paper's measurements on Pythia models.
Discussion (0). Sign in to comment.