REVIEW 5 cited by
Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large language models (LLMs) have performed well on several reasoning benchmarks, including ones that test analogical reasoning abilities. However, it has been debated whether they are actually performing humanlike abstract reasoning or instead employing less general processes that rely on similarity to what has been seen in their training data. Here we investigate the generality of analogy-making abilities previously claimed for LLMs (Webb, Holyoak, & Lu, 2023). We take one set of analogy problems used to evaluate LLMs and create a set of "counterfactual" variants-versions that test the same abstract reasoning abilities but that are likely dissimilar from any pre-training data. We test humans and three GPT models on both the original and counterfactual problems, and show that, while the performance of humans remains high for all the problems, the GPT models' performance declines sharply on the counterfactual set. This work provides evidence that, despite previously reported successes of LLMs on analogical reasoning, these models lack the robustness and generality of human analogy-making.
Forward citations
Cited by 5 Pith papers
-
The Metanym Game: A Self-Contained, Self-Consistent LLM Peer-Community Benchmark for Structural Intelligence
The metanym game lets LLMs generate and judge novel analogies with no fixed test set, and a single SVD of their mutual ratings yields factual-competence scores that correlate r=0.92 with GPQA Diamond.
-
LLM world models are mental: Output layer evidence of brittle world model use in LLM mechanical reasoning
LLMs estimate pulley mechanical advantage above chance via a pulley-counting heuristic, but fail to distinguish functional from connected-but-nonfunctional systems, indicating brittle world-model use.
-
Understanding and Benchmarking Artificial Intelligence: OpenAI's o3 Is Not AGI
A conceptual analysis arguing that OpenAI's o3 solves ARC-AGI by brute-force search over predefined operations, so its high score is not evidence of AGI, and proposes a new definition and benchmark for intelligence.
-
A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future
A historical review that organizes multimodal explainability methods into four chronological eras and three explainability types, extending coverage to generative LLMs.
-
Aligning Generalisation Between Humans and Machines
A perspective paper maps how humans and machines generalize differently and argues that aligning these generalization behaviors is essential for human-AI teaming.
Discussion (0). Continue with ORCID to comment.