REVIEW 9 cited by
Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The emergent few-shot reasoning capabilities of Large Language Models (LLMs) have excited the natural language and machine learning community over recent years. Despite of numerous successful applications, the underlying mechanism of such in-context capabilities still remains unclear. In this work, we hypothesize that the learned \textit{semantics} of language tokens do the most heavy lifting during the reasoning process. Different from human's symbolic reasoning process, the semantic representations of LLMs could create strong connections among tokens, thus composing a superficial logical chain. To test our hypothesis, we decouple semantics from the language reasoning process and evaluate three kinds of reasoning abilities, i.e., deduction, induction and abduction. Our findings reveal that semantics play a vital role in LLMs' in-context reasoning -- LLMs perform significantly better when semantics are consistent with commonsense but struggle to solve symbolic or counter-commonsense reasoning tasks by leveraging in-context new knowledge. The surprising observations question whether modern LLMs have mastered the inductive, deductive and abductive reasoning abilities as in human intelligence, and motivate research on unveiling the magic existing within the black-box LLMs. On the whole, our analysis provides a novel perspective on the role of semantics in developing and evaluating language models' reasoning abilities. Code is available at {\url{https://github.com/XiaojuanTang/ICSR}}.
Forward citations
Cited by 9 Pith papers
-
What Does Chain-of-Thought Contribute at Probe Time? Evidence for Local Co-Occurrence Activation
CoT probe-time gains arise primarily from lexical activation and short-range token co-occurrence rather than sentence-level logical derivation.
-
CXXCrafter: An LLM-Based Agent for Automated C/C++ Open Source Software Building
An LLM-driven agent, CXXCrafter, automatically builds 587 of 752 C/C++ open-source projects (78%), beating default build commands (39%) and bare LLMs (32 to 38%).
-
Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMs
UCerF scores LLM fairness by both correctness and confidence, and SynthBias provides 31,756 gender-occupation coreference samples for benchmark testing.
-
A Systematic Evaluation of Traditional Privacy Policy Analysis Tools Against LLMs
Prompt-only LLMs match or outperform six specialized privacy-policy tools on most tasks, according to a 10-policy benchmark with several validity caveats.
-
Can Large Models Teach Student Models to Solve Mathematical Problems Like Human Beings? A Reasoning Distillation Method via Multi-LoRA Interaction
A multi-LoRA distillation method with knowledge generation, deep reasoning, and iterative consistency checking improves small-model math reasoning accuracy on GSM8K.
-
Lightweight Language Models are Prone to Reasoning Errors for Complex Computational Phenotyping Tasks
Small language models show logical and faithfulness errors in chain-of-thought explanations during acute respiratory failure phenotyping, and biased prompts shift their answers.
-
Integrating Quantized LLMs into Robotics Systems as Edge AI to Leverage their Natural Language Processing Capabilities
llama_ros brings quantized LLMs to ROS 2 via llama.cpp, enabling local edge-AI text generation, tokenization, and embeddings in robotics.
-
ReflectEvo: Improving Meta Introspection of Small LLMs by Learning Self-Reflection
ReflectEvo shows that small language models can improve their reasoning by fine-tuning on their own self-generated reflections, but the headline BIG-bench gains depend on oracle feedback and an unclear data split.
-
Deep Learning in Classical and Quantum Physics
A graduate-level lecture-note review of deep learning methods and their applications in classical and quantum physics, with hands-on examples.
Discussion (0). Continue with ORCID to comment.