REVIEW 32 cited by
Self-Reflection in LLM Agents: Effects on Problem-Solving Performance
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
In this study, we investigated the effects of self-reflection in large language models (LLMs) on problem-solving performance. We instructed nine popular LLMs to answer a series of multiple-choice questions to provide a performance baseline. For each incorrectly answered question, we instructed eight types of self-reflecting LLM agents to reflect on their mistakes and provide themselves with guidance to improve problem-solving. Then, using this guidance, each self-reflecting agent attempted to re-answer the same questions. Our results indicate that LLM agents are able to significantly improve their problem-solving performance through self-reflection ($p < 0.001$). In addition, we compared the various types of self-reflection to determine their individual contribution to performance. All code and data are available on GitHub at https://github.com/matthewrenze/self-reflection
Forward citations
Cited by 32 Pith papers
-
SkillSmith: Learning to Compose Parametric Skills and Textual Knowledge
SkillSmith, an LLM augmented to ingest prefix-weights and text, generates target-task prefix-weights that beat text-only and weight-only baselines, especially as fine-tuning initialization.
-
NeSyFS: A Neuro-symbolic Fast-Slow Thinking Framework for LLM Agent under Partial Observability
NeSyFS combines a knowledge-graph belief state, a reflection step, and TSMC-style particle planning to improve LLM agent success rates on ALFWorld, WebShop, and ScienceWorld.
-
Reflection or Re-Generation? Why LLM Revision Fails Where Human Revision Succeeds
Matched two-pass experiments show human revisers improve on objective and subjective tasks, while LLM self-revision yields near-zero information gain on objective tasks and negative information gain on subjective task...
-
It Matters How You Say It: Exploring Rhetorical Patterns for AI-Assisted Information Evaluation
In a 98-participant fact-checking study, the rhetorical style of AI advice changed accuracy, confidence, and preference, with step-by-step explanations helping most and user preference diverging from performance.
-
Specialty-Specific Medical Language Model for Immune-Mediated Diseases
Across 1,140 traces, multi-step tool-calling agents show high tool-sequence consistency (TSS≈0.87) but lower argument consistency (AC≈0.69), and only TSS predicts task success.
-
VineLM: Trie-Based Fine-Grained Control for Agentic Workflows
VineLM uses an annotated execution trie plus cascade profiling and online re-rooting to select models per stage invocation in agentic workflows, improving the cost-latency-accuracy frontier by up to 18% accuracy at fi...
-
Towards Agents That Know When They Don't Know: Uncertainty as a Control Signal for Structured Reasoning
An LLM agent using retrieval and summary uncertainty as training rewards and inference filters produces more factual, useful multi-omics summaries and better downstream survival predictions.
-
Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Platforms
Murakkab uses declarative workflow specs and a profile-guided MILP optimizer to reduce GPU, energy, and cost for agentic workflow serving while meeting percentile-defined SLOs.
-
Extremal Testing for Network Software using LLMs
LLM-generated extremal tests, produced by first eliciting input validity constraints and then violating them, uncovered new bugs in mature network software implementations.
-
Legal Requirements Translation from Law
A prompt-based method using GPT-4o and selected examples turns legal paragraphs into executable Python objects, passing about 89% of unit tests on 13 U.S. data breach notification laws and beating a JSON extraction baseline.
-
AdapThink: Adaptive Thinking Preferences for Reasoning Language Model
AdapThink is an RL post-training framework that adaptively reduces overthinking and underthinking in reasoning language models by rewarding confidence-appropriate reasoning depth and diverse training samples.
-
Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection Learning
A 7B open-source model fine-tuned on GPT-4-verified tool-use trajectories plus reflection data reports state-of-the-art pass rates on StableToolBench and high error-correction rates on a new benchmark.
-
Cell-o1: Training LLMs to Solve Single-Cell Reasoning Puzzles with Reinforcement Learning
A 7B model trained with reasoning distillation and reinforcement learning reaches 32.9% batch-level accuracy on a new single-cell annotation benchmark, versus 19.0% for OpenAI's o1.
-
Zoom-Refine: Boosting High-Resolution Multimodal Understanding via Localized Zoom and Self-Refinement
A training-free two-step pipeline, localize a bounding box then refine the answer from a high-resolution crop, improves MLLM accuracy on high-resolution image benchmarks.
-
Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning
A GRPO-based method that rewards only self-reflection tokens, not answer tokens, improves LLM accuracy on function calling and Countdown math tasks using only binary success/failure feedback.
-
RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving
RepoMaster, a repository-aware code agent, lifts the task pass rate from 40.7% to 62.9% and cuts token use by about 95% versus OpenHands on the new GitTaskBench benchmark.
-
USTBench: Benchmarking and Dissecting Spatiotemporal Reasoning of LLMs as Urban Agents
USTBench is the first benchmark that decomposes urban spatiotemporal reasoning into understanding, forecasting, planning, and reflection, and shows LLMs struggle most with planning and reflection.
-
MARCO: Meta-Reflection with Cross-Referencing for Code Reasoning
MARCO combines cross-problem knowledge accumulation with cross-agent lesson sharing to improve LLM code reasoning at inference time.
-
R-TOFU: Unlearning in Large Reasoning Models
R-TOFU shows that answer-level unlearning is insufficient for large reasoning models because residual knowledge persists in chain-of-thought traces.
-
A Diagnostic Framework for AI Agent Behavior
A two-layer diagnostic framework for AI agent behavior: distinguishing foundational computational substrate from behavioral modulation layer.
-
A Survey of Reasoning and Agentic Systems in Time Series with Large Language Models
The authors organize LLM-based time series reasoning into three exclusive topologies (direct, chain, branch) crossed with four objectives, and use them to label 125 papers, benchmarks, and resources.
-
How Can Input Reformulation Improve Tool Usage Accuracy in a Complex Dynamic Environment? A Study on $\tau$-bench
IRMA reformulates tool-agent inputs with memory, domain constraints, and tool suggestions, and reports improved tau-bench pass^5 reliability over ReAct, function calling, and self-reflection.
-
HKGAI-V1: Towards Regional Sovereign Large Language Model for Hong Kong
A DeepSeek-based model fine-tuned for Hong Kong outperforms general models on Hong Kong benchmarks, but most of those benchmarks are self-authored and unreleased.
-
SciSage: A Multi-Agent Framework for High-Quality Scientific Survey Generation
SciSage, a multi-agent reflection-based framework, is reported to outperform previous LLM survey generators on coherence and citation F1, while a new benchmark, SurveyScope, enables standardized evaluation.
-
From Emergence to Control: Probing and Modulating Self-Reflection in Language Models
Self-reflection in LLMs can be steered up or down by a single activation-space vector, improving accuracy when amplified and cutting output length when suppressed.
-
Reasoning or Overthinking: Evaluating Large Language Models on Financial Sentiment Analysis
On financial sentiment classification, zero-shot LLMs match human labels better without chain-of-thought reasoning than with it.
-
Self-Correcting Code Generation Using Small Language Models
CoCoS, an online RL method with accumulated and fine-grained unit-test rewards, enables 1B-parameter language models to self-correct generated code and outperforms existing self-correction baselines on MBPP, HumanEval...
-
Can Past Experience Accelerate LLM Reasoning?
LLMs can cut reasoning compute by up to 56% on similar questions when memory of past correct solutions is combined with adaptive early stopping, though the benchmark is small and synthetic.
-
DuaShepherd: Integrating Stepwise Correctness and Potential Rewards for Mathematical Reasoning
A reward model that multiplies stepwise correctness and potential scores improves best-of-N verification accuracy for math reasoning.
-
ChemAU: Harness the Reasoning of LLMs in Chemical Research with Adaptive Uncertainty Estimation
ChemAU adds a position penalty to token-level uncertainty estimates so that flagged reasoning steps are corrected by a fine-tuned chemistry model, reporting improved accuracy on GPQA, MMLU-Pro, and SuperGPQA chemistry...
-
Invisible Tokens, Visible Bills: The Urgent Need to Audit Hidden Operations in Opaque LLM Services
A taxonomy and auditing framework for hidden operations that opaque LLM APIs bill users for, with proposals for commitment-based, predictive, behavioral, and hardware-based verification.
-
ReflectEvo: Improving Meta Introspection of Small LLMs by Learning Self-Reflection
ReflectEvo shows that small language models can improve their reasoning by fine-tuning on their own self-generated reflections, but the headline BIG-bench gains depend on oracle feedback and an unclear data split.
Discussion (0). Continue with ORCID to comment.