REVIEW 6 cited by
Assessing Judging Bias in Large Reasoning Models: An Empirical Study
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large Reasoning Models (LRMs) like DeepSeek-R1 and OpenAI-o1 have demonstrated remarkable reasoning capabilities, raising important questions about their biases in LLM-as-a-judge settings. We present a comprehensive benchmark comparing judging biases between LLMs and LRMs across both subjective preference-alignment datasets and objective fact-based datasets. Through investigation of bandwagon, authority, position, and distraction biases, we uncover four key findings: (1) despite their advanced reasoning capabilities, LRMs remain susceptible to the above biases; (2) LRMs demonstrate better robustness than LLMs specifically on fact-related datasets; (3) LRMs exhibit notable position bias, preferring options in later positions; and (4) we identify a novel "superficial reflection bias" where phrases mimicking reasoning (e.g., "wait, let me think...") significantly influence model judgments. To address these biases, we design and evaluate three mitigation strategies: specialized system prompts that reduce judging biases by up to 19\% in preference alignment datasets and 14\% in fact-related datasets, in-context learning that provides up to 27\% improvement on preference tasks but shows inconsistent results on factual tasks, and a self-reflection mechanism that reduces biases by up to 10\% in preference datasets and 16\% in fact-related datasets, with self-reflection proving particularly effective for LRMs. Our work provides crucial insights for developing more reliable LLM-as-a-Judge frameworks, especially as LRMs become increasingly deployed as automated judges.
Forward citations
Cited by 6 Pith papers
-
Who Endorsed It? Measuring Authority Bias Across Expertise Levels in Language Models
Language models accept misleading endorsements more readily from higher-expertise persona sources, and the effect can be partially steered away by subtracting a residual-stream vector.
-
One Token to Fool LLM-as-a-Judge
LLM reward models falsely accept empty 'master key' responses such as ':' or 'Thought process:' across many models, and a fine-tuning defense reduces these false positives to near zero.
-
Beyond Gold Standards: Epistemic Ensemble of LLM Judges for Formal Mathematical Reasoning
A taxonomy-guided ensemble of LLM judges correlates with human ratings of autoformalizations (up to 0.662 on Isabelle/HOL) better than coarse-grained judges and reference metrics, but validation is partly in-sample an...
-
PulseReddit: A Novel Reddit Dataset for Benchmarking MAS in High-Frequency Cryptocurrency Trading
MAS traders using Reddit sentiment from PulseReddit beat traditional baselines in the reported bull-market backtests, but the gains are small, most runs lose money, and the evaluation has critical flaws.
-
A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
A structured literature survey concluding that reasoning capabilities do not automatically make LLMs more trustworthy and can introduce new vulnerabilities in safety, robustness, and privacy.
-
Strategic Reflectivism In Intelligent Systems
Strategic Reflectivism holds that intelligent systems should allocate reflective reasoning tactically, weighing its benefits against its costs.
Discussion (0). Continue with ORCID to comment.