Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perception

Jinsong Guo; Kam-Fai Wong; Lingzhi Wang; Luyang Lin

Investigating Bias in LLM-Based Bias Detection: Disparities between LLMs and Human Perception

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.14896 v2 pith:B5YMNIS4 submitted 2024-03-22 cs.CY

Luyang Lin , Lingzhi Wang , Jinsong Guo , Kam-Fai Wong This is my paper · ORCID

classification cs.CY

keywords biasllmsdetectionmediabiasesmodelsacrosscritical

verification ladder T0 review T1 audit T2 compute T3 formal

0 comments

read the original abstract

The pervasive spread of misinformation and disinformation in social media underscores the critical importance of detecting media bias. While robust Large Language Models (LLMs) have emerged as foundational tools for bias prediction, concerns about inherent biases within these models persist. In this work, we investigate the presence and nature of bias within LLMs and its consequential impact on media bias detection. Departing from conventional approaches that focus solely on bias detection in media content, we delve into biases within the LLM systems themselves. Through meticulous examination, we probe whether LLMs exhibit biases, particularly in political bias prediction and text continuation tasks. Additionally, we explore bias across diverse topics, aiming to uncover nuanced variations in bias expression within the LLM framework. Importantly, we propose debiasing strategies, including prompt engineering and model fine-tuning. Extensive analysis of bias tendencies across different LLMs sheds light on the broader landscape of bias propagation in language models. This study advances our understanding of LLM bias, offering critical insights into its implications for bias detection tasks and paving the way for more robust and equitable AI systems

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Too Human to Model:The Uncanny Valley of LLMs in Social Simulation -- When Generative Language Agents Misalign with Modelling Principles
cs.CY 2025-07 conditional novelty 7.0 of 10

A position paper contends that LLM agents, despite their human-like talk, are often too rich in detail to serve as scientific models, and proposes conditions where they still excel.
Analysis of Propaganda in Tweets From Politically Biased Sources
cs.SI 2025-07 conditional novelty 6.0 of 10

Journalists at politically extreme news outlets tweet propaganda-like language more often than those at mild outlets, and large language models outperform a fine-tuned BERT in detecting it.
Bias Amplification in RAG: Poisoning Knowledge Retrieval to Steer LLMs
cs.LG 2025-06 reject novelty 6.0 of 10

A retrieval-augmented generation system can be poisoned with reward-optimized biased documents and vector-space manipulation to substantially increase biased LLM outputs.
Software Engineering for Large Language Models: Research Status, Challenges and the Road Ahead
cs.SE 2025-06 conditional novelty 4.0 of 10

A literature review organizes LLM development into a six-phase software engineering lifecycle and identifies challenges and research directions for each phase.
Relative Bias: A Comparative Framework for Quantifying Bias in LLMs
cs.CL 2025-05 conditional novelty 4.0 of 10

A model is 'relatively biased' when its responses deviate from the consensus of a baseline LLM set, and this deviation can be scored by embedding distances or LLM judges plus equivalence tests.

Pith tools