Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation

Fan Zhang; Fengxian Ji; Jingpu Yang; Juanfan Wu; Min Peng; Qianqian Xie; Xiuying Chen; Yuke Li; Yu Xie; Zhexuan Cui

LLM judges reward writing style over scientific substance in idea evaluation.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 23:17 UTC pith:RACFST22

load-bearing objection Useful diagnostic benchmark for style bias in LLM judges of scientific ideas, but the variant validity check covers only 6 of 15 types and has a 320/240 mismatch—referee with major revision. the 4 major comments →

arxiv 2608.01666 v2 pith:RACFST22 submitted 2026-08-03 cs.CL cs.AI

Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation

Fengxian Ji , Yuke Li , Jingpu Yang , Juanfan Wu , Fan Zhang , Zhexuan Cui , Yu Xie , Min Peng

show 3 more authors

Qianqian Xie Xiuying Chen Zhuohan Xie

This is my paper

classification cs.CL cs.AI

keywords LLM-as-Judgestyle biasscientific idea generationevaluation benchmarksubstance recognitionadversarial robustnessstyle extractorautomated research evaluation

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that LLM judges used to select scientific ideas are systematically swayed by how an idea is written, not just by what it claims, and that this distortion changes which ideas survive downstream screening. To test this, the authors build SciStyleBench: 600 source scientific ideas, each rewritten into 15 style variants meant to keep the same scientific content, evaluated under three retrieval settings totaling 9,000 instances per setting. They define three metrics: Style Bias Index (how much scores move under style-only change), Substance Recognition Rate (whether judges rank substantively stronger ideas higher), and Adversarial Win Rate (whether plainly presented strong ideas beat stylistically packaged weak ones). Direct judges score 0.566/0.504/0.554 on these metrics, meaning style shifts scores roughly as much as substance differences. A plug-in module, SciStyleExtractor, that detects style type and injects bias-control hints into a frozen judge improves the balance to 0.501/0.759/0.899, showing that style-aware auxiliary signals can make judges focus more on substance.

Core claim

On the paper's own terms, the central discovery is that LLM-as-Judge systems for research ideas are not substance-first evaluators: style perturbation alone moves dimension-level and overall scores, changes Top-K selection, and shifts rankings by two to three positions on average. Style-induced Top-K membership changes rise from 4.6% at Top-5 to 15.9% at Top-30, meaning identical content can enter or leave the selected set depending on rhetorical packaging. The paper further claims that adding a trained style extractor, which labels the input's style type, quantifies deviation from neutral presentation, and emits a bias-control hint before the frozen judge scores, substantially raises substa

What carries the argument

SciStyleBench's core machinery is the controlled style-only perturbation: each idea is modeled as I = <S, T> (substance S, presentation T), and a perturbation p produces <S, T_p> with T_p ≠ T while S is held fixed; the evaluation shift Δp = J(I_p, K) − J(I, K) measures style-induced bias. Three metrics jointly characterize the judge: SBI quantifies score instability under style shift, SRR measures correct ranking of Enriched > Plain > Hollow/Flawed, and AWR measures whether plain high-substance ideas beat persuasive hollow ones. The mitigation module SciStyleExtractor is plug-and-play: it predicts style type, deviation, and a bias-control instruction, and passes these structured signals to a

Load-bearing premise

The central premise is that the 15 perturbations change only presentation style and not scientific substance; the paper checks this with human reviewers on only 240 sampled pairs (the text also mentions 320), so if a substantial share of the 9,000 variants actually alter content, the measured style bias would mix in genuine substance differences.

What would settle it

Take a random sample of the 9,000 style-variant pairs, have blinded domain experts mark each pair as same-substance or altered-substance, and recompute SBI/SRR/AWR on the subset certified as same-substance. If the certified subset shows markedly smaller style shifts, the reported bias conflates content changes with style; if it reproduces the reported gaps, the claim is confirmed.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

Existing idea-selection pipelines that rely on direct LLM judging inherit a style lottery: identical scientific content can enter or leave Top-K sets based on rhetorical packaging.
Adding a style-aware extractor in front of a frozen judge is a lightweight alternative to retraining the judge, and it can improve substance recognition without collapsing score variance.
The three-metric joint criterion means future judge evaluations should not report score stability alone; a judge that assigns nearly identical scores to everything can trivially achieve low SBI.
External context, whether fixed-domain or retrieved, does not reliably remove style sensitivity, so retrieval augmentation alone is not a debiasing fix.
Substance-perturbed and hybrid variants remain the hardest cases, indicating that judges are most confusable when persuasive presentation is combined with hollow or flawed content.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The paper's content-preservation check covers only a small slice of the variant space, so a natural next test is to extend human review to all 9,000 pairs and recompute the bias metrics after excluding pairs where human reviewers detect content changes.
Because the extractor is trained per judge, its gains may partly reflect adaptation to that judge's specific scoring patterns; testing on held-out judges not seen during training would separate generic style awareness from judge-specific overfitting.
The same style-versus-substance confound likely operates in other scientific evaluation tasks, such as paper review, grant scoring, and hypothesis ranking; the SciStyleBench taxonomy could be adapted into a stress test for those settings.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful diagnostic benchmark for style bias in LLM judges of scientific ideas, but the variant validity check covers only 6 of 15 types and has a 320/240 mismatch—referee with major revision.

read the letter

Here's my read. The paper's real contribution is a benchmark, SciStyleBench, that tests LLM judges on complete scientific ideas under controlled stylistic perturbations—holding content fixed while changing framing, verbosity, confidence, etc.—and then measures score shifts, substance discrimination, and Top-K membership changes. Prior work on style-over-substance bias mostly used generic responses, so this is a genuine step forward for the AI-scientist evaluation community. The three metrics (SBI, SRR, AWR) are coherent, and the paper correctly warns that low SBI can be score collapse; the OpenReviewer case where SBI is near zero but SRR and AWR are also near zero makes that point cleanly. The extractor module—predicting style type and deviation, then injecting a bias-control hint into a frozen judge—is a sensible mitigation and the reported improvements are meaningful if the variant labels are right.

The soft spot is exactly the variant labels. The central claim requires that all 15 variants preserve scientific substance and change only presentation. The human validation in Table 3 covers only 6 of the 15 types. It omits Novelty Emphasis (B4) and Application Framing (B5), which could introduce new content claims, and it omits the Deceptive Hollow, Confident Flaw, and Verbose Enriched variants—the ones that define the ground truth for SRR and AWR. If those generations miss their intended labels, the headline numbers partly measure generation artifacts instead of judge behavior. There's also an internal inconsistency: the text says 320 pairs were sampled, but Table 3 sums to 240, so the aggregate 92.5% preservation rate is not anchored. On the extractor, no train/test split is disclosed; the ideal style signal comes from the same taxonomy used for evaluation, so the mitigation numbers could be optimistic. The intro promises theoretical analysis of aggregation dilution that I couldn't find in the body.

These are fixable, not fatal. The core diagnostic—direct judges are style-sensitive and weak at substance discrimination—is plausible and the benchmark concept is useful. This paper is for people building automated research pipelines or LLM-based review systems; it should go to peer review with a request for major revision. I would not rely on the headline numbers until the validation gap is closed.

Referee Report

4 major / 5 minor

Summary. The paper investigates whether LLM-as-a-judge evaluations of scientific ideas are influenced by surface-level presentation style rather than scientific substance. It introduces SciStyleBench, a benchmark with 600 source ideas, 15 controlled style/substance variants, and three background settings (no context, fixed-domain, idea-specific retrieval). It proposes three metrics—Style Bias Index (SBI), Substance Recognition Rate (SRR), and Adversarial Win Rate (AWR)—and a trained module, SciStyleExtractor, that injects style-related signals into a frozen judge to reduce style bias. The main reported results are that direct LLM judges achieve SBI/SRR/AWR of 0.566/0.504/0.554, while SciStyleExtractor improves these to 0.501/0.759/0.899. The paper also shows that style transformations change Top-K membership (4.6% at Top-5 to 15.9% at Top-30), which motivates the practical importance of the problem.

Significance. If the results hold, this is a useful and timely contribution: it provides a controlled diagnostic benchmark for a real problem in LLM-based scientific idea evaluation, a three-metric framework that explicitly guards against score collapse, and a concrete mitigation module. The three-stage context manipulation and the Top-K analysis are strengths, as is the reporting of standard errors in Table 1. However, the significance is contingent on two conditions: that the variant generations truly preserve or alter scientific substance as intended, and that SciStyleExtractor is not trained on the same instances on which it is evaluated. Both conditions are currently not established in the manuscript, so the headline comparisons should be treated as provisional.

major comments (4)

[Rewrite Validation/Table 3] The validation is internally inconsistent: the text says 320 source-variant pairs were sampled, but Table 3 sums to 240. Only 6 of 15 variant types are validated (Paraphrase, Plain Core, Verbose, Grand Narrative, Overconfident, Ultimate Hype). B4/B5 (style-only but content-adjacent) and C1/C2/C3/H1/H2/H3 (the very labels used in SRR/AWR, Eqs. (3)-(4)) are not. If these generations fail their intended labels, the headline SBI/SRR/AWR numbers partly measure generation artifacts. Provide per-type validation for all 15 variants and reconcile the counts.
[Extractor-Aligned Training and Ablation] The extractor is trained on 2,000 pairs from SciStyleBench and then evaluated on SciStyleBench, with no train/test split reported. The ideal style signal z* in Eq. (5) comes from the same 15-variant taxonomy used to build the benchmark, so training and evaluation may share the same labels. This contamination risk could inflate the reported SBI/SRR/AWR gains. Specify which ideas/variants/settings are used for training and confirm that Table 1/4 numbers are on held-out data.
[Eq. (2)] The Plain terms cancel algebraically, so SBI(p) = E|J(I_p)-J(I_para)|. The Plain variant is therefore not a reference; SBI measures deviation from the paraphrase variant, not from the unperturbed idea. This contradicts the text's claim that SBI answers 'when substance is unchanged, how much does the score change because of style?' Redefine the metric (e.g., use E|J(I_p)-J(I_plain)|) or revise the interpretation of all SBI values.
[Main Result/Table 1] The abstract and text quote aggregate numbers (0.566/0.504/0.554 for Direct; 0.501/0.759/0.899 for SciStyleExtractor), but Table 1 has only per-judge, per-setting values. It is unclear whether the averages include all six judges or only the 'four general-purpose judges', and whether settings are equally weighted. Give the exact aggregation rule and include an aggregated row in the table.

minor comments (5)

[Introduction/Contributions] The contribution list refers to 'SBI, ADR, and SRR', but ADR is never defined and the metrics section defines AWR. Replace ADR with AWR throughout.
[Table 2] The column heading 'Overall Top-50 Five-score Top-5' concatenates several conditions. Spell out what each column measures and add a note on how the Spearman correlations were computed.
[Figures 4-5] Annotations such as '+0.10' and 'Better rank shift' are difficult to read at print size; provide a separate legend or larger fonts.
[Ablation] Give full training hyperparameters (LoRA rank, learning rate, batch size, selection procedure for the 2,000 training pairs) so the extractor can be reproduced.
[Experimental Setup] Clarify whether the 15 variants per idea include the original Identity variant or only the 14 perturbed variants plus identity; this affects how the 9,000 instances per setting are counted.

Circularity Check

3 steps flagged

Substance ground truth and extractor training signal both come from the authors' own 15-type taxonomy; validation covers only 6 types and is internally inconsistent, so SRR/AWR and the extractor's reported gains are partially self-referential, though direct-judge measurements are genuinely empirical.

specific steps

self definitional [SciStyleMetrics (Eq. 3, Eq. 4); SciStyleStage 'Style Perturbation Space']
"Let Qsub contain ordered pairs (a, b) in which a has stronger substance than b, including Enriched over Plain, Plain over Hollow, and Plain over Flawed ... C1 (Hollow), which removes substantive support while retaining persuasive presentation; C2 (Flawed), which introduces logical weaknesses; and C3 (Enriched), which adds substantive information."

SRR and AWR treat the authors' variant taxonomy as ground truth: Enriched>Plain>Hollow/Flawed and high-substance-plain > low-substance-persuasive are definitions written into the perturbation space, not externally established properties. The generated variants are machine-written (DeepSeek-V3); whether C1/C2/C3/H1/H2/H3 actually realize the intended substance ordering is asserted, not validated. Table 3 validates only Paraphrase, Plain Core, Verbose, Grand Narrative, Overconfident, and Ultimate Hype; no Hollow, Flawed, Enriched, Deceptive Hollow, Confident Flaw, or Verbose Enriched pairs are evaluated. The metric therefore partially measures the judge's agreement with the authors' labels for unvalidated LLM-generated text rather than independently verified substance.
fitted input called prediction [SciStyleExtractor 'Extractor-Aligned Training' (Eqs. 5-6); Ablation]
"For each input (Ip, K), the teacher judge receives an ideal style signal z*_p, whereas the student judge receives the predicted signal zp = f_phi(Ip, K)... Minimizing this objective teaches the extractor to generate auxiliary style signals that make the student approximate the teacher's debiased evaluation behavior."

The 'ideal style signal' z* is the authors' own 15-type taxonomy label (type, deviation from neutral, bias hint) attached to each benchmark variant. The extractor is fitted to reproduce these labels on 2,000 training pairs from the same benchmark, and its mitigation claim is then reported as SBI/SRR/AWR improvements on that same benchmark, with no disclosed train/evaluation split. The metrics SBI/SRR/AWR are defined against the same taxonomy (Plain/Paraphrase reference, Q_sub, Q_adv). Hence the reported gains are an in-distribution check of the judge following the extractor's replayed taxonomy labels, not an independent test of style-invariance. The reduction is partial: results vary across judges (e.g., Llama-3.1-8B SBI rises to 0.912), so it is not strictly forced.
other [Rewrite Validation paragraph; Table 3]
"Second, to validate that our perturbations modify presentation style while preserving scientific content, we randomly sampled 320 source-variant pairs for human evaluation. ... As shown in Table 3, 93.1% of pairs preserved the original scientific substance, 90.9% maintained equivalent perceived quality, and 97.9% were identified as presentation-level variations. [Table 3] Overall 240 92.5 89.7 97.8"

The substance-preservation premise that anchors the whole style/substance interpretation is supported only by this validation, which is internally inconsistent: the text says 320 sampled pairs but Table 3 sums to 240 (6 variants x 40); the text quotes 93.1/90.9/97.9 while the table reports 92.5/89.7/97.8; and the validated types exclude all C and most H variants, including Novelty Emphasis (B4) and Application Framing (B5) which could introduce new content claims. The load-bearing premise is therefore not reliably anchored for the variant types that define SRR and AWR, making the reported diagnostic numbers partially functions of generation artifacts rather than pure measurements.

full rationale

The paper's core diagnostic direct-judge measurements are not circular: SBI/SRR/AWR are computed from frozen LLM judges with no access to the taxonomy, so values like SBI 0.566 and SRR 0.504 are genuine empirical observations. The circularity concerns are (1) the ground-truth substance ordering for SRR/AWR is definitional for the 15-type taxonomy (C1/C2/C3/H1/H2/H3 are assumed to realize their names), and the paper's own human validation covers only 6 of 15 types, excludes all C types, and is internally inconsistent (320 vs 240 pairs; mismatched percentages); (2) the SciStyleExtractor is trained on the same taxonomy's labels (ideal style signal z*) and evaluated on the same benchmark's metrics with no disclosed split, so the reported SBI/SRR/AWR gains are in-distribution label-conditional behavior rather than independent generalization; and (3) no load-bearing self-citation or imported uniqueness theorem appears, so patterns 3-5 score zero. Because the extractor's gains are judge-dependent and imperfect (SBI drops only from 0.566 to 0.501 and rises for some judges), the mitigation is not mathematically forced, which caps the severity. Score 3: partial self-referentiality in ground-truth construction and extractor evaluation, with the central diagnostic measurement retaining independent empirical content.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical or mathematical entities. The main assumptions are about the validity of the generated variants and the experimental protocol. The extractor's ideal style signal and the unmotivated train/test split are the most fragile assumptions.

axioms (4)

domain assumption LLM-generated style variants preserve scientific substance and change only presentation style.
Core to interpreting SBI as style-induced bias; validated on only 240 pairs for 6 of 15 variants (Rewrite Validation, Table 3).
domain assumption Human judgments of scientific quality are reliable ground truth for validating variants.
Used to claim 90.9% perceived-quality-equivalence in Table 3; no inter-annotator agreement metric is reported.
ad hoc to paper The 'ideal style signal' z* accurately separates style from substance and correctly debiases the teacher judge.
Used in extractor training (Eq. 6); z* is defined by the authors' own taxonomy, and its debiasing effect is assumed rather than independently verified.
ad hoc to paper The SciStyleBench evaluation is not contaminated by training data for the extractor.
The paper never describes a train/test split; the Ablation mentions 2,000 training pairs, leaving open the possibility that evaluation uses training data.

pith-pipeline@v1.3.0-daily-deepseek · 15944 in / 11839 out tokens · 123694 ms · 2026-08-04T23:17:16.929477+00:00 · methodology

0 comments

Cite this review

Pith. "Pith review of Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation." pith.science (2026). https://pith.science/paper/RACFST22

@misc{pith2026260801666,
  author       = {Pith},
  title        = {Pith review of: Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RACFST22}},
  note         = {Machine review of arXiv:2608.01666}
}

Share X Bluesky LinkedIn Reddit HN

read the original abstract

However, whether these judges truly evaluate the scientific substance of ideas or are influenced by superficial stylistic presentation remains an open question. To address this question, we propose SciStyleBench, a unified three-component benchmark for diagnosing and mitigating stylistic bias in LLM-based idea evaluation: (i) First, SciStyleStage, a three-stage evaluation environment that applies controlled stylistic perturbations to fixed scientific content across three settings no context, fixed-domain context, and open-domain retrieval context, covering 600 scientific ideas and 15 style variants, with 9,000 evaluation instances per setting; (ii) Second, SciStyleMetrics, a set of quantitative measures, including Style Bias Index (SBI), Substance Recognition Rate (SRR), and Adversarial Win Rate (AWR), to characterize how stylistic variation affects scoring stability, substance discrimination, and ranking robustness; (iii) Third, SciStyleExtractor, a plug-and-play evaluation module that separates presentation style from scientific content by predicting style type and deviation before style-conditioned evaluation, enabling us to assess whether style awareness reduces stylistic bias. Experiments on SciStyleBench show that direct LLM judges remain sensitive to writing style and struggle to distinguish scientific substance. In contrast, SciStyleExtractor reduces SBI from 0.566 to 0.501 while increasing SRR and AWR from 0.504 and 0.554 to 0.759 and 0.899, respectively. These results suggest that robust idea evaluation requires invariance to stylistic variation without sacrificing sensitivity to scientific substance. Overall, SciStyleBench provides a systematic framework for identifying, quantifying, and mitigating stylistic bias in scientific idea evaluation.

Figures

Figures reproduced from arXiv: 2608.01666 by Fan Zhang, Fengxian Ji, Jingpu Yang, Juanfan Wu, Min Peng, Qianqian Xie, Xiuying Chen, Yuke Li, Yu Xie, Zhexuan Cui, Zhuohan Xie.