Pith. sign in

REVIEW 5 major objections 5 minor 37 references

What is Good? Extracting and Testing Implicit Theories of Literary Quality from LLM Reasoning Traces

T0 review · 5 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An LLM's quality judgments are far more sensitive to sentence structure and voice than to vocabulary, despite its reasoning traces praising word choice.

desk verdict A promising but under-validated degradation study: the vocabulary-insensitivity result is plausible, but the manual manipulations are too confounded to carry the weight the paper puts on them. read the letter →

arxiv 2607.20425 v1 pith:5AHIRIVR submitted 2026-04-07 cs.CL

Birger Moëll This is my paper
classification cs.CL
keywords largelanguagemodelsliteraryqualityreasoningtracesdegradationanalysisLLM-as-judgecomputationalstudiesvocabularyparadoxevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks what a reasoning-enabled LLM implicitly thinks makes writing good, by first extracting its stated theory from reasoning traces and then testing it through systematic degradation of literary passages. The central finding is that the model's quality assessments are much more sensitive to structure and voice than to vocabulary: simplifying words like 'spiteful' to 'mean' costs only 0.41 points out of 10, while flattening structure costs 2.78 and genericizing voice costs 2.34. The model's traces praise 'precise vocabulary,' yet its revealed preferences show vocabulary is nearly a non-factor—a discrepancy that matters for anyone using LLMs as writing judges. The same broad pattern appears across five replications and a second model, and combined degradation is catastrophic but subadditive, suggesting a quality floor. These results have direct implications for automated writing feedback and for how much trust to place in LLM-generated rationales.

What carries the argument

The degradation-based sensitivity analysis: five canonical passages (Morrison, García Márquez, Woolf, Fitzgerald, Orwell) each rewritten in six controlled manipulations—vocabulary simplification, rhythm flattening, imagery removal, voice genericization, structure simplification, and combined—and re-scored through the same LLM evaluation pipeline. This turns the model's verbalized theory into a probe: if a feature matters to the evaluator, degrading it should lower the score. The complementary machinery is the reasoning-trace extraction, which mines 30 traces for the model's stated dimensions and quality vocabulary.

What would settle it

Use algorithmic degradation on a set of high-quality, non-canonical passages (to strip out source recognition), varying only word choice via frequency-matched synonyms while keeping sentence length and rhythm constant; if the model's quality score drops by more than about one point on a 10-point scale, the paper's central claim that vocabulary is a secondary quality signal is wrong.

Watch

Extended reading notes

Core claim

Across five independent DeepSeek runs and an exploratory Qwen comparison, the paper demonstrates that the LLM's quality assessments are holistic, author-specific, and primarily structure-driven. Vocabulary simplification produces a mean quality loss of only 0.41 ± 0.46 points—statistically significant but an order of magnitude below structure (2.78) or voice (2.34); imagery removal (0.56) is also nearly inert. Combined degradation is the most destructive (−5.64) but subadditive, with the sum of individual losses (8.1) exceeding the combined effect, interpreted as a quality floor. The model's stated theory from 36,000+ characters of reasoning traces values intentionality, craft, depth, and di

Load-bearing premise

The manually written degraded versions isolate exactly one quality dimension each; if the vocabulary edit also changes rhythm or the structure edit also changes meaning, the sensitivity ranking is an artifact of how the degradations were composed.

Editorial extensions

If this is right

  • Automated writing feedback built on LLM judges will systematically underweight vocabulary and overweight sentence architecture, unless the evaluator is retrained or calibrated.
  • Voice is author-specific in the model's eyes: Woolf collapses under genericization (−5.7) while Orwell barely moves (−0.4), so universal 'voice' prescriptions in feedback tools are unsupported.
  • Reasoning traces overstate the role of lexical choice: the vocabulary paradox means chain-of-thought rationales should not be taken as faithful explanations of LLM scoring.
  • Combined manipulation always crushes scores (−5.64), so multi-feature quality is more than the sum of its parts in practice, even if the worst case is bounded by a floor.
  • Source recognition inflates scores for canonical texts, so benchmark tiers built from famous excerpts may overestimate how well LLMs judge intrinsic quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested extension: the same degradation protocol could be run on human readers. If humans also show vocabulary insensitivity, the result reflects a general aesthetic property rather than an LLM artifact; if not, it is a model-specific bias.
  • The vocabulary finding could be sharpened by using algorithmic, normed degradations (e.g., replacing words by frequency-matched synonyms while keeping sentence rhythm intact) on non-canonical high-quality texts, eliminating the recognition confound.
  • The subadditive combined loss suggests a floor effect; testing progressively weaker combined perturbations (e.g., only two features at a time) would map the interaction surface and tell whether features truly interact or merely saturate.
  • For computational literary studies, the method offers a way to operationalize 'craft' and 'voice' as causally testable dimensions rather than vague critical terms.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper presents a two-study investigation of how reasoning-enabled LLMs evaluate literary quality. Study 1 extracts an 'implicit theory' from DeepSeek R1 reasoning traces over a 30-text benchmark spanning six quality tiers, reporting 79.3% mean tier-classification accuracy across five replications and a stated theory that values intentionality, craft, depth, and voice. Study 2 tests this theory by manually creating six degraded versions of five canonical literary passages and measuring quality-score loss, finding that vocabulary simplification causes the smallest loss (0.41 ± 0.46 points), while structure (2.78) and voice (2.34) are far more impactful, and combined degradation is catastrophic but subadditive. An exploratory comparison with Qwen QwQ shows a similar broad pattern. The paper also includes a familiarity-bias experiment and extensive replication and cross-model validation.

Significance. If the central finding is robust, the paper makes a valuable contribution to computational aesthetics and to understanding LLM evaluator behavior: it demonstrates that LLM quality judgments can be probed via degradation and that stated reasoning about vocabulary may diverge from operative sensitivity. The work's strengths include five independent replications, cross-model validation with QwQ, concrete statistical reporting (CIs, effect sizes), and unusually honest treatment of limitations. The reproducibility package and the clear separation of observation versus sensitivity analysis are also positive. However, the central claim of Study 2 rests on a small, manually constructed set of degradations whose validity and mutual isolation are not established; the statistical analysis also appears to treat replications incorrectly, and a source-recognition confound is acknowledged but not resolved. These issues currently prevent full confidence in the headline 'vocabulary is secondary' result.

major comments (5)
  1. [§4.2, Table 4] The six degradation manipulations are defined by example, not by algorithmic procedure or independent validation. The table shows that the 'voice' version of the Morrison passage changes not only authorial personality but also propositional content and syntax ('haunted'/'malevolent presence' vs. 'spiteful'/'baby's venom'), while the 'structure' version fragments prose and reduces information. Vocabulary simplification, by contrast, preserves syntax and the recognizable canonical opening. If the manipulations differ in severity or inadvertently alter multiple dimensions, the ranking in Table 5 (vocabulary 0.41 vs. structure 2.78, voice 2.34) could be an artifact of how the degradations were written. The paper's own §5.5 lists 'manual degradation' as a limitation, but this is the load-bearing instrument for the central claim; I would need to see either algorithmic control (e.g., matched-le
  2. [§4.6, Table 5] The Wilcoxon signed-rank tests appear to pool all 5 replications × 5 texts as 25 independent observations. Since the same five texts are reused across replications, observations within each text are not independent; clustering or text-level analysis is required. For vocabulary loss, the five text-level means (approximately 0.5, 0.1, 0.7, 0.5, 0.4, from Figure 5) would not yield p = 3.1×10⁻⁴; a one-sample Wilcoxon on n=5 gives p ≈ 0.06 (two-sided). A correctly clustered test could render the 'statistically significant but tiny' vocabulary effect non-significant, which matters because the paper uses this significance to refine the initial 'zero loss' observation. Please report tests based on text-level averages or a mixed model with random effects for text.
  3. [§5.2] The argument that source recognition cannot explain vocabulary insensitivity is incomplete. The paper notes that combined degradation still reduces scores, but this does not rule out the proposed mechanism: vocabulary-simplified passages retain the recognizable canonical opening (e.g., '124 was spiteful...') and hence may benefit from recognition anchoring, whereas structure/voice manipulations destroy those recognition cues. The §5.5 limitation acknowledges this confound is 'particularly acute for the vocabulary finding,' but the discussion in §5.2 dismisses it too quickly. A direct test would evaluate degraded versions of non-canonical but high-quality texts (as the paper itself suggests for future work), or at least report which degraded versions the model still identifies by author/title. Without such a test, the central 'vocabulary paradox' remains vulnerable to the recognition-anch
  4. [Figure 2, §3.4] The confusion matrix reports counts that are incompatible with the stated 5-text-per-tier design. For example, the Literary row shows 70% (7) and 30% (3), summing to 10; similar counts appear for other tiers. If this is a single-run matrix, counts should sum to 5; if it is pooled across replications, they should sum to 25 (or 30). The text says the model achieves 24/30 accuracy, but the figure's numbers do not correspond to any obvious pooling. This is a significant reporting inconsistency that undermines one of the paper's Study 1 contributions; please correct the figure or clarify its construction.
  5. [§3.7 and Figure 3] The dimension-gap numbers are internally inconsistent. The text states craft +5.2 and depth +4.6, with voice +3.1 as the smallest gap; the figure caption repeats these values, but the figure itself shows Prose +5.0, Depth +4.9, Craft +5.0, Voice +2.0, Impact +2.6. These discrepancies affect the claimed 'dimension hierarchy' from Study 1. Please harmonize the numbers.
minor comments (5)
  1. [Abstract and §4.6] The abstract states '80% classification accuracy' while §4.6 reports a mean of 79.3% with a confidence interval [76.0, 82.2]. The rounded value is acceptable, but it would be clearer to state the replicated mean in the abstract as well.
  2. [Table 6] Note explicitly in the table caption that the QwQ voice mean is based on n=4 passages because of the Fitzgerald connection error, to avoid giving an impression of equal n across all degradation types.
  3. [§4.4 / Figure 5] The heatmap values are difficult to parse in the text rendering; ensure that the figure is legible in the final PDF and that values match the prose (e.g., Morrison 'voice −1.7±0.7' appears as 1.7 in the figure but the sign convention is 'loss', so be consistent about sign).
  4. [§5.4] The phrase 'craft (operationalized as sentence structure)' is a strong simplification; Study 1's craft dimension encompasses more than sentence structure. Clarify that the sensitivity analysis probes one operationalization of craft.
  5. [§3.5] The claim that recognition 'correlates with lower accuracy' is based on 66.7% (4/6) vs. 83.3% (20/24); the difference is not statistically meaningful at these sample sizes. The paper does hedge, but the heading is stronger than the evidence. Suggest softening.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study is an observational/sensitivity analysis of LLM behavior; no prediction is derived from a fitted parameter or self-citation chain.

full rationale

The paper's central claims rest on two empirical operations: (1) extracting a stated theory from reasoning traces and (2) measuring score changes under hand-authored degradation. Neither reduces to its inputs. The theory extraction is a summary of observable traces; the degradation experiment compares model scores across text versions. The conclusion that vocabulary loss is smallest (0.41 vs. 2.78 for structure, Table 5) is a direct measurement result, not a consequence of how the vocabulary degradation was defined. The authors explicitly acknowledge the nearest circularity-adjacent threats in Section 5.5: manual degradation choices, source-recognition buffering of vocabulary-simplified passages, and reasoning-trace unfaithfulness. These are disclosed as limitations and confounds, not used as load-bearing derivations. There are no self-citations to prior work by the author, no imported uniqueness theorem, no fitted parameter renamed as a prediction, and no ansatz smuggled in via citation. The unvalidated manipulations and recognition confound are construct-validity and external-validity risks, which belong to correctness assessment rather than circularity analysis. The 'quality floor' explanation for subadditivity is post hoc but does not make the empirical finding tautological. The derivation chain is therefore self-contained and not circular.

Assumptions & free parameters 0 free parameters · 5 assumptions · 1 invented entities

The paper introduces no fitted numerical parameters. The central claims rest on domain assumptions about the validity of manual degradations, the representativeness of the two LLMs, and the use of reasoning traces as credible evidence. The only invented explanatory construct is the 'quality floor'.

assumptions (5)
  • domain assumption The six manual degradations isolate the target features (vocabulary, rhythm, imagery, voice, structure) while holding other quality dimensions constant.
    Section 4.2 describes degradations; no algorithmic control or inter-rater reliability is provided, so this is an untested premise.
  • domain assumption Quality scores from DeepSeek R1 and Qwen QwQ are stable and meaningful measures of the models' judgments, and these two models are representative reasoning-enabled LLMs.
    The entire study operationalizes 'sensitivity' via these scores; both models share overlapping training data (Section 5.5).
  • domain assumption Reasoning traces provide a usable signal of the model's evaluation criteria despite known unfaithfulness.
    Authors acknowledge unfaithfulness (Section 2.5) but still extract the implicit theory from traces.
  • domain assumption The tier labels (Literary, Prestige Journalism, ..., Chan) reflect an ordinal quality hierarchy assumed as ground truth for classification accuracy.
    No human ratings validate the ordering; the paper itself notes this in the no-human-baseline limitation.
  • ad hoc to paper Statistical tests treat observations as independent across replications.
    Section 4.6 uses Wilcoxon tests on the pooled 25 loss values per degradation type without accounting for text-level clustering, which may inflate significance.
invented entities (1)
  • Quality floor
    purpose: Posits a lower bound on quality scores that prevents individual degradation losses from combining additively, explaining subadditivity.
    In Section 5.3 the subadditivity is attributed to a 'quality floor effect' but no independent measurement of the floor is given; it is inferred from the same data it explains.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What is Good? Extracting and Testing Implicit Theories of Literary Quality from LLM Reasoning Traces." pith.science (2026). https://pith.science/paper/5AHIRIVR

@misc{pith2026260720425,
  author       = {Pith},
  title        = {Pith review of: What is Good? Extracting and Testing Implicit Theories of Literary Quality from LLM Reasoning Traces},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5AHIRIVR}},
  note         = {Machine review of arXiv:2607.20425}
}
read the original abstract

What makes writing "good" remains a persistent question in literary studies and computational linguistics. We present a two-study investigation of how reasoning-enabled LLMs evaluate literary quality. In Study 1, we construct a benchmark of 30 real texts spanning six quality tiers, from canonical literature to anonymous forum posts, and extract the model's implicit theory of quality from its reasoning traces. Across five DeepSeek replications, the model achieves 79.3% mean tier-classification accuracy. The traces reveal a consistent stated theory: the model values intentionality over correctness, prioritizing craft, depth, and distinctive voice. A familiarity experiment with style-matched but unrecognizable passages suggests that source recognition may inflate scores, although this is confounded by genuine quality differences between canonical originals and researcher-written pastiches. In Study 2, we probe this theory through systematic degradation of five canonical prose passages. We apply six manipulations - vocabulary simplification, rhythm flattening, imagery removal, voice genericization, structure simplification, and combined degradation - and reevaluate each version. Vocabulary simplification causes the smallest quality loss (0.41 +/- 0.46 points), far below structure (2.78) or voice (2.34) loss. Combined degradation is devastating (-5.64) but subadditive. An exploratory comparison with Qwen QwQ shows the same broad qualitative pattern. Together, these studies suggest that LLM judgments of writing quality are holistic, author-specific, and more sensitive to structural than lexical features, with implications for automated writing feedback and computational aesthetics.

Figures

Figures reproduced from arXiv: 2607.20425 by the authors.

Figure 1
Figure 1. Readability metrics by tier. News writing is most complex; literary prose is moderate. Com [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Confusion matrix. Literary and news achieve perfect recognition. Didion and Orwell are [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Left: Dimension scores by tier. Right: Gap between high and low tiers. Craft (+5.2) and depth [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Average quality loss by degradation type across replications. Vocabulary simplification causes [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Author sensitivity heatmap (mean across replications). Woolf is devastated by voice generi [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Quality trajectories. All authors start at 9–10 but follow different degradation paths. All [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 5 linked inside Pith

  1. [1]

    Rhetoric

    Aristotle. Rhetoric. Dover Publications, -350. Translated by W. Rhys Roberts, 1924

    1924

  2. [2]

    The Well Wrought Urn: Studies in the Structure of Poetry

    Cleanth Brooks. The Well Wrought Urn: Studies in the Structure of Poetry. Harcourt Brace, 1947

    1947

  3. [3]

    Writing with Power: Techniques for Mastering the Writing Process

    Peter Elbow. Writing with Power: Techniques for Mastering the Writing Process. Oxford University Press, 1981

    1981

  4. [4]

    The Art of Fiction: Notes on Craft for Young Writers

    John Gardner. The Art of Fiction: Notes on Craft for Young Writers. Vintage Books, 1991

    1991

  5. [5]

    The technique of clear writing

    Robert Gunning. The technique of clear writing. McGraw-Hill, 1952

    1952

  6. [6]

    A computational analysis of style, affect, and imagery in contemporary poetry

    Justine Kao and Dan Jurafsky. A computational analysis of style, affect, and imagery in contemporary poetry. In Proceedings of the NAACL-HLT Workshop on Computational Linguistics for Literature, pages 8--17, 2012

    2012

  7. [7]

    Automated essay scoring: A survey of the state of the art

    Zixuan Ke and Vincent Ng. Automated essay scoring: A survey of the state of the art. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pages 6300--6308, 2019

    2019

  8. [8]

    Peter Kincaid, Robert P

    J. Peter Kincaid, Robert P. Fishburne Jr, Richard L. Rogers, and Brad S. Chissom. Derivation of new readability formulas for navy enlisted personnel. Research Branch Report, 8 0 (75), 1975

    1975

Show all 37 references
  1. [11]

    Rlaif: Scaling reinforcement learning from human feedback with AI feedback

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with AI feedback. arXiv preprint arXiv:2309.00267, 2024

    2024 arXiv

  2. [12]

    G- E val: NLG evaluation using GPT-4 with better human alignment

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G- E val: NLG evaluation using GPT-4 with better human alignment. In Proceedings of EMNLP, 2023

    2023

  3. [13]

    A survey of modern authorship attribution methods

    Efstathios Stamatatos. A survey of modern authorship attribution methods. Journal of the American Society for Information Science and Technology, 60 0 (3): 0 538--556, 2009

    2009

  4. [14]

    Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting. In Advances in Neural Information Processing Systems, volume 36, 2024

    2024

  5. [15]

    A review of deep-neural automated essay scoring models

    Masaki Uto. A review of deep-neural automated essay scoring models. Behaviormetrika, 48: 0 459--484, 2021

    2021

  6. [17]

    Wilson and Jonathan W

    Timothy D. Wilson and Jonathan W. Schooler. Thinking too much: Introspection can reduce the quality of preferences and decisions. Journal of Personality and Social Psychology, 60 0 (2): 0 181--192, 1991

    1991

  7. [18]

    Judging LLM -as-a-judge with MT-Bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM -as-a-judge with MT-Bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2023

    2023

  8. [19]

    -350 , note=

    Rhetoric , author=. -350 , note=

  9. [20]

    1947 , publisher=

    The Well Wrought Urn: Studies in the Structure of Poetry , author=. 1947 , publisher=

    1947

  10. [21]

    2014 , publisher=

    The Elements of Journalism: What Newspeople Should Know and the Public Should Expect , author=. 2014 , publisher=

    2014

  11. [22]

    1991 , publisher=

    The Art of Fiction: Notes on Craft for Young Writers , author=. 1991 , publisher=

    1991

  12. [23]

    1981 , publisher=

    Writing with Power: Techniques for Mastering the Writing Process , author=. 1981 , publisher=

    1981

  13. [24]

    Research Branch Report , volume=

    Derivation of New Readability Formulas for Navy Enlisted Personnel , author=. Research Branch Report , volume=. 1975 , publisher=

    1975

  14. [25]

    McGraw-Hill , year=

    The Technique of Clear Writing , author=. McGraw-Hill , year=

  15. [26]

    Journal of Applied Psychology , volume=

    A Computer Readability Formula Designed for Machine Scoring , author=. Journal of Applied Psychology , volume=

  16. [27]

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric and others , journal=. Judging

  17. [28]

    RLAIF: Scaling Reinforcement Learning from Human Feedback with

    Lee, Harrison and Phatale, Samrat and Mansoor, Hassan and Lu, Kellie and Mesnard, Thomas and Bishop, Colton and Carbune, Victor and Rastogi, Abhinav , journal=. RLAIF: Scaling Reinforcement Learning from Human Feedback with

  18. [29]

    Liu, Yang and Iter, Dan and Xu, Yichong and Wang, Shuohang and Xu, Ruochen and Zhu, Chenguang , booktitle=. G-

  19. [30]

    Journal of Personality and Social Psychology , volume=

    Thinking Too Much: Introspection Can Reduce the Quality of Preferences and Decisions , author=. Journal of Personality and Social Psychology , volume=

  20. [31]

    arXiv preprint arXiv:2412.19437 , year=

    DeepSeek-V3 Technical Report , author=. arXiv preprint arXiv:2412.19437 , year=

    arXiv

  21. [32]

    Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI) , pages=

    Automated Essay Scoring: A Survey of the State of the Art , author=. Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI) , pages=

  22. [33]

    Behaviormetrika , volume=

    A Review of Deep-neural Automated Essay Scoring Models , author=. Behaviormetrika , volume=. 2021 , publisher=

    2021

  23. [34]

    Journal of the American Society for Information Science and Technology , volume=

    A Survey of Modern Authorship Attribution Methods , author=. Journal of the American Society for Information Science and Technology , volume=

  24. [35]

    Proceedings of the NAACL-HLT Workshop on Computational Linguistics for Literature , pages=

    A Computational Analysis of Style, Affect, and Imagery in Contemporary Poetry , author=. Proceedings of the NAACL-HLT Workshop on Computational Linguistics for Literature , pages=

  25. [36]

    Advances in Neural Information Processing Systems , volume=

    Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting , author=. Advances in Neural Information Processing Systems , volume=

  26. [37]

    arXiv preprint arXiv:2307.13702 , year=

    Measuring Faithfulness in Chain-of-Thought Reasoning , author=. arXiv preprint arXiv:2307.13702 , year=

    arXiv

  27. [38]

    arXiv preprint arXiv:2305.17926 , year=

    Large Language Models are not Fair Evaluators , author=. arXiv preprint arXiv:2305.17926 , year=

    arXiv

  28. [39]

    arXiv preprint arXiv:2309.17012 , year=

    Benchmarking Cognitive Biases in Large Language Models as Evaluators , author=. arXiv preprint arXiv:2309.17012 , year=

    arXiv

  29. [40]

    On the Limitations of Using

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and others , journal=. On the Limitations of Using. 2024 , note=

    2024

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.