Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation

Ariel Gera; Noam Slonim; Noy Sternlicht; Roy Bar-Haim; Tom Hope

REVIEW 3 major objections 4 minor 15 references

Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM judges can now match human raters on debate speech quality

desk verdict Solid, reusable LLM-judge benchmark for long debate speeches, but the 'surpassing humans' claim is an artifact of comparing a deterministic model to noisy human raters. read the letter →

arxiv 2506.05062 v2 pith:VCFARHJ7 submitted 2025-06-05 cs.CL

Noy Sternlicht , Ariel Gera , Roy Bar-Haim , Tom Hope , Noam Slonim This is my paper · ORCID

classification cs.CL

keywords LLM-as-a-judgedebatespeechevaluationbenchmarkargumentqualitychain-of-thoughtpromptingKendall'sTau-Chuman-LLMagreementpersuasivetextgeneration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that rating long debate speeches is a hard, realistic test for LLM-as-a-judge systems, and that on this test larger LLMs can match individual human raters. Over 631 speeches, each scored by 15 crowd annotators, the best models reach or slightly exceed the average human-human agreement, while models below 7B parameters lag. The same models, however, do not judge the way humans do: they hand out lower scores and separate strong from weak speeches more sharply. The paper also shows that speeches written by GPT-4.1 are rated above speeches by human expert debaters, which it reads as evidence of rapid progress and as a warning about persuasive-text misuse.

What carries the argument

The load-bearing object is the Speech Quality Dataset of 631 debate speeches from Slonim et al. (2021), each with 15 crowd ratings; agreement is measured two ways: leave-one-out pairwise weighted Kappa against individual human annotators, and Kendall's Tau-C against the average human score. A chain-of-thought prompt variant asks judges to justify their scores before deciding. The combination separates instance-level agreement, whether the judge can predict a single human rater, from system-level behavior, whether the judge's score distribution and ranking match humans, and this distinction explains why a model can look human-like on one metric and not the other.

What would settle it

Collect fresh ratings from 15 new annotators on the same 631 speeches plus a set of GPT-4.1 speeches, and compare Qwen-72B's leave-one-out Kappa against the new annotators with the new human-human Kappa; if LLM-human agreement falls clearly below the new human-human Kappa, the claim of on-par performance fails. A second check: score a plain-text rewrite of a GPT-4.1 speech with identical content but no headings or signposting; if the rewrite scores much lower, the judge's quality signal is partly stylistic.

Watch

Extended reading notes

Core claim

The paper establishes debate speech evaluation as a benchmark: an LLM judge reads a topic and a ~600-word opening speech, then scores on a 1-5 scale how good the speech is as an opening supporting the topic. Against this benchmark, model size is the main driver: models with at least 7B parameters approach human-level agreement, Qwen-72B surpasses the average pairwise human Kappa, and the strongest judges reach Kendall's Tau-C around 0.6 against average human scores over the full set. Yet all strong judges assign systematically lower scores than humans and enlarge the perceived gap between high- and low-quality speeches, especially penalizing older synthetic sources. When GPT-4.1 writes new speeches on the same topics, five strong judges rate them above human-expert speeches, suggesting that LLMs can now compose argumentative text that judges consider more persuasive.

Load-bearing premise

The benchmark treats the average of 15 crowd annotations as the true quality of each speech, even though individual human raters agree with one another only weakly (pairwise Kappa 0.24); if those ratings are noisy or biased, then 'matching humans' may mostly reflect matching that noise.

Editorial extensions

If this is right

Large LLM judges with 7B or more parameters can replace or supplement human annotators for scoring long debate speeches, saving annotation cost at similar instance-level agreement.
Ranking-aware use of LLM judges is safer than raw scores, because even well-aligned judges systematically score lower than humans.
Models under roughly 7B parameters are not reliable judges for this task, regardless of model family.
Debate-speech quality judged this way may not transfer to multi-turn or interactive debate, which the paper leaves open.
Because modern LLM speeches outrank human expert speeches, safeguards against persuasive misuse become more urgent.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the 0.24 human-human Kappa reflects genuine subjectivity, then 'on par with humans' may be the wrong bar: a useful judge should perhaps predict the average rating better than an individual rater does, and the Tau-C result around 0.6 is the more meaningful ceiling.
The sharp lower-score behavior suggests LLM judges compress the rating scale at the bottom; a calibration step, or comparing within-source rankings instead of absolute scores, could make LLM and human systems more interchangeable.
A testable extension: strip GPT-4.1 speeches of their paragraph structure and signposting while keeping the content, and see whether judge scores drop; if they do, stylistic bias is a major contributor to the 'surpassing humans' result.
The 2020-era speeches and newer training data are confounded; generating speeches with knowledge cutoffs matched to 2020 would separate improved writing skill from broader knowledge.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Two things to know about this one. First, it is the first real benchmark of LLM judges on holistic quality scoring of long debate speeches, using 631 speeches with 15 human annotations each. That is genuinely new, and the paper documents a robust behavioral finding: strong LLMs reproduce the ranking of speeches decently (Tau-C around 0.6 against the average human score) but assign systematically lower scores and sharper distinctions, which matters for anyone using LLM judges to rank systems. Second, the headline 'surpassing humans' claim in Section 5.1 is overstated. The comparison is asymmetric: human-human pairwise Kappa is 0.24, while an LLM is a single deterministic model (temperature 0.01) that can effectively average out idiosyncratic noise. A trivial predictor that rounds the average human score would likely exceed the human baseline, so the Kappa result does not demonstrate superior judging. The Tau-C analysis against the average human score is more robust and is the paper's real evidence.

What it does well: the dataset description is thorough, the metrics (weighted Kappa, Kendall's Tau-C) are appropriate, and the limitations are unusually candid—they report single runs, 4-bit quantization for >8B models, and the absence of human annotations for the GPT-4.1 speeches. They also release data and code, and they include a small run-variability check (three repeats of GPT-4.1, stable within ±0.003). The chain-of-thought and key-point analyses are a nice bonus.

Soft spots, in order of importance. (1) The 'surpassing humans' claim should be qualified or replaced with a comparison to a consensus baseline; this is fixable and does not affect the main benchmark. (2) There are no confidence intervals or significance tests, which is a concern given single runs and quantization—the authors acknowledge it but should show at least bootstrap intervals on the main figures. (3) Section 5.3's conclusion that GPT-4.1 speeches are better than human expert speeches rests entirely on LLM judges; the paper admits this, but the abstract still says models 'may perform at a human level' on generation, which is too strong. Minor: the gold standard is the average of 15 noisy ratings; that is a reasonable target for ranking, but the low pairwise agreement (κ=0.24) should be kept in mind when interpreting absolute 'matching humans' language.

Who is this for? Anyone working on LLM-as-judge, argumentation, or automatic evaluation of long-form text. It deserves a serious referee: the benchmark is reusable, the data is public, and the central ranking/score-distribution findings are well supported. I would accept this for review and ask the authors to fix the 'surpassing humans' interpretation and add uncertainty estimates.

Referee Report

3 major / 4 minor

Summary. The paper introduces Debate Speech Evaluation as a benchmark for LLM-as-a-Judge, using 631 debate speeches from Slonim et al. (2021), each with 15 crowd annotations. The authors run 23 LLMs as judges under zero-shot and chain-of-thought prompts, measuring agreement with human labels via leave-one-out pairwise weighted Kappa and Kendall's Tau-C against average human scores. They report that larger models approach human agreement, with Qwen-72B exceeding the human pairwise Kappa baseline, while also documenting systematic differences: LLM judges give lower scores and sharper distinctions between speech sources. A second contribution evaluates GPT-4.1-generated speeches, finding that five strong LLM judges rate them higher than human-expert speeches. The paper includes detailed appendices on data, prompts, parsing errors, ensembling, and Key Point Analysis of judge rationale.

Significance. If the benchmark claims hold, this is a useful and timely contribution: it extends LLM-as-a-Judge evaluation to long-form argumentative texts, a setting that combines local argument quality with holistic discourse assessment, and it provides a large, publicly available testbed derived from an existing dataset. The paper's strengths include the appropriate choice of Tau-C for ranking agreement over the full dataset, the leave-one-out protocol grounded in the annotator structure, the honest limitations section, and the interpretability analysis via Key Point Analysis. The central results, however, rest on point estimates without uncertainty quantification, and the headline 'surpassing humans' claim is vulnerable to a deterministic-versus-noisy comparison artifact. These issues are fixable, but they currently affect the strength of the main conclusions.

major comments (3)

[§5.1, Fig. 2a] The statement that Qwen-72B 'surpasses human performance' is not supported by the pairwise Kappa comparison as currently presented. With human-human κ=0.24 (Appendix C.1), each human-human pair contributes two independent annotation noise terms, whereas an LLM run at temperature 0.01 is a single deterministic scorer; a model that predicts the central tendency of the annotator pool can obtain higher agreement with a held-out human than two humans obtain with each other even if its underlying judgment is no better than the average human's. The paper should quantify this artifact by adding a consensus-predictor baseline (e.g., the average of the remaining 14 annotators, in a leave-one-out scheme) and by reporting bootstrap confidence intervals for the human and LLM Kappa estimates. Until then, the claim should be restricted to 'higher agreement with individual human labels than the average human-human pair,' rather than phrased as superior judging ability.
[§4, Figs. 2a and 2b] No uncertainty intervals or significance tests are reported for any of the headline metrics, even though each judge was run once and the pairwise Kappa estimates use overlapping annotator sets of unknown size (the 'minimal-sample' threshold is mentioned but its value is not given). Point estimates such as 'a notable jump at 7B' (Fig. 2b), 'Qwen-72B surpasses human performance' (Fig. 2a), and 'o3 still underperforms' (Section 5.1) are therefore not statistically grounded. The small-scale reproducibility check in Appendix A covers only GPT-4.1 and addresses run-to-run variability, not sampling variability over annotator pairs or speeches. Bootstrap or permutation-based tests should be added before drawing comparative conclusions among models or against the human baseline.
[§5.3, §6] The claim that 'modern LLMs can outperform humans in generating high-quality speeches' is based entirely on LLM judges, with no human annotations of the GPT-4.1 speeches; the paper itself acknowledges in the Limitations that stylistic bias and self-bias may have contributed. Because this result is listed among the contributions and restated in the Conclusion, the evidence supports only 'the five selected LLM judges prefer GPT-4.1 speeches over human-expert speeches in this setting.' The wording should be softened throughout the abstract, Section 6, and the contribution list, or human evaluation of a sample of the generated speeches should be added.

minor comments (4)

[§4] Please specify the actual 'minimal-sample' threshold value and report the number of annotator pairs retained, since this threshold is a free parameter that directly shapes Figure 2a.
[Appendix A] The main text should state explicitly that the headline models in Figures 2a and 2b, including Qwen-72B, were run with 4-bit quantization; the limitation is disclosed only in the appendix, and a reader of Section 5.1 would not know that the 'surpassing' result comes from a quantized model.
[Appendix E] The phrase 'a slight improvement of 5 points' is ambiguous; it should read 'an improvement of 0.05 in Tau-C' or similar, since the figure shows 0.40 to 0.45.
[Figure 2] Some model names in the Figure 2 legend are visually merged (e.g., 'GPT-4.1-nanoGPT-4o-mini'); please adjust the layout for readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark claim is an external comparison against human labels, and the cited prior dataset and methods are used as data sources rather than as self-justifying premises.

full rationale

Walking the claimed derivation chain—dataset scores from Slonim et al. (2021), LLM ratings collected under the same prompt, agreement metrics (leave-one-out weighted Kappa and Tau-C versus average human scores), and the behavioral comparisons—I find no step in which an output is defined in terms of the quantity it is supposed to predict. The 'surpassing human performance' claim in Section 5.1 compares a deterministic LLM's agreement with a held-out human against the human-human pairwise baseline; this is an empirical comparison, not a fitted input renamed as a prediction. Even if the comparison is statistically asymmetric (human-human pairs contain two idiosyncratic noise terms while the LLM contributes one deterministic score), that is a validity or correctness concern about the baseline, not a circular reduction: the paper's definitions do not make the LLM's Kappa equal to the human baseline by construction. The dataset and annotation protocol are inherited from Slonim et al. (2021), and some authors overlap with that work; however, the dataset is a publicly released prior resource with published quality-control checks, and the present paper's central benchmark claim is an external comparison against those human labels. Citations to Gera et al. (2024) and Bar-Haim et al. (2020) are used for interpretation and method, not as unverified premises that force the conclusions. No load-bearing self-citation chain or equivalent-by-definition result is present. The acknowledged limitation in Section 5.3 that human annotations for GPT-4.1 speeches are lacking is an honest scope restriction, not a circular step.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No formal derivation is present; this is an empirical benchmark study. The numbers that matter are measurement choices rather than fitted constants. The main hidden load is the reliability of the human labels and the comparability of the agreement metrics.

free parameters (2)

minimal-sample threshold for pairwise Kappa = 50 shared speeches
Chosen in Section 4 to reduce variance in pairwise human agreement; it changes which annotator pairs enter the Kappa estimates and could affect the 'on par with humans' comparison.
Excluded speeches and topics = 78 vetting speeches; 2 topics
Appendix C.2 removes vetting speeches designed to be low-quality and two topics lacking Arg-Human1/2 speeches; this is reasonable but alters benchmark composition and could affect source-level rankings.

assumptions (4)

domain assumption The average of 15 crowd ratings is a stable gold standard for speech quality despite low pairwise agreement (kappa = 0.24).
Used throughout Section 5; if the average hides multimodal disagreement, Tau-C against the average may not reflect true quality.
standard math Weighted Kappa and Kendall's Tau-C are valid for comparing LLM and human scores on a 1-5 ordinal scale.
Section 4; this assumes ordinal properties and appropriate handling of ties, but no distributional checks are given.
domain assumption Human annotators were blind to speech source, so source-level score differences reflect speech content rather than labeling bias.
Inherited from Slonim et al. 2021, Appendix C.1; needed for interpreting source analysis in Figures 5 and 6.
domain assumption LLM judges' scores are comparable to a single human annotator in the leave-one-out substitution.
Section 4 replaces one human with an LLM and averages; this assumes exchangeability despite the distributional differences the paper itself reports.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation." pith.science (2026). https://pith.science/paper/VCFARHJ7

@misc{pith2026250605062,
  author       = {Pith},
  title        = {Pith review of: Debatable Intelligence: Benchmarking LLM Judges via Debate Speech Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VCFARHJ7}},
  note         = {Machine review of arXiv:2506.05062}
}

read the original abstract

We introduce Debate Speech Evaluation as a novel and challenging benchmark for assessing LLM judges. Evaluating debate speeches requires a deep understanding of the speech at multiple levels, including argument strength and relevance, the coherence and organization of the speech, the appropriateness of its style and tone, and so on. This task involves a unique set of cognitive abilities that previously received limited attention in systematic LLM benchmarking. To explore such skills, we leverage a dataset of over 600 meticulously annotated debate speeches and present the first in-depth analysis of how state-of-the-art LLMs compare to human judges on this task. Our findings reveal a nuanced picture: while larger models can approximate individual human judgments in some respects, they differ substantially in their overall judgment behavior. We also investigate the ability of frontier LLMs to generate persuasive, opinionated speeches, showing that models may perform at a human level on this task.

Figures

Figures reproduced from arXiv: 2506.05062 by the authors.

**Figure 1.** Benchmarking data (1) and task (2). We assess the judgment capabilities and behavior of LLMs by analyzing how they rate debate speeches - long texts that argue for or against a controversial topic. words, the task demands a comprehensive understanding of long-form argumentation, making it a compelling testbed for LLMaJ systems. In this work, we introduce a novel benchmarking task for LLM judges: evaluating long-for… view at source ↗

**Figure 2.** We compare LLM-human agreement in two ways: Kappa scores (Figure [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** LLMs, especially stronger ones, generally align better with human annotators when using CoT prompting. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (15 more)

**Figure 4.** Figure 4: Strong LLM judges tend to give lower scores than human annotators (-1 signifies parsing issues). Results for all judges are shown in Appendix F. findings confirm that larger LLMs generally show higher agreement with humans, aligning well with expectations based on mod…

**Figure 5.** Figure 5: Judge ratings of different speech sources, compared to human ratings (rightmost panels). [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Strong judges rate speeches generated by GPT [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: Distribution of the top three pro and con key points in Llama-3.3-70B’s chain-of-thought explanations, grouped by source. Other (pro) and Other (con) denote less frequent points. The relative share of positive key points per source reflects the judge ratings discussed …

**Figure 8.** Figure 8: Speech scoring prompt. We give LLM judges [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

**Figure 9.** Figure 9: Chain of Thought (CoT) speech scoring prompt. We request the LLM judge to give a short justification for the given score. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

**Figure 10.** Figure 10: Speech generation prompt, requesting the [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]

**Figure 11.** Figure 11: Number of parse errors for different prompt [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

**Figure 12.** Figure 12: Results for two judge ensembles and their [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]

**Figure 14.** Figure 14: Key point preprocessing prompt: To improve [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]

**Figure 15.** Figure 15: Distribution of the top three pro and con key points in GPT-4.1-mini’s chain-of-thought explanations, grouped by source. Other (pro) and Other (con) denote less frequent points. The relative share of positive key points per source reflects the judge ratings discussed …

**Figure 16.** Figure 16: Distribution of the top three pro and con key points in Qwen-32B’s chain-of-thought explanations, grouped by source. Other (pro) and Other (con) denote less frequent points. The relative share of positive key points per source reflects the judge ratings discussed in S…

**Figure 17.** Figure 17: Speech-GPT2 speech on the topic “We should increase fuel tax” 18 [PITH_FULL_IMAGE:figures/full_fig_p018_17.png]

**Figure 18.** Figure 18: Human-expert debater speech on the topic “We should increase fuel tax” 19 [PITH_FULL_IMAGE:figures/full_fig_p019_18.png]

**Figure 19.** Figure 19: GPT-4.1 speech on the topic “We should increase fuel tax” 20 [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 13 canonical work pages

[1]

Expert speeches:Speeches delivered by human debate experts received consistently higher average scores than those produced by automatic systems

work page
[2]

Most arguments in this speech support the topic

Control questions:Annotators were also asked to rate a secondary statement — “Most arguments in this speech support the topic” — which served as a control for annotator relia- bility. In addition, “control speeches” of inten- tionally low quality were included to identify inattentive annotators

work page
[3]

Pipeline-set-1

Manual review:A qualitative inspection of 20 speeches (10 high- and 10 low-scoring) confirmed that high scores corresponded to coherent, on-topic content, while low scores reflected off-topic, repetitive, or non- argumentative text. This vetting process yielded 15 human annota- tions per speech. C.2 Data Partition Used We use “Pipeline-set-1”, a subset co...

work page
[4]

Ensure that each sentence conveys only one main idea

work page
[5]

Maintain the logical flow and connection between sentences

work page
[6]

Double-check that all information from the original paragraph is included in your rephrased version

work page
[7]

The argument for reform is strong

Verify that the overall meaning and intent of the original paragraph are preserved. Remember: - Keep sentences short and to the point. - Use clear and simple language. - Avoid complex sentence structures or multiple clauses. - Do not add any new information that wasn't in the original paragraph. - Do not omit any important details from the original paragr...

work page
[9]

In general, results for the CoT prompt seem to be more challenging to parse

We note that parsing errors mostly occur with smaller models and could largely vary with the prompt. In general, results for the CoT prompt seem to be more challenging to parse. for some of the smaller models. The only excep- tions are Llama-3.2-1B and GPT-4.1-nano, whose parsability is significantly improved by using CoT. E Judge Ensembling We inspect en...

work page

Show all 15 references

[10]

Read the paragraph carefully to understand its main ideas and content
[11]

Break down the information into individual concepts or points
[12]

Rewrite each concept as a separate, concise sentence
[2019]

This speech is a good opening speech for supporting the topic

fine-tuned on a dataset of arguments from Gretz et al. (2020). Speeches are formed by automatically concatenating arguments generated by the fine-tuned model. • Arg-Human1,Arg-Human2Speeches com- posed of automatically concatenated, crowd- sourced arguments on debate topics fe...

2020
[2020]

InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7073–7086, Online

Out of the echo chamber: Detecting coun- tering debate speeches. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7073–7086, Online. Association for Computational Linguistics. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Da...

2019
[2023]

Shachar Mirkin, Michal Jacovi, Tamar Lavee, Hong- Kwang Kuo, Samuel Thomas, Leslie Sager, Lili Kotlerman, Elad Venezian, and Noam Slonim

ChatGPT as a factual inconsistency eval- uator for text summarization.arXiv preprint arXiv:2303.15621. Shachar Mirkin, Michal Jacovi, Tamar Lavee, Hong- Kwang Kuo, Samuel Thomas, Leslie Sager, Lili Kotlerman, Elad Venezian, and Noam Slonim. 2018. A recorded debating dataset. I...

2018 arXiv
[2024]

This speech is a good opening speech for supporting the topic

LMUnit: Fine-grained evaluation with natural language unit tests.Preprint, arXiv:2412.13091. Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023. Verbosity bias in preference la- beling by large language models.arXiv preprint arXiv:2310.10076. Yijia Shao, Yuchen...

2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Expert speeches:Speeches delivered by human debate experts received consistently higher average scores than those produced by automatic systems

work page

[2] [2]

Most arguments in this speech support the topic

Control questions:Annotators were also asked to rate a secondary statement — “Most arguments in this speech support the topic” — which served as a control for annotator relia- bility. In addition, “control speeches” of inten- tionally low quality were included to identify inattentive annotators

work page

[3] [3]

Pipeline-set-1

Manual review:A qualitative inspection of 20 speeches (10 high- and 10 low-scoring) confirmed that high scores corresponded to coherent, on-topic content, while low scores reflected off-topic, repetitive, or non- argumentative text. This vetting process yielded 15 human annota- tions per speech. C.2 Data Partition Used We use “Pipeline-set-1”, a subset co...

work page

[4] [4]

Ensure that each sentence conveys only one main idea

work page

[5] [5]

Maintain the logical flow and connection between sentences

work page

[6] [6]

Double-check that all information from the original paragraph is included in your rephrased version

work page

[7] [7]

The argument for reform is strong

Verify that the overall meaning and intent of the original paragraph are preserved. Remember: - Keep sentences short and to the point. - Use clear and simple language. - Avoid complex sentence structures or multiple clauses. - Do not add any new information that wasn't in the original paragraph. - Do not omit any important details from the original paragr...

work page

[8] [9]

In general, results for the CoT prompt seem to be more challenging to parse

We note that parsing errors mostly occur with smaller models and could largely vary with the prompt. In general, results for the CoT prompt seem to be more challenging to parse. for some of the smaller models. The only excep- tions are Llama-3.2-1B and GPT-4.1-nano, whose parsability is significantly improved by using CoT. E Judge Ensembling We inspect en...

work page