DS@GT at Touch\'e: Large Language Models for Retrieval-Augmented Debate

Aaryan Potdar; Anthony Miyaguchi; Conor Johnston

REVIEW 3 major objections 4 minor 7 references

DS@GT at Touch\'e: Large Language Models for Retrieval-Augmented Debate

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Retrieval-augmented LLMs respond with relevant argumentative content but are systematically verbose, and LLM judges give repeatable, strict scores.

desk verdict A solid, transparent system-description working note for the Touché RAD task whose headline 'consistent in evaluation' claim overreaches its evidence. read the letter →

arxiv 2507.09090 v1 pith:PUGGRLPD submitted 2025-07-12 cs.IR cs.CL

Anthony Miyaguchi , Conor Johnston , Aaryan Potdar This is my paper

classification cs.IRcs.CL

keywords largelanguagemodelsretrieval-augmentedgenerationdebateLLMself-evaluationGriceanmaximsargumentativesystemsconversationalAIsharedtask

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

At issue is whether large language models, given a short list of retrieved arguments and the whole debate state, can serve as both debate speakers and debate judges. The paper reports on a system entered in a shared retrieval-augmented debate task, using six publicly available models from three providers. Its central empirical claim is that the models perform well in debates when supplied with related arguments, but they are wordier than the official baseline, and their evaluation scores are consistent across runs. On the official response sub-task, the best run averaged 0.70, above the baseline's 0.62. The authors are careful to note that judging true debate quality ultimately requires an expert experimental setting.

What carries the argument

The mechanism is a two-phase retrieval-augmented generation pipeline. In the response phase, a dense-embedding search service retrieves the top ten argument documents from a claims corpus, and a prompt pairs those documents with the full debate state before asking the model for a short utterance. In the evaluation phase, a second prompt asks the same or another model to score the latest utterance on quantity, quality, manner, and relation in a single structured JSON response, with one call producing all four measures and the result memoized to avoid repeated calls. The pipeline runs through the shared task's standard simulation interface, which supplies the request shapes for both speaking and scoring. This design carries the argument because the same retrieved context is what makes responses relevant, while the single-call structured evaluation is what makes the model's judgments repeatable.

What would settle it

Have independent human experts score the same debate utterances on the four dimensions and correlate their ratings with each model's scores; if the LLM rankings and strictness do not track the experts, then the paper's evaluation finding is an artifact of self-assessment rather than evidence about debate quality.

Watch

Extended reading notes

Core claim

The central finding is two-sided. On the generation side, giving an LLM retrieved arguments and the full debate transcript produces utterances that largely satisfy the relation and quantity standards of the task, yet the models habitually approach or exceed the stated 60-word ceiling, with Claude Opus 4 averaging 56.5 words per utterance and the tightest models still peaking near the limit. On the evaluation side, the same models, asked to score the latest utterance on four Gricean maxims in one structured call, are internally consistent but strict: every model rated the baseline's manner as acceptable while judging relation, quantity, and quality low, producing overall averages from 0.268 to 0.407. The paper interprets this as LLMs excelling at summarizing queried arguments while being verbose, and as LLM evaluation being repeatable even when its absolute judgments are pessimistic.

Load-bearing premise

The whole consistency-and-strictness finding rests on treating LLM self-scores as meaningful measures of debate quality even though they were never calibrated against human expert judgments, a limitation the paper itself acknowledges.

Editorial extensions

If this is right

Retrieval augmentation is sufficient to make off-the-shelf LLMs competitive debaters on relevance, so debate systems do not need task-specific fine-tuning to stay on topic.
A single structured evaluation call is enough to obtain repeatable scores on four debate-quality dimensions, which lowers the cost of evaluating large sets of debate turns.
Because verbosity persists across providers despite explicit length instructions, response length must be controlled by decoding constraints or post-processing rather than by prompt wording alone.
The uniformly low quantity and quality scores from self-evaluation indicate that the official baseline's weakness is visible to LLM judges, even if the absolute scale of that weakness remains unverified.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: the consistency the paper observes may live in the single shared evaluation prompt rather than in the models; re-running the evaluation with varied prompt phrasings would separate prompt stability from model stability.
Inference: comparing the same six models' scores against expert human ratings on a shared utterance set would determine whether the strictness finding says something about the baseline system's actual quality or only about LLM calibration.
Inference: the reported cost projections suggest that a cheap evaluation model could score hundreds of debates for about one dollar, making large-scale debate quality monitoring feasible for small teams.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a workmanlike shared-task system description, and for what it is, it is mostly fine. The genuinely useful parts: they deployed six LLMs from three providers in the RAG debate pipeline, report official leaderboard scores, give word-count statistics on verbosity, and publish cost projections the community can reuse. The code is on GitHub, the pipeline description is clear, and they are transparent that the Cloud LLM setup required a proxy client for TIRA. For a competition notebook, that is a solid contribution.

The verbosity finding is real as a measurement: the models cluster between 44 and 56 words per utterance versus the committee baseline's 23, with low standard deviations for the GPT models. It is worth noting that the response prompt explicitly asks for a 60-word limit, so 'verbose' here partly means 'pushes against the instructed ceiling.' That does not kill the observation, but it is a softer claim than the abstract's phrasing suggests.

The load-bearing problem is 'consistent in evaluation.' The internal evaluation has three issues the authors themselves admit in Sections 5 and 6: it needs expert or human calibration; the scores come from LLMs judging LLM-generated outputs; and the toy evaluation is only two topics, four repeats per model. On top of that, the stress-test note is correct: the evaluation prompt primes low scores by telling the model that 1.0 should be 'extremely rare' and that most responses should fall between 0 and 0.8. That shared prior plausibly explains why all models are strict on the same dimensions. And Table 3 shows standard deviations up to 0.314 on a 0-1 scale, with no ICC or inter-model agreement statistic. So the statement 'consistent in evaluation' is not supported by the current data. It is a prompt-artifact hypothesis, not a measured behavior. The authors' own conclusions section is more careful than the abstract, but the abstract is what people will quote.

Minor: the cost projections mix development and official requests, which the paper acknowledges; the internal generation sample is 18 utterances per model, which is thin but acceptable for a pilot. The leaderboard results provide external grounding for the response sub-task, so the system itself is not merely self-referential.

Who is this for? Someone building a RAG debate system who wants model-selection guidance and leaderboard context. It does not deserve harsh treatment, and it does not deserve to be read as a validated psychological claim about LLM consistency. A serious referee would accept it as a working note while requiring the consistency claim to be reworded or supported with a prompt ablation. My take: accept with minor revision, and make the authors downgrade the abstract sentence.

Referee Report

3 major / 4 minor

Summary. This CLEF 2025 working note describes the DS@GT participation in the Touché Retrieval-Augmented Debate (RAD) task. The authors deploy six LLMs from OpenAI, Anthropic, and Google as both debaters and evaluators. For response generation, the system retrieves the top-10 documents from the ClaimRev corpus and prompts an LLM to produce a counter-argument under a 60-word constraint. For evaluation, a single LLM call produces scores for quantity, quality, manner, and relation on a 0–1 scale. The internal validation reports word-count statistics over 18 utterances per model and LLM-based scores on the committee baseline debate, and the official Touché leaderboard results are reported for both sub-tasks. The paper's central claim is that LLMs perform well in debates when given related arguments, but are verbose in responses yet consistent in evaluation.

Significance. If fully supported, the paper would provide a useful, reproducible system description for retrieval-augmented debate with LLMs and an empirical data point on using LLMs as debate evaluators. The open-source code (https://github.com/dsgt-arc/touche-2025-rad), the official leaderboard results, and the transparent cost analysis are concrete contributions that make the paper a worthwhile notebook submission. However, the headline claims about evaluator consistency and overall debate performance are not yet established by the reported evidence. The paper's own discussion concedes that expert evaluation is required and that results must be grounded in human evaluation, which is exactly the missing calibration step. The verbosity observation is descriptive and supported by word counts, though it is partly an artifact of the prompt's own 60-word instruction.

major comments (3)

[Section 4.1, Figure 2, Sections 5–6] The conclusion that LLM evaluators are "consistent in evaluation" is not supported. The evaluation prompt in Figure 2 imposes a strong shared scoring prior: a score of 1.0 "should be extremely rare" and "most responses should fall between 0 and 0.8." All six evaluators receive this identical prior, so the observed agreement on low scores and on Manner being the highest metric is plausibly an anchoring artifact of the rubric rather than an independent evaluative consistency. No inter-rater reliability statistic (e.g., ICC, Krippendorff's alpha) or repeated-measures agreement measure is reported. The paper itself admits in Section 5 that the task "requires an experimental setting with experts" and in Section 6 that results "have to be grounded in human evaluation." Without a prompt-ablation study or human-calibration step, the abstract's "consistent in evaluation" claim remains self-referential and over-stated.
[Section 4.2, Table 4] The claim that "LLMs perform well in debates when given related arguments" is only partially supported by the official leaderboard. While GPT-4.1 achieves the highest average score (0.70) and Relation scores for all DS@GT runs are high (0.74–0.94), the Quality scores for all DS@GT runs are very low (0.02–0.31) compared with the baseline's 1.00, and four of the five non-GPT-4.1 runs (gemini-2.5-flash, claude-opus-4, gpt-4o, claude-sonnet-4) score at or below the baseline average (0.38–0.50 vs. 0.62). The "perform well" statement should be qualified to specific Gricean maxims (Relation, and in some cases Quantity) and specific models, or the paper should discuss these low Quality scores explicitly.
[Section 4.1, Table 3] The sample sizes are too small and the dispersion too large for the population-level "consistent" conclusion. Generation statistics are based on only 18 utterances per model, and evaluation statistics on only 6 baseline messages repeated 4 times (2 topics × 3 messages). Table 3 reports standard deviations as large as 0.314 (Relation for Gemini Flash and Gemini Pro) on a 0–1 scale, which is nearly as large as the corresponding means (0.269 and 0.223 respectively). The statement that the models "all agree" on the low metrics is not justified without an agreement statistic. A test-retest reliability measure or an inter-evaluator agreement measure (e.g., ICC, Kendall's W) should be reported before claiming consistency.

minor comments (4)

[Figure 4] In the code listing, the second type declaration is also named "Evaluation Request" but should be "Evaluation Response" to match the described interface; as written, the request and response types share a name.
[Section 4.1, Table 2] The text says Claude Opus 4 "peaking at the required maximum of 60 words" but Table 2 lists a maximum of 63 words, and Claude Sonnet 4's maximum is 61 words; the text should be corrected to acknowledge that these models exceeded the stated 60-word limit.
[Section 4.1, Table 2] The word-count analysis for generation is based on only 18 utterances per model, yet the table reports means and standard deviations to four decimal places; the precision of the presented statistics should reflect the small sample size.
[Keywords] The keyword list includes "CEUR-WS," which is not a topical keyword and appears to be a formatting artifact; it should be removed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's headline observations are direct measurements (word counts and repeated evaluation scores), and the official leaderboard provides external grounding for system performance.

full rationale

The abstract's two central observations are empirical summaries, not derivations that reduce to their inputs. Verbosity is measured directly from word counts in Table 2, which are outputs of the generation systems and are not fitted to any target conclusion. Evaluator consistency is measured in Section 4.1 by running the fixed baseline simulations four times per model and reporting standard deviations in Table 3; this is a test-retest reliability summary, and while the lack of human calibration limits the validity of the consistency claim (as the paper itself acknowledges in Sections 5 and 6), the claim is not forced by construction from a fitted parameter or by a self-referential equation. The Figure 2 prompt's instruction that most scores fall between 0 and 0.8 is a potential confound for the 'strictness' observation, but it does not make the observation tautological, since models still varied across metrics and runs. Official leaderboard results in Tables 4 and 5 provide committee-run, external benchmarks that independently support the response sub-task performance. There are no load-bearing self-citations or imported uniqueness theorems; references to Touché organizers, GenIRSim, and TIRA are infrastructural rather than argumentative. Accordingly, no circular step is exhibited.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the validity of LLM self-evaluation and on a small topic sample. No fitted parameters or invented entities are introduced beyond the design choice of retrieving 10 documents.

free parameters (1)

retrieved document count (top-k) = 10
Number of ClaimRev documents injected into each generation prompt; chosen by the authors and not optimized or fitted.

assumptions (3)

domain assumption LLM evaluator scores are treated as valid measures of debate response quality without human calibration.
Section 3.2 and Figure 2; the evaluation claims depend on this.
domain assumption Six randomly selected topics with three turns per model are representative enough to support model comparisons.
Section 3; the internal validation sample is very small.
domain assumption The committee baseline retrieval system with Elasticsearch and Stella embeddings provides adequate evidence for the RAG component.
Section 3.1; no retrieval quality evaluation is reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of DS@GT at Touch\'e: Large Language Models for Retrieval-Augmented Debate." pith.science (2026). https://pith.science/paper/PUGGRLPD

@misc{pith2026250709090,
  author       = {Pith},
  title        = {Pith review of: DS@GT at Touch\'e: Large Language Models for Retrieval-Augmented Debate},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PUGGRLPD}},
  note         = {Machine review of arXiv:2507.09090}
}

read the original abstract

Large Language Models (LLMs) demonstrate strong conversational abilities. In this Working Paper, we study them in the context of debating in two ways: their ability to perform in a structured debate along with a dataset of arguments to use and their ability to evaluate utterances throughout the debate. We deploy six leading publicly available models from three providers for the Retrieval-Augmented Debate and Evaluation. The evaluation is performed by measuring four key metrics: Quality, Quantity, Manner, and Relation. Throughout this task, we found that although LLMs perform well in debates when given related arguments, they tend to be verbose in responses yet consistent in evaluation. The accompanying source code for this paper is located at https://github.com/dsgt-arc/touche-2025-rad.

Figures

Figures reproduced from arXiv: 2507.09090 by the authors.

**Figure 1.** The prompt was used to generate a debate response in the first sub-task. Evidence from the claims database is formatted as yaml at the start of the prompt, while the context of the entire argument is serialized at the end of the prompt [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

**Figure 2.** The prompt is used to generate all measures of the requested debate. The call to the LLM leverages structured output through the chat completion API provided by OpenRouter [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** The function signature of the responding endpoint the GenIRSim system expects for a valid simulation. The messages are passed in OpenAI-compatible chat completion format, with the roles of "user" and "assistant". // Transcript of the simulated debate UserTurn = { utterance :: String, systemResponse :: SystemResponse } Simulation = { userTurns :: [UserTurn] } EvaluationRequest = { simulation :: Simulation, userTurnIn… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: The function interface for the evaluation portion of the GenIRSim API. A separate request is made for quantity, quality, manner, and relation. 4. Results 4.1. Internal Validation In our simulation, which runs over six randomly selected debate topics, there were three t…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

7 extracted references · 4 canonical work pages

[1]

Kiesel, Ç

J. Kiesel, Ç. Çöltekin, M. Gohsen, S. Heineking, M. Heinrich, M. Fröbe, T. Hagen, M. Aliannejadi, T. Erjavec, M. Hagen, M. Kopp, N. Ljubešić, K. Meden, N. Mirzakhmedova, V. Morkevičius, H. Scells, I. Zelch, M. Potthast, B. Stein, Overview of Touché 2025: Argumentation Systems, in: J. C. de Albornoz, J. Gonzalo, L. Plaza, A. García Seco de Herrera, J. Moth...

work page 2025
[2]

A. Khan, J. Hughes, D. Valentine, L. Ruis, K. Sachan, A. Radhakrishnan, E. Grefenstette, S. R. Bowman, T. Rocktäschel, E. Perez, Debating with more persuasive llms leads to more truthful answers, 2024. URL: https://arxiv.org/abs/2402.06782. arXiv:2402.06782

arXiv 2024
[3]

Hagström, D

L. Hagström, D. Saynova, T. Norlund, M. Johansson, R. Johansson, The effect of scaling, retrieval augmentation and form on the factual consistency of language models, 2023. URL: https://arxiv. org/abs/2311.01307. arXiv:2311.01307

arXiv 2023
[4]

X. Liu, P. Liu, H. He, An empirical analysis on large language models in debate evaluation, 2024. URL: https://arxiv.org/abs/2406.00050. arXiv:2406.00050

work page Pith review arXiv 2024
[5]

Kiesel, M

J. Kiesel, M. Gohsen, N. Mirzakhmedova, M. Hagen, B. Stein, Who will evaluate the evaluators? exploring the gen-ir user simulation space, in: International Conference of the Cross-Language Evaluation Forum for European Languages, Springer, 2024, pp. 166–171

work page 2024
[6]

Zhang, J

D. Zhang, J. Li, Z. Zeng, F. Wang, Jasper and stella: distillation of sota embedding models, arXiv preprint arXiv:2412.19048 (2024)

arXiv 2024
[7]

Fröbe, M

M. Fröbe, M. Wiegmann, N. Kolyada, B. Grahm, T. Elstner, F. Loebe, M. Hagen, B. Stein, M. Potthast, Continuous Integration for Reproducible Shared Tasks with TIRA.io, in: J. Kamps, L. Goeuriot, F. Crestani, M. Maistro, H. Joho, B. Davis, C. Gurrin, U. Kruschwitz, A. Caputo (Eds.), Advances in Information Retrieval. 45th European Conference on IR Research ...

work page 2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Kiesel, Ç

J. Kiesel, Ç. Çöltekin, M. Gohsen, S. Heineking, M. Heinrich, M. Fröbe, T. Hagen, M. Aliannejadi, T. Erjavec, M. Hagen, M. Kopp, N. Ljubešić, K. Meden, N. Mirzakhmedova, V. Morkevičius, H. Scells, I. Zelch, M. Potthast, B. Stein, Overview of Touché 2025: Argumentation Systems, in: J. C. de Albornoz, J. Gonzalo, L. Plaza, A. García Seco de Herrera, J. Moth...

work page 2025

[2] [2]

A. Khan, J. Hughes, D. Valentine, L. Ruis, K. Sachan, A. Radhakrishnan, E. Grefenstette, S. R. Bowman, T. Rocktäschel, E. Perez, Debating with more persuasive llms leads to more truthful answers, 2024. URL: https://arxiv.org/abs/2402.06782. arXiv:2402.06782

arXiv 2024

[3] [3]

Hagström, D

L. Hagström, D. Saynova, T. Norlund, M. Johansson, R. Johansson, The effect of scaling, retrieval augmentation and form on the factual consistency of language models, 2023. URL: https://arxiv. org/abs/2311.01307. arXiv:2311.01307

arXiv 2023

[4] [4]

X. Liu, P. Liu, H. He, An empirical analysis on large language models in debate evaluation, 2024. URL: https://arxiv.org/abs/2406.00050. arXiv:2406.00050

work page Pith review arXiv 2024

[5] [5]

Kiesel, M

J. Kiesel, M. Gohsen, N. Mirzakhmedova, M. Hagen, B. Stein, Who will evaluate the evaluators? exploring the gen-ir user simulation space, in: International Conference of the Cross-Language Evaluation Forum for European Languages, Springer, 2024, pp. 166–171

work page 2024

[6] [6]

Zhang, J

D. Zhang, J. Li, Z. Zeng, F. Wang, Jasper and stella: distillation of sota embedding models, arXiv preprint arXiv:2412.19048 (2024)

arXiv 2024

[7] [7]

Fröbe, M

M. Fröbe, M. Wiegmann, N. Kolyada, B. Grahm, T. Elstner, F. Loebe, M. Hagen, B. Stein, M. Potthast, Continuous Integration for Reproducible Shared Tasks with TIRA.io, in: J. Kamps, L. Goeuriot, F. Crestani, M. Maistro, H. Joho, B. Davis, C. Gurrin, U. Kruschwitz, A. Caputo (Eds.), Advances in Information Retrieval. 45th European Conference on IR Research ...

work page 2023