Pith. sign in

REVIEW 4 major objections 6 minor 15 references

ELLIS Alicante at CQs-Gen 2025: Winning the critical thinking questions shared task: LLM-based question generation and selection

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Small open-weight LLMs win critical-question generation task

desk verdict A solid shared-task system paper whose reported win rests on an under-documented manual evaluation and automatic metrics that point the other way. read the letter →

arxiv 2506.14371 v1 pith:UX4WXMRZ submitted 2025-06-17 cs.CL cs.HC

Lucile Favero , Daniel Frases , Juan Antonio Pérez-Ortiz , Tanja Käser , Nuria Oliver This is my paper
classification cs.CLcs.HC
keywords criticalquestiongenerationLLM-as-a-judgeQuestioner-Judgepipelineargumentationschemesdebateinterventionssharedtaskopen-weightLLMsthinking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a two-step pipeline using two small, open-weight language models—a Questioner that generates candidate critical questions and a Judge that selects the best three—can produce critical questions about debate interventions that human judges rate as useful. The authors report that this configuration, Llama 3.1 8B as Questioner and Gemma 2 9B as Judge, ranked first in the CQs-Gen 2025 shared task with a post-annotation score of 67.6. The result matters because it suggests that locally deployable, untuned models can support critical-thinking tools without relying on large proprietary systems, and that separating generation from selection improves question quality.

What carries the argument

The pipeline is the central mechanism: a Questioner LLM generates N candidate questions, then a Judge LLM ranks and selects three. The generation prompt optionally includes argumentation scheme definitions and critical-question templates from Walton et al. (2008); the Judge's prompt instructs it to prefer redundant-but-important questions. The paper's key design choice is the 'Both' configuration, where the Questioner is prompted once with and once without schemes and the candidates are merged before selection.

What would settle it

If an independent set of human judges re-annotated the test questions and the winning submission's manual usefulness score fell substantially below 67.6, or if a different random split of the 189 interventions reversed the ranking between the small-model pipeline and the GPT-4o baseline, the paper's central claim of superiority would be undercut. More directly, rerunning the final submission with the same prompts and models but with the 'Both' scheme configuration removed should drop the manual score if the scheme mixture is truly load-bearing.

Watch

Extended reading notes

Core claim

The central claim is that combining a generative Questioner LLM with an evaluative Judge LLM, both in the 7B–14B parameter range and without fine-tuning, outperforms other submissions to the CQs-Gen 2025 shared task on critical question generation. In the final submission, Llama 3.1 8B generates four candidate questions without argumentation schemes and four with schemes in a single prompt; Gemma 2 9B then selects the three most relevant. After organizers' manual annotation, this submission scored 67.6 and ranked first. The paper also reports that selectively mixing scheme-based and scheme-free prompts ('Both') beats either alone, and that the Judge adds a small but statistically significant gain over random selection.

Load-bearing premise

The method's winning configuration was chosen assuming that questions the automatic metric cannot evaluate may still be genuinely useful, so a lower automatic score does not mean worse questions.

Editorial extensions

If this is right

  • Separating generation from selection improves question quality over direct generation.
  • Selectively adding argumentation schemes to prompts yields better questions than strict enforcement, which reduces diversity.
  • Small, open-weight models in the 7B–14B range can compete with or beat a GPT-4o-based pipeline on this task.
  • The automatic similarity-based evaluation underestimates useful questions, so manual annotation can substantially change rankings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the automatic metric's 'Not able to evaluate' labels hide many useful questions, then leaderboard rankings on this task hinge on human annotation; future shared tasks should consider designing metrics that better credit novel-but-valid critical questions.
  • The Questioner–Judge architecture could be applied to other argumentative domains (e.g., scientific claims, policy documents) with minimal adaptation, since it only needs an intervention text and optional scheme definitions.
  • The near-oracle gap in Judge performance (93.5% oracle vs 59.3% Gemma 2) suggests headroom for better selection methods, e.g., training a small classifier on human preferences or using uncertainty-based selection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper describes a two-stage pipeline for critical question generation in debate interventions: a Questioner LLM generates multiple candidate questions, and a Judge LLM selects the three most useful ones. Using small open-weight models (Llama 3.1 8B as Questioner, Gemma 2 9B as Judge) and including argumentation schemes selectively in prompts, the system was entered into the CQs-Gen 2025 shared task. The authors report that after manual annotation by the organizers, the best submission reached a score of 67.6 and ranked first. The paper also presents internal experiments on model choice, number of candidates, scheme prompting, and Judge selection.

Significance. If the reported shared-task result is fully substantiated, the paper has clear practical significance: a small, open-weight, untuned LLM pipeline outperformed a GPT-4o-based pipeline on a human-evaluated task, which supports locally deployable educational tools that preserve privacy. The paper gives credit for releasing code, running experiments on commodity hardware, and honestly listing limitations, including the mismatch between automatic and human evaluation. However, the internal evidence for the specific architectural choices is noisy, and the central claim of winning rests on a manual re-annotation whose details are not reported in the manuscript.

major comments (4)
  1. [§4.5, Table 5] The central claim that the proposed system won the shared task is not substantiated within the paper. Submission 1 has the lowest automatic test usefulness (36.3%) among the three submissions, while Submission 3 reaches 50.0%; the only evidence for the win is the sentence 'After the manual annotation of the questions by the organizers, the score of the best performing submission rose to 67.6, ranking first in the task.' The paper should report the official scoring formula, the number of annotators and their agreement, how many Not-able-to-evaluate questions were reclassified into each label, and the manual scores for all three submissions. If these details are in the shared-task overview paper, cite the specific table or section; otherwise include them directly.
  2. [§3.2, Tables 1–3] Configuration choices that motivated the final submission are based on single-run automatic scores on Dtest with no variance estimates. The differences among the top rows of Table 1 (57.6 vs. 57.1 vs. 56.8) and between 'Both' and 'Without' in Table 2 (62.4 vs. 57.7) are small, and Table 3 shows overlapping confidence intervals for 4, 6, and 8 candidate questions (59.3±3.36, 57.2±0.88, 57.3±0.76), yet §4.3 concludes that four candidates are best. Report multiple runs with confidence intervals or perform statistical tests for the key comparisons that drive the design decisions.
  3. [§4.4, Table 4] The claim that the Judge improves over random selection by 3.4 percentage points with p < 0.05 (McNemar's test) lacks essential procedural details. State the number of paired items, whether the test was applied to pooled questions or per intervention, how the three runs were aggregated, and the test statistic. Without this information, the statistical claim cannot be verified.
  4. [Limitations and §A.2.1] The paper acknowledges that Not-able-to-evaluate questions may still be useful and that the authors intentionally avoided overfitting to the automatic metric. This assumption is load-bearing because the final submission was chosen despite its lowest automatic test usefulness. A quantitative characterization of the Not-able-to-evaluate category—for example, a manual sample annotation with agreement figures, or a breakdown of the official re-annotation—would directly support this assumption and is currently missing.
minor comments (6)
  1. [§4.5, Table 5] The table header contains a typo, 'Valiadation' for 'Validation,' and the surrounding text says 'DShared train and DShared test' while the table columns are labeled 'Valiadation' and 'Test'; clarify which split each column reports.
  2. [Figures 2 and 3] The axis labels contain '/glyph1197umber' instead of 'Number,' indicating a LaTeX rendering issue that should be fixed.
  3. [§A.2.1] The text refers to 'the column No in Tables 1, 2, and 3,' but the corresponding column is labeled 'NoEval' in the tables; align the terminology.
  4. [Table 1 caption] The caption should explicitly state that rows with '—' for LLMJ correspond to direct generation by LLMQ without a Judge, rather than leaving that interpretation to the reader.
  5. [Tables 3 and §4.3] The meaning of '# quest.' is ambiguous: clarify whether the numbers denote candidate questions per prompt, total candidates including both scheme and non-scheme prompts, or something else, because §4.3 mentions 'four candidate questions per prompt (eight in total).'
  6. [References] The shared-task overview (Figueras et al., 2025) is cited in §2.2 but with no page numbers or URL in the reference list; add the complete proceedings information.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the winning result rests on an externally held-out test set and organizer manual annotation, and local model selection on a development split is standard rather than definitional.

full rationale

The paper is an empirical system description rather than a derivation. Its central claim—that the Llama 3.1 8B Questioner plus Gemma 2 9B Judge configuration ranked first in CQs-Gen 2025—is supported by the organizers' held-out test set and manual annotation, both external to the paper. The authors select hyperparameters on Dtest, a split of the shared training set, and then report the chosen configuration on Dshared_test; this is standard model selection and does not make the final ranking equivalent to the selection criterion. The automatic metric's poor showing for the winning submission and the absence of a manual-annotation breakdown raise statistical robustness concerns, but those are correctness issues, not circularity. The self-citations (e.g., Favero et al. 2024; Figueras and Agerri 2024) are contextual prior work and are not load-bearing; no equation or fitted parameter is reused as a prediction. No circular step was found.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new entities are postulated. The paper's hyperparameters (model pair, number of candidates, prompt template) are tuned on Dtest, a 79-intervention split of the shared training data, which introduces a modest selection bias. The core assumptions are about the validity of the automatic evaluation metric and the transfer of dev-set configuration choices to the official test set.

free parameters (3)
  • Number of candidate questions per prompt = 8 (4 without scheme + 4 with scheme)
    Selected based on Table 3 on Dtest; the differences between 4, 6, and 8 are within confidence intervals.
  • Questioner/Judge model pair = Llama 3.1 8B / Gemma 2 9B
    Selected based on Table 1 on Dtest; the gain over other pairs is small and unreplicated on the official test.
  • Critical question template choice = Template from Figueras and Agerri (2024)
    Chosen after slight improvement on Dtest for the Llama/Gemma pair (Appendix A.4).
assumptions (4)
  • domain assumption Cosine similarity >= 0.6 to a reference question indicates the generated question is Useful
    Used throughout as the automatic evaluation metric; introduced in Section 2.2 and acknowledged as a limitation in Section A.2.1.
  • domain assumption Argumentation schemes provide useful structural guidance for generating critical questions
    Core to the prompt design; Table 2 shows a noisy advantage for the Both configuration (62.4 vs 57.7) on Dtest.
  • domain assumption Small open-weight LLMs can generate and select useful critical questions without fine-tuning
    The whole framework relies on this; fine-tuning attempts were inconclusive (Section A.4).
  • domain assumption LLM-as-judge can select the most useful among candidate questions better than random
    Supported by Table 4 with a reported p<0.05 McNemar test, but the gain is only 3.4 percentage points.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ELLIS Alicante at CQs-Gen 2025: Winning the critical thinking questions shared task: LLM-based question generation and selection." pith.science (2026). https://pith.science/paper/UX4WXMRZ

@misc{pith2026250614371,
  author       = {Pith},
  title        = {Pith review of: ELLIS Alicante at CQs-Gen 2025: Winning the critical thinking questions shared task: LLM-based question generation and selection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UX4WXMRZ}},
  note         = {Machine review of arXiv:2506.14371}
}
read the original abstract

The widespread adoption of chat interfaces based on Large Language Models (LLMs) raises concerns about promoting superficial learning and undermining the development of critical thinking skills. Instead of relying on LLMs purely for retrieving factual information, this work explores their potential to foster deeper reasoning by generating critical questions that challenge unsupported or vague claims in debate interventions. This study is part of a shared task of the 12th Workshop on Argument Mining, co-located with ACL 2025, focused on automatic critical question generation. We propose a two-step framework involving two small-scale open source language models: a Questioner that generates multiple candidate questions and a Judge that selects the most relevant ones. Our system ranked first in the shared task competition, demonstrating the potential of the proposed LLM-based approach to encourage critical engagement with argumentative texts.

Figures

Figures reproduced from arXiv: 2506.14371 by the authors.

Figure 1
Figure 1. Overview of the proposed framework. Given a debate intervention as input, a first LLM, the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Distribution of the number of schemes per [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. shows the distribution of the labels in Dshared train. 10 20 30 40 50 Number of annotated questions per intervention 0.00 0.02 0.04 0.06 0.08 Frequency [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (1 more)
Figure 2
Figure 2. Figure 2: Distribution of the number of annotated ques [PITH_FULL_IMAGE:figures/full_fig_p006_2.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 6 canonical work pages

  1. [1]

    You are a critical judge

    • Qwen 2.5 7B. Qwen 2.5 is a multilingual transformer-based LLM with RoPE, SwiGLU, RMSNorm, and Attention QKV bias, released in September 2024 by the Qwen Team. (Yang et al., 2024). • Llama 3.1 8B, Llama 3.1 is a multilingual large language model optimized for dialogue applications. It supports eight languages and offers a context window of up to 128,000 ...

    work page 2008

  2. [4]

    Blanca Figueras, Jaione Bengoetxea, Maite Heredia, Ekaterina Sviridova, Elena Cabrio, Serena Villata, and Rodrigo Agerri

    Enhancing critical thinking in education by means of a Socratic chatbot.arXiv preprint arXiv:2409.05511. Blanca Figueras, Jaione Bengoetxea, Maite Heredia, Ekaterina Sviridova, Elena Cabrio, Serena Villata, and Rodrigo Agerri

    arXiv

  3. [6]

    Critical Questions Generation: Motivation and Challenges

    Criti- cal questions generation: Motivation and challenges. arXiv preprint arXiv:2410.14335. Michael Gerlich

    work page Pith review arXiv

  4. [7]

    Shasha Guo, Jing Zhang, Xirui Ke, Cuiping Li, and Hong Chen

    Deepseek-r1: In- centivizing reasoning capability in llms via reinforce- ment learning.arXiv preprint arXiv:2501.12948. Shasha Guo, Jing Zhang, Xirui Ke, Cuiping Li, and Hong Chen

    arXiv

  5. [8]

    Diversifying Question Generation over Knowledge Base via External Natural Questions

    Diversifying question generation over knowledge base via external natural questions. arXiv preprint arXiv:2309.14362. Zhijiang Guo, Michael Schlichtkrull, and Andreas Vla- chos

    work page Pith review arXiv

  6. [9]

    Jintao Ling and Muhammad Afzaal

    From generation to judgment: Oppor- tunities and challenges of LLM-as-a-judge.arXiv preprint arXiv:2411.16594. Jintao Ling and Muhammad Afzaal

    arXiv

  7. [10]

    An Explainable Framework for Misinformation Identification via Critical Question Answering

    An ex- plainable framework for misinformation identifica- tion via critical question answering.arXiv preprint arXiv:2503.14626. Odin Monrad Schei, Anja Møgelvang, and Kristine Ludvigsen

    work page Pith review arXiv

  8. [11]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupati- raju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al

    Gemma 3 technical report.arXiv preprint arXiv:2503.19786. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupati- raju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al

    arXiv

Show all 15 references
  1. [12]

    arXiv preprint arXiv:2408.00118

    Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. Douglas Walton, Christopher Reed, and Fabrizio Macagno. 2008.Argumentation schemes. Cam- bridge University Press. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu...

    2008 arXiv

  2. [13]

    5 tech- nical report.arXiv preprint arXiv:2412.15115

    Qwen2. 5 tech- nical report.arXiv preprint arXiv:2412.15115. A Appendix A.1 The datasets Figure 2 shows the distribution of the number of annotated questions per intervention in Dshared train, Figure 3 shows the distribution of the number of schemes per intervention in Dshared...

    arXiv

  3. [15]

    I want G, doing A achieves G, so I should do A

    to clas- sify candidate questions into three categories:Use- ful,Unhelpful, andInvalid, selecting the three ques- tions with the highest predicted probability of being Useful. However, similar to the LLM fine-tuning, the model failed to outperform a random baseline, likely due...

    2008

  4. [2018]

    arXiv preprint arXiv:1810.04805

    Bert: Pre-training of deep bidi- rectional transformers for language understanding. arXiv preprint arXiv:1810.04805. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al

    arXiv

  5. [2023]

    Jacob Devlin

    GPT-4 technical re- port.arXiv preprint arXiv:2303.08774. Jacob Devlin

    arXiv

  6. [2024]

    Linda Elder and Richard Paul

    The Llama 3 herd of models.arXiv preprint arXiv:2407.21783. Linda Elder and Richard Paul. 2020.Critical thinking: Tools for taking charge of your learning and your life. Rowman & Littlefield. Lucile Favero, Juan Antonio Pérez-Ortiz, Tanja Käser, and Nuria Oliver

    2020 arXiv

  7. [2025]

    https://arxiv.org/abs/2505.11341

    Benchmarking Criti- cal Questions Generation: A Challenging Reason- ing Task for Large Language Models — arxiv.org. https://arxiv.org/abs/2505.11341. [Accessed 19-05-2025]. Blanca Calvo Figueras and Rodrigo Agerri

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.