{"id":"d9f38a00-9cf2-4390-ba2e-66810d40bcaf","arxiv_id":"2501.00713","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"A simulated-annealing pipeline that generates and ranks counterspeech candidates with a language-model judge placed first for Basque and in the top three for English, Italian, and Spanish in the MCG-COLING-2025 shared task.","lead":"This paper describes a counterspeech generation system that repeatedly mutates and scores candidate replies using a language-model judge, and reports top results in the MCG-COLING-2025 shared task for Basque, Italian, English, and Spanish. A generalist reader might care because automating counterspeech at scale, especially in low-resource languages, is a candidate tool for fighting online hate speech without removing free expression.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set leakage and post-hoc run selection undermine the reported shared-task rankings","rationale":"The reader's verdict is REJECT with high correctness risk, and the weakest assumption is precisely that the test set can be used for generation and run selection without compromising the evaluation. My stress-test confirms that this is the single most load-bearing concern. Section 3.2's explicit use of the testing set for generation, combined with Section 3.5's use of the official scoring script to select the top three runs, means the reported leaderboard positions are not scientifically valid evidence for the paper's central claim. This is not a disagreement with the shared-task results themselves; the external rankings may be accurate as recorded, but the paper's methodology precludes treating them as a validated demonstration of the proposed method's superiority. The JudgeLM circularity (Sections 3.3–3.4) is a real weakness but would matter even if the evaluation protocol were clean; the test-set leakage is decisive because it undermines the only empirical support for the headline claim. The abstract's claim of fine-tuning on multilingual datasets contradicts Section 3.2's statement that no training was needed, further weakening coherence, but that contradiction is secondary. Given the reader's verdict is already REJECT, my analysis does not change it; hence UNCHANGED. If anything, the concrete evidence of test-set leakage strengthens the rejection, but no verdict adjustment is needed.","tokens_in":8000,"tokens_out":3434,"duration_ms":34192,"concrete_test":"Download the MCG-COLING-2025 test split from the official repository and inspect the file structure. Verify whether each test instance contains a gold counter-narrative column (e.g., 'CN'). If the test files include gold references, then the authors' §3.2 statement that generation used the testing set constitutes direct label leakage. Additionally, check the shared-task instructions or README to determine whether participants were permitted to run the official evaluation script on the test set before final submission; if this was prohibited, the §3.5 run-selection procedure invalidates the reported rankings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of state-of-the-art performance rests on the integrity of the shared-task evaluation. The paper's own description breaks that integrity. Section 3.2 states: 'we elected to only use the testing set (100 HS-CN pairs for each language) of the data provided to generate our answers.' The testing set, per Section 2.3, is a split where 'counter-narratives [are] held out as blind test data' yet the pair notation 'HS-CN pairs' implies the gold CN is present in the released files. Using that split for generation means the system could access the reference counterspeech, either directly or as an initialization string, which is label leakage. Section 3.5 then says: 'we ran the evaluation script given by MCG-COLING to find the top 3 runs.' This is selection on the test set: the authors generated four candidate runs, scored them with the official script on the blind test data, and submitted the three best. That procedure destroys the statistical validity of any subsequent leaderboard comparison, because the chosen runs are optimised on the test labels. Even if the gold CNs were never directly read, the run-selection step alone invalidates the reported rankings as evidence of generalisable state-of-the-art performance. The reader's weakest_assumption correctly identifies this as the load-bearing point; the JudgeLM circularity is secondary because it would only affect the quality of the optimization target, not the validity of the evaluation protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes CODEOFCONDUCT, a counterspeech generation system submitted to the MCG-COLING-2025 shared task. The method uses simulated annealing to iteratively mutate and score candidate responses with the LLM-based JudgeLM evaluator, followed by a round-robin re-ranking stage, and finally produces four run files per language from which the top three are selected for submission. The paper claims state-of-the-art performance, including first place for Basque and top-three ranks in English, Italian, and Spanish. The central evaluation claim is undermined by the procedure described in Sections 3.2 and 3.5: the testing set is used for generation and hyperparameter tuning, and the official evaluation script is run on the four candidate run files to select the three submitted runs. This constitutes selection on the held-out test labels and invalidates the reported leaderboard positions as evidence of generalizable performance.","tokens_in":8292,"tokens_out":4782,"duration_ms":48958,"significance":"If the reported rankings were valid, the simulated-annealing plus LLM-judge pipeline would be an interesting contribution to multilingual counterspeech generation, particularly for a low-resource language such as Basque. The manuscript is unusually transparent about its methodology, and it provides concrete implementation details such as algorithms, hyperparameter tables, model names, and computational cost. These strengths, however, do not compensate for the evaluation protocol: the paper explicitly states that it used the testing set to generate answers and that it ran the official evaluation script to choose the top three runs. Because the final submission is selected after observing test-set scores, the claimed state-of-the-art results cannot be interpreted as an unbiased comparison with other shared-task systems. The paper's contribution is therefore primarily a system description, not a validated empirical claim.","major_comments":[{"comment":"Section 2.3 describes the testing split as '100 HS-CN pairs (counter-narratives held out as blind test data)', while Section 3.2 states that the authors 'elected to only use the testing set (100 HS-CN pairs for each language) of the data provided to generate our answers.' If the released test files contain the gold counter-narratives, this is direct label leakage, because the generation process could access the reference responses. If the counter-narratives are not actually present, the notation 'HS-CN pairs' is seriously misleading. The paper must clarify exactly what fields are in the released test files; in the current form, the central evaluation claim is invalid under the first reading and unverifiable under the second.","section":"§2.3 and §3.2"},{"comment":"Stage 3 of the methodology says the authors 'ran the evaluation script given by MCG-COLING to find the top 3 runs' and then selected, for Basque, the first-, second-, and fourth-ranked answer files for submission. Running the official evaluation script on candidate submissions requires access to the test-set gold references, and selecting runs after observing those scores means the submitted runs are chosen by optimizing on the test labels. The resulting leaderboard positions are therefore not a valid measure of how the system would perform on unseen data. This directly undermines the abstract's claim of state-of-the-art performance across four languages and the claim that the Basque runs 'outclass all the other runs submitted by other groups.'","section":"§3.5"},{"comment":"JudgeLM is used both as the objective function in the simulated-annealing search and in the round-robin re-ranking stage, and it is also listed as one of the official evaluation metrics of the shared task. Consequently, the reported high scores on JudgeLM-based metrics are not an independent assessment of quality: the generation and selection procedures are explicitly optimized toward the same evaluator that is later used to measure success. The paper should report results on metrics that were not used in any selection step, and it should discuss this circularity explicitly, since it affects the interpretation of all JudgeLM-related numbers in the paper.","section":"§3.3, §3.4, and §2.4"},{"comment":"Hyperparameter tuning is described as being performed on 'a small subset of 4 HS instances (one from each language)' taken from the testing set, since Section 3.2 says no other data were used. Tuning hyperparameters on the test set, even on a small subset, is another form of test-set leakage and further weakens the claim that the reported results reflect generalization to the shared-task test data.","section":"§3.3"}],"minor_comments":[{"comment":"Table 2 contains formatting and typographical errors, including 'V ersion' as a column header and 'Meta-Llamaz 3' as a model name; the parameter counts are missing and several entries appear corrupted.","section":"Table 2"},{"comment":"The selection probability P(c') = T^{E(c')} / sum T^{E(c'')} is described as favoring higher-scoring candidates, but this is only true when T > 1. The paper does not specify the initial temperature T0 or the increment ΔT, so the claimed monotonic relationship between score and selection probability is not guaranteed by the stated algorithm.","section":"§3.3, Algorithm 2"},{"comment":"The text says that for Basque 'run 4 ... scored 3.5 points higher than the file for the second rank answers,' but Figure 7 is not described with enough detail to identify which scoring function produced that number; the figure should state the metric and the comparison procedure.","section":"§3.5 and Figure 7"},{"comment":"The conclusion states that the system's success stems from factors such as the simulated-annealing approach handling Basque's agglutinative morphology, but no analysis or ablation is provided to isolate these factors from the selection effects described in Sections 3.2 and 3.5.","section":"§4"}],"recommendation":"reject","confidential_remarks":"The evaluation-protocol problem is not a minor caveat: the authors explicitly disclose that they used the test set for generation and for run selection via the official evaluation script. Since the paper's main contribution is the claimed shared-task ranking, this is a load-bearing error that cannot be repaired by a revision. The paper might be salvageable as a system description if all leaderboard claims are removed and the experiment is reframed as an exploratory study, but in its current form the central empirical assertion is not valid."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this paper's central claim—state-of-the-art counterspeech across four languages—does not survive its own methods section. The authors generated answers using only the testing split (100 HS-CN pairs per language), and then ran the official evaluation script on four candidate run files and submitted the three highest-scoring ones. That is selection on the test set, and it breaks the leaderboard comparison as evidence of generalizable performance. The stress-test note is right to call this load-bearing; the JudgeLM circularity is a real but secondary issue.\n\nWhat is genuinely new: the simulated-annealing loop with Boltzmann-style JudgeLM selection, followed by round-robin re-ranking, is a modest extension of Zubiaga et al. 2024. The shared-task outcome for Basque is also new. The paper provides algorithms, complexity bounds, and hyperparameter tuning details, which is more than most shared-task system descriptions do. The limitations section is honest about computational cost and JudgeLM's multilingual weaknesses.\n\nSoft spots beyond the test-set problem: (1) The abstract claims the model was \"fine-tuned on multilingual datasets,\" but Section 3.2 says \"our model did not require training.\" That is a direct contradiction. (2) JudgeLM serves as both the optimization target in the annealing loop and the re-ranking objective, so the final outputs are selected to please the same judge that later evaluates them. (3) No code or data is released, so none of the internal analysis is reproducible. (4) The hyperparameter tuning was done on four examples, one per language, which is thin but not fatal.\n\nIs any of this salvageable? The shared-task rankings are external evidence, but the paper's own description makes them unverifiable in the manuscript. If the organizers can confirm that using the test split and selecting runs after scoring was permitted, the rankings might stand—but that confirmation would have to come from outside the paper. As written, the methodology invalidates the reported result.\n\nWho this is for: anyone studying evaluation pitfalls in shared tasks, or thinking about whether LLM judges can serve as both optimizer and metric. It is a cautionary example, not a validated system.\n\nRecommendation: this deserves a serious referee because the flaws are clear and the authors might be able to clarify or correct the protocol (e.g., confirm they never accessed gold counter-narratives, or that selection was allowed). But my own verdict is skeptical; I would not cite it as evidence of state-of-the-art performance unless the evaluation integrity is resolved.","headline":"The reported SOTA is undermined by the paper's own methodology: generation on the test set and post-hoc run selection after seeing official scores.","tokens_in":8835,"tokens_out":1528,"would_cite":false,"duration_ms":16747,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CODEOFCONDUCT, a simulated-annealing pipeline driven by an LLM judge, ranks first for Basque, second for Italian, and third for English and Spanish at MCG-COLING-2025.","keywords":["counterspeech generation","hate speech","low-resource languages","Basque","simulated annealing","LLM judge","multilingual NLP","shared task"],"falsifier":"Rerun the pipeline with the gold counter-narratives held out of the generation step and with run selection made only on the development split, then recompute the MCG-COLING-2025 leaderboard; if CODEOFCONDUCT does not keep its Basque top-three position under that protocol, the ranking claim is refuted. A second check would be native-speaker human ratings of the submitted Basque runs against lower-ranked systems, testing whether the JudgeLM scores correspond to perceived quality.","tokens_in":7774,"feed_emoji":"🛡️","tokens_out":9739,"duration_ms":86241,"temperature":0.7,"pith_summary":"The paper sets out to show that counterspeech generation for hate speech can be treated as a search problem rather than a single-pass text-generation problem, and that this search pays off most in low-resource languages. Its system, CODEOFCONDUCT, uses simulated annealing to mutate and expand candidate responses, an LLM judge to score every candidate, and a round-robin tournament to pick the best answers. On the shared task's official evaluation, the system ranked first for Basque, second for Italian, and third for English and Spanish, with all three Basque runs taking the top three positions. The authors contend this makes a no-fine-tuning, test-set-only recipe a viable path for languages where annotated counterspeech data and trained models are scarce. A reader should care because manual counterspeech does not scale, and an automated pipeline that wins in a structurally difficult language suggests the approach transfers.","feed_headline":"Annealing-plus-judge pipeline tops Basque counterspeech task","feed_subtitle":"Same no-fine-tuning recipe places second in Italian and third in English and Spanish.","key_machinery":"The mechanism that carries the argument is the simulated-annealing loop in Algorithm 1, combined with the JudgeLM scoring function. At each iteration the algorithm appends randomly sampled words to candidate counterspeech, scores the results with JudgeLM, selects survivors with probability $P(c')=T^{E(c')}/\\sum_{c''\\in S}T^{E(c'')}$, and uses LLMs to generate fresh variants from the selected candidates; the temperature $T$ and target score $S_{\\mathrm{target}}$ control how long the search explores weak candidates. A second mechanism, the round-robin tournament in Algorithm 3, re-scores the top six candidates per hate-speech instance by pairwise comparison in both orders, averaging the results to reduce position bias and produce a final ranking. The paper then assembles the top-four ranked answers into four run files and uses the official shared-task evaluation script to choose which runs to submit, which is the step that produced the Basque run-4 anomaly.","core_discovery":"On its own terms, the paper's central claim is that a simulated-annealing search over counterspeech candidates, scored by the LLM-based JudgeLM, yields top-ranked outputs in four typologically different languages without any training or additional data. Starting from the background-knowledge sentences supplied with each hate-speech instance, the algorithm appends words sampled from language-specific lists and from the hate speech itself, generates new variants with instruction-tuned LLMs, keeps high-scoring candidates through a Boltzmann-like selection rule, and then re-ranks the survivors in a pairwise round-robin tournament. This pipeline placed first in Basque, second in Italian, and third in English and Spanish in MCG-COLING-2025; for Basque, all three submitted runs outranked the other teams' runs. The paper also reports that its fourth-ranked Basque run scored higher than its second-ranked run under the official metric, which is why the final Basque submission used runs 1, 2, and 4.","pith_inferences":["If JudgeLM is a valid reward signal, the same annealing-plus-judge recipe should transfer to other text-generation tasks with cheap judge-based scoring, such as simplification or style transfer, where candidate mutation is easy to define.","The paper's use of the test set for both generation and run selection leaves the reported rankings dependent on task rules; an independent reproduction that holds out gold counter-narratives during generation and selects runs on development data would show whether the ranking reflects general quality.","The Basque run-4 anomaly hints that score differences of a few points between submitted runs may be within metric noise, so the leaderboard gaps over other teams deserve a direct human-evaluation check before being read as quality differences.","Because Basque required a separate judge (Judge-EUS) while other languages used JudgeLM, the cross-language comparability of scores is uncertain; the Basque first-place result may partly reflect judge-language fit."],"forward_implications":["Low-resource languages can reach competitive counterspeech quality with no fine-tuning and no outside training data: the pipeline needs only the test hate-speech pairs and their background knowledge.","LLM-judge scores can act as a reward signal for discrete search, not just as a final evaluation metric, so generation quality can be improved by exploring and recombining candidate texts.","Pairwise tournament re-ranking changes the identity of the best answers, meaning the final output depends on the set of candidates being compared.","The official metric and the LLM judge can disagree about which run is best, as the Basque fourth-ranked run outscored the second-ranked run, so submission selection is a non-trivial part of the system.","The pipeline is computationally expensive—about ten GPU-hours per language on an A100—which limits its use in real-time moderation."],"supporting_citations":[{"why":"Defines JudgeLM, the LLM-based ranking method used to score every counterspeech candidate in the annealing loop and tournament.","marker":"Zubiaga (2024)"},{"why":"Presents the tournament-style LLM-judge selection method for counter-narrative generation that this paper extends with simulated annealing.","marker":"Zubiaga et al. (2024)"},{"why":"Supplies the Multitarget-CONAN dataset from which the shared task's hate-speech instances and background knowledge are drawn.","marker":"Fanton et al. (2021)"},{"why":"Defines BLEU, one of the traditional metrics in the official shared-task evaluation.","marker":"Papineni et al. (2002)"},{"why":"Defines ROUGE-L, the sentence-level structure metric used in the official evaluation.","marker":"Lin (2004)"},{"why":"Defines BERTScore, the contextual-embedding similarity metric used in the official evaluation.","marker":"Zhang et al. (2019)"},{"why":"Defines the Novelty metric used to measure n-gram novelty in generated counterspeech.","marker":"Wang and Wan (2018)"}],"fun_headline_variants":["Annealing counterspeech model ranks first in Basque","Simulated annealing counterspeech wins Basque shared task","Context-aware annealing tops counterspeech in 4 languages","No-training counterspeech model sweeps Basque runs","JudgeLM-scored annealing leads counterspeech in low-resource languages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that it is legitimate to generate answers from the held-out test set and to choose which runs to submit after seeing their scores on the official evaluation script; if that use of the test set is disallowed by the shared task, or if JudgeLM scores do not reflect counterspeech quality, the reported top rankings collapse.","fun_headline_variants_meta":{"raw":{"variants":["Annealing counterspeech model ranks first in Basque","Simulated annealing counterspeech wins Basque shared task","Context-aware annealing tops counterspeech in 4 languages","No-training counterspeech model sweeps Basque runs","JudgeLM-scored annealing leads counterspeech in low-resource languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000631,"raw_usage":{"total_tokens":2915,"prompt_tokens":950,"completion_tokens":1965,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":1889}},"tokens_in":566,"tokens_out":1965,"duration_ms":14728,"temperature":1.0,"reasoning_tokens":1889,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:43:47.569445+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the pipeline with the gold counter-narratives held out of the generation step and with run selection made only on the development split, then recompute the MCG-COLING-2025 leaderboard; if CODEOFCONDUCT does not keep its Basque top-three position under that protocol, the ranking claim is refuted. A second check would be native-speaker human ratings of the submitted Basque runs against lower-ranked systems, testing whether the JudgeLM scores correspond to perceived quality.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines BLEU, one of the traditional metrics in the official shared-task evaluation."}],"review_version":1}