{"id":"8126a005-0e24-4803-a6b4-4642cce3e072","arxiv_id":"2501.15268","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"This paper proposes an all-in-one lexical simplification dataset with per-sentence complex word and substitute annotations, and a multi-LLM voting method that is claimed to outperform earlier baselines.","lead":"The authors built a small English dataset that marks every hard word in a sentence and lists easier alternatives for each, instead of only one word per sentence. They also tested a multi-large-language-model voting pipeline and report that it beats earlier lexical simplification methods on this dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold-standard construction is partly endogenous to the evaluated models, so the claimed LLM advantage may be an artifact; a blind re-annotation subset would settle it.","rationale":"The most load-bearing condition for the paper's headline claim is that the new all-in-one LS dataset is a fair yardstick. If that fails, both the dataset contribution and the CoLLS comparison lose their probative value. I agree with the reader that §3.3–3.4 create a plausible contamination channel: the candidate pool and LLM pre-annotations are generated by the same model families (LSBert, GPT-3.5, Gemini) that are later evaluated in Table 4. The asymmetry is structural: a substitute is only eligible for gold status if it appears in the top-12 fused pool or is added by an annotator. LSBert's correct substitutes that ranked below the top-12 are invisible to the gold set; GPT-3.5's are pre-inserted. The consistency test (Table 3) and coverage test (95%) are useful but do not isolate this source bias. Meanwhile, 'significantly outperforms' is used without significance testing, and the dataset is unreleased. These are all addressable; hence CONDITIONAL is the right verdict, with the condition that the authors re-validate on a blind, independently constructed gold set and report significance and inter-annotator agreement. The concrete test above would directly settle whether the reported advantage is an artifact of dataset construction.","tokens_in":13734,"tokens_out":4809,"duration_ms":43931,"concrete_test":"Take a random 100-instance subset of the new LS dataset. Build a blind gold set for it by having two expert annotators independently propose substitutes for each complex word with no LLM suggestions and no candidate pool (or a pool from a non-evaluated model such as BART or Flan-T5), then adjudicate disagreements. Re-run the Table 4 evaluation of LSBert, GPT-3.5, GPT-3.5(COT), CoLLS(GPT-3.5), and CoLLS(Llama3) against this blind gold. If CoLLS's F1/F1-20 margin over LSBert persists (within, say, 0.02), the contamination concern is resolved; if the margin shrinks or reverses, the original comparison is confounded by dataset construction. Also report Krippendorff's alpha between original and blind gold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—CoLLS significantly outperforms LSBert and single-prompt LLMs—rests on a gold standard that is not independent of the systems being scored. In §3.3 the pseudo-substitute pool is built from LSBert, GPT-3.5, and Gemini, fused by Eq. (1–2); in §3.4 the same GPT-3.5 and Gemini pre-annotate each candidate as YES/NO, and these recommendations are displayed to human annotators on the annotation website. Table 4 then evaluates LSBert, GPT-3.5, and CoLLS (built on GPT-3.5) against this gold. Consequently, substitutes that GPT-3.5 generates during evaluation are disproportionately likely to already be in the pool and pre-endorsed by GPT-3.5 itself, while valid LSBert substitutes that fell outside the top-12 fusion were never shown to annotators and thus cannot count as correct unless an annotator spontaneously added them. The reported 95% coverage (325/342) mitigates but does not remove this asymmetry: the two 'new human evaluators' produced substitutes after seeing the same task and may share the pool's lexical biases, and the 5% uncovered substitutes are exactly the ones penalized. The abstract's 'significantly outperforms' is asserted without significance tests, compounding the risk. The dataset is not released, so this cannot currently be checked externally. The authors' own Limitations section acknowledges small scale but does not flag the endogenous gold construction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new evaluation paradigm for lexical simplification (LS) based on an 'all-in-one' dataset in which every complex word in a sentence is annotated with a set of suitable simpler substitutes. The dataset is constructed by generating a candidate substitute pool with LSBert, GPT-3.5, and Gemini, fusing the candidates via Eq. (1)–(2), and then having human annotators judge the candidates with the aid of LLM predictions that the same models provide. The paper evaluates LSBert, GPT-3.5, Llama-3, and a proposed multi-LLM collaboration framework CoLLS on this dataset using F1 and a difficulty-weighted F1-20. The main claimed result is that CoLLS significantly outperforms existing baselines and single-prompt LLM methods. The paper also presents an ablation study on the number of LLMs in CoLLS.","tokens_in":14045,"tokens_out":7077,"duration_ms":54336,"significance":"If the dataset and evaluation are sound, the paper would contribute a useful resource and a new end-to-end way to evaluate LS, and it demonstrates that LLM-based single-prompt simplification is competitive and that a multi-LLM collaboration can further improve results. The human-machine annotation workflow is pragmatic, and the explicit checks of precision (94%) and coverage (95%) are good practices. However, because the gold standard is partly constructed from the outputs of the very models being evaluated, and because the headline comparisons lack statistical support, the empirical claims are currently not reliable enough to support the conclusions.","major_comments":[{"comment":"The gold standard is endogenous to the evaluated systems. The pseudo-substitute pool is generated by LSBert, GPT-3.5, and Gemini and fused to a top-12 list via Eq. (1)–(2), and in §3.4 the same GPT-3.5 and Gemini pre-annotate each candidate, with their recommendations displayed to human annotators on the annotation website. Table 4 then scores LSBert, GPT-3.5, and CoLLS (built on GPT-3.5) against this gold. As a result, GPT-3.5-generated substitutes are disproportionately already in the pool and pre-endorsed by GPT-3.5 itself, while valid LSBert substitutes that fell outside the top-12 fusion are never shown to annotators and can only enter the gold if an annotator spontaneously adds them. The reported 95% coverage in §3.5 does not remove this asymmetry: the two new human evaluators produced substitutes after seeing the same task and may share the pool's lexical biases, and the 17 uncovered substitutes are exactly the ones penalized. Consequently, the claim in §5.2 that CoLLS 'significantly outperforms' other methods rests on a benchmark whose construction privileges the models being scored; this needs to be addressed, for example by a blind, independent re-annotation of a held-out subset or by re-building the pool without the evaluated models.","section":"§3.3–§3.4, Eq. (1)–(2), Table 4"},{"comment":"No statistical support is given for the central comparative claim. The results appear to come from a single run of each method; no error bars, standard deviations, or significance tests are reported. On a dataset of only 400 instances, the F1 differences between CoLLS(GPT-3.5) and the best single-prompt methods are in the 0.06–0.08 range, so the abstract's 'significantly outperforms' is unsupported. The paper should either report multiple runs with variance estimates and a paired test, or soften the claim to a descriptive comparison.","section":"§5.1–§5.2, Table 4"},{"comment":"The post hoc removal of complex words without suitable substitutes changes the evaluation target relative to the original CWI-2018 labels and may bias the comparison against LSBert. The paper states that 'if a word does not have a suitable substitute, it cannot be considered a complex word' and filters such words out of the dataset, but LSBert was trained on the original CWI labels, where these words were annotated as complex. Moreover, the 'suitability' determination was made through the same human-machine loop that produced the gold, so the filtering is not independent of the gold construction. The authors should quantify how often filtering occurred and, ideally, run an additional evaluation on the unfiltered word set with a 'no simplification needed' option.","section":"§3.2, §5.1"},{"comment":"The evaluation of LSBert is underspecified. LSBert is a pipeline with a CWI classifier and a substitute generator/ranker, but the paper does not describe how it produces simplified sentences or how its outputs are aligned with the whole-sentence gold. It is unclear whether LSBert's CWI predictions are used to select which words to simplify, how the substitute set is restricted (e.g., whether it is limited to the gold-annotated words), and how ties and multiple substitutes per word are handled. A mismatch between LSBert's identification criterion and the gold's post hoc filtering can depress recall independently of substitute quality. The authors should specify the exact decoding and scoring procedure for LSBert in Table 4.","section":"§5.1 (Baselines)"}],"minor_comments":[{"comment":"There are minor typographical errors: 'paer' should be 'paper', and 'a all-in-one' should be 'an all-in-one'.","section":"§5.4"},{"comment":"'AgentLS' is mentioned in the Baselines paragraph but is never defined, described, or evaluated in Table 4; this appears to be a leftover from a previous draft.","section":"§5.1"},{"comment":"The header '3' and '4' is cryptic; the voting threshold should be clearly defined (e.g., 'at least 3 of 4 models agree').","section":"Table 3"},{"comment":"The ablation uses 'M' for the majority threshold, while elsewhere 'm' is used; the notation should be consistent.","section":"§5.3"},{"comment":"The dataset is not released or linked; given that the paper's proposal is a new evaluation paradigm, releasing the dataset is essential for reproducibility and for the community to adopt it.","section":"General"},{"comment":"The annotation website is described, but there is no information on the number of annotators, their qualifications, or inter-annotator agreement beyond the consistency test with LLMs; a brief description would help assess the reliability of the gold labels.","section":"Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":"The endogenous-gold concern is real and should be the primary focus of the revision. In addition, the paper would benefit from a smaller, more cautious claim about 'significance' until proper tests are added. The dataset release is a prerequisite for any acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's real contribution is an evaluation design: a dataset that marks every simplifiable complex word in a sentence and gives validated substitutes for each, so whole-sentence simplification can be scored end-to-end. That is a genuine gap, and the human-machine annotation protocol—machine-generated candidate pool, LLM pre-annotations, human final judgment plus additions—is a reasonable way to build such a resource. The coverage check they report, 95% of independently produced human substitutes appear in the dataset, shows the pool is not narrow. If the dataset were released, this would be a useful public artifact for the LS community.\n\nThe soft spots are serious, though. First, the gold standard is partly endogenous: the candidate pool is produced by LSBert, GPT-3.5, and Gemini, and the same two LLMs pre-annotate suitability for the human annotators. Table 4 then scores LSBert and GPT-3.5 (and CoLLS built on GPT-3.5) against this gold. So the evaluated models are disproportionately likely to see their own outputs in the gold set, and a valid substitute that none of the three generators happened to produce is invisible unless a human spontaneously adds it. The 95% coverage mitigates this, but the 5% that humans thought of and were missing are exactly the ones that would penalize a model that generates them. Second, \"significantly outperforms\" appears in the abstract with no significance tests, no error bars, and single runs. The ablation is on one dataset with one model. Third, the dataset is only 400 sentences and has not been released, so none of this can be checked externally. The LSBert sentence-level output procedure is also not described.\n\nNone of these are fatal to the core idea. They are fixable: release the data, run significance tests, report inter-annotator agreement, and ideally re-annotate a blind subset of the gold without LLM pre-annotations to measure the bias directly. But as it stands, the empirical claim that CoLLS outperforms the baselines is not yet supported with the evidence in the paper.\n\nFor a reader, this is worth engaging with if they work on lexical simplification or text simplification evaluation. The new evaluation paradigm is the contribution; CoLLS is a fairly standard multi-agent voting assembly. I would send this to peer review with a request for major revision, not desk-reject it.","headline":"A useful new evaluation idea for lexical simplification is undermined by a gold standard contaminated by the evaluated models and a 'significant' claim with no statistics.","tokens_in":14580,"tokens_out":3935,"would_cite":false,"duration_ms":31851,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes a sentence-level all-in-one dataset for lexical simplification and shows that a multi-LLM collaboration with majority voting outperforms both pipeline baselines and single-prompt LLMs.","keywords":["lexical simplification","evaluation dataset","large language models","multi-LLM collaboration","complex word identification","substitute generation and ranking","text simplification","majority voting"],"falsifier":"Re-annotate a random sample of the dataset using only human-generated substitutes and human-only judgments, then rerun the Table 4 comparisons; if LSBert or single-prompt methods match or beat CoLLS on this human-only gold standard, the reported superiority would be an artifact of the LLM-influenced candidate pool rather than a general result.","tokens_in":13524,"feed_emoji":"✂️","tokens_out":5703,"duration_ms":47219,"temperature":0.7,"pith_summary":"The paper proposes a new evaluation paradigm for lexical simplification: instead of scoring each complex word separately, judge the whole sentence by annotating every simplifiable complex word and its acceptable substitutes in one dataset. It shows that large language models can produce simplified sentences with a single prompt, bypassing the traditional three-step pipeline of complex word identification, substitute generation, and ranking. To make such outputs measurable, the authors build a 400-sentence all-in-one dataset by combining CWI 2018 data with machine-generated substitute candidates that human annotators vet and extend. They also introduce CoLLS, a multi-LLM framework that runs identification, generation, and sentence-level validation with majority voting, and report that it outperforms both LSBert and single-prompt LLM baselines.","feed_headline":"Multi-LLM teams beat single prompts at lexical simplification","feed_subtitle":"A new all-in-one dataset scores every complex word in a sentence, giving LLMs an end-to-end test.","key_machinery":"The key machinery is the all-in-one LS dataset, in which each sentence carries a complete list of complex words and a set of simpler substitutes per word, together with the CoLLS collaboration protocol that decomposes simplification into Complex Word Identification, Substitute Generation, and Validation steps, each executed by multiple LLMs with majority voting. Substitutes are scored by a combination formula that re-ranks outputs from LSBert, GPT-3.5, and Gemini, then human annotators judge simplicity and meaning preservation for each candidate and may add their own. The difficulty-weighted F1-20 metric weights each simplification by how many of the 20 original annotators marked the word as complex.","core_discovery":"The central discovery is that a single dataset containing all complex words in each sentence and their substitutes, built by human-machine collaboration, supports direct end-to-end evaluation of sentence-level lexical simplification, and that on this dataset LLM-based methods outperform pipeline-based small-model baselines like LSBert. The paper further claims that multi-LLM collaboration—where one LLM proposes complex words, another generates substitutes, and a third validates the simplified sentence, each with majority voting—significantly outperforms single-prompt LLM methods and existing baselines on both unweighted F1 and the difficulty-weighted F1-20 metric.","pith_inferences":["One implicit consequence is that the pseudo-substitute pool's composition, drawn from LSBert, GPT-3.5, and Gemini, may favor methods using the same candidate-generating families; a cleaner test would compare against a pool built from fully human suggestions only.","If CoLLS-style validation is applied to other sentence-generation tasks, the same propose-generate-verify decomposition could become a generic template for LLM text simplification and controlled text revision.","The F1-20 difficulty-weighting idea could be reused in readability or accessibility evaluations wherever annotator agreement counts are available.","A testable extension is to expand the 400-instance dataset to more genres and languages and check whether the ranking of methods remains stable."],"forward_implications":["Simplification models can be compared end to end on whole sentences rather than word by word, making single-prompt LLM outputs directly measurable.","LLM-based simplification no longer needs to be staged as separate CWI/SG/SR models, since a single prompt already beats a small-model pipeline.","The majority-voting validation step is load-bearing: removing it lowers F1-20, so consensus-based sentence-level checking appears to improve output quality.","Difficulty weighting changes rankings by rewarding models that simplify genuinely hard words, so evaluation should report both F1 and F1-20.","The annotation method extends to other languages by starting from existing CWI annotations and adding machine-generated candidates for humans to vet."],"supporting_citations":[{"why":"Defines the three-step CWI/SG/SR pipeline and surveys lexical simplification, the framework the paper replaces with end-to-end evaluation.","marker":"Paetzold and Specia, 2017b"},{"why":"Provides the CWI 2018 dataset of sentences with all annotated complex words, the source material for the new all-in-one dataset.","marker":"Yimam et al., 2018"},{"why":"Introduces LSBert, the supervised small-model baseline that CoLLS and LLM prompts are compared against.","marker":"Qiang et al., 2021a"},{"why":"Supplies the LSBert-style paraphrase generation method used to generate pseudo-substitute candidates for the dataset.","marker":"Liu et al., 2023"},{"why":"Chain-of-thought prompting technique used in the direct LLM simplification and annotation-assistance prompts.","marker":"Fu et al., 2023"},{"why":"Self-consistency and majority-voting strategy that CoLLS adopts for complex word sets, substitutes, and validation.","marker":"Wang et al., 2023"}],"fun_headline_variants":["All-in-one dataset gives LLMs a true lexical simplification test","Multi-LLM teams beat single-prompt on new LS benchmark","New benchmark makes lexical simplification evaluable for LLMs","Human-machine annotated dataset challenges LLM simplifiers","Multi-LLM collaboration wins on new lexical simplification eval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation gold standard is fair even though it was built from substitutes suggested by the same families of LLMs that are later scored against it, and by LLM-assisted annotations that humans may not have fully corrected.","fun_headline_variants_meta":{"raw":{"variants":["All-in-one dataset gives LLMs a true lexical simplification test","Multi-LLM teams beat single-prompt on new LS benchmark","New benchmark makes lexical simplification evaluable for LLMs","Human-machine annotated dataset challenges LLM simplifiers","Multi-LLM collaboration wins on new lexical simplification eval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1391,"prompt_tokens":831,"completion_tokens":560,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":447,"completion_tokens_details":{"reasoning_tokens":479}},"tokens_in":447,"tokens_out":560,"duration_ms":5266,"temperature":1.0,"reasoning_tokens":479,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:26:39.429208+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of the dataset using only human-generated substitutes and human-only judgments, then rerun the Table 4 comparisons; if LSBert or single-prompt methods match or beat CoLLS on this human-only gold standard, the reported superiority would be an artifact of the LLM-influenced candidate pool rather than a general result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CWI 2018 dataset of sentences with all annotated complex words, the source material for the new all-in-one dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LSBert-style paraphrase generation method used to generate pseudo-substitute candidates for the dataset."}],"review_version":1}