{"id":"c2ddf47b-9f2c-48c2-8367-3b3d2dde8c19","arxiv_id":"2412.01293","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new Sinhala text simplification dataset with 3,000 human-written simplifications is released, and intermediate-task transfer learning on mT5/mBART beats prior zero-resource baselines.","lead":"The paper introduces SiTSE, the first sentence-level text simplification dataset for Sinhala, with 1,000 complex government-document sentences each simplified by three human annotators. It also shows that sequential fine-tuning on auxiliary tasks such as translation and paraphrasing outperforms earlier zero-resource baselines for Sinhala simplification.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SiTSE test sentences are drawn from the same Fernando et al. corpus as the 56k SiTa translation auxiliary data; absent an explicit exclusion, the ITTL gains may stem from seeing test inputs, not from transfer.","rationale":"The reader's weakest assumption is the same one I would flag. After reading the full text, I find no statement anywhere—Section 5, Appendix D, or the repository description—that the 1,000 SiTSE sentences were excluded from the 56k SiTa translation pairs. Since the SiTSE dataset is literally a selection from that corpus, the burden is on the authors to demonstrate non-overlap. The potential contamination is not merely theoretical: the translation step is the highest-contributing auxiliary task, and its input side is the same domain and corpus as the test. A held-out test with overlapping sentences removed would settle whether Trans→En-simp→Si-simp's 39.95 SARI reflects transfer learning or familiarity with the test inputs. I do not see a more load-bearing concern. All other weaknesses—small test size, lack of significance tests, and the known limitations of SARI and BERTScore—are secondary if overlap exists, and the dataset contribution remains valuable regardless of the outcome of this check. The conditional verdict remains appropriate: acceptance should require the overlap analysis and, if needed, the cleaned retraining result.","tokens_in":18075,"tokens_out":3517,"duration_ms":32784,"concrete_test":"Compare every SiTSE complex sentence (original and normalized/tokenized forms) against the Sinhala source side of the 56k SiTa pairs and the full Fernando et al. corpus, reporting exact-match and near-duplicate counts. If any overlap exists, retrain Trans→En-simp→Si-simp and Trans→Si-simp with all overlapping sentences removed from every auxiliary task, then recompute SARI and BERTScore on the full 1,000-sentence test set; if the SARI advantage over TS-Mining falls below roughly 1 point or reverses, the central claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 states that the 1,000 SiTSE complex sentences were selected from the Sinhala side of the Fernando et al. [26] English-Sinhala-Tamil parallel corpus. Section 5 states that the translation auxiliary task used the SiTa corpus with 56,000 parallel sentence pairs 'extracted from official government documents of Sri Lanka [26]', i.e., the same corpus. The paper never reports removing the 1,000 test sentences from the 56k pairs before fine-tuning Trans, Trans→En-simp→Si-simp, or any other ITTL variant. Because the test inputs are likely present verbatim in the translation step, the model is fine-tuned on the exact source sentences it is later asked to simplify; this can improve fluency, adequacy, and SARI for reasons unrelated to intermediate-task transfer, and it breaks the zero-shot framing. The comparisons against TS-Mining and Pivot baselines, which never see the test inputs, are therefore potentially unfair. The central claim that ITTL outperforms previously proposed zero-resource methods hinges on this overlap being absent or immaterial.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SiTSE, a human-curated Sinhala sentence-level text simplification evaluation dataset containing 1,000 complex sentences from official government documents, each with three simplified references. It frames Sinhala text simplification as a zero-resource sequence-to-sequence task and evaluates mT5 and mBART under several regimes: unfine-tuned multilingual models, zero-shot transfer from English simplification data, paraphrase mining (TS-Mining), a pivot-based baseline, and intermediate-task transfer learning (ITTL) with auxiliary tasks including translation, paraphrasing, and English simplification, applied singly or in sequence. The best reported automatic result is the Trans→En-simp→Si-simp ITTL strategy on mT5 with SARI 39.95 (Table 3), and human evaluations on 50 sentences per model also favor the ITTL models. The paper additionally proposes two new error-analysis categories and discusses the limits of existing automatic metrics for simplification.","tokens_in":18274,"tokens_out":6912,"duration_ms":59495,"significance":"If the central empirical claim holds, the paper makes two contributions: a reusable, manually curated Sinhala simplification evaluation resource with multiple references, and evidence that sequential fine-tuning on related sequence-to-sequence tasks can provide a viable route to simplification in a language with no parallel simplification data. The authors provide public code and data, and the dataset's three-reference design is a genuine asset. The paper is also valuable for its frank discussion of the challenges that SARI and BERTScore face in low-resource simplification evaluation. However, the validity of the headline claim is currently contingent on resolving a potentially serious test-set/auxiliary-corpus overlap and on adding statistical support for the reported differences.","major_comments":[{"comment":"The SiTSE test sentences are selected from the Sinhala side of the Fernando et al. [26] English-Sinhala-Tamil parallel corpus (§3), and the Trans auxiliary task uses 56,000 sentence pairs from the same corpus (§5). The paper never states that the 1,000 test sentences were excluded from the translation fine-tuning data, nor does it report any overlap check. If any test sentence (or a close paraphrase) appears in the Trans training data, the Trans→... ITTL models have been fine-tuned on the exact input strings they are later asked to simplify, while the Si-simp, TS-Mining, and Pivot baselines have not seen those sentences. This would directly confound the abstract's central claim that ITTL outperforms previously proposed zero-resource methods. The authors should either verify non-overlap (e.g., by releasing the overlap computation), remove overlapping sentences from the auxiliary data, or re-run the evaluation on a clean test set drawn from a source not used in any auxiliary task, and then report the comparison again.","section":"§3, §5"},{"comment":"All automatic results are single-run point estimates with no error bars, significance tests, or multiple seeds. Several of the differences that motivate the conclusions are small (e.g., SARI 39.95 vs. 39.17 for Trans→En-simp→Si-simp vs. Trans→Si-simp on mT5, Table 3), and the human evaluation uses only 50 randomly selected sentences per model with no reported inter-annotator agreement or significance testing. As presented, the claim that ITTL 'outperforms' the baselines is not statistically supported. The authors should report means and standard deviations over several seeds, run paired significance tests (e.g., bootstrap) for SARI and BERTScore, and provide agreement metrics and significance tests for the human ratings.","section":"§6.1, §5.1, Table 3"},{"comment":"The paper states that the 1,000-sentence dataset was used entirely for testing, leaving nothing for model training (§7), but it does not describe any held-out validation set or a model-selection protocol. Appendix D reports hyperparameters, yet there is no indication of how training steps, checkpoints, or hyperparameter choices were made without looking at the test set. If the test set was used to guide early stopping or hyperparameter selection, this is an additional form of test-set leakage. The authors should specify a development protocol that does not use the evaluation sentences (e.g., a small held-out subset of the auxiliary data, or fixed training schedules) and state explicitly that no test-set information was used to make modeling decisions.","section":"§3, §7, Appendix D"}],"minor_comments":[{"comment":"The mT5 BERTScore for the sqPLMs baseline is reported as 15.21, which is inconsistent with the statement in §6.1 that BERTScore is 'more than 81' for all models except Si Zero-Shot-TS and Pivot; this appears to be a typographical error and should be corrected or clarified.","section":"Table 3"},{"comment":"The table layout is very hard to read: the 'Avg. sent length' column is not clearly distinguished from the SARI and BERTScore columns, and the mBART/mT5 sub-headers are ambiguous in the rendered text. Please reformat the table so that each model x metric cell is unambiguous.","section":"Table 3"},{"comment":"The term 'zero-resource' is used to describe a setup that still relies on 56,000 translation pairs, 7,000 mined paraphrases, and a machine-translated version of Newsela; the paper should define the term precisely (e.g., 'no parallel complex-simple Sinhala data') to avoid the misleading implication that no external data are used.","section":"§4.2, Abstract"},{"comment":"The text says the best average human score is obtained when all auxiliary tasks are sequenced, but the difference between the best and second-best average scores is only 0.01 (3.76 vs. 3.75 in Table 5); this wording should be tempered, especially since the authors themselves note the gain is insignificant.","section":"§6.2.1, Table 5"},{"comment":"Please clarify whether the same 50 sentences were used for human evaluation across all models and whether the evaluators were aware of which model produced each output; these details are important for interpreting the human comparison.","section":"§5.1"},{"comment":"The ACM reference block lists the submission date as 'December 2018' while the arXiv version is dated December 2024; update the date and venue information to avoid confusion.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The most serious risk is the overlap between the SiTSE test set and the 56,000-pair translation auxiliary corpus, both derived from Fernando et al. [26]. If that overlap is real, the headline result would be invalidated, and the paper would need substantial reworking. The good news is that this is verifiable and potentially fixable: the authors can run an exact and near-duplicate overlap check, remove any overlapping sentences, and re-report the comparisons. They should also be required to add significance testing and multiple seeds; the current single-run numbers are a weak basis for the strong 'outperforms' claim. The dataset itself and the human evaluation design are valuable, so the work is worth pursuing, but the revisions are not merely cosmetic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The dataset is the real contribution here: the first sentence-level Sinhala simplification dataset, 1,000 complex sentences with three expert references each, plus public code and data. The annotation process is careful, the statistics are useful, and the two new human error categories are sensible. This alone justifies referee time.\n\nWhat the paper does well is position this in the low-resource TS literature. The baselines are standard (TS-Mining, pivot, zero-shot), the ITTL variants are clearly described, and the human evaluation, though small, is a genuine attempt to look beyond SARI. I also think the authors are honest about the dataset's size and about the difficulty of choosing a single best model.\n\nThe soft spot is load-bearing. The 1,000 SiTSE complex sentences are drawn from the Sinhala side of the Fernando et al. corpus, and the 56,000-pair translation auxiliary corpus is extracted from the same source. The paper never reports excluding the test sentences from the auxiliary data. One correction to the stress-test: in the Trans step the Sinhala side is the target, not the input, so it is not quite true that the model was fine-tuned on the exact source sentences it is later asked to simplify. But it is still contamination: the model was trained to generate those exact Sinhala sentences, which can inflate fluency, adequacy, and copying in ways unrelated to transfer. The comparisons against TS-Mining and Pivot, which never saw those sentences, are therefore potentially unfair. The authors need to check overlap and rerun without it, or at least quantify how much overlap exists.\n\nThe second issue is statistical, and it is real but more minor given the size of the gaps. No error bars, no multiple seeds, and the human evaluation uses 50 sentences with no inter-annotator agreement. Some SARI differences among ITTL variants are small, so I would not hang much on ordering claims. The central claim that ITTL beats the zero-resource baselines, however, rests mostly on the contamination concern, not on noise.\n\nThe citation pattern looks fine. Self-cites point to the actual corpus and prior Sinhala NLP work, and the related work is adequate. The paper is clearly written and the authors know the task.\n\nMy recommendation: send it to peer review. The dataset is a lasting contribution, and the ITTL claim can be rescued or refuted with a clean overlap analysis and some variance estimates. I would not accept it as-is, but I would not desk-reject it either.","headline":"The SiTSE dataset is a genuine resource and the ITTL recipe is worth a look, but the paper's headline result is vulnerable because the test sentences and the translation auxiliary data come from the same corpus, and the paper never says the test set was excluded.","tokens_in":18815,"tokens_out":3673,"would_cite":true,"duration_ms":33355,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper reports that sequentially fine-tuning a multilingual model on translation, English simplification, and then Sinhala simplification enables zero-resource Sinhala text simplification that outperforms earlier zero-resource baselines.","keywords":["text simplification","Sinhala","low-resource NLP","intermediate task transfer learning","sequence-to-sequence models","mT5","mBART","SARI"],"falsifier":"Search the 56,000-sentence auxiliary translation corpus for exact or near-duplicate matches with the 1,000 SiTSE complex sentences; if any overlap exists, retrain the best ITTL configuration on the non-overlapping remainder and check whether its SARI advantage over the simpler baseline disappears.","tokens_in":123,"feed_emoji":"📄","tokens_out":8424,"duration_ms":131818,"temperature":0.7,"pith_summary":"The paper introduces SiTSE, the first sentence-level text simplification benchmark for Sinhala, built from 1,000 complex government sentences each paired with three human-written simplifications. Using it purely as a test set, the authors ask whether a multilingual sequence-to-sequence model can simplify Sinhala with no Sinhala simplification training data at all. They report that sequentially fine-tuning the model on related tasks—Sinhala translation, paraphrasing, and English simplification—before a synthetic Sinhala simplification step outperforms the previously proposed zero-resource baselines. The best configuration reaches a SARI score of 39.95 on mT5 and is favored in human evaluation, suggesting that task transfer can substitute for missing simplification corpora in a low-resource language.","feed_headline":"Transfer learning wins for low-resource Sinhala text simplification","feed_subtitle":"Sequential fine-tuning on translation and English simplification lifts SARI to 39.95 and matches human ratings.","key_machinery":"The carrying mechanism is a sequential fine-tuning recipe applied to pre-trained multilingual encoder-decoder models, mT5-base and mBART50: the model is fine-tuned in stages on auxiliary sequence-to-sequence tasks (Sinhala translation, Sinhala paraphrasing, and English-to-Sinhala simplification) and then on a synthetic Sinhala simplification corpus translated from English. The SiTSE dataset, with its three human references per complex sentence, supplies the held-out evaluation; the ordering and composition of the auxiliary task sequence is the independent variable that determines whether transfer happens.","core_discovery":"The paper's central claim is that intermediate task transfer learning (ITTL) outperforms the previously proposed zero-resource methods for text simplification in Sinhala. Concretely, the authors claim that fine-tuning a multilingual encoder-decoder model first on translation, then on English simplification, then on a machine-translated Sinhala simplification corpus yields both the highest SARI score (39.95 with mT5) and the best human ratings of adequacy and fluency among the tested systems. They further claim that translation is the most useful auxiliary task to run first, while adding paraphrasing as a third task gives no noticeable gain. The implication, on the paper's own terms, is that zero-resource simplification is achievable for Sinhala by reusing existing sequence-to-sequence data rather than waiting for a large native simplification corpus.","pith_inferences":["Beyond the paper: because the evaluation sentences and the auxiliary translation corpus are drawn from the same government-document corpus and no overlap filtering is reported, part of the observed gain could come from the test sentences appearing in fine-tuning; a disjoint test set is the clean way to settle this.","Beyond the paper: the uniformly low human simplicity scores relative to adequacy and fluency suggest the models mostly split and rearrange sentences rather than replace difficult vocabulary, so a lexical-substitution component is a natural next step.","Beyond the paper: the same sequential recipe is likely portable to other under-resourced languages with modest parallel data, but the high near-exact-copy rates imply the outputs will be conservative unless copying is directly penalized.","Beyond the paper: the dataset itself, by including three human simplifications and two added error categories, offers a testbed for reference-free and learned metrics that are presently unavailable for Sinhala."],"forward_implications":["A low-resource language that has access to a parallel corpus and a high-resource simplification dataset can build a workable simplification system without any native simplification training data.","Task ordering is not neutral: starting the auxiliary sequence with translation yields the best results, whereas concatenating all three auxiliary tasks does not improve over two.","The SiTSE benchmark, with three references per sentence and simplification dominated by sentence splitting, gives other researchers a fixed Sinhala test set and a target similar to HSplit.","Automatic scores and human error analysis diverge in places, so claims about simplification quality for Sinhala should be checked with human evaluation rather than SARI alone.","The reported insensitivity to translation language and corpus size suggests the transfer effect comes from the task itself, not from the scale of the auxiliary data."],"supporting_citations":[{"why":"Supplies both the source sentences for SiTSE and the 56,000-pair Sinhala-Tamil-English corpus used for the translation auxiliary task.","marker":"[26]"},{"why":"Defines the paraphrase-mining method used to build the auxiliary paraphrase data and provides the TS-Mining baseline it must beat.","marker":"[47]"},{"why":"The zero-shot cross-lingual simplification method used as a baseline, which the ITTL results are compared against.","marker":"[67]"},{"why":"Introduces intermediate-task training (STILTs), the framework the paper extends to sequential sequence-to-sequence fine-tuning.","marker":"[61]"},{"why":"Prior cross-lingual zero-shot ITTL work that motivates using a single intermediate task before the target task.","marker":"[60]"},{"why":"Provides mT5, the multilingual model that yields the best reported SARI result.","marker":"[88]"},{"why":"Provides mBART, the second multilingual model used in the experiments.","marker":"[41]"},{"why":"Supplies the annotation instructions (extract main idea, split, reorder, substitute) and the multi-reference ASSET design that SiTSE follows.","marker":"[3]"},{"why":"Motivates the choice of BERTScore and SARI and supports the paper's argument that current metrics are unsuited to low-resource simplification.","marker":"[5]"}],"fun_headline_variants":["Transfer learning beats zero-resource for Sinhala simplification","New Sinhala simplification dataset: 3,000 human simplified sentences","Translation fine-tuning lifts Sinhala simplification SARI to 39.95","Sinhala text simplification gets a new dataset and a transfer recipe"],"cache_read_input_tokens":20992,"weakest_assumption_plain":"The results stand on the assumption that the 1,000 test sentences were never seen during any fine-tuning step, but the paper does not report excluding them from the auxiliary corpora that come from the same government-document source.","fun_headline_variants_meta":{"raw":{"variants":["Transfer learning beats zero-resource for Sinhala simplification","New Sinhala simplification dataset: 3,000 human simplified sentences","Translation fine-tuning lifts Sinhala simplification SARI to 39.95","Sinhala text simplification gets a new dataset and a transfer recipe"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000492,"raw_usage":{"total_tokens":2394,"prompt_tokens":900,"completion_tokens":1494,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":516,"completion_tokens_details":{"reasoning_tokens":1418}},"tokens_in":516,"tokens_out":1494,"duration_ms":10606,"temperature":1.0,"reasoning_tokens":1418,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:29:48.926924+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Search the 56,000-sentence auxiliary translation corpus for exact or near-duplicate matches with the 1,000 SiTSE complex sentences; if any overlap exists, retrain the best ITTL configuration on the non-overlapping remainder and check whether its SARI advantage over the simpler baseline disappears.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the paraphrase-mining method used to build the auxiliary paraphrase data and provides the TS-Mining baseline it must beat."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The zero-shot cross-lingual simplification method used as a baseline, which the ITTL results are compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior cross-lingual zero-shot ITTL work that motivates using a single intermediate task before the target task."}],"review_version":1}