{"id":"f5a9ef4b-cf76-4e28-98b8-bd524208c940","arxiv_id":"2607.05259","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"SalAngaBhava is a new, publicly released Sinhala e-commerce review dataset with 1,858 reviews manually annotated at the aspect-sentiment quadruple level for ABSA research.","lead":"The paper introduces SalAngaBhava, the first publicly available Sinhala dataset for aspect-based sentiment analysis (ABSA), containing 1,858 manually annotated e-commerce reviews with aspect terms, opinion terms, and sentiment polarities. It enables fine-grained sentiment analysis in a low-resource language where previously only document-level datasets existed.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The IAA study (Kappa=0.82) covers only 5.4% of the dataset with no reported stratification by review type or category; combined with the uninvestigated ALSC F1 of 0.28, annotation quality on the full dataset is under-validated.","rationale":"The reader's CONDITIONAL verdict is appropriate, and I agree with it. However, I partially disagree with the reader's identification of the weakest assumption. The reader flags the 'sufficiently balanced' claim as the most fragile premise. While this is a genuine overstatement (58 neutral labels out of ~3,970 total), it is a secondary characterization in the abstract and conclusion—it does not undermine the dataset's core value as the first Sinhala ABSA resource.\n\nThe more load-bearing concern is the representativeness of the IAA study. The Kappa of 0.82 is the primary quality signal, but it is computed on 5.4% of the dataset with no reported stratification across two dimensions that plausibly affect annotation difficulty: transliteration status (52% of reviews) and product category (coverage ranges from 4.28% to 100%). The uninvestigated ALSC F1 of 0.28 is a red flag that could indicate the full dataset's annotation quality is lower than the IAA sample suggests.\n\nThat said, this concern does not warrant a REJECT. The dataset is publicly released, the annotation methodology is sound in design, and the issues identified are addressable through additional analysis. The paper makes a legitimate contribution to low-resource Sinhala NLP. The CONDITIONAL verdict correctly captures that the dataset is useful but not yet fully validated.\n\nThe concrete test I propose—stratified Kappa by transliteration status—is feasible with modest additional annotation effort and would directly settle whether the headline Kappa generalizes. If it does, the dataset's quality claim is well-supported. If it does not, the authors would need to qualify their quality claims and potentially re-examine annotations on the transliterated portion.","tokens_in":10919,"tokens_out":3495,"duration_ms":62839,"concrete_test":"Recompute Cohen's Kappa separately for (a) reviews that were transliterated from Romanized Sinhala and (b) reviews originally in native Sinhala script, using the same 3-annotator protocol on at least 50 reviews from each group. Also report the category composition of the existing 100-review IAA sample. If Kappa on transliterated reviews drops below 0.70, or if the IAA sample contains fewer than 30% transliterated reviews (vs. 52% in the full dataset), the headline Kappa of 0.82 does not generalize and the dataset's quality claim needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on two pillars: (1) the dataset is the first publicly available Sinhala ABSA resource with quadruple annotations, and (2) annotation quality is validated by an average pairwise Cohen's Kappa of 0.82 (Section IV.B). Pillar (1) is verifiable and holds. Pillar (2) is the soft spot.\n\nThe IAA study was conducted on only 100 reviews out of 1,858 (5.4%). The paper states these were 'randomly sampled' but does not report their composition along two critical dimensions:\n\nFirst, 52% of the cleaned dataset (5,732/10,989 reviews) underwent automated transliteration from Romanized Sinhala via the Google Transliterator API (Section IV.A). Transliteration quality is never evaluated. If the 100-review IAA sample under-represents transliterated reviews, the Kappa of 0.82 may not generalize to the portion of the dataset where annotation is hardest.\n\nSecond, annotation coverage varies from 4.28% (Electronics) to 100% (Fashion) across categories (Table I). If the IAA sample over-represents categories with higher coverage or clearer aspect vocabulary, the agreement metric is biased upward.\n\nThe ALSC baseline macro F1 of 0.28 (Table IV) is consistent with a hypothesis that annotation quality on the full dataset is lower than the IAA suggests. While this could also reflect mT5-small's limitations on Sinhala, the paper provides no error analysis, no breakdown by review type (transliterated vs. native), and no per-class F1 to distinguish model failure from data quality issues. An F1 of 0.28 on a 3-class task where the majority class constitutes ~79% of labels means the model is performing far below the majority-class baseline, which warrants investigation.\n\nThe 'sufficiently balanced' claim the reader flags is a real overstatement (58 neutral labels = 1.5%), but it is a secondary characterization, not load-bearing for the dataset's existence or utility. The IAA representativeness gap is the more fundamental concern because it undermines the quality signal that justif","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"The paper introduces SalAngaBhava, a Sinhala dataset for aspect-based sentiment analysis (ABSA) comprising 1,858 manually annotated reviews across five e-commerce product domains. The annotations follow the (t, a, o, s) quadruple scheme (target term, aspect category, opinion term, sentiment polarity). The dataset is collected from Daraz.lk and includes both native Sinhala script and Romanized Sinhala that was transliterated via the Google Transliterator API. The paper reports an inter-annotator agreement (IAA) of Cohen's Kappa = 0.82 on a 100-review sample, a LaBSE-based pairwise similarity analysis, product category classification baselines (TF-IDF and FastText), and ABSA baselines using InstructABSA (mT5-small) for aspect term extraction (macro F1 = 0.72) and aspect-level sentiment classification (macro F1 = 0.28). The dataset is publicly released on GitHub and HuggingFace.","tokens_in":11323,"tokens_out":1341,"duration_ms":213449,"significance":"The creation of the first publicly available Sinhala ABSA dataset with quadruple-level annotations is a meaningful contribution to low-resource NLP. The public release on GitHub and HuggingFace supports reproducibility. The adoption of the (t, a, o, s) scheme from Zhang et al. and the inclusion of both explicit and implicit aspect terms align the dataset with contemporary ABSA task formulations. The LaBSE similarity analysis and the product category classification baselines provide useful initial quality signals. However, the significance is tempered by the limited IAA sample size, the severely imbalanced sentiment distribution, and the lack of error analysis for the low ALSC baseline.","major_comments":[{"comment":"Section V.B and Abstract: The abstract states the dataset is 'sufficiently balanced for ABSA research,' but the sentiment distribution is 3,150 positive vs. 762 negative vs. 58 neutral (1.5% neutral). The conclusion (Section VII) repeats the 'well-balanced' claim. This is internally inconsistent with the data reported. The authors acknowledge the neutral class is 'problematic' (Section V.B) but do not reconcile this with the balance claim. The abstract and conclusion should be revised to accurately characterize the distribution rather than claiming balance.","section":null},{"comment":"Section IV.B: The IAA study covers only 100 of 1,858 reviews (5.4%). Given that 51.9% of the cleaned dataset underwent automated transliteration (Section IV.A) and annotation coverage ranges from 4.28% (Electronics) to 100% (Fashion) across categories (Table I), the representativeness of the IAA sample is critical. The paper does not report the composition of the 100-review IAA sample along either dimension. Without stratification information, the Kappa = 0.82 may not generalize to the full dataset, particularly to transliterated reviews where annotation difficulty may be higher. The authors should report the stratification of the IAA sample by review type (transliterated vs. native) and product category, or at minimum acknowledge this as a limitation.","section":null},{"comment":"Section VI, Table IV: The ALSC macro F1 of 0.28 is notably low and could reflect either model limitations (mT5-small on Sinhala) or annotation quality issues on the full dataset. The paper provides no per-class F1 breakdown, no error analysis, and no comparison stratified by review type (transliterated vs. native). Without any analysis distinguishing model failure from data quality, the reader cannot assess whether the low score reflects dataset difficulty or dataset noise. At least a brief error analysis or per-class F1 breakdown should be provided.","section":null}],"minor_comments":[{"comment":"Section V.B: The sentence 'It consists of 3150 positive labels, 762 and 58 negative and neutral labels, respectively' is awkwardly phrased. Consider rewriting as '3,150 positive, 762 negative, and 58 neutral labels.'","section":null},{"comment":"Section I: 'an Indo-Aryan languages' should be 'an Indo-Aryan language.'","section":null},{"comment":"Section IV.B: 'anotation' should be 'annotation.'","section":null},{"comment":"Section IV.A: 'achived' should be 'achieved.'","section":null},{"comment":"Section VII: 'confrim' should be 'confirm.'","section":null},{"comment":"Table I: The total annotated count sums to 1,858, but the total reviews column sums to 10,989. The coverage percentage 16.91% is correct, but the table caption could clarify that 'Coverage' refers to the percentage of total reviews in each category that were annotated.","section":null},{"comment":"Section V.D: 'IF-IDF' should be 'TF-IDF' in the text discussing FastText performance.","section":null},{"comment":"Fig. 1: The Sinhala text in the figure appears to have spacing issues between characters (e.g., 'ද ව ස' rather than 'දවස'). This may be a rendering artifact, but if it reflects the actual data, it could affect downstream tokenization. The authors should verify that the figure accurately represents the dataset text.","section":null},{"comment":"Section III.A: Reference [6] and [14] appear to be by overlapping author groups (Rizvi et al.) on closely related topics. The distinction between these works and SalAngaBhava could be stated more precisely.","section":null},{"comment":"Section IV.C: The assumption that 'the most dominant sentiment' can be reliably determined when conflicting sentiments exist for the same aspect is stated but not justified. A brief note on how 'dominant' was operationalized would strengthen this.","section":null}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution itself is valuable and the public release is commendable. The main concern is the gap between the 'sufficiently balanced' framing in the abstract/conclusion and the actual sentiment distribution (1.5% neutral). This is a presentation issue that could be fixed by revising claims, but combined with the unstratified IAA sample and absent error analysis for the low ALSC baseline, the manuscript needs revision before the dataset quality claims are fully substantiated. The stress-test concern about IAA sample representativeness is valid and should be addressed."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive review. The referee raises three major points: (1) the abstract and conclusion overstate the sentiment balance of the dataset, (2) the IAA sample composition is not reported and may not be representative, and (3) the low ALSC baseline lacks error analysis. We agree with all three points and will revise the manuscript accordingly. Specifically, we will correct the 'well-balanced' and 'sufficiently balanced' claims to accurately characterize the distribution as realistically skewed with a severely underrepresented neutral class; we will report the stratification of the IAA sample by review type and product category and add an explicit limitation if representativeness cannot be fully guaranteed; and we will add a per-class F1 breakdown and a brief error analysis for the ALSC baseline, including stratification by review type where feasible. All revisions will be incorporated in the next version of the manuscript.","responses":[{"response":"The referee is correct. The sentiment distribution (3,150 positive, 762 negative, 58 neutral) is heavily skewed, and describing it as 'sufficiently balanced' or 'well-balanced' in the abstract and conclusion is inaccurate and internally inconsistent with the data we report in Section V.B. We will revise both the abstract and the conclusion to remove the balance claim and instead characterize the distribution accurately as reflecting the natural class imbalance of e-commerce reviews, with a severely underrepresented neutral class. We will also ensure the language is consistent across all sections of the paper.","revision_made":"yes","referee_comment":"Section V.B and Abstract: The abstract states the dataset is 'sufficiently balanced for ABSA research,' but the sentiment distribution is 3,150 positive vs. 762 negative vs. 58 neutral (1.5% neutral). The conclusion (Section VII) repeats the 'well-balanced' claim. This is internally inconsistent with the data reported. The authors acknowledge the neutral class is 'problematic' (Section V.B) but do not reconcile this with the balance claim. The abstract and conclusion should be revised to accurately characterize the distribution rather than claiming balance."},{"response":"We agree that the composition of the IAA sample should be reported. We will add a breakdown of the 100-review IAA sample by review type (native Sinhala vs. transliterated vs. code-mixed) and by product category to Section IV.B. If the sample turns out not to be stratified along these dimensions, we will explicitly acknowledge this as a limitation and note that the Kappa value may not fully generalize, particularly to transliterated reviews. We will also discuss the potential for higher annotation difficulty on transliterated text and how this might affect agreement.","revision_made":"yes","referee_comment":"Section IV.B: The IAA study covers only 100 of 1,858 reviews (5.4%). Given that 51.9% of the cleaned dataset underwent automated transliteration (Section IV.A) and annotation coverage ranges from 4.28% (Electronics) to 100% (Fashion) across categories (Table I), the representativeness of the IAA sample is critical. The paper does not report the composition of the 100-review IAA sample along either dimension. Without stratification information, the Kappa = 0.82 may not generalize to the full dataset, particularly to transliterated reviews where annotation difficulty may be higher. The authors should report the stratification of the IAA sample by review type (transliterated vs. native) and product category, or at minimum acknowledge this as a limitation."},{"response":"We agree that the low ALSC macro F1 of 0.28 requires further analysis. We will add a per-class F1 breakdown (positive, negative, neutral) to Table IV or in the surrounding text. We will also conduct a brief error analysis examining a sample of misclassified instances, distinguishing between errors likely attributable to model limitations (e.g., mT5-small's limited Sinhala capacity, class imbalance effects) and those potentially reflecting annotation ambiguity. Where feasible, we will also report performance stratified by review type (transliterated vs. native). This analysis will help readers assess whether the low score reflects dataset difficulty, model limitations, or data quality issues.","revision_made":"yes","referee_comment":"Section VI, Table IV: The ALSC macro F1 of 0.28 is notably low and could reflect either model limitations (mT5-small on Sinhala) or annotation quality issues on the full dataset. The paper provides no per-class F1 breakdown, no error analysis, and no comparison stratified by review type (transliterated vs. native). Without any analysis distinguishing model failure from data quality, the reader cannot assess whether the low score reflects dataset difficulty or dataset noise. At least a brief error analysis or per-class F1 breakdown should be provided."}],"tokens_in":10772,"tokens_out":1002,"duration_ms":98608,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"This is the first publicly available Sinhala ABSA dataset with manual quadruple-level annotations (target, aspect, opinion, sentiment) across five e-commerce domains. That's a genuine gap filled. The dataset is released on GitHub and HuggingFace, the annotation scheme is well-motivated (splitting target term from aspect category to handle Sinhala inflection is a sensible design choice), and the LaBSE similarity analysis provides reasonable evidence of semantic coherence. The related work section clearly distinguishes this from prior Sinhala sentiment work that only operated at comment or document level. Credit where it's due: the resource exists, it's public, and the annotation guidelines are linguistically informed. For low-resource Sinhala NLP, this is a useful contribution. The soft spots are real but concentrated. The IAA study covers 100 of 1,858 reviews (5.4%) with no reported stratification by review type or category. Since 52% of the dataset was transliterated from Romanized Sinhala via Google Transliterator API and transliteration quality is never evaluated, the Kappa of 0.82 may not generalize to the harder portion of the data. This is the most fundamental concern — it's the quality signal that justifies the dataset's existence, and it's under-validated. The stress-test note lands here. The ALSC macro F1 of 0.28 is also uninvestigated. On a 3-class task where positive is 79% of labels, 0.28 is well below majority-class baseline. This could be mT5-small's limitations on Sinhala, or it could reflect annotation noise on the full set. The paper provides no error analysis, no per-class F1, no breakdown by transliterated vs. native reviews. That needs at least a paragraph of discussion. The 'sufficiently balanced' claim in the abstract is an overstatement — 58 neutral labels (1.5%) is not balanced by any definition. The authors acknowledge this in Section V.B but still repeat the claim in the conclusion. This is a framing problem, not a fatal one. The dataset is realistic for e-commerce; it's just not balanced. The reader's CONDITIONAL verdict is about right. The core contribution holds, but the quality validation needs strengthening before this should be considered fully validated: stratify the IAA sample (or at least report its composition), add error analysis on the baseline, and fix the 'balanced' language. This paper is for Sinhala NLP researchers and anyone working on low-resource ABSA. It deserves a serious referee who can push on the validation gaps.","headline":"First Sinhala ABSA dataset with quadruple annotations — real contribution, but quality validation is thin where it matters most.","tokens_in":11886,"tokens_out":622,"would_cite":false,"duration_ms":55794,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"First Sinhala ABSA dataset released with 1,858 annotated reviews","keywords":["aspect-based sentiment analysis","Sinhala NLP","low-resource languages","dataset annotation","e-commerce reviews","sentiment quadruple","inter-annotator agreement"],"falsifier":"If independent annotators given the same guidelines cannot reproduce the reported Cohen's Kappa of 0.82, or if the aspect category annotations prove unstable under review (the Jaccard similarity of 0.63 already suggests notable variability), the dataset's reliability for fine-grained ABSA benchmarking would be undermined.","tokens_in":11076,"feed_emoji":"🛒","tokens_out":1257,"duration_ms":113530,"temperature":0.7,"pith_summary":"The authors introduce SalAngaBhava, the first publicly available dataset for aspect-based sentiment analysis (ABSA) in Sinhala, the primary language of Sri Lanka. ABSA goes beyond sentence-level sentiment by identifying specific product features—such as battery life, display quality, or price—and the opinion expressed toward each one individually. The dataset contains 1,858 e-commerce product reviews from five domains (Electronics, Home Appliances, Grocery, Fashion, and Skincare), manually annotated by native Sinhala speakers with quadruple-level labels: target term, aspect category, opinion term, and sentiment polarity. The annotation scheme adapts the standard ABSA quadruple format to handle Sinhala's highly inflected morphology by separating the surface form of an aspect word (target term) from its canonical category (aspect), preventing lexical variants from fragmenting the label space. The authors report an average pairwise Cohen's Kappa of 0.82 across annotators, and validate the dataset's semantic coherence through LaBSE embedding similarity analysis showing that same-aspect pairs cluster tightly (median cosine ~0.83) while different-aspect pairs separate clearly (median cosine ~0.44). Baseline experiments using InstructABSA (mT5-small) yield macro F1 of 0.72 for aspect term extraction and 0.28 for sentiment classification, establishing reference points for future work. The full corpus of 10,989 cleaned reviews is released alongside the annotated subset.","feed_headline":"First Sinhala ABSA dataset released with 1,858 reviews","feed_subtitle":"Quadruple-level annotations across five e-commerce domains fill a gap for a language spoken by 17 million people, with 0.82 annotator kappa.","key_machinery":"The (t, a, o, s) sentiment quadruple: target term (t) captures the aspect word as it appears in text, aspect (a) maps it to a canonical category, opinion term (o) identifies the sentiment-bearing phrase, and sentiment (s) assigns positive/negative/neutral polarity. This decomposition is the mechanism that makes ABSA tractable in a highly inflected language where a single concept like 'battery' can appear in many morphological forms.","core_discovery":"The central contribution is the dataset itself and the demonstration that fine-grained, quadruple-level ABSA annotation is feasible and reliable for Sinhala, a morphologically rich, low-resource language. The key adaptation is splitting the traditional aspect term into a target term (surface form) and an aspect (canonical category) to handle Sinhala inflection, which prevents the same semantic feature from appearing under dozens of lexical variants and fragmenting training signal. The inter-annotator agreement of 0.82 at the quadruple level and the clear semantic separation in embedding space together argue that the annotation guidelines produce consistent, machine-learnable labels.","pith_inferences":["The baseline ALSC macro F1 of 0.28 suggests that sentiment classification on this dataset is substantially harder than aspect extraction (0.72), likely due to the severe class imbalance and the relatively small training signal for negative and especially neutral classes. Future models may need targeted data augmentation or class-balanced loss functions to improve on this baseline.","The category-level annotation coverage imbalance (4.28% for Electronics vs. 100% for Fashion) means that domain-specific aspect taxonomies are unevenly populated. Models trained on the current dataset may perform well on Fashion-specific aspects but poorly on Electronics-specific ones, a limitation that could be addressed by targeted annotation of under-covered categories.","The LaBSE embedding analysis showing same-aspect pairs clustering at 0.83 cosine similarity regardless of sentiment polarity suggests that aspect and sentiment occupy separable semantic dimensions in the embedding space, which could motivate disentangled representation approaches for Sinhala ABSA."],"forward_implications":["The dataset enables direct comparison of Sinhala ABSA performance against English benchmarks like SemEval-2014, since the annotation scheme follows the same quadruple tradition, allowing cross-lingual transfer studies.","The 10,989 unannotated reviews released alongside the 1,858 annotated ones provide a pool for semi-supervised or weakly supervised ABSA model development.","The target-term/aspect split designed for Sinhala inflection could serve as a template for other morphologically rich low-resource languages where lexical variation fragments the label space.","The heavily skewed sentiment distribution (3,150 positive vs. 58 neutral) provides a realistic testbed for evaluating model robustness under class imbalance, a known challenge in real-world ABSA deployment."],"fun_headline_variants":["SalAngaBhava: A new benchmark for Sinhala aspect-based sentiment","New Sinhala dataset enables fine-grained sentiment analysis with 0.82 kappa","Splitting aspect terms handles Sinhala inflection in new ABSA dataset","First Sinhala ABSA dataset achieves 0.82 agreement on quadruple annotations","Quadruple-level ABSA proves reliable for low-resource Sinhala text"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper claims the dataset is 'sufficiently balanced for ABSA research,' but the sentiment distribution is heavily skewed: 3,150 positive labels versus 762 negative and only 58 neutral (1.5% of all labels). The neutral class is so underrepresented that training models to detect neutral sentiment reliably would be difficult. The authors acknowledge this could be 'problematic' but maintain the balanced claim in the abstract and conclusion.","fun_headline_variants_meta":{"raw":{"variants":["SalAngaBhava: A new benchmark for Sinhala aspect-based sentiment","New Sinhala dataset enables fine-grained sentiment analysis with 0.82 kappa","Splitting aspect terms handles Sinhala inflection in new ABSA dataset","First Sinhala ABSA dataset achieves 0.82 agreement on quadruple annotations","Quadruple-level ABSA proves reliable for low-resource Sinhala text"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":1578,"prompt_tokens":584,"completion_tokens":994,"prompt_tokens_details":null},"tokens_in":584,"tokens_out":994,"duration_ms":19200,"temperature":1.0,"reasoning_tokens":858,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-07T21:01:13.235968+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If independent annotators given the same guidelines cannot reproduce the reported Cohen's Kappa of 0.82, or if the aspect category annotations prove unstable under review (the Jaccard similarity of 0.63 already suggests notable variability), the dataset's reliability for fine-grained ABSA benchmarking would be undermined.","supporting_citations":[],"review_version":1}