REVIEW 3 major objections 10 minor 27 references
SalAngaBhava: A Sinhala Market Dataset for Aspect-based Sentiment Analysis
T0 review · 3 major / 10 minor · reviewed 2026-07-07 · glm-5.2
Pith's one-line read First Sinhala ABSA dataset released with 1,858 annotated reviews
desk verdict First Sinhala ABSA dataset with quadruple annotations — real contribution, but quality validation is thin where it matters most. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The (t, a, o, s) sentiment quadruple: target term (t) captures the aspect word as it appears in text, aspect (a) maps it to a canonical category, opinion term (o) identifies the sentiment-bearing phrase, and sentiment (s) assigns positive/negative/neutral polarity. This decomposition is the mechanism that makes ABSA tractable in a highly inflected language where a single concept like 'battery' can appear in many morphological forms.
What would settle it
If independent annotators given the same guidelines cannot reproduce the reported Cohen's Kappa of 0.82, or if the aspect category annotations prove unstable under review (the Jaccard similarity of 0.63 already suggests notable variability), the dataset's reliability for fine-grained ABSA benchmarking would be undermined.
Extended reading notes
Core claim
The central contribution is the dataset itself and the demonstration that fine-grained, quadruple-level ABSA annotation is feasible and reliable for Sinhala, a morphologically rich, low-resource language. The key adaptation is splitting the traditional aspect term into a target term (surface form) and an aspect (canonical category) to handle Sinhala inflection, which prevents the same semantic feature from appearing under dozens of lexical variants and fragmenting training signal. The inter-annotator agreement of 0.82 at the quadruple level and the clear semantic separation in embedding space together argue that the annotation guidelines produce consistent, machine-learnable labels.
Load-bearing premise
The paper claims the dataset is 'sufficiently balanced for ABSA research,' but the sentiment distribution is heavily skewed: 3,150 positive labels versus 762 negative and only 58 neutral (1.5% of all labels). The neutral class is so underrepresented that training models to detect neutral sentiment reliably would be difficult. The authors acknowledge this could be 'problematic' but maintain the balanced claim in the abstract and conclusion.
Editorial extensions
If this is right
- The dataset enables direct comparison of Sinhala ABSA performance against English benchmarks like SemEval-2014, since the annotation scheme follows the same quadruple tradition, allowing cross-lingual transfer studies.
- The 10,989 unannotated reviews released alongside the 1,858 annotated ones provide a pool for semi-supervised or weakly supervised ABSA model development.
- The target-term/aspect split designed for Sinhala inflection could serve as a template for other morphologically rich low-resource languages where lexical variation fragments the label space.
- The heavily skewed sentiment distribution (3,150 positive vs. 58 neutral) provides a realistic testbed for evaluating model robustness under class imbalance, a known challenge in real-world ABSA deployment.
Reading between the lines
- The baseline ALSC macro F1 of 0.28 suggests that sentiment classification on this dataset is substantially harder than aspect extraction (0.72), likely due to the severe class imbalance and the relatively small training signal for negative and especially neutral classes. Future models may need targeted data augmentation or class-balanced loss functions to improve on this baseline.
- The category-level annotation coverage imbalance (4.28% for Electronics vs. 100% for Fashion) means that domain-specific aspect taxonomies are unevenly populated. Models trained on the current dataset may perform well on Fashion-specific aspects but poorly on Electronics-specific ones, a limitation that could be addressed by targeted annotation of under-covered categories.
- The LaBSE embedding analysis showing same-aspect pairs clustering at 0.83 cosine similarity regardless of sentiment polarity suggests that aspect and sentiment occupy separable semantic dimensions in the embedding space, which could motivate disentangled representation approaches for Sinhala ABSA.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces SalAngaBhava, a Sinhala dataset for aspect-based sentiment analysis (ABSA) comprising 1,858 manually annotated reviews across five e-commerce product domains. The annotations follow the (t, a, o, s) quadruple scheme (target term, aspect category, opinion term, sentiment polarity). The dataset is collected from Daraz.lk and includes both native Sinhala script and Romanized Sinhala that was transliterated via the Google Transliterator API. The paper reports an inter-annotator agreement (IAA) of Cohen's Kappa = 0.82 on a 100-review sample, a LaBSE-based pairwise similarity analysis, product category classification baselines (TF-IDF and FastText), and ABSA baselines using InstructABSA (mT5-small) for aspect term extraction (macro F1 = 0.72) and aspect-level sentiment classification (macro F1 = 0.28). The dataset is publicly released on GitHub and HuggingFace.
Significance. The creation of the first publicly available Sinhala ABSA dataset with quadruple-level annotations is a meaningful contribution to low-resource NLP. The public release on GitHub and HuggingFace supports reproducibility. The adoption of the (t, a, o, s) scheme from Zhang et al. and the inclusion of both explicit and implicit aspect terms align the dataset with contemporary ABSA task formulations. The LaBSE similarity analysis and the product category classification baselines provide useful initial quality signals. However, the significance is tempered by the limited IAA sample size, the severely imbalanced sentiment distribution, and the lack of error analysis for the low ALSC baseline.
major comments (3)
- Section V.B and Abstract: The abstract states the dataset is 'sufficiently balanced for ABSA research,' but the sentiment distribution is 3,150 positive vs. 762 negative vs. 58 neutral (1.5% neutral). The conclusion (Section VII) repeats the 'well-balanced' claim. This is internally inconsistent with the data reported. The authors acknowledge the neutral class is 'problematic' (Section V.B) but do not reconcile this with the balance claim. The abstract and conclusion should be revised to accurately characterize the distribution rather than claiming balance.
- Section IV.B: The IAA study covers only 100 of 1,858 reviews (5.4%). Given that 51.9% of the cleaned dataset underwent automated transliteration (Section IV.A) and annotation coverage ranges from 4.28% (Electronics) to 100% (Fashion) across categories (Table I), the representativeness of the IAA sample is critical. The paper does not report the composition of the 100-review IAA sample along either dimension. Without stratification information, the Kappa = 0.82 may not generalize to the full dataset, particularly to transliterated reviews where annotation difficulty may be higher. The authors should report the stratification of the IAA sample by review type (transliterated vs. native) and product category, or at minimum acknowledge this as a limitation.
- Section VI, Table IV: The ALSC macro F1 of 0.28 is notably low and could reflect either model limitations (mT5-small on Sinhala) or annotation quality issues on the full dataset. The paper provides no per-class F1 breakdown, no error analysis, and no comparison stratified by review type (transliterated vs. native). Without any analysis distinguishing model failure from data quality, the reader cannot assess whether the low score reflects dataset difficulty or dataset noise. At least a brief error analysis or per-class F1 breakdown should be provided.
minor comments (10)
- Section V.B: The sentence 'It consists of 3150 positive labels, 762 and 58 negative and neutral labels, respectively' is awkwardly phrased. Consider rewriting as '3,150 positive, 762 negative, and 58 neutral labels.'
- Section I: 'an Indo-Aryan languages' should be 'an Indo-Aryan language.'
- Section IV.B: 'anotation' should be 'annotation.'
- Section IV.A: 'achived' should be 'achieved.'
- Section VII: 'confrim' should be 'confirm.'
- Table I: The total annotated count sums to 1,858, but the total reviews column sums to 10,989. The coverage percentage 16.91% is correct, but the table caption could clarify that 'Coverage' refers to the percentage of total reviews in each category that were annotated.
- Section V.D: 'IF-IDF' should be 'TF-IDF' in the text discussing FastText performance.
- Fig. 1: The Sinhala text in the figure appears to have spacing issues between characters (e.g., 'ද ව ස' rather than 'දවස'). This may be a rendering artifact, but if it reflects the actual data, it could affect downstream tokenization. The authors should verify that the figure accurately represents the dataset text.
- Section III.A: Reference [6] and [14] appear to be by overlapping author groups (Rizvi et al.) on closely related topics. The distinction between these works and SalAngaBhava could be stated more precisely.
- Section IV.C: The assumption that 'the most dominant sentiment' can be reliably determined when conflicting sentiments exist for the same aspect is stated but not justified. A brief note on how 'dominant' was operationalized would strengthen this.
Simulated Author's Rebuttal
We thank the referee for a careful and constructive review. The referee raises three major points: (1) the abstract and conclusion overstate the sentiment balance of the dataset, (2) the IAA sample composition is not reported and may not be representative, and (3) the low ALSC baseline lacks error analysis. We agree with all three points and will revise the manuscript accordingly. Specifically, we will correct the 'well-balanced' and 'sufficiently balanced' claims to accurately characterize the distribution as realistically skewed with a severely underrepresented neutral class; we will report the stratification of the IAA sample by review type and product category and add an explicit limitation if representativeness cannot be fully guaranteed; and we will add a per-class F1 breakdown and a brief error analysis for the ALSC baseline, including stratification by review type where feasible. All revisions will be incorporated in the next version of the manuscript.
read point-by-point responses
-
Referee: Section V.B and Abstract: The abstract states the dataset is 'sufficiently balanced for ABSA research,' but the sentiment distribution is 3,150 positive vs. 762 negative vs. 58 neutral (1.5% neutral). The conclusion (Section VII) repeats the 'well-balanced' claim. This is internally inconsistent with the data reported. The authors acknowledge the neutral class is 'problematic' (Section V.B) but do not reconcile this with the balance claim. The abstract and conclusion should be revised to accurately characterize the distribution rather than claiming balance.
Authors: The referee is correct. The sentiment distribution (3,150 positive, 762 negative, 58 neutral) is heavily skewed, and describing it as 'sufficiently balanced' or 'well-balanced' in the abstract and conclusion is inaccurate and internally inconsistent with the data we report in Section V.B. We will revise both the abstract and the conclusion to remove the balance claim and instead characterize the distribution accurately as reflecting the natural class imbalance of e-commerce reviews, with a severely underrepresented neutral class. We will also ensure the language is consistent across all sections of the paper. revision: yes
-
Referee: Section IV.B: The IAA study covers only 100 of 1,858 reviews (5.4%). Given that 51.9% of the cleaned dataset underwent automated transliteration (Section IV.A) and annotation coverage ranges from 4.28% (Electronics) to 100% (Fashion) across categories (Table I), the representativeness of the IAA sample is critical. The paper does not report the composition of the 100-review IAA sample along either dimension. Without stratification information, the Kappa = 0.82 may not generalize to the full dataset, particularly to transliterated reviews where annotation difficulty may be higher. The authors should report the stratification of the IAA sample by review type (transliterated vs. native) and product category, or at minimum acknowledge this as a limitation.
Authors: We agree that the composition of the IAA sample should be reported. We will add a breakdown of the 100-review IAA sample by review type (native Sinhala vs. transliterated vs. code-mixed) and by product category to Section IV.B. If the sample turns out not to be stratified along these dimensions, we will explicitly acknowledge this as a limitation and note that the Kappa value may not fully generalize, particularly to transliterated reviews. We will also discuss the potential for higher annotation difficulty on transliterated text and how this might affect agreement. revision: yes
-
Referee: Section VI, Table IV: The ALSC macro F1 of 0.28 is notably low and could reflect either model limitations (mT5-small on Sinhala) or annotation quality issues on the full dataset. The paper provides no per-class F1 breakdown, no error analysis, and no comparison stratified by review type (transliterated vs. native). Without any analysis distinguishing model failure from data quality, the reader cannot assess whether the low score reflects dataset difficulty or dataset noise. At least a brief error analysis or per-class F1 breakdown should be provided.
Authors: We agree that the low ALSC macro F1 of 0.28 requires further analysis. We will add a per-class F1 breakdown (positive, negative, neutral) to Table IV or in the surrounding text. We will also conduct a brief error analysis examining a sample of misclassified instances, distinguishing between errors likely attributable to model limitations (e.g., mT5-small's limited Sinhala capacity, class imbalance effects) and those potentially reflecting annotation ambiguity. Where feasible, we will also report performance stratified by review type (transliterated vs. native). This analysis will help readers assess whether the low score reflects dataset difficulty, model limitations, or data quality issues. revision: yes
Circularity Check
No circularity: this is a dataset contribution paper with no derivation chain that reduces to its own inputs.
full rationale
SalAngaBhava is a dataset construction paper, not a derivation or prediction paper. There is no chain of equations or theoretical claims where an output reduces to an input by construction. The paper's main claims—(1) first public Sinhala ABSA dataset with quadruple annotations, (2) annotation quality validated by Cohen's Kappa=0.82, (3) dataset suitability for ABSA tasks—are each supported by independent evidence: the dataset itself is publicly released and verifiable; the IAA study uses three independent annotators on a held-out sample; the baseline results (Table IV, InstructABSA macro-F1 of 0.72/0.28) are standard downstream model evaluations, not fitted-then-predicted quantities; the pairwise similarity analysis (Fig 6) uses LaBSE embeddings as an external tool to measure semantic coherence, which is an independent measurement not defined in terms of the dataset's own labels; and the product category classification (Table III) uses TF-IDF and FastText as external models to verify category separability. The self-citations (de Silva [7], De Mel and de Silva [8], [16], Peiris and De Silva [24]) are to prior Sinhala NLP work that establishes context (e.g., the absence of Sinhala ABSA datasets, transliteration methodology), none of which are load-bearing for a mathematical derivation. No step in the paper reduces to its own inputs by definition or by fitted-then-predicted construction.
Assumptions & free parameters
free parameters (3)
- Aspect category inventory
- Annotation coverage per category
- IAA sample size (100 reviews)
assumptions (4)
- domain assumption The Google Transliterator API correctly converts Romanized Sinhala to Unicode Sinhala script with sufficient accuracy for annotation purposes.
- domain assumption Native Sinhala speakers with proficiency in language understanding can reliably annotate aspect-sentiment quadruples following the defined guidelines.
- domain assumption The e-commerce review distribution on Daraz is representative of Sinhala consumer sentiment in general.
- ad hoc to paper When conflicting sentiments are expressed for the same aspect, a single 'most dominant' sentiment can be reliably determined.
Cite this review
Pith. "Pith review of SalAngaBhava: A Sinhala Market Dataset for Aspect-based Sentiment Analysis." pith.science (2026). https://pith.science/paper/CK6TW5WY
@misc{pith2026260705259,
author = {Pith},
title = {Pith review of: SalAngaBhava: A Sinhala Market Dataset for Aspect-based Sentiment Analysis},
year = {2026},
howpublished = {\url{https://pith.science/paper/CK6TW5WY}},
note = {Machine review of arXiv:2607.05259}
}
read the original abstract
Sentiment analysis has been a primary domain under Natural Language Processing (NLP) from its inception as it plays a vital role in both real-world and research applications. In high-resource languages, this has been extended a step further, and instead of predicting sentiment at the sentence level, models have been developed to detect more fine-grained sentiments at aspect level. However, in order to conduct this fine-grained Aspect-based Sentiment Analysis (ABSA), datasets annotated with aspects and sentiments toward the said aspects is required. Such datasets are lacking for low-resources languages among which, we can count Sinhala, an Indo-Aryan languages used primarily in Sri Lanka. In this work, we introduce, SalAngaBhava, a new Sinhala Aspect-based Sentiment Analysis dataset which contains Sinhala product reviews that are manually labeled with aspect terms and the associated sentiments (positive, negative, neutral). The data was collected from domain-relevant sources such as user-generated reviews and comments, and was annotated following carefully defined guidelines to ensure consistency and quality. The dataset consists of sentences and aspect-sentiment pairs, encompassing a considerable range of aspects from several domains. The analysis confirms that the dataset is well-structured and sufficiently balanced for ABSA research. This dataset can be used as a benchmark and facilitates further studies related to Sinhala natural language processing, and low-resource sentiment analysis tasks.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Semeval-2014 task 4: Aspect based sentiment analysis,
M. Pontiki, D. Galaniset al., “Semeval-2014 task 4: Aspect based sentiment analysis,” inSemEval. Dublin, Ireland: Association for Computational Linguistics, 2014, pp. 27–35
work page 2014
-
[2]
Semeval-2015 task 12: Aspect based sentiment analysis,
M. Pontiki, D. Galanis, H. Papageorgiou, S. Manandhar, and I. Androut- sopoulos, “Semeval-2015 task 12: Aspect based sentiment analysis,” in SemEval, P. Nakov, T. Zesch, D. Cer, and D. Jurgens, Eds., 2015, pp. 486–495
work page 2015
-
[3]
Semeval-2016 task 5: Aspect based sentiment analysis,
M. Pontiki, D. Galaniset al., “Semeval-2016 task 5: Aspect based sentiment analysis,” inSemEval, 2016, pp. 19–30
work page 2016
-
[4]
Aspect-based sentiment analysis on mobile application reviews,
S. Gunathilaka and N. De Silva, “Aspect-based sentiment analysis on mobile application reviews,” in2022 22nd International Conference on Advances in ICT for Emerging Regions (ICTer). IEEE, 2022, pp. 183– 188
work page 2022
-
[5]
Uzabsa: Aspect-based sentiment analysis for the uzbek language,
S. G. Matlatipov, J. Rajabovet al., “Uzabsa: Aspect-based sentiment analysis for the uzbek language,” inLREC-COLING, 2024, pp. 394– 403
work page 2024
-
[6]
A. Rizvi, N. Thaminduet al., “Enhancing multilingual sentiment anal- ysis with explainability for sinhala, english, and code-mixed content,” arXiv preprint arXiv:2504.13545, 2025
work page Pith review arXiv 2025
-
[7]
Survey on Publicly Available Sinhala Natural Language Processing Tools and Research,
N. de Silva, “Survey on Publicly Available Sinhala Natural Language Processing Tools and Research,”arXiv preprint arXiv:1906.02358v26, 2026
-
[8]
GeeSanBhava: Sentiment Tagged Sinhala Music Video Comment Data Set,
Y . De Mel and N. de Silva, “GeeSanBhava: Sentiment Tagged Sinhala Music Video Comment Data Set,” inInternational Conference on Computational Collective Intelligence. Springer, 2025, pp. 157–171
work page 2025
Show all 27 references
-
[9]
A unified generative framework for aspect-based sentiment analysis,
H. Yan, J. Daiet al., “A unified generative framework for aspect-based sentiment analysis,” inACL-IJCNLP, 2021, pp. 2416–2429
2021
-
[10]
Aspect based sentiment analysis for service industry,
A. Maroof, S. Wasiet al., “Aspect based sentiment analysis for service industry,”IEEE Access, 2024
2024
-
[11]
Bert4gcn: Using bert intermediate layers to augment gcn for aspect-based sentiment classification,
Z. Xiao, J. Wuet al., “Bert4gcn: Using bert intermediate layers to augment gcn for aspect-based sentiment classification,”arXiv preprint arXiv:2110.00171, 2021
2021 arXiv
-
[12]
Target-aspect-sentiment joint detection for aspect-based sentiment analysis,
H. Wan, Y . Yanget al., “Target-aspect-sentiment joint detection for aspect-based sentiment analysis,” inAAAI, vol. 34, no. 05, 2020, pp. 9122–9129
2020
-
[13]
Sentiment analysis for sin- hala language using deep learning techniques,
L. Senevirathne, P. Demotteet al., “Sentiment analysis for sin- hala language using deep learning techniques,”arXiv preprint arXiv:2011.07280, 2020
2011 arXiv
-
[14]
Keyword extraction, and aspect clas- sification in sinhala, english, and code-mixed content,
F. A. Rizvi, T. Navojithet al., “Keyword extraction, and aspect clas- sification in sinhala, english, and code-mixed content,”arXiv preprint arXiv:2504.10679, 2025
2025 arXiv
-
[15]
Aspect sentiment quad prediction as para- phrase generation,
W. Zhang, Y . Denget al., “Aspect sentiment quad prediction as para- phrase generation,”arXiv preprint arXiv:2110.00796, 2021
2021 arXiv
-
[16]
Linguistic analysis of sinhala youtube comments on sinhala music videos: A dataset study,
W. De Mel and N. de Silva, “Linguistic analysis of sinhala youtube comments on sinhala music videos: A dataset study,”arXiv preprint arXiv:2501.18633, 2025
2025 arXiv
-
[17]
Aspect-based sentiment analysis advance- ments and applications in code-mixed text and gujarati language process- ing,
S. Viththalani and K. Patel, “Aspect-based sentiment analysis advance- ments and applications in code-mixed text and gujarati language process- ing,” inInternational Journal of Engineering Trends and Technology, vol. 73, 2025
2025
-
[18]
Edurabsa: An education review dataset for aspect-based sentiment analysis tasks,
Y . C. Hua, P. Dennyet al., “Edurabsa: An education review dataset for aspect-based sentiment analysis tasks,”arXiv preprint arXiv:2508.17008, 2025
2025 arXiv
-
[19]
Sentihood: Targeted aspect based sentiment analysis dataset for urban neighbourhoods,
M. Saeidi, G. Bouchardet al., “Sentihood: Targeted aspect based sentiment analysis dataset for urban neighbourhoods,” inCOLING, 2016, pp. 1546–1556
2016
-
[20]
A coefficient of agreement for nominal scales,
J. Cohen, “A coefficient of agreement for nominal scales,”Educational and psychological measurement, vol. 20, no. 1, pp. 37–46, 1960
1960
-
[21]
The distribution of the flora in the alpine zone. 1,
P. Jaccard, “The distribution of the flora in the alpine zone. 1,”New phytologist, vol. 11, no. 2, pp. 37–50, 1912
1912
-
[22]
Dimabsa: Building multilingual and multido- main datasets for dimensional aspect-based sentiment analysis,
L.-H. Lee, L.-C. Yuet al., “Dimabsa: Building multilingual and multido- main datasets for dimensional aspect-based sentiment analysis,”arXiv preprint arXiv:2601.23022, 2026
2026 arXiv
-
[23]
Language-agnostic BERT sentence embedding,
F. Feng, Y . Yanget al., “Language-agnostic BERT sentence embedding,” inACL, 2022, pp. 878–891
2022
-
[24]
Synthesis and evaluation of a domain-specific large data set for dungeons & dragons,
A. Peiris and N. De Silva, “Synthesis and evaluation of a domain-specific large data set for dungeons & dragons,” inPACLIC, 2022, pp. 415–424
2022
-
[25]
Text classification using machine learning methods,
B. Oancea, “Text classification using machine learning methods,” in KNOWCON. Palack `y University Olomouc, 2023, p. 137
2023
-
[26]
Enriching Word Vectors with Subword Information,
P. Bojanowski, E. Graveet al., “Enriching Word Vectors with Subword Information,”TACL, vol. 5, pp. 135–146, 2017
2017
-
[27]
Instructabsa: Instruction learning for aspect based sentiment analysis,
K. Scaria, H. Guptaet al., “Instructabsa: Instruction learning for aspect based sentiment analysis,” inNAACL-HLT, 2024, pp. 720–736
2024
Reviewed July 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.