Czech Dataset for Complex Aspect-Based Sentiment Analysis Tasks

Jakub \v{S}m\'id; Ond\v{r}ej Pra\v{z}\'ak; Pavel Kr\'al; Pavel P\v{r}ib\'a\v{n}

REVIEW 3 major objections 1 minor 1 references

Czech Dataset for Complex Aspect-Based Sentiment Analysis Tasks

T0 review · 3 major / 1 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new 3.1K-review Czech benchmark puts aspect-based sentiment analysis into the SemEval-2016 linked format, enabling cross-lingual comparisons.

desk verdict Genuinely useful Czech ABSA resource; the linked-annotation layer is the new contribution, but the 90% agreement figure needs kappa and per-layer evidence before cross-lingual claims lean on it. read the letter →

arxiv 2508.08125 v1 pith:SELDWU32 submitted 2025-08-11 cs.CL

Jakub Šmíd , Pavel Přibáň , Ondřej Pražák , Pavel Král This is my paper · ORCID

classification cs.CL

keywords aspect-basedsentimentanalysisCzechdatasetSemEval-2016target-aspect-categorydetectionrestaurantreviewscross-lingualevaluationannotationcorpusunsupervisedlearning

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to establish that a Czech dataset can support the complex, linked version of aspect-based sentiment analysis (ABSA), not just the basic subtasks. To do that, it converts an older Czech restaurant-review corpus into the SemEval-2016 format, where each aspect term, category, and polarity is explicitly connected, and adds 3.1K fresh annotations with about 90% inter-annotator agreement. If the dataset is sound, it gives Czech a benchmark that can be plugged into cross-lingual comparisons and used to train models on tasks such as target-aspect-category detection. The 24M unlabeled Czech reviews are a secondary resource for unsupervised learning.

What carries the argument

The load-bearing object is the SemEval-2016-style linked annotation record. Instead of storing aspect terms, categories, and polarities as independent labels, each review is annotated with a single unified structure that explicitly connects a target mention to its aspect category and its sentiment polarity. That linkage is what turns the dataset from a collection of basic ABSA subtasks into a resource for complex tasks such as target-aspect-category detection, and it is also what makes the corpus directly comparable, format-wise, with equivalent datasets in other languages.

What would settle it

Take a random sample of a few hundred released reviews, have a fresh pair of annotators label them from scratch using the published annotation scheme, and compare against the released annotations; if pairwise agreement on the full linked target-aspect-category tuples is much below 90% or chance-corrected agreement is low, the claimed reliability and conversion fidelity are not supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that a Czech-language ABSA benchmark can support complex linked sentiment tasks, not just the separate basic subtasks. It presents 3.1K manually annotated restaurant reviews in which aspect terms, aspect categories, and polarity are annotated in a unified SemEval-2016-style format, so that each sentiment element is linked to the others. The corpus is derived from an older Czech dataset that only had separate labels for basic tasks, and the new format is what allows tasks such as target-aspect-category detection. Two trained annotators reached about 90% agreement, and the paper reports Transformer-based monolingual baselines plus an error analysis. It also makes 2

Load-bearing premise

The annotations are only as good as the human labelling: that the roughly 90% agreement reflects true correctness, and that converting the older dataset's separate labels into one linked SemEval-style format did not change what was originally annotated.

Editorial extensions

If this is right

Same-format Czech data means systems trained or evaluated on SemEval-2016 English data can be run on Czech without reformatting.
Linked target-aspect-category annotations enable joint models rather than separate extraction-then-polarity pipelines.
Published Transformer baselines give reference numbers for future Czech ABSA work.
The 24M unlabeled Czech reviews can support unsupervised pretraining or domain adaptation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper: the linked format opens a direct route to cross-lingual transfer tests between Czech and the other SemEval-2016 languages, which this paper itself does not report.
Beyond the paper: the 24M unlabeled reviews could support domain-adaptive pretraining; a testable extension is to measure whether pretraining on them improves linked ABSA scores beyond the reported baselines.
Beyond the paper: re-annotating a subset of the original reviews with the new scheme and checking label-by-label agreement would quantify how much of the new resource's value comes from the linked format versus fresh annotation work.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The real product here is a linked-annotation Czech ABSA benchmark in SemEval-2016 format. That is genuinely new: the predecessor dataset had separate labels for basic tasks, and this one unifies them for target-aspect-category detection and other complex tasks. Format compatibility is a concrete plus, since it enables direct comparison with English and other SemEval-2016 resources. The paper also delivers 3.1K manually annotated reviews, 24M unlabeled reviews, and Transformer baselines with error analysis. As a resource contribution, this is solid.

The main soft spot, which the stress-test note gets right, is reliability evidence for the linked layer. Raw 90% agreement is inflated by label imbalance in ABSA, and the linked target-aspect-category annotations are exactly where alignment errors would hide. There is no kappa, no per-category or per-layer agreement, no adjudication protocol, and no guideline excerpts in the abstract. If the authors converted older labels rather than independently annotating the links, raw agreement would not catch conversion mistakes. This is a fixable problem: report chance-corrected agreement, break it down by layer and category, and describe the linking process.

One caveat: the full text in the copy I had was unreadable mojibake, so I could not verify baseline numbers, split construction, license details, or the download link. The reader's conditional verdict is appropriate. Nothing in the abstract suggests a defect, but the verification burden is on the authors.

This paper is for the ABSA and lower-resource NLP communities. It deserves a serious referee, not a desk reject. My recommendation: send it to peer review with the explicit request that the authors supply kappa, per-layer agreement, and a transparent description of how the linked annotations were produced and adjudicated. The resource is worth having, but its usability depends on that evidence.

Referee Report

3 major / 1 minor

Summary. The paper introduces a new Czech ABSA dataset consisting of 3.1K manually annotated restaurant reviews, built on an older Czech ABSA corpus and reformatted into the SemEval-2016 unified annotation format to support complex linked tasks such as target-aspect-category detection. The authors report approximately 90% raw inter-annotator agreement, release 24M unlabeled reviews, and provide Transformer-based monolingual baselines with error analysis. The dataset and code are made available for non-commercial research.

Significance. If the annotation quality and format claims hold, this is a useful resource: it provides the first Czech ABSA dataset in a SemEval-2016-compatible format, enabling cross-lingual comparison and evaluation of linked ABSA tasks in Czech. The release of 24M unlabeled reviews is a substantial additional resource for unsupervised and semi-supervised methods. The inclusion of multiple Transformer baselines is a practical benefit, and the public release supports reproducibility.

major comments (3)

[Abstract] The only reliability evidence reported for the linked annotations is 'an inter-annotator agreement rate of approximately 90%'. This is a raw percentage, which is known to be inflated by label imbalance in ABSA annotation. The paper must report chance-corrected agreement (e.g., Cohen's kappa or Krippendorff's alpha) and per-layer/per-category agreement for target spans, aspect categories, polarities, and especially the target-aspect-category link. Without this, the reported baseline scores and the claimed cross-lingual comparability rest on an unverified quality assertion.
[Abstract / annotation procedure] The dataset is built on an older Czech dataset that contained separate labels for basic ABSA tasks, and the new contribution is the unified linked format. The manuscript must clarify whether the links were produced by fresh annotation of the unified format or by mechanically combining the predecessor's separate labels. If derived mechanically, link-alignment errors would be invisible to the reported agreement. Please provide annotation guidelines, adjudication protocol, and a worked example showing how target, aspect category, and polarity are linked.
[Full text (all sections)] The submitted full text is heavily corrupted by an encoding issue, leaving most of the paper unreadable. I cannot verify the dataset statistics, annotation details, baseline configurations, hyperparameters, or the error analysis. This is not a cosmetic defect; it makes the technical content uncheckable. A clean, properly encoded manuscript is required for any further review.

minor comments (1)

[Abstract] The abstract says '24M reviews without annotations'; please clarify whether this counts raw review documents or individual sentences/reviews and whether these unlabeled reviews come from the same domain and time period as the 3.1K annotated set.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the dataset and baselines are self-contained.

full rationale

The paper introduces a manually annotated Czech ABSA dataset and reports baseline evaluations. The abstract and readable portions do not derive any predicted quantity from a fitted parameter or from a self-citation chain. The predecessor Czech dataset is used as source material, but the new contribution is the unified linked annotation produced by two annotators, with agreement reported as an empirical quality measure. Even if the agreement metric is not chance-corrected, that is a measurement-quality concern, not circularity. Baseline results are standard train/test evaluations on the new dataset, not predictions derived from the same labels that define the dataset. No equation or claim in the available text reduces to its own input by construction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

This is a resource paper, so the ledger is light: no fitted numbers and no invented entities. The free parameters list is empty because the design choices that do exist, such as the aspect category set, are inherited from SemEval-2016 rather than fitted to data. The assumptions recorded are the three domain-level presuppositions any dataset paper carries: the annotation scheme transfers to the new language, the source reviews are sound, and the reported agreement metric reflects label quality.

assumptions (3)

domain assumption The SemEval-2016 restaurant annotation scheme transfers cleanly to Czech reviews.
The paper adopts the SemEval-2016 format specifically to enable cross-lingual comparison; this presupposes that the category set and the linking rules fit Czech restaurant-review language, which the abstract asserts but does not demonstrate.
domain assumption The older Czech dataset's reviews are valid source material.
The new dataset is built upon the older Czech dataset; the abstract does not describe whether reviews were re-annotated, filtered, extended, or kept as-is.
domain assumption About 90% inter-annotator agreement implies usable label quality.
The abstract reports agreement as the quality credential without giving kappa, disagreement resolution, adjudication steps, or per-category breakdowns.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Czech Dataset for Complex Aspect-Based Sentiment Analysis Tasks." pith.science (2026). https://pith.science/paper/SELDWU32

@misc{pith2026250808125,
  author       = {Pith},
  title        = {Pith review of: Czech Dataset for Complex Aspect-Based Sentiment Analysis Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SELDWU32}},
  note         = {Machine review of arXiv:2508.08125}
}

read the original abstract

In this paper, we introduce a novel Czech dataset for aspect-based sentiment analysis (ABSA), which consists of 3.1K manually annotated reviews from the restaurant domain. The dataset is built upon the older Czech dataset, which contained only separate labels for the basic ABSA tasks such as aspect term extraction or aspect polarity detection. Unlike its predecessor, our new dataset is specifically designed for more complex tasks, e.g. target-aspect-category detection. These advanced tasks require a unified annotation format, seamlessly linking sentiment elements (labels) together. Our dataset follows the format of the well-known SemEval-2016 datasets. This design choice allows effortless application and evaluation in cross-lingual scenarios, ultimately fostering cross-language comparisons with equivalent counterpart datasets in other languages. The annotation process engaged two trained annotators, yielding an impressive inter-annotator agreement rate of approximately 90%. Additionally, we provide 24M reviews without annotations suitable for unsupervised learning. We present robust monolingual baseline results achieved with various Transformer-based models and insightful error analysis to supplement our contributions. Our code and dataset are freely available for non-commercial research purposes.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

1 extracted references · 1 canonical work pages

[1]

� �� ...

work page Pith review arXiv 2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

� �� ...

work page Pith review arXiv 2025