Pith. sign in

REVIEW 4 major objections 6 minor 33 references

The 2021 Tokyo Olympics Multilingual News Article Dataset

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Pith's one-line read This paper introduces OG2021, a manually annotated multilingual news dataset of 10,940 articles covering 1,350 sub-events of the 2021 Tokyo Olympics, and argues it fills a gap in benchmarks for multilingual news clustering in…

desk verdict A genuinely new multilingual news clustering benchmark with a documented pipeline, but the missing inter-annotator agreement means the gold-standard labels are not yet demonstrated to be reliable. read the letter →

arxiv 2502.06648 v2 pith:YSLLM2R4 submitted 2025-02-10 cs.IR cs.AIcs.CL

classification cs.IRcs.AIcs.CL
keywords multilingualnewsdataseteventclusteringTokyoOlympics2021manualannotationbenchmarkhigh-frequencyeventsarticlesclusterevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces OG2021, a new publicly available dataset of 10,940 news articles about the 2021 Tokyo Olympics, written in nine languages and grouped into 1,350 manually verified event clusters. The authors argue this fills a gap: existing news datasets are mostly monolingual or classify articles by broad topic rather than specific events, and the one comparable event-clustering dataset the survey found spreads its events thinly over a long time range. The Olympics, with many simultaneous sub-events in a short window and similar locations, provides a high-frequency setting that stresses clustering algorithms. The paper's central claim is that OG2021 gives researchers a reusable benchmark for evaluating online multilingual news clustering, with each article's cluster ID serving as ground truth.

What carries the argument

The load-bearing mechanism is a three-stage annotation pipeline. Retrieval first filters articles by language, publication window, and required Wikipedia concepts (Olympic Games, Japan, plus one of seven specified sports) through a news-aggregation service. Automatic clustering then runs an online single-pass algorithm that represents each article by a multilingual sentence-embedding vector, a set of extracted named entities, and its publication datetime, and assigns it to a cluster only if content similarity, entity overlap, and temporal proximity all exceed tuned thresholds. Manual annotation completes the pipeline: human annotators review every automatic cluster, following criteria based on who, what, where, when, and optionally how an event unfolded, and are allowed to merge, split, relocate, or delete articles. The cluster IDs produced by this combined process are what makes the dataset a benchmark rather than just a collection of articles.

What would settle it

Re-annotate a random sample of the 1,350 clusters with independent annotators using the paper's stated criteria, then compute pairwise agreement between the original cluster IDs and the re-annotation, for example with BCubed F1 or the adjusted Rand index. Low agreement would show that the dataset's labels do not provide a stable benchmark for clustering.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that a dense, multilingual, real-time news stream can be captured and manually curated into a usable benchmark. The dataset contains 10,940 articles in English, Portuguese, Spanish, French, Russian, German, Slovenian, Arabic, and Chinese, published between July 1 and August 14, 2021, sourced from 1,918 publishers. The articles are grouped into 1,350 clusters corresponding to Olympic sub-events, with the largest cluster, the opening ceremony, containing 499 articles and about 28% of clusters containing articles in at least two languages. The paper reports that the cluster labels were produced by automatic online clustering followed by a full manual review in which annotators merged, split, relocated, or removed clusters according to journalistic who/what/where/when/how criteria, so the final cluster IDs represent the intended ground truth for evaluating clustering algorithms.

Load-bearing premise

The cluster labels are trustworthy: the manual annotators, using machine translation when needed, consistently applied the who/what/where/when/how criteria so that articles about the same sub-event receive the same cluster ID, but the paper reports no inter-annotator agreement or error analysis to back that consistency.

Editorial extensions

If this is right

  • A clustering algorithm can now be scored against 1,350 event clusters in a setting where many sub-events occur simultaneously and in similar locations, which is a harder test than most existing news benchmarks provide.
  • Because about 28% of clusters contain at least two languages, the dataset supports evaluating cross-lingual event clustering rather than only monolingual grouping.
  • The per-language cluster distributions and language co-occurrence statistics give a concrete profile of how reporting on the same event differs across languages and scripts.
  • The same retrieval-and-annotation pipeline could be rerun on future Olympics or other time-dense global events to produce comparable benchmarks and track how event coverage evolves.
  • Researchers can use the dataset not only for clustering evaluation but also for studying the dynamics of the 2021 Olympics, including media perspectives and the challenges of COVID-era competition.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that its annotation criteria define an operational notion of 'event' centered on who/what/where/when with 'how' optional; that protocol could be extracted and reused for building event datasets on other topics.
  • A natural testable extension is to measure inter-annotator agreement on a sample of clusters; the paper reports no such agreement, so its absence is the main open question for label reliability.
  • The public version of the dataset omits article bodies, so users who cannot access the research version would evaluate on titles, URLs, and metadata alone; this split itself could be studied as a low-resource clustering scenario.
  • Because the pipeline depends on a multilingual sentence-embedding model, benchmark scores will likely be sensitive to that model's language coverage; comparing alternative embedding models on this dataset would reveal which language pairs drive clustering errors.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces OG2021, a multilingual news article dataset covering the 2021 Tokyo Olympics, containing 10,940 articles from 1,918 publishers in nine languages, grouped into 1,350 event clusters. The authors describe a three-stage pipeline: retrieval of articles via the Event Registry API using criteria on language, publication date, and Wikipedia concepts; automatic online clustering of articles into event clusters; and manual annotation and evaluation where annotators review, merge, split, relocate, and delete clusters according to criteria based on the journalistic questions who, what, where, when, and how. The paper reports descriptive statistics and technical validation of the dataset, and releases public and research versions through CLARIN.SI. The intended contribution is a benchmark for evaluating online multilingual news clustering algorithms in a high-frequency event setting.

Significance. If the cluster labels are reliable, OG2021 fills a genuine gap in multilingual news event clustering benchmarks, especially for temporally dense and simultaneous sub-events. The paper's strengths include transparent retrieval criteria, public release on a recognized repository, and basic technical validation such as language distributions, cluster-size statistics, and language co-occurrence. The main limitation is that the benchmark value rests entirely on the manual cluster labels, and the paper provides no evidence of their reliability beyond a description of the annotation procedure. Addressing this omission would make the dataset a valuable community resource.

major comments (4)
  1. [Manual annotation and evaluation] The paper does not report the number of annotators, their language coverage, how disagreements were resolved, or any inter-annotator agreement statistic. Since the manual cluster labels are the core benchmark contribution, the absence of such evidence means the central claim that articles sharing a cluster ID report on the same sub-event is unverified. I recommend adding an inter-annotator agreement study, even on a subsample, and a detailed description of the annotation team and disagreement resolution process.
  2. [News cleanup] The text states that articles are split into seven datasets, one per sports concept, but does not specify how articles that mention multiple of the sports in the retrieval criteria are assigned. If such articles appear in more than one sports dataset, they may be processed multiple times and could appear multiple times in the final CSV, which would affect the reported article count, cluster count, and cluster-size statistics. The authors should clarify the assignment rule and confirm that the final dataset contains unique article IDs with no duplicates.
  3. [Manual annotation and evaluation] Because annotators started from clusters produced by an automatic clustering algorithm, the final cluster IDs may inherit the algorithm's biases. The paper mentions that annotators could merge, split, relocate, and eliminate clusters, but it reports no statistics on how often these actions were taken. Without information on, for example, the number of clusters merged or split, or the number of articles relocated, it is impossible to judge how much the manual stage changed the automatic output, and thus how independent the final labels are from the automatic initialization. I request that these statistics be reported, or that the authors provide other evidence of the manual stage's substantial contribution.
  4. [Automatic news clustering] The clustering thresholds for content similarity, entity overlap, and temporal proximity are said to be chosen to optimize precision while maintaining recall, but the actual threshold values are not reported. Since these thresholds determine the initial clusters that annotators review, their absence impairs reproducibility of the pipeline. I recommend reporting the exact thresholds or providing a configuration file in the released code.
minor comments (6)
  1. [Manual annotation and evaluation] Typo: 'the annotators are tasks to group articles' should be 'the annotators are tasked to group articles'.
  2. [Table 1] The entry for Miranda et al. lists the time range as 'Dec 2013 - Avg 2015'; 'Avg' should likely be 'Aug'.
  3. [Background & Summary] The phrase 'it's news article metadata' should be 'its news article metadata'.
  4. [Figure 5] The language co-occurrence matrix is visually dense; consider providing the values in a table or with larger font to improve readability.
  5. [Data Records] The description of the BODY variable states it is 'truncated' but then says the public version does not include it at all; this is confusing and should be clarified as 'available only in the research version'.
  6. [Manual annotation criteria] The phrase 'The intentionally ambiguous criterion' is unclear; it likely means 'flexible' or 'subjective' rather than 'ambiguous'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the dataset labels are the product of manual annotation, not a fitted prediction or self-citation reduction.

full rationale

The paper's central claim is the creation of OG2021, a manually annotated multilingual news dataset. The derivation chain is a data-construction pipeline rather than a predictive argument. Automatic clustering (citing the first author's algorithm [25]) is used only to produce an initial cluster structure, and the paper explicitly states that annotators could relocate articles, merge, split, and eliminate clusters: 'Annotators had the flexibility to remove specific articles from the dataset, relocate articles between clusters, merge, divide, and eliminate entire clusters.' The final cluster IDs are therefore not equivalent to the automatic clustering output by construction; no fitted parameter is renamed as a prediction, and the paper does not evaluate the authors' own clustering algorithm against the dataset. The self-citations to Event Registry, wikification, and the clustering algorithm are tool citations, not load-bearing justifications of the dataset's validity, and no uniqueness theorem or ansatz is imported from prior work by the same authors. The absence of inter-annotator agreement statistics is a reliability limitation, not evidence of circularity; it concerns the quality of the human ground truth rather than a reduction of the claimed result to its own inputs.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The dataset pipeline relies on the correctness of Event Registry retrieval and wikification, on the validity of the retrieval filter, and on the consistency of manual event annotation. No inter-annotator reliability metric is reported, so label quality is assumed. The automatic clustering thresholds are tuned but not specified, though manual correction limits their influence on the final labels.

free parameters (1)
  • Automatic clustering thresholds (content similarity, entity overlap, temporal proximity) = not reported
    The thresholds were 'chosen to optimize precision while maintaining a reasonable recall score' (Methods, Automatic news clustering). The values are not published; they influence the automatic clusters that annotators then correct, so the final labels do not depend on them directly.
assumptions (3)
  • domain assumption Event Registry retrieval with the required Wikipedia concepts reliably returns articles about the 2021 Olympics and the selected sports.
    All 36k retrieved articles are selected by the API filters for language, date range, and concepts (Methods, Criteria definition and News collection); any missed or falsely linked articles affect coverage and composition.
  • domain assumption The manual annotation criteria based on who, what, where, when, and how produce consistent event groupings.
    The paper states the criteria are 'intentionally ambiguous' and that the 'how' criterion is optional (Manual annotation and evaluation); no inter-annotator agreement is reported, so consistency is assumed.
  • domain assumption The automatic clustering algorithm's content embeddings, named entities, and publication dates provide useful initial clusters that can be corrected by human review.
    The automatic step uses SBERT and WikiNEuRal with tuned thresholds (Automatic news clustering), but manual corrections make this assumption less central to the final labels.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The 2021 Tokyo Olympics Multilingual News Article Dataset." pith.science (2026). https://pith.science/paper/YSLLM2R4

@misc{pith2026250206648,
  author       = {Pith},
  title        = {Pith review of: The 2021 Tokyo Olympics Multilingual News Article Dataset},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YSLLM2R4}},
  note         = {Machine review of arXiv:2502.06648}
}
read the original abstract

In this paper, we introduce a dataset of multilingual news articles covering the 2021 Tokyo Olympics. A total of 10,940 news articles were gathered from 1,918 different publishers, covering 1,350 sub-events of the 2021 Olympics, and published between July 1, 2021, and August 14, 2021. These articles are written in nine languages from different language families and in different scripts. To create the dataset, the raw news articles were first retrieved via a service that collects and analyzes news articles. Then, the articles were grouped using an online clustering algorithm, with each group containing articles reporting on the same sub-event. Finally, the groups were manually annotated and evaluated. The development of this dataset aims to provide a resource for evaluating the performance of multilingual news clustering algorithms, for which limited datasets are available. It can also be used to analyze the dynamics and events of the 2021 Tokyo Olympics from different perspectives. The dataset is available in CSV format and can be accessed from the CLARIN.SI repository.

Figures

Figures reproduced from arXiv: 2502.06648 by the authors.

Figure 1
Figure 1. The schematic overview of the OG2021 development. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The OG2021 article distribution by date. The majority of articles were published between the official start of [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. The OG2021 article distribution by size. Globally, about 95% of clusters contain 25 or fewer articles. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The OG2021 cluster distribution per language. Almost 28% of the clusters contain two or more languages. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: The OG2021 language co-occurrence in clusters. All language pairs appear together in at least one cluster. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 31 canonical work pages

  1. [1]

    Newsweeder: Learning to filter netnews

    Ken Lang. Newsweeder: Learning to filter netnews. In Machine Learning Proceedings 1995, pages 331–339. Morgan Kaufmann, 1995

  2. [2]

    Reuters-21578 text categorization collection, 1997

    David Lewis. Reuters-21578 text categorization collection, 1997

  3. [3]

    Antonio Gulli. AG news. http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html,

  4. [4]

    Practical solutions to the problem of diagonal dominance in kernel document clustering

    Derek Greene and Pádraig Cunningham. Practical solutions to the problem of diagonal dominance in kernel document clustering. In Proceedings of the 23rd International Conference on Machine learning (ICML), pages 377–384. ACM Press, 2006

  5. [5]

    RCV1: A new benchmark collection for text categoriza- tion research

    David D Lewis, Yiming Yang, Tony G Rose, and Fan Li. RCV1: A new benchmark collection for text categoriza- tion research. Journal of Machine Learning Research, 5:361–397, 2004

  6. [6]

    TDT2 multilanguage text version 4.0, 2001

    Charles Wayne, George R Doddington, Jonathan G Fiscus, Mark Liberman, Jennifer Alabiso, David Graff, and Christopher Cieri. TDT2 multilanguage text version 4.0, 2001. 10 The 2021 Tokyo Olympics Multilingual News Article Dataset

  7. [7]

    NELA-GT-2018: A large multi-labelled news dataset for the study of misinformation in news articles

    Jeppe Nørregaard, Benjamin D Horne, and Sibel Adalı. NELA-GT-2018: A large multi-labelled news dataset for the study of misinformation in news articles. In Proceedings of the International AAAI Conference on Web and Social Media, volume 13, pages 630–638. Association for the Advancement of Artificial Intelligence (AAAI), 2019

  8. [8]

    Zero-shot learning for cross- lingual news sentiment classification

    Andraž Pelicon, Marko Pranji´c, Dragana Miljkovi´c, Blaž Škrlj, and Senja Pollak. Zero-shot learning for cross- lingual news sentiment classification. Applied sciences (Basel, Switzerland), 10:5993, 2020

Show all 33 references
  1. [9]

    NELA-GT-2019: A large multi-labelled news dataset for the study of misinformation in news articles

    Maurício Gruppi, Benjamin D Horne, and Sibel Adalı. NELA-GT-2019: A large multi-labelled news dataset for the study of misinformation in news articles. Preprint at https://arxiv.org/abs/2003.08444, 2020

  2. [10]

    The POLUSA dataset: 0.9M political news articles balanced by time and outlet popularity

    Lukas Gebhard and Felix Hamborg. The POLUSA dataset: 0.9M political news articles balanced by time and outlet popularity. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, pages 467–468. ACM, 2020

  3. [11]

    Sculpting Data for ML: The first act of Machine Learning

    Rishabh Misra and Jigyasa Grover. Sculpting Data for ML: The first act of Machine Learning. Independently Published, 2021

  4. [12]

    News category dataset

    Rishabh Misra. News category dataset. Preprint at https://arxiv.org/abs/2209.11429, 2022

  5. [13]

    Efficient discrimination between closely related languages

    Jörg Tiedemann and Nikola Ljubeši´c. Efficient discrimination between closely related languages. In Proceedings of COLING 2012, pages 2619–2634, 2012

  6. [14]

    Creating a persian-english comparable corpus

    Homa Baradaran Hashemi, Azadeh Shakery, and Heshaam Faili. Creating a persian-english comparable corpus. In Multilingual and Multimodal Information Access Evaluation, Lecture notes in computer science, pages 27–39. Springer Berlin Heidelberg, 2010

  7. [15]

    A multilingual simplified language news corpus

    Renate Hauser, Jannis Vamvas, Sarah Ebling, and Martin V olk. A multilingual simplified language news corpus. In Proceedings of the 2nd Workshop on Tools and Resources to Empower People with REAding DIfficulties (READI) within the 13th Language Resources and Evaluation Confere...

  8. [16]

    MassiveSumm: a very large-scale, very multilingual, news summarisation dataset

    Daniel Varab and Natalie Schluter. MassiveSumm: a very large-scale, very multilingual, news summarisation dataset. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10150–10161. Association for Computational Linguistics, 2021

  9. [17]

    CC- news-en: A large english news corpus

    Joel Mackenzie, Rodger Benham, Matthias Petri, Johanne R Trippas, J Shane Culpepper, and Alistair Moffat. CC- news-en: A large english news corpus. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. ACM, 2020

  10. [18]

    A diverse multilingual news headlines dataset from around the world

    Felix Leeb and Bernhard Schölkopf. A diverse multilingual news headlines dataset from around the world. Preprint at https://arxiv.org/abs/2403.19352, 2024

  11. [19]

    Event registry: learning about world events from news

    Gregor Leban, Blaz Fortuna, Janez Brank, and Marko Grobelnik. Event registry: learning about world events from news. In Proceedings of the 23rd International Conference on World Wide Web. ACM, 2014

  12. [20]

    Gdelt: Global data on events, location, and tone, 1979–2012

    K Leetaru and P A Schrodt. Gdelt: Global data on events, location, and tone, 1979–2012. ISA Annual Convention, 2013

  13. [21]

    Multilingual clustering of streaming news

    Sebastião Miranda, Art¯urs Znotin, š, Shay B Cohen, and Guntis Barzdins. Multilingual clustering of streaming news. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages 4535–4544. Association for Computational Linguistics, 2018

  14. [22]

    News across languages - cross-lingual document similarity and event tracking

    Jan Rupnik, Andrej Muhic, Gregor Leban, Primoz Skraba, Blaz Fortuna, and Marko Grobelnik. News across languages - cross-lingual document similarity and event tracking. Journal of Artificial Intelligence Research, 55:283–316, 2016

  15. [23]

    Annotating documents with relevant wikipedia concepts

    Janez Brank, Gregor Leban, and Marko Grobelnik. Annotating documents with relevant wikipedia concepts. In Proceedings of Slovenian KDD Conference on Data Mining and Data Warehouses (SiKDD), 2017

  16. [24]

    Semantic annotation of documents based on wikipedia concepts

    Janez Brank, Gregor Leban, and Marko Grobelnik. Semantic annotation of documents based on wikipedia concepts. Informatica, 42, 2018

  17. [25]

    News stream clustering using multilingual language models

    Erik Novak. News stream clustering using multilingual language models. In The Proceedings of the Conference on Data Mining and Data Warehouses (SiKDD), 2021

  18. [26]

    Sentence-BERT: Sentence embeddings using siamese BERT-networks

    Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)...

  19. [27]

    Augmented SBERT: Data augmenta- tion method for improving bi-encoders for pairwise sentence scoring tasks

    Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. Augmented SBERT: Data augmenta- tion method for improving bi-encoders for pairwise sentence scoring tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computat...

  20. [28]

    WikiNEu- Ral: Combined neural and knowledge-based silver data creation for multilingual NER

    Simone Tedeschi, Valentino Maiorca, Niccolò Campolungo, Francesco Cecconi, and Roberto Navigli. WikiNEu- Ral: Combined neural and knowledge-based silver data creation for multilingual NER. In Findings of the Association for Computational Linguistics: EMNLP 2021 , pages 2521–25...

  21. [29]

    The news articles reporting on the 2021 tokyo olympics data set OG2021 (public), 2024

    Erik Novak, Erik Calcina, Dunja Mladeni´c, and Marko Grobelnik. The news articles reporting on the 2021 tokyo olympics data set OG2021 (public), 2024. Slovenian language resource repository CLARIN.SI

  22. [30]

    The news articles reporting on the 2021 tokyo olympics data set OG2021 (research), 2024

    Erik Novak, Erik Calcina, Dunja Mladeni´c, and Marko Grobelnik. The news articles reporting on the 2021 tokyo olympics data set OG2021 (research), 2024. Slovenian language resource repository CLARIN.SI

  23. [31]

    A comparison of extrinsic clustering evaluation metrics based on formal constraints

    Enrique Amigó, Julio Gonzalo, Javier Artiles, and Felisa Verdejo. A comparison of extrinsic clustering evaluation metrics based on formal constraints. Information Retrieval, 12:461–486, 2009

  24. [32]

    The code for creating the OG2021 dataset, 2024

    Erik Novak, Matevž Matjašec, and Erik Calcina. The code for creating the OG2021 dataset, 2024. 12

  25. [2004]

    Accessed: 2024-08-05

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.