{"id":"afde47eb-2ece-4be8-ac5f-b73704072c59","arxiv_id":"1908.09785","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The authors introduce a 317-article Bulgarian news toxicity dataset and show that a stacked classifier using LSA, BERT, and other features outperforms the majority-class baseline.","lead":"This paper creates a small dataset of Bulgarian news articles labeled with eight toxicity types plus non-toxic, and trains a meta-classifier that reaches 59% accuracy and 39.7% macro-F1. It is one of the few attempts to detect toxic news outside English, offering a starting resource for Bulgarian.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Source–label confound: non-toxic articles come from media with no Media Scan listings, and the meta-classifier includes media-only features, so the reported 59.0% accuracy may detect source-flag status rather than article toxicity.","rationale":"The reader correctly identified unverified non-toxic labels as a weakness, but the more load-bearing issue is the confound between the label and the source medium combined with the deliberate inclusion of media-level features in the classifier. Section 3 shows that non-toxic articles are drawn exclusively from media without Media Scan listings, while toxic articles come from listed media; Section 4.4 and Section 5.3 then include source information in both individual and meta-classifiers. The media-only setup (setup 12) already achieves 42.04% accuracy, so part of the reported performance is attributable to source identity. This does not require assuming author misconduct; the paper is transparent about the data collection and releases code and data, which makes the concern testable. The multi-label/single-label inconsistency noted by the reader is also real, but the source confound more directly threatens the central claim that the classifier detects toxicity in news articles. A conditional verdict remains appropriate because the dataset contribution is still potentially valuable for Bulgarian toxicity research, provided the evaluation is re-run without source features and ideally on source-matched, manually verified labels.","tokens_in":14564,"tokens_out":5738,"duration_ms":60932,"concrete_test":"Re-run the released pipeline for Table 3 setup 14 with the media-only model (setup 12) and all six media features excluded, keeping only posteriors from text-based models (setups 2–5 and 7–10). If accuracy and macro-F1 fall substantially below the reported 59.06/39.70, the claimed improvement is partly a source-identity shortcut; to fully rule out label confounding, additionally evaluate on a source-matched, manually verified sample where toxic and non-toxic articles come from the same media.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing weakness is a source–label confound in dataset construction, which is more specific than unverified non-toxic labels alone. Section 3 (Data) states that toxic articles come from roughly 150 media listed in Media Scan, while the 96 non-toxic articles were 'fetched from media without toxicity examples in Media Scan.' Thus the toxic/non-toxic label is almost perfectly correlated with whether the source medium has ever been flagged by Media Scan. Section 4.4 then introduces six media-level features (editor, responsible person, bg server, popularity, domain person, days existing), and Section 5.3 explicitly uses the posterior probabilities of the media-only model (setup 12) in the meta-classifier. Table 3 shows that this media-only model alone reaches 42.04% accuracy, a 12-point gain over the majority baseline. Consequently, the headline result of 59.06% accuracy and 39.70% macro-F1 may reflect a source-reputation shortcut rather than article-level toxicity detection. Because the non-toxic set is curated by absence of a listing, the evaluation cannot distinguish 'this article is non-toxic' from 'this article's medium has never been sanctioned by Media Scan.' The paper's central claim that the detector recognizes toxic content is therefore not established by the current experimental design.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a multi-class toxicity detection system for Bulgarian news articles. The authors construct a dataset of 317 articles, of which 221 are labelled into eight toxic categories based on the Media Scan repository and 96 are labelled non-toxic from media without Media Scan toxicity listings. They compare feature sets including LSA, BERT, XLM, USE, ElMo, NELA, stylometric features, and media-level features, and then build a meta-classifier over posterior probabilities of the individual models. The best system (setup 14) is reported to reach 59.06% accuracy and 39.70% macro-F1, which the authors claim are sizable improvements over the majority-class baseline (30.30% accuracy, 5.17% macro-F1).","tokens_in":14851,"tokens_out":4124,"duration_ms":43070,"significance":"If validated, the paper would make a useful contribution as one of the few resources for Bulgarian toxicity detection: it introduces a new multi-label dataset, systematically compares several representations, and releases code and data. The multi-class setup with eight toxicity types plus a non-toxic class is more differentiated than much prior English-focused work. However, the central empirical claim is currently undermined by a source-label confound in the construction of the non-toxic set and by insufficient detail about potential information leakage in the LSA component. The significance of the contribution is therefore contingent on addressing these methodological issues.","major_comments":[{"comment":"The dataset construction introduces a confound between article-level toxicity and source reputation that prevents the headline results from being interpreted as evidence of article-level toxicity detection. Section 3 states that toxic articles come from media listed in Media Scan, while the 96 non-toxic articles were \"fetched from media without toxicity examples in Media Scan.\" This makes the toxic/non-toxic label almost perfectly correlated with whether the source medium has ever been flagged, rather than with the content of the individual article. The media features introduced in Section 4.4 (editor, responsible person, bg server, popularity, domain person, days existing) are then used both as a standalone model (setup 12 in Table 3) and as inputs to the meta-classifier, whose input list in Section 5.3 explicitly includes the posteriors of setup 12. Setup 12 alone reaches 42.04% accuracy, a 12-point gain over the majority baseline, showing that source-level information carries substantial predictive signal. Consequently, the reported 59.06% accuracy and 39.70% macro-F1 may reflect a source-reputation shortcut rather than detection of toxic content. To support the article-level claim, the authors should re-collect or re-annotate non-toxic articles from the same media that also have toxic articles, or otherwise control for source in the evaluation, and should report results without media features.","section":"Section 3 (Data), Section 4.4 (Media Features), Section 5.3 (Meta Classifier), Table 3"},{"comment":"The description of the LSA model is ambiguous about whether the SVD is fitted inside each cross-validation fold. The text says \"We trained a Latent Semantic Analysis (LSA) model on our data,\" but it does not state that the TF-IDF and SVD transformations are recomputed on the training portion of each fold in the 5-fold cross-validation described in Section 5.1. If the SVD is fit on all 317 articles before splitting, test-fold articles contribute to the learned low-dimensional space, which is a form of information leakage. This matters because LSA (setup 5) is the best individual text representation and feeds into the meta-classifier. The authors should clarify the exact fitting procedure; if the model is fit on the full data, the LSA results and the meta-classifier results must be recomputed with nested fitting.","section":"Section 4.1 (LSA)"},{"comment":"No measures of variability or statistical significance are reported for any of the cross-validation results. With only 317 articles and nine classes, differences such as the 3.5% absolute accuracy gain of the meta-classifier over the best individual LSA model (59.06% vs. 55.59%) could easily fall within fold-to-fold variance. The paper repeatedly emphasizes \"sizable improvements\" over the baseline, but this claim is not supported without per-fold results, standard deviations, or a paired significance test between setup 14 and the relevant baselines and individual models.","section":"Section 5.1, Table 3"}],"minor_comments":[{"comment":"The construction of the meta-classifier is described only as \"we made sure that we do not leak information about the labels when training the meta classifier.\" Please specify exactly how the posterior probabilities are produced (e.g., out-of-fold predictions from the inner cross-validation) so that the procedure is reproducible.","section":"Section 5.3"},{"comment":"There is a typo in the example: \"Januarty 1, 2005\" should read \"January 1, 2005.\"","section":"Section 4.4"},{"comment":"The word \"1024-demnsional\" should be \"1024-dimensional.\"","section":"Section 4.5"},{"comment":"The phrase \"ElMo, BERT, xand XLM\" contains a typo: \"xand\" should be \"and.\"","section":"Section 7 (Conclusion)"},{"comment":"The statement that 15,000 additional experiments were run for fine-tuning is not accompanied by the hyperparameter search space or the selection criterion; adding this information would improve reproducibility.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The source-label confound is the main load-bearing issue and is fixable: the authors could add non-toxic articles from the same sources that have toxic articles, remove media-level features from the evaluation, and report results separately for source-controlled subsets. The potential LSA leakage should also be resolved by clarifying or correcting the fitting procedure. The dataset and code release are valuable, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper's lasting value is the dataset — 317 Bulgarian news articles labeled with eight toxicity classes, plus code and features. That's a real contribution for a low-resource language. The paper also does a broad, sensible feature comparison (BERT, XLM, ELMo, USE, LSA, NELA, stylometry, media metadata) and is transparent about releasing code and data.\n\nNow the problem: the evaluation. The non-toxic articles are taken exclusively from media with no toxic listings in Media Scan, while toxic articles come from media that have been flagged. So the article label is nearly determined by whether the source has ever been sanctioned. The meta-classifier then includes six media-level features, and the media-only model alone reaches 42% accuracy — 12 points over the majority baseline. That strongly suggests the headline 59% accuracy is in large part a source-reputation shortcut, not article-level toxicity detection. The authors even note that the media features help, but they don't recognize it as a confound.\n\nSmaller issues: the non-toxic labels are unverified (absence of a listing is not the same as clean), the paper says the task is multi-label but the experiments are single-label, LSA is described as trained on 'our data' without clarifying fold separation, and there are no error bars or significance tests.\n\nWhat's still valid? The dataset itself is useful if you treat the labels cautiously. The Media Scan curation is real manual work, and the article text is released. For Bulgarian, a future worker could re-annotate non-toxic articles from the same sources and avoid the confound. The feature comparison is also a reasonable baseline survey, though the numbers should be taken with a grain of salt.\n\nI think the paper deserves a serious referee — not for the claims as written, but because the resource is worth building on and the confound is addressable. A referee should ask for an ablation that removes media features, an evaluation on held-out sources, or at least a clear discussion of the limitation. If the authors fix the framing, the paper becomes a modest but honest contribution.\n\nIn short: cite it for the dataset, not for the accuracy figure. Bring it to reading group if you want a teaching example of source confounding.","headline":"A useful new dataset for Bulgarian toxicity detection, but the evaluation is confounded by source selection, so the headline accuracy likely overstates content-based detection.","tokens_in":15354,"tokens_out":2482,"would_cite":true,"duration_ms":26521,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Bulgarian news toxicity detector reaches 59% accuracy across nine labels.","keywords":["toxicity detection","Bulgarian","news articles","multi-class classification","meta-classifier","BERT","low-resource language","fake news"],"falsifier":"Ask independent Bulgarian-speaking annotators to label the 96 non-toxic articles, or to re-label all 317; if a large share of the assumed-clean articles turn out to be toxic, retraining on corrected labels would likely drop the reported 59.0% accuracy and 39.7 macro-F1. More narrowly, the confusion matrix's heavy misclassification of fake news as conspiracy can be checked against a larger sample of those two classes.","tokens_in":14394,"feed_emoji":"📰","tokens_out":7682,"duration_ms":73592,"temperature":0.7,"pith_summary":"This paper tries to show that toxic content in Bulgarian news articles can be recognized automatically, even though almost all prior toxicity-detection work has targeted English. The authors built a new dataset of 317 articles, 221 toxic and 96 non-toxic, manually categorized into eight toxic classes plus a non-toxic class by an existing Bulgarian media-monitoring archive. They trained separate classifiers on different text representations—BERT, XLM, ElMo, Universal Sentence Encoder, LSA, stylometric and media features—and combined their predicted probabilities in a meta-classifier. That meta-classifier reaches 59.0% accuracy and 39.7 macro-F1, against a majority-class baseline of 30.3% accuracy and 5.2 macro-F1. If the result holds, it suggests a practical Bulgarian-language filter for harmful news is within reach with a small labeled corpus.","feed_headline":"59% accuracy in a nine-way Bulgarian toxicity detector","feed_subtitle":"A meta-classifier over language models beats the majority baseline by nearly 29 accuracy points on a 317-article dataset.","key_machinery":"The load-bearing mechanism is the meta-classifier: a logistic regression model whose inputs are the posterior class probabilities produced by several base classifiers, rather than raw text. Because the dataset is small (317 articles), the authors train one model per feature type and then let the meta-classifier learn how to weight those models' predictions. The base models cover complementary views of the text: BERT and XLM transformer representations, ElMo contextual word vectors, Universal Sentence Encoder embeddings, LSA low-rank vectors, NELA style and content features, stylometric counts, and six metadata features about the news medium. This stacking step is what lifts accuracy from 30.3% at the baseline to 59.0%.","core_discovery":"The paper's central claim is that a nine-way classifier can distinguish Bulgarian news articles into non-toxic and eight toxic categories—fake news, sensationalism, hate speech, conspiracy theories, anti-democratic, pro-authoritarian, defamation, and delusion—using only a few hundred labeled examples. The evidence is a 317-article dataset whose toxic labels come from a five-year human-curated archive of Bulgarian media, supplemented by 96 articles taken from media with no listed toxic items. The best model is not any single text representation but a meta-classifier: a logistic regression model trained on the posterior probabilities of several base classifiers built from BERT, XLM, ElMo, Universal Sentence Encoder, LSA, stylometric, NELA, and media features. It reports 59.0% accuracy and 39.7 macro-F1, compared with 30.3% accuracy and 5.2 macro-F1 for the majority-class baseline; the best single representation, LSA, reaches 55.6% accuracy and 42.1 macro-F1. The paper also reports that oversampling and a feed-forward neural network did not improve over logistic regression on this small dataset.","pith_inferences":["The paper's treatment of the non-toxic class is the least verified part of the data; an obvious extension is to have those 96 articles independently annotated, which would test whether the accuracy gain is real or an artifact of a clean-by-assumption class.","Since the source archive allows multiple toxicity labels per article, reformulating the task as multi-label classification could better match the data and may improve recall on rare classes like delusion and anti-democratic.","The finding that English-translation features nearly match Bulgarian-native ones suggests a cheap recipe for other low-resource languages: translate, apply English transfer models, and stack the probabilities; this recipe is testable on a second language."],"forward_implications":["A nine-way toxic/non-toxic classifier for Bulgarian reaches 59.0% accuracy and 39.7 macro-F1, roughly doubling the majority baseline's F1, on a 317-article dataset.","The meta-classifier outperforms every single representation, including LSA (55.6% accuracy, 42.1 macro-F1), so combining diverse feature models appears to be the most reliable path on small in-language datasets.","English-translation features (BERT, USE, ElMo, NELA) perform close to or better than Bulgarian-native features, meaning English resources can be reused for low-resource toxicity detection.","The confusion matrix shows the model is weakest on the rare toxic classes: the three smallest classes together cover less than 18% of the dataset and are rarely predicted.","Because the dataset and code are released, the reported numbers can be reproduced and the classifier can be extended or rebalanced in future work."],"supporting_citations":[{"why":"Supplies the pretrained BERT embeddings used for title and body representations.","marker":"Devlin et al., 2019"},{"why":"Supplies XLM cross-lingual representations used as one feature type.","marker":"Lample and Conneau, 2019"},{"why":"Supplies Universal Sentence Encoder embeddings applied to English translations.","marker":"Cer et al., 2018"},{"why":"Supplies ElMo contextual embeddings for title and body.","marker":"Peters et al., 2018"},{"why":"Provides the NELA toolkit whose 129 features capture style, sentiment, topic, and morality.","marker":"Horne et al., 2018a"},{"why":"Defines the large feature set the paper says it uses when analyzing news articles.","marker":"Horne et al., 2018b"},{"why":"Prior Bulgarian fake-news and click-bait detector, the only Bulgarian toxicity baseline the paper extends.","marker":"Karadzhov et al., 2017a"}],"fun_headline_variants":["Bulgarian toxicity detector: meta-classifier hits 59% accuracy","Nine-way Bulgarian news classifier reaches 59% accuracy","Meta-model beats baseline by 29 points on Bulgarian toxic news","From ElMo to BERT: ensemble tackles Bulgarian toxicity","Bulgarian toxic news: multi-class approach outperforms majority label"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The non-toxic training articles are assumed to be non-toxic only because they come from media that have no toxic articles listed in the monitoring archive; no one independently verified that these 96 articles are actually clean, so the non-toxic class and the headline accuracy figures could be built on mislabeled examples.","fun_headline_variants_meta":{"raw":{"variants":["Bulgarian toxicity detector: meta-classifier hits 59% accuracy","Nine-way Bulgarian news classifier reaches 59% accuracy","Meta-model beats baseline by 29 points on Bulgarian toxic news","From ElMo to BERT: ensemble tackles Bulgarian toxicity","Bulgarian toxic news: multi-class approach outperforms majority label"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000802,"raw_usage":{"total_tokens":3540,"prompt_tokens":976,"completion_tokens":2564,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":2492}},"tokens_in":592,"tokens_out":2564,"duration_ms":17744,"temperature":1.0,"reasoning_tokens":2492,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:01:09.694799+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask independent Bulgarian-speaking annotators to label the 96 non-toxic articles, or to re-label all 317; if a large share of the assumed-clean articles turn out to be toxic, retraining on corrected labels would likely drop the reported 59.0% accuracy and 39.7 macro-F1. More narrowly, the confusion matrix's heavy misclassification of fake news as conspiracy can be checked against a larger sample of those two classes.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies ElMo contextual embeddings for title and body."}],"review_version":1}