{"id":"93a35326-0558-4233-a3ae-6509b1c92d0a","arxiv_id":"2509.05719","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A survey and benchmark of Farsi subjective NLP finds few public datasets, missing demographic labels, and highly variable LLM performance across datasets.","lead":"This paper maps 110 studies on Farsi sentiment, emotion, and toxicity detection, finding only 15 public datasets and little demographic metadata. It then tests three open LLMs and a fine-tuned encoder on nine datasets, showing unstable scores across models and tasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Survey completeness is the load-bearing assumption: capping Google Scholar at top-10 results per query may miss public Farsi datasets, so the scarcity claim could be overstated.","rationale":"The reader's weakest-assumption analysis identified exactly this issue: restricting Google Scholar to top-10 hits could miss relevant datasets and make the scarcity claim overstated. My read agrees with that assessment. The concern is real but does not overturn the paper; it warrants a conditional verdict, which the reader already issued. The paper is transparent about many limitations, and the experimental findings—even if limited—still provide useful evidence about LLM performance on Farsi subjective tasks. The concrete test would settle whether the survey is sufficiently complete; until then, the scarcity claim should be treated as provisional. I do not see grounds to reject the paper, as the core contribution remains valuable as a survey of what is currently known, provided its completeness is not overclaimed.","tokens_in":13881,"tokens_out":3178,"duration_ms":41697,"concrete_test":"Re-run the 140 query combinations with the same SerpApi setup but retrieving the top 50 results per query instead of the top 10, and add searches of Persian-language academic repositories (SID, Magiran, Ensani, Irandoc) and Iranian conference proceedings. For each newly found paper, check whether it describes a publicly available EA/SA/TD dataset and whether it includes demographic metadata. If more than five additional public datasets (or any demographic-rich datasets) are discovered, the central scarcity and demographic-gap claims need to be revised or qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim—that Farsi subjective NLP suffers from a lack of publicly available datasets—rests on a literature search that (a) draws mostly from ACL Anthology and (b) restricts Google Scholar results to the top 10 per keyword phrase via SerpApi (Section 3). If relevant datasets appear on later result pages or in venues not indexed by these sources (e.g., Iranian journals/conferences indexed by SID, Magiran, Ensani), the survey's count of only 15 public datasets could be an underestimate. The authors do not report recall checks, snowballing from references, or searching Persian-language databases. Since the conclusion 'the volume of data is insufficient to significantly improve a language's prospects in NLP' is directly proportional to the completeness of this inventory, the entire scarcity narrative inherits this uncertainty. This is not an internal inconsistency, but it is a correctness risk: the claim could be overstated if the search was too narrow. The issue is specifically load-bearing because the paper's contribution is a survey that concludes scarcity, and the experimental evaluation is secondary and explicitly limited by the same dataset inventory.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper surveys 110 publications on three subjective NLP tasks in Farsi (sentiment analysis, emotion analysis, toxicity detection), classifying them by dataset creation, public availability, annotation framework, modality, source, and demographic metadata. It reports 15 publicly available Farsi datasets across the three tasks and evaluates three open-source decoder-only LLMs (Llama-3-8B, Mixtral-8x7B, Qwen2-7B) in a zero-shot setting, together with fine-tuned XLM-RoBERTa, on nine datasets, including an experiment on translating data to English and comparing two prompt templates. The main conclusions are that Farsi subjective NLP suffers from scarce public datasets and missing demographic/annotation documentation; that LLMs perform poorly and unstably across datasets and models, especially on emotion analysis; and that fine-tuning consistently improves performance.","tokens_in":14161,"tokens_out":7908,"duration_ms":83640,"significance":"If the survey inventory is reliable, the paper provides a useful and much-needed resource map for Farsi subjective NLP, identifying specific gaps in emotion analysis, toxicity detection, and demographic metadata. The experimental benchmark is a reasonable first comparison of open encoder and decoder models on these tasks, and the authors make the list of reviewed papers publicly available, which is a reproducibility asset. The paper also credibly shows that translation to English does not consistently help and that prompt variation has limited impact on emotion analysis. However, the central survey claim depends on a search completeness assumption that is not validated, and the experimental conclusions are not supported by statistical estimates of variability.","major_comments":[{"comment":"The paper's central claim of a 'lack of publicly available datasets' (Abstract) rests on a literature search restricted to ACL Anthology and the top-10 Google Scholar results per query via SerpApi. No snowballing from references, no recall check against existing Farsi surveys, and no queries to Persian-language repositories (e.g., SID, Magiran, Ensani) are reported. Because the count of 15 public datasets (Table 1) is the load-bearing evidence for the survey's main conclusion, the possibility of undercounting is a substantive correctness risk. Please add a validation/recall check against known Farsi surveys (e.g., Rajabi & Valavi 2021; Asgarnezhad & Monadjemi 2021; Borowczyk 2023), report coverage statistics, or soften the scarcity claim accordingly.","section":"Section 3"},{"comment":"Table 4 reports single macro-F1 values without error bars, confidence intervals, or significance tests. The abstract's 'highly unstable across datasets and models' is a claim about variability, but the paper provides no variance estimates, no repeated runs, and no information about sampling temperature or seeds for the decoder-only models. It is therefore impossible to distinguish genuine model/dataset instability from evaluation noise. Please report repeated-run statistics, standard deviations, and ideally significance tests for the main comparisons (e.g., fine-tuned XLM-R vs. zero-shot LLMs).","section":"Section 5.2 / Table 4"},{"comment":"Prompt template selection was performed on subsamples of EmoPars and MirasOpinion, and the same datasets are then used again in the final evaluation reported in Table 4. If the prompt-selection subsamples were not disjoint from the final evaluation subsets, the reported advantage of template (II) may be inflated by test-set tuning. Please state whether the selection data were held out, or use a separate development set for prompt selection.","section":"Section 5.1 / Section 4.2.1"}],"minor_comments":[{"comment":"Typographical issues: 'disguss' should be 'disgust'; the 'Farsi.Task' run-in is a formatting error; the EmoPars labels 'E - [disgust] + [hatred]' do not match the prose description that mentions 'wonder' (or surprise) instead of disgust. Please align the table with the dataset's actual label set.","section":"Table 1"},{"comment":"The sentence 'Only authors of three datasets (Yazdani and Shekofteh, 2022) provide detailed documentation...' is grammatically garbled and does not identify which three datasets are meant. Please rewrite and name the datasets.","section":"Section 3.2"},{"comment":"Inconsistent terminology: 'Mixtral-7B' vs. 'Mixtral-8x7B', 'SentiPars' vs. 'SentiPers', and 'NLBB' should be 'NLLB' (No Language Left Behind). Please standardize.","section":"Throughout"},{"comment":"The heading 'Model hyperparameters' has no content under it; the hyperparameters appear in prose later. Either move the prose under the heading or remove the empty heading.","section":"Appendix B.3"},{"comment":"Llama 3 is cited twice as 'Dubey et al., 2024' and 'Grattafiori et al., 2024' with the same title and same arXiv identifier (arXiv:2407.21783). These appear to be duplicate references; please merge.","section":"References"},{"comment":"The header 'T emplate A vg. F1 (I) (II)' is confusing. It should clearly indicate separate columns for Template (I) and Template (II) F1 scores.","section":"Table 3"},{"comment":"The conclusion 'fine-tuning consistently improves performance across all tasks' is supported only at the task-average level in Table 4; on Pars-OFF, Qwen2-7B (0.925) exceeds XLM-RoBERTa (0.854). Please qualify the claim.","section":"Section 6"},{"comment":"The Limitations section acknowledges biases in evaluation datasets but does not mention the survey search's reliance on top-10 Google Scholar results or the absence of Persian-language database queries. This is an important limitation of the main survey claim and should be stated.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"The paper's main contribution is the survey, and the experimental evaluation is secondary. The stress-test concern about search completeness is real and load-bearing: without a recall check, the '15 public datasets' count and the resulting scarcity narrative may be overstated. The experimental section is also methodologically thin (no variance estimates, prompt selection on evaluation data), but these issues are fixable. I would not reject on these grounds; however, the authors should be asked to either substantially validate their search procedure or carefully weaken the scarcity claim. The adoption of a co-authored annotation framework (Plaza-del Arco et al., 2024) is acceptable, though the dependence on that framework could be made more explicit for novelty assessment."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You asked for my read on the Farsi subjective tasks paper. Here's the short version: it's genuinely useful for anyone working on Farsi NLP or low-resource subjective tasks, but it needs a revision to tighten the evidence behind the scarcity narrative and to report uncertainty in the experiments.\n\nThe paper is the first multi-task survey of Farsi sentiment, emotion, and toxicity detection, and that alone is a real contribution. The authors combed through 110 publications, built a transparent inventory of 15 public datasets, and ran a comparative evaluation of three open-source LLMs plus a fine-tuned XLM-RoBERTa. The dataset table with metadata is practical, the translation experiments are a sensible addition, and the finding that fine-tuning consistently beats zero-shot LLMs is credible. They are also honest about limitations, which I appreciate.\n\nThe soft spots are real but not fatal. First, the survey completeness is load-bearing. Search is limited to ACL Anthology and Google Scholar top-10 results per query via SerpApi, with no snowballing or coverage of Persian-language venues indexed by SID, Magiran, or Ensani. That could undercount public datasets, and the abstract's claim that data volume is insufficient to improve Farsi's NLP prospects depends directly on that count. The authors flag the limitation, but the claim is stronger than the search warrants. I'd soften the causal framing.\n\nSecond, the experiments lack statistical grounding. The prompt template was selected after comparing two options on subsamples of the same datasets used in the final evaluation, which can inflate scores. More importantly, there are no error bars, confidence intervals, or significance tests. The headline result—model performance is highly unstable across datasets and models—is plausible from Table 4, but without variance estimates it's hard to know how much is noise. Also, the abstract's phrasing about data volume causing poor prospects goes beyond what the data can show.\n\nThese are fixable issues. The survey and data inventory are valuable, the evaluation is reasonable as a first pass, and the authors are transparent about their assumptions. I'd send this to review with a request for major revision: report standard deviations across seeds, run a quick recall check on the survey search, and temper the causal claim. It deserves referee time, and with those changes it would be a solid reference for Farsi NLP and low-resource subjective tasks broadly.","headline":"A useful first survey and evaluation of Farsi subjective NLP, but the scarcity claim leans on a search method that could undercount datasets and the experiments lack variance reporting.","tokens_in":14587,"tokens_out":1549,"would_cite":true,"duration_ms":19551,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Farsi is de facto low-resource for subjective NLP despite its 127 million speakers, with only 15 public datasets and highly unstable LLM performance.","keywords":["Farsi NLP","subjective tasks","sentiment analysis","emotion analysis","toxicity detection","low-resource languages","demographic metadata","LLM evaluation"],"falsifier":"Search beyond the top-10 Google Scholar results and outside ACL Anthology—local Iranian conferences, institutional repositories, and non-indexed venues—for public Farsi emotion, sentiment, or toxicity datasets; if those searches turn up many additional public datasets with demographic metadata, the scarcity claim weakens. Alternatively, rerun the benchmark on a broader stratified sample of Farsi texts and observe whether LLM F1 scores remain stable across datasets; stability would contradict the paper's 'highly unstable' claim.","tokens_in":13842,"feed_emoji":"📊","tokens_out":7147,"duration_ms":66831,"temperature":0.7,"pith_summary":"Farsi has more than 127 million speakers and a large Wikipedia presence, which usually earns it a \"middle-resource\" label in NLP. This paper argues that the label hides a deeper problem: for subjective tasks—emotion analysis, sentiment analysis, and toxicity detection—Farsi is effectively low-resource. Reviewing 110 publications, the authors find only 15 publicly available datasets, most lacking demographic metadata such as age and gender. In their benchmark, zero-shot LLMs perform poorly and unstably, especially on emotion, while a fine-tuned multilingual encoder model consistently outperforms them. The paper concludes that raw text volume is not enough to lift a language's prospects in subjective NLP.","feed_headline":"Farsi subjective NLP: only 15 public datasets in 110 papers","feed_subtitle":"A survey and benchmark show LLM scores swing across datasets; fine-tuning beats zero-shot.","key_machinery":"The paper's argument runs through two instruments. First, a survey inventory that adapts a prior annotation framework to code each of 110 papers by task, dataset availability, source, size, labels, modality, and presence of demographic metadata. Second, a comparative benchmark: three open-source LLMs (Llama-3-8B, Mixtral-8x7B, Qwen2-7B) in zero-shot prompting with two templates, plus XLM-RoBERTa fine-tuned separately on each of nine datasets, with a Farsi-to-English translation condition. The macro-averaged F1 scores across this task-by-dataset-by-model matrix carry the instability claim, and the fine-tuning comparison carries the finding that fine-tuning consistently improves performance.","core_discovery":"Resource status should be judged task by task, not by corpus size. Farsi's abundant web text and Wikipedia do not yield usable supervised data for emotion analysis, sentiment analysis, or toxicity detection. A structured review of 110 publications surfaces only 15 public datasets (7 EA, 5 SA, 3 TD), many from the same narrow sources (Twitter/Instagram and Digikala reviews); only two include demographic information, and few document annotator agreement. In evaluation, LLMs show highly unstable macro-averaged F1 scores: emotion near 0.2–0.4, sentiment 0.42–0.68, toxicity 0.56–0.94 depending on dataset and model. Fine-tuning XLM-RoBERTa improves every task, while translating Farsi to English gi","pith_inferences":["Inference: LLM 'stability' claims for low-resource languages should be treated as dataset-specific until shown otherwise; model rankings from one Farsi dataset are unlikely to transfer.","Inference: The poor emotion-analysis results may stem partly from label-scheme mismatch, since the datasets do not align with standard emotion frameworks; cross-lingual emotion benchmarks may be hard to interpret even with more data.","Inference: The dominance of social-media and e-commerce sources suggests that datasets from other registers, such as news, literature, or spoken language, could shift both task difficulty and model rankings.","Inference: A testable extension would be to collect demographic-aware Farsi annotations and check whether age- and gender-conditioned evaluation reveals systematic bias that current datasets cannot expose."],"forward_implications":["Language-resource classifications should become task-specific: a language can be text-rich yet supervision-poor for subjective tasks.","Translation to English is not a reliable remedy for low-resource subjective NLP, since it did not consistently improve LLM scores.","Fine-tuned encoder-only models are a stronger default than zero-shot open LLMs on all three Farsi tasks in this study.","New Farsi subjective dataset creation should prioritize demographic metadata and annotator documentation, not just dataset size.","Evaluations of Farsi subjective NLP should report per-dataset results, because averages hide large swings across datasets.",""],"supporting_citations":[{"why":"Supplies the language-resource classification that positions Farsi as having a strong web presence but insufficient labeled data, framing the paper's motivating puzzle.","marker":"[Joshi et al., 2020]"},{"why":"Provides the annotation framework the paper adapts to categorize datasets by framework, language, modality, source, and size.","marker":"[Plaza-del Arco et al., 2024]"},{"why":"Prior evaluation showing low GPT performance on the ArmanEmo emotion dataset; motivates extending evaluations to open-source models and more datasets.","marker":"[Abaskohi et al., 2024]"},{"why":"Proposes translating non-English datasets to English to boost multilingual LLM performance; the paper tests and challenges this strategy for Farsi.","marker":"[Etxaniz et al., 2024]"},{"why":"Introduces XLM-RoBERTa, the multilingual encoder model fine-tuned on each of the nine datasets in the benchmark.","marker":"[Conneau et al., 2020]"},{"why":"Provides the Llama-3-8B-Instruct model evaluated in zero-shot subjective-task prompting.","marker":"[Dubey et al., 2024]"},{"why":"Provides the Mixtral-8x7B-Instruct model evaluated in zero-shot subjective-task prompting.","marker":"[Jiang et al., 2024]"},{"why":"Provides the Qwen2-7B-Instruct model evaluated in zero-shot subjective-task prompting.","marker":"[Yang et al., 2024]"},{"why":"Supplies the Phate toxicity dataset and the prompt template used for toxicity detection experiments.","marker":"[Delbari et al., 2024]"}],"fun_headline_variants":["Farsi subjective NLP: 110 papers, only 15 public datasets","LLM scores swing across Farsi sentiment, toxicity datasets","Fine-tuning beats zero-shot on Farsi subjective tasks","Farsi's web text can't fix its subjective-data gap"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The survey's completeness rests on restricting the search largely to ACL Anthology and the top 10 Google Scholar results per keyword; if many public Farsi subjective-task datasets sit outside those channels, the scarcity claim would be overstated, and the evaluation also assumes the nine selected datasets are representative of Farsi subjective tasks, a limitation the authors acknowledge.","fun_headline_variants_meta":{"raw":{"variants":["Farsi subjective NLP: 110 papers, only 15 public datasets","LLM scores swing across Farsi sentiment, toxicity datasets","Fine-tuning beats zero-shot on Farsi subjective tasks","Farsi's web text can't fix its subjective-data gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000818,"raw_usage":{"total_tokens":3398,"prompt_tokens":703,"completion_tokens":2695,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":447,"completion_tokens_details":{"reasoning_tokens":2625}},"tokens_in":447,"tokens_out":2695,"duration_ms":21150,"temperature":1.0,"reasoning_tokens":2625,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T05:06:20.610180+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Search beyond the top-10 Google Scholar results and outside ACL Anthology—local Iranian conferences, institutional repositories, and non-indexed venues—for public Farsi emotion, sentiment, or toxicity datasets; if those searches turn up many additional public datasets with demographic metadata, the scarcity claim weakens. Alternatively, rerun the benchmark on a broader stratified sample of Farsi texts and observe whether LLM F1 scores remain stable across datasets; stability would contradict the paper's 'highly unstable' claim.","supporting_citations":[{"cited_title":"Cercas Curry, Amanda Cercas Curry, and Dirk Hovy","cited_arxiv_id":null,"evidence_quote":"Provides the annotation framework the paper adapts to categorize datasets by framework, language, modality, source, and size."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior evaluation showing low GPT performance on the ArmanEmo emotion dataset; motivates extending evaluations to open-source models and more datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Phate toxicity dataset and the prompt template used for toxicity detection experiments."}],"review_version":1}