{"id":"8ad404a9-99b7-4dcf-8b31-1d81d891ed32","arxiv_id":"2412.00425","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":1.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A literature review that catalogs sarcasm detection datasets, word-embedding strategies, and neural models, but adds no new experimental results.","lead":"This paper surveys sarcasm detection research, summarizing datasets, embedding methods, and modeling approaches from prior published work. It might interest generalists as a quick map of a difficult NLP problem that still confuses current systems.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The survey's central claim to be a reliable map of sarcasm detection is undercut by irreconcilable performance figures for KnowleNet: Section 5.6 reports 88.87/86.33 while Table 4 reports 92.69/91.21. Until these numbers are traced to the source, the comparative tables cannot be trusted.","rationale":"Reader's weakest assumption is exactly the load-bearing one: the survey is only as good as the figures and attributions it inherits. The KnowleNet internal contradiction and the SarcPrompt reference error are not peripheral; both sit in the comparative artifact (Tables 3 and 4) that constitutes the paper's contribution. They do not, however, destroy the paper: many other rows and the prose summaries can be checked independently, and the errors are correctable. The absence of original experiments is normal for a survey and is not itself a defect. A targeted re-extraction from the cited sources would settle whether these are isolated typos or systematic transcription failure; the current evidence warrants keeping the paper conditional pending that check. No objection to the reader's verdict is raised.","tokens_in":9629,"tokens_out":2737,"duration_ms":25673,"concrete_test":"Pull the published KnowleNet paper (Yue et al., Information Fusion 100, 101921) and the original Cai et al. dataset paper, and re-extract the headline Accuracy/F1 for KnowleNet and CMGCN from their result tables. Then check which value (88.87/86.33 in §5.6 or 92.69/91.21 in Table 4) matches the source and whether either refers to a different test split or metric definition. Independently locate the SarcPrompt paper (e.g., by searching the prompt-tuning template 'Actually [MASK]') and confirm the reference numbered [16] should point to it. If the source values fix one of the two figures and the bibliographic entry is corrected, the CONDITIONAL verdict stands; if the source contains yet a third value, the tables need a full audit.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Since this paper contributes no new experiments, its value rests entirely on accurate transcription and attribution of reported results. That assumption is demonstrably unsafe. In Section 5.6, KnowleNet is said to achieve Accuracy 88.87% and F1 86.33% on the Cai et al. multimodal dataset, with CMGCN at 87.23/83.45; the same approaches in Table 4 are listed as 92.69/91.21 and 87.55/84.16, respectively. No experimental setup explains the gap. A second attribution failure: Section 5.4 introduces SarcPrompt by citing reference [16], but [16] is the KnowleNet paper by Yue et al., and Section 5.6 also cites [16] for KnowleNet. The SarcPrompt paper is not in the bibliography, so the survey cannot be used to locate the method it describes. If the tables contain unverifiable metrics and the references do not uniquely identify methods, the central 'map of the field' function fails in exactly the places a reader would rely on it. These are not cosmetic typos: they corrupt the quantitative comparisons that are the survey's main deliverable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript is a literature survey of sarcasm detection in NLP. It reviews linguistic/context-based approaches, word embeddings, multimodal methods, and graph-based networks; describes datasets including MUStARD, SARC, the Cai et al. multimodal dataset, FLUTE, and WITS; and provides two summary tables (Table 3 for datasets, Table 4 for benchmarked approaches). It closes with future directions such as LLM-based detection, multilingual extension, synthetic data, and metaphor-aware methods. The survey's contribution is organizational rather than empirical.","tokens_in":9904,"tokens_out":8046,"duration_ms":70677,"significance":"If its reported figures and attributions were accurate, the survey would be a convenient entry point to the field. Its useful features include concise descriptions of representative methods, reproduction of key quantitative results, and a compact dataset table. However, the paper's value rests entirely on faithful transcription from primary sources, and the discrepancies detailed below compromise exactly that deliverable. The absence of a documented survey methodology also limits the paper's reproducibility as a scholarly map of the area.","major_comments":[{"comment":"Section 5.6 reports KnowleNet accuracy 88.87% and F1 86.33% on the Cai et al. dataset, with CMGCN at 87.23%/83.45%, while Table 4 lists the same two systems as 92.69%/91.21% and 87.55%/84.16%. No experimental setup, split, or citation is given that would reconcile these numbers. Because the comparative tables are the survey's core deliverable, the authors must trace both sets of values to the original papers and correct one of them.","section":"Section 5.6 and Table 4"},{"comment":"The paragraph beginning '[16] introduces SarcPrompt' attributes SarcPrompt to reference [16], but [16] is Yue et al.'s KnowleNet paper, which is also correctly cited in Section 5.6 for KnowleNet. The actual SarcPrompt source is not in the bibliography, so the description cannot be located by a reader. Relatedly, Table 2 lists 'Ili et al. 0.79' with no matching reference. The authors need to fix these attribution and citation failures throughout the paper.","section":"Section 5.4 and References"},{"comment":"The SARC entry states 'The total size of the corpus is 533 Million' without specifying a unit (comments, tokens, or bytes), and this figure is inconsistent with the original SARC paper, which describes 1.3 million sarcastic comments from a larger Reddit corpus. Please verify the number against Khodak et al. and state the unit explicitly; if the intended figure is 55 million comments, Table 3 must be corrected.","section":"Section 3.2.2 and Table 3"},{"comment":"The survey calls Table 3 and Table 4 'complete lists' of datasets and approaches, but Table 3 contains only five datasets and omits several datasets cited elsewhere in the paper (e.g., the Twitter data of Ptácek et al. [24] and Riloff et al. [26] used in experiments described in Sections 5.2 and 5.4). The word 'complete' should either be removed or the tables expanded to include all datasets and methods mentioned.","section":"Section 3.1 and Section 5 preamble"},{"comment":"The paper provides no search strategy, inclusion/exclusion criteria, coverage dates, or statement of how the sixteen papers were selected. Since the survey's stated goal is to map the field, the absence of a documented methodology prevents a reader from judging whether the selection is representative or exhaustive. A short 'Survey Methodology' paragraph should be added.","section":"Overall (Sections 1 and 7)"}],"minor_comments":[{"comment":"The author name is spelled 'Agarwal' in the text but 'Agrawal' in the reference list; standardize to the author's actual spelling.","section":"Section 2.2"},{"comment":"The sentence 'and The paper elaborates...' contains an incorrect capital letter and a missing connector; rephrase.","section":"Section 5.4"},{"comment":"Contractions are used as possessives: 'it's apparent sarcasm' and 'it's multi-modal data' should be 'its apparent sarcasm' and 'its multi-modal data'.","section":"Sections 3.1 and 5.5"},{"comment":"The KnowleNet method description reads 'Graph-based semantic network semantic.'; the stray period and truncated phrase should be cleaned up.","section":"Table 4"},{"comment":"The news headlines dataset is dated 2018 in Table 4 but 2022 in reference [23]; align the year.","section":"References and Table 4"},{"comment":"The table does not identify what 'Proposed' refers to; label it as 'RoBERTa-RCNN' as done in the text.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"This submission reads as an early arXiv preprint rather than a polished survey. The repeated citation and transcription errors, especially the KnowleNet metrics and SarcPrompt reference, make it unsuitable for publication in its current form, though they are fixable within the manuscript's scope. I would encourage the journal to require the authors to verify every entry in Tables 2-4 against the source papers, to add a survey methodology statement, and to run a consistency check on all in-text citation numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a conventional literature survey with no new experiments, so judge it as a map, not a result. On that standard it is mostly readable and covers the expected ground: the dataset table and the approach table give a newcomer a starting point, and the summaries of Joshi et al., Bamman et al., and KnowleNet are accurate enough at the level of general description. The paper earns credit for assembling this material in one place; that is its actual value.\n\nThe soft spots are exactly where a map has to be reliable, and they are not cosmetic. KnowleNet's accuracy/F1 in Section 5.6 is 88.87/86.33, while Table 4 lists 92.69/91.21 for the same model on the same dataset. No setup detail explains the gap, so a reader cannot tell which number comes from the original paper. Section 5.4 introduces SarcPrompt by citing reference [16], which is actually the KnowleNet paper; the SarcPrompt paper is not in the bibliography. That breaks the attribution chain for anyone trying to locate the method. There are smaller errors too: the author name is given as 'Agarwal' in one place and 'Agrawal' in the reference, and the character-bias observation in the MUStARD section reproduces the original paper's analysis without flagging it as such. None of these kill the survey's usefulness as a bibliography, but together they mean the two tables cannot be trusted until every figure is rechecked against its source.\n\nThe paper is what it claims to be: a survey. It introduces no new data, method, or derivation, so its significance is a 4 at best. But a correct survey of this kind has a place, and a demanding referee could get it into that shape. The inconsistencies are concrete and fixable, and the scope is appropriate. I would not desk-reject it; I would send it back with a requirement that the authors trace each reported metric to its source, fix the SarcPrompt citation, and flag any analysis that is quoted from the original dataset papers.\n\nWho gets value? A graduate student looking for an entry point into sarcasm detection will find this useful after the corrections. A specialist in the area will not learn much, but can use it as a quick reference. I would not cite it in my own work in the next year; I would cite the originals.\n\nRecommendation: engage with it, but only with the corrections enforced.","headline":"A useful but sloppy survey of sarcasm detection; the tables need verification before the paper can be relied on as a map of the field.","tokens_in":10406,"tokens_out":2170,"would_cite":false,"duration_ms":21357,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This survey compiles the datasets, word-embedding schemes, and neural architectures that define automatic sarcasm detection, and argues that they are best understood as attempts to model incongruity at multiple levels.","keywords":["sarcasm detection","natural language processing","literature survey","context incongruity","multimodal sarcasm","word embeddings","deep learning","datasets"],"falsifier":"Re-read the cited papers and compare their reported metrics to Tables 3 and 4: the survey's central claim is falsified by any mismatch, such as KnowleNet's accuracy appearing as 88.87% in the text versus 92.69% in Table 4, or the SarcPrompt passage citing reference [16] (KnowleNet) instead of a prompt-tuning paper.","tokens_in":9455,"feed_emoji":"🙄","tokens_out":8580,"duration_ms":76649,"temperature":0.7,"pith_summary":"Sarcasm is hard even for humans, with an average accuracy of 81.6%, and this paper surveys how automatic systems have tried to close that gap. It organizes the field into four strands: linguistic and context-based methods, word-embedding schemes, multimodal approaches, and graph-based networks, and it catalogues the principal datasets (SARC from Reddit, MUStARD from TV shows, a multimodal Twitter corpus, and others) with their reported performance. The survey's aim is to give a reader a reliable map of datasets and methods so that future work can build on the current state of the art. If its transcriptions are accurate, the tables provide a convenient comparative reference for the field.","feed_headline":"Survey maps the landscape of automatic sarcasm detection","feed_subtitle":"One place to find the key datasets, the leading models, and the open problems in teaching machines to read irony.","key_machinery":"The load-bearing objects are the two comparative tables. Table 3 catalogues sarcasm datasets with their source, modality, size, and the metric reported in the original paper; Table 4 catalogues detection approaches with their model, training data, method, and reported accuracy and F1. These tables carry the survey's argument that the field can be surveyed at a glance, and they are also where a reader can check the survey's reliability by comparing the transcribed numbers against the source papers.","core_discovery":"The survey's organizing claim is that sarcasm detection is best understood as the detection of incongruity—between words of opposite polarity, between an utterance and its situational context, and between modalities such as text and image. It asserts that the datasets and approaches catalogued in Tables 3 and 4 represent the current landscape of the field, from the SARC Reddit corpus and MUStARD TV-show dataset, through topic-enriched and affective word embeddings, to attention-based and multimodal knowledge-fusion models. The paper contributes no new experiments; its contribution is the organization, description, and comparison of existing reported results, and it identifies future directions including fine-tuned large language models, GPT-Vision, multilingual data, and synthetic sarcasm generation.","pith_inferences":["Beyond the survey's own claims, a fair comparison of the systems in Table 4 would require re-running them under a shared protocol, since the survey reprints scores from papers with different splits and preprocessing.","The MUStARD character-bias result implies that other small-cast multimodal datasets should use speaker-independent splits to avoid rewarding speaker recognition rather than sarcasm detection.","If the cited 81.6% human accuracy is a real ceiling, future systems should be evaluated against human agreement, not merely against the previous state of the art.","A direct test of the survey's future-work suggestions would be to generate synthetic sarcastic examples with a modern LLM and measure whether adding them to SARC or MUStARD improves out-of-domain generalization."],"forward_implications":["Text-only sarcasm detection gains about 8% in F-score when context-incongruity features are added to lexical and pragmatic features, so incongruity is a tractable and effective signal.","For Twitter sarcasm, adding author historical information raises accuracy from 75.4% with tweet-only features to 84.9%, and all feature groups together reach 85.1%, so who is speaking matters almost as much as what is said.","Multimodal models on MUStARD cut error rate by 12.9% in F-score over single-modality models, but the gains shrink under speaker-independent splits, revealing that part of the apparent advantage is character bias.","Topic-enriched (LDA2Vec) and affective word embeddings outperform standard word2vec, fastText, and GloVe embeddings for sarcasm detection; sentiment embeddings work best on short texts and emotion embeddings on longer ones.","On the Cai et al. multimodal Twitter dataset, graph-based and knowledge-fusion models (cross-modal GCN, KnowleNet) report the highest accuracies in the survey, above 87%, indicating that modeling inter-modal incongruity is the current frontier."],"supporting_citations":[{"why":"Supplies the context-incongruity feature set and the reported 8% F-score improvement over lexical and pragmatic features.","marker":"[3]"},{"why":"Provides the SARC self-annotated Reddit corpus used for text-based sarcasm benchmarks and the human accuracy baseline.","marker":"[4]"},{"why":"Establishes contextualized sarcasm detection with author, audience, and response features, including the accuracy progression from 75.4% to 85.1%.","marker":"[6]"},{"why":"Contributes the MUStARD multimodal TV-show dataset and the 12.9% F-score error reduction claim for multimodal over unimodal models.","marker":"[9]"},{"why":"Defines intra- and inter-modality incongruity modeling with BERT on Twitter image-text data.","marker":"[10]"},{"why":"Introduces the cross-modal graph convolutional network for multimodal sarcasm detection, reported as a top performer in Table 4.","marker":"[12]"},{"why":"Describes KnowleNet, the knowledge-fusion multimodal model that the survey reports as state of the art, along with its ConceptNet-based knowledge fusion.","marker":"[16]"},{"why":"Provides the multimodal Twitter sarcasm dataset with image attributes used by KnowleNet, CMGCN, and the hierarchical fusion baseline.","marker":"[18]"}],"fun_headline_variants":["Sarcasm detection survey: from datasets to multimodal models","Teaching machines to read irony: a field survey","Incongruity as key: survey of sarcasm detection","Sarcasm AI landscape: methods, data, and future paths"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The survey's entire map rests on the assumption that the accuracy and F1 scores it copied from the original papers are correct, comparable, and assigned to the right methods; it re-runs no experiments to check them.","fun_headline_variants_meta":{"raw":{"variants":["Sarcasm detection survey: from datasets to multimodal models","Teaching machines to read irony: a field survey","Incongruity as key: survey of sarcasm detection","Sarcasm AI landscape: methods, data, and future paths"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1180,"prompt_tokens":754,"completion_tokens":426,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":370,"completion_tokens_details":{"reasoning_tokens":359}},"tokens_in":370,"tokens_out":426,"duration_ms":4494,"temperature":1.0,"reasoning_tokens":359,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:22:54.832203+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-read the cited papers and compare their reported metrics to Tables 3 and 4: the survey's central claim is falsified by any mismatch, such as KnowleNet's accuracy appearing as 88.87% in the text versus 92.69% in Table 4, or the SarcPrompt passage citing reference [16] (KnowleNet) instead of a prompt-tuning paper.","supporting_citations":[{"cited_title":"Modeling Intra and Inter-modality Incongruity for Multi-Modal Sarcasm Detection","cited_arxiv_id":null,"evidence_quote":"Defines intra- and inter-modality incongruity modeling with BERT on Twitter image-text data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the context-incongruity feature set and the reported 8% F-score improvement over lexical and pragmatic features."},{"cited_title":"A Large Self-Annotated Corpus for Sarcasm","cited_arxiv_id":"1704.05579","evidence_quote":"Provides the SARC self-annotated Reddit corpus used for text-based sarcasm benchmarks and the human accuracy baseline."},{"cited_title":"”Contextualized sarcasm detection on twitter.” Proceedings of the International AAAI Conference on Web and Social Media.Vol","cited_arxiv_id":null,"evidence_quote":"Establishes contextualized sarcasm detection with author, audience, and response features, including the accuracy progression from 75.4% to 85.1%."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the cross-modal graph convolutional network for multimodal sarcasm detection, reported as a top performer in Table 4."},{"cited_title":"”KnowleNet: Knowledge fusion network for multimodal sarcasm detection.” Information Fusion 100 (2023): 101921","cited_arxiv_id":null,"evidence_quote":"Describes KnowleNet, the knowledge-fusion multimodal model that the survey reports as state of the art, along with its ConceptNet-based knowledge fusion."},{"cited_title":"”Multi-modal sarcasm detection in twitter with hierar- chical fusion model.” Proceedings of the 57th annual meeting of the association for computational linguistics","cited_arxiv_id":null,"evidence_quote":"Provides the multimodal Twitter sarcasm dataset with image attributes used by KnowleNet, CMGCN, and the hierarchical fusion baseline."}],"review_version":1}