{"id":"ef2df649-53dd-4c7c-980e-874e417d04f2","arxiv_id":"2411.09214","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Zero-shot GPT-3.5 Turbo prompting achieves macro-F1 0.756 on HASOC 2024 English hate speech classification, ranking 5th.","lead":"This system paper reports using GPT-3.5 Turbo with a zero-shot prompt to classify English tweets as hateful or not, achieving around 0.75 macro-F1 on the HASOC 2024 test set. It is a shared-task report with a simple method, but it lacks baselines, error bars, and released code.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The robustness/reliability conclusion rests on three single stochastic runs whose 0.005 Macro-F1 spread is within expected sampling noise; no repeated trials, confidence intervals, or per-class metrics are reported.","rationale":"The reader's weakest assumption correctly identifies that three runs without error bars or repeated trials are insufficient to establish the paper's robustness and reliability claim. My stress-test agrees with this assessment. The reported Macro-F1 scores themselves are plausibly official shared-task results and are not demonstrably wrong, so the appropriate verdict remains CONDITIONAL: the central numbers may stand, but the paper's interpretive claims and reproducibility require additional evidence. I considered whether the absence of a baseline comparison or the unspecified GPT-3.5 Turbo model snapshot might be more load-bearing. Missing baselines weaken the 'strong performance' framing, but the shared-task ranking provides an external anchor, so it is less central than the stability claim. The unspecified model snapshot is a reproducibility concern but would not invalidate the reported scores if the runs were made within a short time window. The most load-bearing issue is that the robustness/reliability conclusion is directly derived from a 0.005 spread across three stochastic runs, and that derivation is not statistically justified. The proposed concrete test would settle the concern by measuring within-temperature variance; if that variance is large, the central interpretive claim fails even though the raw scores may be correct. No change to the reader's verdict is needed; CONDITIONAL remains the appropriate outcome.","tokens_in":7516,"tokens_out":4864,"duration_ms":56640,"concrete_test":"On the same 888-instance test set with the same prompt and the same fixed GPT-3.5 Turbo model snapshot, run the classifier at least 10 times at temperature 0.7 and compute Macro-F1 for each run. If the range of Macro-F1 across these identical-configuration runs is comparable to or larger than the 0.005 spread observed across temperatures, then the observed cross-temperature differences do not demonstrate stability. Additionally, compute bootstrap 95% confidence intervals for each of the three original runs; if the intervals overlap substantially, the 'minimal variance' conclusion is not statistically supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, stated in the abstract and Section 7, is that the model is 'robust and reliable' because three runs at temperatures 0.7, 0.8, and 0.9 produced Macro-F1 scores of 0.756, 0.751, and 0.754 (Section 6, Table 1). The load-bearing assumption is that these three runs adequately characterize the stochastic behavior of GPT-3.5 Turbo. This assumption is insecure. Each reported score is a single sample from a stochastic decoding process, and the experimental design varies temperature and run simultaneously, so the observed spread of 0.005 cannot be separated from sampling noise. With only 888 test tweets, a difference of 0.005 in Macro-F1 is plausibly within the sampling error of a binary classifier; a small number of label flips can move the metric by this amount. Without repeated runs at a fixed temperature, bootstrap confidence intervals, or any statistical test, the claim that the model 'consistently performs well' and exhibits 'minimal variance' is underdetermined. The absence of per-class precision and recall also weakens the assertion that the model 'balances precision and recall across all classes.' This is not an accusation that the scores are fabricated; it is a statement that the experimental evidence does not support the robustness/reliability interpretation with the reported data alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes a zero-shot GPT-3.5 Turbo prompt-based system submitted to the English hate speech detection task of HASOC at FIRE 2024. The authors classify 888 English tweets into 'Hate and Offensive' (HOF) and 'Non Hate-Offensive' (NOT). They report three runs at temperatures 0.7, 0.8, and 0.9, with Macro-F1 scores of 0.756, 0.751, and 0.754, respectively, and state that their team ranked 5th in the shared task. The central claim is that the small score spread demonstrates the model's robustness and reliability.","tokens_in":7760,"tokens_out":5485,"duration_ms":54871,"significance":"If the reported performance is reproducible, the result is a useful data point showing that a zero-shot proprietary LLM can be competitive with specialized fine-tuned systems in this shared task. The paper reports the exact prompt and the observed API scores, which supports reproducibility to a degree. However, no code, test data, or statistically grounded evaluation is provided, and the robustness claim is not supported by three single-sample runs. The contribution is primarily a short system description rather than a methodological advance.","major_comments":[{"comment":"The three 'runs' vary temperature simultaneously, so each reported score is a single stochastic sample at a different hyperparameter setting. The 0.005 spread cannot separate sampling noise from temperature effects. The abstract and Section 7 claim 'robustness and reliability' and 'minimal variance' without repeated trials at a fixed temperature, confidence intervals, or a statistical test. This evidence is load-bearing for the paper's main conclusion and is currently insufficient to support it.","section":"Section 6, Table 1"},{"comment":"The paper claims a 'high level of performance' and reports a 5th-place ranking, but it does not provide the official leaderboard scores, the scores of other participating systems, or any baseline such as TF-IDF with a linear classifier or fine-tuned BERT. Without this context, the reader cannot assess whether Macro-F1 of about 0.75 is strong. At minimum, include the shared-task overview table or the range and median of participant scores.","section":"Section 6"},{"comment":"The assertion that the model 'balances precision and recall across all classes' is not supported by the reported results, since only the Macro-F1 aggregate is given. Per-class precision and recall, or a confusion matrix for at least one run, are needed to substantiate this claim.","section":"Abstract and Section 7"}],"minor_comments":[{"comment":"Equations (1) through (5) describe a generic transformer architecture and are not used in the analysis; they should be removed or replaced with a description of the actual prompt/API configuration, including model version, max tokens, and output parsing.","section":"Section 5.1, Eqs. (1)-(5)"},{"comment":"The paper states that the model was run at three temperature values but does not report other decoding parameters or how non-conforming outputs were handled; specifying these details is needed for reproducibility.","section":"Section 5.1"},{"comment":"The dataset description provides only the tweet count; report the label distribution and any preprocessing steps, since Macro-F1 can be sensitive to class imbalance.","section":"Section 3"},{"comment":"The conclusion says the work is 'extending our analysis to other languages,' but no multilingual experiments or results appear in the manuscript; either add them or remove this claim.","section":"Section 7"},{"comment":"Reference [14] uses generic placeholder names ('J. Smith, J. Doe'), and the related-work sentence attributing logistic regression to Badjatiya et al. [17] and Chiu et al. [18] does not match the cited works; verify and correct these citations.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This is a very short system-description paper. The main weakness is statistical, not methodological: the robustness claim rests on three single-sample runs without error bars or baselines. If the authors add repeated trials, confidence intervals, and baseline comparisons, the paper could become acceptable as a workshop-style contribution; for a standard journal the contribution may still be too thin. Reference [14] appears to be a placeholder and should be checked carefully for scholarly integrity."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this is a modest shared-task system description, not a research contribution. The authors ran GPT-3.5 Turbo zero-shot on the FIRE 2024 HASOC English hate speech task, tried three temperatures, got Macro-F1 around 0.75, and ranked 5th. That is the whole paper. It is honest about being a successor to earlier GPT-3 work, and the prompt is simple and clearly stated.\n\nWhat it does well: it gives a clean minimal description of method and scores, cites the relevant prior art (Chiu et al. on GPT-3, the HASOC overview papers), and does not hide that it is a zero-shot baseline. The scores are plausible as API outputs.\n\nThe soft spots are real and not minor. The abstract and conclusion claim robustness and reliability based on three single runs whose Macro-F1 spread is 0.005. That spread is within expected sampling noise, not a stability result. Temperature and run are varied simultaneously, so even the temperature effect is confounded. No repeated runs at a fixed temperature, no confidence intervals, no per-class precision and recall. The absence of baselines also makes it hard to interpret 0.75, though the rank of 5th provides some context. The long transformer tutorial in Section 5 adds nothing and reads like padding. No code or data are released, so the numbers cannot be independently verified.\n\nThe narrow claim that the model scored about 0.75 on this test set is fine; the overclaim is the robustness interpretation. That is fixable with language changes and optionally a few repeated runs.\n\nWho is this for? People tracking shared-task results, or wanting a quick data point on GPT-3.5 Turbo zero-shot for hate speech. It does not advance the field, but it is a legitimate data point. I would send it to peer review in a shared-task or workshop track, with the expectation that the authors tone down the robustness claim and either add error bars or drop the word reliable. A serious editor does not need to desk-reject this; a careful reviewer can handle it in one pass.\n\nOn balance: worth a referee, not worth much more.","headline":"A thin but honest shared-task note: GPT-3.5 Turbo zero-shot scores 0.75 Macro-F1 on English hate speech, with a robustness claim that three runs cannot support.","tokens_in":8279,"tokens_out":2840,"would_cite":false,"duration_ms":26350,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Zero-shot prompting with GPT-3.5 Turbo classifies English tweets as hate/offensive or not with macro-F1 scores of 0.756, 0.751, and 0.754, placing 5th in the English task.","keywords":["hate speech detection","GPT-3.5 Turbo","zero-shot prompting","prompt engineering","Macro-F1","English tweets","offensive language classification","large language models"],"falsifier":"Rerun the same prompt on the same 888 tweets many times per temperature and compute the standard deviation of macro-F1; if the spread is large or the average is no better than a majority-class baseline, the stability claim fails. Alternatively, examine a confusion matrix: if the score is driven by predicting one class almost always, the claim of balanced precision and recall is wrong.","tokens_in":7323,"feed_emoji":"🛡️","tokens_out":8405,"duration_ms":82354,"temperature":0.7,"pith_summary":"This paper argues that a single fixed prompt to a large language model, with no fine-tuning and no feature engineering, can serve as a workable hate speech detector for English tweets. The authors send each tweet to GPT-3.5 Turbo in zero-shot mode with a prompt that asks for one of two labels, and they vary only the temperature parameter across runs. On 888 tweets from the social media platform X, the three runs achieve macro-F1 scores of 0.756, 0.751, and 0.754. The close agreement among the three scores is offered as evidence that the approach is consistent, not a fluke of one sampling setting. The practical point is that a deployable classifier can be obtained by editing a prompt rather than training a model.","feed_headline":"No fine-tuning: GPT-3.5 Turbo hits 0.756 on hate speech","feed_subtitle":"One prompt, three temperature settings, no training data: stable hate speech scores on English tweets.","key_machinery":"The carrying object is the zero-shot prompt itself, one fixed string into which a tweet is inserted, together with GPT-3.5 Turbo's transformer stack that turns the prompt into a label. The paper steps through tokenization, embedding, positional encoding, multi-head attention, feedforward layers, and decoding as the internal machinery that produces the output text. The only experimental lever is temperature, which changes the randomness of the sampled output token; the authors run it at 0.7, 0.8, and 0.9 and then map whatever text the model returns to the two submission labels HOF and NOT.","core_discovery":"On its own terms, the discovery is that zero-shot prompting of GPT-3.5 Turbo is a viable route to English hate speech and offensive language detection. Using the prompt 'Please Check whether the Tweet-<Tweet> is Hate and Offensive or Non Hate-Offensive. Only state Hate and Offensive or Non Hate-Offensive', the model labels each tweet, and the labels are mapped to HOF and NOT. The reported macro-F1 values are 0.756 for temperature 0.7, 0.751 for temperature 0.8, and 0.754 for temperature 0.9, with the submission placing 5th in that task's English track. The near-flat scores across the three temperatures are read by the authors as minimal variance and therefore as reliability of the prompting approach.","pith_inferences":["The paper does not compare its scores against a majority-class baseline, so a reader cannot yet tell how much of the 0.75 macro-F1 reflects real signal; reporting a confusion matrix would resolve this.","If the stability across temperatures holds more broadly, prompt wording is likely a larger source of variance than sampling temperature, so a small grid of alternative phrasings would be a natural next test.","The introduction motivates multilingual and code-mixed cases, but only English results are reported; applying this exact prompt protocol to Hinglish or German-English would test whether the zero-shot advantage extends beyond English."],"forward_implications":["A hate speech detector can be deployed by writing a prompt and calling an API, with no labeled training data, no preprocessing pipeline, and no model training.","Temperature in the 0.7-0.9 range shifts macro-F1 by only about 0.005, so the result is not visibly tied to one sampling configuration.","Changing the task or label set would amount to editing the prompt, which makes rapid iteration cheap for moderators.","At the reported scores, the zero-shot approach is competitive enough to rank 5th in the English track of the shared task, despite using none of the task's training data."],"supporting_citations":[{"why":"Introduces GPT-3 and few-shot prompting, the model family and inference paradigm that GPT-3.5 Turbo and this paper's zero-shot protocol rely on.","marker":"[32]"},{"why":"Earlier work detecting hate speech with GPT-3, the direct predecessor this paper extends to GPT-3.5 Turbo with zero-shot prompting.","marker":"[18]"},{"why":"Overview of the shared task that defines the English hate speech classification setup, the two labels, and the evaluation metric.","marker":"[23]"},{"why":"Companion task overview that supplies the official results context, including the ranking in which the submission placed 5th.","marker":"[24]"},{"why":"BERT transfer learning baseline for hate speech detection that sets the transformer comparison point for the zero-shot approach.","marker":"[16]"}],"fun_headline_variants":["Zero-shot GPT-3.5 Turbo tames hate speech with 0.756 F1","Prompt-only hate speech detection: GPT-3.5 Turbo ranks 5th","No training data, no problem: GPT-3.5 Turbo on hate speech","Stable F1 across three temps: GPT-3.5 Turbo for hate speech"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that three runs with different temperatures are enough to establish that the model is stable and reliable; the paper offers no error bars, confidence intervals, or baseline comparison to support that reading.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot GPT-3.5 Turbo tames hate speech with 0.756 F1","Prompt-only hate speech detection: GPT-3.5 Turbo ranks 5th","No training data, no problem: GPT-3.5 Turbo on hate speech","Stable F1 across three temps: GPT-3.5 Turbo for hate speech"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000665,"raw_usage":{"total_tokens":3061,"prompt_tokens":999,"completion_tokens":2062,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":1971}},"tokens_in":615,"tokens_out":2062,"duration_ms":15832,"temperature":1.0,"reasoning_tokens":1971,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:53:31.665152+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the same prompt on the same 888 tweets many times per temperature and compute the standard deviation of macro-F1; if the spread is large or the average is no better than a majority-class baseline, the stability claim fails. Alternatively, examine a confusion matrix: if the score is driven by predicting one class almost always, the claim of balanced precision and recall is wrong.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces GPT-3 and few-shot prompting, the model family and inference paradigm that GPT-3.5 Turbo and this paper's zero-shot protocol rely on."},{"cited_title":"Ghosh, N","cited_arxiv_id":null,"evidence_quote":"Overview of the shared task that defines the English hate speech classification setup, the two labels, and the evaluation metric."},{"cited_title":"Raihan, K","cited_arxiv_id":null,"evidence_quote":"Companion task overview that supplies the official results context, including the ranking in which the submission placed 5th."},{"cited_title":"Mozafari, R","cited_arxiv_id":null,"evidence_quote":"BERT transfer learning baseline for hate speech detection that sets the transformer comparison point for the zero-shot approach."}],"review_version":1}