Pith. sign in

REVIEW 3 major objections 5 minor 38 references

Controlling Out-of-Domain Gaps in LLMs for Genre Classification and Generated Text Detection

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read This paper claims that modern large language models still show an out-of-domain accuracy gap in few-shot in-context learning, and that explicitly instructing the model to ignore topic and use stylistic indicators reduces that gap by up to…

desk verdict Useful, honestly presented demonstration that style-control prompts shrink OOD gaps in few-shot ICL; the headline 20-point gain is probably inflated by prompt tuning on the same test set. read the letter →

arxiv 2412.20595 v1 pith:HVNPLN6V submitted 2024-12-29 cs.CL cs.AI

classification cs.CLcs.AI
keywords out-of-domaingapin-contextlearningfew-shotgenreclassificationgeneratedtextdetectionpromptcontroldomaintransferlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the out-of-domain (OOD) gap in text classification, known from smaller BERT-era models, still affects modern large language models used with in-context learning: when the five demonstration examples come from one topic (say, sport) and the test texts come from another (say, history), accuracy drops. It then shows a fix: give the model a detailed list of style- and structure-based indicators to use, tell it to ignore topical content and text length, and require that its criteria be applicable to any topic. On genre classification, this reduces the OOD gap by up to 7 percentage points; on generated-text detection, by up to 20 percentage points, and in two of four models the gap disappears. The practical stakes are that labeled examples from one domain can be reused in another without fine-tuning, saving annotation effort. The claims are made for two tasks, six model variants, and a 5-shot setting.

What carries the argument

The mechanism that carries the argument is prompt control over which predictive indicators the model may use. The detailed-control prompt lists concrete stylistic and structural features to attend to, such as formality, tone, sentence length, purpose, person perspective, voice, citations, and personal experience, while explicitly prohibiting topical content and text length and insisting that criteria must apply to any topic. The test apparatus is a topic-model split of a large natural-genre corpus into on-topic and off-topic documents, which creates measurable OOD transfer scenarios. Chain-of-Thought prompting, defined here as a two-stage instruction asking the model to articulate criteria before classifying, serves as the baseline that this controlled prompting is compared against.

What would settle it

An experiment that would settle it: rerun the 5-shot comparison with the same topics but 50 test texts per topic, several genre pairs per topic, and synthetic texts from several different generators (not only Claude 3 Sonnet); if the detailed-control advantage does not persist, the central claim is an artifact of the original small sample.

Watch

Extended reading notes

Core claim

The central claim is that modern LLMs reproduce the OOD gap observed in earlier PLMs, and that the gap can be controlled by prompt design rather than by model retraining. The authors test two non-topical tasks—genre classification and generated-text detection—using a corpus with topic labels from a topic model, and compare a basic chain-of-thought prompt with prompts that add simple or detailed control over which features the model may use. The detailed-control prompt, which names stylistic criteria (formality, tone, sentence structure, purpose, perspective, voice, citations, personal experience) and explicitly forbids topic and length, is the best configuration for the more capable models. It reduces the OOD gap by up to 7 percentage points in genre classification and up to 20 percentage points in generated-text detection, and eliminates the detection gap entirely for two of the four models tested. Ablations show the effect depends on the detailed instruction rather than on generic chain-of-thought: removing the style-feature list cuts the gain by more than half, while paraphrasing the prompt leaves the trend stable.

Load-bearing premise

The results rest on a small benchmark: 10 test texts per topic, one randomly chosen genre pair per topic, and topic-model scores that decide which texts count as on-topic or off-topic; if those scores are noisy or the 10-text samples are not representative, the measured OOD gaps and the 7-20 point gains could be miscalibrated.

Editorial extensions

If this is right

  • The method offers a recipe for cross-topic use of few-shot examples: a task-specific list of permitted indicators plus an explicit ban on topical content.
  • More capable models (GPT-4o, Claude 3 Opus, Claude 3.5 Sonnet) gain more from detailed control than older or smaller models, suggesting prompt control and model scale interact.
  • The same control approach should transfer to other non-topical classification tasks, such as sentiment, authorship, or gender classification, because the methodology is task-independent.
  • The OOD gap remains non-zero in most conditions, so prompt control reduces rather than eliminates the cost of domain shift.
  • Ablated and rephrased prompts confirm that the specificity of the instructions, not the exact wording, is what drives the improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's numeric gap estimates rest on only 10 test texts per topic and a single random genre pair per topic, so the 7-20 point figures should be read as approximate; a larger test sample would give a sharper estimate.
  • Because the AI-generated side of the detection data was written by a single generator (Claude 3 Sonnet), the 20-point gain could partly reflect that generator's stylistic fingerprint; testing other generators would show whether the control prompt transfers.
  • The paper does not test languages other than English; since genre conventions and topic cues differ across languages, the benefit of topical suppression could be stronger or weaker in translation.
  • The contrast between simple and detailed control suggests a testable hypothesis: models with weaker instruction-following may need more explicit examples of what 'topic' means before they can ignore it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper investigates out-of-domain (OOD) performance gaps in few-shot in-context learning (ICL) for two non-topical classification tasks: genre classification and generated-text detection. Using six LLMs from the GPT and Claude families, the authors first confirm that accuracy drops when demonstration examples are drawn from a different topic than the test texts. They then propose a 'Detailed Control' prompt that instructs the model to focus on stylistic and structural indicators and to ignore topical content. The main reported result is that this prompt reduces the OOD gap by up to 20 percentage points in the generated-text detection task and by up to 7 points in genre classification, with ablation and paraphrase-robustness checks supporting the contribution.

Significance. If the effect is reliable, the paper offers a practical, fine-tuning-free intervention for improving domain transfer in few-shot ICL for non-topical classification. The study covers a useful range of models and tasks, provides a replication repository, and includes ablation studies and prompt-rephrasing robustness checks, which are strengths. The main finding, however, rests on a single headline cell in Table 1 and on prompt development that is not protected against selection on the evaluation set. The paper would be a valuable empirical contribution after the statistical reliability and prompt-selection concerns are addressed.

major comments (3)
  1. [Section 2.6, Table 1, Table 2] The method's headline effect size is vulnerable to prompt-selection bias. The authors state in Section 2.6 that the prompts 'were developed based on our preliminary experiments,' and all evaluations, ablations, and rephrasing checks in Tables 1 and 2 use the same 25-topic, 10-test-text benchmark. Because the Detailed Control prompt contains many free components (formality, structure, purpose, perspective, voice, citations, explicit topic prohibitions), tuning it on this same evaluation set can inflate the reported 7–20 point gap reductions. The rephrasing experiments in Table 2 vary only the surface form of an already-tuned prompt and therefore do not address this selection. Please provide a held-out evaluation (e.g., prompts fixed before any test topics are examined, or a separate development set) or report the prompt development history and show results for a pre-registered prompt.
  2. [Section 2.5, Table 1] The statistical evidence for the central claim is underreported. Each cell in Table 1 is based on 10 test texts per topic (250 classifications per condition), but the paper gives only point accuracies, with no confidence intervals, per-topic variance, or significance tests for the detailed-control versus baseline differences. The 'up to 20 percentage points' claim is a single cell (GPT-4o, generated-text detection), and its uncertainty is not quantified. Report bootstrap or per-topic confidence intervals and significance tests for the main comparisons, not only for the OOD gap itself.
  3. [Section 2.4, Table 1] The generated-text detection task has two design features that threaten the validity of the claimed effect. First, the synthetic texts were generated by Claude 3 Sonnet, and Claude 3 Sonnet is also one of the evaluated detectors, so generator–detector overlap may inflate detection accuracy for that model and potentially for models from the same family. Second, three of six genres are excluded because they were 'trivially distinguishable'; this changes the task distribution and limits generalization to the full genre set. Please discuss these threats and, if possible, test whether the gap reduction persists when the generator and detector are from different families and when the excluded genres are included with a more difficult generation protocol.
minor comments (5)
  1. [Section 2.6.1] The baselines are asymmetric: the genre-classification baseline uses a two-stage Chain-of-Thought prompt, while the generated-text-detection baseline is single-stage without CoT. The authors should justify this asymmetry or use parallel baselines for both tasks.
  2. [Table 2] The column headers in Table 2 are difficult to parse; the first row of numbers appears to lack clear alignment with the column descriptions. Please reformat the table or add a caption that explains each column (including the three rephrased runs for each prompt type).
  3. [Section 1 and Section 4.2] The abstract and introduction refer to 'GPT-4', while the experiments use 'GPT-4o'; Section 4.2 mentions 'GPT-4.5'. Please standardize the model naming throughout.
  4. [Section 3.3] The text says the approach 'consistently enhances domain transfer performance,' but in Table 1, Claude 3.5 Haiku's genre classification with simple control (69.6) is below its baseline (70.8), and GPT-3.5's generated-text detection with detailed control (66.0) is essentially unchanged from the baseline (64.8). Please qualify the consistency claim or add significance tests to show these differences are within noise.
  5. [Figure 2] The y-axis starts at 59.2 rather than 0, which visually exaggerates the difference between baseline and detailed control. Please use a full-scale axis or add error bars.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported OOD-gap reductions are new empirical measurements, not derivations from the reused benchmark or from self-citations.

full rationale

The paper's central claims are empirical measurements reported in Tables 1 and 2, not derivations from definitions or from prior results. The reuse of Roussinov and Sharoff (2023) supplies the corpus, topic model, and evaluation protocol, but the OOD gap in modern LLMs and the reductions from prompt control are newly measured on those data. No equation or construction in the paper reduces a reported improvement to a fitted parameter or to the benchmark definition: the on-topic/off-topic split is an experimental condition, and the accuracy values are model outputs, not identities. The statement in Section 2.6 that the prompts 'were developed based on our preliminary experiments' raises a legitimate concern about selection on the evaluation set, and Section 2.5 similarly reports that five-shot was chosen 'after our preliminary investigations'; however, this is a threat to external validity and statistical confidence, not circular reasoning, because the paper does not present the resulting accuracy as an independent prediction and does not fit a parameter that is then renamed as the outcome. Self-citations are used as sources of data and methodology and as related work, not as the sole justification for the headline effect. The central empirical comparison is self-contained against the reported benchmark, so no specific circular step can be quoted and no circularity score above zero is warranted.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No mathematical derivation exists; the central claim is an empirical comparison. The unpaid-for inputs are the prior corpus and topic model, the hand-selected prompt, the synthetic generated text data, and the small test design.

free parameters (3)
  • number of demonstration examples (shots) = 5
    Selected as the best compromise between prompt size and performance in preliminary investigations (Section 2.5); the headline results are specific to this hand-picked setting.
  • Detailed Control prompt contents = hand-crafted style/topic instruction lists
    The prompt was developed from preliminary experiments and described as empirically found to be the best (Sections 2.6 and 3.3). The wording is part of the treatment, and no separate prompt-development holdout is reported.
  • test set size per topic = 10 texts (5 per class)
    Determined by the smallest dataset in the inherited corpus (Section 2.5); with one random genre pair per topic, accuracy estimates have high variance.
assumptions (4)
  • domain assumption Natural genre annotations and topic-model scores from Roussinov and Sharoff (2023) define valid on-topic/off-topic splits.
    The benchmark is inherited from the authors' prior paper; no independent validation of the topic/genre separation is provided (Sections 2.2 and 2.3).
  • domain assumption Binary pairwise classification with 10 test texts per topic gives reliable accuracy estimates.
    No confidence intervals or repeated sampling are reported; a single random genre pair per topic may not represent the classification difficulty (Section 2.5).
  • ad hoc to paper Claude 3 Sonnet can generate texts on the same topic and in the same style such that human/LLM detection is nontrivial and comparable across genres.
    The synthetic generated-text data is produced by one commercial model, and three genres are excluded because generation was trivially distinguishable (Section 2.4).
  • domain assumption LLM API behavior during the experiments is stable enough for cross-prompt comparisons.
    Black-box commercial APIs are acknowledged as a limitation in Section 6; temporal drift could change the measured gaps.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Controlling Out-of-Domain Gaps in LLMs for Genre Classification and Generated Text Detection." pith.science (2026). https://pith.science/paper/HVNPLN6V

@misc{pith2026241220595,
  author       = {Pith},
  title        = {Pith review of: Controlling Out-of-Domain Gaps in LLMs for Genre Classification and Generated Text Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HVNPLN6V}},
  note         = {Machine review of arXiv:2412.20595}
}
read the original abstract

This study demonstrates that the modern generation of Large Language Models (LLMs, such as GPT-4) suffers from the same out-of-domain (OOD) performance gap observed in prior research on pre-trained Language Models (PLMs, such as BERT). We demonstrate this across two non-topical classification tasks: 1) genre classification and 2) generated text detection. Our results show that when demonstration examples for In-Context Learning (ICL) come from one domain (e.g., travel) and the system is tested on another domain (e.g., history), classification performance declines significantly. To address this, we introduce a method that controls which predictive indicators are used and which are excluded during classification. For the two tasks studied here, this ensures that topical features are omitted, while the model is guided to focus on stylistic rather than content-based attributes. This approach reduces the OOD gap by up to 20 percentage points in a few-shot setup. Straightforward Chain-of-Thought (CoT) methods, used as the baseline, prove insufficient, while our approach consistently enhances domain transfer performance.

Figures

Figures reproduced from arXiv: 2412.20595 by the authors.

Figure 1
Figure 1. Domain transfer assessment methodology adapted from ( [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Accuracy comparison between GPT-4o base [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 25 canonical work pages

  1. [1]

    Shriyansh Agrawal, Lalit Mohan Sanagavarapu, and Y Raghu Reddy. 2019. Fact-fine grained assessment of web page credibility. In TENCON 2019-2019 IEEE Region 10 Conference (TENCON), pages 1088--1097. IEEE

  2. [2]

    Erin L Allwein, Robert E Schapire, and Yoram Singer. 2000. Reducing multiclass to binary: A unifying approach for margin classifiers. Journal of machine learning research, 1(Dec):113--141

  3. [3]

    Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The WaCky wide web: a collection of very large linguistically processed web-crawled corpora. Language Resources and Evaluation, 43(3):209--226

  4. [4]

    Blei, Andrew Y

    David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet allocation. Journal of Machine Learning Research, 3:993--1022

  5. [5]

    John Blitzer, Mark Dredze, and Fernando Pereira. 2007. http://www.aclweb.org/anthology/P07-1056 Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification . In Proc ACL, pages 440--447, Prague, Czech Republic. Association for Computational Linguistics

  6. [6]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

  7. [7]

    S \'e bastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. arxiv. arXiv preprint arXiv:2303.12712

  8. [8]

    Christopher Cieri and Mark Liberman. 2002. Language resources creation and distribution at the Linguistic Data Consortium . In Proc LREC, pages 1327--1333. Las Palmas, Spain

Show all 38 references
  1. [9]

    Kevin Crowston, Barbara Kwasnik, and Joseph Rubleske. 2010. Problems in the use-centered development of a taxonomy of web genres. In Alexander Mehler, Serge Sharoff, and Marina Santini, editors, Genres on the Web: Computational Models and Empirical Studies. Springer

  2. [10]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

  3. [11]

    Dieng, Francisco J

    Adji B. Dieng, Francisco J. R. Ruiz, and David M. Blei. 2020. https://doi.org/10.1162/tacl_a_00325 Topic modeling in embedding spaces . Transactions of the Association for Computational Linguistics, 8:439--453

  4. [12]

    Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: Long papers), pages ...

  5. [13]

    Aleksandra Edwards and Jose Camacho-Collados. 2024. https://aclanthology.org/2024.lrec-main.879 Language models for text classification: Is in-context learning enough? In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources an...

  6. [14]

    Luciano Floridi and Massimo Chiriatti. 2020. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30(4):681--694

  7. [15]

    Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. 2019. Gltr: Statistical detection and visualization of generated text. arXiv preprint arXiv:1906.04043

  8. [16]

    Andrew Gordon and Reid Swanson. 2009. https://www.aaai.org/Papers/Workshops/2009/WS-09-01/WS09-01-003.pdf Identifying personal stories in millions of weblog entries . In Proceedings of International Conference on Weblogs and Social Media, San Jose, CA

  9. [17]

    Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019. https://doi.org/10.18653/v1/S19-2145 S em E val-2019 task 4: Hyperpartisan news detection . In Proceedings of the 13th International Workshop on...

  10. [18]

    John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. 2023 a . A watermark for large language models. In International Conference on Machine Learning, pages 17061--17084. PMLR

  11. [19]

    John Kirchenbauer, Jonas Geiping, Yuxin Wen, Manli Shu, Khalid Saifullah, Kezhi Kong, Kasun Fernando, Aniruddha Saha, Micah Goldblum, and Tom Goldstein. 2023 b . On the reliability of watermarks for large language models. arXiv preprint arXiv:2306.04634

  12. [20]

    Taja Kuzman and Nikola Ljube s i \'c . 2023. Automatic genre identification: a survey. Language Resources and Evaluation, pages 1--34

  13. [21]

    Taja Kuzman, Igor Mozeti c , and Nikola Ljube s i \'c . 2023. Automatic genre identification for robust enrichment of massive text collections: Investigation of classification methods in the era of large language models. Machine Learning and Knowledge Extraction, 5(3):1149--1175

  14. [22]

    Taja Kuzman, Peter Rupnik, and Nikola Ljube s i \'c . 2022. https://aclanthology.org/2022.lrec-1.170 The GINCO training dataset for web genre identification of documents out in the wild . In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1584...

  15. [23]

    Mikhail Lepekhin and Serge Sharoff. 2022. Estimating confidence of predictions of individual classifiers and their ensembles for the genre classification task. Proceedings of LREC

  16. [24]

    Quanyu Long, Wenya Wang, and Sinno Jialin Pan. 2023. Adapt in contexts: Retrieval-augmented domain adaptation via in-context learning. arXiv preprint arXiv:2311.11551

  17. [25]

    Max M \"u ller-Eberstein, Rob Van Der Goot, and Barbara Plank. 2021. Genre as weak supervision for cross-lingual dependency parsing. arXiv preprint arXiv:2109.04733

  18. [26]

    Georg Rehm, Marina Santini, Alexander Mehler, Pavel Braslavski, R \"u diger Gleim, Andrea Stubbe, Svetlana Symonenko, MIRKO LUIGI AURELIO Tavosanis, Vedrana Vidulin, et al. 2008. Towards a reference corpus of web genres for the evaluation of genre identification systems. In Pr...

  19. [27]

    Samuel R \"o nnqvist, Valtteri Skantsi, Miika Oinonen, and Veronika Laippala. 2021. https://aclanthology.org/2021.nodalida-main.16 Multilingual and zero-shot is closing in on monolingual web register classification . In Proceedings of the 23rd Nordic Conference on Computationa...

  20. [28]

    Dmitri Roussinov and Serge Sharoff. 2023. BERT goes off-topic: Investigating the domain transfer challenge using genre classification. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore

  21. [29]

    Serge Sharoff. 2018. Functional text dimensions for the annotation of Web corpora. Corpora, 13(1):65--95

  22. [30]

    Benno Stein, Sven Meyer zu Eissen, and Nedim Lipka. 2011. Web genre analysis: Use cases, retrieval models, and implementation issues. Genres on the web: Computational models and empirical studies, pages 167--189

  23. [31]

    Jade Goldstein Stewart and J Callan. 2009. Genre oriented summarization. Ph.D. thesis, Carnegie Mellon University, Language Technologies Institute, School of …

  24. [32]

    Ruixiang Tang, Yu-Neng Chuang, and Xia Hu. 2024. The science of detecting llm-generated text. Communications of the ACM, 67(4):50--59

  25. [33]

    Marlies Van der Wees, Arianna Bisazza, and Christof Monz. 2018. Evaluation of machine translation performance across multiple genres and languages. In International Conference on Language Resources and Evaluation, pages 3822--3827. European Language Resources Association (ELRA)

  26. [34]

    Vladimir Vapnik. 1995. The nature of statistical learning theory. Springer science & business media

  27. [35]

    Lean Wang, Lei Li, Damai Dai, Deli Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.609 Label words are anchors: An information flow perspective for understanding in-context learning . In Proceedings of the 2023 Conference o...

  28. [36]

    Rongsheng Wang, Haoming Chen, Ruizhe Zhou, Han Ma, Yaofei Duan, Yanlan Kang, Songhua Yang, Baoyu Fan, and Tao Tan. 2024. Llm-detector: Improving ai-generated chinese text detection with open-source llm instruction tuning. arXiv preprint arXiv:2402.01158

  29. [37]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903

  30. [38]

    Lifan Yuan, Yangyi Chen, Ganqu Cui, Hongcheng Gao, Fangyuan Zou, Xingyi Cheng, Heng Ji, Zhiyuan Liu, and Maosong Sun. 2024. Revisiting out-of-distribution robustness in nlp: Benchmarks, analysis, and llms evaluations. Advances in Neural Information Processing Systems, 36

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.