{"id":"518cbd75-33f6-4559-9c75-07051c625345","arxiv_id":"2508.20771","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Cognitive distortion detection trained on English clinical data drops sharply on Dutch adolescent forum posts, and domain-adapted training recovers part of the loss.","lead":"Researchers tested how well AI models trained on English therapy conversations can detect cognitive distortions in posts by Dutch teenagers on a forum. They found that the change in writing style hurt accuracy more than the change in language, and that a domain adaptation method recovered the most performance.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"EN→KT gap attributed to register is confounded by label prior shift: KT has 39% positive vs EN 63%, and no threshold calibration is reported; a threshold sweep on KT predictions would test whether the 'struggle' is calibration rather than register.","rationale":"Reader's weakest_assumption (machine translation preserving register) is legitimate, but I judge it secondary: even if MT were perfect, the NL-vs-KT comparison would still be confounded by the label-prior difference. The prior shift is directly observable in Table 5 (63.1% vs 39.3% positive) and is not controlled anywhere in Section 4.1. All reported F1 scores use implicit 0.5 thresholds from EN training; no precision-recall or calibration analysis appears. The conclusion about register is the paper's key novel claim, so failure of this control would materially change the interpretation, though it would not necessarily invalidate the practical recommendation that target-domain data/adaptation helps. Hence CONDITIONAL rather than REJECT. I also agree with the Reader's noted Empath feature-selection leak, but that issue inflates a comparison baseline rather than threatening the central language-vs-register attribution; the prior-shift confound is more directly tied to the central claim and is easily testable.","tokens_in":14869,"tokens_out":8823,"duration_ms":89055,"concrete_test":"Using the 5-fold predictions of the best EN-only model (e.g., XLMR FT from Table 1), compute the raw positive-class probability on each KT test fold and sweep the decision threshold (e.g., 0.05 to 0.95 in 0.05 steps), reporting weighted F1 at the best threshold (or after isotonic calibration on a held-out fold). If best-threshold F1 approaches the EN+KT adapter/DCCL range (≥0.65) instead of the reported ~0.54, then the register gap is substantially a calibration/prior artifact and the paper's attribution needs revision. As a control, run the same sweep on NL: the language-only gap should remain small regardless of threshold.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing assumption is that the EN→KT performance drop measures register, not label prior. Table 5 shows EN is 63.1% distorted while KT is 39.3% distorted. Section 4.1 trains on EN and evaluates on KT with a fixed decision threshold; no calibration or threshold sweep is reported. A model with perfectly transferable representations but an EN-calibrated threshold will under-predict the positive class on KT, deflating weighted F1. Because NL is a translation of EN, it inherits EN's prior, so the EN→NL vs EN→KT comparison cannot separate the effect of register from the effect of prior shift. The conclusion that 'register is a bigger challenge than language' is therefore not yet supported; EN+KT methods could be winning partly by implicitly recalibrating to the KT prior. A compounding, related untested assumption is that Google Translate (Section 3.2) preserves the EN register in NL; if it does not, even the language-only condition is confounded.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how cognitive distortion detection models generalize across language and register, using an English therapist Q&A dataset (EN), a Dutch machine translation of it (NL), and a newly collected Dutch adolescent forum dataset (KT). It compares prompting, instruction tuning, fine-tuning, and domain-adaptation methods (adapters, Empath features, and Domain Confused Contrastive Learning, DCCL). The main findings are that EN-trained models transfer to NL but drop sharply on KT, suggesting register is more challenging than language, and that DCCL yields the best KT performance among the tested methods.","tokens_in":15202,"tokens_out":4680,"duration_ms":46201,"significance":"If the conclusions hold, the paper would provide a useful new dataset (KT) and evidence that cross-register generalization in mental-health text classification needs more than multilingual fine-tuning. The inclusion of multiple baselines, statistical tests, and an MMD analysis of representation alignment is a strength. However, the central claims are currently weakened by an unaddressed label-prior shift, a target-label leakage in the Empath feature selection, and an untested machine-translation confound, so the significance of the reported effects is not yet established.","major_comments":[{"comment":"The claim that the EN→KT drop shows 'register is a bigger challenge than the language shift' is confounded by label prior shift. EN has 63.1% distorted posts while KT has 39.3% (Table 5). Models trained on EN and evaluated on KT with a fixed decision threshold will under-predict the positive class, deflating weighted F1. NL is a translation of EN and inherits EN's prior, so EN→NL vs EN→KT cannot separate register from prior. Please report calibration/threshold-sweep results, precision-recall curves, or balanced metrics (e.g., macro F1) to support the register-specific interpretation.","section":"Section 4.1, Tables 1 and 5"},{"comment":"The Empath baseline selects 68 features using a paired t-test on the full KT dataset, including test labels, before the cross-validation splits. This is target-label leakage and likely inflates Empath's F1. Additionally, Appendix E.1 shows DCCL vs Empath is not statistically significant (p=0.0637), so the statement in Section 4.3 that 'DCCL performs best' is not supported against Empath. The feature selection must be performed inside each training fold or on a held-out portion, and the significance interpretation should be adjusted accordingly.","section":"Section 4.2 and Appendix C.4"},{"comment":"The construction of NL via Google Translate assumes the translation preserves the register and distortion cues of the original EN text. If translation changes formality, sentence structure, or lexical cues, the EN→NL condition is not a clean language-only shift, and the comparison with EN→KT cannot isolate register. The authors should validate translation quality (e.g., human evaluation, back-translation) or at least explicitly discuss this as a limitation in the main text rather than only in passing.","section":"Section 3.2"}],"minor_comments":[{"comment":"Typos: 'Kintertelefoon' in the Conclusion, 'da◎abase' in Table 2, and a reference formatting inconsistency in the bibliography (e.g., 'Michael Han Daniel Han and Unsloth team').","section":"Throughout"},{"comment":"McNemar tests are only reported for the three best methods. For completeness, report significance comparisons between DCCL and the other EN+KT baselines (e.g., XLMR FT, LLaMA IT) to support the claim that DCCL is preferable.","section":"Appendix E.1"},{"comment":"The 'Random' baseline is not described. Clarify whether it is majority-class, balanced random, or something else, and how the error bars are computed.","section":"Tables 1 and 2"},{"comment":"The row 'XLMR ots' (off-the-shelf XLM-RoBERTa) is not introduced in the methods. Briefly explain what this model is and why it is included as a reference.","section":"Table 3"},{"comment":"The 'Rewriting' method is described as using LLaMA to rewrite EN sentences in the style of a Dutch teenager, but no details are given about the quality of the rewrites or the number of examples used. A short analysis would help interpret the null result in Table 2.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important applied problem and introduces a potentially valuable dataset. However, the main comparative conclusions are vulnerable to the label-prior confound and the Empath leakage. The fixes (threshold sweep, within-fold feature selection, translation validation) are feasible and should be requested before considering publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper does careful work and gives us a new Dutch adolescent dataset, but its central claim—that register shift hurts more than language shift—isn't actually established by the experiments as run. The label-prior confound is real, and the Empath baseline leaks target labels.\n\nWhat's genuinely new: a 450-post annotated De Kindertelefoon corpus, with a transparent annotation protocol and reasonable IAA. The paper also compares a wide range of methods—prompting, instruction tuning, fine-tuning, adapters, a contrastive domain adaptation method—and reports error bars, McNemar tests, and MMD analysis. That's more than most papers in this space do.\n\nWhere it gets soft:\n\n1. The language-vs-register comparison is confounded by label prior. EN has 63% distorted posts; KT has 39%. Everything is evaluated with a fixed threshold from the EN-trained model. With a fixed threshold, a model with perfectly transferable representations but EN-calibrated scores will under-predict on KT and get a lower weighted F1. Because NL is a translation of EN, it inherits EN's prior, so the EN→NL vs EN→KT comparison cannot separate register from prior shift. This is the load-bearing comparison, and the stress-test note is right: a threshold sweep or AUC comparison would settle it. Until that's done, \"register is a bigger challenge than language\" is an overstatement.\n\n2. The Empath baseline selects its 68 features via a t-test on the full KT set, including labels, before the CV splits. That is target-label leakage. It likely inflates Empath's F1, and since DCCL vs Empath isn't significant (p=0.0637), DCCL's advantage is not demonstrated.\n\n3. The NL dataset is machine-translated, and the paper never tests whether the translation preserves the register (and the distortion cues) of the English original. If it doesn't, even the \"language-only\" condition is noisy.\n\nThese are addressable, not fundamental. The dataset is small (450 posts), and data/code are restricted-access only, which limits reproducibility. The limitations section is honest about some of this, but not the prior-shift issue.\n\nWho this is for: researchers applying NLP to youth mental health, especially anyone thinking about cross-lingual or cross-register transfer. The dataset itself is valuable. The empirical claims need a revision before I'd rely on them.\n\nRecommendation: send it to peer review—it deserves a serious referee—but expect major revisions. The authors should calibrate thresholds or report AUC, fix the feature-selection leakage, test the MT condition, and ideally release at least the annotations under a reasonable license.","headline":"A careful empirical study with a useful new dataset, but the core register-vs-language claim is undercut by a label-prior confound, and the best method's edge over a leaky baseline isn't significant.","tokens_in":15603,"tokens_out":3579,"would_cite":false,"duration_ms":34559,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multilingual model transfers across languages but not across registers; domain adaptation closes the gap on Dutch adolescent forum posts.","keywords":["cognitive distortions","cross-lingual transfer","cross-register generalization","domain adaptation","contrastive learning","adolescent mental health","multilingual models","social media text"],"falsifier":"Have human raters compare the machine-translated Dutch test set to the English original for formality and distortion cues; if the translations read more formal or alter the cues, the experiment no longer separates language from register. Alternatively, train on English adult text and test on English teen forum posts: if DCCL's advantage disappears when language is held constant, the method's benefit comes from confusing language, not from adapting to register.","tokens_in":14849,"feed_emoji":"🧠","tokens_out":7939,"duration_ms":76205,"temperature":0.7,"pith_summary":"This paper asks whether automated detection of cognitive distortions—irrational thought patterns that can amplify depression and anxiety—can move from English adult clinical text to the informal, Dutch-language forum posts of adolescents. The authors show that a multilingual model trained on English therapist Q&A transfers across language almost intact, but collapses to near-random accuracy on Dutch teen forum posts, so writing register, not language, is the main obstacle. Among the remedies they test, the strongest is a domain-adaptation technique that forces the model to stop using language and style as a shortcut and focus on the distortion signal itself, raising F1 from about 0.5 to 0.73 on the teen forum data. If this finding holds, automated early-warning tools for youth mental health are plausible, but they appear to require a small amount of target-register training data.","feed_headline":"Register change, not language, defeats distortion detectors","feed_subtitle":"On Dutch teen forums, adult-clinical models fail until domain adaptation recovers performance.","key_machinery":"Domain Confused Contrastive Learning (DCCL): a training objective that adds small learnable perturbations to sentence embeddings, feeds the perturbed embeddings to a domain classifier that tries to tell English adult text from Dutch teen text, and maximizes that classifier's loss so the perturbation hides the domain. A contrastive loss keeps original and perturbed projections close; a classification loss on the original embedding keeps the distortion signal; and a consistency loss keeps predictions stable. This forces the encoder to drop language and style cues and keep only task-relevant cues, which is exactly what cross-register transfer requires. The paper also uses Empath lexical feature","core_discovery":"On the paper's own terms, the discovery is that cognitive distortion detection generalizes poorly across registers despite multilingual models' apparent cross-lingual strength. Using the English Therapist Q&A dataset as the source and a Dutch adolescent forum as the target, the authors find that all baseline methods—prompting, instruction tuning, and fine-tuning—drop to near-random F1 on the teen posts, while the same models lose only a few points on a machine-translated Dutch version of the English data. A small set of 450 annotated target posts changes the picture: fine-tuning on English plus target-register data with Domain Confused Contrastive Learning (DCCL) achieves an F1 of 0.73, and","pith_inferences":["If register is the binding constraint, the same DCCL recipe should transfer to style-shifted settings where the language is unchanged—for example, English adult clinical text to English teen forum text—providing a cleaner test of the register hypothesis.","The machine-translated NL set is the linchpin of the language-versus-register comparison; a human-translated or back-translated version with formality ratings would tell whether translation itself changed the register and therefore the detected cues.","The paper's future-work goal of identifying exact distorted spans could build directly on DCCL's aligned representations, which already separate distortion classes while hiding domain.","A practical deployment implication the authors mention but do not develop: a screening tool biased toward recall, as the Empath baseline is, may be preferable for moderator support, and the recall-versus-precision trade-off should be tuned per use case."],"forward_implications":["Deployable detection on youth forums is within reach: roughly 450 labeled target-register posts, combined with English source data and DCCL, lift weighted F1 from near-random (~0.5) to 0.73.","Language shift alone is mild for a strong multilingual encoder (EN to NL drops from 0.74 to 0.73), so buying more translated or parallel data will not fix register failures.","Prompt-based LLM methods, including instruction tuning with detailed distortion definitions, underperform supervised fine-tuning and are not a substitute for target-register adaptation.","The best-tested method achieves high precision and recall together (DCCL: 0.74/0.73), indicating the model is not simply over-predicting the positive class, whereas the Empath baseline trades precision for recall.","DCCL's advantage over adapter fine-tuning is statistically significant after Bonferroni correction, while its edge over Empath is not significant in these experiments."],"supporting_citations":[{"why":"Supplies the English Therapist Q&A source dataset with cognitive distortion labels and the ten distortion definitions used for annotation and prompts.","marker":"Shreevastava and Foltz (2021)"},{"why":"Provides XLM-RoBERTa, the multilingual encoder that all fine-tuning, adapter, Empath, and DCCL experiments build on.","marker":"Conneau et al., 2020"},{"why":"Contributes the DCCL objective (domain-confusion, contrastive, and consistency losses with their coefficients) that the paper adapts for language-plus-register adaptation.","marker":"Long et al., 2022"},{"why":"Provides the adapter modules used as a parameter-efficient baseline in the generalization experiments.","marker":"Houlsby et al., 2019"},{"why":"Supplies the Empath lexical categories used as features in the Empath baseline.","marker":"Fast et al., 2016"},{"why":"Motivates lexical-feature approaches to cognitive distortion detection and informs the choice of Empath features.","marker":"Simms et al., 2017"},{"why":"Provides the LLaMA-3.1 model used for prompting, instruction tuning, and fine-tuning baselines.","marker":"Touvron et al., 2023"},{"why":"Prior work whose observed weakness of prompt-based methods is corroborated by the paper's prompt-based results.","marker":"Jiang et al., 2024"}],"fun_headline_variants":["Same language, new register: distortion detectors collapse","450 teen posts revive distortion detection after register shock","Cross-lingual holds, cross-register fails for distortion spotting","Teen forum text stumps distortion models; 450 posts turn tide"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The claim that register rather than language causes the performance drop rests on the assumption that the machine-translated Dutch version of the English data keeps the original's writing style and distortion labels; if translation altered formality or the cues that mark distortion, the language-versus-register contrast is confounded.","fun_headline_variants_meta":{"raw":{"variants":["Same language, new register: distortion detectors collapse","450 teen posts revive distortion detection after register shock","Cross-lingual holds, cross-register fails for distortion spotting","Teen forum text stumps distortion models; 450 posts turn tide"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000308,"raw_usage":{"total_tokens":1535,"prompt_tokens":622,"completion_tokens":913,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":366,"completion_tokens_details":{"reasoning_tokens":845}},"tokens_in":366,"tokens_out":913,"duration_ms":9492,"temperature":1.0,"reasoning_tokens":845,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T14:48:52.081647+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human raters compare the machine-translated Dutch test set to the English original for formality and distortion cues; if the translations read more formal or alter the cues, the experiment no longer separates language from register. Alternatively, train on English adult text and test on English teen forum posts: if DCCL's advantage disappears when language is held constant, the method's benefit comes from confusing language, not from adapting to register.","supporting_citations":[],"review_version":1}