{"id":"575edf65-0731-42ef-a24c-89932d518886","arxiv_id":"2601.07988","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Longitudinal NLP evaluation should separate generalization to unseen people from generalization to future days, and report between-person vs within-person metrics, because document-level splits can reverse conclusions.","lead":"This paper argues that standard NLP evaluation—randomly splitting documents and pooling all predictions—can mislead when documents come from the same people over time, and proposes separating evaluation by people and time. On daily diaries from 238 participants, it shows conclusions can reverse depending on whether models are tested on new people or future days.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LOCF imputation for missing diaries may drive the headline reversal in Table 2; robustness under alternative missingness handling is needed.","rationale":"The reader's weakest assumption and my analysis converge on the same load-bearing concern: LOCF imputation under informative missingness can distort within-person temporal signal and differentially affect evaluation regimes. This concern is directly tied to the paper's strongest empirical claim—the Table 2 reversal—because imputed inputs are not the same across random, cross-sectional, and prospective splits, and the model's ability to exploit person-specific cues from carried-forward text depends on whether those people were seen in training. The paper's own Limitations explicitly flag non-random missingness, so this is not an external or adversarial critique but an internal consistency check. I considered other concerns (e.g., the 20-person sample size, single dataset, minimal transformer) but these affect generalizability rather than the validity of the demonstrated reversal; the LOCF issue, if confirmed, could invalidate the central demonstration itself. The proposed concrete test—re-running Table 2 under alternative missingness handling—would settle the question without requiring new data, since the authors have access to the private dataset. Because the concern is addressable and does not force a verdict change from the reader's CONDITIONAL, I recommend no change to the verdict; the paper should add this robustness check to justify its central claim.","tokens_in":18746,"tokens_out":7908,"duration_ms":84451,"concrete_test":"Re-run the Table 2 comparison on the same 20-person cohort, with identical splits and the same RoBERTa-large ridge model, but replace LOCF with: (a) listwise deletion of person-days with missing language (report the resulting N per split), (b) person-mean imputation of language vectors, and (c) LOCF plus a binary missingness indicator as an extra feature. If the reversal—model MAE < baseline MAE under the random split and > baseline under the cross-sectional split—persists across all three treatments, the conclusion is robust to missingness handling. If the cross-sectional model matches or beats the baseline under any alternative, the headline claim is an artifact of LOCF and the paper must qualify its central conclusion.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2 states that missing language is imputed by carrying forward the most recent available transcript (last observation carried forward), while missing outcomes are excluded. This imputation is applied to the 20-person demonstration in Table 2. If missingness is informative about PTSD severity—as the paper's Limitations explicitly concede ('intensive longitudinal data are rarely missing-at-random')—then LOCF can differentially affect the three evaluation regimes. In the traditional random split, the model has already seen the same person's training documents, so a carried-forward input still carries person-specific stylistic and content cues that help prediction. In the cross-sectional split, the model has never seen the test person, so a carried-forward input is stale and likely misaligned with current symptoms, degrading performance and widening the gap between traditional and cross-sectional MAE. This could turn a real evaluation-regime difference into an artifact of imputation. Similarly, for the prospective and sequence-modeling results, repeated carried-forward vectors create artificial autocorrelation, which can make autoregressive and history-based models appear more beneficial than they would with genuine daily language. The paper neither reports missingness rates by split nor provides any alternative missingness analysis, despite flagging the risk itself. Therefore the central empirical reversal (model beats baseline under random split, loses under cross-sectional split) may be partly driven by LOCF rather than purely by the evaluation protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that standard document-level i.i.d. evaluation is inappropriate for longitudinal NLP data, where documents are nested within persons and ordered in time. It proposes a four-part paradigm: (1) evaluation splits targeting cross-sectional (unseen people) and prospective (future time) generalization; (2) between- and within-person metric decompositions; (3) sequence inputs with history by default; (4) model internals supporting different 'coarseness' of latent state. Using 17k daily diary transcripts and PTSD symptom scores from 238 participants, the authors demonstrate that a traditional random document split reverses the conclusion about a RoBERTa-large model vs. a mean baseline relative to a cross-sectional split (Table 2), that pooled metrics hide whether models capture person means vs. temporal dynamics (Figure 1, Table 3), and that optimal representation dimensionality, history length, and temporal inductive bias depend on the generalization regime (Figures 2-4).","tokens_in":19096,"tokens_out":5856,"duration_ms":57805,"significance":"If the central empirical finding is robust, the paper makes a valuable methodological contribution: it gives NLP a concrete vocabulary and set of procedures for choosing splits and metrics in person-indexed, time-ordered data, and it connects a known leakage problem (Geva et al. 2019) to longitudinal mental-health NLP. Strengths include matched train/test sizes across regimes (Table 2), hyperparameter selection on regime-matched development splits (Appendix C), explicit metric definitions (Table 3), and falsifiable claims about regime-dependent conclusions. The paper does not overclaim model performance; it is transparent about its simple model classes and single dataset. The main risk is that the headline reversal is built on a 20-person constructed subsample and on last-observation-carried-forward imputation, with no sensitivity analysis.","major_comments":[{"comment":"The headline reversal in Table 2 rests on LOCF imputation for missing language, yet the paper's Limitations concede that intensive longitudinal data are rarely missing-at-random. With a random document split, a carried-forward input still contains person-identifying cues from the same individual's earlier text; under a cross-sectional split the test person is unseen, so carried-forward inputs are stale, potentially widening the traditional-vs-cross-sectional gap. No missingness rates by split or alternative analysis (complete-case, missingness indicator, multiple imputation) is reported. Because this reversal is the paper's central empirical claim, please add a missingness sensitivity analysis and report missingness rates; if the reversal persists without LOCF, the claim is much stronger.","section":"§2 Missingness handling; §3 Table 2"},{"comment":"The demonstration uses 20 participants, a single split per regime, and p<.001 only for model vs. baseline, not for the reversal (traditional vs. cross-sectional / prospective). The point estimates .520/.757/.455 are striking, but with n=20 people, one random draw could be unrepresentative. Report bootstrap CIs or repeated split draws for the ΔMAE differences, and ideally repeat the comparison on the full 238-participant sample or several 20-person samples. Also state how the 'randomly masked' instances were chosen and whether results are stable across seeds.","section":"§3 Table 2 and Appendix B.1"},{"comment":"The 'state coarseness' conclusion is based on a deliberately minimal transformer (1 layer, 1 head, no positional embeddings). Without positional information and with only a causal mask, the model cannot represent order or lag distances, so it is not a strong test of 'interaction-rich sequence models.' The claim that prospective generalization benefits from temporal interactions should be framed as specific to this minimal architecture; otherwise the coarseness taxonomy in contribution (4) is overstated. If the authors want to support the stronger claim, add a positional-encoding ablation or a slightly larger transformer.","section":"§4 Figure 4 and Appendix C.2"}],"minor_comments":[{"comment":"Typo: 'samping' should be 'sampling'.","section":"§3"},{"comment":"The flattened metric equation is typeset with ambiguous parentheses; define f as applied to pooled person-day instances.","section":"Table 3"},{"comment":"'le 1024' should be '≤ 1024'.","section":"Appendix C.1"},{"comment":"The 'Sensitive human-subject data and governance' paragraph is duplicated almost verbatim twice in the manuscript.","section":"Ethical Considerations"},{"comment":"The task switch from same-day nowcasting (§3) to one-day-ahead forecasting (§4) should be stated at the first use of Figure 2, since Figure 2's caption does not say forecasting (the SMAPE version in Figure 5 does).","section":"§4 / Figure 2"},{"comment":"The caption's p<.001 should specify what is being tested and the number of paired observations; the main text says one-sided paired t-test vs. baseline but the caption alone is ambiguous.","section":"Table 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for an NLP methodology venue. I would support publication after the missingness and statistical-robustness analyses are added. The self-citation density is high but mostly to the authors' own relevant prior work; I do not see a citation-policy problem. The duplicated ethics paragraph should be cleaned up."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline result—that document-level random splits can reverse conclusions about whether a model beats a mean baseline—is the kind of thing that should make people in longitudinal NLP stop and check their evaluation protocol. The paper builds the case carefully: regime-matched dev tuning, matched train/test sizes, and a clear distinction between cross-sectional and prospective generalization. The between/within-person metric decomposition is not new in psychology, but it is well adapted here and genuinely missing from most NLP evaluation. That is the real contribution.\n\nThe soft spot is the one you flagged: LOCF imputation for missing daily language. The paper itself concedes that intensive longitudinal data are rarely missing at random, and LOCF is applied to the 20-person subsample behind Table 2. It is entirely plausible that carried-forward inputs help in the traditional random split (where the test person is already in training) and hurt in the cross-sectional split (where the test person is unseen), widening the gap that produces the reversal. The paper reports no missingness rates by split and no alternative imputation analysis. That makes the central empirical demonstration less convincing than it should be. This is fixable, not fatal—the framework survives, but the motivating reversal needs to be shown not to be an artifact of imputation.\n\nOther soft spots are minor: the statistical test is reported only as p<.001 without procedure details; the transformer is deliberately minimal (no positional embeddings), which is defensible given the 'coarseness' question but limits scope; and the empirical load rests on one private dataset, with data/code not available, though the sensitivity of diary text makes that partly understandable. The authors are honest about all of this in the Limitations.\n\nBottom line: this is a serious methodology paper for anyone working on mental-health or social-media NLP. It deserves a careful peer review, and the reviewer should push hard on the LOCF robustness check and on reporting missingness. If those come back clean, the reversal claim becomes strong. The paper is worth citing for the framework regardless.","headline":"Useful framework for longitudinal NLP evaluation, but the headline reversal in Table 2 needs a robustness check for LOCF imputation before I'd fully trust it.","tokens_in":19535,"tokens_out":2406,"would_cite":true,"duration_ms":26077,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Standard document-level NLP evaluation can reverse whether a model beats a simple baseline once the goal is generalizing to unseen people or future days.","keywords":["longitudinal NLP","behavioral sequences","evaluation splits","cross-sectional generalization","prospective generalization","between-person vs within-person","temporal context","PTSD symptom prediction"],"falsifier":"Re-run the Table 2 comparison on the same data using only complete-case person-days (no last-observation-carried-forward imputation). If the cross-sectional result no longer reverses—i.e., the model no longer underperforms the training-mean baseline on unseen people—the central empirical demonstration collapses. A complementary check: generate synthetic longitudinal data with known person and time effects and verify that random document splits reverse model-vs-baseline leadership only when person-specific signal leaks; if the reversal appears under the true generative model, the demonstrated e","tokens_in":18694,"feed_emoji":"📊","tokens_out":5698,"duration_ms":53802,"temperature":0.7,"pith_summary":"This paper argues that treating documents as independent, unordered samples breaks down when the documents are repeated daily diaries from a small set of people: the data form person-indexed, time-ordered behavioral sequences. It shows empirically that the standard random document split can produce reversed conclusions about a model's usefulness. The same model that beats a training-set-mean baseline under a random document split (MAE .520 vs .660) underperforms that baseline when tested on unseen people (MAE .757 vs .619), and beats it again for future days of seen people (MAE .455 vs .598). The proposed fix is an evaluation paradigm with cross-sectional and prospective splits, between-person and within-person metrics, and sequence-aware models with different coarseness of temporal state. If right, papers evaluating longitudinal human language need to state which generalization target they claim.","feed_headline":"Random data splits can reverse model verdicts","feed_subtitle":"A daily-diary PTSD model that beats baseline on random splits loses on unseen people and wins again on future days.","key_machinery":"The central object is the 'behavioral sequence': a document is treated as one observation (xi,t, yi,t) indexed by person i and time t, with dependencies from person-level baselines and temporal autocorrelation. The evaluation machinery has two axes: cross-sectional splits (train and test on disjoint people) and prospective splits (train on days up to a cutoff, test on later days for the same people), plus their combination. Metrics are decomposed into between-person (compare each person's mean prediction to mean truth) and within-person (average per-person temporal fit), so a paper can say whether a model learned who is higher or when someone is higher. For modeling, the machinery compares t","core_discovery":"The paper's central discovery is that evaluation protocol, not just model quality, determines the inferred value of longitudinal NLP models. On 17,051 daily diary transcripts from 238 participants paired with daily PTSD symptom severity, a typical document-level model (a large pretrained encoder with a regression head) outperformed a training-set-mean baseline under a conventional random document split (MAE .520 vs .660). The conclusion reversed under a cross-sectional split that holds out entire people: the model underperformed the baseline (MAE .757 vs .619; p<.001). Under a prospective split that tests future days for seen people, the model again beat the baseline (MAE .455 vs .598; p<.00","pith_inferences":["This reversed-conclusion phenomenon should be expected in any NLP dataset where labels or references come from a finite set of human sources—annotator-disjoint splits in crowdsourced NLU, or institution-specific summaries—not only in repeated-measures mental-health diaries.","The between-/within-person decomposition doubles as a shortcut diagnostic: a large gap between flattened and within-person performance flags a model that leans on stable person-level baselines rather than tracking day-to-day change.","A direct testable extension is to benchmark existing longitudinal NLP tasks under cross-sectional and prospective splits and compare rank orderings; if rankings shift, current leaderboards for those tasks are likely overstating real-world readiness.","Because the paper imputes missing diary days by carrying forward the last available language, its conclusions should be re-checked with complete-case or multiple-imputation analyses; the direction of the reversal could be sensitive to that choice."],"forward_implications":["Random document splits should be avoided whenever documents are repeated measures from a limited set of people: they can both inflate apparent accuracy and reverse conclusions about whether a model beats a trivial baseline.","Reporting only pooled (flattened) metrics can hide the source of performance; the prospective correlation of r=.655 was driven mostly by between-person signal, with within-person r only .297.","Incorporating prior-day language history generally improves prediction, but the best history length and representation size depend on the generalization target, not just the dataset.","No single temporal inductive bias wins in all regimes: autoregressive models are best for unseen people, interaction-based transformers for future days of seen people, and pooled summaries for the combined hardest setting.","Evaluation design is part of the scientific claim: a paper evaluated only with random document splits cannot support claims about generalization to new people or future time."],"fun_headline_variants":["Evaluation splits flip PTSD model's verdicts","How you split data decides your NLP model's worth","Random splits fool PTSD diary model","Cross-sectional vs prospective: model verdict reverses","Diary model wins, loses, wins again based on split"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The empirical results rest on the assumption that carrying forward the last available diary entry for missing days does not systematically distort the within-person temporal signal—an assumption the paper itself flags as fragile because intensive longitudinal data are rarely missing at random.","fun_headline_variants_meta":{"raw":{"variants":["Evaluation splits flip PTSD model's verdicts","How you split data decides your NLP model's worth","Random splits fool PTSD diary model","Cross-sectional vs prospective: model verdict reverses","Diary model wins, loses, wins again based on split"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000146,"raw_usage":{"total_tokens":1021,"prompt_tokens":748,"completion_tokens":273,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":203}},"tokens_in":492,"tokens_out":273,"duration_ms":3744,"temperature":1.0,"reasoning_tokens":203,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T10:56:55.993970+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Table 2 comparison on the same data using only complete-case person-days (no last-observation-carried-forward imputation). If the cross-sectional result no longer reverses—i.e., the model no longer underperforms the training-mean baseline on unseen people—the central empirical demonstration collapses. A complementary check: generate synthetic longitudinal data with known person and time effects and verify that random document splits reverse model-vs-baseline leadership only when person-specific signal leaks; if the reversal appears under the true generative model, the demonstrated e","supporting_citations":[],"review_version":1}