Pith. sign in

REVIEW 3 major objections 6 minor 37 references

From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLP

T0 review · 3 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Standard document-level NLP evaluation can reverse whether a model beats a simple baseline once the goal is generalizing to unseen people or future days.

desk verdict Useful framework for longitudinal NLP evaluation, but the headline reversal in Table 2 needs a robustness check for LOCF imputation before I'd fully trust it. read the letter →

arxiv 2601.07988 v2 pith:S5KLBRAL submitted 2026-01-12 cs.CL cs.AIcs.LG

Adithya V Ganesan , Vasudha Varadarajan , Oscar NE Kjell , Whitney R Ringwald , Scott Feltman , Benjamin J Luft , Roman Kotov , Ryan L Boyd
show 1 more author
H Andrew Schwartz
This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords longitudinalNLPbehavioralsequencesevaluationsplitscross-sectionalgeneralizationprospectivebetween-personvswithin-persontemporalcontextPTSDsymptomprediction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that treating documents as independent, unordered samples breaks down when the documents are repeated daily diaries from a small set of people: the data form person-indexed, time-ordered behavioral sequences. It shows empirically that the standard random document split can produce reversed conclusions about a model's usefulness. The same model that beats a training-set-mean baseline under a random document split (MAE .520 vs .660) underperforms that baseline when tested on unseen people (MAE .757 vs .619), and beats it again for future days of seen people (MAE .455 vs .598). The proposed fix is an evaluation paradigm with cross-sectional and prospective splits, between-person and within-person metrics, and sequence-aware models with different coarseness of temporal state. If right, papers evaluating longitudinal human language need to state which generalization target they claim.

What carries the argument

The central object is the 'behavioral sequence': a document is treated as one observation (xi,t, yi,t) indexed by person i and time t, with dependencies from person-level baselines and temporal autocorrelation. The evaluation machinery has two axes: cross-sectional splits (train and test on disjoint people) and prospective splits (train on days up to a cutoff, test on later days for the same people), plus their combination. Metrics are decomposed into between-person (compare each person's mean prediction to mean truth) and within-person (average per-person temporal fit), so a paper can say whether a model learned who is higher or when someone is higher. For modeling, the machinery compares t

What would settle it

Re-run the Table 2 comparison on the same data using only complete-case person-days (no last-observation-carried-forward imputation). If the cross-sectional result no longer reverses—i.e., the model no longer underperforms the training-mean baseline on unseen people—the central empirical demonstration collapses. A complementary check: generate synthetic longitudinal data with known person and time effects and verify that random document splits reverse model-vs-baseline leadership only when person-specific signal leaks; if the reversal appears under the true generative model, the demonstrated e

Watch

Extended reading notes

Core claim

The paper's central discovery is that evaluation protocol, not just model quality, determines the inferred value of longitudinal NLP models. On 17,051 daily diary transcripts from 238 participants paired with daily PTSD symptom severity, a typical document-level model (a large pretrained encoder with a regression head) outperformed a training-set-mean baseline under a conventional random document split (MAE .520 vs .660). The conclusion reversed under a cross-sectional split that holds out entire people: the model underperformed the baseline (MAE .757 vs .619; p<.001). Under a prospective split that tests future days for seen people, the model again beat the baseline (MAE .455 vs .598; p<.00

Load-bearing premise

The empirical results rest on the assumption that carrying forward the last available diary entry for missing days does not systematically distort the within-person temporal signal—an assumption the paper itself flags as fragile because intensive longitudinal data are rarely missing at random.

Editorial extensions

If this is right

  • Random document splits should be avoided whenever documents are repeated measures from a limited set of people: they can both inflate apparent accuracy and reverse conclusions about whether a model beats a trivial baseline.
  • Reporting only pooled (flattened) metrics can hide the source of performance; the prospective correlation of r=.655 was driven mostly by between-person signal, with within-person r only .297.
  • Incorporating prior-day language history generally improves prediction, but the best history length and representation size depend on the generalization target, not just the dataset.
  • No single temporal inductive bias wins in all regimes: autoregressive models are best for unseen people, interaction-based transformers for future days of seen people, and pooled summaries for the combined hardest setting.
  • Evaluation design is part of the scientific claim: a paper evaluated only with random document splits cannot support claims about generalization to new people or future time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This reversed-conclusion phenomenon should be expected in any NLP dataset where labels or references come from a finite set of human sources—annotator-disjoint splits in crowdsourced NLU, or institution-specific summaries—not only in repeated-measures mental-health diaries.
  • The between-/within-person decomposition doubles as a shortcut diagnostic: a large gap between flattened and within-person performance flags a model that leans on stable person-level baselines rather than tracking day-to-day change.
  • A direct testable extension is to benchmark existing longitudinal NLP tasks under cross-sectional and prospective splits and compare rank orderings; if rankings shift, current leaderboards for those tasks are likely overstating real-world readiness.
  • Because the paper imputes missing diary days by carrying forward the last available language, its conclusions should be re-checked with complete-case or multiple-imputation analyses; the direction of the reversal could be sensitive to that choice.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that standard document-level i.i.d. evaluation is inappropriate for longitudinal NLP data, where documents are nested within persons and ordered in time. It proposes a four-part paradigm: (1) evaluation splits targeting cross-sectional (unseen people) and prospective (future time) generalization; (2) between- and within-person metric decompositions; (3) sequence inputs with history by default; (4) model internals supporting different 'coarseness' of latent state. Using 17k daily diary transcripts and PTSD symptom scores from 238 participants, the authors demonstrate that a traditional random document split reverses the conclusion about a RoBERTa-large model vs. a mean baseline relative to a cross-sectional split (Table 2), that pooled metrics hide whether models capture person means vs. temporal dynamics (Figure 1, Table 3), and that optimal representation dimensionality, history length, and temporal inductive bias depend on the generalization regime (Figures 2-4).

Significance. If the central empirical finding is robust, the paper makes a valuable methodological contribution: it gives NLP a concrete vocabulary and set of procedures for choosing splits and metrics in person-indexed, time-ordered data, and it connects a known leakage problem (Geva et al. 2019) to longitudinal mental-health NLP. Strengths include matched train/test sizes across regimes (Table 2), hyperparameter selection on regime-matched development splits (Appendix C), explicit metric definitions (Table 3), and falsifiable claims about regime-dependent conclusions. The paper does not overclaim model performance; it is transparent about its simple model classes and single dataset. The main risk is that the headline reversal is built on a 20-person constructed subsample and on last-observation-carried-forward imputation, with no sensitivity analysis.

major comments (3)
  1. [§2 Missingness handling; §3 Table 2] The headline reversal in Table 2 rests on LOCF imputation for missing language, yet the paper's Limitations concede that intensive longitudinal data are rarely missing-at-random. With a random document split, a carried-forward input still contains person-identifying cues from the same individual's earlier text; under a cross-sectional split the test person is unseen, so carried-forward inputs are stale, potentially widening the traditional-vs-cross-sectional gap. No missingness rates by split or alternative analysis (complete-case, missingness indicator, multiple imputation) is reported. Because this reversal is the paper's central empirical claim, please add a missingness sensitivity analysis and report missingness rates; if the reversal persists without LOCF, the claim is much stronger.
  2. [§3 Table 2 and Appendix B.1] The demonstration uses 20 participants, a single split per regime, and p<.001 only for model vs. baseline, not for the reversal (traditional vs. cross-sectional / prospective). The point estimates .520/.757/.455 are striking, but with n=20 people, one random draw could be unrepresentative. Report bootstrap CIs or repeated split draws for the ΔMAE differences, and ideally repeat the comparison on the full 238-participant sample or several 20-person samples. Also state how the 'randomly masked' instances were chosen and whether results are stable across seeds.
  3. [§4 Figure 4 and Appendix C.2] The 'state coarseness' conclusion is based on a deliberately minimal transformer (1 layer, 1 head, no positional embeddings). Without positional information and with only a causal mask, the model cannot represent order or lag distances, so it is not a strong test of 'interaction-rich sequence models.' The claim that prospective generalization benefits from temporal interactions should be framed as specific to this minimal architecture; otherwise the coarseness taxonomy in contribution (4) is overstated. If the authors want to support the stronger claim, add a positional-encoding ablation or a slightly larger transformer.
minor comments (6)
  1. [§3] Typo: 'samping' should be 'sampling'.
  2. [Table 3] The flattened metric equation is typeset with ambiguous parentheses; define f as applied to pooled person-day instances.
  3. [Appendix C.1] 'le 1024' should be '≤ 1024'.
  4. [Ethical Considerations] The 'Sensitive human-subject data and governance' paragraph is duplicated almost verbatim twice in the manuscript.
  5. [§4 / Figure 2] The task switch from same-day nowcasting (§3) to one-day-ahead forecasting (§4) should be stated at the first use of Figure 2, since Figure 2's caption does not say forecasting (the SMAPE version in Figure 5 does).
  6. [Table 2 caption] The caption's p<.001 should specify what is being tested and the number of paired observations; the main text says one-sided paired t-test vs. baseline but the caption alone is ambiguous.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: empirical evaluation claims are self-contained; self-citations are motivational, not load-bearing.

full rationale

The paper's central claim — that random document-level splits can reverse conclusions relative to cross-sectional/prospective splits — is supported by a direct experiment (Table 2) on held-out data, not by a derivation from prior work. No equation is fit to the test set and then reported as a prediction: hyperparameters are selected on regime-matched dev splits (Appendix C: 'the development set is split from the training data using the same evaluation regime as the corresponding test set'), and the headline MAE comparisons use a fixed RoBERTa ridge model and a train-mean baseline. The self-citations (e.g., V Ganesan et al. 2021 for 'effective sample size' and dimensionality; Boyd and Schwartz 2021 for verbal behavior) are background motivation and are not used to define the outcomes or forbid alternatives. The only caveat with the feel of circularity is the admitted LOCF imputation for missing language (Section 2: 'we impute x_{i,t} by carrying forward the most recent available language observation for that participant'), which could differentially affect evaluation regimes if missingness is informative — the paper itself concedes 'intensive longitudinal data are rarely missing-at-random.' That is a validity/robustness concern, not circularity: it is a confound in an empirical measurement, not a prediction that equals its input by construction. Aside from this, no step in the paper's argument reduces to a self-citation chain, a fitted parameter renamed as prediction, or a definition equating input and output. Score 1 reflects only the presence of a few non-load-bearing self-citations and the flagged missingness caveat; there is no substantive circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper makes empirical claims rather than deriving theory, so the ledger mainly records assumptions about data validity, missingness, and hyperparameter choices that shape the reported patterns. No invented entities.

free parameters (5)
  • Ridge penalty λ = selected from {10^-2..10^5} on regime-matched dev
    Controls all model comparisons; chosen per split/regime on dev (Appendix C).
  • PCA language dimensionality d = 32–1024 grid; peaks at 64–128 cross-sectional, ~512 prospective
    Figure 2/5 conclusion that optimal capacity depends on generalization target.
  • History length h = 1–32 days grid
    Figure 3/6 core sequence-modeling results.
  • Transformer hyperparameters = d'=32, weight decay=1, attn dropout=0.3, out dropout=0.1, lr=1e-3
    Tuned on dev; only one minimal architecture tested, limiting coarseness comparison (Appendix C.2).
  • Temporal cutoff τ = 60 or 63 days
    Prospective split boundary chosen by hand; influences all prospective results.
assumptions (4)
  • domain assumption Daily diary transcripts and PCL self-report scores measure PTSD-relevant verbal behavior and symptoms at daily resolution
    Section 2 treats transcripts as repeated measures of outcome-related behavior; if the diary prompts themselves (e.g., asking about unwanted memories) trivially reveal symptoms, the task becomes easier and the evaluation conclusions may not transfer.
  • ad hoc to paper Missing language days can be imputed by last observation carried forward without biasing within-person comparisons
    Section 2 Missingness handling; the paper's Limitations admit missingness is rarely at random, but no sensitivity analysis is provided.
  • domain assumption Whisper ASR transcription noise does not systematically alter between- vs within-person language signal
    Limitations note transcription errors can differentially affect within-person vs between-person signals.
  • domain assumption Ridge regression on PCA-reduced RoBERTa representations is a fair 'typical model' and enough to reveal evaluation effects
    Appendix C; more expressive models might behave differently, though the paper limits claims to simple transparent models.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLP." pith.science (2026). https://pith.science/paper/S5KLBRAL

@misc{pith2026260107988,
  author       = {Pith},
  title        = {Pith review of: From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLP},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S5KLBRAL}},
  note         = {Machine review of arXiv:2601.07988}
}
abstract

While NLP typically treats documents as independent and unordered samples, in longitudinal studies, this assumption rarely holds: documents are nested within authors and ordered in time, forming person-indexed, time-ordered $\textit{behavioral sequences}$. Here, we demonstrate the need for and propose a longitudinal modeling and evaluation paradigm that consequently updates four parts of the NLP pipeline: (1) evaluation splits aligned to generalization over people ($\textit{cross-sectional}$) and/or time ($\textit{prospective}$); (2) accuracy metrics separating between-person differences from within-person dynamics; (3) sequence inputs to incorporate history by default; and (4) model internals that support different $\textit{coarseness}$ of latent state over histories (pooled summaries, explicit dynamics, or interaction-based models). We demonstrate the issues ensued by traditional pipeline and our proposed improvements on a dataset of 17k daily diary transcripts paired with PTSD symptom severity from 238 participants, finding that traditional document-level evaluation can yield substantially different and sometimes reversed conclusions compared to our ecologically valid modeling and evaluation. We tie our results to a broader discussion motivating a shift from word-sequence evaluation toward $\textit{behavior-sequence}$ paradigms for NLP.

Figures

Figures reproduced from arXiv: 2601.07988 by the authors.

Figure 1
Figure 1. Flattened metrics can mask whether models learn people or dynamics. Left: For three illustrative users from cross-sectional test set, predictions covary within individuals, but the between-person relationship of person-level means is reversed in direction; a pooled document-level fit conflates these effects, depicted by the flattened fit. Right: Decomposing performance into between-person and within-person MAE/r acr… view at source ↗
Figure 2
Figure 2. Between- and within-person MAE vs. rep￾resentation size. Error shows a U-shaped trend across regimes: cross-sectional and cross-sectional & prospec￾tive perform best with small representations (64–128), whereas prospective benefits from larger size (∼512). Representation capacity depended on the gen￾eralization target (people vs. time). We first considered the typical NLP modeling setup that maps an isolated documen… view at source ↗
Figure 3
Figure 3. Between- and within-person MAE vs. history length. Using longer history generally improves performance, but the best history–capacity trade-off depends on the regime: prospective benefits from longer context with larger representations (≥512), while cross-sectional and cross-sectional & prospective improve primarily with longer context at smaller size (∼64). Across splits, the average best size is 128. els achieved … view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: AR vs. BoE vs. Transformer across history length (128 dims per day). Between-person (top) and within-person (bottom) MAE as a function of history h. AR performs best for cross-sectional generalization, while modeling temporal interactions (Transformer) yields the large…
Figure 5
Figure 5. Figure 5: Between- (top) and Within-Person (bottom) SMAPE as a function of hidden dimension size. Fore￾casting performance follows a U-shaped trend as a func￾tion of hidden dimension size of language across all three evaluation sets. While a typical model requires only 64 dimens…
Figure 6
Figure 6. Figure 6: Between- and Within-Person SMAPE as a function of history length for Auto Regressive Model. Predictive Performance improves with modeling the temporal dynamics of linguistic behavior. For Prospective test set, between- and within-person performance improves with longer…
Figure 7
Figure 7. Figure 7: Between- and Within-Person metrics as a function of history length for Bag of Embeddings Model. Predictive Performance improves with modeling the temporal context of linguistic behavior. For Prospective test set, between- and within-person performance improves with lon…
Figure 8
Figure 8. Figure 8: Comparison of Auto Regressive, Bag of Embeddings and Transformer model using Between- (top) and Within-Person (bottom) SMAPE as a function of history length. 0.19 0.20 0.20 0.20 0.21 0.21 Between-Person SMAPE Cross-sectional Test Set 0.18 0.20 0.22 0.24 Prospective Tes…
Figure 9
Figure 9. Figure 9: AR vs. BoE vs. Transformer across history length (128 dims per day). Between-person (top) and within-person (bottom) SMAPE as a function of history h. Modeling temporal interactions (Transformer) offers marginal improvements over modeling dynamics in cross-sectional sp…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 1 linked inside Pith

  1. [1]

    Tell me about the best part of your day

  2. [2]

    InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), pages 1–16

    Methodological gaps in predicting mental health states from social media: Triangulating di- agnostic signals. InProceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI), pages 1–16. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for da...

    arXiv 2019

  3. [3]

    Describe when you felt most sad today

  4. [4]

    Describe when you felt most scared or ner- vous today

  5. [5]

    Describe when you felt most annoyed today

  6. [6]

    Tell me about the worst part of your day

  7. [7]

    How did you feel physically today? Did you have any pain, discomfort, or other physical symptoms? Please elaborate

  8. [8]

    How did you get along with others today? Please elaborate

Show all 37 references
  1. [9]

    Did you have any unwanted, disturbing mem- ories of a past stressful experience? Tell me about this

  2. [10]

    Describe when you felt most happy today

  3. [11]

    Did you feel on guard today? What made you feel this way?

  4. [12]

    How did it go?

    Describe anything that cheered you up today. How did it go?

  5. [13]

    What happened today that you can feel thank- ful for? Tell me about this. Rating ScalesIn addition to the open-ended di- ary, participants completed daily self-report rating scales capturing (i) post-stressor/PTSD-related ex- periences and overall stress that day, and (ii) exp...

  6. [14]

    Today, did you avoid anything because it would have made you uncomfortable? Tell me more about what you did and why

  7. [18]

    Today, I had repeated, disturbing, and un- wanted memories of the stressful experience

  8. [19]

    Today, I felt very upset because something reminded me of the stressful experience

  9. [20]

    Today, I avoided memories, thoughts, or feel- ings related to the stressful experience

  10. [21]

    Today, I avoided external reminders of the stressful experience (for example, people, places, conversations, activities, objects, or situations)

  11. [22]

    Today, I felt distant or cut off from other peo- ple

  12. [23]

    Today, I had strong negative feelings such as fear, horror, anger, guilt, or shame

  13. [24]

    Today, I felt jumpy or easily startled

  14. [25]

    superalert

    Today, I was “superalert” or watchful or on guard

  15. [26]

    Overall, how stressed did you feel today? Part 2 (Daily stressor checklist).Participants then indicated which of the following troublesome or stressful events occurred everyday (check all that apply):

  16. [27]

    Had tension or argument with spouse, partner or close family

  17. [28]

    Had tension or argument with others (e.g., co- worker, friend, etc)

  18. [29]

    A lot of demands at home

  19. [30]

    A lot of demands at job

  20. [31]

    A lot of demands made by family

  21. [32]

    Caring for a sick family member

  22. [33]

    Problems with transportation

  23. [34]

    Financial or money problem(s)

  24. [35]

    Health-related event(s)

  25. [36]

    Other troublesome things happened to me

  26. [37]

    about the text itself

    No troublesome or stressful things happened to me today B Data Pre-processing B.1 Longitudinal Evaluation The violations of independence assumption arise whenever many documents are produced or labeled by a limited set of humans, and they are amplified in longitudinal data whe...

    2019

  27. [1979]

    https://www.hhs.gov/ohrp/ regulations-and-policy/belmont-report/

    The belmont report: Ethical principles and guidelines for the protection of human sub- jects of research. https://www.hhs.gov/ohrp/ regulations-and-policy/belmont-report/ . Last reviewed Aug 26, 2024. U.S. Department of Health & Human Ser- vices. 2025. Guidance regarding metho...

    2024

  28. [2008]

    Ecological momentary assessment.Annu. Rev. Clin. Psychol., 4(1):1–32. Khushboo Singh, Vasudha Varadarajan, Adithya V . Ganesan, August Håkan Nilsson, Nikita Soni, Syeda Mahwish, Pranav Chitale, Ryan L. Boyd, Lyle Ungar, Richard N. Rosenthal, and H. Andrew Schwartz. 2025. Syste...

    2025

  29. [2019]

    InProceed- ings of the 2019 ACM Conference on Fairness, Ac- countability, and Transparency (FAccT), pages 79– 88

    A taxonomy of ethical tensions in inferring mental health states from social media. InProceed- ings of the 2019 ACM Conference on Fairness, Ac- countability, and Transparency (FAccT), pages 79– 88. Stevie Chancellor and Munmun De Choudhury. 2020. Methods in predictive techniqu...

    2019

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.