{"id":"548d36b4-25b6-4b8a-9b1f-4af7df938638","arxiv_id":"2502.05670","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Across heavy NP shift, dative alternation and multiple PP shift, LLM ordering preferences correlate with human judgments, but particle movement preferences do not.","lead":"Large language models mostly prefer the same constituent orders in English that humans do, with particle movement as a clear exception. The study compares 12 models across four sentence-reordering phenomena and against 126 human raters.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 4 reports absolute Spearman correlations; without signed values, anti-aligned models are indistinguishable from aligned ones, so the 'generally align' claim is not yet supported.","rationale":"The reader's weakest_assumption is the dative-alternation length confound: the shifted form adds the preposition 'to', so M_preference for DA carries a systematic bias from the extra token, and the GPT-2 DA correlation of 0.651 is the highest in the paper. That concern is real and well-founded, and I do not dispute it. However, I think the more load-bearing issue is the unreported sign of the reported correlations. The central claim is about the direction of alignment, and absolute Spearman values cannot support it. If the authors had reported signed rhos and all were negative, the DA confound would be the main remaining validity threat; but without signed values, the headline claim is unverifiable. The two concerns compound: the DA confound can inflate the magnitude of a correlation, and the absolute-value presentation can mask its direction. Either issue alone warrants conditional acceptance; together they mean the paper should be accepted only after the authors release signed correlations and a length-normalized DA analysis. I therefore keep the reader's CONDITIONAL verdict unchanged.","tokens_in":13135,"tokens_out":8207,"duration_ms":81175,"concrete_test":"Recompute the Table 4 results as signed Spearman rhos between M_preference and the human naturalness scores, with the sign convention that negative rho means the model and humans prefer the same constituent order (since M_preference > 0 means unshifted-preferred and human score < 4 means unshifted-preferred). If any |rho| values correspond to positive signed rhos, the model is anti-aligned for that shift, and the 'generally align' claim must be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LLM M_preference scores align with human naturalness judgments. Section 7.2 and Table 4 report only 'Spearman correlation values (absolute)'. M_preference is defined (Section 5.1) as Mscore(unshifted) − Mscore(shifted), with positive values meaning the unshifted sentence is preferred. Human scores are 1 = first sentence more natural and 7 = second more natural; the paper's examples and Table 1 list unshifted first. Therefore, under genuine alignment, the signed Spearman rho must be negative. Reporting |rho| conflates correctly aligned models (negative rho) with anti-aligned models (positive rho). A positive rho would mean the model prefers the shifted order exactly where humans prefer the unshifted order, which is the opposite of the paper's claim. Because no signed values are given, the reader cannot determine whether the headline result is driven by true alignment or by an artifact of taking absolute values. This is particularly problematic for particle movement, where the paper reports 'low' correlations: if the underlying PM rhos are positive, the finding is not a weak failure but an active reversal, which the absolute-value presentation hides. The missing sign information is a load-bearing gap in the primary evidence for the abstract's 'generally align' claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares LLM preferences on four types of English constituent movement (Heavy NP Shift, Particle Movement, Dative Alternation, and Multiple PP Shift) against human naturalness judgments. Model preferences are computed as the difference in full-sequence log-probabilities between unshifted and shifted sentence pairs (M_preference), using frozen autoregressive models across several families and sizes. Human judgments come from a Prolific crowdsourced study on a subset of the sentence pairs. The authors report that LLMs generally align with human ordering preferences except for particle movement, that syllable weight is the most important weight measure in explaining model preferences, and that instruction-tuned models align with humans less than their base counterparts. The paper also provides a large synthetic dataset and a smaller mined dataset, with code and data publicly available.","tokens_in":13368,"tokens_out":3648,"duration_ms":38647,"significance":"If the headline result holds, this is a valuable large-scale benchmark for syntactic ordering preferences in LLMs, with implications for psycholinguistic modeling and for claims about human-like inductive biases in language models. The paper's strengths include the use of raw probability scores from frozen pretrained models (so nothing is fitted to the human judgments), the breadth of model families and scales, the combination of synthetic and naturally occurring data, and the public release of code and data. The finding that instruction-tuned models correlate less with human judgments than their base counterparts is a useful and somewhat counterintuitive contribution. However, two load-bearing issues in the current presentation—the use of absolute Spearman correlations and the non-minimal-pair dative alternation stimuli—mean the central 'generally align' claim is not yet actually demonstrated by the reported evidence.","major_comments":[{"comment":"The paper reports only absolute Spearman correlations between LLM M_preference scores and human naturalness judgments. This is not sufficient to support the claim that LLMs 'generally align' with humans. Given the definitions in Section 5.1 and the human-rating scale described in Section 7.1, M_preference = Mscore(U) - Mscore(S), with positive values indicating preference for the unshifted form, while the human score is 1 when the first (unshifted, as listed in Table 1) sentence is more natural and 7 when the second (shifted) sentence is more natural. Under genuine alignment, the signed Spearman rho should be negative. Reporting |rho| conflates correctly aligned models (negative rho) with anti-aligned models (positive rho), and the 'low' particle-movement correlations could in fact be active reversals. The abstract's central claim cannot be evaluated from Table 4 as presented. Please report signed correlations with a clear sign convention, or provide scatterplots with axes that make the direction of association visible.","section":"Section 7.2, Table 4"},{"comment":"The dative alternation (DA) stimuli are not a minimal pair: the shifted form 'He sent a gift to her' adds the preposition 'to' compared with the unshifted 'He sent her a gift'. Because Mscore is the unnormalized sum of per-token log-probabilities over sequences of different lengths, the DA M_preference difference necessarily includes a systematic length and lexical confound that is independent of ordering preference. This confound could inflate (or deflate) the reported DA correlations, including the highest human-model correlation in the paper (GPT-2, DA, rho = 0.651 in Table 4). The paper should either normalize by sequence length, use a matched control condition, or explicitly quantify the contribution of the added token to the model score difference before drawing conclusions about DA alignment.","section":"Section 5.1, Table 1"}],"minor_comments":[{"comment":"The human judgment study description is sparse; please report the number of sentence pairs per shift type after exclusions, the number of judgments per pair, the agreement threshold applied, and how the 500 pairs were sampled from the larger synthetic and mined datasets.","section":"Section 7.1"},{"comment":"The note for MPP states that 'the unshifted and shifted form for MPP is ambiguous; the unshifted and shifted forms cannot be derived separately given an example.' This is confusing as written, since the table does provide two distinct rows; please clarify what ambiguity is intended.","section":"Table 1"},{"comment":"The claim that syllable weight is 'often the most important predictor' is based on raw differences in R-squared between the full GAMM and ablated models, without confidence intervals or significance tests; for several rows the differences are small (e.g., Table 3, DA rows), so this claim currently rests on descriptive differences only.","section":"Section 6.2, Table 3"},{"comment":"The textual descriptions ('converge above 0', 'plateau below zero') are not easy to verify from the small excerpt of Figure 5; please ensure the figure is legible with labeled axes and visible trend lines, and consider adding individual data point density.","section":"Section 5.2 and Figure 5"},{"comment":"The statement that Table 4 shows 'relatively high correlation effects' lacks a benchmark or effect-size interpretation; a correlation of 0.3-0.5 may be modest depending on the context, and adding confidence intervals or a comparison with baseline correlations would strengthen the interpretation.","section":"Section 8.3"}],"recommendation":"major_revision","confidential_remarks":"The two major issues are both fixable within the manuscript's scope: reporting signed correlations (with the sign convention made explicit) and controlling for the DA length/lexical confound. The absolute-value reporting is the most serious concern because it genuinely obscures whether the headline result is alignment or, in some cases, anti-alignment. The GAMM significance point is secondary but should also be addressed for the weight-analysis claims to be solid."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a solid, broad empirical follow-up to Futrell and Levy's LSTM work: twelve open models, four shift types, a large synthetic corpus, and fresh human judgments. The finding that instruction-tuned models generally match human naturalness ratings worse than their base counterparts is genuinely interesting and runs against the authors' own prior. The GAMM analysis of which weight measure matters is a nice addition. Code and data are released. That is real value.\n\nBut I have two reservations, and one of them is load-bearing. The stress-test note is right: Table 4 reports absolute Spearman correlations. Since M_preference is positive when the model prefers the unshifted form, and human ratings are 1 for unshifted and 7 for shifted, true alignment should give negative rho. Taking absolute values means an anti-aligned model (positive rho) would look just as \"aligned\" as a correctly aligned one. The authors never report signed correlations, so the central claim \"LLMs generally align with human preferences\" is not actually supported by the table as printed. The scatter plots in the appendix may show the sign, but the headline number is uninterpretable. This needs fixing before the paper can make its main claim.\n\nSecond, the dative alternation pairs are not minimal: the shifted form adds \"to\", so M_preference includes a length and lexical confound. The paper does not normalize by sequence length or control for the added preposition. This could inflate the DA correlations, including the highest one in the table (GPT-2 at 0.651). It's not fatal, but it should be addressed, perhaps by a length-controlled baseline or by scoring only the shared span.\n\nThe \"generally align\" claim is also stated more strongly than the numbers warrant. Correlations range from about 0.2 to 0.65 across models and shift types; many are weak. The abstract's \"largely\" is doing a lot of work. That said, the pattern is consistent for HNPS and MPP, and the PM exception is real.\n\nThe reader's circularity concern is unfounded: nothing is fitted to the human judgments, and the GAMMs are descriptive. The citation pattern is fine.\n\nVerdict: conditional accept. The empirical contribution is worth refereeing, but the absolute-correlation issue is a serious flaw in how the main result is presented. If the authors report signed values and the sign is consistent with alignment for most models, the paper is publishable after minor revisions. I would send it to review.","headline":"A broad, useful empirical follow-up to Futrell and Levy, but the main claim rests on absolute correlations that hide whether models are aligned or anti-aligned.","tokens_in":13924,"tokens_out":2562,"would_cite":true,"duration_ms":24632,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Across three of four constituent-ordering phenomena, LLM preferences correlate with human naturalness judgments; particle movement is the outlier.","keywords":["constituent ordering","heavy NP shift","dative alternation","particle movement","language models","human-model alignment","end-weight","syllable weight"],"falsifier":"Recompute the dative-alternation correlations after normalizing $M_{\\mathrm{score}}$ by sequence length or after matching the two forms on token count; if the reported correlations (for example, GPT-2's 0.651) drop to near zero, the ordering-preference alignment is an artifact of the length confound rather than a shared ordering preference.","tokens_in":12913,"feed_emoji":"🧠","tokens_out":7311,"duration_ms":67837,"temperature":0.7,"pith_summary":"Constituent ordering in English is flexible after the verb, and a prominent theory says heavier constituents tend to come last. This paper tests whether large language models share that tendency by comparing each model's log-probability preference between shifted and unshifted sentence pairs with crowdsourced human naturalness ratings on the same pairs, covering heavy NP shift, dative alternation, multiple PP shift, and particle movement. The central claim is that models largely do align with human preferences: Spearman correlations are positive and moderate (roughly 0.3–0.6) for the first three phenomena, while particle movement is the clear exception. The paper also reports that instruction-tuned models align less with humans than their base counterparts, contrary to the authors' initial hypothesis.","feed_headline":"LLMs share human word-order preferences, except particle movement","feed_subtitle":"Model preferences track human naturalness ratings on three movement types; instruction tuning offers no boost.","key_machinery":"The central object is $M_{\\mathrm{preference}}$, the difference between a model's summed log-probability of the unshifted and shifted versions of the same sentence; a positive value means the model prefers the unshifted order. It is computed over roughly 400,000 synthetic minimal pairs plus mined Penn Treebank-2 sentences, across four movement types. The same design supports a regression analysis: generalized additive mixed models (GAMMs) with verb-level random effects are fit to $M_{\\mathrm{preference}}$ as a function of word length, syllable weight, token length, and modifier weight, and each predictor is ablated in turn to see which weight measure the model's preferences actually depend on. Human naturalness ratings on 500 sentence pairs, collected on a 1–7 scale from 126 native speakers, form the comparison baseline, and the Spearman correlation between mean human scores and $M_{\\mathrm{preference}}$ is the headline agreement statistic.","core_discovery":"The paper claims that, when a sentence can be reordered without changing its truth conditions, a language model's relative preference for the two orders — measured as $M_{\\mathrm{preference}} = M_{\\mathrm{score}}(\\text{unshifted}) - M_{\\mathrm{score}}(\\text{shifted})$, the difference of full-sequence log-probabilities — tracks human naturalness judgments for heavy NP shift, dative alternation, and multiple PP shift, with absolute Spearman correlations typically between 0.3 and 0.6 across the GPT-2, Llama-3, Mistral, OLMo, and BabyLM-style model families. Particle movement is the outlier, with low human-model agreement and a non-monotonic weight effect in which shifting motivation first rises and then falls. The paper further finds that among several weight measures, syllable weight best explains model shifting behavior (word length best for dative alternation), and that instruction-tuned models consistently show lower agreement with human judgments than their base counterparts. These results are presented as evidence that weight-based end-weight ordering principles, long studied in human psycholinguistics, are largely induced by autoregressive language models, while the failure of particle movement marks a genuine boundary.","pith_inferences":["Because the dative-alternation score conflates ordering with the added preposition 'to', the paper's highest human-model correlations are the least clean; a length-normalized re-analysis could revise those numbers.","The instruction-tuning gap suggests a testable prediction for future models: methods that emphasize naturalistic generation or register diversity might restore or improve base-model-level alignment on these small-scale word-order choices.","The same $M_{\\mathrm{preference}}$ protocol could be run on languages with freer post-verbal ordering to see whether human-model agreement holds cross-linguistically, which the paper notes as outside its scope.","If syllable weight genuinely drives model preferences, perturbing syllable counts while holding token counts fixed should move model scores predictably; that experiment would separate a phonological proxy from purely token-level statistics."],"forward_implications":["If model preferences genuinely track human rankings, the same weight-based theory of end-weight ordering applies to how transformer language models evaluate sentences, not just to how humans produce them.","Syllable weight being the best single predictor suggests that model ordering behavior is not purely token-level and may reflect an induced proxy for phonological or prosodic heaviness.","Instruction-tuned models correlating less with humans than base models implies that human-feedback training does not automatically make syntactic micro-preferences more human-like; it can move them away.","Particle movement is a distinct case: human-model agreement is low, and beyond a weight threshold additional weight no longer motivates shifting, so the general end-weight story does not cover all movement types."],"supporting_citations":[{"why":"Supplies the surprisal-based preference metric and the earlier LSTM finding that models track human post-verbal order preferences, which this paper extends to more models and graded weight data.","marker":"Futrell and Levy (2018)"},{"why":"Provides the weight-based theory of post-verbal constituent ordering and the claim that length and modifier weight jointly predict shifts, which this paper tests against LLMs.","marker":"Wasow and Arnold (2003)"},{"why":"Introduces relative weight as the relevant quantity for ordering, motivating the weight-ratio design used throughout the experiments.","marker":"Wasow (1997b)"},{"why":"Human judgment study establishing weight as a primary, ceiling-limited factor in heavy NP shift; it frames the human baseline and the gradient and ceiling discussion.","marker":"Medeiros et al. (2021)"},{"why":"Penn Treebank-2 supplies the naturally occurring sentence pairs mined for the realistic data split.","marker":"Marcus et al. (1995)"},{"why":"Establishes the use of model probability scores as proxies for human acceptability judgments, the methodological premise of the human-model comparison.","marker":"Linzen et al. (2016)"},{"why":"Supports the decision to use raw probability scores rather than prompting instruction-tuned models, protecting the measurement from meta-linguistic prompting artifacts.","marker":"Hu and Levy (2023)"}],"fun_headline_variants":["LLMs match human word-order tastes, except particle shift","Language models mirror human ordering, except particle shift","Human-like constituent order in LLMs, with one exception: particles","LLMs show human-like ordering, but fail on particle movement"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that $M_{\\mathrm{preference}}$ isolates ordering preference; for dative alternation the shifted form adds the word 'to', so the score also changes with length and lexical content, and if that confound is large the reported human-model correlations for dative alternation could overstate genuine ordering alignment.","fun_headline_variants_meta":{"raw":{"variants":["LLMs match human word-order tastes, except particle shift","Language models mirror human ordering, except particle shift","Human-like constituent order in LLMs, with one exception: particles","LLMs show human-like ordering, but fail on particle movement"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000466,"raw_usage":{"total_tokens":2331,"prompt_tokens":956,"completion_tokens":1375,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":1307}},"tokens_in":572,"tokens_out":1375,"duration_ms":10648,"temperature":1.0,"reasoning_tokens":1307,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T18:25:24.032468+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the dative-alternation correlations after normalizing $M_{\\mathrm{score}}$ by sequence length or after matching the two forms on token count; if the reported correlations (for example, GPT-2's 0.651) drop to near zero, the ordering-preference alignment is an artifact of the length confound rather than a shared ordering preference.","supporting_citations":[],"review_version":1}