{"id":"87138829-cd5c-438f-9563-e22a08c83131","arxiv_id":"2608.08227","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Humans and Llama 3.3 70B both rate 'only' sentences higher than 'even' sentences, but the model's responses are uniformly extreme and lack human-like variability.","lead":"A behavioral study compared how humans and a large language model rate ability from sentences using 'even' and 'only', finding that both produce stable differences but the model is more extreme and less variable. The result suggests that similar aggregate outputs from humans and LLMs may come from different underlying mechanisms, which matters for using language models as models of human cognition.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The LLM's zero response variability may be an artifact of the single-integer response prompt; without a token-probability or continuous-output control, the claim that its mechanism differs from humans is not yet established.","rationale":"I agree with the reader that the weakest assumption is the faithfulness of the LLM prompt. The paper's central claim has two legible parts: (1) humans and the LLM both track the even/only scalar difference across response configurations, and (2) the LLM's lack of response variability indicates a different underlying mechanism. Part (1) is reasonably supported by the reported particle main effect and follow-up comparisons, subject to the usual missing-data and code-availability caveats. Part (2) is the load-bearing inference, and it is not secured by the current design. Repeated sampling at temperature 1.0 tests the variance of the model's output distribution under a fixed response format; it does not test whether the discrete single-integer format itself forced the model into a single canonical answer. The temperature sweep broadens the sampling temperature but not the output space or the task framing, so it cannot separate internal concentration from instruction-following bias. A concrete check using token logits, a continuous response scale, or distributional output would settle this. If the degeneracy persists across formats, the author's conclusion is strengthened; if it does not, the 'different mechanism' conclusion becomes a claim about prompt sensitivity rather than about cognitive representations. This concern does not change the overall verdict from CONDITIONAL, because either way the paper needs an additional control before the central claim can be accepted as stated.","tokens_in":3295,"tokens_out":4619,"duration_ms":46630,"concrete_test":"Use a local checkpoint of Llama 3.3 70B and, for the same 60 sentences, compute the full token probability distribution over the five integer outputs under the original single-integer instruction. Then repeat with (a) a continuous 0--100 rating prompt and (b) a prompt asking for a probability distribution over the five labels. If the modal label's probability remains above 0.99 in all formats, the paper's variability-based mechanism claim is supported; if the continuous or distributional formats produce non-degenerate variance, the reported lack of variability is partly a response-format artifact and the mechanism claim needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central inference that the LLM's underlying mechanism differs from humans rests entirely on its lack of response variability under repeated sampling at temperature 1.0. This inference is valid only if the observed degeneracy reflects the model's internal judgment distribution rather than the instruction to 'respond with a single integer from 1 to 5.' A prompt that demands one discrete integer can induce a collapse onto a canonical label even when the model's internal representation is graded, particularly because the model is optimized to follow instructions and produce a single best answer. The reported temperature sweep from 0.0 to 2.0 does not address this concern: it varies sampling randomness but not the output constraint, so it cannot distinguish a genuinely concentrated internal distribution from a response-format artifact. The conclusion that matching aggregate ratings is insufficient evidence for shared cognitive representations would survive only if the lack of variability persists when the model is allowed to express graded uncertainty, such as through token probabilities or a continuous response scale.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This extended abstract reports a cross-system comparison of scalar ability judgments for sentences containing the focus particles 'even' and 'only'. Human participants (N=108) and Llama-3.3-70B rated 60 sentences on 5-point Likert scales under four response-scale configurations that varied spatial format (horizontal/vertical) and label mapping (standard/reversed). Both humans and the LLM rated 'only' sentences higher than 'even' sentences, with the LLM producing more extreme and less variable responses. The authors interpret the stable particle effect as evidence for semantic/evaluative rather than spatial-format-driven judgments, and the LLM's lack of response variability as evidence that its underlying mechanism differs from humans'.","tokens_in":3429,"tokens_out":3199,"duration_ms":31674,"significance":"If the central claim is established, the paper contributes a useful cautionary result: aggregate matching of average human and LLM ratings does not entail shared cognitive representations. The repeated-sampling design and the temperature-sweep robustness check are sensible probes of model variability, and the explicit reporting of a null result for spatial-format reversal is a strength in transparency. However, the significance is moderate because the key inference about mechanism difference currently rests on a single, potentially confounded observation, and because several statistical and design details are insufficiently specified.","major_comments":[{"comment":"The inference that the LLM's lack of variability reflects a different underlying mechanism is not yet established because the only response format used was a prompt asking the model to 'respond with a single integer from 1 to 5.' A discrete-output instruction can collapse a graded internal distribution onto a canonical label, even when the model's token probabilities are diffuse. The temperature sweep (in the Discussion) does not address this concern because it varies sampling randomness while keeping the same single-integer output constraint; it cannot distinguish a genuinely concentrated distribution from a response-format artifact. The authors should either report the token-level probabilities of the rating options from the model's softmax (e.g., for '1' through '5' in each condition) or add a control condition that allows graded responses (e.g., a continuous slider or a distribution output). Without such evidence, the mechanism-difference claim is unsupported.","section":"Method / Results"},{"comment":"The paper claims that spatial configuration did not affect scalar judgments, but it never reports a statistical test of the Format or Mapping factors or their interactions with Particle. The only reported effects are the main effect of Particle and the Particle×Source interaction. The statement that changing orientation or label mapping 'did not eliminate or reverse this pattern' is not a substitute for a formal test; an absence of reversal is compatible with a small but real spatial effect. The authors should report the full fixed-effects table for the mixed model, including main effects of Format and Mapping and all two- and three-way interactions, and ideally model comparisons (e.g., with vs. without spatial terms) to support the claim that scalar judgments are driven primarily by semantic structure.","section":"Results"},{"comment":"The abstract states that the dataset contains 'approximately 100 items,' but the Method section says 60 sentences (30 with 'even' and 30 with 'only'). This discrepancy is not cosmetic: the item sample size directly affects statistical power and the generality of the item-level random effects. The authors must reconcile the count and report the exact number of items used in each condition.","section":"Abstract / Method"},{"comment":"The temperature sweep is described only in the Discussion ('a follow-up temperature sweep on a subset of items') and no results are reported for it. Since this negative result is load-bearing for the claim that sampling noise cannot produce human-like variability, the authors should move the description to the Method and provide the number of items, the temperatures tested, the variability metric (e.g., standard deviation or entropy of sampled responses), and the actual stability values across temperatures. Without these details, the claim that 'increasing temperature alone was insufficient' is not verifiable.","section":"Discussion"}],"minor_comments":[{"comment":"The design lacks a neutral control condition without 'even' or 'only' (e.g., 'Mike can bake a cake' with no particle). Such a baseline would help interpret the absolute ratings and confirm that the effect is specifically due to the particles rather than to some other aspect of the ability statements.","section":"Method"},{"comment":"The reference for Llama 3.3 70B is given as Touvron et al. (2023), which describes the original Llama architecture, not Llama 3.3. The authors should cite the correct model card or a technical report for Llama 3.3.","section":"References"},{"comment":"The sentence about transformer-based language models processing 'textual input tokens simultaneously through bidirectional attention' is inaccurate for Llama, which is an autoregressive, left-to-right model. The discussion of spatial orientation may still hold, but the architectural description should be corrected.","section":"Introduction"},{"comment":"The assignment of the 108 human participants to the four response-scale conditions is not specified; it should state whether participants were between-subjects assigned to one condition each and how items were randomized across trials.","section":"Method"},{"comment":"The figure caption states that error bars are 95% CIs and are absent for the LLM in 'only' conditions, but the figure itself is not included in the manuscript. Please ensure the figure is present and that the caption describes the aggregation units (participant-level vs. item-level CIs).","section":"Results / Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short extended abstract and, as such, currently lacks the statistical detail expected of a full journal article. The central idea is interesting, but the mechanism-difference claim is confounded by the response-format issue; I would want to see either token-probability evidence or a continuous-output control before endorsing the conclusion. The missing spatial-effect test is also easily fixable with the existing data. I recommend major revision rather than rejection because the core empirical observation (the particle effect in both humans and the LLM, and the LLM's extreme responses) is plausible and worth reporting properly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a small but clean extended abstract showing that Llama 3.3 70B reproduces the human aggregate pattern for 'even' vs 'only' ability ratings across four response-scale formats, while producing zero response variability. That last point—the LLM's degenerate output distribution—is the interesting bit, but it is also the part that is not fully supported.\n\nWhat is actually new: the specific combination of focus particles and response-scale format manipulation has not been tested on an LLM before, and the between-source comparison is executed straightforwardly. The linear mixed-effects analysis is appropriate for the repeated-measures structure, and the reported statistics support the main effects. The temperature sweep is a sensible robustness check and the authors are honest that temperature didn't induce human-like variability. Good.\n\nThe soft spots are real but not fatal. The abstract says 'approximately 100 items' while the Method says 60 sentences; that needs fixing. There is no neutral baseline, so we cannot tell whether both particles shift away from a neutral midpoint or only one does; this limits the interpretation of the 'evaluative' account. The bigger concern, flagged also by a stress-test, is the prompt: the LLM was instructed to 'respond with a single integer from 1 to 5.' That instruction can easily collapse a graded internal distribution onto a canonical label—particularly when the model is trained to follow instructions—so the lack of variability might be a response-format artifact, not a representational property. The temperature sweep doesn't answer this because it only varies sampling randomness, not the output constraint. The authors stop short of claiming a hard mechanistic difference; they say the result 'raises questions,' which is appropriately tentative. Still, the conclusion that aggregate matching is insufficient evidence for shared representations would be on firmer ground if the model were given a way to express graded uncertainty, e.g., via token probabilities or a continuous scale.\n\nThe citation pattern is fine; the prior work on LLM variability is acknowledged. This paper is for readers working on LLMs as cognitive models and on scalar semantics. It is a preliminary result, but the empirical design is sound enough to deserve referee time. I would send it to review and ask for the item-count fix, a neutral baseline, and either a control condition or a toned-down claim about the mechanism.","headline":"A compact extended abstract with a clean aggregate result and a real confound in the LLM variability claim; worth a round of revision, not a desk reject.","tokens_in":3958,"tokens_out":2500,"would_cite":false,"duration_ms":24121,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A large language model reproduces the average human pattern of scalar judgments for 'even' and 'only', yet its complete lack of response variability reveals that matching aggregates is not evidence for shared cognitive representations.","keywords":["focus particles","scalar inferences","even","only","large language models","response variability","polarity correspondence","human-LLM comparison"],"falsifier":"Check the model's token-level probability distribution over the five response integers before any forced decoding: if 'only' items place substantial probability mass on ratings other than 5 even though every sampled integer is 5, then the categorical rating is an artifact of the response format, and the paper's mechanism-difference conclusion loses its main evidence.","tokens_in":3084,"feed_emoji":"🧠","tokens_out":7426,"duration_ms":61335,"temperature":0.7,"pith_summary":"The paper asks whether humans and a large language model build the same kind of scalar representations when reading sentences with the focus particles 'even' and 'only.' In a rating study with 60 sentences presented under four response-scale layouts, both humans and the LLM reliably rated 'only' sentences higher than 'even' sentences regardless of scale orientation or label mapping, supporting an evaluative, semantically driven account over a spatial-coding account. The paper argues, however, that the LLM's aggregate agreement is misleading: the model produced maximally extreme ratings with zero variability across repeated samples, even when sampling temperature was swept from 0.0 to 2.0. The conclusion is that matching average human judgments does not establish shared cognitive representations, because the LLM lacks the individual-level variability that is a defining feature of human behavior.","feed_headline":"LLM matches human 'even' and 'only' ratings but never varies","feed_subtitle":"Average agreement is not enough: the model's zero response spread signals a different mechanism.","key_machinery":"The study's machinery is a paired contrast between two focus particles—'even,' which highlights an unexpected or low-probability alternative, and 'only,' which restricts the set of valid alternatives—used to elicit scalar ability ratings across four response-scale configurations (horizontal or vertical layout, standard or reversed label mapping). The argument separates two rival accounts: spatial coding, which predicts ratings should shift with the physical arrangement of the scale, and the polarity correspondence principle, which predicts stable particle differences because conceptual polarity aligns with response polarity. The decisive diagnostic is response variability: human ratings are graded across individuals, whereas the LLM's repeated samples are nearly constant, so the spread of responses—not the mean—carries the claim that the model's mechanism differs.","core_discovery":"The central claim is that sensitivity to the scalar implications of 'even' and 'only' is stable across response-scale configurations in both humans and an LLM, but that the two systems arrive at this stability through different mechanisms. Human participants rated people described with 'even' at 2.07 on average and with 'only' at 4.31, while Llama 3.3 70B rated them at 1.41 and 5.00, respectively; the differential was larger for the model (3.58 vs. 2.20) and every 'only' trial received the scale maximum. Reversing or rotating the response scale did not reverse the pattern, which the paper reads as evidence that scalar judgments are driven by the evaluative content of the particles rather than by spatial magnitude coding. The absence of response variability—confirmed by a temperature sweep from 0.0 to 2.0—leads the authors to conclude that repeated samples from a single model reflect noise around one fixed judgment, not the genuine individual differences seen across people, and therefore that aggregate human-model agreement is not sufficient evidence for shared representations.","pith_inferences":["A direct extension the paper does not run: ask the model to output a full probability distribution over the 1-5 scale, or to answer with a continuous slider, and compare the spread to human ratings. If spread appears under those formats, the zero-variability result would be a response-format artifact rather than a representational difference.","The stability of LLM judgments across spatial layouts is plausibly trivial given transformers lack embodied spatial history; a sharper test would manipulate the set of alternatives in the linguistic context (e.g., replacing 'only' with 'even' in identical frames) and see whether the model's categorical response shifts.","The paper's logic implies that human gradedness is itself informative: individual differences in how strongly 'only' raises ability ratings are real variation, so future model evaluations should target the distribution of human responses, not a normative single judgment."],"forward_implications":["If the paper's reading is right, scalar judgments about 'even' and 'only' are governed by semantic/evaluative structure, not by the spatial layout of a response scale.","Aggregate benchmarks that compare only average human and LLM ratings can overstate cognitive similarity; response distributions must be compared as well.","Sampling one language model many times at high temperature does not reproduce human individual differences, so such sampling is not a substitute for collecting human data.","Temperature adjustment alone will not make current LLMs produce human-like graded judgments when the model's probability mass is already concentrated on one response.","Claims that LLMs mirror human semantic representations need to be checked against human-like variability, not just matched means."],"supporting_citations":[{"why":"Sets up the spatial-coding account that the scale-layout manipulation is designed to test.","marker":"Dehaene et al., 1993"},{"why":"Supplies the polarity correspondence principle that predicts stable particle differences across configurations, which the results support.","marker":"Proctor & Cho, 2006"},{"why":"Provides the transformer architecture rationale for why spatial scale orientation may not affect LLMs the way it affects humans.","marker":"Vaswani et al., 2017"},{"why":"Identifies the Llama 3.3 70B model used as the LLM participant.","marker":"Touvron et al., 2023"},{"why":"Documents that LLMs can match average human performance while failing to reproduce human behavioral variability, the pattern this study extends.","marker":"Qiu et al., 2025"},{"why":"Motivates the follow-up temperature sweep by examining LLM behavior under temperature variations.","marker":"Rodriguez et al., 2026"},{"why":"Motivates the research question by showing LLMs develop structured internal linguistic representations.","marker":"Kryvosheieva et al., 2025"}],"fun_headline_variants":["Model and humans agree on 'even'/'only' means, not variability","LLM's fixed scalar judgments signal a different mechanism than humans","Zero variance in LLM responses despite human-like averages","Humans vary, model doesn't, in scalar 'even'/'only' judgments"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion rests on the assumption that the instruction to 'respond with a single integer from 1 to 5' captures the same judgment humans made; if the discrete-response prompt itself collapsed the model's uncertainty onto scale endpoints, then the observed mechanism difference would be an artifact of task framing rather than a genuine representational difference.","fun_headline_variants_meta":{"raw":{"variants":["Model and humans agree on 'even'/'only' means, not variability","LLM's fixed scalar judgments signal a different mechanism than humans","Zero variance in LLM responses despite human-like averages","Humans vary, model doesn't, in scalar 'even'/'only' judgments"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001101,"raw_usage":{"total_tokens":4553,"prompt_tokens":867,"completion_tokens":3686,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":3609}},"tokens_in":483,"tokens_out":3686,"duration_ms":21124,"temperature":1.0,"reasoning_tokens":3609,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:13:58.289818+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check the model's token-level probability distribution over the five response integers before any forced decoding: if 'only' items place substantial probability mass on ratings other than 5 even though every sampled integer is 5, then the categorical rating is an artifact of the response format, and the paper's mechanism-difference conclusion loses its main evidence.","supporting_citations":[],"review_version":1}