{"id":"a7a5721a-52c1-410d-b261-3057ee8a1ad5","arxiv_id":"2411.14103","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"NLI benchmarks still discriminate between LLMs of different sizes and are not saturated, while the Jensen-Shannon distance between model and human label distributions shrinks with scale.","lead":"This paper tests whether older natural-language-inference benchmarks still tell us anything about modern large language models, and finds that they do: the benchmarks separate strong models from weaker ones and track progress during training. It also reports that model probability distributions grow closer to human label distributions as models increase in scale, though the gap remains larger than the gap between two groups of humans.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The JSD result in §4.5 rests on an unspecified softmax-over-letters proxy for label distributions; if option-position bias differs by scale, the scale effect could be an artifact.","rationale":"The reader's weakest-assumption analysis correctly flags the single from-scratch pretraining run per scale as a threat to the training-dynamics claims in §4.2 and Figure 5. That is a real limitation, especially because the custom datamix is unreleased and no error bars are provided. However, the paper's most original and headline-worthy finding is the scale-sensitive JSD between model and human label distributions. That claim rests on a measurement whose construction is not specified in the methods: §3.3 describes only NLL-based choice prediction over option letters, yet §4.5 reports probability distributions over the three semantic labels. If the model distribution is a softmax over three single-letter NLLs, it is not obviously comparable to the human annotation distributions from ChaosNLI, and the comparison could be dominated by prompt format, tokenizer behavior, or option-position bias. This is a load-bearing methodological gap that the paper does not address with any control analysis. I do not reject the paper because the concern is addressable and the accuracy-based discriminability results are plausibly robust. The verdict should remain conditional, with the condition made explicit: the authors must specify and validate the model-distribution computation for JSD. I partially agree with the reader because their identified weakness is valid but, in my reading, secondary to the JSD proxy issue for the paper's central new claim.","tokens_in":13150,"tokens_out":4675,"duration_ms":45197,"concrete_test":"Recompute JSD for the ChaosNLI subsets using (a) softmax over option letters as in the paper, (b) softmax over full label strings ('Entailment'/'Neutral'/'Contradiction'), and (c) permuted label orders (e.g., B/C/A) for a subset of models and training checkpoints. If the ranking of models by JSD or the training-curve trend in Figure 5 changes materially—for example, the scale effect reverses or the decrease during training disappears—the central claim is not robust; if all three variants agree, the concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.5's central finding—that model–human JSD decreases with scale and training—requires a model probability distribution over the three labels, but the paper never specifies how this distribution is obtained. The evaluation setup in §3.3 computes only the NLL of the option letter ('A'/'B'/'C'), and the natural reading is that JSD uses a softmax over these three letter NLLs. This is not a validated stand-in for a label distribution: it ignores probability mass of the full answer string, is sensitive to tokenization of single letters, and is vulnerable to position/letter bias (models often prefer 'A' regardless of content). Since the prompt orders labels as 'A. Entailment', 'B. Neutral', 'C. Contradiction', any systematic letter prior will inflate or deflate JSD relative to human distributions in a way that could correlate with model scale. The paper reports no checks with permuted option orders or with full-label scoring. Without this, the headline JSD result—the most novel claim in the abstract—could be an artifact of the choice proxy rather than evidence about human-aligned distributions. This concern is independent of the single-run training issue: even the final-model scale effect in Figure 4b depends on this proxy.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether NLI benchmarks remain informative for evaluating large language models. It evaluates five NLI benchmarks (SNLI, MNLI, HANS, ANLI, αNLI) on six base models (Llama-3.1 8B/70B/405B and Mistral 7B, Mixtral 8x7B/8x22B) using a multiple-choice NLL scoring setup. The reported results are that accuracy discriminates model scale, few-shot examples are needed for reasonable performance, accuracy develops during pre-training (with low monotonicity for the 8B model), contamination does not meaningfully inflate scores, and errors on the strongest model often coincide with human disagreement. Using ChaosNLI, the paper compares model probability distributions with human label distributions via Jensen-Shannon divergence (JSD), reporting that JSD decreases with scale and during training, although model-human JSD remains substantially above human-human JSD. The central recommendation is that NLI benchmarks, and especially distributional agreement with human labels, remain useful signals for LLM evaluation.","tokens_in":13352,"tokens_out":4955,"duration_ms":48410,"significance":"If the JSD finding is robust, it provides a potentially practical training-monitoring statistic and a way to quantify alignment with human label variation, directly extending the earlier ChaosNLI result that model-human JSD did not improve with scale. The discriminability, few-shot, and contamination analyses are useful and would help practitioners decide whether to include NLI benchmarks in evaluation suites. The paper makes good use of public benchmarks and an established contamination methodology, and the manual error analysis connects accuracy saturation to genuine label ambiguity rather than model deficiency. However, the headline JSD claim currently depends on an unspecified and unvalidated proxy for model label distributions, and the training-dynamics claims rest on single from-scratch runs with a custom unreleased datamix. These issues need to be resolved before the central conclusions can be taken at face value.","major_comments":[{"comment":"The stress-test concern about the JSD computation lands. Section 4.5 reports Jensen-Shannon divergences between model softmax distributions and human label distributions, but the paper never states how the model distribution over labels is obtained. The evaluation in §3.3 computes only the negative log-likelihood of the option letter ('A'/'B'/'C'), so the natural reading is that JSD is computed from a softmax over these letter NLLs. This proxy is not validated: it ignores probability mass of the full answer string, is sensitive to tokenization of single letters, and is vulnerable to position and letter priors. Because the prompt template in Table 5 always orders the options as A. Entailment, B. Neutral, C. Contradiction, any systematic letter preference can inflate or deflate JSD in a way that correlates with scale. The paper reports no checks with permuted option orders or with full-label scoring. Please specify the exact distribution used and add such checks; without them, the headline scale and training effects in Figures 4b and 5 could be artifacts of the proxy rather than evidence about human-aligned distributions.","section":"§4.5, with §3.3 and Table 5"},{"comment":"The training-dynamics and JSD-during-training claims rest on a single from-scratch pre-training run each for the 8B and 70B models, stopped at 2T tokens with the authors explicitly noting that they did not have the budget for a full run. Appendix D describes the data only as a custom mix of public web, code, and reasoning data, without sufficient detail for replication. One run per scale provides no estimate of run-to-run variance, and the unreleased datamix makes it impossible for others to assess how representative the curves are. Please either add multiple runs or seeds, use checkpoints from a released model series that span training, or clearly reframe these sections as a single-configuration case study and soften the general conclusion that JSD decreases steadily during training.","section":"§4.2, Figure 5, Appendix D"}],"minor_comments":[{"comment":"The sentence listing the benchmarks says 'five different NLI benchmarks – αNLI, ANLI, HANS, and MNLI', omitting SNLI; the list should include all five.","section":"Conclusion"},{"comment":"The text refers to 'the three possible labels Entailment, Neutral, and Contradiction', but αNLI is a two-choice task and the ChaosNLI αNLI subset presumably has two labels; clarify how the JSD is computed for the binary tasks.","section":"§4.5"},{"comment":"The x-axis labeled 'shots' should be made explicit as 'number of few-shot examples', and the number of examples used for each shot setting should be stated in the caption or main text rather than only in a footnote.","section":"Figure 1 and footnote 3"},{"comment":"The manual error analysis covers 40 predictions from a single model (Llama-3.1 405B) on MNLI only; the generalization that 'most of the mistakes in MNLI' are human-disagreement cases should be hedged accordingly.","section":"§4.4"},{"comment":"The phrase 'contamination does not play a participatory role' is unclear; use 'does not play a significant role' or similar, and fix the typo 'indicitive'.","section":"§4.3"},{"comment":"The sentence 'the benchmarks MNLI and RTE and RTE were used' appears to have a duplicated phrase; remove the repetition.","section":"§2.1"}],"recommendation":"major_revision","confidential_remarks":"The paper has good breadth and uses public resources well, but the central JSD claim needs a precisely specified and validated distribution before publication. The authors should be encouraged to release the JSD computation code and to add option-order permutation checks; the training-dynamics section may need to be repositioned as a single-run case study unless additional evidence is provided. The scope is appropriate for a venue that welcomes empirical analysis of evaluation methodology."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read of the NLI paper. The genuinely new thing is the JSD result: larger models and more training make model label distributions closer to human label distributions, contradicting Nie et al. (2020b). That is the result to talk about, and also the one I'd want to verify before betting on it. The paper never says how the model distribution is computed in §4.5. The evaluation setup only computes NLL over the option letter ('A'/'B'/'C'), so the natural reading is that JSD uses a softmax over those three letter scores. That proxy ignores the probability mass of the actual answer text and is exposed to letter-position bias. The prompt always orders 'A. Entailment, B. Neutral, C. Contradiction'. If models differ in their prior preference for 'A' across scales, the scale effect could be an artifact. No permuted option-order check, no full-label scoring reported. This is fixable, but it is load-bearing for the paper's most novel claim.\n\nWhat the paper does well: the accuracy analysis is solid. NLI benchmarks do discriminate between Llama and Mistral models of different sizes, most are not saturated, ANLI stays hard. The contamination audit using EPG is careful and shows no real score inflation. The from-scratch 8B/70B training curves are new, even if the monotonicity numbers show these benchmarks are not great for closely spaced checkpoints. The manual error analysis on Llama 405B is small but illustrative: many 'errors' are cases where humans also disagree.\n\nSoft spots are mostly the usual ones. One training run per scale, stopped at 2T tokens because of budget, on a custom datamix that is not released—so the training-dynamics and JSD-over-training conclusions rest on a single trajectory. No error bars or significance tests anywhere, so some apparent scale effects could be within-run noise. The manual analysis covers 40 examples. No code or data released, which makes the robustness checks I'd want impossible to run myself.\n\nThe circularity concern is minor: the monotonicity metric and contamination method come from overlapping-author papers, but those are measurement tools, not the results being derived.\n\nOverall, the main accuracy claims hold up. The JSD scale effect is promising but needs a couple of robustness checks before I'd trust it. I'd send this to peer review and ask the authors to specify the distribution, try permuted option orders, and report variance. Also temper the training-dynamics conclusions to match the single-run evidence.","headline":"The JSD scale effect is the paper's real news, but the underspecified model distribution behind it makes that finding the one to verify before betting on it.","tokens_in":13945,"tokens_out":4024,"would_cite":true,"duration_ms":34926,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"NLI benchmarks still separate strong from weak LLMs, and model–human agreement on ambiguous cases improves as models scale.","keywords":["natural language inference","LLM evaluation","benchmark saturation","Jensen-Shannon divergence","ChaosNLI","human label distributions","model scale","training dynamics"],"falsifier":"If a larger model evaluated on ChaosNLI subsets showed a JSD no lower than a smaller model's, or if JSD rose during later training while accuracy kept improving, the claim that JSD is a scale-sensitive evaluation signal would be falsified. A complementary check would be to pre-train an 8B and a 70B model on a standard public data mixture beyond 2 trillion tokens to see whether NLI accuracy still rises late in training.","tokens_in":12933,"feed_emoji":"📊","tokens_out":6217,"duration_ms":55247,"temperature":0.7,"pith_summary":"Natural language inference benchmarks were once the standard way to test whether a model understands language, but they have largely vanished from LLM evaluation suites. This paper asks whether that abandonment was premature, and answers yes: across five NLI benchmarks and six models from the Llama and Mistral families, accuracy separates models of different size and quality, develops over pre-training, and is not inflated by training-data contamination. The paper also takes a distributional view, using ChaosNLI's 100-way human labels to measure the Jensen-Shannon divergence between a model's softmax distribution and the human label distribution. It finds that this divergence decreases with scale and during training, yet even the largest models remain much further from human labels than two human populations are from each other. That gap is the paper's main new signal: NLI evaluation still has headroom, and model–human divergence is a statistic worth tracking.","feed_headline":"NLI benchmarks still separate strong from weak LLMs","feed_subtitle":"Across five inference tasks, model–human agreement improves with scale and during training.","key_machinery":"The central object is the Jensen-Shannon divergence (JSD) between a model's softmax distribution over NLI labels and the human label distribution recorded in ChaosNLI, a benchmark supplying 100 human annotations per example for subsets of MNLI, SNLI, and αNLI. JSD is a symmetric, bounded (0 to 1) distance between two probability distributions, chosen because, unlike KL divergence, it treats the two distributions symmetrically. The paper also relies on a choice-based evaluation protocol (computing negative log-likelihood over letter options) and on the monotonicity statistic (Kendall tau against an idealized increasing curve) to judge whether a benchmark can track training progress.","core_discovery":"NLI tasks still provide a discriminating, non-saturated, and contamination-free signal for evaluating LLMs. Concretely, on ANLI, even the largest models remain below 70% accuracy, and across MNLI, SNLI, and HANS, accuracy gaps between 8B, 70B, and 405B models are substantial, while αNLI saturates early. A second discovery is distributional: when a model's softmax probabilities over entailment, neutral, and contradiction are compared with the full human label distribution (100 annotations per example from ChaosNLI), the Jensen-Shannon divergence is far lower than chance or earlier BERT-scale models, and — contrary to earlier findings — it decreases steadily both with model scale and over the course of pre-training. The paper interprets this as evidence that NLI benchmarks remain informative for model development and that the JSD between model and human labels is a promising, scale-sensitive evaluation statistic.","pith_inferences":["An implication the authors leave implicit: if JSD to human labels keeps falling with scale, it could become a training-time monitoring signal or even a target, nudging models toward calibrated uncertainty rather than majority-label accuracy.","The authors note the jury idea in passing; a consequence they do not spell out is that a panel of LLM judges may share systematic biases, so jury–human agreement should be checked against human label distributions before using panels for evaluation.","Because the training dynamics rest on one custom datamix, a natural test is to re-run the 8B and 70B curves on a standard public mix; the paper's monotonicity table suggests NLI accuracy may not be reliable for checkpoint selection at small scales regardless.","A testable extension: if JSD is a genuine quality signal, models fine-tuned or post-trained to minimize JSD on ChaosNLI should also become better at selective prediction and calibration under distribution shift."],"forward_implications":["NLI benchmarks still separate models of different size and quality: on ANLI the best models stay below 70% accuracy, while MNLI and SNLI show clear gaps across the Llama and Mistral families.","Performance on these benchmarks develops during pre-training, so they can track long-horizon training progress, with the caveat that their monotonicity at 8B scale is low.","Contamination does not inflate the scores: 8-gram overlap analysis shows virtually no estimated performance gain from contaminated examples.","On ambiguous examples, model disagreement with the human majority label is largest when human entropy is high, and models align more with the majority label as they scale.","The JSD between model softmax distributions and human label distributions decreases with both scale and training time, yet stays far above human–human JSD, so there is headroom for improvement."],"supporting_citations":[{"why":"Supplies ChaosNLI, the 100-annotator label distributions, and the JSD baseline analysis that this paper extends and partially contradicts.","marker":"Nie et al. (2020b)"},{"why":"Reported low GPT-3 scores on ANLI, the key prior result that motivated questioning whether NLI is useful for LLMs.","marker":"Brown et al. (2020)"},{"why":"Provides MNLI, one of the two main benchmarks for the error analysis and majority-label correction experiments.","marker":"Williams et al. (2018)"},{"why":"Provides HANS, the synthetic adversarial benchmark testing whether models rely on shallow lexical and syntactic heuristics.","marker":"McCoy et al. (2019)"},{"why":"Provides ANLI, the hardest benchmark in the study, on which even the largest models stay under 70%.","marker":"Nie et al. (2020a)"},{"why":"Provides αNLI, the abductive-reasoning benchmark that the paper finds near-saturated at 70B.","marker":"Bhagavatula et al. (2020)"},{"why":"Supplies the monotonicity metric (rank correlation against a monotone curve) used to quantify training-signal quality.","marker":"Madaan et al. (2024)"},{"why":"Provides the Llama 3 model family, the pre-training hyperparameter settings, and the prompt-robustness evidence for the choice-based evaluation protocol.","marker":"Dubey et al. (2024)"},{"why":"Supplies the ConTAM contamination-scoring methodology and the estimated performance gain measure used to test data leakage.","marker":"Singh et al. (2024)"},{"why":"Provides SNLI, the first large-scale NLI dataset and one of the benchmarks central to the accuracy-versus-entropy analysis.","marker":"Bowman et al. (2015)"}],"fun_headline_variants":["NLI tasks still expose LLM quality gaps","NLI signals not saturated, still separate LLMs","LLM-human inference agreement grows with scale","NLI benchmarks: discriminative and scale-sensitive","Inference tasks still reveal model differences"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training-dynamics and JSD-vs-scale conclusions rest on a single from-scratch pre-training run for each of two model sizes, stopped at 2 trillion tokens using a custom, unreleased data mixture; if that run is not representative of typical pre-training, the curves in Figures 2 and 5 may not generalize.","fun_headline_variants_meta":{"raw":{"variants":["NLI tasks still expose LLM quality gaps","NLI signals not saturated, still separate LLMs","LLM-human inference agreement grows with scale","NLI benchmarks: discriminative and scale-sensitive","Inference tasks still reveal model differences"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000822,"raw_usage":{"total_tokens":3579,"prompt_tokens":908,"completion_tokens":2671,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":2602}},"tokens_in":524,"tokens_out":2671,"duration_ms":20269,"temperature":1.0,"reasoning_tokens":2602,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:31:51.305075+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If a larger model evaluated on ChaosNLI subsets showed a JSD no lower than a smaller model's, or if JSD rose during later training while accuracy kept improving, the claim that JSD is a scale-sensitive evaluation signal would be falsified. A complementary check would be to pre-train an 8B and a 70B model on a standard public data mixture beyond 2 trillion tokens to see whether NLI accuracy still rises late in training.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides αNLI, the abductive-reasoning benchmark that the paper finds near-saturated at 70B."}],"review_version":1}