{"id":"aec3f6c6-6237-44e9-a196-7caead9a96d1","arxiv_id":"2509.10417","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"On ASAP 2.0, long-context transformer and state-space models all reached quadratic weighted kappa above the 0.745 human-human baseline, led by Longformer at 0.798 and Mamba at 0.797.","lead":"placeholder","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No evidence that long context matters: 512-token DeBERTa matches long-context models, and no full-vs-truncated comparison is provided.","rationale":"The reader's weakest assumption correctly identifies the missing length distribution as a necessary condition for the long-context motivation. I agree that Table 1's mean word counts are insufficient. However, the more directly load-bearing problem is that the experimental design never isolates context length as a variable. DeBERTa-Base, a 512-token model, performs statistically indistinguishably from the long-context models in the reported QWK values, and no full-versus-truncated comparison is run for any single architecture. This means the paper's own evidence undermines its abstract's validity claim about truncation. This concern does not change the verdict: the paper remains CONDITIONAL, because the missing controlled comparison and significance testing are exactly the conditions that would need to be added. I mark agreement as partial because the reader focused on the length distribution rather than the controlled comparison, though both point to the same underlying gap: the data do not demonstrate that long contexts are needed or beneficial for AES.","tokens_in":9116,"tokens_out":4116,"duration_ms":36527,"concrete_test":"Fine-tune Longformer (or ModernBERT) twice with identical hyperparameters and seeds: once on full-length ASAP 2.0 essays and once on the same essays truncated to 512 tokens. Report QWK with bootstrap confidence intervals for both, and also report token-length percentiles (fraction >512, >1024, >2048) using the model's tokenizer. If full-length QWK is not significantly higher than truncated QWK, or if fewer than about 10% of essays exceed 512 tokens, the central 'long context matters' claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that long-context architectures are viable for AES, motivated by the assertion that 512-token truncation undermines scoring validity (§1, Abstract). Table 2 in §4 provides no direct test of this. The only 512-token model, DeBERTa-Base, achieves QWK 0.790, within 0.008 of the best long-context model (Longformer 0.798) and 0.007 of Mamba (0.797), with no confidence intervals or significance tests. No experiment compares the same model on full versus truncated inputs, so architecture and pretraining differences are fully confounded with context length. Table 1 reports only mean word counts (376.1 train, 342.7 test), not the length distribution; if most essays fall below 512 tokens, the long-context premise is vacuous. Section 5 concedes the goal 'should not be about getting higher agreement,' yet the only outcome reported is QWK; the asserted validity benefit for organization is never measured. The paper supports the narrow claim that several models beat the human baseline, but not the load-bearing claim that long contexts matter for AES.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates several pretrained language models for automated essay scoring on the ASAP 2.0 dataset, arguing that long-context architectures are needed because 512-token truncation may undermine scoring validity. It compares DeBERTa-Base (512 tokens), XLNet-Base, Longformer, ModernBERT, Mamba-130m, and Llama-3.2-8B, reporting quadratic weighted kappa (QWK) against a human-human agreement of 0.745. All models reportedly surpass this baseline, with Longformer highest at 0.798 and Mamba-130m at 0.797. The discussion claims that long-context models are viable for AES and that state-space models offer efficiency advantages, while conceding that the goal of long-context modeling 'should not be about getting higher agreement.'","tokens_in":9313,"tokens_out":4050,"duration_ms":35878,"significance":"If the central long-context claim were established, this would be a useful benchmark for AES on a modern, long-essay dataset and a notable demonstration that a 130M state-space model competes with much larger attention-based models. The paper has several strengths: it covers a diverse set of architectural families, uses a recent dataset with more long essays than the original ASAP, reports against a human baseline, and is transparent about training choices such as frozen Mamba components and prompt variations for the generative model. However, the experimental design does not isolate context length from architecture or pretraining, the reported QWK differences are small and presented without uncertainty, and the asserted validity benefit for organization is never measured. The significance is therefore conditional on additional experiments that directly test the long-context motivation.","major_comments":[{"comment":"The claim that long-context models are 'well-suited for handling lengthy student essays' is not supported by the reported comparisons. The 512-token DeBERTa-Base achieves QWK 0.790, within 0.008 of the best long-context model (Longformer, 0.798) and 0.007 of Mamba-130m (0.797). Because no experiment compares the same model on full versus truncated inputs, architecture family and pretraining are fully confounded with context length; the small observed differences could be due to model family rather than context. Please add a paired full-versus-truncated comparison, for example Longformer or ModernBERT trained on 512-token inputs versus the same model on full-length inputs, and report the proportion of essays that exceed 512 tokens.","section":"Section 4, Table 2"},{"comment":"The abstract states that 'many responses in the ASAP 2.0 dataset are too long for most language models,' but Table 1 reports only mean word counts (376.1 train, 342.7 test). Without the distribution of essay lengths, the reader cannot determine whether the long-context premise is vacuous for this dataset; if most essays fall below 512 tokens, the motivation and the comparisons between long-context and 512-token models lose their force. Please report the proportion of essays exceeding 512 tokens (and, ideally, a length histogram) per prompt and grade, separately for train and test, and clarify what happens to essays longer than the implemented context caps such as the 8k bound noted in Table 2.","section":"Section 2.7, Table 1"},{"comment":"The paper reports single QWK point estimates with no confidence intervals, standard errors, or significance tests. The differences among models (0.784 to 0.798) and between models and the human baseline (0.745) are small, so the statements that 'Longformer achiev[ed] the highest overall QWK' and that 'all models surpassed this baseline' are not statistically supported. Report bootstrap intervals or a paired significance test over essays, and consider multiple training seeds. Without this, the model ordering and the claim of surpassing human agreement are not actionable.","section":"Sections 4 and 5"},{"comment":"The paper concedes that using long context 'should not be about getting higher agreement,' yet it does not measure the asserted validity benefit. The abstract and introduction claim that truncation 'undermines the model's ability to fully capture and evaluate organizational elements of the scoring rubric,' but no experiment, rubric trait score, or qualitative analysis of organization is reported. If the contribution is a validity argument rather than an agreement improvement, the paper should include a direct measurement or analysis of the organizational dimension, or the claims should be revised to match the evidence presented.","section":"Section 5"}],"minor_comments":[{"comment":"The ModernBERT context length is given as '8196'; this is presumably a typo for 8192 and should be corrected.","section":"Section 2.4"},{"comment":"The QLoRA configuration is underspecified: the rank, scaling factor, target linear layers (beyond the Lq, Lk, Lv mention), quantization bit-width, and exact hyperparameters are not reported, which prevents reproduction of the Llama-3.2-8B result.","section":"Section 3.2"},{"comment":"The observation that 'full model training seemed to readily lead to model collapse' for Mamba would be more useful if accompanied by a precise description of the failure mode (e.g., loss divergence, degenerate predictions) and the exact set of frozen layers.","section":"Section 3.1"},{"comment":"The data split procedure is unclear: the paper mentions a training set and a development set, but does not state explicitly whether the train/test split is random, stratified by prompt, or otherwise controlled. Please clarify so that prompt overlap between train and test can be ruled out.","section":"Section 3"},{"comment":"There are a number of typographical and formatting errors, including 'pretaining' in Section 3 and the missing closing parenthesis in Equation (4); a careful proofread is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical benchmark, but it is framed as a validation of long-context AES while lacking the direct comparison that would justify that frame. The required experiments—full-versus-truncated evaluation, length distribution reporting, and uncertainty quantification—are feasible within the dataset and the models already used, so major revision rather than rejection seems appropriate. I do not see any integrity concerns; the issue is experimental design and reporting. The paper might also benefit from repositioning as a multi-model benchmark for ASAP 2.0 rather than a long-context validity study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a decent empirical benchmark of five long-context scoring models on the new ASAP 2.0 corpus, and the Mamba result is worth knowing. But the headline claim that long context matters for AES is not tested. The only 512-token model, DeBERTa, hits 0.790 QWK, within 0.008 of Longformer's 0.798, and no confidence intervals are given. Without a same-model truncated-versus-full comparison, architecture and pretraining are fully confounded with context length. The stress-test note lands; it is not a manufactured concern.\n\nWhat the paper does well: it is the first to evaluate this specific set of architectures on ASAP 2.0, a new and larger corpus with longer essays. The training setups are clearly separated for discriminator and generative models, and the Mamba fine-tuning trick—freezing the SSM and conv layers, training only the embedding and in/out projections—is a practical detail that others may reuse. The prose is readable and the related work is appropriately cited.\n\nThe soft spots are real but fixable. There are no variance estimates or significance tests, so the ranking of models is unsupported. Table 1 gives only mean word counts (376 train, 343 test); if most essays fall under 512 tokens, the long-context premise weakens substantially. The authors admit in Section 5 that the goal \"should not be about getting higher agreement,\" yet the only outcome they report is QWK; the claimed validity benefit for organizational quality is never measured. QLoRA configuration (rank, alpha, adapter targets) is not specified, which hurts reproducibility. These are not fatal flaws; they are the difference between a usable benchmark and a convincing one.\n\nThis is a paper for AES practitioners who want to know which open-source models work on the new corpus, and for people curious about state-space models for scoring. It is not a general NLP contribution. I would send it to peer review, but I would require the authors to add confidence intervals, essay length percentiles, and a 512-token truncation baseline for at least the encoder models before publication. With those, it becomes a solid reference result.","headline":"Useful benchmark of long-context AES models on the new ASAP 2.0 corpus, but the 'long context matters' claim is untested: a 512-token DeBERTa nearly matches the best long-context model.","tokens_in":9837,"tokens_out":3210,"would_cite":true,"duration_ms":26397,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Long-context language models score full essays above human-rater agreement.","keywords":["automated essay scoring","long-context language models","ASAP 2.0","quadratic weighted kappa","Mamba state-space model","Longformer","essay truncation","linear-complexity sequence models"],"falsifier":"Stratify QWK by essay length: if models with 8k contexts do not beat DeBERTa's truncated 512-token predictions on essays longer than 512 tokens, or if the human-baseline comparison flips on those essays, the paper's central claim about long context collapses.","tokens_in":8910,"feed_emoji":"📝","tokens_out":6242,"duration_ms":50749,"temperature":0.7,"pith_summary":"This paper tries to show that automated essay scoring does not have to truncate long student essays at the 512-token limit imposed by early transformers. On the ASAP 2.0 benchmark, it compares six model families that extend or replace the fixed-length attention mechanism, and reports that every one surpasses the human-rater agreement of 0.745. Longformer reaches the top quadratic weighted kappa of 0.798, while Mamba-130m scores 0.797 despite having only 130 million parameters. The authors read this as evidence that long-context architectures are viable for scoring full essays, and that linear-complexity state-space models are a practical choice at scale.","feed_headline":"Long-context models top human-rater agreement on essays","feed_subtitle":"On ASAP 2.0, all six models beat QWK 0.745; a 130M Mamba nearly tied the best transformer.","key_machinery":"The load-bearing machinery is a set of architectural fixes to the transformer's fixed context: Longformer's sliding-window local attention with a few global tokens; XLNet's recurrent hidden-state concatenation; ModernBERT's and Llama's rotary position embeddings trained at extended lengths; and Mamba's selective state-space recurrence, which replaces attention with a linear-complexity update. The evaluation harness is the quadratic weighted kappa, computed against a human-human baseline of 0.745 introduced with the ASAP 2.0 corpus. Mamba's selective state-space model does the main argumentative work: it gives near-top agreement at 130M parameters and linear scaling, which the authors cite to make long-context scoring practical.","core_discovery":"The central discovery is that on the ASAP 2.0 essays, long-context language models can score full-length student texts with rater agreement at or above human-human agreement, so truncation is not required for good machine scoring. The paper reports QWK of 0.798 for Longformer, 0.797 for Mamba-130m, 0.792 for Llama-3.2-8B, 0.790 for DeBERTa and ModernBERT, and 0.784 for XLNet, against a human baseline of 0.745. The authors emphasize Mamba's result because it comes from a 130M-parameter state-space model with linear complexity rather than quadratic attention, and they treat the value of long contexts as validity rather than raw agreement: a model that never sees an essay's organization cannot faithfully score the organization trait. A secondary finding is that a large instruction-tuned generative model, Llama-3.2-8B, does not outscore compact encoders, though it can generate feedback.","pith_inferences":["The paper does not report a token-length distribution, so whether the long-context advantage actually comes from long essays is untested; a reader could check by stratifying QWK on essays above and below 512 tokens.","Mamba's strong agreement despite frozen state-space weights hints that long-range essay organization may be captured by compressed state rather than explicit pairwise attention; ablating state size or window width would test that.","If the 0.798 versus 0.784 spread is within run-to-run noise, the practical conclusion shifts from 'Longformer is best' to 'most long-context designs are interchangeable', which strengthens a cost-based selection argument.","The validity framing suggests a testable extension: score organization and local language traits separately, feeding full text to the former and truncated text to the latter, to see where long context earns its cost."],"forward_implications":["Scoring systems can ingest entire essays instead of 512-token prefixes, so rubric traits such as organization and argument flow are evaluated on the text a human rater would read.","Linear-complexity models like Mamba become attractive for large-scale assessment because they combine near-top agreement with lower compute and 2–8x speedups over transformers.","A diverse set of long-context designs—selective attention, recurrence, rotary embeddings, and state-space models—yields similar agreement, so the choice among them can be driven by efficiency, privacy, and deployment constraints.","Generative LLMs add feedback capabilities beyond scoring, which could matter more than a small QWK difference.","Ensembling these architecturally different models is a natural next step, since their errors are likely decorrelated."],"supporting_citations":[{"why":"Supplies the ASAP 2.0 dataset and the human-human QWK baseline of 0.745 that all models are compared against.","marker":"(Crossley et al., 2025)"},{"why":"Defines Longformer's sliding-window plus global attention, the architecture that achieves the highest overall QWK.","marker":"(Beltagy et al., 2020)"},{"why":"Defines Mamba's selective state-space model, the linear-complexity architecture that performs competitively at 130M parameters.","marker":"(Gu and Dao, 2024)"},{"why":"Supplies XLNet's recurrence-based long-context method, one of the fixed-length circumventors tested.","marker":"(Yang et al., 2019)"},{"why":"Defines ModernBERT's rotary-position-embedding long-context encoder, another tested architecture.","marker":"(Warner et al., 2024)"},{"why":"Provides DeBERTa, the strong 512-token baseline that anchors the truncation comparison.","marker":"(He et al., 2021)"},{"why":"Provides the generative Llama-3.2-8B model used for parameter-efficient scoring.","marker":"(AI@Meta, 2024)"},{"why":"Establishes QWK and human-rater agreement as the evaluation criterion for automated essay scoring.","marker":"(Williamson et al., 2012)"}],"fun_headline_variants":["Long-context models outscore human raters on essays","130M Mamba ties huge transformers on essay scoring","No truncation: long-context LMs score essays above humans","Small Mamba matches big models on full-length essays","Essay scoring: long context beats truncation, matches humans"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison only matters if a large enough share of ASAP 2.0 essays exceeds the 512-token cutoff; the reported mean word counts alone do not show that, since means near 350 words leave room for most essays to fit within 512 tokens.","fun_headline_variants_meta":{"raw":{"variants":["Long-context models outscore human raters on essays","130M Mamba ties huge transformers on essay scoring","No truncation: long-context LMs score essays above humans","Small Mamba matches big models on full-length essays","Essay scoring: long context beats truncation, matches humans"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1190,"prompt_tokens":880,"completion_tokens":310,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":229}},"tokens_in":496,"tokens_out":310,"duration_ms":3153,"temperature":1.0,"reasoning_tokens":229,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:53:55.411468+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Stratify QWK by essay length: if models with 8k contexts do not beat DeBERTa's truncated 512-token predictions on essays longer than 512 tokens, or if the human-baseline comparison flips on those essays, the paper's central claim about long context collapses.","supporting_citations":[{"cited_title":"Burleigh, and Jules King","cited_arxiv_id":null,"evidence_quote":"Supplies the ASAP 2.0 dataset and the human-human QWK baseline of 0.745 that all models are compared against."}],"review_version":1}