{"id":"853546f6-61ca-4cb6-801f-2f370d7dcd39","arxiv_id":"2606.12801","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Fine-tuning Gemma-3-27B-it with LoRA on a 120-essay TOEFL subset produces RMSE 0.474, QWK 0.828 and 90.56% agreement within 0.5 on the 360-essay test set, outperforming a larger LLaMA model and prior GPT-3.5 baseline.","lead":"The paper presents AiAWE, an open-source system that fine-tunes the Gemma-3-27B LLM with LoRA on 120 TOEFL essays to score the remaining 360 with RMSE 0.474 and QWK 0.828. A smart generalist might read it to see how accessible open models can handle real educational assessment tasks without proprietary services.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Unverifiable 120/360 split on proprietary ETS data risks leakage or post-hoc selection bias","rationale":"The reader's weakest_assumption correctly isolates the single point that must hold for every quantitative claim to be credible. No other internal inconsistency (architecture comparison, LoRA config, or inference details) is load-bearing once the evaluation split is accepted; the proprietary nature simply makes that acceptance unverifiable without external data access.","tokens_in":1832,"tokens_out":319,"duration_ms":14028,"concrete_test":"Request the full 480-essay ETS collection under the same license used by the authors, apply the exact 120/360 partition described in the paper, and recompute RMSE, QWK, and agreement; if any metric shifts by >15% or if essay-level overlap is found, the reported superiority is not reproducible.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"All headline metrics (RMSE 0.474, QWK 0.828, 90.56% agreement) and the claim of outperforming the prior GPT-3.5 baseline rest on evaluation of a 360-essay test set drawn from a 480-essay proprietary collection after the authors selected a 120-essay training subset. Because the raw essays are not released, neither leakage (topic/ID overlap between subsets) nor selection effects (post-hoc choice of a favorable partition) can be audited. The LoRA adaptation and quantization details are public, but they cannot compensate for an uninspectable held-out set.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript presents AiAWE, an open-source automated writing evaluation system that applies LoRA adaptation to instruction-tuned LLMs (primarily Gemma-3-27B-it, with comparison to LLaMA-3.3-70B) for scoring argumentative TOEFL essays. Using a proprietary ETS collection of 480 essays, the authors fine-tune on a 120-essay training subset and report results on the remaining 360-essay test set under matched inference settings: the Gemma model attains RMSE 0.474, QWK 0.828, and 90.56% agreement within ±0.5 of human scores, outperforming both the larger LLaMA model and a prior fine-tuned GPT-3.5 baseline on the same dataset. The work also releases the production system, LoRA adapters, and fine-tuning configurations.","tokens_in":1974,"tokens_out":576,"duration_ms":25416,"significance":"If the reported metrics hold under the chosen split, the paper supplies concrete evidence that open-weight models under LoRA can reach rubric-aligned performance competitive with or superior to proprietary fine-tuning, while running on consumer hardware. The public release of adapters, code, and the deployed system constitutes a clear reproducibility asset for the AWE community. The secondary observations—that model scale is not a reliable predictor under identical LoRA settings and that adaptation behavior differs across architectures—are empirically grounded contributions.","major_comments":[{"comment":"Dataset and experimental setup (abstract and § on data): the 120-essay training subset is drawn from the 480-essay proprietary ETS collection with no description of the selection procedure (random, score-stratified, prompt-balanced, or otherwise). Because every headline metric (RMSE 0.474, QWK 0.828, 90.56% agreement) and the claim of outperforming the prior GPT-3.5 baseline rests exclusively on the 360-essay held-out set, the absence of an auditable partition leaves open the possibility of post-hoc selection effects or unintended leakage; this is load-bearing for the central empirical claims.","section":"Dataset and experimental setup"}],"minor_comments":[{"comment":"Abstract: the phrase 'identical inference quantization' is used without stating the bit-width or method (e.g., 4-bit GPTQ), which would directly aid replication of the reported numbers.","section":"Abstract"},{"comment":"The manuscript would benefit from an explicit statement of whether the 360-essay test set matches the exact test partition used in the cited prior GPT-3.5 work, beyond the general claim of 'the same dataset'.","section":"Results and comparison"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the careful and constructive review. We address the single major comment below and will incorporate the requested clarification in the revised manuscript.","responses":[{"response":"We agree that the manuscript should have documented the partitioning procedure. The 120-essay training subset was obtained by simple random sampling (without replacement) from the full 480-essay collection prior to any model training, hyperparameter tuning, or evaluation; the remaining 360 essays formed the test set. No score stratification, prompt balancing, or post-hoc adjustments were applied. We will revise the data section to state this explicitly. Because the underlying ETS data remain proprietary, a fully public audit trail of the exact essay IDs is not feasible, but the pre-experiment random split eliminates the possibility of leakage or selection effects on the reported metrics.","revision_made":"yes","referee_comment":"[Dataset and experimental setup] Dataset and experimental setup (abstract and § on data): the 120-essay training subset is drawn from the 480-essay proprietary ETS collection with no description of the selection procedure (random, score-stratified, prompt-balanced, or otherwise). Because every headline metric (RMSE 0.474, QWK 0.828, 90.56% agreement) and the claim of outperforming the prior GPT-3.5 baseline rests exclusively on the 360-essay held-out set, the absence of an auditable partition leaves open the possibility of post-hoc selection effects or unintended leakage; this is load-bearing for the central empirical claims."}],"tokens_in":1547,"tokens_out":338,"duration_ms":26286,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The key takeaway is that this work ships usable LoRA adapters for two open LLMs that score argumentative essays, with the Gemma-3-27B-it version reaching RMSE 0.474 and QWK 0.828 on the 360-essay test set, beating both the LLaMA counterpart and an earlier GPT-3.5 baseline. The open-source release of the AiAWE system makes this more than just another benchmark paper.\n\nThey do a clean job of holding the LoRA config and inference settings fixed across models, which lets them show that scale alone does not predict performance here. The fact that the full system, adapters, and configs are released means others can actually run and extend it without starting from scratch. The production app running on consumer hardware is a practical plus that sets it apart from many closed systems.\n\nThe main limitation is the data split. The training set is only 120 essays from a proprietary ETS collection of 480, and the test set is the rest. Without access to the raw essays, it's impossible to rule out topic overlap or that the partition was selected to look good. That undercuts how strongly we can take the outperformance claim, though the metrics themselves are reported on a held-out portion.\n\nThis is for researchers or developers working on automated writing evaluation or fine-tuning LLMs for scoring tasks. Someone building an edtech tool would get immediate value from the released code and the direct model comparisons.\n\nIt is worth sending to peer review. The empirical results and open artifacts are solid enough to merit referee input, even if the data access issue needs addressing in revision. The paper engages honestly with the literature on prior GPT baselines and reports the numbers clearly.","headline":"The paper ships public LoRA adapters and code for a Gemma-based essay scorer that hits RMSE 0.474 and QWK 0.828 on a 360-essay ETS hold-out, beating the LLaMA version and a prior GPT baseline, but the proprietary 120/360 split cannot be audited for leakage or selection effects.","tokens_in":2424,"tokens_out":461,"would_cite":false,"duration_ms":17687,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A LoRA-adapted open-weight Gemma model scores TOEFL essays with 90.56 percent agreement to human raters.","keywords":["automated writing evaluation","LoRA adaptation","large language models","essay scoring","Gemma model","TOEFL essays","open-source system","quadratic weighted kappa"],"falsifier":"Evaluating the same fine-tuned models on a fresh collection of TOEFL essays drawn from a different source or on a different random split of the 480 essays and obtaining agreement rates well below the reported 90.56 percent would undermine the performance claim.","tokens_in":2727,"feed_emoji":"📝","tokens_out":752,"duration_ms":16409,"temperature":0.7,"pith_summary":"The paper shows that fine-tuning the Gemma-3-27B-it model with LoRA on a 120-essay subset of an ETS TOEFL dataset produces strong scoring performance on the remaining 360 essays. The resulting system reports a root mean square error of 0.474, quadratic weighted kappa of 0.828, and 90.56 percent agreement within half a point of human scores, beating both a larger LLaMA-3.3-70B model under identical adaptation and a prior GPT-3.5 baseline. A reader would care because the result indicates that accessible open-weight models can perform rubric-aligned essay evaluation without proprietary systems or large training sets. The work further reports that model scale does not reliably predict success under this method and that the same LoRA settings yield different outcomes across model families.","feed_headline":"Open LLM with LoRA matches human essay scores at 90 percent","feed_subtitle":"Gemma-3-27B fine-tuned on 120 TOEFL essays outperforms larger LLaMA and GPT-3.5 on 360 held-out samples.","key_machinery":"LoRA adaptation of instruction-tuned large language models for producing rubric-aligned numeric scores on argumentative essays.","core_discovery":"The central claim is that the LoRA-adapted Gemma-3-27B-it model, fine-tuned on 120 TOEFL Independent Writing essays, achieves RMSE of 0.474, quadratic weighted kappa of 0.828, and 90.56 percent agreement within plus or minus 0.5 of human scores on the 360-essay test set, outperforming the identically adapted LLaMA-3.3-70B model and the fine-tuned GPT-3.5 baseline from prior work on the same data.","pith_inferences":["This approach could allow smaller institutions to deploy automated essay feedback without depending on closed commercial APIs.","The architecture-specific differences in LoRA response point to a need for systematic model selection when adapting LLMs to other structured judgment tasks.","Extending the method to non-English essays or different rubrics would test whether the reported gains hold beyond the TOEFL Independent Writing prompt."],"forward_implications":["Open-weight LLMs can match or exceed proprietary fine-tuning for rubric-aligned scoring tasks.","Model scale is not a reliable predictor of downstream performance under LoRA adaptation.","Identical LoRA hyperparameters produce qualitatively different adaptation behaviors across model architectures.","The resulting scoring system runs on consumer-grade hardware and can be made publicly available with open adapters and code."],"fun_headline_variants":["Gemma-3-27B with LoRA achieves 90.56 percent human agreement on essays","LoRA-adapted Gemma-3-27B scores 360 TOEFL essays at 90.56 percent agreement","Gemma model under LoRA outperforms LLaMA-3.3-70B on writing evaluation","LoRA fine-tuning on 120 essays produces 0.828 kappa with Gemma-3-27B","Open-weight LLMs match proprietary models for rubric scoring after LoRA"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The fixed 120-essay training split and 360-essay test split from the proprietary ETS dataset gives an unbiased measure of how well the adapted model generalizes.","fun_headline_variants_meta":{"raw":{"variants":["Gemma-3-27B with LoRA achieves 90.56 percent human agreement on essays","LoRA-adapted Gemma-3-27B scores 360 TOEFL essays at 90.56 percent agreement","Gemma model under LoRA outperforms LLaMA-3.3-70B on writing evaluation","LoRA fine-tuning on 120 essays produces 0.828 kappa with Gemma-3-27B","Open-weight LLMs match proprietary models for rubric scoring after LoRA"]},"model":"grok-4.3","cost_usd":0.011586,"raw_usage":{"total_tokens":5123,"prompt_tokens":762,"num_sources_used":0,"completion_tokens":121,"cost_in_usd_ticks":115862000,"prompt_tokens_details":{"text_tokens":762,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":4240,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":762,"tokens_out":121,"duration_ms":25826,"temperature":1.0,"reasoning_tokens":4240,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T05:50:31.765336+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Evaluating the same fine-tuned models on a fresh collection of TOEFL essays drawn from a different source or on a different random split of the 480 essays and obtaining agreement rates well below the reported 90.56 percent would undermine the performance claim.","supporting_citations":[],"review_version":1}