{"id":"e398a279-ede1-441d-95ff-71601798fe73","arxiv_id":"2501.05222","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A new benchmark of revised scientific paragraphs with human-written revision instructions shows that detailed instructions improve LLM revisions under reference-based similarity metrics compared with generic labels.","lead":"The authors introduce ParaRev, a dataset of 48,203 pairs of original and revised scientific paragraphs, with 641 paragraphs manually annotated with revision labels and shorter instructions for 258. Their experiments compare LLM revisions guided by detailed instructions versus generic labels, reporting higher reference-based similarity for the instruction condition across several models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified; the conditional verdict with softened quality claims and requested human/statistical checks is appropriate.","rationale":"The reader's weakest_assumption correctly identifies the reliance on reference-based metrics as the load-bearing assumption: if these metrics do not measure revision quality, the claimed superiority of instructions over labels is not established. The paper itself concedes this in Section 5. I reviewed the full text for internal inconsistencies, circular reasoning, or unsupported steps in the dataset construction and annotation protocol and found none beyond what the reader flagged. The dataset is released, the extraction code is open, and the annotation agreement is honestly reported. The conditional verdict already requires softening the claim and adding human evaluation, which is the appropriate remedy. Since the reader's concern and required changes are already in the conditional verdict, I see no additional adjustment needed.","tokens_in":8150,"tokens_out":939,"duration_ms":8288,"concrete_test":"Run a human preference study on a random sample of the 258-paragraph evaluation set (e.g., 50-100 paragraphs), where expert annotators compare instruction-guided vs label-guided outputs from the same model, blinded, with ties allowed; if instruction-guided outputs are not preferred at a meaningful rate, the central claim should be weakened to 'instructions yield higher reference-based similarity,' as the conditional verdict requires.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that detailed personalised instructions improve revision quality over generic labels, but the evidence is reference-based metrics on a small evaluation set, and the CopyInput no-edit baseline scores highest. The paper candidly acknowledges this in Section 5, noting that the metrics penalize valid deviations from a single gold revision. The reader's condition to soften the claim and add human/task-level evaluation is the right corrective. I do not find an additional internal inconsistency or a more load-bearing flaw. The dataset construction is transparent, the annotation agreement is reported with appropriate caveats, and the main comparison (instructions vs labels) shows consistent gains across models and metrics, including a statistically significant test for three of four models. The absence of significance testing for CoEdit and the small 258-paragraph evaluation set are limitations the paper itself flags.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"ParaRev is a dataset of 48,203 pairs of original and revised scientific paragraphs extracted from CASIMIR, with 641 manually annotated paragraphs receiving revision-intention labels and 258 paragraphs forming an evaluation subset annotated with free-text revision instructions. The paper proposes a paragraph-level revision task in which a model is guided by a detailed, personalized instruction, and compares this with a baseline that maps coarse labels to general instructions. Using zero-shot CoEdit, Mistral, Llama-3, and GPT-4o, the authors report that detailed instructions yield higher ROUGE-L, SARI, and BERTScore against the gold revision, and conclude that detailed instructions significantly improve revision quality.","tokens_in":8231,"tokens_out":6041,"duration_ms":57852,"significance":"The resource is potentially useful: the selection pipeline and annotation protocol are described transparently, the code and data are released, and the evaluation includes a no-edit control baseline and significance tests for three models. If the main claim were supported, the paper would provide a reusable benchmark for paragraph-level revision and a template for instruction annotation. However, the evidence for 'quality' rests on reference-based similarity metrics on which the CopyInput no-edit baseline scores highest, so the current experiments demonstrate similarity to the gold revision rather than revision quality. The paper's own Section 5 acknowledges this limitation; the abstract and conclusion nevertheless state the stronger claim.","major_comments":[{"comment":"The central empirical claim is not supported by the reported metrics. In Table 1, the CopyInput no-edit baseline obtains the highest scores on ROUGE-L (78.49), SARI (60.69), and BERTScore (95.98) by substantial margins, and the authors themselves note that the metrics penalize revisions that deviate from the gold standard even if valid. Therefore the consistent instruction-over-label gains show only that instructions produce outputs closer to the author's gold revision, not that they are higher-quality revisions. The abstract's 'significantly improves the quality... no matter the model or the metric considered' overstates the evidence. A human or task-based evaluation (e.g., expert preference, targeted instruction compliance, or downstream acceptability) is needed, or the claim should be restricted to 'closer to the gold revision under reference metrics.'","section":"Section 5, Table 1"},{"comment":"The statistical evidence is reported incompletely. Table 1 provides no p-values, confidence intervals, or effect sizes; it only marks three of four models with a dagger. For CoEdit, the instruction-versus-label differences (ROUGE-L 67.50 to 67.70; SARI 39.56 to 39.68; BERTScore 93.88 to 93.93) are tiny and presumably not significant. Since the abstract claims improvement 'no matter the model or the metric considered,' this is a direct overstatement. Please report exact p-values, include variance or confidence intervals, and address multiple comparisons across models and metrics.","section":"Section 5, Table 1"},{"comment":"The instruction-annotation protocol introduces a form of target leakage. Annotators are asked to write instructions 'as they would when guiding an LLM to revise the first version of the paragraph into the second,' which means they saw the gold target paragraph while writing the instruction. The instructions can therefore encode the exact target changes (for example, 'Combine sentences 3 and 4 into a really short one' in Table 3). The comparison with generic labels then conflates the effect of detailed instructions with the effect of providing information derived from the gold revision. This does not invalidate ParaRev as a dataset, but it weakens the claim that the protocol is a template for realistic user guidance. Please discuss this leakage explicitly and, ideally, add a condition in which instructions are written without access to the target paragraph.","section":"Section 3.3"}],"minor_comments":[{"comment":"The instruction distribution reporting is ambiguous: '328 have no instruction, 55 have one, and 258 have two' likely refers to the number of annotation passes containing an instruction rather than to distinct instructions per paragraph. Please clarify how many paragraphs have one versus two instruction annotations, and state which instruction is used for each evaluation instance when two are present.","section":"Section 4"},{"comment":"There are several language and formatting issues: 'This limitation need to be address' should read 'needs to be addressed,' and 'significative improvement' should be 'significant improvement.' The row label 'CopyInput- no edits' contains an unusual dash.","section":"Section 5"},{"comment":"The paragraph-selection thresholds are described as 'obtained empirically,' but the manuscript does not report the development sample size or the procedure used to set them. Please provide this information so readers can assess the risk of overfitting the selection criteria to the corpus.","section":"Appendix A"},{"comment":"The instruction guidelines say 'Detailed lists of changes are not allowed,' but some example instructions in Table 3 contain two directives (e.g., 'Combine sentences 3 and 4... Improve the choice of wording'). Please define more precisely what counts as a 'detailed list.'","section":"Section 3.3"},{"comment":"The reference for Du et al. (2022a) contains an apparent typo: 'Vipul Runderstandaheja' should likely be 'Vipul Raheja.' The same reference also has a spacing issue in 'V olume 1.'","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid dataset contribution, and the authors are unusually candid about the limitations of reference-based metrics. However, the abstract and conclusion currently claim quality improvements that the experiments do not establish, given that the no-edit baseline scores highest. The required changes—softening the quality claim, adding human or task-level evaluation, and addressing the target-leakage issue in the instruction condition—are substantial enough to warrant a major revision rather than a minor one. If the authors can either provide human evaluation or carefully restrict the claims, the paper could be publishable as a resource paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [Name],\n\nQuick take on arXiv:2501.05222. The contribution is the dataset, and that's solid: 48k revised paragraph pairs built from CASIMIR, with a 258-paragraph evaluation subset carrying human-written revision instructions. The construction is transparent (thresholds in Appendix A, annotation setup, agreement stats), the taxonomy is reasonable, and the authors are candid that the labels/instructions are limited to revisions that don't add new content. That's a real resource for anyone working on paragraph-level editing, and the protocol for writing instructions from gold pairs is reusable.\n\nWhat the paper does well: it frames the task cleanly and runs a fair comparison—same models, same prompts, only the instruction changes. The result that detailed instructions beat generic label prompts is consistent across models and statistically significant for three of the four models. The CopyInput baseline scoring highest is itself a useful warning about reference-based metrics in this setting, and the authors say so plainly in Section 5.\n\nThe soft spots are real but not disqualifying. The central claim in the abstract—'significantly improves quality, no matter the model or metric'—is broader than the evidence, because 'quality' is measured only by similarity to a single gold revision, and the no-edit baseline wins. The evaluation set is small (258 paragraphs), CoEdit's gains are tiny and not significance-tested, and there's no human or task-level evaluation to confirm that the instruction-guided revisions are actually better for a writer. The paper itself flags most of this in Limitations and in Section 5, which I found honest, not evasive.\n\nSo the right fix is not to reject the resource but to recalibrate the claim: the dataset is valuable, and personalized instructions help models move closer to the gold edit under these metrics, but 'quality' needs a human/rubric confirmation before the stronger wording stands. I'd also want variance and effect sizes reported, and significance testing for all models, not just three.\n\nWho is this for? Researchers working on text editing, writing assistance, and instruction-based generation. It's a step up from sentence-level datasets, and the annotation protocol is worth copying. This deserves a serious referee; it doesn't deserve a desk reject. I'd recommend major revision with a softened abstract and an added human evaluation, or at minimum a clear statement that the metrics only measure closeness to gold, not end-user value.\n\nBest,\n[Name]","headline":"ParaRev is a genuinely useful paragraph-revision dataset, and the paper is honest about its own metrics problem—worth serious review, but the headline claim needs softening.","tokens_in":8773,"tokens_out":2004,"would_cite":true,"duration_ms":19145,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that giving language models detailed, personalized revision instructions for whole paragraphs improves the quality of their automated revisions compared to generic revision labels, and introduces ParaRev, a dataset of…","keywords":["paragraph revision","scientific writing","revision instruction","revision dataset","LLM evaluation","writing assistance","text revision","reference-based metrics"],"falsifier":"Present blinded human annotators with a sample of revisions generated under the instruction and label conditions, plus the unedited original, and ask them to rate quality and appropriateness. The central claim would be falsified if instruction-guided revisions are not preferred over label-guided ones, or if the no-edit original is rated as good as or better than both.","tokens_in":7940,"feed_emoji":"✍️","tokens_out":5734,"duration_ms":49157,"temperature":0.7,"pith_summary":"The paper argues that scientific text revision should be framed at the paragraph level rather than the sentence level, guided by detailed, personalized revision instructions instead of generic labels. To support this, it introduces ParaRev, a dataset of 48,203 author-revised paragraph pairs extracted from the CASIMIR corpus, with an evaluation subset of 258 paragraphs manually annotated with revision intention labels and instructions. Across four models (CoEdit, Llama-3, Mistral, GPT-4o), the paper reports that detailed instructions produce higher ROUGE-L, SARI, and BERTScore than generic label-based prompts, with statistically significant gains for the three LLMs. The paper acknowledges that its evaluation metrics are reference-based and penalize valid deviations, as shown by the no-edit CopyInput baseline scoring highest on all metrics.","feed_headline":"Detailed instructions improve AI paragraph revision","feed_subtitle":"Specific revision commands beat generic labels across LLMs, though no-edit baselines still score highest.","key_machinery":"The load-bearing object is ParaRev, a new corpus of 48,203 aligned paragraph pairs extracted from the CASIMIR corpus, plus an annotation protocol that assigns each paragraph pair up to two labels from a nine-category revision taxonomy (Rewriting light/medium/heavy, Concision, Development, Content addition/substitution/deletion, Unusable) and a free-text revision instruction written by the annotator. Instructions are written as concise commands to an LLM and may specify position and intensity of edits; paragraphs involving new content (Development, Content addition, Content substitution) receive no instruction. The evaluation subset consists of 258 paragraphs with instructions, on which the paper compares zero-shot prompting of CoEdit, Llama-3, Mistral, and GPT-4o under two prompt conditions, scored with ROUGE-L, SARI, and BERTScore.","core_discovery":"The central claim is that shifting the revision task to paragraph scope and providing detailed, personalized instructions to the model yields more effective automated revisions than giving a generic instruction mapped from a revision label. The evidence comes from a controlled comparison on 258 paragraph pairs, where each model is prompted either with the paragraph's hand-written revision instruction or with a generic instruction derived from its label. The instruction condition outperforms the label condition across ROUGE-L, SARI, and BERTScore for all tested models, with the gap reaching statistical significance for Mistral, Llama-3, and GPT-4o. The authors interpret this as confirmation that specific guidance about what to change, where, and how intensely leads models to produce edits closer to the human gold revision.","pith_inferences":["A direct test of whether the instruction advantage survives human preference judgment, rather than reference-based similarity, would clarify whether the gain is real or an artifact of metric design.","The annotation protocol could be extended to automatically generate instructions from diffs between paragraph pairs, enabling silver training data at the scale of the full 48k corpus.","The CopyInput ceiling suggests that future revision benchmarks should consider multiple acceptable revisions or task-based evaluation instead of a single gold reference.","Paragraph-level instructions may be most valuable when they specify position and intensity, so a follow-up analysis could isolate which instruction properties drive the improvement."],"forward_implications":["ParaRev provides a reusable benchmark of 48,203 paragraph-level revision pairs with an instruction-annotated evaluation subset for scientific writing assistance.","Using detailed personalized instructions improves automated revision quality relative to generic label-based prompts across CoEdit, Llama-3, Mistral, and GPT-4o, with statistically significant gains for the three LLMs.","Paragraph-level revision, as opposed to sentence-level, enables more comprehensive changes such as merging, splitting, or reorganizing sentences while preserving context.","The instruction-annotation protocol serves as a template for building future revision datasets with explicit revision intentions.","The automatically annotated 'silver' extension of ParaRev could be used to fine-tune open-source models specifically for paragraph-level revision."],"supporting_citations":[{"why":"Supplies the CASIMIR corpus of revised articles from which all ParaRev paragraph pairs are extracted.","marker":"(Jourdan et al., 2024)"},{"why":"Provides a sentence-level revision taxonomy that motivates the paragraph-level taxonomy design.","marker":"(Du et al., 2022b)"},{"why":"Provides another sentence-level revision taxonomy and corpus used to motivate the label set.","marker":"(Jiang et al., 2022)"},{"why":"Introduces CoEdit, a baseline model, and the label-to-general-instruction mapping approach that the comparison is based on.","marker":"(Raheja et al., 2023)"},{"why":"Defines the ROUGE-L metric used to score revision quality.","marker":"(Lin, 2004)"},{"why":"Defines the SARI metric used to score revision quality.","marker":"(Xu et al., 2016)"},{"why":"Defines the BERTScore metric used to measure similarity between generated and gold revisions.","marker":"(Zhang et al., 2020)"}],"fun_headline_variants":["ParaRev: detailed instructions beat generic labels for paragraph revision","Specific revision prompts boost AI paragraph editing quality","Paragraph-level revision: detailed instructions outperform generic labels","New dataset shows detailed revision guidance improves AI writing edits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The experimental conclusion depends on the assumption that ROUGE-L, SARI, and BERTScore computed against a single gold revision are meaningful measures of revision quality; the no-edit CopyInput baseline scores highest on all three metrics, so if these metrics are not valid, the reported gains of instructions over labels do not demonstrate higher-quality revisions.","fun_headline_variants_meta":{"raw":{"variants":["ParaRev: detailed instructions beat generic labels for paragraph revision","Specific revision prompts boost AI paragraph editing quality","Paragraph-level revision: detailed instructions outperform generic labels","New dataset shows detailed revision guidance improves AI writing edits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000145,"raw_usage":{"total_tokens":1111,"prompt_tokens":810,"completion_tokens":301,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":426,"completion_tokens_details":{"reasoning_tokens":240}},"tokens_in":426,"tokens_out":301,"duration_ms":3505,"temperature":1.0,"reasoning_tokens":240,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:13:47.799255+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Present blinded human annotators with a sample of revisions generated under the instruction and label conditions, plus the unedited original, and ask them to rate quality and appropriateness. The central claim would be falsified if instruction-guided revisions are not preferred over label-guided ones, or if the no-edit original is rated as good as or better than both.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CASIMIR corpus of revised articles from which all ParaRev paragraph pairs are extracted."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides another sentence-level revision taxonomy and corpus used to motivate the label set."}],"review_version":1}