Pith. sign in

REVIEW 3 major objections 5 minor 18 references

ParaRev: Building a dataset for Scientific Paragraph Revision annotated with revision instruction

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that giving language models detailed, personalized revision instructions for whole paragraphs improves the quality of their automated revisions compared to generic revision labels, and introduces ParaRev, a dataset of…

desk verdict ParaRev is a genuinely useful paragraph-revision dataset, and the paper is honest about its own metrics problem—worth serious review, but the headline claim needs softening. read the letter →

arxiv 2501.05222 v1 pith:ZHJMDFI5 submitted 2025-01-09 cs.CL

Léane Jourdan , Nicolas Hernandez , Richard Dufour , Florian Boudin , Akiko Aizawa This is my paper · ORCID
classification cs.CL
keywords paragraphrevisionscientificwritinginstructiondatasetLLMevaluationassistancetextreference-basedmetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that scientific text revision should be framed at the paragraph level rather than the sentence level, guided by detailed, personalized revision instructions instead of generic labels. To support this, it introduces ParaRev, a dataset of 48,203 author-revised paragraph pairs extracted from the CASIMIR corpus, with an evaluation subset of 258 paragraphs manually annotated with revision intention labels and instructions. Across four models (CoEdit, Llama-3, Mistral, GPT-4o), the paper reports that detailed instructions produce higher ROUGE-L, SARI, and BERTScore than generic label-based prompts, with statistically significant gains for the three LLMs. The paper acknowledges that its evaluation metrics are reference-based and penalize valid deviations, as shown by the no-edit CopyInput baseline scoring highest on all metrics.

What carries the argument

The load-bearing object is ParaRev, a new corpus of 48,203 aligned paragraph pairs extracted from the CASIMIR corpus, plus an annotation protocol that assigns each paragraph pair up to two labels from a nine-category revision taxonomy (Rewriting light/medium/heavy, Concision, Development, Content addition/substitution/deletion, Unusable) and a free-text revision instruction written by the annotator. Instructions are written as concise commands to an LLM and may specify position and intensity of edits; paragraphs involving new content (Development, Content addition, Content substitution) receive no instruction. The evaluation subset consists of 258 paragraphs with instructions, on which the paper compares zero-shot prompting of CoEdit, Llama-3, Mistral, and GPT-4o under two prompt conditions, scored with ROUGE-L, SARI, and BERTScore.

What would settle it

Present blinded human annotators with a sample of revisions generated under the instruction and label conditions, plus the unedited original, and ask them to rate quality and appropriateness. The central claim would be falsified if instruction-guided revisions are not preferred over label-guided ones, or if the no-edit original is rated as good as or better than both.

Watch

Extended reading notes

Core claim

The central claim is that shifting the revision task to paragraph scope and providing detailed, personalized instructions to the model yields more effective automated revisions than giving a generic instruction mapped from a revision label. The evidence comes from a controlled comparison on 258 paragraph pairs, where each model is prompted either with the paragraph's hand-written revision instruction or with a generic instruction derived from its label. The instruction condition outperforms the label condition across ROUGE-L, SARI, and BERTScore for all tested models, with the gap reaching statistical significance for Mistral, Llama-3, and GPT-4o. The authors interpret this as confirmation that specific guidance about what to change, where, and how intensely leads models to produce edits closer to the human gold revision.

Load-bearing premise

The experimental conclusion depends on the assumption that ROUGE-L, SARI, and BERTScore computed against a single gold revision are meaningful measures of revision quality; the no-edit CopyInput baseline scores highest on all three metrics, so if these metrics are not valid, the reported gains of instructions over labels do not demonstrate higher-quality revisions.

Editorial extensions

If this is right

  • ParaRev provides a reusable benchmark of 48,203 paragraph-level revision pairs with an instruction-annotated evaluation subset for scientific writing assistance.
  • Using detailed personalized instructions improves automated revision quality relative to generic label-based prompts across CoEdit, Llama-3, Mistral, and GPT-4o, with statistically significant gains for the three LLMs.
  • Paragraph-level revision, as opposed to sentence-level, enables more comprehensive changes such as merging, splitting, or reorganizing sentences while preserving context.
  • The instruction-annotation protocol serves as a template for building future revision datasets with explicit revision intentions.
  • The automatically annotated 'silver' extension of ParaRev could be used to fine-tune open-source models specifically for paragraph-level revision.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of whether the instruction advantage survives human preference judgment, rather than reference-based similarity, would clarify whether the gain is real or an artifact of metric design.
  • The annotation protocol could be extended to automatically generate instructions from diffs between paragraph pairs, enabling silver training data at the scale of the full 48k corpus.
  • The CopyInput ceiling suggests that future revision benchmarks should consider multiple acceptable revisions or task-based evaluation instead of a single gold reference.
  • Paragraph-level instructions may be most valuable when they specify position and intensity, so a follow-up analysis could isolate which instruction properties drive the improvement.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. ParaRev is a dataset of 48,203 pairs of original and revised scientific paragraphs extracted from CASIMIR, with 641 manually annotated paragraphs receiving revision-intention labels and 258 paragraphs forming an evaluation subset annotated with free-text revision instructions. The paper proposes a paragraph-level revision task in which a model is guided by a detailed, personalized instruction, and compares this with a baseline that maps coarse labels to general instructions. Using zero-shot CoEdit, Mistral, Llama-3, and GPT-4o, the authors report that detailed instructions yield higher ROUGE-L, SARI, and BERTScore against the gold revision, and conclude that detailed instructions significantly improve revision quality.

Significance. The resource is potentially useful: the selection pipeline and annotation protocol are described transparently, the code and data are released, and the evaluation includes a no-edit control baseline and significance tests for three models. If the main claim were supported, the paper would provide a reusable benchmark for paragraph-level revision and a template for instruction annotation. However, the evidence for 'quality' rests on reference-based similarity metrics on which the CopyInput no-edit baseline scores highest, so the current experiments demonstrate similarity to the gold revision rather than revision quality. The paper's own Section 5 acknowledges this limitation; the abstract and conclusion nevertheless state the stronger claim.

major comments (3)
  1. [Section 5, Table 1] The central empirical claim is not supported by the reported metrics. In Table 1, the CopyInput no-edit baseline obtains the highest scores on ROUGE-L (78.49), SARI (60.69), and BERTScore (95.98) by substantial margins, and the authors themselves note that the metrics penalize revisions that deviate from the gold standard even if valid. Therefore the consistent instruction-over-label gains show only that instructions produce outputs closer to the author's gold revision, not that they are higher-quality revisions. The abstract's 'significantly improves the quality... no matter the model or the metric considered' overstates the evidence. A human or task-based evaluation (e.g., expert preference, targeted instruction compliance, or downstream acceptability) is needed, or the claim should be restricted to 'closer to the gold revision under reference metrics.'
  2. [Section 5, Table 1] The statistical evidence is reported incompletely. Table 1 provides no p-values, confidence intervals, or effect sizes; it only marks three of four models with a dagger. For CoEdit, the instruction-versus-label differences (ROUGE-L 67.50 to 67.70; SARI 39.56 to 39.68; BERTScore 93.88 to 93.93) are tiny and presumably not significant. Since the abstract claims improvement 'no matter the model or the metric considered,' this is a direct overstatement. Please report exact p-values, include variance or confidence intervals, and address multiple comparisons across models and metrics.
  3. [Section 3.3] The instruction-annotation protocol introduces a form of target leakage. Annotators are asked to write instructions 'as they would when guiding an LLM to revise the first version of the paragraph into the second,' which means they saw the gold target paragraph while writing the instruction. The instructions can therefore encode the exact target changes (for example, 'Combine sentences 3 and 4 into a really short one' in Table 3). The comparison with generic labels then conflates the effect of detailed instructions with the effect of providing information derived from the gold revision. This does not invalidate ParaRev as a dataset, but it weakens the claim that the protocol is a template for realistic user guidance. Please discuss this leakage explicitly and, ideally, add a condition in which instructions are written without access to the target paragraph.
minor comments (5)
  1. [Section 4] The instruction distribution reporting is ambiguous: '328 have no instruction, 55 have one, and 258 have two' likely refers to the number of annotation passes containing an instruction rather than to distinct instructions per paragraph. Please clarify how many paragraphs have one versus two instruction annotations, and state which instruction is used for each evaluation instance when two are present.
  2. [Section 5] There are several language and formatting issues: 'This limitation need to be address' should read 'needs to be addressed,' and 'significative improvement' should be 'significant improvement.' The row label 'CopyInput- no edits' contains an unusual dash.
  3. [Appendix A] The paragraph-selection thresholds are described as 'obtained empirically,' but the manuscript does not report the development sample size or the procedure used to set them. Please provide this information so readers can assess the risk of overfitting the selection criteria to the corpus.
  4. [Section 3.3] The instruction guidelines say 'Detailed lists of changes are not allowed,' but some example instructions in Table 3 contain two directives (e.g., 'Combine sentences 3 and 4... Improve the choice of wording'). Please define more precisely what counts as a 'detailed list.'
  5. [References] The reference for Du et al. (2022a) contains an apparent typo: 'Vipul Runderstandaheja' should likely be 'Vipul Raheja.' The same reference also has a spacing issue in 'V olume 1.'

Circularity Check

1 steps flagged · score 6.0 of 10

Partial circularity: instruction condition is written from the gold target and evaluated against that same target.

  1. self definitional [Section 3.3 (Instructions) combined with Section 5 (Impact of task definition on revision)]
    "Annotators are asked to write concise, simple instructions as they would when guiding an LLM to revise the first version of the paragraph into the second. [...] To assess the quality of revisions, we employed traditional sentence revision metrics, ROUGE-L (Lin, 2004) and SARI (Xu et al., 2016), alongside Bertscore (Zhang et al., 2020) to measure similarity between the generated and gold revised paragraphs. [...] This approach penalizes revisions that deviate from the gold standard, even if they result in valid improvements."

    The annotation protocol fixes each instruction by looking at the gold target paragraph, so the instruction is a human paraphrase of the exact edits separating source from target. The evaluation metric then scores outputs by similarity to that same target. The instruction condition is therefore, by construction, given a description of the reference revision, while the label condition is not. The measured gain of instructions over labels is largely an information-leakage effect: the model is scored against the text that was used to write its prompt.

full rationale

No load-bearing self-citation chain is present: CASIMIR (Jourdan et al., 2024) is used as an external corpus with documented extraction heuristics, and the annotation agreement is reported against Krippendorff's threshold rather than asserted by fiat. The central circular issue is the evaluation loop: annotators write instructions after seeing the gold revised paragraph, and the metrics compare generated revisions to that same paragraph. This makes the label-versus-instruction comparison a test of how much target information is leaked into the prompt, not a clean test of revision quality. The paper itself flags the related metric problem, noting that CopyInput, which performs no edits, scores highest and that valid deviations from a single gold reference are penalized. The abstract's 'no matter the model or metric considered' is further weakened by CoEdit's small, non-significant gains. These are validity and overclaim concerns in addition to the partial construction issue. Because the dataset itself is grounded in real author revisions from CASIMIR/OpenReview and the instruction-vs-label comparison is only one component of the paper, the circularity is partial rather than total.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central empirical claim rests on the fidelity of CASIMIR alignments, the validity of the gold-revision reference, the validity of reference-based metrics, and the hand-tuned threshold choices used to filter paragraphs. The thresholds and metric assumptions are the most fragile parts; they are partially acknowledged in Sections 5 and 7.

free parameters (1)
  • Hand-crafted paragraph selection thresholds (Appendix A) = size >= 250 chars; sentence-edit >= 25%; paragraph-edit >= 10%; >90%-edited sentences <= 40% or 200 chars; >=20% edit…
    These thresholds were set empirically from observations of a subset of CASIMIR (Section 3.1, Appendix A). They determine which 48,203 of 1,889,810 paragraph pairs enter ParaRev and thereby shape every downstream result.
assumptions (3)
  • domain assumption CASIMIR sentence alignments and paragraph IDs reliably reconstruct coherent paragraph pairs.
    Section 3.1 builds ParaRev from CASIMIR paragraph IDs; if alignments are noisy, paragraph pairs contain segmentation artifacts, which the authors try to filter via Appendix A rules and the 'Unusable' label.
  • domain assumption The author's revised paragraph is the correct reference standard for scoring.
    Section 5 scores all model outputs against the gold revised paragraph; this equates 'good revision' with 'close to the author's revision'.
  • domain assumption ROUGE-L, SARI, and BERTScore are valid proxies for revision quality.
    The paper's comparative claim rests on these metrics; the authors themselves note in Section 5 that they penalize valid deviations, weakening this axiom.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ParaRev: Building a dataset for Scientific Paragraph Revision annotated with revision instruction." pith.science (2026). https://pith.science/paper/ZHJMDFI5

@misc{pith2026250105222,
  author       = {Pith},
  title        = {Pith review of: ParaRev: Building a dataset for Scientific Paragraph Revision annotated with revision instruction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZHJMDFI5}},
  note         = {Machine review of arXiv:2501.05222}
}
read the original abstract

Revision is a crucial step in scientific writing, where authors refine their work to improve clarity, structure, and academic quality. Existing approaches to automated writing assistance often focus on sentence-level revisions, which fail to capture the broader context needed for effective modification. In this paper, we explore the impact of shifting from sentence-level to paragraph-level scope for the task of scientific text revision. The paragraph level definition of the task allows for more meaningful changes, and is guided by detailed revision instructions rather than general ones. To support this task, we introduce ParaRev, the first dataset of revised scientific paragraphs with an evaluation subset manually annotated with revision instructions. Our experiments demonstrate that using detailed instructions significantly improves the quality of automated revisions compared to general approaches, no matter the model or the metric considered.

Figures

Figures reproduced from arXiv: 2501.05222 by the authors.

Figure 1
Figure 1. Definitions of the traditional sentence revision [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Example of a revised paragraph with its asso [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. The data pipeline: annotation, paragraph revi [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Distribution of labels across the dataset overall and degree of modification of the articles. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 7 canonical work pages

  1. [1]

    Tatsuya Amano, Valeria Ram \' rez-Casta \ n eda, Violeta Berdejo-Espinola, Israel Borokini, Shawan Chowdhury, Marina Golivets, Juan David Gonz \'a lez-Trujillo, Flavia Monta \ n o-Centellas, Kumar Paudel, Rachel Louise White, et al. 2023. The manifold costs of being a non-native english speaker in science. PLoS Biology, 21(7):e3002184

    work page 2023

  2. [2]

    Mike D'Arcy, Alexis Ross, Erin Bransom, Bailey Kuehl, Jonathan Bragg, Tom Hope, and Doug Downey. 2023. https://arxiv.org/abs/2306.12587 Aries: A corpus of scientific paper edits made in response to peer reviews . Preprint, arXiv:2306.12587

    work page Pith review arXiv 2023

  3. [3]

    Wanyu Du, Zae Myung Kim, Vipul Runderstandaheja, Dhruv Kumar, and Dongyeop Kang. 2022 a . https://doi.org/10.18653/v1/2022.in2writing-1.14 Read, revise, repeat: A system demonstration for human-in-the-loop iterative text revision . In Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022), pages 96--108, Dubl...

    doi:10.18653/v1/2022.in2writing-1.14 2022

  4. [4]

    Wanyu Du, Vipul Raheja, Dhruv Kumar, Zae Myung Kim, Melissa Lopez, and Dongyeop Kang. 2022 b . https://doi.org/10.18653/v1/2022.acl-long.250 Understanding iterative revision from human-written text . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3573--3590, Dublin, Ireland. Associ...

    doi:10.18653/v1/2022.acl-long.250 2022

  5. [5]

    Takumi Ito, Tatsuki Kuribayashi, Hayato Kobayashi, Ana Brassard, Masato Hagiwara, Jun Suzuki, and Kentaro Inui. 2019. https://doi.org/10.18653/v1/W19-8606 Diamonds in the rough: Generating fluent sentences from early-stage drafts for academic writing assistance . In Proceedings of the 12th International Conference on Natural Language Generation, pages 40-...

    doi:10.18653/v1/w19-8606 2019

  6. [6]

    Chao Jiang, Wei Xu, and Samuel Stevens. 2022. arxivedits: Understanding the human revision process in scientific writing. In Proceedings of EMNLP 2022

    work page 2022

  7. [7]

    L \'e ane Jourdan, Florian Boudin, Nicolas Hernandez, and Richard Dufour. 2024. https://aclanthology.org/2024.lrec-main.257 CASIMIR : A corpus of scientific articles enhanced with multiple author-integrated revisions . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 202...

    work page 2024

  8. [8]

    Léane Jourdan, Florian Boudin, Richard Dufour, and Nicolas Hernandez. 2023. http://ceur-ws.org/Vol-3617/#paper-04 Text revision in scientific writing assistance: A review . In 13th International Workshop on Bibliometric-enhanced Information Retrieval (BIR), number 3617 in CEUR Workshop Proceedings, pages 22--36, Aachen

    work page 2023

Show all 18 references
  1. [9]

    Klaus Krippendorff. 2018. Content analysis: An introduction to its methodology. Sage publications

    2018

  2. [10]

    Ilia Kuznetsov, Jan Buchmann, Max Eichler, and Iryna Gurevych. 2022. https://doi.org/10.1162/coli_a_00455 Revise and resubmit: An intertextual model of text-based collaboration in peer review . Computational Linguistics, 48(4):949--986

    2022 doi

  3. [11]

    Jingjing Li, Zichao Li, Tao Ge, Irwin King, and Michael Lyu. 2022. https://doi.org/10.18653/v1/2022.in2writing-1.7 Text revision by on-the-fly representation optimization . In Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022)...

    2022 doi

  4. [12]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  5. [13]

    Masato Mita, Keisuke Sakaguchi, Masato Hagiwara, Tomoya Mizumoto, Jun Suzuki, and Kentaro Inui. 2024. https://aclanthology.org/2024.bea-1.21 Towards automated document revision: Grammatical error correction, fluency edits, and beyond . In Proceedings of the 19th Workshop on In...

    2024

  6. [14]

    Vipul Raheja, Dhruv Kumar, Ryan Koo, and Dongyeop Kang. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.350 C o E d IT : Text editing by task-specific instruction tuning . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5274--5291, Singapo...

    2023 doi

  7. [15]

    Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. https://doi.org/10.1162/tacl_a_00107 Optimizing statistical machine translation for text simplification . Transactions of the Association for Computational Linguistics, 4:401--415

    2016 doi

  8. [16]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://openreview.net/forum?id=SkeHuCVFDr Bertscore: Evaluating text generation with bert . In International Conference on Learning Representations

    2020

  9. [17]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  10. [18]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.