{"id":"559ddb84-a88f-42c9-9fa8-704f9172f68b","arxiv_id":"2411.18811","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"NewsEdits 2.0 introduces an edit-intention taxonomy and text-based models that predict factual updates in news revisions, enabling LLMs to abstain from answering with outdated facts at near-oracle accuracy.","lead":"This paper introduces a taxonomy for classifying why news articles are revised, separating factual updates from stylistic and narrative edits, and trains models to tag millions of revision pairs. It then shows that predictions about likely factual updates can help large language models abstain from answering questions with outdated information.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Near-oracle abstention claim rests on a GPT-4 judge whose final abstain/answer labels were validated only indirectly on 100 questions; the reported variant differences are small enough that judge error could flip them.","rationale":"The reader's conditional verdict is appropriate. The paper builds a real three-stage pipeline, and the update-prediction result is not obviously invalid: models do beat GPT baselines on the gold test set, and the high-precision subset is at least directionally plausible. However, the most load-bearing step for the abstract's headline claim is the abstention evaluation, where the ground truth is not human-annotated but generated by a GPT-4 judge. The reader identified the judge as the weakest assumption; I agree that this is the right place to look, but the more precise vulnerability is that validation was done on the two sub-questions separately, not on the derived abstain/answer decision, and the reported performance differences between variants are only a few F1 points. Small systematic judge biases could therefore change the qualitative conclusion. A secondary concern is that the high-precision subset (precision .74) lacks a stated threshold, coverage, or confidence intervals, and appears to be selected post hoc; this weakens the 'high precision' claim but is less central than the judge issue because the downstream result depends mainly on ranking quality. I do not think the paper should be rejected: the failure mode is testable, the authors are transparent about noise, and the comparison against a uniform-warning baseline provides a reasonable control. But the conditional should remain: release the judge-validation data, report agreement on the final decision, and provide confidence intervals for the key comparisons before the near-oracle claim is taken at face value.","tokens_in":19440,"tokens_out":8349,"duration_ms":83030,"concrete_test":"Sample roughly 300 question-sentence-variant instances, stratified across No-Conflict, Maybe-Conflict, and Likely-Conflict and across all four prompting variants. Have two independent human annotators (or a second, different LLM judge) label the final abstain/answer decision directly from the old sentence, revised sentence, and question, blind to the prompting variant. Compute agreement on the final decision and re-estimate Table 8's Micro/Macro F1 and Table 9's abstention rates using these human labels. If Update Prediction no longer shows a meaningful calibration advantage over Uniform Warning (e.g., the 95% confidence interval for the abstention-rate gap in No-Conflict includes 0), the near-oracle claim should be downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that update predictions make LLM abstention 'near-oracle' is evaluated against ground-truth labels produced by a GPT-4 judge. A question is considered correctly abstained only if the judge says the question is not answerable from the old sentence OR not consistent with the revised sentence. The judge was validated on only 100 manually labeled questions, and only per-question kappas (>0.74) are reported, not agreement on the derived final abstain/answer decision. Because the final decision is an OR of two imperfect binary labels, judge errors can amplify, and the accuracy gaps that carry the paper's claim are very small: w. Update Pred vs. Uniform Warning differs by 0.6 points in Maybe-Conflict and 2.4 points in Likely-Conflict; Oracle is within 1.6 points of Update Pred in every row. If the judge is lenient toward abstention for any question whose revised sentence differs, or if No-Conflict questions are underrepresented in the 100-question validation, then Table 8 overstates the value of the update predictor and 'near-oracle' becomes an artifact of judge bias. Table 9 shows the real benefit is better calibration of abstention rates, not raw accuracy; that calibration claim also needs a human-validated final decision to be trustworthy.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces NewsEdits 2.0, a taxonomy of edit intentions (Factual, Style, Narrative, Other) with 20 fine-grained categories, developed with professional journalists. The authors annotate 9,200 sentence-revision pairs, train LED-based sequence-to-sequence models to tag edit intentions, then use silver labels from these models to build a large training set for predicting, from an older sentence alone, whether a fact in it will update in a later version. They evaluate this predictor on held-out gold annotations, report a high-precision region at 0.74 precision, and finally test whether these update-likelihood predictions can improve LLM abstention in a RealTimeQA-style setup, reporting near-oracle accuracy.","tokens_in":19652,"tokens_out":8534,"duration_ms":66683,"significance":"The main contributions are substantial: a new edit-intention taxonomy grounded in journalism practice, a 9,200-pair gold annotation with professional journalists, a 4-million-pair silver-labeled corpus, a text-only factual-update prediction task, and a downstream LLM abstention task. If the results hold, they provide evidence for the hypothesis that linguistic features signal factual fluidity, and a practical way to make LLMs more cautious with outdated documents. The pipeline is largely reproducible from public components, and the authors promise code release. However, the empirical support for the two strongest claims ('high precision' and 'near-oracle abstention') is not yet methodologically solid: the precision figure comes from a post hoc subset, and the abstention evaluation rests on a judge whose final decision has not been validated and whose errors could flip the small reported differences.","major_comments":[{"comment":"The final abstain/answer decision is computed as an OR of two GPT-4 judgments: “Is this question answerable given the old sentence?” and “Is the answer consistent with the revised sentence?” The validation on 100 questions reports per-question kappas (>0.74) but not agreement on the derived final decision. Because errors in either sub-judgment propagate through the OR, the small accuracy gaps in Table 8 (e.g., 0.6 points in Maybe-Conflict and 2.4 points in Likely-Conflict between Update Pred and Uniform Warning; Update Pred within 1.6 points of Oracle in every row) are within the likely error band of the judge. The “near-oracle” claim needs human validation of the final decision, ideally with the judge blind to the experimental condition, plus a confusion matrix for the judge-versus-human final labels.","section":"Section 5.1 (Abstention Rate Evaluations)"},{"comment":"No confidence intervals or significance tests are reported, and the sizes of the gold test set (Part 2) and the question set (Part 3) are not stated in the main text. The differences that carry the paper’s claims are small — for instance, in Table 8 the Uniform Warning and Update Pred variants differ by only 0.6 points in Maybe-Conflict micro-F1, and Update Pred is essentially tied with Oracle. Without an N, a confidence interval, or a significance test, these point estimates cannot support the claim of near-oracle performance.","section":"Tables 4, 8, 9"},{"comment":"The advertised precision of .74 is computed on a post hoc high-confidence subset of sentences. The manuscript does not explain how the confidence threshold is chosen or whether it is tuned on a held-out set. If the threshold is selected after looking at the gold test labels, the .74 is an optimistic estimate, and the abstract’s claim to predict updates “with high precision” should be explicitly scoped to the high-likelihood region rather than to the general prediction task.","section":"Section 4.2 and Figure 4"},{"comment":"The factual-update predictor is trained on silver labels generated by the authors’ own edit-intention model, and the Oracle variant in the Part 3 experiments is defined using the same NewsEdits 2.0 gold taxonomy. Since the Part 2 test labels and the Oracle labels share that taxonomy, systematic biases in the schema (or in the silver-labeling step) are shared across training, evaluation, and the upper-bound condition. This is not a fatal circularity because the Part 2 predictor is evaluated on held-out gold annotations, but the manuscript should discuss this shared-schema limitation and, if possible, include a robustness check using a narrower definition of factual updates (e.g., only Event Update and Correction labels).","section":"Sections 4.1 and 5.1"},{"comment":"The FT Longformer Direct Context row reports Macro F1 = 87.8 with class F1s of 22.3 (Fact) and 93.0 (Not Fact), which is arithmetically inconsistent (the average of the two class F1s is about 57.7). This looks like a typo, but it prevents the reader from trusting the table. In addition, the text “At 41.2 F1-score, the journalist sets a moderately higher upper bound” should identify that 41.2 is the Fact-class F1, not the macro F1; used without this clarification it understates the model’s macro F1 of 58.0.","section":"Table 4 and Section 4.2"}],"minor_comments":[{"comment":"There are typos: “absention” in the abstract and “acnkowledge” in Section 4.1; please correct them.","section":"Abstract and Section 4.1"},{"comment":"The “Avg.” column is identical to the Micro F1 column in every row; please state what it averages or remove it.","section":"Table 8"},{"comment":"The final inter-annotator agreement on the full 9,200-pair annotation is not reported; only the training criterion (kappa > .6 on a 50-pair gold set) is given. Please report the kappa or other agreement measure on the final annotated data.","section":"Section 3.2"},{"comment":"The number of sentence pairs and the number of questions used in the LLM-QA experiment are not reported in the main text; please add these counts, as they are needed to assess the reliability of the point estimates in Tables 8 and 9.","section":"Section 5.1"},{"comment":"The evaluation prompt instructs the judge to answer question 2 with n/a when question 1 is “no”, but the main text says to abstain if either answer is “no”; please specify how n/a is mapped to the final abstain/answer label.","section":"Appendix D.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for a CL journal, and the resource contribution (NewsEdits 2.0, the annotated corpus, and the silver-labeled corpus) is likely to be valuable to the community. The main risk is that the headline claims go beyond what the current evaluation supports; the Part 3 experiment in particular needs the judge validation and uncertainty analysis described in my major comments. The authors' ethical and limitations sections are unusually thoughtful, and I found no citation concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe real contribution here is the NewsEdits 2.0 taxonomy and the gold-annotated data: 9,200 sentence-revision pairs labeled by professional journalists, separating factual, stylistic, and narrative edits. That resource is reusable, and the paper is honest about how noisy the modeling is. It reports moderate F1s and still pushes through to a downstream QA abstention experiment. I give it credit for that transparency.\n\nWhat is new: the taxonomy, the silver-labeled corpus of 4 million revision pairs, and the task of predicting factual updates from old article text alone. The result that a fine-tuned Longformer beats GPT-3.5/GPT-4 at this, while a journalist is only moderately better, is a solid empirical point. The predictor is evaluated on held-out gold annotations, which partly answers the circularity objection about training on silver labels from the authors' own tagger. That circularity is real but not damning.\n\nSoft spots, in order.\n\nFirst, the near-oracle abstention claim is load-bearing and rests on a GPT-4 judge validated on only 100 questions. The reported kappas are per-question, not on the derived final abstain/answer decision, which is an OR of two imperfect binary labels. Judge errors can amplify, and the differences between Update Pred and Oracle are tiny (roughly 0.6-1.6 points across rows). A lenient judge toward abstention could flip the conclusion. This needs human validation of the final decision and confidence intervals. The real value may be better calibration of abstention rates, but that also needs a human-validated final decision.\n\nSecond, the 0.74 precision headline comes from a post hoc high-confidence subset. The paper is upfront about this and includes a precision-pooling figure, but the abstract's \"high precision\" should come with recall/coverage stated.\n\nThird, there are no error bars anywhere, and Part 3 sample sizes are not clear in the main text. The tagger's Fact macro-F1 around 50 means downstream results inherit a lot of noise.\n\nFourth, code and data are promised but no artifacts appear in the manuscript.\n\nCitation pattern is fine: heavy use of the authors' own NewsEdits corpus is expected, and related work on Wikipedia and student-essay edit intentions is covered.\n\nWho this is for: computational journalism, news revision, temporal QA, and LLM abstention under stale information. It deserves a serious referee. My recommendation: send to peer review, with a required major revision focused on the abstention evaluation.","headline":"Useful taxonomy and corpus for news edit intentions, but the near-oracle abstention claim rests on a thinly validated GPT-4 judge and small differences.","tokens_in":20236,"tokens_out":3599,"would_cite":true,"duration_ms":33554,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that linguistic features in an old news draft predict which facts will later be updated, and that these predictions let an LLM abstain from stale questions almost as well as knowing ground truth.","keywords":["news revision history","factual updates","edit-intention taxonomy","LLM abstention","question answering","news discourse","silver-label corpus","temporal reasoning"],"falsifier":"Re-run the abstention experiment with an independently human-annotated set of at least 1,000 question-answer pairs (or with judges from two different LLMs) and compare abstention F1; if the update-prediction variant no longer matches the oracle condition, the near-oracle claim fails. A second check: replace the silver labels used for training the factual-update predictor with gold labels from the annotated test set and see whether the high-precision region and downstream abstention gains persist.","tokens_in":19188,"feed_emoji":"📰","tokens_out":6488,"duration_ms":50948,"temperature":0.7,"pith_summary":"News articles are living documents: as events unfold, facts get corrected, expanded, or replaced, and an LLM answering from an old draft can confidently give outdated answers. This paper tests the hypothesis that the text of an article itself carries linguistic signals of 'factual fluidity' — that a sentence about an advisory or a developing event is more likely to be factually rewritten than a sentence describing a historical detail or a quote. To test this, the authors build the NewsEdits 2.0 taxonomy, a schema of edit intentions drawn up with professional journalists, annotate over 9,200 revision pairs, and train an ensemble model that tags edits at 75.1 micro-F1. Using those tags to silver-label a large revision corpus, they train a predictor of whether a sentence in an older draft will receive a factual update, and find high precision (0.74) in the model's high-confidence region. They then show that when an LLM is given these update-likelihood predictions, it abstains from answering potentially outdated questions at near-oracle accuracy.","feed_headline":"Linguistic cues reveal which news facts are about to change","feed_subtitle":"A taxonomy of edit intentions and a confidence signal let LLMs refuse stale questions at near-oracle accuracy.","key_machinery":"The central object is the NewsEdits 2.0 taxonomy, a four-coarse, twenty-fine-grained schema of edit intentions (Factual, Style, Narrative/Contextual, Other) developed with professional journalists to separate fact updates from stylistic and narrative rewrites. Around that schema the paper builds a pipeline: a sequence-to-sequence LED (Longformer) model tags sentence-revision pairs with edit intentions; its tags silver-label a large corpus; a second Longformer, fed only the old sentence and article, predicts the probability of a factual update; and those probabilities are binned into low/medium/high and injected into an LLM's prompt for a RealTimeQA-style abstention task. The load-bearing mechanism is the two-stage transfer: semantic edit-intention labels learned on a small gold-annotated set are used to train the factual-fluidity predictor at scale, and the predictor's high-precision region drives the downstream abstention gains.","core_discovery":"The central claim is that factual updates to news sentences are not random: they are signalled by linguistic cues in the original text, and a model trained on revision histories can learn those cues. The paper isolates fact updates from stylistic and narrative edits using the NewsEdits 2.0 edit-intention taxonomy, and shows that pretrained LLMs (GPT-3.5, GPT-4) perform poorly at this prediction task, while a fine-tuned Longformer reaches a macro-F1 of 0.58 overall and precision of 0.74 when predictions are restricted to its most confident sentences. In the paper's use-case, giving GPT-4 a binned update-likelihood score for the retrieved sentence leads to abstention behavior that matches or slightly exceeds the oracle condition, where GPT-4 is told the ground-truth update status.","pith_inferences":["The two-stage pipeline (annotate edit intentions, then predict factual updates) could transfer to other versioned text domains—product documentation, wikis, legal filings—where future changes may also leave linguistic traces; this is an inference, not a paper claim.","A newsroom tool could surface only the model's highest-confidence update candidates for human re-verification, a natural productization of the reported high-precision region that the paper does not propose.","Because the abstention experiments used a single LLM and a judge validated on only 100 questions, testing the gains across other LLMs and with larger human-judged sets would show whether the near-oracle result generalizes; this is an extension, not a paper claim."],"forward_implications":["Retrieval-augmented QA systems could use the update-likelihood signal to abstain or flag uncertainty on sentences likely to go stale, reducing the spread of outdated information.","The NewsEdits 2.0 schema and the silver-labeled corpus of 4 million revision pairs provide a reusable resource for studying style and narrative edits, not just factual ones.","The finding that sentence-level cues carry most of the predictive signal means lightweight, text-only models can flag fluid facts without external retrieval or knowledge bases.","The near-oracle abstention result implies that, in the high-confidence region, a cheap text-only model can stand in for ground-truth update knowledge."],"supporting_citations":[{"why":"Supplies the NewsEdits corpus of article revision histories and the initial syntactic edit-action prediction task that this paper extends with semantic labels.","marker":"Spangher et al., 2022"},{"why":"Defines the RealTimeQA-style setup and the goal of abstaining when retrieved information is outdated, which the paper's use-case builds on.","marker":"Kasai et al., 2022"},{"why":"Provides an edit-intention schema for student essays that influenced the NewsEdits 2.0 taxonomy.","marker":"Zhang and Litman, 2015"},{"why":"Provides the Wikipedia edit-intention schema that the taxonomy builds on and adapts to news-specific categories.","marker":"Yang et al., 2017"},{"why":"Supplies the Longformer (LED) architecture used for edit-intention tagging and factual-update prediction.","marker":"Beltagy et al., 2020"},{"why":"Supplies the news discourse schema used to define narrative/contextual edit categories and to interpret linguistic cues.","marker":"Van Dijk, 1998"},{"why":"Provides the definition of events used in the Factual Edit subcategories.","marker":"Doddington et al., 2004"},{"why":"Supplies quote/source detection models used as features for edit-intention tagging and the social-interest categories in the exploratory analysis.","marker":"Spangher et al., 2023"},{"why":"Supplies the adversarial NLI model used as a textual-entailment feature for edit-intention tagging.","marker":"Nie et al., 2020"}],"fun_headline_variants":["Predict news updates from language alone","Spot soon-outdated news facts","Cues in news text predict fact updates","Learn which news facts are likely to change","News text hints at upcoming fact changes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automated judge used to score abstentions—a GPT-4 evaluator validated on only 100 manually labeled questions—correctly decides when a question is answerable from the old sentence and when its answer conflicts with the revised sentence. If that judge is biased, the reported near-oracle abstention accuracy could be inflated.","fun_headline_variants_meta":{"raw":{"variants":["Predict news updates from language alone","Spot soon-outdated news facts","Cues in news text predict fact updates","Learn which news facts are likely to change","News text hints at upcoming fact changes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000376,"raw_usage":{"total_tokens":2001,"prompt_tokens":940,"completion_tokens":1061,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":1000}},"tokens_in":556,"tokens_out":1061,"duration_ms":9362,"temperature":1.0,"reasoning_tokens":1000,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T10:51:34.887164+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the abstention experiment with an independently human-annotated set of at least 1,000 question-answer pairs (or with judges from two different LLMs) and compare abstention F1; if the update-prediction variant no longer matches the oracle condition, the near-oracle claim fails. A second check: replace the silver labels used for training the factual-update predictor with gold labels from the annotated test set and see whether the high-precision region and downstream abstention gains persist.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the NewsEdits corpus of article revision histories and the initial syntactic edit-action prediction task that this paper extends with semantic labels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides an edit-intention schema for student essays that influenced the NewsEdits 2.0 taxonomy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Wikipedia edit-intention schema that the taxonomy builds on and adapts to news-specific categories."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the news discourse schema used to define narrative/contextual edit categories and to interpret linguistic cues."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the definition of events used in the Factual Edit subcategories."},{"cited_title":"Identifying Informational Sources in News Articles","cited_arxiv_id":"2305.14904","evidence_quote":"Supplies quote/source detection models used as features for edit-intention tagging and the social-interest categories in the exploratory analysis."}],"review_version":1}