Pith. sign in

REVIEW 3 major objections 5 minor 2 references

AtomEval: Validity-Aware Atomic Evaluation of Adversarial Claim Rewriting in Fact Verification

T0 review · 3 major / 5 minor · reviewed 2026-07-13 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Conventional attack success rates overstate how well language models can rewrite false claims to fool fact checkers, because many rewrites change the false claim itself rather than preserve it.

desk verdict Clean fixed-evidence re-evaluation that shows ASR is often inflated by proposition-changing rewrites; VASR and the diagnostics are the real contribution. read the letter →

arxiv 2604.07967 v3 pith:DUC3BMNI submitted 2026-04-09 cs.CL cs.AI

Hongyi Cen , Mingxin Wang , Yule Liu , Jingyi Zheng , Hanze Jia , Tan Tang This is my paper · ORCID
classification cs.CLcs.AI
keywords adversarialclaimrewritingfactverificationattacksuccessrateatomicevaluationpropositionpreservationSROMVASRFEVER
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

When large language models rewrite already-refuted claims so that fact verifiers stop labeling them false, the usual attack success rate (ASR) can look high for the wrong reason: many of those rewrites alter, weaken, or correct the original false statement instead of keeping it. AtomEval decomposes claims into subject–relation–object–modifier atoms and applies a one-way preservation check so a rewrite counts as valid only if the original false proposition remains recoverable. It reports validity-aware attack success rate (VASR), which multiplies verifier prediction flips by that preservation gate, and adds diagnostics that separate evidence drift and scope loss from valid but non-minimal additions. On FEVER refuted-claim rewriting under fixed evidence, the ASR–VASR gap is large—for example omission-style rewrites can score near 78% ASR but near 0% VASR—and surface similarity or fluency filters do not recover proposition-level validity. A small repair probe shows the diagnostics point to editable failures, while also showing how hard it is to keep the verifier fooled once the original proposition is restored. The paper’s aim is a stable evaluation target for adversarial rewriters that must balance evasion with proposition preservation.

What carries the argument

Subject–relation–object–modifier (SROM) atoms and a one-way binary preservation gate H(C, C′): every original atom must remain recoverable from the rewrite via natural-language entailment plus constraint checks on dates, quantities, exclusivity, and related truth-critical modifiers; VASR multiplies the verifier flip indicator by H.

What would settle it

On a larger stratified human audit, if expert valid/invalid labels systematically disagree with AtomEval’s gate—especially on borderline scope and entity cases—so that re-scoring with human validity erases or reverses the large ASR–VASR gaps for omission-style and related attacks, the inflation claim fails.

Watch

Extended reading notes

Core claim

Under fixed-evidence rewriting of correctly classified FEVER REFUTED claims, conventional ASR substantially overestimates valid adversarial success because many prediction-flipping rewrites change, weaken, or correct the original false proposition. AtomEval’s VASR counts only flips that also pass a one-way atomic preservation gate, and its diagnostics make that ASR–VASR gap interpretable rather than merely measurable.

Load-bearing premise

The method treats automatic subject–relation–object–modifier extraction plus a frozen entailment model with fixed rules for dates, quantities, and exclusivity as enough to decide whether the original false claim is still present in a rewrite.

Editorial extensions

If this is right

  • Robustness numbers for fact verifiers should report VASR (or an equivalent proposition-preservation filter), not raw ASR alone.
  • Sentence-level similarity and fluency screens will not recover the same valid attacks as atomic preservation checks.
  • Constraint-weakening and omission-style attacks will be re-ranked downward once proposition-changing rewrites are filtered out.
  • Evidence-drift, scope-loss, evidence-entanglement, and unsupported-addition labels give a concrete vocabulary for diagnosing and repairing invalid rewrites.
  • Adversarial generators should be evaluated for the trade-off between verifier evasion and attacked-proposition preservation, not evasion alone.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same ASR inflation is likely to appear in open-retrieval or multi-hop settings once claim rewrites can also steer which evidence is fetched.
  • Papers that claim high ASR against claim-only verifiers may need validity-aware reporting as a standard check, not an optional diagnostic.
  • Training or filtering generators against an AtomEval-style gate would create a measurable evasion–preservation trade-off curve rather than a single ASR score.
  • The SROM-plus-entailment design may transfer to other claim-side transformations (paraphrase, style shift) beyond the five FEVER-adapted attack families studied here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes AtomEval, a validity-aware protocol for evaluating fixed-evidence adversarial rewriting of FEVER REFUTED claims. Claims are decomposed into SROM atoms; a one-way preservation gate H(C,C′) (Eqs. 1–3) requires every original atom to remain recoverable from the rewrite via NLI entailment plus consistency checks. Validity-aware attack success rate (VASR) multiplies the usual prediction-flip indicator by H, so only proposition-preserving evasions count. Four post-hoc diagnostics (EvDrift, ScopeLoss, EvEnt, UnverAdd) explain invalid raw successes and non-minimal valid ones. On 400 REFUTED claims, two generators, five attack families, and two verifiers, raw ASR is often far higher than VASR (e.g., Mistral Omission-style 78.40 ASR vs 0.27 VASR on Gemma-7B). Surface filters (S-ASR, P-ASR) do not close the gap. Human audit (κ=0.84, BinaryGate F1 87.18%, extractor Acc 94%) and ablations support the gate; a small repair probe shows diagnostics are actionable but ASR retention under validity is hard.

Significance. If the result holds, it changes how adversarial claim rewriting should be scored: ASR alone is an unreliable robustness signal when LLM rewriters can alter, weaken, or correct the attacked proposition. The contribution is a concrete, auditable operationalization (SROM + one-way gate + VASR + diagnostics) with a unified fixed-evidence protocol, human validation, and large, interpretable ASR–VASR gaps. Strengths include the stratified human audit, extractor comparison, validity-decision ablation (Table 4), and diagnostic breakdowns that make the inflation explainable rather than merely measured. The work supplies a clearer evaluation target for validity-constrained rewriters and for claims about verifier robustness under claim-side attacks.

major comments (3)
  1. [§3.3 Eqs. (1)–(3); Table 5; §5] §3.3 Eqs. (1)–(3) and Appendix C: the central validity claim rests on automatic SROM extraction plus a frozen DeBERTa-v3 NLI check with hand-specified Cons constraints as the operational definition of “original false proposition remains recoverable.” The audit (extractor Acc 94%, BinaryGate F1 87.18%, Table 4) supports aggregate use, but borderline scope/entity/implicit-reference cases can still flip H. The manuscript should quantify sensitivity of the headline ASR–VASR gaps (Table 5) to gate errors—e.g., error-propagation bounds, human-gated VASR on the audit subset, or confidence intervals—so the inflation conclusion is not solely dependent on the automatic gate.
  2. [§4.1–4.2; Table 5] §4.1–4.2 and Table 5: the attack set is 400 REFUTED FEVER test claims, with verifier-specific attackable subsets (n=375 Gemma, n=331 BERT). The extreme gaps (especially Omission-style) are load-bearing for the claim that ASR is substantially inflated. The paper should report sampling procedure, confidence intervals or bootstrap uncertainty on ASR/VASR, and whether results are stable under alternative attack-set draws or claim-length/entity strata, so the magnitude of inflation is not tied to a single fixed sample of 400.
  3. [§6.2 Diagnostic-guided repair] §6.2 repair probe: only 40 invalid raw successes are repaired; all pass the gate after edit, but only 9 retain verifier evasion. This is used to argue that diagnostics are actionable and that proposition-preserving evasion is hard. With n=40 and a single GPT-4.1 editor, the probe is too small to support that trade-off as a general finding. Either enlarge and stratify the probe (by family/generator/verifier) or clearly demote it to a qualitative illustration so it does not over-extend the main FEVER re-evaluation.
minor comments (5)
  1. [Table 5; §6.1] Table 5: S-ASR (SBERT ≥0.65) and P-ASR (GPT-2 PPL ≤100) are useful controls, but the thresholds are free parameters. Briefly justify or sensitivity-check them so readers do not treat them as canonical validity proxies.
  2. [Figure 1; Table 9; Appendix B] Figure 1 and Table 9 case studies are clear; ensure all diagnostic tags in the case table are defined in the main text before first use, not only in Appendix B.
  3. [Limitations] Limitations correctly flag English FEVER scope and extractor dependence; a short note on whether multi-hop or multi-atom claims systematically lower BinaryGate agreement would help readers gauge external risk.
  4. [§3.3 Eq. (3)] Notation: H(C,C′) is product over atoms (Eq. 3); state explicitly that empty A(C) is excluded or handled, and that multi-atom claims require all atoms preserved (one-way, not bijection).
  5. [Appendix A–D] Reproducibility: release of the SROM extractor, audit labels, and attack prompts (Appendix D) would strengthen the evaluation protocol claim; if not released at submission, state the planned artifact package.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: VASR is a defined evaluation metric validated against human labels, not a prediction forced by its inputs or by self-citation.

full rationale

AtomEval is an evaluation protocol paper, not a first-principles derivation. Its central empirical claim—that conventional ASR on fixed-evidence FEVER REFUTED rewrites is inflated by proposition-changing rewrites—is measured, not derived. VASR (Eq. 5) is defined as the product of the raw flip indicator and the one-way preservation gate H(C,C′) (Eqs. 1–3); H is operationalized via SROM extraction plus frozen DeBERTa-v3 NLI and Cons checks, then validated against adjudicated human VALID/INVALID labels (BinaryGate F1 87.18%, extractor Acc 94%, ablations in Table 4). The extractor is trained on a FEVER-derived corpus explicitly disjoint from the 400-claim attack set (Appendix A); diagnostics (EvDrift, ScopeLoss, EvEnt, UnverAdd) are post-hoc and do not enter H or VASR. No parameter is fitted to maximize the ASR–VASR gap and then reported as a prediction. Self-citations (e.g., Liu et al. 2025a for attack-family taxonomy) supply protocol context, not a uniqueness theorem or load-bearing premise that forces the inflation result. There is no self-definitional loop, fitted-input-as-prediction, uniqueness import, ansatz smuggling, or renaming of a known result as a new derivation. The paper is self-contained against its human audit and reported measurements.

Assumptions & free parameters 3 free parameters · 4 assumptions · 3 invented entities

The central claim is empirical-methodological: ASR overcounts invalid rewrites under a stated validity criterion. It rests on domain choices about what counts as a valid attack (proposition preservation under fixed evidence), on standard NLI/extraction tooling, and on a few comparison thresholds that do not define VASR itself. No physical entities are postulated; the invented objects are evaluation constructs validated against human labels.

free parameters (3)
  • S-ASR SBERT similarity threshold
    Set to ≥0.65 as a permissive surface filter for comparison only; not used in VASR, but affects how strongly the paper can claim surface proxies fail.
  • P-ASR GPT-2 perplexity threshold
    Set to ≤100 as a permissive fluency filter for comparison only; same role as the SBERT cutoff.
  • Attack-set size and sampling (400 REFUTED FEVER test claims)
    Hand-chosen evaluation scale; results are conditional on this sample and on initially correct verifier subsets (n=375/331).
assumptions (4)
  • domain assumption A valid adversarial rewrite of a REFUTED claim must preserve the original attacked false proposition under the same fixed evidence, not merely flip the verifier label.
    Stated as the evaluation objective in §3.1; this is the normative criterion that makes VASR the target metric.
  • ad hoc to paper One-way recoverability of every original SROM atom via NLI entailment plus Cons checks is an adequate operationalization of proposition preservation (Eqs. 1–3).
    AtomEval’s binary gate; alternative notions (bidirectional equivalence, human intent, multi-hop reason) could disagree on edge cases.
  • domain assumption Fixed-evidence, claim-only, single-pass black-box rewriting isolates claim-side validity from retrieval dynamics.
    Threat model §3.2; excludes evidence manipulation and open retrieval by design.
  • domain assumption A frozen NLI model can judge whether a rewrite premise entails a rendered atomic proposition for truth-critical content.
    Implementation Appendix C uses DeBERTa-v3 MNLI/FEVER/ANLI; standard tooling assumption with known error modes.
invented entities (3)
  • AtomEval / VASR (validity-aware attack success rate) independent evidence
    purpose: Count only verifier-flipping rewrites that pass the one-way attacked-proposition preservation gate.
    Core metric introduced by the paper; independent handle is agreement with human VALID labels on the audit sample, not an external physical measurement.
  • SROM atom representation for claims independent evidence
    purpose: Decompose claims into subject–relation–object–modifier units for preservation checking and diagnostics.
    Methodological representation trained/annotated for this protocol; related to prior atomic factuality work but specialized here with modifiers and one-way attack validity.
  • EvDrift / ScopeLoss / EvEnt / UnverAdd diagnostics
    purpose: Explain invalid raw successes and non-minimal valid rewrites without changing VASR.
    Post-hoc labels defined in Appendix B; useful analysis tools rather than independently measured natural kinds.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AtomEval: Validity-Aware Atomic Evaluation of Adversarial Claim Rewriting in Fact Verification." pith.science (2026). https://pith.science/paper/DUC3BMNI

@misc{pith2026260407967,
  author       = {Pith},
  title        = {Pith review of: AtomEval: Validity-Aware Atomic Evaluation of Adversarial Claim Rewriting in Fact Verification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DUC3BMNI}},
  note         = {Machine review of arXiv:2604.07967}
}
read the original abstract

Large language models (LLMs) can rewrite refuted claims to evade evidence-based fact verifiers, but conventional attack success rate (ASR) can be inflated when rewrites change, weaken, or correct the false proposition they are supposed to preserve. We introduce AtomEval, a validity-aware evaluation protocol for fixed-evidence adversarial claim rewriting. AtomEval represents claims as subject--relation--object--modifier (SROM) atoms, applies a one-way preservation gate to separate valid verifier evasion from proposition-changing rewrites, and reports validity-aware attack success rate (VASR), which counts only verifier-evasive rewrites that preserve the original false proposition. AtomEval further provides fine-grained diagnostics that explain both proposition-level failures and non-minimal valid rewrites. On FEVER refuted-claim rewriting, AtomEval exposes and explains ASR inflation: many apparent attacks fool the verifier by altering, weakening, or correcting the proposition they should preserve. By making attacked-proposition preservation explicit and measurable, AtomEval provides a stable evaluation target for evaluating adversarial rewriters that must balance verifier evasion with proposition preservation.

Figures

Figures reproduced from arXiv: 2604.07967 by the authors.

Figure 1
Figure 1. Valid and invalid adversarial rewrites of a refuted claim. A valid rewrite pre￾serves the original false proposition (red), whereas an invalid rewrite drifts toward the evidence￾supported fact (green) or introduces hallucinated content (blue). textual similarity alone can misclassify seman￾tically corrupted rewrites as successful attacks. In our analysis, such cases account for a sub￾stantial portion of attacks judg… view at source ↗
Figure 1
Figure 1. Regions in azimuthal angle defined with respect to a J/ψ candidate. In events containing a J/ψ candidate, the multiplicity can be separated into several azimuthal regions relative to the J/ψ emission direction. These azimuthal regions are as labelled in [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Overview of AtomEval, a validity-aware evaluation framework for adversarial claim rewriting in fact verification. Given an original claim, its retrieved evidence, and an adversarial rewrite, AtomEval decomposes the original and rewritten claims into atomic facts (SROM tuples) and evaluates them using a hard structural gate together with soft semantic degradation metrics. Blue, red, and green correspond to the origin… view at source ↗
Figures from the paper (11 more)
Figure 2
Figure 2. Figure 2: Raw inclusive J/ψ yield as a function of the selection on the BDT non-prompt output probability for 39 Ntrks < 46, and pT > 1 GeV/c. The error bars indicate statistical uncertainties. The lines correspond to the prompt (dashed), non-prompt (dot-dashed) and total (conti…
Figure 3
Figure 3. Figure 3: The modular prompt generation pipeline. The final adversarial prompt is synthesized by [PITH_FULL_IMAGE:figures/full_fig_p012_3.png]
Figure 4
Figure 4. Figure 4: Self-normalized prompt (left) and non-prompt (right) J/ψ yields with pT > 1 GeV/c as a function of self-normalized charged-particle multiplicity at midrapidity. The data are shown separately for the INEL>0 and V0M 0–0.1% event classes. The yields are compared to PYTHIA…
Figure 5
Figure 5. Figure 5: Prompt J/ψ yield for 1 < pT < 4 GeV/c (left), 4 < pT < 8 GeV/c (middle) and pT > 8 GeV/c (right) as a function of self-normalized charged-particle multiplicity within |η| < 0.9 for the INEL>0 and V0M 0–0.1% event classes. The data are compared to PYTHIA 8 using differe…
Figure 6
Figure 6. Figure 6: Non-prompt J/ψ yield for 1 < pT < 4 GeV/c (left), 4 < pT < 8 GeV/c (middle) and pT > 8 GeV/c (right) as a function of self-normalized charged-particle multiplicity within |η| < 0.9 for the INEL>0 and V0M 0–0.1% event classes. The data are compared to PYTHIA 8 using dif…
Figure 7
Figure 7. Figure 7: The fractions of J/ψ mesons from beauty feed-down with pT > 1 GeV/c as a function of the self￾normalized charged-particle multiplicity in |η| < 0.9 in the toward (top-left), transverse (top-right), and away (bottom left) azimuthal regions with respect to the J/ψ moment…
Figure 8
Figure 8. Figure 8: Self-normalized prompt J/ψ yields with pT > 1 GeV/c as a function of the self-normalized charged￾particle multiplicity in |η| < 0.9 measured in the toward (left), transverse (middle), and away (right) azimuthal regions with respect to the J/ψ momentum direction. The da…
Figure 9
Figure 9. Figure 9: Self-normalized non-prompt J/ψ yields with pT > 1 GeV/c as a function of the self-normalized charged-particle multiplicity in |η| < 0.9 measured in the toward (left), transverse (middle), and away (right) azimuthal regions with respect to the J/ψ momentum direction. Th…
Figure 10
Figure 10. Figure 10: Self-normalized prompt J/ψ yields in the pT intervals 1 < pT < 4 GeV/c (left column), 4 < pT < 8 GeV/c (middle column), and pT > 8 GeV/c (right column) as a function of the self-normalized charged-particle multiplicity in |η| < 0.9 measured in toward (top row), transv…
Figure 11
Figure 11. Figure 11: Self-normalized non-prompt J/ψ yields in the pT intervals 1 < pT < 4 GeV/c (left column), 4 < pT < 8 GeV/c (middle column), and pT > 8 GeV/c (right column) as a function of the self-normalized charged￾particle multiplicity in |η| < 0.9 measured in toward (top row), tr…
Figure 12
Figure 12. Figure 12: Ratio between the pT-integrated prompt J/ψ and prompt D0 yields as a function of charged-particle multiplicity within |η| < 0.5 in several collision systems [52, 96, 97]. This is compared to the prediction from the Statistical Hadronization Model (SHMc) [98]. The vert…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 1 linked inside Pith

  1. [1]

    Pepa Atanasova, Jakob Grue Simonsen, Christina Li- oma, and Isabelle Augenstein

    PMLR. Pepa Atanasova, Jakob Grue Simonsen, Christina Li- oma, and Isabelle Augenstein. 2022. Fact checking with insufficient evidence.Transactions of the Asso- ciation for Computational Linguistics, 10:746–763. Pepa Atanasova, Dustin Wright, and Isabelle Augen- stein. 2020. Generating label cohesive and well- formed adversarial claims. InProceedings of th...

    arXiv 2022

  2. [2]

    repair_claim

    GEM: Generative enhanced model for adver- sarial attacks. InProceedings of the Second Work- shop on Fact Extraction and VERification (FEVER), pages 20–26, Hong Kong, China. Association for Computational Linguistics. Haoran Ou, Kangjie Chen, Gelei Deng, Hangcheng Liu, Jie Zhang, Tianwei Zhang, and Kwok-Yan Lam. 2026. Deceive-afc: Adversarial claim attacks ...

    arXiv 2026

Pith tools

Reviewed July 13, 2026 · model on record in the stance chip above.