{"id":"8f1320e1-2994-43ac-b1fa-d759970e2d0b","arxiv_id":"2506.07446","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AFEV iteratively decomposes complex claims into atomic facts, verifies each with reranked evidence and dynamic demonstrations, and reports state-of-the-art results on five fact verification benchmarks.","lead":"This paper introduces AFEV, a fact-checking pipeline that breaks complex claims into smaller atomic facts, checks each one against retrieved evidence, and then combines the results. The authors report top accuracy on five fact verification benchmarks, though the gains over prior systems are modest and no code is released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The interpretability half of the central claim is untested, and Eq. (2)'s rationale feedback can inject content absent from the original claim, so the 'state-of-the-art accuracy and interpretability' claim is not established.","rationale":"For the central claim to hold, AFEV must both beat all relevant systems and produce decompositions whose rationales are faithful to the original claim. The accuracy side is benchmarked, though comparisons with the most closely related frameworks (RAC, FIRE, and the authors' own MRRFV) are missing. The more internal weakness is the interpretability side: the paper's own design, Eq. (2), feeds LLM-generated rationales back into extraction, and Section 5.7 celebrates adding 'Lionel Messi' from a rationale even though it is not explicit in the original claim. If a rationale is wrong, every downstream atomic fact is conditioned on that error, and the final aggregate can decide a different claim than the one posed. The ablations do not isolate this failure: 'w/o Rationales' removes the feedback and loses only 0.13 LA on HOVER, while 'w/o Iterative Extraction' shows a larger drop, so the paper provides no evidence that rationale feedback is what helps. A human audit of atomic-fact faithfulness and rationale consistency would settle whether the iterative mechanism improves or corrupts decomposition, and whether the interpretability claim is justified. Because the reader already returned a CONDITIONAL verdict focused on the same feedback-loop assumption, this stress-test does not change the verdict; it sharpens the required condition: direct evaluation of rationale faithfulness and contamination frequency before the interpretability and error-propagation claims can be accepted. A secondary internal inconsistency (Table 5 and Table 6 report different values for '-w/o Iterative Extraction', 77.66 vs 77.04) further supports the need for code and reproducibility details, but it is not the primary load-bearing concern.","tokens_in":16599,"tokens_out":11750,"duration_ms":145367,"concrete_test":"Run AFEV on 150 randomly sampled claims from HOVER and 150 from LIAR-PLUS, then have two independent annotators judge each extracted atomic fact and its rationale chain: (1) Is every entity and relation in the atomic fact entailed by, or at least licensed by, the original claim? (2) Does the rationale support the assigned atomic label without introducing unsupported statements? Compute the fraction of atomic facts containing content absent from the original claim and the inter-annotator agreement. Then rerun the full system against a variant that conditions extraction only on C and F_{1:t-1}, dropping the y,r feedback; if the full system is not significantly better on claims with faithful atomic facts, the Eq. (2) feedback loop is not justified. If the injected-content rate is high, the interpretability claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract claims state-of-the-art accuracy and interpretability, but interpretability is never measured: there is no human evaluation, no faithfulness metric, and no error analysis of rationales. The one qualitative example, Section 5.7, exposes the risk. Equation (2) conditions the extractor on previous rationales r_{1:t-1}, and Figure 6 shows Fact3 incorporating 'Lionel Messi' from the rationale of Fact2 rather than from the original claim. Because the reasoner is GPT-3.5, an earlier rationale can hallucinate an entity or relation; conditioning the next extraction on it then injects a condition that was never part of the input claim. The final aggregation (Eq. 10) then judges a statement that is not the original claim, so both correctness and interpretability can be corrupted. The ablation evidence for the feedback loop is also weak: removing rationales ('w/o Rationales' in Table 5) costs only 0.13 LA (78.87 vs 78.74), so the claimed benefit is marginal while the failure mode is unmeasured. Since the paper reports no direct test of rationale faithfulness or of contamination frequency, the interpretability half of the headline claim, and the claim that iterative feedback reduces error propagation, are unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AFEV, a fact-verification framework that iteratively decomposes a complex claim into atomic facts, conditions each new atomic fact on previously extracted facts and their labels and rationales (Eq. 2), retrieves and reranks evidence per atomic fact, dynamically selects demonstrations, and aggregates per-fact verdicts into a final label (Eq. 10). The framework is evaluated on HOVER, PolitiHop, LIAR-PLUS, RAWFC, and LIAR, with additional retrieval, ablation, hyperparameter-sensitivity, and efficiency studies. The authors claim state-of-the-art accuracy and interpretability in the abstract and conclusion.","tokens_in":16888,"tokens_out":6410,"duration_ms":74397,"significance":"If the reported accuracy numbers are stable, AFEV is a useful contribution to LLM-based fact verification: it combines iterative decomposition with lightweight evidence reranking and dynamic demonstrations, and it shows gains over strong baselines such as CURE, VMASK, and RAFTS on five benchmarks. The experimental design is broad, the ablations cover all major modules, and the paper gives concrete prompts and implementation details. However, the interpretability half of the headline claim is not evaluated at all, and the case study in Section 5.7 illustrates a contamination mechanism through Equation (2) that could undermine the claimed error-reduction benefit. The accuracy claim itself rests on small margins and an underspecified significance test. These issues are fixable but require additional evidence or a rescoping of the claims.","major_comments":[{"comment":"The interpretability claim and the claim that iterative feedback reduces error propagation are not supported. Equation (2) conditions the extraction of the next atomic fact on previous rationales r_{1:t-1}; the case study in §5.7 explicitly shows that Fact3 includes 'Lionel Messi' because it appeared in the rationale of Fact2, not in the original claim. If a rationale contains a hallucinated entity or relation, that content enters later atomic facts and is ultimately passed to the final aggregation in Eq. (10), so the system can end up judging a statement that is not a decomposition of the original claim. The paper reports no human evaluation of rationales, no faithfulness metric, and no estimate of how often this contamination occurs. Therefore the 'interpretability' half of the headline claim, and the claim that the feedback loop reduces rather than amplifies errors, are unverified. The authors should either provide direct measurements of rationale faithfulness and contamination frequency or remove/qualify the interpretability and error-reduction claims.","section":"§5.7, Eq. (2), Eq. (10)"},{"comment":"The statistical evidence for the state-of-the-art accuracy claim is weak as reported. The tables state that the improvement over the best baseline is statistically significant by a sign test with p<0.01, but the test is not described: there is no indication of the number of paired samples, whether the test is per claim or per dataset, or whether it accounts for multiple comparisons. The observed margins are small (e.g., LIAR F1 43.9 vs 42.0 in Table 3; PolitiHop label accuracy 74.14 vs 72.34 in Table 2), and no error bars, standard deviations, or multi-seed results are given. Because these numbers are the sole support for the central accuracy claim, the authors should provide a precise account of the significance test, report variance over independent runs, and state whether hyperparameters such as k=2 and d=1 were fixed before evaluating all datasets.","section":"§5.2, Tables 2 and 3"},{"comment":"The ablation supporting the rationale feedback loop is overstated. Removing rationales from the extractor input changes HOVER label accuracy from 78.87 to 78.74, a decrease of 0.13 point, while removing iterative extraction entirely decreases accuracy by 1.21 points and removing atomic-fact extraction by 1.83 points. The text's statement that removal of each component leads to a 'noticeable decline' is therefore inaccurate for the rationales component. Moreover, if the feedback loop is intended to reduce error propagation, the 0.13-point effect and the contamination risk shown in §5.7 should be analyzed with error bars and an error analysis before concluding that the mechanism is beneficial.","section":"§5.4, Table 5"}],"minor_comments":[{"comment":"The 'w/o Iterative Extraction' ablation reports label accuracy 77.66 and Macro-F1 77.81 in Table 5, but label accuracy 77.04 and Macro-F1 76.96 in Table 6. Please reconcile these numbers.","section":"Tables 5 and 6"},{"comment":"The sentence that AFEV achieves state-of-the-art retrieval performance across all datasets is not supported by the table, since on HOVER the Pipeline baseline has a higher retrieval F1 (0.9427 vs 0.9423). Please correct the claim or clarify the metric being compared.","section":"§5.3, Table 4"},{"comment":"The caption identifies VMASK as the best baseline, but on HOVER the best baseline is CURE (76.98 vs 74.38). Please correct the caption or the significance description.","section":"Table 2 caption"},{"comment":"Please specify the exact GPT-3.5 model version, decoding temperature, and the number of initial retrieval candidates k'; also report whether k=2 and d=1 were tuned only on HOVER and then frozen for the other datasets, and provide standard deviations over multiple runs.","section":"§5.1.4 and §5.5"},{"comment":"The symbol d is used both for the number of dynamic demonstrations in Eq. (8) and for the embedding dimension in the complexity expression O(k'd); please disambiguate.","section":"§4.4"},{"comment":"The header 'Preprint submitted to Nuclear Physics B' appears to be a leftover template line and should be removed or corrected.","section":"Title page"}],"recommendation":"major_revision","confidential_remarks":"The mismatch between the submission line ('Nuclear Physics B') and the content may indicate a template error; the editor may wish to verify the submission venue. More substantively, the paper's claimed interpretability is entirely unsupported by evaluation, and the case study shows a concrete contamination mechanism. If the authors cannot add faithfulness measurements or a human study, the interpretability and error-reduction claims should be removed from the abstract and conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on Zheng et al. The genuinely new piece is the specific assembly: iterative atomic fact extraction conditioned on previous rationales, an LLM-supervised reranker, and dynamic demonstration retrieval. Each component is borrowed from existing work, but the combination is not present in RAC or FIRE, and the benchmark numbers are new. The paper is also careful in places: the ablations and efficiency study are useful, and reporting a sign test against the best baseline on each dataset is a step in the right direction. The accuracy gains over VMASK and RAFTS are small but consistent across five benchmarks, and I think the central accuracy claim is plausible.\n\nThe soft spots are real. The interpretability half of the headline claim is unsupported. There is no faithfulness metric, no human evaluation of the rationales, and no error analysis. The single qualitative example actually highlights the risk: Fact3 incorporates \"Lionel Messi\" from the rationale of Fact2 rather than from the original claim. If that rationale were hallucinated, the error would propagate into the next extraction and into the final aggregation. Nothing in the paper measures how often the feedback loop corrects versus corrupts. The ablation evidence is weak too: removing rationales costs only 0.13 label accuracy on HOVER, so the benefit of the loop is marginal while the failure mode is untested. The stress-test note gets this right.\n\nTwo other gaps stand out. First, there is no comparison with RAC and FIRE, the closest iterative atomic-fact frameworks. For a state-of-the-art claim, that is a notable omission. Second, there are no error bars or multiple seeds, and no code release. Hyperparameters are tuned on HOVER and applied elsewhere; that is acceptable, but the lack of variance reporting makes the margins hard to interpret. Minor: the \"Submitted to Nuclear Physics B\" template remnant is sloppy, and an author affiliation from a surgery department is odd, but neither affects the science.\n\nBottom line: the accuracy claim is plausible, the interpretability claim is not established. This deserves a serious referee who will ask for code, variance, a RAC/FIRE comparison, and a human faithfulness evaluation. I would send it to review rather than desk reject.","headline":"A solid, well-written pipeline paper with small but consistent accuracy gains, whose \"interpretability\" claim is never actually measured.","tokens_in":17389,"tokens_out":2527,"would_cite":false,"duration_ms":31252,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By iteratively decomposing complex claims into atomic facts and feeding each verified label and rationale back into the next extraction step, AFEV claims state-of-the-art accuracy and interpretability on five fact-verification datasets.","keywords":["fact verification","atomic fact extraction","multi-hop reasoning","evidence retrieval","large language models","iterative decomposition","interpretability","claim verification"],"falsifier":"Take a set of complex claims, run AFEV once normally, and run it again after manually replacing one entity in the first rationale with a plausible but false entity before the second extraction step. If final verification accuracy stays the same, Eq. (2)'s feedback is not doing the load-bearing work the paper attributes to it; if accuracy changes, the loop propagates rationale errors. Either result settles whether the feedback loop is the source of the reported gains.","tokens_in":16405,"feed_emoji":"✅","tokens_out":9822,"duration_ms":92640,"temperature":0.7,"pith_summary":"This paper tries to establish that the hard part of verifying complex claims is decomposition, and that decomposition should be iterative and driven by verification feedback rather than performed once. It claims AFEV, a pipeline that extracts one atomic fact at a time, retrieves and reranks evidence for that fact, verifies it with an LLM reasoner, and feeds the label and rationale back into the next extraction step, outperforms both multi-granular supervised systems and LLM reasoning baselines. Across LIAR-PLUS, HOVER, PolitiHop, RAWFC, and LIAR, the reported gains are state-of-the-art on label accuracy, macro-F1, and retrieval F1. A sympathetic reader should care because the method promises to turn unverifiable long claims into a sequence of small, inspectable verification steps with a rationale for each step.","feed_headline":"Verified atomic facts beat whole-claim checks on five datasets","feed_subtitle":"Feeding each verified fact back into the next extraction step beats multi-hop and LLM baselines.","key_machinery":"The load-bearing mechanism is the iterative extraction loop defined in Eq. (2): $F_t = \\text{Extractor}(C, F_{1:t-1}, y_{1:t-1}, r_{1:t-1})$, where $F_t$ is the next atomic fact, $C$ is the original claim, and $y_{1:t-1}, r_{1:t-1}$ are the labels and rationales from earlier verifications. Around this loop the pipeline places a retrieve-then-rerank step that keeps only the top evidence pieces for each atomic fact, a dynamic instance retriever that fetches the most similar training claim as an in-context demonstration, and a reasoner that returns both a label and a rationale. Atomic facts are compact, independently verifiable units, which is what allows evidence and demonstrations to stay small and precise. The loop terminates once the extracted facts cover the claim, so the number of iterations is controlled.","core_discovery":"The central discovery is that iterative atomic fact extraction with feedback reduces error accumulation in complex claim verification. Instead of decomposing a claim once and then retrieving evidence for all fragments at once, the extractor generates the next atomic fact conditioned on the original claim, the previously extracted facts, and their verification labels and rationales. The reasoner simultaneously produces a factuality label and an evidence-grounded rationale per atomic fact; the rationale can introduce implicit information, such as an entity named only in earlier evidence, that sharpens the next decomposition. The paper reports that this feedback loop, together with an evidence reranker and dynamically retrieved demonstrations, achieves state-of-the-art results on all five datasets, including 83.73 label accuracy on LIAR-PLUS, 78.87 on HOVER, and 74.14 on PolitiHop, and 60.2 F1 on RAWFC and 43.9 F1 on LIAR.","pith_inferences":["An untested extension would measure decomposition fidelity directly by checking whether atomic facts plus rationales reconstruct the original claim; the paper reports no such metric.","The feedback loop's safety depends on rationale quality, so a controlled corruption test of rationales would reveal whether the loop reduces errors or silently propagates hallucinated entities.","Because the extractor and reasoner are untrained LLM calls, swapping in a different base model could test whether the iterative structure or the particular model carries the gains.","The same iterative decomposition could transfer to long-form question answering or claim generation, where verifying one component should inform the next; the paper does not test these settings."],"forward_implications":["Complex claims can be verified as a sequence of smaller, independently checkable facts rather than as one monolithic judgment.","Feeding verification labels and rationales back into extraction reduces redundant or broken atomic facts, which the paper identifies as the mechanism behind the accuracy gains.","Reranking evidence per atomic fact means each sub-claim needs only one or two precise evidence pieces, lowering noise compared with retrieving for the whole claim.","The pipeline outputs a label and rationale for every atomic fact, so the final verdict can be traced back to individual evidence-grounded steps.","On the five benchmarks tested, the reported numbers place the method ahead of both multi-granular supervised baselines and the strongest LLM prompting baselines."],"supporting_citations":[{"why":"It supplies the CURE multi-granular baseline and the evidence-retrieval evaluation protocol the paper compares against.","marker":"[34]"},{"why":"It supplies VMASK, the strongest multi-granular baseline that AFEV must beat on LIAR-PLUS, HOVER, and PolitiHop.","marker":"[40]"},{"why":"It supplies RAFTS, the strongest LLM baseline that AFEV must beat on RAWFC and LIAR.","marker":"[21]"},{"why":"It defines the HoVer dataset used for the main accuracy and ablation experiments.","marker":"[26]"},{"why":"It supplies the LIAR-PLUS benchmark with six-label veracity annotations.","marker":"[29]"},{"why":"It supplies the LIAR benchmark that tests the pipeline without annotated evidence.","marker":"[28]"},{"why":"It supplies the RAWFC benchmark with weakly structured evidence for LLM-baseline comparisons.","marker":"[27]"},{"why":"It supplies the PolitiHop political multi-hop benchmark and its fine-grained claim structure.","marker":"[3]"},{"why":"It defines RAC, the prior atomic-fact decomposition method whose limitations motivate the iterative feedback design.","marker":"[9]"},{"why":"It motivates combining retrieval, rethinking, and revision inside the verification loop.","marker":"[10]"}],"fun_headline_variants":["Iterative atomic fact feedback cuts errors in claim verification","AFEV: feedback-driven atomic facts beat whole-claim baselines","Atomic fact loops achieve state-of-the-art on five datasets","Feeding verified facts back improves complex claim checks","Decompose, verify, feed back: AFEV tops five benchmarks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The feedback loop assumes the labels and rationales generated for earlier atomic facts are reliable enough to condition later extraction; if an earlier rationale contains a hallucinated entity or a wrong relationship, subsequent atomic facts can be extracted from that contaminated information, and errors can accumulate instead of shrinking.","fun_headline_variants_meta":{"raw":{"variants":["Iterative atomic fact feedback cuts errors in claim verification","AFEV: feedback-driven atomic facts beat whole-claim baselines","Atomic fact loops achieve state-of-the-art on five datasets","Feeding verified facts back improves complex claim checks","Decompose, verify, feed back: AFEV tops five benchmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000154,"raw_usage":{"total_tokens":1183,"prompt_tokens":888,"completion_tokens":295,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":212}},"tokens_in":504,"tokens_out":295,"duration_ms":3827,"temperature":1.0,"reasoning_tokens":212,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:33:02.422028+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of complex claims, run AFEV once normally, and run it again after manually replacing one entity in the first rationale with a plausible but false entity before the second extraction step. If final verification accuracy stays the same, Eq. (2)'s feedback is not doing the load-bearing work the paper attributes to it; if accuracy changes, the loop propagates rationale errors. Either result settles whether the feedback loop is the source of the reported gains.","supporting_citations":[{"cited_title":"Consistent Multi-Granular Rationale Extraction for Explainable Multi-hop Fact Verification","cited_arxiv_id":"2305.09400","evidence_quote":"It supplies the CURE multi-granular baseline and the evidence-retrieval evaluation protocol the paper compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies VMASK, the strongest multi-granular baseline that AFEV must beat on LIAR-PLUS, HOVER, and PolitiHop."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies RAFTS, the strongest LLM baseline that AFEV must beat on RAWFC and LIAR."},{"cited_title":"Jiang, S","cited_arxiv_id":null,"evidence_quote":"It defines the HoVer dataset used for the main accuracy and ablation experiments."},{"cited_title":"Alhindi, S","cited_arxiv_id":null,"evidence_quote":"It supplies the LIAR-PLUS benchmark with six-label veracity annotations."},{"cited_title":"liar, liar pants on fire","cited_arxiv_id":null,"evidence_quote":"It supplies the LIAR benchmark that tests the pipeline without annotated evidence."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the RAWFC benchmark with weakly structured evidence for LLM-baseline comparisons."},{"cited_title":"Ostrowski, A","cited_arxiv_id":null,"evidence_quote":"It supplies the PolitiHop political multi-hop benchmark and its fine-grained claim structure."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It motivates combining retrieval, rethinking, and revision inside the verification loop."}],"review_version":1}