{"id":"1bdd5fd3-6f7f-426e-816a-802d233794d6","arxiv_id":"2505.06324","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A zero-shot textual entailment prompt ('Does the REFERENCE entail the CLAIM?') edges out prior baselines on AttributionBench (73.8 ID, 83.43 OOD with flan-ul2), while per-layer attention probes reduce mostly to trivial all-positive classifiers.","lead":"This paper tests whether asking a language model a simple yes/no question, 'does the reference support the claim,' makes citation checking more reliable, reporting small gains on a public benchmark. It also probes attention layers inside a small model for attribution signals, with results that mostly collapse to trivial always-positive classifiers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline F1 gains are not established: Table 1's ID average contradicts its own per-dataset F1s, and the 0.2-point ID delta is unreplicated and within plausible noise.","rationale":"The reader's CONDITIONAL verdict already captures the main exposure: the zero-shot deltas are single-run comparisons against published numbers, and the attention experiment lacks a held-out split. My pass adds that the headline improvement is internally inconsistent (ID-Avg 73.8 vs the macro mean 74.9; OOD 2.4% vs the 2.62% over the published best baseline), which strengthens the case for requiring code and raw scores before trusting the abstract. The attention claim is even weaker: in Table 3, F1=66.67 with FP=100 and FN=0 on balanced 84/84 data is exactly the trivial all-positive classifier, so Section 4.3 provides no evidence that attention helps attribution. These are not reasons to dismiss the possibility that the cheap textual-entailment prompt works—the OOD gain is large—but they are reasons to require the audit described above. Since the reader already asked for code, variance, and correction of numeric inconsistencies, the appropriate disposition remains conditional, so I leave the verdict unchanged.","tokens_in":7167,"tokens_out":11457,"duration_ms":111106,"concrete_test":"Obtain the authors' evaluation script and raw per-instance predictions for the Table 1 flan-ul2 run; recompute the ID macro-average from the four per-dataset F1 values and rerun the identical prompt on the published FLAN-UL2 zero-shot baseline across at least 5 seeds. If the script does not reproduce 73.8, or if the 0.2-point ID F1 gap is within the seed-to-seed standard deviation of the baseline, the claimed improvement is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—a 0.27% ID and 2.4% OOD improvement over the best AttributionBench baseline—is not supported by the numbers as printed. In Table 1, the flan-ul2 per-dataset F1s are 55, 75.2, 84.16, and 85.38; taking the simple mean, which is the formula used for every other ID-Avg row in that table (e.g., the FLAN-UL2 baseline 59.4/72.5/82.5/80.1 gives 73.6), yields 74.9, not the reported 73.8. The abstract's 0.27% ID gain is computed from the 73.8 value, so the headline number does not match the underlying data. In Table 2, the best published OOD baseline is FLAN-UL2 at 81.3 and the method is 83.43, a 2.62% relative gain; the abstract's 2.4% instead matches a comparison to the authors' own gpt4-o row (81.48). All F1 values come from single runs with no seeds, confidence intervals, or significance tests, so the 0.2-point ID gap could be sampling noise on roughly 1,600 binary examples. Because the central quantitative claim is thus internally inconsistent and statistically unreplicated, it is currently unverified.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper addresses automatic attribution evaluation, formulated as the binary task of deciding whether a claim is supported by a reference document, as instantiated by the AttributionBench benchmark. The authors propose two techniques. First, a zero-shot prompting method that recasts attribution as textual entailment (\"Does the REFERENCE entail the CLAIM?\"), evaluated with gpt4-o and flan-ul2 on the ID and OOD subsets of AttributionBench; the abstract claims a 0.27% ID and a 2.4% OOD relative improvement over the best baselines. Second, an attention-based analysis in which the attention weights of each layer of flan-t5-small are fed to a fully connected binary classifier, with per-layer F1 reported on the LFQA subset. The paper concludes that the entailment prompt outperforms existing zero-shot baselines and that attention layers, except layer 4 and layers 8 through 11, improve attribution.","tokens_in":7248,"tokens_out":10075,"duration_ms":97340,"significance":"If the reported results were statistically robust, the paper would offer a genuinely simple and reproducible zero-shot baseline for attribution: the prompt template is minimal, the evaluation uses the fixed public AttributionBench labels, and the method would be cheap to deploy. The attention analysis, however, is explicitly preliminary. The main limitation on significance is the size of the reported effects: a 0.2-point ID F1 gain on roughly 1,610 examples and OOD gains on subsets of 162 to 1,013 examples are within the range of typical evaluation noise, and all numbers come from single unseeded runs. Section 6 is candid about computational constraints, but the claims in the abstract outrun the statistical evidence in Sections 4.1 and 4.2. The contributions are potentially useful as an empirical note, not as an established improvement over the state of the art.","major_comments":[{"comment":"The reported ID average for the proposed flan-ul2 row is arithmetically inconsistent with the row's own per-dataset F1 values: (55 + 75.2 + 84.16 + 85.38) / 4 = 74.9, not the reported 73.8. Because the abstract's headline \"0.27% improvement\" is the relative difference between 73.8 and the FLAN-UL2 baseline of 73.6, the central ID claim rests on an internally inconsistent number. The average must be recomputed consistently, and either the per-dataset values or the reported average corrected.","section":"Table 1 and §4.2"},{"comment":"The claimed 2.4% OOD improvement is not stated against the best baseline. The best published OOD zero-shot baseline in Table 2 is FLAN-UL2 at 81.3, against which 83.43 is a 2.62% relative gain; the value 2.4% corresponds instead to comparing 83.43 with the authors' own gpt4-o row (81.48). The relative gain should be computed with respect to an explicitly named baseline, and absolute F1 deltas should be reported alongside relative percentages.","section":"Abstract, Table 2, §4.2"},{"comment":"All F1, FP, and FN values come from single runs with no seeds, repeats, confidence intervals, or significance tests. The ID gain is 0.2 F1 points over roughly 1,610 binary examples, and the OOD gains are computed on subsets of 162 to 1,013 examples; at these sample sizes the reported deltas are within the noise of the evaluation. Repeated runs with variance estimates and, where applicable, paired significance tests are needed before the improvements claimed in the abstract can be considered established; otherwise the improvement claims should be withdrawn or substantially tempered.","section":"§4.1 and §4.2"},{"comment":"The attention experiment does not support its stated conclusion. First, no train/test split is described for the fully connected classifiers, and the caption indicates that the evaluation uses the 168 LFQA examples (84 per class); if the classifiers were trained on the same 168 instances, the results are circular. Second, the layers with the highest F1 (layers 1-3, 5-7, 11, 12) report FP near 100 and FN near 0, meaning they assign every instance to the positive class; on balanced classes such a constant-positive predictor attains F1 of about 66.7, which is exactly the reported range of values, so no attention-based signal beyond the class prior is demonstrated. Third, the text's claim of lower false positives than the zero-shot baseline is contradicted by the table: the baseline has FP = 17.85 while most attention rows have FP = 100. Fourth, the abstract and §4.3 state that performance is worse at \"layer 4 and layers 8 through 11\", but layer 11 has F1 = 66.67, above the baseline F1 = 20; the underperforming layers are 4, 8, 9, and 10. The attention claim should be re-analyzed or removed.","section":"§3.2 and Table 3"}],"minor_comments":[{"comment":"The tables contain garbled numeric entries (for example, Table 1's gpt4-o row shows \"64.71421.2\" and Table 2 shows \"88.2417.28 7.4\"); the tables should be re-typeset and every value checked against the raw model output.","section":"Tables 1 and 2"},{"comment":"The text states that the method \"outperforms the baselines in both ID and OOD sets,\" but in Table 1 the proposed flan-ul2 row is below the FLAN-UL2 baseline on ExpertQA (55 vs. 59.4); the claim should be restricted to the aggregate averages and to the specific datasets where it holds.","section":"§4.2"},{"comment":"The model citations are imprecise: flan-ul2 should be cited to the UL2 (Tay et al., 2022) and FLAN (Chung et al., 2022) papers rather than to Raffel et al. (2020), and the gpt4-o model is not the GPT-4 model of Achiam et al. (2023); the appropriate model documentation should be cited.","section":"References"},{"comment":"The description does not specify how attention weights are aggregated (over heads, tokens, or positions) before being fed to the fully connected layer; the preprocessing should be stated explicitly.","section":"§3.2"},{"comment":"The FP and FN columns are described in Section 4.1 as rates, but Table 3 reports them as percentages; the units and denominators should be clarified.","section":"§4.1 and Table 3"},{"comment":"The prompt template asks the model to \"Answer the question,\" but the template contains no question, only a REFERENCE and a CLAIM; the wording should be aligned with the actual prompt.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The two advertised contributions are both empirical and both currently outrun their evidence. The zero-shot entailment idea is reasonable, but the headline gain is a 0.2-point ID delta that is both internally inconsistent with Table 1's own arithmetic and statistically unreplicated; the attention analysis demonstrates only that a constant-positive classifier beats a weak zero-shot baseline on one subset. I would give the authors the opportunity to fix the arithmetic, add repeated runs with variance and significance testing, and re-analyze or cut the attention section. If the corrected and replicated numbers no longer beat the published baselines, the paper should not proceed. The manuscript is also on the short side for a full-length empirical paper; a workshop-format submission may be a better fit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read on 2505.06324. The idea is clean: frame attribution as textual entailment with the prompt \"Does the REFERENCE entail the CLAIM?\" and test it on AttributionBench. That is a legitimate, cheap thing to measure, and the authors are explicit that they follow the same zero-shot setup as the benchmark, which makes cross-paper comparison meaningful in principle. The related work is on point, and the paper is easy to read.\n\nThe problems are in the numbers. Table 1 lists flan-ul2 per-dataset F1s of 55, 75.2, 84.16, and 85.38, but the reported ID average is 73.8. The simple mean is 74.9. That's an internal inconsistency in the headline result. Fixing it makes the claimed ID gain bigger (about 1.3 points over the published baseline), but there are still no error bars, seeds, or significance tests, so the gain is unreplicated and within plausible noise. The abstract's 2.4% OOD figure is also mis-sourced: it matches a comparison to the authors' own gpt4-o row (81.48) rather than the best published baseline (81.3). That's a minor issue, but it matters for honesty of reporting.\n\nThe attention section is the weakest part. On the balanced 84/84 LFQA subset, any classifier that predicts \"yes\" for everything gets F1=66.67. Many rows in Table 3 have FP=100, FN=0, which is exactly that degenerate solution. The paper presents these as \"outperforming the baseline\" when the zero-shot F1 is only 20, but 66.67 is not evidence that attention helps; it is evidence that the probe collapsed. No train/test split is described for the per-layer linear classifiers, so overfitting is a real risk. The Limitations section concedes compute constraints, which explains the small scale but not the arithmetic error or the missing experimental controls.\n\nSo: the prompt variant is worth a quick replication by anyone already running AttributionBench, but as a paper the central claim is not established. I would not cite it as evidence without running my own control first. The right editorial decision is to desk reject in the current form, while inviting a resubmission that fixes the table arithmetic, reports variance, and replaces the attention analysis with a non-degenerate probe (majority baseline, held-out split). The underlying question is real; the evidence here is not.","headline":"Simple entailment prompt for attribution is worth a quick test, but the paper's own tables don't support the claimed improvements, and the attention analysis reduces to an always-positive classifier.","tokens_in":8002,"tokens_out":4440,"would_cite":false,"duration_ms":48083,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Framing attribution as a textual entailment question lets a zero-shot LLM match or beat prior attribution benchmarks without fine-tuning.","keywords":["attribution","textual entailment","zero-shot prompting","large language models","citation verification","AttributionBench","FLAN-UL2","attention layers"],"falsifier":"Rerun the AttributionBench zero-shot FLAN-UL2 baseline and the paper's entailment prompt under the same decoding settings and metric implementation; if the baseline reproduces at 73.8 or above, or if repeated runs of the entailment prompt vary by more than 0.2 F1 points, the claimed in-distribution improvement is not demonstrated.","tokens_in":6770,"feed_emoji":"📄","tokens_out":8788,"duration_ms":82379,"temperature":0.7,"pith_summary":"This paper claims that LLM attribution—deciding whether a generated claim is supported by a cited reference—can be handled by a simple zero-shot textual entailment prompt. Using the prompt 'Does the REFERENCE entail the CLAIM?' with the FLAN-UL2 model, it reports the best average $F_1$ on both the in-distribution (73.8) and out-of-distribution (83.43) sets of AttributionBench, edging out the strongest published baselines (73.6 and 81.3). The paper also reports a preliminary attention-based classifier using flan-t5-small on one subset, where attention features from most layers beat the zero-shot baseline $F_1$. The intended upshot is that expensive fine-tuning and specialized training are not necessary for basic citation-claim verification, and that attention layers carry usable attribution signal.","feed_headline":"A yes/no entailment prompt improves LLM citation attribution","feed_subtitle":"Zero-shot FLAN-UL2 tops the best published F1 averages on AttributionBench without any fine-tuning.","key_machinery":"The central object is the textual entailment relation $\\mathrm{Entail}(S_1, S_2)$, implemented as a zero-shot prompt: 'Answer the question with ONLY a YES or NO. Does the REFERENCE entail the CLAIM?' The reference text is the premise, the claim is the hypothesis, and the model's yes/no decision is the attribution label. This collapses attribution into a judgment the LLM can make from its pretrained language understanding, without task-specific fine-tuning. For the attention analysis, the machinery is a per-layer binary classifier: attention weights from each of flan-t5-small's 12 layers feed a fully connected layer that predicts the attribution class, isolating where attribution signal lives.","core_discovery":"The paper's central claim is that the binary attribution task defined in AttributionBench—whether a claim is attributable to a given reference—is adequately solved by reframing it as textual entailment. With the reference as premise and the claim as hypothesis, an instruction-tuned LLM asked a yes/no entailment question yields $F_1 = 73.8$ on the in-distribution average and $F_1 = 83.43$ on the out-of-distribution average, compared with the best baseline values of 73.6 and 81.3. On individual subsets, the method sets the best reported scores on Stanford-GenSearch and LFQA (ID) and on AttrScore-GenSearch and HAGRID (OOD). For the attention half, the paper claims that with flan-t5-small on the LFQA subset, linear classifiers over layer-wise attention weights beat the zero-shot baseline $F_1$ at most layers, with exceptions at layer 4 and layers 8 through 11.","pith_inferences":["The in-distribution gain (0.2 $F_1$ points) is smaller than typical run-to-run variation on such benchmarks, so the stronger out-of-distribution gain (2.1 points) is the more informative result, but it rests on comparing with published numbers rather than a re-run baseline.","The layer-wise attention results should be read cautiously: they come from one 168-instance subset with balanced classes but extreme false-positive and false-negative values at several layers, so they show feasibility, not a stable mechanism.","A natural testable extension would be to use the same entailment prompt on a per-sentence citation task and check whether yes/no judgments align with human citation-quality judgments, or to combine attention features from multiple layers to see whether the dip at layers 8 through 11 disappears."],"forward_implications":["Attribution can be treated as a textual entailment problem, so any instruction-tuned LLM capable of yes/no entailment judgments can be used for citation verification without training data.","The reported out-of-distribution average (83.43) suggests the entailment formulation transfers across datasets with different claim-and-reference styles better than the previous zero-shot baselines.","Because the attention classifier performs above baseline on most layers of flan-t5-small, attribution signal appears distributed across the model rather than concentrated in a single layer, which could inform where to probe or intervene.","Simple prompt templates are enough to improve on established benchmarks; the gains do not require specialized attribution modules or retrieval changes."],"supporting_citations":[{"why":"Defines AttributionBench, the binary attribution task, the evaluation metrics, and the zero-shot baselines the paper compares against.","marker":"Li et al. (2024)"},{"why":"Introduces the T5 text-to-text architecture underlying the FLAN models used in the zero-shot and attention experiments.","marker":"Raffel et al. (2020)"},{"why":"Identifies the GPT-4 model family used for the gpt-4o zero-shot comparison system.","marker":"Achiam et al. (2023)"}],"fun_headline_variants":["Entailment reframing tops baselines for LLM citation attribution","Attention weights improve attribution in small LLMs","Zero-shot entailment outperforms baselines on AttributionBench","Entailment prompt boosts citation attribution for LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the published baseline F1 values were computed under exactly the same evaluation conditions as the paper's own runs, so the reported gains—especially the 0.2-point in-distribution difference—reflect the method rather than protocol or sampling noise.","fun_headline_variants_meta":{"raw":{"variants":["Entailment reframing tops baselines for LLM citation attribution","Attention weights improve attribution in small LLMs","Zero-shot entailment outperforms baselines on AttributionBench","Entailment prompt boosts citation attribution for LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001559,"raw_usage":{"total_tokens":6230,"prompt_tokens":950,"completion_tokens":5280,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":5214}},"tokens_in":566,"tokens_out":5280,"duration_ms":41566,"temperature":1.0,"reasoning_tokens":5214,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:58:15.028526+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the AttributionBench zero-shot FLAN-UL2 baseline and the paper's entailment prompt under the same decoding settings and metric implementation; if the baseline reproduces at 73.8 or above, or if repeated runs of the entailment prompt vary by more than 0.2 F1 points, the claimed in-distribution improvement is not demonstrated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the T5 text-to-text architecture underlying the FLAN models used in the zero-shot and attention experiments."}],"review_version":1}