{"id":"55ae419c-3732-417c-a815-03211ce4f6c2","arxiv_id":"1908.05267","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"FEVER fact-verification models rely heavily on claim wording bias, drop sharply on a new symmetric test set, and improve modestly when training is reweighted to flatten n-gram-label correlations.","lead":"Fact-checking models trained on the FEVER dataset can score well by exploiting wording cues in the claim instead of reading the evidence. The authors build a symmetric test set that removes those cues and introduce a reweighting method that partly restores evidence-based reasoning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The symmetric test set's central property—claim-only accuracy at chance—is asserted but never empirically checked; a claim-only BERT baseline on it would settle whether the reported drop reflects bias removal or distribution shift.","rationale":"The paper's qualitative discovery is credible: a claim-only BERT at 61.7% versus 33.3% majority, the LMI table showing negation bigrams predicting REFUTES, and the random-embedding InferSent control all independently support the existence of claim-only cues in FEVER. My concern targets the second pillar, the symmetric test set, which is the load-bearing instrument for the quantitative claims: the large drop on the symmetric set and the reweighting gains of 3.4/3.3 points. The reader's weakest_assumption identifies the same pillar. I sharpen it: even granting perfect fluency and labels, the construction's guarantee is only marginal n-gram independence. The paper asserts a stronger property—claim-only prediction equals random guessing—without testing it. This is not a nitpick; it is the key control. If a claim-only BERT scores, say, 58% on the symmetric set, then the set still contains exploitable cues, and the \"unbiased\" evaluation is only partially unbiased. The reweighting method has a separate circularity (Eq. 3 optimizes for the same n-gram independence the test set encodes), but the claim-only baseline is the more fundamental check. If that check passes, the conditional verdict is appropriate; if it fails, the quantitative conclusions need substantial revision, though the existence of bias in FEVER would remain. Therefore I do not change the reader's verdict.","tokens_in":8734,"tokens_out":8710,"duration_ms":92189,"concrete_test":"Using the same claim-only BERT setup as Section 2 (same architecture, same fine-tuning protocol on FEVER training claims), evaluate it on the claims extracted from the 956-pair symmetric test set, with evidence omitted. Compute accuracy against the pair labels and test whether it is statistically distinguishable from 50% (exact binomial test or bootstrap confidence interval). If the interval excludes 50% (or accuracy exceeds roughly 55%), the symmetric set is not unbiased for this model class and the performance drop in Table 3 cannot be attributed to bias removal. If it is within noise of 50%, the instrument is validated for the claim-only baseline and the conditional verdict can stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3 asserts that in the symmetric test set \"determining veracity with the claim alone in this setting would be equivalent to a random guess.\" The construction guarantees only marginal independence between individual claim n-grams and labels: each original or generated sentence appears once with label L and once with the inverse label, so p(l|w)=0.5 for any n-gram contained in a single sentence. That is not sufficient for a trained neural classifier, which can exploit longer-range lexical patterns, syntax, sentence length, punctuation, or stylistic regularities of the manually authored generated claims. The paper never runs a claim-only model on the symmetric set, so the central drop (NSMN 81.8% to 58.7%, Table 3) is not shown to be caused by bias removal; it could be a distribution-shift artifact. The selection of source pairs from NSMN-correct cases compounds this: the FEVER DEV column is a different, easier distribution than the symmetric claims. Partial annotation of 285 pairs checks label agreement, not the stronger property that no claim-only model can beat chance. A claim-only BERT baseline on the symmetric set is the missing control that would validate the instrument.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies annotation bias in the FEVER fact-verification dataset. The authors train a claim-only BERT classifier and obtain 61.7% accuracy versus a 33.3% majority baseline, and use LMI-based n-gram analysis to show that REFUTES-associated bigrams such as 'did not' and 'refused to' are stable across train and dev. They construct a symmetric test set by manually generating counterfactual claim-evidence pairs for 239 original pairs, and report that evidence-aware models drop from roughly 81-86% on FEVER dev to 56-59% on the new set. They then propose an instance-reweighting regularizer (Eq. 3) that flattens n-gram-label correlations, and report absolute gains of 3.4 and 3.3 points for ESIM and BERT on the symmetric set. The paper concludes that FEVER models exploit claim-side cues and that the proposed reweighting improves evidence-based verification.","tokens_in":8899,"tokens_out":7999,"duration_ms":80256,"significance":"The bias diagnosis is a solid and valuable contribution: the claim-only BERT baseline, the LMI analysis, and the consistent p(l|w) patterns in train and dev are reproducible evidence of annotation artifacts. The proposed symmetric test set and the model-agnostic reweighting objective are useful ideas and could become community resources. The paper is also clearly written and ships code and data via a public repository. However, the experimental support for the central claims is incomplete: the symmetric test set's validity is asserted rather than verified, the source pairs are selected using the very model whose bias is being studied, and the reweighting gains are measured on a set constructed from the same n-gram marginal criterion that the objective minimizes. These issues are fixable, but they are load-bearing for the paper's evaluation claims.","major_comments":[{"comment":"The central claim that the symmetric test set makes claim-only prediction equivalent to random guessing is asserted but never tested. The construction guarantees only marginal independence: for any n-gram appearing in the data, p(label|n-gram)=0.5 by construction. A trained neural classifier can exploit longer-range lexical patterns, syntax, sentence length, punctuation, and stylistic regularities introduced by the manual generation process, none of which are controlled by the construction. The paper reports no claim-only BERT or InferSent accuracy on the symmetric set, so the drop from 81.8% to 58.7% for NSMN in Table 3 is not shown to be caused by bias removal rather than distribution shift. Please add this control: a claim-only model should be at or near chance on the symmetric set; if it is not, the performance drop cannot be attributed to elimination of claim-side cues.","section":"Section 3, Table 3"},{"comment":"The source pairs for the symmetric set were 'randomly picked from the cases which NSMN correctly predicts.' This introduces selection bias: the symmetric set is generated from an easier, model-agreeing subset of FEVER, while the FEVER DEV column in Table 3 reports accuracy on the full dev set. The 81.8% versus 58.7% comparison therefore conflates distribution shift with removal of claim-side bias. Please report accuracy on the matched 239 original source pairs, and either draw source pairs independently of model correctness or demonstrate that the selected subset is representative of the dev distribution.","section":"Section 5, footnote 7, Table 3"},{"comment":"The reweighting objective minimizes exactly the per-n-gram label bias that the symmetric test set is designed to neutralize, and Table 4 confirms that the objective is satisfied (reweighted p(l|w) around 0.33-0.35 for the top bigrams). The 3.3- and 3.4-point gains in Table 3 therefore partly measure how well the method satisfies its own objective rather than a general improvement in evidence-based verification. To support the external claim, please validate on an unbiased set constructed independently of the n-gram marginal criterion, and compare with simpler debiasing baselines such as removing or downweighting high-LMI n-grams, or training on n-gram-balanced subsets.","section":"Section 4, Eq. (3), Table 4"},{"comment":"All reported accuracies are single runs on 956 pairs with no confidence intervals or significance tests. The absolute gains of 3.3 and 3.4 points correspond to roughly 32 examples in a binary split, which could be within run-to-run noise for neural models. Please report means and standard deviations over multiple random seeds, and use a paired test such as McNemar's for the BASE versus R.W comparisons.","section":"Section 5, Table 3"},{"comment":"Only 285 of 956 pairs (30%) were annotated, with 94% agreement and 2% grammatical errors. This checks label agreement, but not the stronger property that every generated claim is factually coherent and that every cross pair carries the inverse label. Since the entire argument depends on the correctness of the unannotated pairs, please either release full annotations, enlarge the annotated sample, or at minimum report annotation results separately for the generated and cross pairs, and examine whether annotation errors correlate with model errors.","section":"Section 5, Symmetric Test Set"}],"minor_comments":[{"comment":"The sentence 'This new test set completely eliminates the ability of models to rely on cues from claims' overstates what is shown; the construction eliminates the specific n-gram-level label correlations identified in Section 2, not all possible claim-only cues. Please soften the wording to match what is proven.","section":"Section 3"},{"comment":"The column heading 'GENERATED' is ambiguous: the text sometimes refers to the full SYMMETRIC TEST SET (956 pairs) and sometimes to the generated pairs only. Please define exactly which subset is scored in each column and give the number of examples.","section":"Table 3"},{"comment":"The optimization procedure for Eq. (3) is not described; please state how the objective is solved, including the optimizer, the range of the regularization coefficient lambda, and any convergence criteria, so that the preprocessing step is reproducible.","section":"Section 4, Eq. (3)"},{"comment":"The indicator I[w_j^(i)] is used for n-gram presence, but if a claim contains w_j multiple times the counting convention is ambiguous; please clarify whether this is a binary presence indicator or an occurrence count.","section":"Section 4, Eq. (2)"},{"comment":"The description of NSMN as 'the leading evidence-aware system in the FEVER Shared Task' should be qualified with the year and, if possible, a citation to the shared task report.","section":"Section 1"}],"recommendation":"major_revision","confidential_remarks":"The bias diagnosis in Section 2 is solid and should be preserved; the main risk is that the symmetric test set has not been validated as an unbiased instrument. The missing claim-only control on the symmetric set is the critical experiment, and I would like to see it in the revision before the evaluation claims are accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline finding holds up: FEVER has claim-only artifacts, and the paper shows it clearly with a claim-only BERT at 61.7% versus 33.3% majority, plus LMI bigram correlations that transfer from train to dev. That part is solid and worth citing.\n\nThe genuinely new pieces are the symmetric test set and the reweighting regularizer. The idea of balancing every n-gram across labels is clever, and the manual construction is a real effort. Releasing the data and code helps reproducibility.\n\nThe soft spots are mostly about the evaluation instrument. The paper asserts that claim-only prediction on the symmetric set is at chance, but never runs a claim-only model on it. The construction balances individual n-gram marginals, not the longer-range lexical or syntactic patterns a neural classifier can use. So the drop from 81.8 to 58.7 for NSMN is consistent with bias removal, but it could also partly be distribution shift from the manually generated claims. A claim-only BERT baseline on the symmetric set would settle this and should have been in the paper. Also, the original pairs were chosen from NSMN-correct cases, which makes the comparison to FEVER dev not apples-to-apples.\n\nThe reweighting results are modest and reported as single runs without confidence intervals. The regularizer and the test set encode the same n-gram independence criterion, so part of the 3.3-3.4 point gain is the model learning to satisfy the objective's own measure. That's not fatal, but it tempers the claim of general debiasing.\n\nOn balance, this is a solid empirical contribution with a real methodological gap. The bias diagnosis is reproducible and the test set, once validated with the missing control, would be a reusable resource. It deserves serious peer review, with the request for a claim-only baseline on the symmetric set and error bars.","headline":"FEVER bias finding is real and worth citing, but the symmetric test set needs a claim-only baseline before its measured drop can be read as proof of unbiasedness.","tokens_in":9529,"tokens_out":3442,"would_cite":true,"duration_ms":31448,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FEVER claims leak their labels, and a symmetric test set proves it.","keywords":["fact verification","dataset bias","annotation artifacts","FEVER","symmetric test set","reweighting regularization","claim-only classification","natural language inference"],"falsifier":"Re-annotate all 956 symmetric pairs with fresh annotators: if agreement falls well below 94%, or if crossing original and generated sentences fails to invert the label in many cases, the symmetry property is broken. A complementary check is to train a claim-only classifier on reweighted data and see whether it still beats 50% on the symmetric set; if it does, unmeasured cues remain.","tokens_in":8468,"feed_emoji":"⚖️","tokens_out":12285,"duration_ms":107497,"temperature":0.7,"pith_summary":"This paper argues that the FEVER fact-verification benchmark is easier than it looks: crowdworker editing habits left giveaway phrasing in claims, so a classifier that reads only the claim reaches 61.7% accuracy against a 33.3% majority baseline. The giveaway is concentrated in negation bigrams like “did not,” “failed to,” and “unable to,” which are strongly associated with the REFUTES label. To force models to use the evidence, the authors build a symmetric test set in which every n-gram is equally likely under either label; on it, evidence-aware FEVER-trained models fall from roughly 81–86% to 56–59%. They then introduce a model-agnostic reweighting scheme that flattens n-gram–label correlations and recovers 3.3–3.4 absolute points on the symmetric set. If this is right, published FEVER scores overstate how much reasoning the models actually do.","feed_headline":"Claim-only AI hits 61.7% on FEVER fact-checking","feed_subtitle":"A new symmetric test set drops top models below 60%, and reweighting regains ground.","key_machinery":"Two mechanisms carry the argument. The first is the symmetric test set: for each original claim-evidence pair, the authors manually generate a counterpart with the same SUPPORTS or REFUTES label but a contrary fact, and then create two cross pairs by swapping claims with evidence from the other pair. The resulting four combinations make $p(\\mathrm{label}\\mid w)=1/2$ for every n-gram in either input, so claim-only guessing is a coin flip. The second is a reweighting regularization in which Local Mutual Information $\\operatorname{LMI}(w,l)=p(w,l)\\log(p(l\\mid w)/p(l))$ identifies giveaway n-grams, and weights $\\alpha$ are optimized to minimize total n-gram-label bias across the training claims; training loss for instance i is then scaled by $1+\\alpha^{(i)}$, which is architecture-independent.","core_discovery":"The paper’s central claim is that FEVER’s labels are partially predictable from the claim alone because of annotation artifacts, not because of contextual reasoning or even world knowledge. The evidence is threefold: a claim-only BERT beats the majority baseline by a wide margin; high-LMI bigrams in claims, especially negations, predict REFUTES with probabilities around 0.8–0.9 in both training and development; and the same models drop dramatically on a symmetric test set where $p(\\mathrm{label}\\mid w)=1/2$ by construction. The paper further claims that its reweighting regularization—scaling each training example’s loss by a weight that makes giveaway n-grams label-uniform—improves symmetric-set accuracy of ESIM and BERT by 3.4 and 3.3 absolute points, with a small cost on the original biased dev set.","pith_inferences":["Likely consequence: other fact-verification and NLI benchmarks built by similar crowdworker editing may carry the same giveaway-phrase bias, and the paper’s symmetric pairing is a cheap diagnostic for detecting it before expensive training.","The reweighting flattens only n-gram-level cues; semantic or world-knowledge shortcuts that do not reduce to surface bigrams would survive it, so combining reweighting with counterfactual data augmentation or adversarial filtering is a natural follow-up.","Because only 30% of the symmetric pairs were independently validated, the headline drop estimates carry annotation noise; full re-annotation or a larger symmetric set would tighten the numbers."],"forward_implications":["FEVER leaderboard scores on the original benchmark overstate evidence-based reasoning; the symmetric set offers a cleaner lower-bound test and should be added alongside the retrieval-based evaluation.","The reweighting method is independent of model architecture, so it can be dropped into any classification objective where training data carry annotation artifacts.","Debiased training trades a small amount of accuracy on the biased dev set for robustness on the unbiased set, so gains should be reported on both.","A model that performs well on the symmetric set is using the evidence rather than the surface form of the claim, which is what fact verification is supposed to measure."],"supporting_citations":[{"why":"Defines FEVER and provides the biased benchmark dataset that the paper analyzes and re-evaluates.","marker":"Thorne et al., 2018"},{"why":"Provides the NSMN evidence-aware classifier whose drop from 81.8% to 58.7% on the symmetric set anchors the bias claim.","marker":"Nie et al. (2019)"},{"why":"Supplies BERT, used both for the claim-only probe (61.7%) and as an evidence-aware classifier in the reweighting experiments.","marker":"Devlin et al., 2019"},{"why":"Supplies the ESIM architecture that serves as the second evidence-aware classifier and reweighting testbed.","marker":"Chen et al., 2017"},{"why":"Establishes annotation artifacts in NLI datasets, motivating the analysis of similar artifacts in FEVER.","marker":"Gururangan et al., 2018"},{"why":"Introduces hypothesis-only baselines in NLI, the direct precedent for claim-only evaluation.","marker":"Poliak et al., 2018"},{"why":"Defines Local Mutual Information, the measure the paper uses to identify giveaway n-grams.","marker":"Evert, 2005"},{"why":"Provides the instance-reweighting approach to label bias that the paper adapts into its regularization scheme.","marker":"Jiang and Nachum (2019)"}],"fun_headline_variants":["Fact-check models lean on claim cues, not evidence","Claim-only bias exposed in FEVER; reweighting fixes it","New test set reveals claim-only shortcuts in fact-checking","Reweighting removes claim bias from fact verification models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The symmetric test set is a valid unbiased instrument, which requires that every generated and crossed claim-evidence pair is fluent, factual, correctly labeled, and genuinely inverse when crossed; the authors checked only 30% of the 956 pairs (94% agreement, 2% grammar issues), and the source pairs were chosen only from cases NSMN already predicted correctly.","fun_headline_variants_meta":{"raw":{"variants":["Fact-check models lean on claim cues, not evidence","Claim-only bias exposed in FEVER; reweighting fixes it","New test set reveals claim-only shortcuts in fact-checking","Reweighting removes claim bias from fact verification models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000581,"raw_usage":{"total_tokens":2682,"prompt_tokens":835,"completion_tokens":1847,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":451,"completion_tokens_details":{"reasoning_tokens":1789}},"tokens_in":451,"tokens_out":1847,"duration_ms":12779,"temperature":1.0,"reasoning_tokens":1789,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:19:54.025280+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate all 956 symmetric pairs with fresh annotators: if agreement falls well below 94%, or if crossing original and generated sentences fails to invert the label in many cases, the symmetry property is broken. A complementary check is to train a claim-only classifier on reweighted data and see whether it still beats 50% on the symmetric set; if it does, unmeasured cues remain.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines FEVER and provides the biased benchmark dataset that the paper analyzes and re-evaluates."},{"cited_title":"Combining Fact Extraction and Verification with Neural Semantic Matching Networks","cited_arxiv_id":"1811.07039","evidence_quote":"Provides the NSMN evidence-aware classifier whose drop from 81.8% to 58.7% on the symmetric set anchors the bias claim."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ESIM architecture that serves as the second evidence-aware classifier and reweighting testbed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Local Mutual Information, the measure the paper uses to identify giveaway n-grams."}],"review_version":1}