{"id":"9bd045db-a92f-46c3-8033-ca4fd776580e","arxiv_id":"2608.07400","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new SEC-filing QA benchmark shows that retrieval models lose 13 to 20.5 points of ranking accuracy when plausible but wrong disclosures are used as distractors.","lead":"FinRank is a new benchmark of 1,185 finance questions on SEC filings, each paired with the supporting passages and with deliberately confusing \"hard negative\" passages from other firms or periods. The benchmark shows that current retrieval systems, including a 7-billion-parameter embedder, often fail to pick the right evidence, which matters for any finance tool that must cite its sources.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-annotator gold labels mean the 13.0–20.5 pt hardness premium is not yet established as a stable benchmark property; an independent re-annotation/IAA check is required before FinRank's central claim can be accepted.","rationale":"The reader's weakest assumption—gold labels may be wrong—is also the load-bearing point for the benchmark's headline result. The entire contribution reduces to the claim that curated hard negatives are harder than random ones, and that claim is only meaningful if the hard negatives are actually non-supporting. With single-annotator authorship, sampled review, and no IAA, the negative labels are not independently verified. This is not a novelty dispute: the paper is transparent about the curated-pool regime, releases reproducible scripts, a repair log, per-passage hashes, and hn_taxonomy.json, and its metadata-filtered BM25 baseline is honestly reported. Those strengths do not substitute for annotation validity. I considered whether the metadata-filtered BM25 result (which removes 92.9% of hard negatives) undercuts the benchmark's practical difficulty, but the authors explicitly frame the unfiltered pool as the no-metadata regime, so that is a disclosed design choice rather than an internal inconsistency. The concern is therefore about external validity of the labels, not about internal logic. A stratified double-annotation study with reported IAA, plus multi-seed variance reporting for the hard-vs-random contrast, would settle it. Absent that, the appropriate verdict remains CONDITIONAL, unchanged from the reader.","tokens_in":22618,"tokens_out":7088,"duration_ms":76784,"concrete_test":"Re-annotate a stratified sample of 250 records (oversampling 10-Q, quantitative, and Hard records) with two fresh annotators shown only the question and the in-record candidate set L_r, asking them to label each passage SUPPORTING or NON-SUPPORTING; compute Cohen's kappa on passage labels. Then recompute pairwise accuracy and the hard-vs-random gap on the subset where both annotators agree with the released labels, and rerun the random-negative baseline over 5–10 seeds to obtain a confidence interval. If kappa is below about 0.6, or the gap narrows by more than 3–5 points on consensus labels, the hardness premium is not a stable property of FinRank.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that FinRank's curated hard negatives are genuinely harder than random distractors, so the 13.0–20.5 percentage-point drop in pairwise accuracy (Section 7.3, Figure 3B) is evidence that the benchmark isolates evidence discrimination. This inference depends entirely on gold labels: each supporting passage must be truly supporting and each hard negative truly non-supporting for its own question. Those labels are single-annotator. Section 3.5 states records were authored by five business students with only sampled review and no inter-annotator-agreement statistic; Appendix A.10 concedes the methodology 'does not by itself certify the correctness of any individual question, answer, supporting passage, or hard negative.' Because the same annotator wrote the question, chose the positives, and hand-picked the hard negatives, 'hard' and 'non-supporting' are not independently established. Section 3.7 already finds 442 hard negatives byte-identical to a supporting passage of another record, and Appendix C flags 56 label/question-id disagreements plus 65 excluded records including reliance on non-filing sources. If even a small fraction of hard negatives actually answer their own question (e.g., a competitor's nearly identical disclosure for a templated metric), model pairwise accuracy is artificially depressed and the hardness premium inflated, directly undermining the headline contribution. The single seed-42 random-negative sample and the absence of confidence intervals throughout Section 7 compound this: the exact 13.0–20.5 pt gap is not statistically characterized.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FinRank, a benchmark of 1185 question–answer records over SEC 10-K and 10-Q filings from 22 companies. Each record includes a reference answer, gold supporting passages, and human-curated hard negatives drawn from confusable disclosures in other filings, other periods, or other sections. The benchmark defines separate retrieval, reranking, and hard-negative discrimination tasks, and the paper reports baselines spanning TF-IDF, BM25, dense encoders, and a cross-encoder. The headline empirical result is that replacing random negatives with the curated hard negatives lowers pairwise ranking accuracy by 13.0–20.5 percentage points across all evaluated models (Section 7.3, Figure 3B). The release includes normalization scripts, split files, a repair log, and a hard-negative taxonomy, and the paper is explicit that the retrieval corpus is a curated pool of passages rather than full filing text.","tokens_in":22888,"tokens_out":6942,"duration_ms":61918,"significance":"If the gold labels hold up, FinRank fills an actual gap: it is, to my knowledge, the first financial QA benchmark to release per-question curated hard negatives and to score retrieval and reranking directly against them. The stratified metadata (topic, difficulty, reasoning type, evidence scope), the query-rewrite field, the deterministic normalization pipeline, and the reproducible baseline harness are genuine strengths, as is the candid discussion of the corpus definition's scope. The empirical baseline results are useful reference points for the community. However, the central claim that shipped hard negatives 'demonstrably degrade' ranking rests entirely on single-annotator gold labels with no inter-annotator agreement statistic, and the headline hardness premium is reported without confidence intervals. These issues must be addressed before the benchmark's main contribution can be considered established.","major_comments":[{"comment":"The gold labels (supporting passages and hard negatives) are each authored by a single annotator with only sampled author review and no formal inter-annotator agreement statistic, and Appendix A.10 explicitly states that the methodology does not certify the correctness of any individual question, answer, supporting passage, or hard negative. The central claim of Section 7.3 and Figure 3(B) that curated hard negatives are 'genuinely harder' and cause a 13.0–20.5-point pair-wise accuracy drop assumes these labels are correct; if even a small fraction of hard negatives actually support their own question, the drop is inflated, and if gold passages are misidentified, the drop is mismeasured. The manuscript honestly lists this as a limitation, but the abstract and Section 7.3 present the hardness premium as an established result. Please add a stratified double-annotation study with reported agreement on a representative sample, or, failing that, rephrase the headline claim as conditional on single-annotator labels and show that the hardness premium survives on a verified subset.","section":"Sections 3.5, 9, and Appendix A.10"},{"comment":"The hardness premium is reported as a single point estimate per model with no confidence intervals, and the random-negative baseline is a single draw at seed 42 (Section 6.2). The 13.0–20.5-point gaps are large, but without error bars the reader cannot judge whether the relative ordering of models by premium is stable or whether the gap could shrink under resampling. Please report bootstrap confidence intervals over records (or queries) and, ideally, repeat the random-negative sampling with multiple seeds and report the variance of the premium.","section":"Section 7.3 and Figure 3(B)"},{"comment":"The paper reports that 442 hard negatives (about 7.3%) are byte-identical to supporting passages of a different record, and notes that such passages 'may legitimately be relevant to more than one question.' The pairwise-accuracy metric in Section 7.3 nonetheless treats every hard negative as non-supporting for its own question. Because these overlaps could include passages that are actually relevant to the query, the hardness premium should be recomputed on the subset of records after removing or relabeling these overlapping hard negatives (using the released hn_taxonomy.json), or the paper should demonstrate that the premium is unchanged when this subset is excluded.","section":"Section 3.7 and Section 7.3"}],"minor_comments":[{"comment":"The caption cites 'the 6021 hard negatives' while the figure itself reports 'N = 6,052'; please reconcile these counts.","section":"Figure 1 caption and figure body"},{"comment":"Section 4 says 88% of records come from 10-K filings, while Section 9 says 87%; Table 14 gives 1048/1185 = 88.4%, so Section 9 should be corrected.","section":"Section 4 vs. Section 9"},{"comment":"The percentage for the 442 overlapping hard negatives is given as '~7.3%'; please compute this percentage against the same total used in Figure 1, since the caption and the figure currently report different totals.","section":"Section 3.7"},{"comment":"The statement that 'records are averaged with equal weight' is slightly ambiguous; please clarify that pairwise accuracy is first computed within each record and then averaged across records, rather than pooling all pairs globally.","section":"Section 7.3"},{"comment":"The global pool C is said to contain 5230 unique passages, but the average counts of 1.96 supporting passages and 5.08 hard negatives per record imply larger raw totals before deduplication; please state the deduplication effect explicitly for reproducibility.","section":"Section 3.6"}],"recommendation":"major_revision","confidential_remarks":"The paper is transparent about its limitations and the dataset release is well engineered. The main concern is the gap between the confident abstract claim that curated hard negatives 'demonstrably degrade ranking' and the single-annotator label evidence. A double-annotation study with reported agreement is feasible (the authors already list it as future work) and should be required before the headline claim is accepted. The benchmark itself is likely to be useful to the community, and the baseline harness is a strength. The paper's scope is appropriate for an NLP or IR venue, but the central empirical claim needs stronger support."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"FinRank is worth a serious look. As far as I can tell it is the first financial QA benchmark that ships per-question human-curated hard negatives drawn from comparable filings, and it makes retrieval, reranking, and negative discrimination separately measurable. The construction is unusually transparent: a shared criteria document, a deterministic normalization pipeline with a machine-readable repair log, per-passage hashes, a hard-negative taxonomy, and fully reproducible baselines with code. The authors are also honest about what the pooled corpus is—a curated stress test, not full-document retrieval—and they flag the single-annotator labels and the 442 cross-record positive overlaps themselves. That level of care earns real credit.\n\nThe headline result—every model loses 13–20.5 points of pairwise accuracy when random negatives are replaced with curated ones—is the empirical justification for shipping hard negatives as a first-class asset. It is a measurement on released data, so there is no derivation-level circularity; the only self-referential element is that hardness is defined and measured on the curated set itself, which is normal for a benchmark resource. The soft spot is the one the authors own in Section 9: gold labels are single-annotator with sampled review and no inter-annotator-agreement statistic, and the same annotator wrote the question, chose the positives, and picked the hard negatives. If even a small fraction of hard negatives actually support their own question—templated metrics across competitors make this plausible—the premium is inflated. The 442 byte-identical overlaps with other records' positives are partly mitigated by releasing hn_taxonomy.json, and the 11 within-record duplicates were removed, but the broader label-quality question remains. Compounding this, the random-negative contrast uses one seed (42) and the main tables carry no confidence intervals, so the exact 13–20.5 pt gap is not statistically characterized. Both are fixable with a stratified double-annotation study and variance reporting.\n\nThese concerns do not sink the paper. The benchmark is a reproducible resource, the provenance-sensitive evaluation axis is a real gap in the existing financial QA landscape, and the limitations section is unusually candid. Who benefits: anyone building retrieval or reranking systems for SEC filings, and anyone who wants a benchmark where evidence attribution is separated from answer generation. It deserves a serious referee, with the request that the annotation follow-up and confidence intervals be addressed before the hardness premium is treated as a settled property.","headline":"FinRank is a carefully built, genuinely useful benchmark whose novel asset—per-question curated hard negatives—is real, but its headline hardness premium rests on single-annotator labels, so treat the 13–20.5 pt gap as provisional until a double-annotation check lands.","tokens_in":23404,"tokens_out":1512,"would_cite":true,"duration_ms":15455,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FinRank, a benchmark over 10-K and 10-Q filings, shows that hand-curated hard negatives cut every tested system's pairwise ranking accuracy by 13.0 to 20.5 percentage points compared with random negatives.","keywords":["financial question answering","information retrieval","hard negatives","SEC filings","benchmark","retrieval-augmented generation","evidence grounding","10-K and 10-Q filings"],"falsifier":"Re-annotate a stratified random sample of about 200 records with independent financial analysts and adjudicate disagreements; if a substantial share of gold supporting passages are judged irrelevant or a substantial share of hard negatives are judged relevant, the reported hardness premium would be partly an artifact of label noise rather than model confusion.","tokens_in":22448,"feed_emoji":"📊","tokens_out":7934,"duration_ms":63082,"temperature":0.7,"pith_summary":"FinRank is a financial question-answering benchmark built from 1,185 manually written question–answer pairs over the 10-K and 10-Q filings of 22 companies. Its central claim is that the hardest part of financial QA over SEC filings is not composing an answer but identifying the correct evidence, because near-identical disclosures recur across companies, reporting periods, and filing types. To make that measurable, FinRank releases, for every question, gold supporting passages and human-curated hard negatives—plausible but wrong passages drawn mostly from competitors' filings. Baselines on the benchmark show the point: every evaluated model loses 13.0 to 20.5 percentage points of pairwise ranking accuracy when random negatives are replaced with the curated ones, and even a 7-billion-parameter embedder finds only 44.8% of gold evidence in its top ten. The paper argues that existing financial QA benchmarks, which score answer correctness over supplied snippets, miss this provenance-sensitive failure mode entirely.","feed_headline":"Curated distractors cut financial QA ranking by up to 20.5 points","feed_subtitle":"New SEC-filing benchmark shows models struggle to tell a company's evidence from a competitor's near-identical disclosure.","key_machinery":"The mechanism that carries the argument is the hard-negative contrast embedded in the benchmark design: for each record $r$, the in-record candidate set $L_r = G_r \\cup HN_r$ (gold supporting passages plus hand-curated hard negatives) isolates reranking from first-stage recall, while the global pooled corpus $C$ (5,230 deduplicated passages) defines first-stage retrieval. The decisive quantity is the hardness premium—the drop in pairwise accuracy on (positive, hard-negative) pairs compared with (positive, random-negative) pairs—which is what shows the curated distractors are genuinely confusable and justifies shipping them as a first-class benchmark asset.","core_discovery":"FinRank claims to be the first financial QA benchmark to release, for every question, a dedicated set of human-selected, semantically confusable hard-negative passages, and to score discrimination against curated versus random distractors directly. The benchmark comprises 1,185 records over 10-K and 10-Q filings of 22 companies; each record pairs a question and reference answer with gold supporting passages, rich metadata, and on average 5.08 hard negatives. Executed baselines show that ranking within a curated pool of confusable filing passages is hard: the strongest evaluated system, a 7B instruction-tuned embedder, reaches 44.8% Recall@10 on the 5,230-passage pooled corpus, sub-billion-parameter encoders gain at most 3.5 points over BM25, and a finance-adapted embedder trails BM25 by 9.7 points. In the per-record reranking task, pairwise accuracy falls 13.0–20.5 percentage points when random negatives are replaced with curated hard negatives, with 79.9% of the hard negatives drawn from same-industry, different-company filings. The authors conclude that evidence discrimination rather than answer composition is the primary bottleneck in financial QA over regulatory filings.","pith_inferences":["A natural extension the paper leaves implicit is using FinRank's curated hard negatives as training data: the literature it cites shows hard negatives improve dense retrievers, so the benchmark's value may be as much in training as in evaluation.","Because gold labels are single-annotator with sampled review and no inter-annotator agreement statistic, part of the measured hardness gap could be label noise; a stratified double-annotation study would show whether the gap shrinks once labels are adjudicated.","The pooled corpus $C$ contains only annotator-selected positives and curated distractors, so FinRank scores are not directly comparable to full-document retrieval; an exhaustive filing-level chunking pass would bridge the benchmark to production RAG settings.","The 10-Q records are all from oil and gas and predominantly 2025, so document-type gaps are confounded with sector, year, and annotator; controlled splits would be needed before attributing the 10-Q deficit to document type."],"forward_implications":["Retrieval, reranking, and hard-negative discrimination become separately measurable tasks, so a system that produces a plausible answer from the wrong evidence is exposed at the ranking stage rather than masked by answer-correctness metrics.","Existing financial QA benchmarks that score numerical or end-to-end correctness over supplied snippets likely miss the dominant failure mode of provenance-sensitive filing questions.","Metadata pre-filtering by ticker, year, and document type removes 92.9% of hard negatives and lifts BM25 Recall@10 from 32.1 to 55.0, yet still misses nearly half of gold evidence at k=10, so within-filing semantic discrimination remains a real bottleneck.","Aggregate scores on FinRank primarily reflect the dominant strata (10-K, 2025, qualitative, pharmaceutical and oil-and-gas records), so per-stratum reporting is needed to avoid hiding weakness on 10-Q, multi-passage, and quantitative questions."],"supporting_citations":[{"why":"The closest prior open-book financial QA benchmark over SEC filings; FinRank contrasts its answer-correctness scoring with evidence-level evaluation.","marker":"Islam et al., 2023"},{"why":"The financial RAG retrieval benchmark with expert queries and gold evidence; FinRank adds per-question curated hard negatives it lacks.","marker":"Choi et al., 2025"},{"why":"Supplies the BM25 sparse baseline that FinRank uses as the reference point for dense retrievers.","marker":"Robertson and Zaragoza, 2009"},{"why":"Defines dense passage retrieval architecture that FinRank evaluates and extends to hard-negative discrimination.","marker":"Karpukhin et al., 2020"},{"why":"Provides the Sentence-BERT dense retriever baseline evaluated on FinRank.","marker":"Reimers and Gurevych, 2019"},{"why":"Supplies the cross-encoder reranking architecture used in FinRank's reranking baselines.","marker":"Nogueira and Cho, 2019"},{"why":"Establishes the hard-negative training principle that FinRank repurposes from training signal to evaluation asset.","marker":"Xiong et al., 2021"},{"why":"Frames the RAG setting FinRank is designed to evaluate, motivating its focus on grounding and attribution.","marker":"Lewis et al., 2020"}],"fun_headline_variants":["FinRank: evidence discrimination is the real bottleneck in financial QA","Curated hard negatives drop financial QA reranking by up to 20.5 points","FinRank: 7B model only 44.8% recall on SEC evidence retrieval","New SEC-filing benchmark exposes models' evidence blindness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the gold labels are correct—that each question's supporting passages really support the answer and each hard negative really is non-supporting—since records were single-authored with only sampled review and no formal inter-annotator agreement, and 442 hard negatives are byte-identical to other records' supporting passages.","fun_headline_variants_meta":{"raw":{"variants":["FinRank: evidence discrimination is the real bottleneck in financial QA","Curated hard negatives drop financial QA reranking by up to 20.5 points","FinRank: 7B model only 44.8% recall on SEC evidence retrieval","New SEC-filing benchmark exposes models' evidence blindness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001272,"raw_usage":{"total_tokens":5262,"prompt_tokens":1063,"completion_tokens":4199,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":679,"completion_tokens_details":{"reasoning_tokens":4119}},"tokens_in":679,"tokens_out":4199,"duration_ms":30172,"temperature":1.0,"reasoning_tokens":4119,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T05:04:42.335912+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a stratified random sample of about 200 records with independent financial analysts and adjudicate disagreements; if a substantial share of gold supporting passages are judged irrelevant or a substantial share of hard negatives are judged relevant, the reported hardness premium would be partly an artifact of label noise rather than model confusion.","supporting_citations":[{"cited_title":"Proceedings of the 6th ACM International Conference on AI in Finance (ICAIF '25) , publisher =","cited_arxiv_id":null,"evidence_quote":"The financial RAG retrieval benchmark with expert queries and gold evidence; FinRank adds per-question curated hard negatives it lacks."}],"review_version":1}