{"id":"6e167c80-d73d-48a8-8d84-390a19a6b9d7","arxiv_id":"1909.00596","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Attentive Ranker, a self-attention model over documents scored by pre-trained relevance and answer-verification discriminators, achieves 72.30% on ARC Easy and 44.72% on ARC Challenge and improves downstream QA with its ranked documents.","lead":"This paper introduces Attentive Ranker, a neural network that re-ranks Wikipedia and book passages retrieved for science multiple-choice questions while simultaneously predicting the answer. On the ARC Easy and Challenge benchmarks it reports state-of-the-art accuracy and shows that its rankings improve a downstream BERT question-answering model by up to 7.79 points on validation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim rests on a single validation-selected run; the ARC Challenge margin of 0.10 pp is within noise.","rationale":"The reader identifies transferability of the SQuAD 2.0 and RACE discriminators as the weakest assumption. I partially disagree: the paper provides direct empirical evidence on this point. Table 5 shows that removing either semantic discriminator substantially reduces test accuracy, and Tables 6 and 7 show that documents ranked by the Attentive Ranker improve a downstream BERT model compared with TF-IDF or doc2vec ranking. That is not a proof that the discriminators capture the intended notion of relevance, but it is evidence that the transferred representations carry useful signal under the ARC evaluation objective. The more load-bearing weakness is statistical. Section 4 reports large variance and a model-selection protocol that keeps the lowest validation loss across random initializations. On small datasets, this procedure can inflate test accuracy, and no error bars or significance tests are provided. The ARC Challenge margin over the previous leaderboard entry is 0.10 percentage points on 1,172 questions, which is less than one standard error. The Easy margin is larger, but still needs multi-seed confirmation given the reported variance. The validation-versus-test gap in Table 6 (+7.79 vs +2.92) further suggests selection overfitting. A multi-seed rerun with confidence intervals would settle whether the SOTA claim is robust. Because the architecture is coherent, the ablations are informative, and the failure mode is addressable, the conditional verdict stands; my concern is a different, and arguably more direct, reason for conditionality than the reader's transferability concern.","tokens_in":11375,"tokens_out":3671,"duration_ms":101640,"concrete_test":"Run the full Attentive Ranker pipeline (same retrieval, discriminators, and hyperparameters) for at least 10 random seeds on ARC Challenge, applying the authors' own lowest-validation-loss selection per seed, and report the mean and standard deviation of test accuracy. Compute a 95% confidence interval for the mean; if that interval includes 44.62%, the claimed first place over the previous SOTA is within noise. For ARC Easy, repeat the same multi-seed procedure and compare the interval against 68.90%. This directly tests whether the reported point estimates reflect a systematic advantage or a favorable random initialization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—first place on ARC Easy and ARC Challenge—is not statistically supported by the reported numbers. Section 4 explicitly states that the authors experienced “a large variance in performance” and that they trained the models multiple times with different random initializations, keeping only the weights with the lowest validation loss. With only 1,119 ARC Challenge training questions and 1,172 test questions, this selection procedure produces a point estimate biased upward relative to the expected accuracy of the method, and no variance, confidence interval, or significance test is reported. On ARC Challenge, 44.72% versus the previous SOTA 44.62% is a difference of roughly one test question; the approximate 95% confidence interval is about ±2.8 percentage points, so the claimed first place is statistically indistinguishable from the prior leaderboard entry. The same issue affects ARC Easy (72.30% versus 68.90%, with an interval of roughly ±1.8 pp), and the downstream-ranking claim shows a similar signature: Table 6 reports +7.79 on validation Top-10 but only +2.92 on test, consistent with selection-based overfitting. Even if the SQuAD 2.0/RACE transferability assumption is accepted—and the ablations in Table 5 do provide empirical support for it—the headline superiority is not established by the evidence as reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses open-book multiple-choice question answering in a setting where no supporting documents are provided. The proposed system first retrieves N documents per candidate answer using a lexical Lucene/TF-IDF engine, then scores each document with three discriminators: the TF-IDF lexical score, a BERT-large document relevance discriminator trained on an answerability version of SQuAD 2.0, and a BERT-large answer verifier discriminator trained on RACE. These scores are combined by a self-attention based neural network, the Attentive Ranker, which is trained end-to-end on ARC to predict the correct answer while implicitly learning to rank documents. The paper reports state-of-the-art test accuracy on ARC Easy (72.30%) and ARC Challenge (44.72%), claims first place on both leaderboards, and shows that the ranked documents improve a downstream BERT reader over TF-IDF and doc2vec rankings. The ranked documents, source code, and trained models are publicly released.","tokens_in":11647,"tokens_out":4479,"duration_ms":43346,"significance":"If the reported results hold, the paper makes a useful engineering contribution: it demonstrates that semantic re-ranking of lexically retrieved documents, using discriminators pre-trained on SQuAD 2.0 and RACE, can improve multiple-choice science QA, and the public release of ranked documents and code supports reproducibility. The ablation in Table 5 is clean and provides empirical evidence that both semantic discriminators contribute to accuracy beyond the lexical baseline. The main weakness is statistical: the headline state-of-the-art claims rest on single validation-selected runs without confidence intervals, and the ARC Challenge margin over the previous leaderboard entry is approximately one test question. The downstream-ranking claim is also stated in the abstract and conclusion using a validation-set gain that is much larger than the corresponding test gain.","major_comments":[{"comment":"The central claim of state-of-the-art accuracy is not statistically supported by the reported results. The text in §4 states that the authors \"experienced a large variance in performance\" and trained models multiple times with different random initializations, keeping only the weights with the lowest validation loss. This model-selection procedure makes the reported test accuracy an upward-biased point estimate, yet no variance, confidence interval, or significance test is reported. On ARC Challenge, 44.72% versus the previous SOTA of 44.62% is a difference of roughly one question out of 1,172; the approximate 95% confidence interval for a single-run accuracy is about ±2.8 percentage points. The same concern applies to the ARC Easy comparison (72.30% vs. 68.90%, interval roughly ±1.8 points). To support the \"first place\" contribution, the authors should report the mean and standard deviation over random seeds, confidence intervals (e.g., bootstrap), and ideally a paired significance test against comparison systems with matched retrieval settings.","section":"§4, Tables 3–4"},{"comment":"The contribution claiming that the ranked documents boost downstream classifiers \"by up to 7%\" is based on the validation-set gain (+7.79 at Top 10) while the corresponding test gain is +2.92, a gap the paper itself attributes to overfitting of hyperparameters. The abstract and conclusion restate the larger validation number without that qualification. The headline should be the test-set improvement (+2.92 on Top 10), or the paper should report a proper evaluation protocol for the downstream ranking claim, including validation/test consistency and error bars.","section":"§4, Table 6; §5"},{"comment":"The transferability of the two semantic discriminators to ARC is asserted rather than measured: no experiment shows that the DRD and ARD scores correlate with human judgments of relevance on ARC documents, and the only evidence is the cumulative accuracy gain in Table 5. I do not regard this as fatal, because the ablation is consistent with the discriminators being useful, but the paper should add at least a qualitative error analysis or a ranking-quality metric (e.g., nDCG against annotated ARC relevance) to substantiate the semantic-ranking interpretation of the gains.","section":"§3.3–3.4, Table 5"}],"minor_comments":[{"comment":"The captions of Figure 2 and Figure 3 are identical, but the figures appear to depict different stages (document scoring/attention versus the final decision network); the captions should be differentiated.","section":"Figures 2–3"},{"comment":"The softmax in Eq. (3) is applied to a vector W_P K + b_P; please state explicitly that the softmax is over the N document columns, since the notation is otherwise ambiguous.","section":"§3.6, Eq. (3)"},{"comment":"There are typos in the table headers: \"TFD\" should be \"TF-IDF\" in Table 5, and \"dov2vec\" should be \"doc2vec\" in Table 7.","section":"Tables 5 and 7"},{"comment":"The reference list contains an incomplete entry for Radford (2018) and a stray line \"translation. arXiv preprint arXiv:1406.1078.\" left over from the Cho et al. entry; these should be cleaned up.","section":"References"},{"comment":"The row \"BERT (previous SOTA - Microsoft)\" lacks a citation or leaderboard submission identifier; since leaderboard entries are time-sensitive, the paper should specify the date and the exact entry it compares against.","section":"§4, Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a systems/leaderboard contribution, and the statistical weakness is the main barrier to acceptance. The code and data release are genuine strengths, as is the clean ablation in Table 5. I would encourage the editor to consider whether the 0.10 percentage point ARC Challenge margin should be descoped to \"state-of-the-art-comparable\" even after the authors add confidence intervals, since the leaderboard context is inherently noisy and the model-selection procedure biases the point estimate upward."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nShort version: this paper has a genuinely useful idea — cheap semantic re-ranking of the top-N lexical documents using two off-the-shelf discriminators plus a self-attention layer — and the ablation and downstream ranking experiments are clean. But the headline claim of state-of-the-art on ARC Easy and Challenge is not supported by the reported numbers. The authors admit to large variance and pick the best validation run, yet give no error bars or significance tests. On Challenge, 44.72% vs. the Microsoft BERT's 44.62% is one test question; the uncertainty is roughly ±2.8 pp. So \"first place\" is noise.\n\nWhat's actually new: using SQuAD 2.0 answerability and RACE answer verification as features for document relevance, then learning an attention weight per document while training to answer. That's a reasonable transfer setup, and the discriminators are trained on non-ARC labels, so the circularity burden is low. Table 5 is the best part: adding the two semantic discriminators gives a large, sensible jump over TF-IDF alone, especially on Challenge. The downstream ranking comparison (Tables 6 and 7) is also valuable — replacing TF-IDF/doc2vec ranks with the Attentive Ranker's ranks improves a separate BERT model, and they've published the ranked documents.\n\nSoft spots, in order: (1) Statistics. Single best-of-k runs, no CIs, no significance test. The Easy gain (72.30 vs 68.90) is more robust than Challenge, but still within ~1.8 pp. The paper should report mean±std over multiple runs or at least a paired test. (2) The transfer assumption — SQuAD 2.0 and RACE relevance as proxy for ARC science relevance — is asserted rather than directly validated. The ablation mitigates this, but a small analysis of what the discriminators actually score would help. (3) Minor: the \"up to 20%\" claim in the ablation discussion is overstated (actual total is 18 pp on Challenge). Also the GitHub link lacks a commit hash; for reproducibility you'd want the exact snapshot.\n\nBottom line: this is a solid within-subfield contribution, worth a serious referee. I'd recommend the editor send it to review with a request to soften the SOTA claim and add statistical grounding. I'd cite it for the ranking architecture, not for the leaderboard numbers.\n\nBest,\n[You]","headline":"A useful ranking architecture and clean ablation, but the SOTA claim is within noise — referee it, but ask for statistics.","tokens_in":12162,"tokens_out":2496,"would_cite":true,"duration_ms":23270,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A self-attention network that latently learns to rank retrieved documents while predicting answers reaches 72.30% on ARC Easy and 44.72% on ARC Challenge, surpassing previous state-of-the-art systems.","keywords":["multiple-choice question answering","semantic document ranking","self-attention","transfer learning","BERT","ARC dataset","information retrieval","answer verification"],"falsifier":"On ARC Challenge, take the questions where the Attentive Ranker selects a different top document than TF-IDF; if the selected document does not contain the correct answer substantially more often than the TF-IDF top document, the semantic ranking claim is falsified. A broader test would run the same pipeline on another multiple-choice QA dataset with noisy retrieval and check whether the ranker's documents improve a downstream reader relative to TF-IDF ranking.","tokens_in":11071,"feed_emoji":"🧠","tokens_out":5868,"duration_ms":57376,"temperature":0.7,"pith_summary":"Answering science multiple-choice questions without provided supporting text usually relies on a lexical search for Wikipedia paragraphs, but lexical retrieval often puts the wrong document on top. This paper claims that the fix is not better retrieval but better ranking: a self-attention network, the Attentive Ranker, takes the top handful of lexically retrieved documents and latently learns which ones actually support the correct answer while being trained only to pick the right choice. On the ARC Easy and ARC Challenge benchmarks this pushes accuracy to 72.30% and 44.72%, ahead of previous published systems. The paper further claims that the ranked document lists transfer: feeding them to a downstream BERT-based classifier improves accuracy by up to 7 percentage points over the same classifier using TF-IDF-ranked documents. If these claims hold, semantic re-ranking of retrieved evidence is a cheap, reusable upgrade for open-book multiple-choice QA.","feed_headline":"Semantic ranking lifts both ARC science-QA leaderboards","feed_subtitle":"Attentive Ranker hits 72.3% on Easy and 44.7% on Challenge; its ranked documents boost other models by up to 7%","key_machinery":"The load-bearing object is the Attentive Ranker, a self-attention layer over per-document discriminator scores. For each candidate answer, the top N retrieved documents (N=40 in the main experiments) are encoded as columns of a matrix of discriminator scores; a learned projection maps the scores into a higher-dimensional space, then a key-value attention mechanism computes a weighted average of the projected document vectors, with attention weights shared across all candidate answers. A feed-forward network turns the attended vector into an answer score. What makes this carry the argument is the feedback loop: because the only training signal is the correct answer, the attention weights must learn to concentrate on documents that actually support the correct answer, while the pre-trained discriminators supply a starting notion of relevance that the ranker refines.","core_discovery":"The central claim is that a neural module, the Attentive Ranker, can learn to rank supporting documents semantically without any document-level relevance labels, purely from the downstream objective of choosing the correct answer among four choices. Each retrieved document is scored by three pre-trained discriminators—a TF-IDF lexical score, a document relevance discriminator trained on answerable versus unanswerable SQuAD 2.0 questions, and an answer verifier trained on RACE—and the ranker attends over those score vectors with weights shared across candidate answers. The attended representation is used to score each candidate answer, so the network must place useful documents high in order to answer correctly. The paper reports that this joint loop yields 72.30% accuracy on ARC Easy and 44.72% on ARC Challenge, exceeding the previous best published results on both leaderboards, and that documents re-sorted by the ranker improve a downstream BERT classifier by up to 7 percentage points compared with TF-IDF or doc2vec sorting.","pith_inferences":["An implication the paper leaves implicit is that the same latent-ranking loop could apply to other retrieval-augmented tasks, such as open-domain claim verification, where the output is a verdict rather than a multiple-choice answer.","The ablation results suggest that the semantic discriminators contribute far more on the harder Challenge questions than on Easy ones, so a testable extension is to measure whether the ranker's benefit grows with question difficulty in other datasets.","Because the ranked documents are public, a natural next test is to train a retrieval model directly on these rankings and see whether it surpasses the original lexical engine on questions outside science exams."],"forward_implications":["Using the top 40 retrieved documents instead of only the top one improves accuracy on both ARC Easy and ARC Challenge, confirming that relevant documents are buried in the retrieval list but can be surfaced by learned ranking.","A downstream BERT classifier performs better when fed documents ranked by the Attentive Ranker than when fed documents ranked by TF-IDF or doc2vec, with gains up to about 7.8 percentage points on ARC Challenge validation.","The published ranked documents can be adopted off-the-shelf by other multiple-choice QA models, giving them a semantic ranking without retraining the ranker.","The joint training loop, in which answer prediction supervises document ranking, offers a general recipe for retrieval-augmented QA when relevance labels are unavailable."],"supporting_citations":[{"why":"Introduces the ARC Easy and ARC Challenge datasets and the ARC Corpus, which define the benchmark and one of the two external knowledge bases used for retrieval.","marker":"Clark et al., 2018"},{"why":"Provides the SQuAD 2.0 answerable-versus-unanswerable split that is adapted to train the document relevance discriminator.","marker":"Rajpurkar et al., 2018"},{"why":"Supplies the RACE reading comprehension dataset used to train the answer verifier discriminator.","marker":"Lai et al., 2017"},{"why":"Provides the BERT architecture used for both semantic discriminators and for the downstream classifier in the ranking comparison experiments.","marker":"Devlin et al., 2018"},{"why":"Contributes the essential-term retrieval approach and the book collection knowledge base that the Attentive Ranker builds upon and compares against.","marker":"Pîrtoacă et al., 2018"},{"why":"Documents the finding that lexical retrieval often returns irrelevant or insufficient support for a large fraction of questions, motivating semantic re-ranking.","marker":"Zhang et al., 2018"},{"why":"Holds the previous state-of-the-art results on ARC that the Attentive Ranker claims to surpass.","marker":"Sun et al., 2018"},{"why":"Introduces the self-attention mechanism that inspires the key-value attention used by the Attentive Ranker.","marker":"Vaswani et al., 2017"}],"fun_headline_variants":["No labels needed: Ranker tops ARC Easy and Challenge","Attentive Ranker wins ARC without document labels","Self-attention ranking hits SOTA on ARC science QA","Doc ranker from answer signal tops ARC leaderboards","Ranked documents boost BERT by 7% and top ARC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the notion of 'this document helps answer this question' learned from SQuAD 2.0 answerability and RACE answer verification transfers to ARC science questions, which come from a different domain and retrieval setting.","fun_headline_variants_meta":{"raw":{"variants":["No labels needed: Ranker tops ARC Easy and Challenge","Attentive Ranker wins ARC without document labels","Self-attention ranking hits SOTA on ARC science QA","Doc ranker from answer signal tops ARC leaderboards","Ranked documents boost BERT by 7% and top ARC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00023,"raw_usage":{"total_tokens":1459,"prompt_tokens":898,"completion_tokens":561,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":514,"completion_tokens_details":{"reasoning_tokens":480}},"tokens_in":514,"tokens_out":561,"duration_ms":5706,"temperature":1.0,"reasoning_tokens":480,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T05:42:25.451781+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On ARC Challenge, take the questions where the Attentive Ranker selects a different top document than TF-IDF; if the selected document does not contain the correct answer substantially more often than the TF-IDF top document, the semantic ranking claim is falsified. A broader test would run the same pipeline on another multiple-choice QA dataset with noisy retrieval and check whether the ranker's documents improve a downstream reader relative to TF-IDF ranking.","supporting_citations":[],"review_version":1}