REVIEW 3 major objections 5 minor 5 references
Answering questions by learning to rank -- Learning to rank by answering questions
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A self-attention network that latently learns to rank retrieved documents while predicting answers reaches 72.30% on ARC Easy and 44.72% on ARC Challenge, surpassing previous state-of-the-art systems.
desk verdict A useful ranking architecture and clean ablation, but the SOTA claim is within noise — referee it, but ask for statistics. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Attentive Ranker, a self-attention layer over per-document discriminator scores. For each candidate answer, the top N retrieved documents (N=40 in the main experiments) are encoded as columns of a matrix of discriminator scores; a learned projection maps the scores into a higher-dimensional space, then a key-value attention mechanism computes a weighted average of the projected document vectors, with attention weights shared across all candidate answers. A feed-forward network turns the attended vector into an answer score. What makes this carry the argument is the feedback loop: because the only training signal is the correct answer, the attention weights must learn to concentrate on documents that actually support the correct answer, while the pre-trained discriminators supply a starting notion of relevance that the ranker refines.
What would settle it
On ARC Challenge, take the questions where the Attentive Ranker selects a different top document than TF-IDF; if the selected document does not contain the correct answer substantially more often than the TF-IDF top document, the semantic ranking claim is falsified. A broader test would run the same pipeline on another multiple-choice QA dataset with noisy retrieval and check whether the ranker's documents improve a downstream reader relative to TF-IDF ranking.
Extended reading notes
Core claim
The central claim is that a neural module, the Attentive Ranker, can learn to rank supporting documents semantically without any document-level relevance labels, purely from the downstream objective of choosing the correct answer among four choices. Each retrieved document is scored by three pre-trained discriminators—a TF-IDF lexical score, a document relevance discriminator trained on answerable versus unanswerable SQuAD 2.0 questions, and an answer verifier trained on RACE—and the ranker attends over those score vectors with weights shared across candidate answers. The attended representation is used to score each candidate answer, so the network must place useful documents high in order to answer correctly. The paper reports that this joint loop yields 72.30% accuracy on ARC Easy and 44.72% on ARC Challenge, exceeding the previous best published results on both leaderboards, and that documents re-sorted by the ranker improve a downstream BERT classifier by up to 7 percentage points compared with TF-IDF or doc2vec sorting.
Load-bearing premise
The load-bearing premise is that the notion of 'this document helps answer this question' learned from SQuAD 2.0 answerability and RACE answer verification transfers to ARC science questions, which come from a different domain and retrieval setting.
Editorial extensions
If this is right
- Using the top 40 retrieved documents instead of only the top one improves accuracy on both ARC Easy and ARC Challenge, confirming that relevant documents are buried in the retrieval list but can be surfaced by learned ranking.
- A downstream BERT classifier performs better when fed documents ranked by the Attentive Ranker than when fed documents ranked by TF-IDF or doc2vec, with gains up to about 7.8 percentage points on ARC Challenge validation.
- The published ranked documents can be adopted off-the-shelf by other multiple-choice QA models, giving them a semantic ranking without retraining the ranker.
- The joint training loop, in which answer prediction supervises document ranking, offers a general recipe for retrieval-augmented QA when relevance labels are unavailable.
Reading between the lines
- An implication the paper leaves implicit is that the same latent-ranking loop could apply to other retrieval-augmented tasks, such as open-domain claim verification, where the output is a verdict rather than a multiple-choice answer.
- The ablation results suggest that the semantic discriminators contribute far more on the harder Challenge questions than on Easy ones, so a testable extension is to measure whether the ranker's benefit grows with question difficulty in other datasets.
- Because the ranked documents are public, a natural next test is to train a retrieval model directly on these rankings and see whether it surpasses the original lexical engine on questions outside science exams.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper addresses open-book multiple-choice question answering in a setting where no supporting documents are provided. The proposed system first retrieves N documents per candidate answer using a lexical Lucene/TF-IDF engine, then scores each document with three discriminators: the TF-IDF lexical score, a BERT-large document relevance discriminator trained on an answerability version of SQuAD 2.0, and a BERT-large answer verifier discriminator trained on RACE. These scores are combined by a self-attention based neural network, the Attentive Ranker, which is trained end-to-end on ARC to predict the correct answer while implicitly learning to rank documents. The paper reports state-of-the-art test accuracy on ARC Easy (72.30%) and ARC Challenge (44.72%), claims first place on both leaderboards, and shows that the ranked documents improve a downstream BERT reader over TF-IDF and doc2vec rankings. The ranked documents, source code, and trained models are publicly released.
Significance. If the reported results hold, the paper makes a useful engineering contribution: it demonstrates that semantic re-ranking of lexically retrieved documents, using discriminators pre-trained on SQuAD 2.0 and RACE, can improve multiple-choice science QA, and the public release of ranked documents and code supports reproducibility. The ablation in Table 5 is clean and provides empirical evidence that both semantic discriminators contribute to accuracy beyond the lexical baseline. The main weakness is statistical: the headline state-of-the-art claims rest on single validation-selected runs without confidence intervals, and the ARC Challenge margin over the previous leaderboard entry is approximately one test question. The downstream-ranking claim is also stated in the abstract and conclusion using a validation-set gain that is much larger than the corresponding test gain.
major comments (3)
- [§4, Tables 3–4] The central claim of state-of-the-art accuracy is not statistically supported by the reported results. The text in §4 states that the authors "experienced a large variance in performance" and trained models multiple times with different random initializations, keeping only the weights with the lowest validation loss. This model-selection procedure makes the reported test accuracy an upward-biased point estimate, yet no variance, confidence interval, or significance test is reported. On ARC Challenge, 44.72% versus the previous SOTA of 44.62% is a difference of roughly one question out of 1,172; the approximate 95% confidence interval for a single-run accuracy is about ±2.8 percentage points. The same concern applies to the ARC Easy comparison (72.30% vs. 68.90%, interval roughly ±1.8 points). To support the "first place" contribution, the authors should report the mean and standard deviation over random seeds, confidence intervals (e.g., bootstrap), and ideally a paired significance test against comparison systems with matched retrieval settings.
- [§4, Table 6; §5] The contribution claiming that the ranked documents boost downstream classifiers "by up to 7%" is based on the validation-set gain (+7.79 at Top 10) while the corresponding test gain is +2.92, a gap the paper itself attributes to overfitting of hyperparameters. The abstract and conclusion restate the larger validation number without that qualification. The headline should be the test-set improvement (+2.92 on Top 10), or the paper should report a proper evaluation protocol for the downstream ranking claim, including validation/test consistency and error bars.
- [§3.3–3.4, Table 5] The transferability of the two semantic discriminators to ARC is asserted rather than measured: no experiment shows that the DRD and ARD scores correlate with human judgments of relevance on ARC documents, and the only evidence is the cumulative accuracy gain in Table 5. I do not regard this as fatal, because the ablation is consistent with the discriminators being useful, but the paper should add at least a qualitative error analysis or a ranking-quality metric (e.g., nDCG against annotated ARC relevance) to substantiate the semantic-ranking interpretation of the gains.
minor comments (5)
- [Figures 2–3] The captions of Figure 2 and Figure 3 are identical, but the figures appear to depict different stages (document scoring/attention versus the final decision network); the captions should be differentiated.
- [§3.6, Eq. (3)] The softmax in Eq. (3) is applied to a vector W_P K + b_P; please state explicitly that the softmax is over the N document columns, since the notation is otherwise ambiguous.
- [Tables 5 and 7] There are typos in the table headers: "TFD" should be "TF-IDF" in Table 5, and "dov2vec" should be "doc2vec" in Table 7.
- [References] The reference list contains an incomplete entry for Radford (2018) and a stray line "translation. arXiv preprint arXiv:1406.1078." left over from the Cho et al. entry; these should be cleaned up.
- [§4, Table 4] The row "BERT (previous SOTA - Microsoft)" lacks a citation or leaderboard submission identifier; since leaderboard entries are time-sensitive, the paper should specify the date and the exact entry it compares against.
Circularity Check
No circular derivation: the ranking model is trained end-to-end on ARC answer labels and evaluated against external leaderboards, with discriminators trained on SQuAD 2.0 and RACE.
full rationale
No circularity is present in the paper's derivation chain. The central claim, state-of-the-art accuracy on ARC Easy and ARC Challenge, is evaluated against external leaderboard entries and baselines, not against the model's own training objective. The two semantic discriminators are trained on external datasets: the DRD is trained on an adapted SQuAD 2.0 answerable/unanswerable task, and the ARD is trained on RACE. Section 3.4 explicitly says that 'both the DRD and ARD discriminators have been trained on different datasets than the final multiple choice QA model,' so their scores are not fitted to ARC labels. The Attentive Ranker's objective is answer prediction, with document ranking learned as a latent byproduct; the phrase 'Answering questions by learning to rank -- Learning to rank by answering questions' describes joint training, not an equation that assumes its own conclusion. The same-author citations, mainly the book collection used as a retrieval corpus and related work on essential terms, are data provenance and background, not unverified premises that force the results. The reported practice of training multiple times and keeping the lowest-validation-loss weights is a statistical selection issue, not definitional circularity, and the paper's own comparison to external baselines makes the evaluation self-contained.
Assumptions & free parameters
free parameters (4)
- Projection dimension D =
32
- Number of retrieved documents N =
40 (20 from ARC Corpus, 20 from book collection)
- BERT fine-tuning hyperparameters =
Learning rate 3e-5, warm-up 10%, max sequence length 425 or 512, batch size 10 or 24
- Attentive Ranker training hyperparameters =
50 epochs, batch size 128, Adam optimizer, D=32
assumptions (5)
- domain assumption SQuAD 2.0 unanswerable questions mark paragraphs as irrelevant to the question
- domain assumption RACE paragraphs are sufficient to verify whether a candidate answer is correct
- domain assumption Relevance and answer-verification signals transfer from SQuAD 2.0 and RACE to ARC science questions
- domain assumption The top 40 lexically retrieved documents contain at least one document sufficient to answer each question
- standard math Self-attention operations (softmax, tanh, ReLU, linear projections) can compose the discriminator scores into a useful relevance weighting
Cite this review
Pith. "Pith review of Answering questions by learning to rank -- Learning to rank by answering questions." pith.science (2026). https://pith.science/paper/5EB723WA
@misc{pith2026190900596,
author = {Pith},
title = {Pith review of: Answering questions by learning to rank -- Learning to rank by answering questions},
year = {2026},
howpublished = {\url{https://pith.science/paper/5EB723WA}},
note = {Machine review of arXiv:1909.00596}
}
read the original abstract
Answering multiple-choice questions in a setting in which no supporting documents are explicitly provided continues to stand as a core problem in natural language processing. The contribution of this article is two-fold. First, it describes a method which can be used to semantically rank documents extracted from Wikipedia or similar natural language corpora. Second, we propose a model employing the semantic ranking that holds the first place in two of the most popular leaderboards for answering multiple-choice questions: ARC Easy and Challenge. To achieve this, we introduce a self-attention based neural network that latently learns to rank documents by their importance related to a given question, whilst optimizing the objective of predicting the correct answer. These documents are considered relevant contexts for the underlying question. We have published the ranked documents so that they can be used off-the-shelf to improve downstream decision models.
Reference graph
Works this paper leans on
-
[1]
Białecki, A., Muir, R., Ingersoll, G., & Imagination, L. (2012, August). Apache lucene
work page 2012
-
[4]
In SIGIR 2012 workshop on open source information retrieval (p. 17). Banerjee, P ., Pal, K. K., Mitra, A., & Baral, C. (2019). Careful selection of knowledge to solve open book question answering . arXiv preprint arXiv:1907.10738. Brysbaert, M., Warriner, A.B., & Kuperman, V . (2014). Concreteness ratings for 40 thousand generally known English word lemma...
arXiv 2019
-
[5]
Rajpurkar, P ., Zhang, J., Lopyrev, K., & Liang, P . (2016). Squad: 100,000+ questions for machine comprehension of text . arXiv preprint arXiv:1606.05250. Rajpurkar, P ., Jia, R., & Liang, P . (2018). Know What You Don't Know: Unanswerable Questions for SQuAD. arXiv preprint arXiv:1806.03822. translation. arXiv preprint arXiv:1406.1078. Sun, K., Y u, D.,...
arXiv 2016
-
[8]
Pirtoaca, G. S., Ruseti, S., & Rebedea, T. (2018). Improving multi -choice question answering by identifying essential terms in questions. Romanian Journal of Human -Computer Interaction, 11(2), 145-162. Radford, A. (2018). Improving Language Understanding by Generative Pre-Training. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2...
work page 2018
-
[1780]
Jansen, P ., Sharp, R., Surdeanu, M., & Clark, P . (2017). Framing qa as building and ranking intersentence answer justifications . Computational Linguistics, 43(2), 407-449. https://www.doi.org/10.1162/COLI_a_00287 Khashabi, D., Khot, T., Sabharwal, A., & Roth, D. (2017). Learning What is Essential in Questions . CoNLL. Kingma, D. P ., & Ba, J. (2014). A...
arXiv 2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.