{"id":"960f405b-c0ac-4c58-aa4c-b93ef301d8d7","arxiv_id":"2504.13615","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An answer-paragraph-selection-based retriever improves non-factoid QA in four Indic languages on three LLMs, with average semantic gains of 4% and token-level ROUGE gains of 47% over a full-context baseline, while using less compute.","lead":"Context-shortening pipelines built from an answer-paragraph selection model improve LLM answers to long-context non-factoid questions in Hindi, Tamil, Telugu, and Urdu, compared with feeding the full context. The paper quantifies the gains on three open LLMs and releases code, fine-tuned checkpoints, and a new multilingual semantic similarity metric.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The A1-vs-B comparison is measured only on examples where B did not OOM and on a 1100-item subset; the headline 4%/47% gains could shift under full test coverage.","rationale":"The reader's weakest assumption is exactly the OOM-restricted comparison, and I agree that this is the most load-bearing threat to the paper's central claim. The OOM restriction removes the longest contexts from B's side, and those are the cases the paper's motivation targets most directly. I considered whether the concern should be elevated to REJECT: it should not, because the authors disclose the restriction, provide code and checkpoints, run a contamination check, and include a translated NaturalQA experiment (though without a reported table). The direction of any bias is not established, so the claim remains plausible. I also considered contamination as a separate load-bearing flaw: it is a secondary gap because only B's drop is reported, not the A1-vs-B comparison on filtered examples. The verdict stays conditional pending full-coverage and significance reporting, with no change from the reader's verdict.","tokens_in":20650,"tokens_out":8216,"duration_ms":77754,"concrete_test":"Re-run A1 and B on the full 1100-question subset (or a fresh random sample from the 6k test set) using hardware/software that lets B complete on every example (e.g., 80GB A100s, activation checkpointing, or sequence parallelism), and recompute the A1-vs-B deltas for STS-MuTe and the average ROUGE score, with bootstrap confidence intervals. If the 4% and 47% averages persist on the full set—and on the contamination-filtered examples—the concern is settled; if they shrink or reverse, the abstract's scope must be narrowed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is an A1-over-B improvement, so the comparison must cover a representative slice of the test distribution. Section 5 explicitly states that 'B led to Out of Memory (OOM) errors on many test examples' and therefore B and A1 were compared on the 'test examples common to both' within the 1100-question subset. If Table 1's means are computed on that common subset, the reported 4% STS-MuTe and 47% ROUGE improvements exclude exactly the longest examples, where B is most likely to fail and where context-shortening is arguably most relevant. The paper provides no evidence about B's behavior on the OOM examples, no significance tests, and no error bars; some per-model gains are small (e.g., gemma-7b semantic STS-MuTe 0.60 to 0.61). The self-reported contamination filter also drops B's performance by 13%, 2.5%, and 22% by model, but the paper does not report the A1-vs-B comparison on the filtered examples, leaving open the possibility that part of the relative gain is contamination-driven. This does not invalidate the claim, but it means the headline numbers are conditional on the hardware-restricted subset.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies context-shortening for non-factoid question answering in Hindi, Tamil, Telugu, and Urdu. It builds on the authors' MuNfQuAD dataset and compares four retrieval-based pipelines (APS, OIE+APS, coref+APS, OIE+coref+APS) against a baseline that feeds the full long context to Gemma-2B, Gemma-7B, and Llama-3.1-8B, in both zero-shot and LoRA fine-tuned settings. The authors report that the APS-only pipeline (A1) improves the average semantic score (STS-MuTe) by about 4% and ROUGE by about 47% over the full-context baseline, while also reducing GPU memory and inference time. They also introduce a new semantic similarity metric, report an LLM-as-a-judge evaluation, and use LIME and SHAP to analyze the APS model's rationales. Code, checkpoints, and resources are released. The headline comparison is limited to a 1100-question subset of the 6k+ test set and, within that subset, to the examples on which the full-context baseline did not run out of memory.","tokens_in":20954,"tokens_out":4714,"duration_ms":43787,"significance":"If the reported improvement is robust, this is a useful practical contribution: it shows that a fine-tuned paragraph selector can reduce long contexts for non-factoid QA in low-resource languages while matching or exceeding full-context LLM performance at substantially lower computational cost. The paper is transparent in several important ways: the APS training data is excluded from evaluation (a proper held-out split), ablations against BM25 and dense retrieval are included, computational footprints are reported, and a contamination analysis is attempted. These strengths make the central claim worth taking seriously. The main weakness is that the central A1-vs-B comparison is restricted to the subset where the baseline did not OOM, and no uncertainty quantification is provided; the headline numbers are therefore conditional on the hardware-restricted subset rather than unconditional claims about the test distribution.","major_comments":[{"comment":"The central claim that A1 outperforms B is evaluated only on the test examples common to both, because B OOMs on many long examples. The paper states: 'The resource intensive nature of B led to Out of Memory (OOM) errors on many test examples' and therefore B and A1 were compared on the 'test examples common to both' among the 1100. Since the OOM examples have average context lengths of 1238, 1189, and 1069 tokens by model (reported in §5), the comparison excludes exactly the longest contexts, where context-shortening is arguably most relevant. To support the abstract's 4% and 47% claims, the authors should either run B on the full 1100-item subset (e.g., with chunking, a longer-context model, or reduced batch size) or provide evidence that the common subset is representative in terms of context length, language, and question category distributions. As written, the reported gains are estimates conditional on the non-OOM subset.","section":"§5, Tables 1 and 3"},{"comment":"The contamination filter shows that B's zero-shot scores drop by 13%, 2.5%, and 22% when test data that were publicly available before each model release are excluded, but the paper does not report A1's filtered scores or the A1-vs-B comparison on the filtered examples. This matters because the relative gain could change if contamination inflates B and A1 differently. The filtered sets are small (78, 197, and 18 examples), but the manuscript should still report the filtered comparison for at least STS-MuTe and ROUGE, with the small sample sizes acknowledged, so that readers can see whether the relative improvement survives the filter.","section":"§8, contamination analysis"},{"comment":"Several per-model differences are small (e.g., gemma-7b-it STS-MuTe: B 0.60 vs A1 0.61; BERTScore 0.66 vs 0.69 for gemma-7b-it), and no confidence intervals, bootstrap estimates, or significance tests are provided. Since the headline is an average over three models and four languages, the 4% semantic improvement may be within run-to-run noise. Please report per-language, per-model results with bootstrap confidence intervals or pairwise significance tests on the common subset, or explicitly state that the observed differences are not tested for significance.","section":"§5, Table 1"}],"minor_comments":[{"comment":"The LLM-as-a-judge protocol should state whether the order of option1 and option2 is randomized across examples, and should report agreement with human judgments if available; the current text only gives preference percentages.","section":"§5, LLM-as-a-judge"},{"comment":"Please clarify the relation between the 'over 40K' question-answer pairs after excluding the APS training data, the 'more than 29K' training questions, and the 'more than 6k' test questions; these numbers do not obviously sum, and a precise split description would help reproducibility.","section":"§3 and §4, data splits"},{"comment":"There are minor typos (e.g., 'The A1 approach does not only shows better performance' in §5) and duplicate bibliography entries (Kolluru et al. 2022a/2022b and Ribeiro et al. 2016a/2016b); please clean these before publication.","section":"§2 and References"},{"comment":"The text describing Figure 6 gives a concrete example of 7 out of 10 tokens above a threshold, but the figure caption does not explain the bucketing clearly; please make the caption self-contained and state which languages and how many examples the 8.5k question-paragraph rationale computation covers.","section":"§6, Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a CL/NLP venue, and the central comparison is not circular: the held-out split from the APS training data and the transparent baseline comparisons are genuine strengths. The main concern is that the headline numbers are computed on the subset where the baseline did not OOM, and the paper does not quantify the resulting selection bias. This is fixable within the manuscript's scope by either running the baseline on all 1100 examples or clearly reframing the claims as conditional on the non-OOM subset and adding uncertainty estimates. The heavy reliance on the authors' own prior resources (MuNfQuAD, IndIE, Transmucores) is a limitation to flag in the text, but not a reason for rejection given the held-out evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version first. This is a legitimate empirical paper about context-shortening for non-factoid QA in Hindi, Tamil, Telugu, and Urdu. The best part is the retriever-based pipeline (APS) is tested against a full-context baseline, with ablations against BM25 and dense retrieval, compute footprint reporting, and a genuinely candid limitations section. The STS-MuTe metric is a reasonable addition, and the released checkpoints for gemma-2b/7b and llama-3.1 are reusable. If you work on multilingual QA, there is real value here.\n\nThe central claim—that APS-based shortening beats feeding the whole context—is plausible and the direction is consistent across three LLMs. But the headline numbers are softer than they look. The baseline B OOMs on many test examples, and the A1-vs-B comparison is restricted to the examples where B fit in memory. Those are exactly the shorter contexts. The paper gives the average token counts for the OOM examples (1238, 1189, 1069 per model) but not B's performance on them, so the reported 4% and 47% gains are conditional on hardware. On top of that, there are no error bars or significance tests, and some per-model gains are tiny (gemma-7b STS-MuTe goes from 0.60 to 0.61). The contamination check is a point in their favor, but it only reports how much B drops (13%, 2.5%, 22%) after excluding pretraining-era data; they don't give the A1-vs-B comparison on that filtered set, so we can't rule out that part of the relative gain is contamination-driven.\n\nNone of this kills the paper. The evaluation is transparent about what was done, and the limitations section reads like the authors know where the weak points are. The heavy reliance on their own prior tools (APS, coref, MuNfQuAD) is a dependence rather than a flaw, since those resources are published and the held-out split is proper. But the headline claim should be treated as provisional.\n\nSend it to review. The right reviewers will ask for the full-non-OOM comparison, significance tests, and the contamination-filtered A1-vs-B table. This is the kind of paper that gets noticeably better with one solid revision round, and the released artifacts make it worth the referee time.","headline":"The headline gains (4% semantic, 47% token-level) are real but measured only on examples where the full-context baseline fit in memory; the paper is a solid, transparent empirical study whose central claim is plausible, but the headline numbers need a fuller comparison before being taken at face value.","tokens_in":21456,"tokens_out":2934,"would_cite":true,"duration_ms":26358,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Shrinking the context with an answer-paragraph retriever improves non-factoid QA in four Indic languages.","keywords":["non-factoid question answering","long-context QA","answer paragraph selection","Indic languages","context shortening","multilingual retrieval","low-resource NLP","explainable NLP"],"falsifier":"Run the baseline B and the A1 pipeline on every one of the 1,100 sampled questions on hardware that avoids out-of-memory errors and compare average STS-MuTe and ROUGE; if the gains shrink, reverse, or vanish on the previously excluded examples, the paper's central claim fails. A second check is to compare B and A1 on the translated Natural-QA subset with contexts above 8,000 tokens, where the paper reports only topline results for A1.","tokens_in":20454,"feed_emoji":"🧠","tokens_out":8163,"duration_ms":63407,"temperature":0.7,"pith_summary":"This paper sets out to establish that the main obstacle to non-factoid question answering in low-resource languages is the length of the context, not the generative ability of the LLM. It claims that a trained Answer Paragraph Selection (APS) model, which scores each paragraph against the question and keeps the top five, yields better answers than feeding the full long context to the model. Across Gemma-2b, Gemma-7b and Llama-3.1-8b without fine-tuning, the shortened-context pipeline improves average semantic similarity by 4% and token-level overlap scores by 47%, while also reducing memory and latency. The same pipeline with fine-tuning gives about 2% improvement on both metric families. The paper also uses LIME and SHAP to claim that the APS model's confident decisions rest on nearly all tokens in the chosen paragraph, whereas low-confidence decisions rest on only a few.","feed_headline":"Shorter contexts beat long ones for Indic-language QA","feed_subtitle":"A paragraph-selection retriever lifts non-factoid answering in Hindi, Tamil, Telugu and Urdu while cutting compute.","key_machinery":"The central object is the Answer Paragraph Selection (APS) model, a cross-encoder that reads a question and a paragraph together and outputs a relevance score. The paper uses a fine-tuned XLM-R based checkpoint to score every paragraph in the long context, ranks them, and keeps the top five as the LLM's context. Around this core sit four retrieval variants: APS alone (A1), verbalized OpenIE triples re-ranked by APS (A2), coreference-linked paragraphs re-ranked by APS (A3), and coreference clusters of verbalized triples re-ranked by APS (A4). The evaluation side is carried by STS-MuTe, a new semantic metric that averages cosine similarities from three multilingual embedding models with BERTScore, and by standard ROUGE variants for token-level comparison.","core_discovery":"On its own terms, the discovery is that a multilingual paragraph-scoring model can serve as a question-specific context compressor for non-factoid QA in Hindi, Tamil, Telugu, and Urdu. The A1 pipeline, built from a fine-tuned XLM-R based APS model, outperforms the full-context baseline B on the common test examples for all three LLMs, with a 4% average gain in STS-MuTe (a composite of BERTScore, USE, LaBSE, and LASER similarities) and a 47% average gain in ROUGE scores when the LLMs are not fine-tuned; fine-tuning shrinks the gain to about 2%. The token-level gain is large but the semantic gain is modest, and the paper attributes the semantic gain to the retriever's ability to find the paragraphs that actually answer the question. In an ablation, fine-tuned APS beats both BM25 and an embedding-based vectorstore retriever, and 75% of its top-k paragraphs differ from BM25's, supporting the claim that the selection is semantic rather than lexical.","pith_inferences":["Editorial inference: the A1 pipeline should transfer to other low-resource languages with a cross-lingual encoder, because it depends on no OIE or coreference tools; the paper only tests Hindi, Tamil, Telugu, and Urdu.","Editorial inference: the true improvement over the baseline is probably closer to the 4% semantic gain than the 47% token-level gain, because token-level metrics penalize legitimate paraphrase in LLM answers.","Editorial inference: a direct stress test is to run the baseline on the OOM examples with larger GPUs; if the baseline wins there, the central comparison is an artifact of memory limits rather than a property of long contexts.","Editorial inference: using the APS score as a reward model for preference tuning, which the paper names as future work, is a testable next step: it would show whether retriever confidence aligns with downstream answer quality."],"forward_implications":["If the gain is real, long-context QA systems for low-resource languages can be improved and made cheaper at the same time by inserting a paragraph-selection retriever before generation.","The result implies that semantic metrics and token-level metrics move together in this setting, so the improvement is not just a paraphrase effect.","The explainability finding suggests the APS score can be used as a confidence signal: low-scoring paragraphs are likely to be irrelevant, so the retriever can safely discard them.","Because verbalizing OIE triples did not help, triple extraction is probably not a useful context-shortening step for non-factoid Indic QA, despite its success in knowledge-grounded English tasks.","The 75% disagreement with BM25 indicates that lexical retrieval alone is insufficient for non-factoid Indic QA, supporting the need for learned semantic retrieval."],"supporting_citations":[{"why":"Supplies the MuNfQuAD dataset, the question categories, and the fine-tuned APS model used in every retrieval variant.","marker":"Mishra et al. 2024b"},{"why":"Provides the Gemma instruction-tuned models used as two of the three generators.","marker":"Team et al. 2024b"},{"why":"Provides Llama 3.1 8B, the third LLM in the evaluation.","marker":"Dubey et al. 2024"},{"why":"Supplies XLM-R, the base encoder for the fine-tuned APS model.","marker":"Conneau et al. 2020"},{"why":"Provides ROUGE, the token-level overlap metric used throughout the evaluation.","marker":"Lin 2004"},{"why":"Provides LIME, one of the two post-hoc explainability methods applied to the APS model.","marker":"Ribeiro et al. 2016b"},{"why":"Provides SHAP, the second explainability method applied to the APS model.","marker":"Lundberg and Lee 2017b"},{"why":"Supplies Gen2OIE, one of the two multilingual OpenIE tools that produce triples for A2 and A4.","marker":"Kolluru et al. 2022b"},{"why":"Supplies the multilingual coreference resolution model used to build A3 and A4.","marker":"Mishra et al. 2024a"},{"why":"Supplies the Natural Questions QA pairs translated into six Indic languages for the longer-context evaluation.","marker":"Kwiatkowski et al. 2019"}],"fun_headline_variants":["Indic non-factoid QA: shorter contexts win by 47%","Context-shortening lifts Indic QA token scores 47%","Cut contexts, cut compute: better Indic QA","Shorter context, higher token score for Indic QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison that carries the paper's headline numbers was run only on the subset of a 1,100-question sample where the full-context baseline did not run out of memory, so the load-bearing premise is that those completed examples represent the full test distribution.","fun_headline_variants_meta":{"raw":{"variants":["Indic non-factoid QA: shorter contexts win by 47%","Context-shortening lifts Indic QA token scores 47%","Cut contexts, cut compute: better Indic QA","Shorter context, higher token score for Indic QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000519,"raw_usage":{"total_tokens":2579,"prompt_tokens":1078,"completion_tokens":1501,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":694,"completion_tokens_details":{"reasoning_tokens":1432}},"tokens_in":694,"tokens_out":1501,"duration_ms":11106,"temperature":1.0,"reasoning_tokens":1432,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:03:21.004408+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the baseline B and the A1 pipeline on every one of the 1,100 sampled questions on hardware that avoids out-of-memory errors and compare average STS-MuTe and ROUGE; if the gains shrink, reverse, or vanish on the previously excluded examples, the paper's central claim fails. A second check is to compare B and A1 on the translated Natural-QA subset with contexts above 8,000 tokens, where the paper reports only topline results for A1.","supporting_citations":[],"review_version":1}