{"id":"1ad2343a-908f-4efe-a683-326454a1acb1","arxiv_id":"1908.10449","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Reframing machine reading comprehension as an interactive, partially observable environment where agents reveal hidden sentences via commands, and showing a DQN-based baseline can learn to seek answers.","lead":"This paper turns standard reading comprehension datasets into interactive games where an agent must reveal hidden sentences by issuing commands, then answer a question. The authors show that a reinforcement-learning agent can learn to seek information this way, and they argue the setup better matches real-world web search.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No non-learning baselines in Table 3: DQN F1 alone does not show the agent learned to seek information.","rationale":"The reader's weakest-assumption (external validity of the synthetic environment) is a genuine limitation, but it is explicitly hedged in Section 6 ('we hypothesize', 'we believe'), so it does not directly falsify any asserted result. The more immediately load-bearing gap is internal: the paper's own central sentence in the Abstract ('we show how the interactive corpora can be used to train a model that seeks relevant information through sequential decision making') is an assertion about what the agent learned, and Table 3 is the only evidence offered. Because the action space in the 'QUERY from question' condition is just the set of question tokens, and because the environment's Ctrl+F transition is a deterministic exact-match jump, a hand-coded search policy is a natural and strong control. Without such controls, the observed F1 values are equally compatible with the agent having learned a reasonable search policy and with it having converged to a simple loop over question tokens. The reader's rationale already lists missing baselines and variance; this check would settle the issue directly. The full-document QANet upper bound further separates the difficulty of the interaction game from the underlying answerer. For these reasons the concern does not change the CONDITIONAL verdict; it sharpens the condition.","tokens_in":12329,"tokens_out":6767,"duration_ms":71231,"concrete_test":"Run four controls on the iSQuAD and iNewsQA hard-mode, QUERY-from-question setting: (1) stop immediately and answer from the first visible sentence; (2) uniform-random action selection over the same action space; (3) a fixed loop that issues Ctrl+F for each token of the question in order (skipping repeats) and stops when the observed sentence contains the answer or the 20-step budget is exhausted; (4) the same QANet answerer with the full paragraph visible as an upper bound. Compare F1 and F1info against the DQN rows of Table 3 using at least 3 seeds. If DQN does not beat the best control by more than the seed variance, the 'learns to seek' claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The claim that the agent 'seeks relevant information through sequential decision making' (Abstract) rests entirely on the F1 numbers in Table 3, but that table contains no non-learning controls. In the 'QUERY from question' setting (Sec. 3.3) the Ctrl+F action is restricted to a single token drawn from the question and deterministically jumps to the next sentence containing that token (Sec. 3.2). With a 20-step budget, a trivial policy that issues Ctrl+F for each question token in order, or repeatedly takes next in easy mode, could plausibly produce F1 scores comparable to the DQN rows (e.g., 0.524 hard-mode iSQuAD, 0.352 iNewsQA) without any learned search strategy. The paper also omits a stop-at-first-sentence baseline and a full-document QANet upper bound, so the reported F1 cannot be decomposed into (a) the answerer's quality and (b) the agent's learned search behavior. Since F1info is consistently much higher than F1 (Table 3), the difference is almost entirely the search policy, which is exactly the component that needs a control. Section 6 candidly calls the experiments preliminary, but the missing controls still leave the central behavioral claim underdetermined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes iMRC, a reformulation of static extractive MRC datasets (SQuAD v1.1 and NewsQA) as interactive, partially observable environments. A document is split into sentences, only a small window or memory of sentences is observable at a time, and the agent must issue commands (previous, next, Ctrl+F QUERY, stop) to reveal further text before producing an extractive answer. The authors define easy and hard modes, three QUERY-generation settings, and a POMDP formulation. As a baseline, they adapt QA-DQN with a QANet-style answerer and train it with DQN and A2C, with optional memory queues and a 'sufficient information' reward. The experiments report training curves and test F1 and F1info scores for iSQuAD and iNewsQA across these settings, and the paper discusses trends such as easy mode outperforming hard mode, DQN generally outperforming A2C, and larger memories helping.","tokens_in":12590,"tokens_out":4612,"duration_ms":50961,"significance":"If the empirical claims are supported, this is a useful contribution to the study of information-seeking behavior in NLP: it provides a simple, general recipe for converting existing MRC datasets into interactive benchmarks, releases the data and code publicly, and formulates a clean RL baseline. The evaluation uses held-out test splits of standard datasets, and the sufficient-information reward is used only as reward shaping, not as the evaluation metric, so there is no circularity in the main performance measure. The work also honestly discusses the preliminary nature of the experiments. However, the central behavioral claim that the agent 'seeks relevant information through sequential decision making' is underdetermined by the current experiments because no non-learning controls, no full-document QA upper bound, and no variance estimates are reported.","major_comments":[{"comment":"The central claim that the agent learns to seek relevant information is not supported without non-learning control policies. In easy mode, the next action deterministically scans sentences within the 20-step budget; in hard mode, a policy that issues Ctrl+F for question tokens in order is available. None of these trivial baselines appears, so the reported DQN F1 scores (e.g., 0.524 hard-mode iSQuAD, 0.352 hard-mode iNewsQA) cannot be decomposed into the answerer's contribution and the learned search policy. I ask for random-action, first-sentence-only, stop-at-first-sentence, and exhaustive-scan baselines to be added.","section":"Section 5.2, Table 3"},{"comment":"The interpretation that information gathering is the main bottleneck relies on F1info, but there is no full-document QANet upper bound and no control for the answerer's training distribution. The answerer is trained only on terminal observations that contain the ground-truth answer, yet at test time it must produce a span from arbitrary terminal observations, including those without sufficient information. This train/test mismatch could itself explain much of the gap between overall F1 and F1info, and it should be addressed by reporting a full-document baseline, a stop-at-first-sentence baseline, or an answerer trained on all terminal observations.","section":"Section 5.2, Table 3 and Section 4.3.3"},{"comment":"All experimental results are from single runs, with no multiple seeds, error bars, or significance tests. Differences such as 0.524 versus 0.551 in hard-mode iSQuAD, or 0.352 versus 0.367 in hard-mode iNewsQA, may be within noise. The paper should report variance across seeds or at least state the number of runs used, so that the claimed trends (memory size, query source, DQN over A2C) can be assessed.","section":"Section 5.2, Table 3"}],"minor_comments":[{"comment":"Easy mode does not specify what Ctrl+F does when there is no next occurrence of the query, or whether the current sentence is revisited if it already contains the query; this should be stated explicitly for reproducibility.","section":"Section 3.2"},{"comment":"F1info is defined only in prose; the caption should state formally whether it is a conditional F1 computed only over episodes whose terminal memory contains the answer, or an upper-bound oracle measure.","section":"Section 5.2 and Table 3"},{"comment":"The training curves show no error bars or indication of the number of runs; please clarify whether these are single runs and whether the trends are stable across seeds.","section":"Figures 2 and 3"},{"comment":"The statement that supervised learning 'requires human labeled or heuristically generated trajectories' and that generalization can be limited is plausible but is presented as fact without experimental support; it should be softened or paired with a reference that demonstrates the limitation for the cited methods.","section":"Section 2, Related Works"},{"comment":"The paragraph on scaling to web-level QA via hyperlink-click actions is appropriately framed as a hypothesis, but the abstract's phrasing 'We believe that this setting can contribute in scaling models to web-level QA scenarios' is consistent with that framing; consider making the abstract equally explicit that web-level scaling is not demonstrated in this paper.","section":"Section 6, Discussion and Future Work"}],"recommendation":"major_revision","confidential_remarks":"The paper presents a timely and reusable task formulation, and the public release of code and data is a real strength. The missing non-learning baselines and absence of variance reporting are the main obstacles: without them, the central behavioral claim is not yet established. I do not see grounds for rejection because the issues are addressable with additional experiments and clearer reporting within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, the one thing to know: the contribution here is the task and the public datasets, not the agent results. iMRC turns SQuAD and NewsQA into partially observable interactive environments where an agent issues commands (next, previous, Ctrl+F query, stop) to reveal sentences before answering. The occlusion idea is simple and general, and the released iSQuAD/iNewsQA plus code give the community a concrete starting point. That is a real contribution, extending QAit from synthetic text games to natural language.\n\nThe paper also does a few things well. The experiment design separates F1 when the answer is in view (F1info) from overall F1, so you can see the bottleneck is search, not span extraction. The ablation across easy/hard mode, query source, memory size, and DQN vs A2C is sensible, and the trends are consistent: easy beats hard, DQN generalizes better than A2C, more memory usually helps. The authors are candid in Section 6 that the experiments are preliminary.\n\nThe soft spot is exactly what the stress-test flags: no non-learning baselines in Table 3. In easy mode (avg 5 sentences, 20-step budget) repeatedly hitting next would reveal the whole document. In hard mode, cycling Ctrl+F over question tokens could plausibly match the DQN F1s. Without a random-action control and a first-sentence baseline, the abstract's claim that the agent 'learns to seek relevant information through sequential decision making' is underdetermined. There is also no full-document QANet upper bound, so we cannot decompose F1 into answerer quality versus search policy; and no multiple seeds or error bars, which matters for RL results. The transfer to web-level QA is asserted, not demonstrated, and the single-token Ctrl+F query is a long way from that vision.\n\nNone of this sinks the paper. The task formulation stands on its own and the baseline is a reasonable first attempt. But the empirical support for the behavioral claim needs work. I would send it to review, and the referee should ask for trivial baselines, an upper bound, seeds, and ideally a small open-domain or multi-document demo before the transfer claim appears.","headline":"The paper's real contribution is the new interactive MRC task and public datasets; the RL agent's search behavior is plausible but under-proven because the experiments lack non-learning baselines and variance reporting.","tokens_in":13116,"tokens_out":2365,"would_cite":true,"duration_ms":22132,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that reframing reading-comprehension datasets as interactive, partially observable environments allows a reinforcement-learning agent to learn to seek relevant information through sequential decisions.","keywords":["machine reading comprehension","interactive environment","partial observability","reinforcement learning","information seeking","question answering","SQuAD","NewsQA"],"falsifier":"Run an agent trained on iMRC against a real open-domain QA setup in which it must issue search commands over a large corpus, and compare it with a non-interactive baseline that receives a retrieved passage: if the interactive agent gains nothing or its performance does not track its iMRC F1info, the transfer claim fails. A sharper within-paper test would ablate the sufficient-information reward and check whether the sequential policy still learns to seek.","tokens_in":12132,"feed_emoji":"🔍","tokens_out":8049,"duration_ms":72902,"temperature":0.7,"pith_summary":"The paper argues that reading-comprehension benchmarks are too static: the supporting document is fully visible, so models can answer by spotting surface similarities between question and text, and they never learn to gather information the way a person would. To fix this, the authors convert existing datasets into interactive, partially observable environments called iMRC: a document is split into sentences, almost all of them are hidden, and an agent must issue commands — move to the previous or next sentence, search with a Ctrl+F query, or stop — to reveal glimpses of the text before answering. The task is formalized as a partially observable Markov decision process, and a baseline agent couples a reinforcement-learning action generator with a QANet-style answerer. Trained on the resulting iSQuAD and iNewsQA benchmarks, the agent learns to seek the sentences that contain answers, and its errors are dominated by information gathering rather than by answer extraction. If this reframing transfers, it offers a route toward MRC systems that can search large, partially observable knowledge sources such as the web.","feed_headline":"Reading agents learn to seek answers when text is hidden","feed_subtitle":"A partially observable reading game reframes SQuAD and NewsQA, exposing search as the bottleneck.","key_machinery":"The load-bearing mechanism is the iMRC environment itself: a POMDP in which each document is split into sentences, all but one are hidden, and the agent reveals text by issuing one of a small set of commands (previous, next, Ctrl+F QUERY, stop). The agent's encoder produces an attention-aggregated representation of the question and current observation; an action generator scores action and query tokens, and a QANet-style answerer produces head and tail pointers over the observed text when the agent stops. Two design elements carry the learning: a bounded memory queue that stores recently seen sentences so the agent can reason over context larger than one sentence, and a shaped 'sufficient information reward' that pays the agent for stopping with the ground-truth answer substring in view. These convert the sparse, delayed reward of question answering into a signal that guides exploration.","core_discovery":"The paper's central claim is that sequential information seeking can be learned and measured in natural-language MRC by occluding documents and letting the agent act. Concretely, iMRC withholds all but the first sentence of a paragraph, gives the agent a small command set, and asks it to stop when it has enough context to answer; easy mode adds previous/next navigation, hard mode forces the agent to use Ctrl+F queries alone. The authors show that a baseline agent trained with DQN and A2C on this POMDP improves its answer F1 as it learns to find the answer sentence, that scores are consistently higher when the terminating observation actually contains the answer (the F1info measure), and that larger action spaces and harder navigation modes degrade performance. The discovery is therefore that the hard part of partially observable comprehension is deciding where to look, not reading what is found, and that a simple word-level search action can be acquired through reinforcement learning.","pith_inferences":["Editorial inference: the paper's own numbers imply that the shaped 'sufficient information reward' may be doing much of the work; a testable ablation that removes that reward and compares learning curves would isolate the contribution of the sequential policy.","Editorial inference: if the interactive framing transfers, the natural next step — which the paper names as future work — is replacing single-token vocabulary queries with latent vector queries and fuzzy matching, which would bring the setup closer to learned information retrieval and could sharply reduce exploration cost.","Editorial inference: because iMRC is built on documents that are short enough for a 20-step budget to cover, the current benchmarks may understate the difficulty of web-scale search; converting long-document or multi-hop datasets such as HotpotQA would be a stronger stress test of the information-seeking claim.","Editorial inference: a comparison of an iMRC-trained agent against the same answerer given the full document would quantify how much performance is lost to active search, and whether the sequential policy is worth its added complexity."],"forward_implications":["Information gathering, not answer extraction, is the bottleneck in partially observable reading: across nearly all settings the reported F1 is lower than F1info, the score an agent achieves when it stops on a sentence containing the answer.","Action-space size controls difficulty: agents do best when the query token is drawn from the question (a small action space) and worst when it is drawn from the full vocabulary, and they do better in easy mode with navigation commands than in hard mode with Ctrl+F only.","Exploration method matters for generalization: DQN-like explicit exploration generalizes to held-out questions markedly better than A2C's implicit sampling, suggesting that exploration strategy is a primary determinant of interactive MRC success.","Larger memory helps: keeping three or five observed sentences in a queue almost always improves test F1 over a single-slot memory, since the environment becomes closer to fully observed.","The reframing is dataset-agnostic: because it only requires splitting documents into sentences and hiding them, nearly any extractive MRC dataset can be converted into an interactive information-seeking benchmark without new annotation."],"supporting_citations":[{"why":"supplies the SQuAD dataset that iSQuAD is built from, defining the extractive QA task and answer-span format that iMRC reframes.","marker":"(Rajpurkar et al., 2016)"},{"why":"supplies the NewsQA dataset that iNewsQA is built from, providing the harder, longer-document test bed.","marker":"(Trischler et al., 2016)"},{"why":"contributes the QA-DQN architecture and the interactive question-answering framework that the baseline agent is adapted from.","marker":"(Yuan et al., 2019)"},{"why":"contributes the QANet answerer, specifically the transformer stacks used to predict answer head and tail positions.","marker":"(Yu et al., 2018)"},{"why":"provides Rainbow, the deep Q-learning algorithm whose extensions (prioritized replay, noisy nets, multi-step returns) the DQN agent uses.","marker":"(Hessel et al., 2017)"},{"why":"provides A2C, the actor-critic algorithm used as the alternative training strategy for the action generator.","marker":"(Mnih et al., 2016)"},{"why":"documents the shallow-matching failure mode (e.g., predicting the only date in the paragraph for when-questions) that motivates the interactive reframing.","marker":"(Sugawara et al., 2018)"},{"why":"provides the earlier program of synthetic information-seeking tasks that iMRC extends to natural-language MRC.","marker":"(Bachman et al., 2016)"}],"fun_headline_variants":["Occluded text teaches reading agents to hunt for answers","Information seeking agents learn to navigate hidden documents","Partially observed text makes AI learn search strategies","Reading bots master peek-a-boo to answer questions","Interactive reading games expose search as key skill"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results rest on the assumption that the synthetic interactive environment — deterministic moves, one-word queries, sentence-sized glimpses, a 20-step budget — is a faithful stand-in for real-world information seeking; if that proxy is too artificial, progress on iMRC would not demonstrate that models can scale to web-level question answering.","fun_headline_variants_meta":{"raw":{"variants":["Occluded text teaches reading agents to hunt for answers","Information seeking agents learn to navigate hidden documents","Partially observed text makes AI learn search strategies","Reading bots master peek-a-boo to answer questions","Interactive reading games expose search as key skill"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1213,"prompt_tokens":866,"completion_tokens":347,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":482,"completion_tokens_details":{"reasoning_tokens":275}},"tokens_in":482,"tokens_out":347,"duration_ms":4256,"temperature":1.0,"reasoning_tokens":275,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:42:30.291778+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an agent trained on iMRC against a real open-domain QA setup in which it must issue search commands over a large corpus, and compare it with a non-interactive baseline that receives a retrieved passage: if the interactive agent gains nothing or its performance does not track its iMRC F1info, the transfer claim fails. A sharper within-paper test would ablate the sufficient-information reward and check whether the sequential policy still learns to seek.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"contributes the QA-DQN architecture and the interactive question-answering framework that the baseline agent is adapted from."},{"cited_title":"What Makes Reading Comprehension Questions Easier?","cited_arxiv_id":"1808.09384","evidence_quote":"documents the shallow-matching failure mode (e.g., predicting the only date in the paragraph for when-questions) that motivates the interactive reframing."}],"review_version":1}