{"id":"9e1d5e83-c17d-42cd-803a-e77c46629b3d","arxiv_id":"1908.10909","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"QAit turns question answering into an interactive text-game task, and the paper's baselines show current agents cannot generalize beyond memorized games, while humans can.","lead":"This paper introduces QAit, a dataset of text-based games where an AI must explore rooms and manipulate objects to answer questions such as 'Is ghargh edible?'. It tests whether reading systems can gather information through interaction, and shows current neural agents mostly fail while humans succeed.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never verifies that TextWorld observations omit attribute values; if templates leak adjectives like 'sharp' or 'closed', attribute questions can be answered without the interaction that the central procedural-knowledge claim depends on.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern, and I agree it is the right one. It targets the premise that distinguishes QAit from static MRC: that attribute answers must be discovered through interaction rather than read off the observation. The paper's public code and on-the-fly generation are genuine strengths, and the full-information control (Sec. 4.4) supports the claim that the difficulty is in the interactive setting. But without an audit of observation text, the attribute results are ambiguous: if TextWorld templates leak attribute adjectives or state descriptions, the benchmark still requires navigation and exploration, but it no longer tests the procedural object-interaction knowledge that the paper emphasizes. The small human baseline (3 participants per condition) and single-seed zero-shot selection are secondary concerns; they affect confidence in 'relative ease' but do not threaten the task definition as directly. Since the reader already issued a CONDITIONAL verdict, my stress-test does not move it. The proposed leakage check would either resolve the concern or force a re-scoped interpretation of the attribute question results.","tokens_in":18338,"tokens_out":5700,"duration_ms":65898,"concrete_test":"Use the public qait_public/TextWorld code to regenerate the 500 held-out attribute-question games (or a reproducible sample). For each game, take the first observation string in which the made-up object name appears; run a simple keyword/pattern classifier that predicts each of the nine Table 2 attributes from phrases in that observation alone (e.g., 'sharp', 'open', 'closed', 'diced', 'cooked', 'fried', 'edible', 'drinkable', 'on/in' constructions). Compare against ground-truth state attributes. If recall on any attribute is substantially above chance, leakage is confirmed. Then rerun the Table 4 attribute accuracy on the subset of games where the classifier cannot determine the answer; if any baseline rises above 0.55, the reported difficulty is partly a text-leakage artifact. If all recalls are at chance, the reader's concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that QAit tests procedural, interactive information seeking (Abstract; Sec. 2.4: 'On encountering an object, the player must interact with it to discover its attributes') depends on attribute answers being absent from observation text. Section 2.3.1 only says object names are replaced with made-up words; it does not say attribute-bearing adjectives or state descriptions are removed. TextWorld's templated output commonly includes such phrases ('a sharp knife', 'a closed oven', 'a diced potato', 'It is ...'), and Table 1 itself contains 'There is an open screen door' and 'a diced potato.' If any of the nine attributes in Table 2 are readable from the observation string, a model that has located the object can answer without executing eat/take/cut/cook. The paper provides no audit or ablation of observation strings, and Appendix D's heuristic attribute conditions (e.g., must try cut to know sharp) would be unnecessary for leaked attributes. Thus the reported attribute near-randomness could be caused by exploration/RL sample complexity rather than by the absence of declarative information, and the benchmark's procedural-knowledge interpretation for attribute questions is unsupported. This is not a consensus dispute; it is an unverified internal premise that can be checked directly.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces QAit, a text-based interactive question answering task built with TextWorld. An agent must issue text commands to navigate a partially observable environment and gather information needed to answer location, existence, and attribute questions. The authors propose a QA-DQN baseline with DQN, DDQN, and Rainbow variants, compare against a random baseline and human participants, and report that RL agents perform near chance on attribute questions across training settings while humans achieve high accuracy. The paper also includes a full-information control showing that a standard MRC model (Match-LSTM) answers all question types well when a concatenated observation document is provided, and a sufficient-information analysis for location questions.","tokens_in":18567,"tokens_out":4251,"duration_ms":44515,"significance":"If the central claims are supported, QAit is a valuable and reproducible benchmark for studying procedural knowledge and interactive information seeking, with a public dataset and code, on-the-fly generation of training games, and a held-out zero-shot evaluation. The paper's evidence that current deep RL agents fail to generalize on attribute questions is useful and well documented, and the full-information control is a fair diagnostic. However, the benchmark's procedural-knowledge interpretation for attribute questions rests on an unverified premise about observation-string leakage, and the human-ease claim is based on a very small sample. These issues materially affect the interpretation of the headline results, though they appear fixable with additional analysis rather than requiring a change of scope.","major_comments":[{"comment":"The claim that attribute answers require interaction is not verified against leakage in the generated observation strings. The paper states that object names are replaced with made-up words for attribute questions, but it does not report whether attribute-bearing descriptions (e.g., 'open' for openable, 'diced' for cuttable/processed, 'sharp' for sharp, or 'It is edible') are excluded from TextWorld's templated observations. Table 1 itself contains 'There is an open screen door' and 'a diced potato.' If any of the nine attributes in Table 2 is directly readable from the observation string, a static reading model that has located the object could answer attribute questions without executing eat/take/cut/cook, and the reported near-random attribute accuracies would reflect exploration and RL sample complexity rather than the absence of declarative information. The heuristic conditions in Appendix D, which assume the agent must attempt cut/eat/take to determine attributes, would be unnecessary for leaked attributes. Please provide an audit of observation strings (e.g., the frequency of attribute-relevant adjectives in observations versus command feedback) or an ablation in which a non-interactive baseline reads only the concatenation of observation strings (without command feedback) and attempts to answer attribute questions.","section":"Sec. 2.3.1, Sec. 2.4, App. D"},{"comment":"The human baseline is measured on only 3 participants per condition and reported as a point estimate with no variance or significance test. With 3 participants, the random-map attribute accuracy of 0.750 is statistically indistinguishable from chance, and the perfect 1.000 scores carry wide error bars. The abstract's claim that humans solve QAit 'with relative ease' is load-bearing for the benchmark's validity and should be supported by a larger participant pool with per-participant accuracy distributions and a test against chance performance.","section":"Sec. 3.2, Table 4"},{"comment":"Zero-shot test performance is reported for only a single seed per condition: the model with the random seed yielding the highest training accuracy is selected for evaluation. This selection procedure biases the reported test accuracy upward and makes the near-chance attribute results more difficult to interpret, since the reported number may be the best of three rather than a typical run. Please report mean test accuracy plus standard error (and, ideally, all seed results) for the held-out sets under each training setting.","section":"Sec. 4, Tables 4, 9, 10"}],"minor_comments":[{"comment":"The statement that answers are 'one word' is slightly misleading for location questions, where the answer can be any word in an observation string, but must be a container or holder name; this could be clarified in the text.","section":"Sec. 2.4"},{"comment":"The full-information setup relies on a 'heuristic trajectory' that is guaranteed to observe sufficient information, but the construction of this trajectory is not described in the main text; please specify how it is generated and whether it includes only observations or also command feedback.","section":"Sec. 4.4"},{"comment":"The statistics in Table 3 are averages over 10,000 games, but no variance or range is reported for the random-map settings; adding interquartile ranges or standard deviations would help readers gauge the spread of game difficulty.","section":"Table 3"},{"comment":"The limitation paragraph acknowledges templated English but does not mention the potential leakage of attribute information in observations; this should be explicitly listed as a limitation and addressed by the audit suggested above.","section":"Sec. 6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid, well-written benchmark proposal with a public release and reasonable baselines. The main risk is the unverified premise that attribute values are not recoverable from the observation text; if the audit or ablation requested in the major comments reveals leakage, the paper's central procedural-knowledge interpretation would need substantial revision. The small human sample is a secondary but real concern. I think the paper deserves a major revision rather than rejection, because the issue is directly testable and the authors have the tools and access to TextWorld to resolve it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about this paper. First, QAit is a genuinely new benchmark that fills a gap: text-based interactive QA with procedurally generated games, on-the-fly training data, and a held-out zero-shot split. The central difficulty result looks solid across the board. Second, the paper's claim that attribute questions require interactive discovery is not actually checked. I think the stress-test note lands: nothing in the paper rules out attribute leak in the observation text. Table 1 itself shows \"closed oven\" and \"diced potato\" — state descriptions that imply openable and cuttable. If such phrases appear in observations, a model that has located the object can answer some attribute questions without performing eat/take/cut/cook. That weakens the procedural-knowledge interpretation for at least a subset of attributes.\n\nWhat is actually new: the integration of TextWorld, on-the-fly generation, and question answering is a real contribution. The zero-shot split with 500 held-out games is a reproducible evaluation standard. The full-information control (Match-LSTM gets over 90% accuracy) cleanly shows the bottleneck is information gathering, not reading final text. The \"sufficient information bonus\" is a nice diagnostic: it exposes that agents overfit to question-answer mappings on small training sets. The code and data are released, which I take as serious evidence.\n\nSoft spots, in proportion. The leak issue is real and directly checkable; the paper should audit the observation templates or explain why attribute adjectives are absent. Until then, \"interact to discover attributes\" is an unverified premise. The human baseline is thin: three participants per cell, no variance, and the random-map attribute score is 0.75. That is a pilot, not strong evidence of \"relative ease.\" Test accuracy also comes from a single seed chosen by training accuracy; tolerable for a benchmark paper, but worth stating as a caveat.\n\nNone of this is fatal. Even if some attributes leak, the task still demands locating objects in a partially observable world, and the location/existence results show the agents' failure is not only about attribute discovery. The benchmark is a useful, controlled testbed for exploration and generalization in text-based RL.\n\nWho this is for: people working on text-based RL, embodied QA, and MRC generalization. It deserves a serious referee, with the leak audit and human variance as the main revision requests. I would send it to review.","headline":"A useful, reproducible benchmark for interactive text-based QA, but the procedural-knowledge claim for attribute questions is under-verified because observation strings may leak attribute values.","tokens_in":19123,"tokens_out":3573,"would_cite":true,"duration_ms":35676,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that machine reading comprehension is missing its interactive, information-seeking component, and proposes QAit, a text-based benchmark where an agent must explore a generated world before answering questions about the…","keywords":["Question Answering with Interactive Text","procedural knowledge","text-based games","machine reading comprehension","deep reinforcement learning","information seeking","generalization"],"falsifier":"Sample a batch of QAit games and scan the raw observation strings for attribute-denoting words (sharp, edible, drinkable, cookable, openable, portable, heat source) associated with the question's target object. If such words appear before any interaction, a static reading model could answer attribute questions above chance, and the reported human-machine gap would reflect surface leakage rather than a procedural reasoning deficit.","tokens_in":18144,"feed_emoji":"🎮","tokens_out":8492,"duration_ms":74524,"temperature":0.7,"pith_summary":"This paper introduces a question-answering benchmark in which a model must explore a partially observable text-based environment—moving, taking, eating, and cutting objects—before answering questions about where objects are, whether they exist, and what attributes they have. It argues that standard machine reading benchmarks omit the information-seeking component of comprehension, because static documents place all necessary text in the reader's hands and reward shallow word matching. The central empirical claim is that current deep reinforcement learning agents solve these interactive tasks barely better than chance, especially on attribute questions, while human players answer nearly perfectly. If true, the real bottleneck in reading comprehension is not extracting declarative facts from text but learning the procedure for gathering evidence, and QAit provides a controlled setting in which that procedure can be studied.","feed_headline":"RL agents can't answer questions that require exploring a text world","feed_subtitle":"Interactive question answering separates reading from evidence gathering; current agents can't do the gathering.","key_machinery":"The central object is QAit (Question Answering with Interactive Text), a generated corpus of text-based games with three question types—location, existence, and attribute—and two difficulty levels, one with a fixed map and one with a randomly sampled map. The mechanism that carries the argument is the interaction loop: at each turn the agent receives an observation string and feedback from its previous command, chooses an action, modifier, and object from small lexicons, and finally issues a wait command to stop and answer. Because questions are generated with access to the underlying game state, ground-truth answers are known, and a 'sufficient information bonus' measures whether the agent has actually seen the evidence needed to answer. For location questions that means observing the queried object; for attribute questions it means executing a small set of probing commands, such as trying to cut an object to test sharpness. This bonus is the diagnostic that exposes memorization, since a model can be correct on training questions while having gathered no information at all.","core_discovery":"The paper's central discovery is that the interactive information-gathering step, not the reading step, is where current models fail. When the same game is flattened into a static document that contains all observations a successful trajectory would see, an off-the-shelf reading model answers 86 to 93 percent of test questions across the three types. When the model has to issue text commands to reveal those observations itself, accuracy on the same question types falls to roughly the level of random guessing, and no baseline significantly beats chance on attribute questions even when trained on unlimited procedurally generated games. Humans who play the games score near perfect on location and existence questions and 75 percent on attribute questions. The failure mode is visible in the sufficient-information bonus: agents can reach 100 percent training accuracy while their bonus stays near zero, meaning they are memorizing question-answer pairs without ever acquiring the information the question requires.","pith_inferences":["Beyond the paper: the random made-up object names are the likely load-bearing device that forces interaction on attribute questions; a direct ablation that swaps real names into the observation text would test whether the measured difficulty is procedural or an artifact of naming.","Beyond the paper: the sufficient-information bonus could serve as a predictive diagnostic, since the paper's results show it diverges from training accuracy before test performance does; future agents could be selected or early-stopped on this signal.","Beyond the paper: because the games are generated from a grammar, the same benchmark could be extended with multi-step cooking or tool-use chains, which would make the procedural knowledge required more compositional and could be used to measure how agent capability scales with procedure length."],"forward_implications":["Models that score well on static reading benchmarks should not be credited with comprehension in settings where evidence must be actively sought; QAit separates the extraction skill from the search-and-gather skill.","The near-chance attribute-question performance implies that current deep reinforcement learning agents have not learned the information-seeking subroutine that human players use, even with unlimited generated training games.","The strong full-information result implies the difficulty is the interaction procedure itself, not the reading or answer module, so progress on QAit requires advancing the information-gathering policy rather than the encoder.","The memorization success on small training sets implies that finite static corpora cannot induce procedural knowledge; on-the-fly generation is necessary, but the paper shows it is not sufficient with current algorithms."],"supporting_citations":[{"why":"Supplies the text-based game generator that defines the environment dynamics and the templated observation text on which QAit is built.","marker":"Côté et al. (2018)"},{"why":"Establishes the deep reinforcement learning approach to text-based games that the QA-DQN agent extends.","marker":"Narasimhan et al. (2015)"},{"why":"Provides the encoder and answer module architecture used in the QA-DQN baseline.","marker":"Yu et al. (2018)"},{"why":"Represents the static-document MRC setting the paper argues rewards word matching rather than information seeking.","marker":"Rajpurkar et al. (2016)"},{"why":"Supplies the philosophy of controlled bAbI-style tasks that justifies templated language as a way to isolate reasoning behavior.","marker":"Weston et al. (2015)"},{"why":"Motivates the use of randomly drawn made-up object names to prevent agents from memorizing attribute values.","marker":"Anand et al. (2018)"},{"why":"Provides the Rainbow DQN algorithm whose near-chance performance on attribute questions anchors the claim that current RL agents cannot solve QAit.","marker":"Hessel et al. (2017)"},{"why":"Supplies the Match-LSTM reading model whose high accuracy on full-information documents isolates the interaction bottleneck.","marker":"Wang and Jiang (2016)"}],"fun_headline_variants":["Question answering fails when models must gather info themselves","Interactive text Q&A: humans excel, RL agents flounder","Reading isn't enough: RL agents fail interactive Q&A","Text games expose RL agents' inability to seek answers","Exploration is the bottleneck: RL agents can't answer interactive questions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the observation strings never state the attribute values being asked about—for instance, a knife is never described as sharp in the room text—so an agent truly must perform actions to discover the answer; the paper provides no analysis or ablation confirming this.","fun_headline_variants_meta":{"raw":{"variants":["Question answering fails when models must gather info themselves","Interactive text Q&A: humans excel, RL agents flounder","Reading isn't enough: RL agents fail interactive Q&A","Text games expose RL agents' inability to seek answers","Exploration is the bottleneck: RL agents can't answer interactive questions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000898,"raw_usage":{"total_tokens":3838,"prompt_tokens":882,"completion_tokens":2956,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":2875}},"tokens_in":498,"tokens_out":2956,"duration_ms":22801,"temperature":1.0,"reasoning_tokens":2875,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:31:07.380021+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample a batch of QAit games and scan the raw observation strings for attribute-denoting words (sharp, edible, drinkable, cookable, openable, portable, heat source) associated with the question's target object. If such words appear before any interaction, a static reading model could answer attribute questions above chance, and the reported human-machine gap would reflect surface leakage rather than a procedural reasoning deficit.","supporting_citations":[{"cited_title":"M., van Merriënboer, B., Joulin, A., and Mikolov, T","cited_arxiv_id":null,"evidence_quote":"Supplies the philosophy of controlled bAbI-style tasks that justifies templated language as a way to isolate reasoning behavior."}],"review_version":1}