{"id":"5e63f42a-bc2c-4ec8-968f-8b80979925ac","arxiv_id":"2505.11893","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"RLAP uses a trained Q-value estimator to adaptively order LLM subtask execution and reports accuracy gains on MRC, IE, and text completion benchmarks.","lead":"This paper trains a small reinforcement-learning model to choose the order in which a large language model performs subtasks, and tests it on reading comprehension, information extraction, and sentence ordering. The method improves results on several benchmarks, but the gains are small on some datasets and the core idea extends the authors' earlier RL4IE system.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MRC results conflate learned reading order with the multi-step sentence protocol; the missing fixed-order ablation leaves the core adaptive-planning claim untested for MRC.","rationale":"Good-faith reading: RLAP is a coherent DQN-based planner; the IE results with RLAP-RL versus RLAP-random/sequence are the strongest support for the adaptive-order mechanism, and the S2P/SFB results show a workable ordering module. The central claim, however, is that the Actor's Q-values capture linguistic features that determine the best next subtask, and that this is what improves LLM performance across MRC, IE, and STC. The load-bearing weak spot is the MRC experiment: because the LLM reads all sentences before answering, the only treatment is sentence order. Without a fixed-order control under the same multi-step protocol, the Table 2 gains cannot be attributed to the learned ordering. This is not an internal inconsistency; it is an unisolated variable. The reader already flagged this as the second premise of the weakest assumption, so my read agrees partially, focusing on the MRC control rather than the general Q-value transfer question. The concrete test above would settle it. The verdict should remain conditional: pending the fixed-order ablation and significance testing, the support from IE is real, but the general claim across task types is not yet established.","tokens_in":16740,"tokens_out":6185,"duration_ms":67691,"concrete_test":"Run the Section 4.1 MRC evaluation with the identical multi-step RLAP prompt and LLM but with sentences presented in the original document order (RLAP-sequence), plus a second control that shuffles sentences according to the Actor's Q-values but with Q trained on random rewards. Compare these against RLAP and IO on SQuAD2.0 and RACE-H. If original-order accuracy is statistically indistinguishable from RLAP, the adaptive order contributes nothing on MRC and the central claim must be restricted to IE/STC; if original-order is lower by the same margins as in Table 2, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 (Table 2) evaluates RLAP for MRC against a single-pass IO prompt and ToT-BFS, but it never runs the same multi-step sentence-by-sentence protocol with sentences in the original document order or any fixed order. This omission is load-bearing because, in this setup, the LLM answers only after every sentence has been read; the Actor's only effect is the permutation in which sentences appear in the final prompt. Any performance difference between RLAP and IO/ToT-BFS could therefore come from the stage-wise protocol itself (structured prompt, 'processed/candidate' framing, recency effects from sentence segmentation) rather than from the learned Q-order. The IE experiments include exactly the needed control, RLAP-sequence versus RLAP-RL (Table 3), and those results support adaptive ordering; the MRC table has no equivalent. Since the paper's headline claim is that a lightweight Actor trained on linguistic features selects better subtask orders across tasks, an MRC result that does not isolate the order variable cannot carry that claim. A fixed-order baseline would settle whether the 0.06-3.92 percentage-point gains on SQuAD2.0, CMRC18, C3-mix, and RACE-H reflect the Actor's choices or merely the multi-step prompting format.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RLAP, a reinforcement-learning framework for adaptive multi-step planning in LLM-based NLP tasks. The task is modeled as an MDP in which a lightweight Actor network, trained with deep Q-learning, selects the next subtask to execute by estimating Q-values over natural-language state-action sequences, while the LLM acts as the task executor. The framework is instantiated for machine reading comprehension (MRC), information extraction (IE), and sentence-level text completion (STC). Experiments on multiple datasets report accuracy and F1 gains over fixed-order baselines (ChatIE, CoT), LLM-planning baselines (ToT-BFS), and the authors' earlier RL4IE, along with ablations (RLAP-random, RLAP-sequence) for IE.","tokens_in":16999,"tokens_out":6348,"duration_ms":60620,"significance":"If validated, RLAP is a useful contribution: it provides a general recipe for injecting a small, trainable policy into LLM pipelines without fine-tuning the LLM, and the IE ablations in Table 3 give credible evidence that the learned ordering, not just the multi-step protocol, drives performance. The attention to state construction and reward design for three task types is valuable. However, the central claim that the Actor adaptively selects better subtask orders is not fully supported because the MRC experiments lack a fixed-order control under the same multi-step protocol, and because the quantitative results are reported without error bars or significance tests despite many small gains. These issues are addressable and do not undermine the plausibility of the underlying idea.","major_comments":[{"comment":"The MRC experiments never run the same sentence-by-sentence protocol with sentences in original or any fixed order. Since the LLM answers only after all sentences have been read, the sole effect of RLAP on the final answer is the sentence permutation. The comparison against IO and ToT-BFS therefore conflates the learned ordering with the multi-step protocol itself (the 'processed/candidate' framing, the final answer prompt, and recency effects from sentence segmentation). Add an RLAP-sequence baseline that uses the identical prompt template and LLM execution but processes sentences in the original document order, and an RLAP-random baseline, as done for IE in Table 3. Without this control, the reported +0.06 to +3.92 percentage-point gains on SQuAD2.0, CMRC18, C3-mix, and RACE-H cannot be attributed to the Actor's adaptive planning.","section":"Section 4.1, Table 2"},{"comment":"All quantitative results come from a single training run and a single evaluation pass, with no standard deviations, confidence intervals, or significance tests. Several headline improvements are small (CMRC18 general +0.06 percentage points, RACE-H general +0.22, Mistral-7B on NYT10 +1.4, ACE05 +0.7), and these could easily fall within run-to-run variance of the LLM or the RL training. To support the language 'significantly outperforms' and 'remains stable,' report means and variances over multiple random seeds (and multiple LLM decoding runs if sampling is used), and apply paired bootstrap or similar tests on the test sets. State whether decoding is greedy or sampling-based.","section":"Section 4.1-4.3, Tables 2-4 and Figure 5"}],"minor_comments":[{"comment":"In line 24, the loss is written as L(θ) = (y_j − Q(S_t, a_t; θ))^2, but the sampled transition uses state S_j, so it should be Q(S_j, a_j; θ).","section":"Section 3.3, Algorithm 1"},{"comment":"The sentence 'treating each subtask as an action that can and can only be executed once..' contains a duplicated period; also, the wording 'can and can only be executed once' is clearer as 'that can be executed at most once' or 'that must be executed exactly once.'","section":"Section 3.2"},{"comment":"The statement 'we set the input of LLMs to be plain prompts without chain-of-thought (CoT) and in-context examples' is ambiguous because the IO and CoT baselines include three in-context examples. Clarify that this applies to the RLAP executor's prompts, not to the baseline configurations.","section":"Section 4, first paragraph"},{"comment":"The 'Improvement' row reports relative gains over RL4IE (e.g., (77.27−58.83)/58.83 ≈ 31.3%), but the caption does not state whether these are relative or absolute, or against which baseline. Please specify.","section":"Table 3 caption"},{"comment":"The task setup says 'we select 2-3 questions for each context, treating a set of question-answer-context as a sample.' For multiple-choice datasets, clarify what 'answer' means in this sample definition and how accuracy is aggregated when a sample contains multiple questions.","section":"Section 4.1"},{"comment":"The claim that a single linear projection of a fixed-size encoded state-action sequence captures the linguistic features needed for ordering rests on an assumption that is only indirectly tested through final accuracy. Adding an analysis of the learned Q-values (e.g., correlation with ground-truth optimal order on a development set, or qualitative examples of selected orderings) would make the mechanism more transparent.","section":"Section 3.3 and Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper shares substantial methodology with the authors' prior RL4IE (arXiv:2406.11455), which already performs adaptive RL-based slot ordering for IE. The novelty of RLAP over RL4IE is not sharply delineated: the main additions appear to be the use of an LLM as the environment, episode-level rewards, and extension to MRC and STC. The authors should clarify what is new relative to that line in both the introduction and the related-work section. Also, the MRC missing-control issue is, in my view, the most serious technical gap and should be resolved before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Tom,\n\nHere is my read on RLAP. The paper applies the authors' existing adaptive-subtask-ordering idea (RL4IE) to MRC and sentence text completion, and adds an episode-level reward on top of the stepwise reward. That is a legitimate extension, not a new paradigm, but it is empirically useful.\n\nThe strongest evidence is the IE section. The ablation comparing RLAP-RL to RLAP-random and RLAP-sequence is the right control, and the RL version wins consistently across five datasets and two LLMs. That tells me the learned ordering is doing real work, and the large gains on HacRED (77.27 vs 58.83 for RL4IE with Qwen2.5-14B) are impressive. I also credit the authors for using exact-match F1 and for keeping the LLM frozen.\n\nThe soft spot is the MRC experiment, and the stress-test note is right: there is no fixed-order multi-step baseline. In the MRC setup, the LLM only answers after reading all sentences; the Actor's only influence is the permutation of the sentences. The comparison against IO and ToT-BFS cannot separate the effect of a learned reading order from the effect of the stage-wise sentence-by-sentence protocol itself (with its 'processed/candidate' framing and recency effects). On CMRC18, RLAP beats IO by 0.06 points; that is noise, not evidence. To claim that the Actor improves MRC, they need to run the same protocol with sentences in original document order and with a random fixed order. The IE table has this control; the MRC table does not.\n\nAnother general issue: everything is single-run with no error bars or significance tests, so most of the 'improvements' are not statistical claims. The abstract says 'significantly outperforms' but the numbers on several datasets do not justify that word.\n\nThe framework itself is coherent. The Q-value approximation over flattened state-action text is a reasonable design choice, though the paper never directly validates that the learned Q-values generalize; it checks only final task accuracy. That is a limitation, but not a fatal one.\n\nNovelty is incremental relative to RL4IE and to adaptive ordered IE [13], and the authors do cite both. The citation pattern is honest.\n\nThis will be useful to people working on RL-based planning for LLMs and on multi-step IE. Bottom line: it deserves a serious referee. I would send it to review with a request for revision: add the fixed-order MRC baseline, report variance or significance, and temper the claims. The IE results are worth publishing on their own.","headline":"A solid extension of the authors' RL4IE line with a strong IE ablation, but the MRC results don't isolate learned order from the multi-step protocol, so the central claim is only partly supported.","tokens_in":17549,"tokens_out":2834,"would_cite":true,"duration_ms":27336,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RLAP claims a lightweight Q-learning Actor can choose each subtask order from linguistic features, improving LLM accuracy without fine-tuning.","keywords":["reinforcement learning","large language models","multi-step planning","adaptive planning","Q-learning","Markov decision process","machine reading comprehension","information extraction"],"falsifier":"Take a held-out set of small contexts (4-6 sentences or slots), enumerate all possible subtask orders, run the frozen LLM on each order, and compare the accuracy of RLAP's chosen order with the best-order accuracy and with fixed and random orders. If RLAP does not approach the best order, or if random or fixed orders already match it, the claim that learned linguistic-feature Q-values drive the gains is refuted.","tokens_in":16535,"feed_emoji":"🧭","tokens_out":8689,"duration_ms":82318,"temperature":0.7,"pith_summary":"This paper proposes RLAP, a method for making multi-step LLM problem-solving adaptive to each task instance. It models a decomposable NLP task as an MDP, embeds a frozen LLM as the environment, and trains a lightweight Actor model to estimate Q-values for pairs of the current state and each candidate next subtask. At inference, the Actor picks the subtask with the highest Q-value, so the order in which a context is read, slots are extracted, or sentences are concatenated depends on the linguistic features of that instance rather than on a preset order or the LLM's own planning. The paper reports consistent accuracy gains over fixed-order and LLM-search baselines on machine reading comprehension, information extraction, and sentence-level text completion, with larger gains on complex instances. A sympathetic reader would take the central claim to be that order choice is a learnable, task-independent policy that can be decoupled from the LLM.","feed_headline":"A learned Actor model picks the best subtask order for LLMs","feed_subtitle":"Deep Q-learning reorders reading, extraction, and completion steps to lift accuracy, with no LLM fine-tuning.","key_machinery":"The load-bearing object is the Actor model: a pretrained language model plus a linear projection from a sequence representation to a scalar Q-value, written as $\\hat{Q}(S_t,a_t)=W h+b$. Its input is the flattened state dictionary concatenated with a candidate action, wrapped in special tokens. It is trained with double deep Q-learning, using a discounted target with a target network, experience replay, and epsilon-greedy exploration. The action space shrinks by one each step because every subtask is executed exactly once; rewards are either stepwise (matching ground truth at each substep) or episode-level (matching the final answer), with the discount factor set accordingly. This machinery converts linguistic features into a quantitative ordering criterion that does not depend on the LLM's planning ability.","core_discovery":"RLAP's central claim is that the optimal ordering of subtasks in a multi-step NLP task is a function of the instance's linguistic features, and that this function can be learned by a lightweight model without modifying the LLM. The paper models the solving process as an MDP in which states are dictionaries of task definition, original text, intermediate results, and requirements; actions are subtasks that can be executed once; and transitions are determined by the LLM's outputs. The Actor model flattens each state-action pair into a natural-language sequence, encodes it with a pretrained encoder, and applies a linear layer to produce a Q-value, trained by double deep Q-learning with epsilon-greedy exploration and stepwise or episode-level rewards. At inference, the LLM simply executes the subtask with the highest Q-value at each step, and the episode ends when the action space is empty. The experiments on MRC, IE, and STC claim that this adaptive ordering outperforms fixed-order prompting, tree-search self-planning, and the prior RL-based planner on the tested datasets.","pith_inferences":["An implication the paper leaves implicit: if the Q-value mapping transfers beyond the three task families, the same Actor-plus-frozen-LLM separation could steer any decomposable task, because the LLM never needs retraining.","A testable extension: inspect the learned Q-values on MRC; if they rank sentences by lexical overlap with the question, the policy reduces to relevance ranking and a simpler deterministic baseline would replicate RLAP.","Another extension: since the Actor is trained on rewards produced by one specific executor LLM, the same Actor may not transfer to a different LLM; retraining per executor is a likely requirement, and the paper does not test cross-executor transfer.","The stated future directions of pre-training and multimodal tasks suggest using the Actor's ordering signal as a curriculum selector or modality-attention controller, which could be evaluated directly in those settings."],"forward_implications":["If the learned ordering policy transfers, closed-source LLMs can be steered to better task accuracy by a small external planner, avoiding fine-tuning and catastrophic forgetting.","A single trained Actor per task type and language can be reused across datasets with the same state and action structure, because training samples are pooled across datasets of the same type.","For tasks whose final answer is a concatenation of substep results, stepwise rewards with discount factor $\\gamma=1$ make the Q-values rank entire order chains; for end-task rewards, the discount factor propagates final correctness back through intermediate states.","Because the LLM is frozen and only the Actor is trained, the framework can be applied to any decomposable task with a well-defined state and action space without additional LLM supervision.","The reported complex-case gains suggest the method is most valuable where instance complexity makes the choice of next subtask harder for a fixed or LLM-driven order."],"supporting_citations":[{"why":"Provides the fixed-order multi-turn baseline for information extraction that RLAP compares against.","marker":"[39]"},{"why":"Provides the tree-search self-planning baseline that RLAP compares against on MRC and STC tasks.","marker":"[46]"},{"why":"Supplies the prior RL-based adaptive information-extraction planner whose reward design RLAP extends.","marker":"[9]"},{"why":"Supplies the pretrained encoder used as the Actor backbone for information-extraction tasks.","marker":"[6]"},{"why":"Supplies the multilingual encoder backbone used for MRC and sentence-filling tasks.","marker":"[50]"},{"why":"Provides the experience-replay deep Q-learning machinery used to train the Actor model.","marker":"[24]"},{"why":"Provides the double Q-learning target-network update used in the Actor training procedure.","marker":"[36]"},{"why":"Supplies the open LLM used as the task executor embedded in the reported experiments.","marker":"[43]"}],"fun_headline_variants":["Instance-aware subtask ordering boosts LLM accuracy","RL-learned planner reorders steps per task instance","Adaptive planning: RL decides the best step sequence for LLMs","Lightweight Q-learning adapts LLM subtask order on the fly","No LLM fine-tuning: RL picks optimal step order per instance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework assumes that a fixed-size embedding of the flattened state-and-action text captures the linguistic features that determine the best next subtask, and that the Q-values learned on training instances transfer to test instances; for MRC it additionally assumes that reordering sentences changes the final answer in a way the reward signal can learn.","fun_headline_variants_meta":{"raw":{"variants":["Instance-aware subtask ordering boosts LLM accuracy","RL-learned planner reorders steps per task instance","Adaptive planning: RL decides the best step sequence for LLMs","Lightweight Q-learning adapts LLM subtask order on the fly","No LLM fine-tuning: RL picks optimal step order per instance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000301,"raw_usage":{"total_tokens":1761,"prompt_tokens":996,"completion_tokens":765,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":680}},"tokens_in":612,"tokens_out":765,"duration_ms":7685,"temperature":1.0,"reasoning_tokens":680,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:45:21.254948+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out set of small contexts (4-6 sentences or slots), enumerate all possible subtask orders, run the frozen LLM on each order, and compare the accuracy of RLAP's chosen order with the best-order accuracy and with fixed and random orders. If RLAP does not approach the best order, or if random or fixed orders already match it, the claim that learned linguistic-feature Q-values drive the gains is refuted.","supporting_citations":[],"review_version":1}