ASK in the Dark: Uncertainty-Gated LLM Assistance under Partial Observability

Adriano Veloso; Francisco Galuppo; Guilherme Lima; Juarez Monteiro; Nathan Gavenski; Odinaldo Rodrigues

REVIEW 3 major objections 8 minor 14 references

ASK in the Dark: Uncertainty-Gated LLM Assistance under Partial Observability

T0 review · 3 major / 8 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Under partial observability, small language models stay silent until you give them trajectory context; then they correct the policy.

desk verdict Clean diagnosis that vanilla ASK dies under partial observability from bare prompts, not gating or model size; stateful context fixes it enough for real but modest gains with 2B models. read the letter →

arxiv 2607.02686 v1 pith:W323DMC5 submitted 2026-07-02 cs.AI cs.LG

Juarez Monteiro , Nathan Gavenski , Guilherme Lima , Francisco Galuppo , Odinaldo Rodrigues , Adriano Veloso This is my paper

classification cs.AIcs.LG

keywords partialobservabilityreinforcementlearninglanguagemodelsuncertaintyestimationMonteCarloDropoutPOMDPselectivequeryingpromptdesign

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement-learning agents that see only a narrow local view often act confidently on incomplete information. Language models could supply the missing global reasoning, but only if they are asked at the right moments and given enough history. Prior uncertainty-gated methods query a small language model when the policy is unsure, yet under partial observability the model almost never proposes a different action: its overwrite rate collapses to zero because it is shown only the same egocentric snapshot the policy already has. This paper shows the bottleneck is context, not model size. By feeding the language model a growing partial map, visited cells, action history and a short reasoning budget, the same small models begin to issue independent corrections. Success rises from 53 percent to 70 percent in FourRooms and from 89 percent to 93 percent in DoorKey; accuracy on a memory-heavy card game matches the language-model-only ceiling. A 2-billion-parameter model matches or beats its 4-billion counterpart, confirming that prompt design and selective gating matter more than scale.

What carries the argument

ASK+: an uncertainty-gated query that fires when Monte-Carlo-dropout total predictive entropy exceeds a tuned threshold, then prompts a small language model with a stateful, enriched observation that includes a partially revealed map, visit counts, action history and a short chain-of-thought budget.

What would settle it

Replace the stateful prompt with the original bare egocentric prompt while keeping the identical entropy gate and models; if overwrite rate and success gains disappear, the context diagnosis is confirmed. Conversely, if a well-calibrated entropy signal still yields near-zero overwrites even with full trajectory context, the gating claim fails.

Watch

Extended reading notes

Core claim

Vanilla uncertainty-gated language assistance fails under partial observability because the bare egocentric prompt supplies too little context for genuine reasoning, producing near-zero overwrite rates. Supplying trajectory-aware state (partial map, visits, action history) plus structured chain-of-thought turns the small language model into an active consultant that occasionally overrides the policy and measurably raises success, while the same predictive-entropy gate remains informative in POMDPs.

Load-bearing premise

That total predictive entropy from a memoryless policy reliably marks the moments when the language model’s supplied history can resolve the residual ambiguity rather than introduce new errors.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing worth knowing is that they nail the failure mode of their own prior method. Vanilla ASK’s overwrite rate collapses to ~0 across three POMDPs because the SLM only sees an egocentric snapshot and just echoes the policy. That is a context problem, not capacity or entropy failure. ASK+ supplies a partial map, visit counts, action history, and a short constrained CoT; overwrite becomes nonzero and success moves from 53% to 70% on FourRooms and 89% to 93% on DoorKey. 2B matches or beats 4B once the prompt is fixed. That is the actual contribution.

What they do well: the ablations are thorough and honest. Table 2 shows which prompt layer matters per environment (rationale for FourRooms, stateful for DoorKey, enriched features for HigherLower). Figure 2 and Table 3 show τ and IR behave as expected, and the system still helps even with weak PPO checkpoints. Code is public, thresholds are tuned only on validation seeds, PPO is trained independently. The claim that total predictive entropy remains a usable action-uncertainty signal in POMDPs is supported by the IR scaling and the fact that gains appear only after the prompt is enriched.

Soft spots, in proportion: gains are real but modest; overwrite rates stay low (0.02 range). HigherLower gives the SLM privileged deck counts the memoryless policy never sees, so that environment is less clean as a pure test of the gate. Environments are three standard MiniGrid/POPGym tasks; no claim of broad generality. Free parameters (τ, N, token budget) exist, as expected for this style of work. None of these overturn the central diagnosis.

This is for people building hybrid RL+SLM agents who care about selective querying under partial observability. It is a solid workshop paper, not a theory breakthrough. I would bring it to reading group if the group is working on this stack. It deserves a serious referee rather than desk rejection; the experiments are careful enough and the diagnosis is useful. Engage if the topic is live for you; skip if you only care about large-scale planning or pure POMDP theory.

Referee Report

3 major / 8 minor

Summary. The paper diagnoses why uncertainty-gated SLM assistance (ASK) fails under partial observability: with bare egocentric prompts, overwrite rates collapse to ~0 across MiniGrid-FourRooms, DoorKey-8×8, and POPGym-HigherLower, so the SLM never contributes independent actions. The authors attribute this to a context bottleneck rather than model capacity, and propose ASK+, which supplies trajectory-aware state (partially revealed map, visit counts, action history) plus constrained chain-of-thought. They further argue that MC-Dropout total predictive entropy remains a usable action-uncertainty trigger in POMDPs. Empirically, ASK+ raises FourRooms success from ~53–55% to ~70%, DoorKey from 89% to 93%, and matches the SLM-only upper bound on HigherLower (~74%), while Qwen3.5-2B matches or exceeds the 4B model. Ablations isolate prompt components, τ, and PPO suboptimality; code and fixed seed splits are provided.

Significance. If the diagnosis holds, the result is practically useful: selective SLM assistance can be made viable in POMDPs by prompt interface design rather than by scaling the language model or retraining the policy. Strengths include a clean failure-mode diagnosis (near-zero OR for vanilla ASK), complementary environments, held-out validation for τ, independent PPO training, public code, and systematic ablations (prompt layers, τ grid, suboptimal checkpoints). The finding that a 2B model matches 4B under the same gating/prompt regime is a concrete efficiency claim. Gains are modest and environment-dependent, and overwrite rates remain low even for ASK+, so the work is better read as a careful systems/empirical contribution than as a large algorithmic leap; within that scope it is solid and actionable.

major comments (3)

Table 1 and §5: ASK+ reports large success gains (e.g., FourRooms 0.54→0.70) while OR stays very small (0.02; IR≈0.20). Because OR is defined over all steps, conditional overwrite is only ~10% of queries. The narrative that the SLM becomes an “active consultant that occasionally corrects” is plausible for sparse-reward navigation, but the paper does not quantify whether the few overwrites are high-impact, whether they help more than they hurt, or whether any residual gain comes from other factors (different effective intervention sets under the same τ, invalid-action filtering, etc.). A short trajectory-level analysis of overwrite steps (success conditional on overwrite, critical-turn examples) is needed to make the mechanism load-bearing rather than inferred.
Abstract and §3.2: the claim that the work “establishes” that predictive entropy “measures action uncertainty rather than state uncertainty and remains informative in POMDPs” overstates what is shown. The method uses total predictive entropy of a memoryless MLP under MC Dropout and argues conceptually that both aleatoric and epistemic components justify querying; experiments show that IR scales with PPO quality (Table 3) and that τ trades off IR vs reward (Figure 2). That supports usefulness of the signal, not a separation of action vs state uncertainty. Soften the language to “remains a useful selective trigger” unless an explicit isolation experiment is added.
§4 / Table 1: evaluation uses one PPO training run per environment and 100 fixed test seeds; reported ± values are episode-level dispersion, not training-seed variability, and no significance tests accompany the 4–17 point success gaps. For DoorKey (0.89→0.93) and HigherLower (0.72–0.74 band) this matters for the strength of the headline claims. At minimum, report multi-seed PPO training or bootstrap/paired tests over episodes, and clarify that stds are not training uncertainty.

minor comments (8)

Author block / contact lines: G. Lima is listed with adriano@kunumi.com; A. Veloso’s email appears duplicated. Fix affiliations and emails.
§4: define OR more explicitly as fraction of all timesteps (hence OR ≤ IR) and, ideally, also report OR|query = OR/IR so readers can compare conditional correction rates across environments with very different IR.
Table 2: τ is held fixed across prompt variants “for consistency,” but optimal τ may interact with prompt quality (rationale changes action distribution). A brief note or one re-tuned τ check for the best prompt would strengthen the ablation.
HigherLower enriched prompt (§A.2) injects count-derived P(next higher/lower) and remaining deck statistics. This is fair if those counts are computable from the observation stream, but it creates a strong information asymmetry vs the memoryless PPO. State this explicitly in §5 when comparing to SLM-only.
Figure 2: mark Optuna-selected τ more clearly (caption says “marked with a line” but values are hard to read); add the selected numeric τ in the legend or caption.
§3.1: “rationale” budget is described as 10 tokens in the method text and “max 15 words” / “max 12 words” in Appendix prompts—align the stated budget.
Related work is appropriate but brief on memory-augmented POMDP baselines (RNN/transformer policies). A one-paragraph contrast—why not simply give the policy recurrence instead of an external SLM—would help position the hybrid design.
Minor prose: “bare egocentric prompt” is repeated heavily; a few sentences in §1/§5 could be tightened. Keywords list is fine.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical method paper with independent PPO training, held-out threshold tuning, and new POMDP measurements; self-citation of ASK is ordinary prior-work baseline, not load-bearing.

full rationale

The paper is a standard empirical RL/LLM-assistance study. PPO is trained independently of the SLM; the gating threshold τ is chosen by Bayesian optimization (Optuna) exclusively on a held-out validation seed split and never touches the test episodes; success/reward/IR/OR are measured on fresh seeds. The central diagnostic (vanilla ASK overwrite rate ≈ 0 under partial observability) and the proposed fix (stateful/enriched/rationale prompts) are new experimental observations, not quantities fitted and then re-reported as predictions. Self-citation of the original ASK framework [1] (overlapping authors) merely supplies the baseline method being extended; it does not import a uniqueness theorem, force an ansatz, or redefine the reported metrics. No equation reduces a claimed result to its own inputs by construction. Score 0 is therefore the correct, proportionate finding.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The central empirical claim rests on a small number of standard RL/LLM assumptions plus a few free parameters (entropy threshold, MC samples, prompt token budget) that are tuned on validation data. No new physical entities are postulated; the invented constructs are engineering interfaces (stateful prompt, total-entropy gate).

free parameters (3)

entropy threshold τ = environment-dependent, range [0.1, 2.0]
Selected independently per environment–model pair by Optuna on validation seeds; directly controls intervention rate and therefore the reported performance–cost trade-off.
MC Dropout sample count N
Number of stochastic forward passes used to estimate predictive entropy; fixed but not ablated in the main tables.
rationale token budget = 10 tokens / max 15 words
Hard limit of 10–15 words / ~10 tokens for the SLM’s internal thinking step; chosen by hand to keep small-model latency low.

assumptions (3)

domain assumption Monte-Carlo Dropout on the policy MLP yields a usable estimate of total predictive entropy that correlates with action unreliability under partial observability.
Inherited from Gal & Ghahramani and the original ASK paper; asserted to remain informative in POMDPs (Section 3.2) without a formal proof that it separates aleatoric from epistemic components cleanly.
domain assumption A memoryless PPO policy trained for 2e6 steps with default Stable-Baselines3 hyper-parameters is a fair and representative base policy for the three chosen environments.
Standard practice, but the absolute performance numbers (and therefore the relative gains of ASK+) depend on this training regime.
ad hoc to paper The partially-revealed map, visit counts and action history supplied in the prompt are a faithful and non-leaking summary of the information the agent could have accumulated.
Engineering choice of ASK+; if the map construction leaks future information or is incomplete, the SLM’s advantage is artificial.

invented entities (1)

ASK+ stateful prompt interface (partial map + visits + history + constrained CoT)
purpose: Convert the SLM from a passive validator into an active consultant under partial observability.
The core engineering contribution; its value is demonstrated only inside the three evaluated environments.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ASK in the Dark: Uncertainty-Gated LLM Assistance under Partial Observability." pith.science (2026). https://pith.science/paper/W323DMC5

@misc{pith2026260702686,
  author       = {Pith},
  title        = {Pith review of: ASK in the Dark: Uncertainty-Gated LLM Assistance under Partial Observability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W323DMC5}},
  note         = {Machine review of arXiv:2607.02686}
}

read the original abstract

Reinforcement learning agents operating under partial observability must act on incomplete information, making them natural candidates for guidance from small language models (SLMs) that carry broad reasoning priors. Yet integrating SLM guidance into this setting has proven difficult: across all test environments, vanilla uncertainty-gated approaches achieve an overwrite rate at or near zero, meaning the SLM almost never contributes an independent action. We trace this failure to the bare egocentric prompt, which provides insufficient context for genuine reasoning, and identify it as a context problem rather than a capacity problem. We propose ASK+, which supplies the SLM with trajectory-aware context (a partially revealed map, visited positions, and action history) and structured chain-of-thought reasoning, converting it from a passive redundancy check into a more informative consultant that occasionally corrects the policy. We further establish that the predictive entropy signal used for selective querying measures action uncertainty rather than state uncertainty and remains informative in POMDPs, making uncertainty-gated assistance viable beyond fully observable settings. The stateful prompt drives substantial gains: on DoorKey, where vanilla ASK matches PPO (both 89%), ASK+ reaches 93% success; on FourRooms, success climbs from 53% to 70%; on HigherLower, accuracy reaches 73.7%, matching the SLM-only upper bound. Across all environments, Qwen3.5-2B matches or exceeds Qwen3.5-4B, confirming that prompt design and selective gating dominate the impact of model scale, enabling guidance without large models.

Figures

Figures reproduced from arXiv: 2607.02686 by the authors.

**Figure 1.** Decision flow of ASK+. At each step, the PPO policy produces a default action 𝑎, and MC Dropout estimates the total predictive entropy 𝐻(𝑜). If 𝐻(𝑜) ≥ 𝜏 , the SLM is queried via a prompt 𝑃 𝑟 constructed from the current observation, the PPO suggestion, and the episode state 𝜎; its output 𝑎 ′ overrides 𝑎 when valid. The episode state 𝜎 (dashed arrow) accumulates observations, visited positions, and action history thr… view at source ↗

**Figure 2.** shows reward and IR across values of 𝜏 . In FourRooms and DoorKey, low 𝜏 yields high IR and degraded reward, whereas in HigherLower it improves average reward. In all environments, high 𝜏 recovers PPO-level performance as IR approaches zero, since the SLM is bypassed entirely. These results confirm that 𝜏 predictably controls the intervention rate, with its optimal value being environment-dependent rather than unive… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 2 linked inside Pith

[1]

Monteiro, N

J. Monteiro, N. Gavenski, G. Zuin, A. Veloso, When to ask: Uncertainty-gated language assistance for reinforcement learning, in: 2026 International Joint Conference on Neural Networks (IJCNN), 2026

2026
[2]

M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al., Do as i can, not as i say: Grounding language in robotic affordances, arXiv preprint arXiv:2204.01691 (2022)

arXiv 2022
[3]

Valmeekam, K

K. Valmeekam, K. Stechly, A. Gundawar, S. Kambhampati, A systematic evaluation of the planning and scheduling abilities of the reasoning model o1, Transactions on Machine Learning Research (2025)

2025
[4]

Chevalier-Boisvert, B

M. Chevalier-Boisvert, B. Dai, M. Towers, R. Perez-Vicente, L. Willems, S. Lahlou, S. Pal, P. S. Castro, J. K. Terry, Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks, Advances in Neural Information Processing Systems 36 (2023) 73383–73394

2023
[5]

Morad, R

S. Morad, R. Kortvelesy, M. Bettini, S. Liwicki, A. Prorok, Popgym: Benchmarking partially observable reinforcement learning, arXiv preprint arXiv:2303.01859 (2023)

arXiv 2023
[6]

L. P. Kaelbling, M. L. Littman, A. R. Cassandra, Planning and acting in partially observable stochastic domains, Artificial intelligence 101 (1998) 99–134

1998
[7]

Y. Gal, Z. Ghahramani, Dropout as a bayesian approximation: Representing model uncertainty in deep learning, in: international conference on machine learning, PMLR, 2016, pp. 1050–1059

2016
[8]

URL: https://qwen.ai/blog?id= qwen3.5

Qwen Team, Qwen3.5: Towards native multimodal agents, 2026. URL: https://qwen.ai/blog?id= qwen3.5

2026

Show all 14 references

[9]

Raffin, A

A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, N. Dormann, Stable-baselines3: Reliable reinforcement learning implementations, Journal of machine learning research 22 (2021) 1–8

2021
[10]

Gavenski, J

N. Gavenski, J. Monteiro, F. Galuppo, A. Veloso, O. Rodrigues, When in doubt, plan it out: Committed small language model deliberation for reactive reinforcement learning, 2026. URL: https://arxiv.org/abs/2606.16995.arXiv:2606.16995

2026
[11]

Akiba, S

T. Akiba, S. Sano, T. Yanase, T. Ohta, M. Koyama, Optuna: A next-generation hyperparameter optimization framework, in: Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, 2019, pp. 2623–2631

2019
[12]

Carta, C

T. Carta, C. Romac, T. Wolf, S. Lamprier, O. Sigaud, P.-Y. Oudeyer, Grounding large language models in interactive environments with online reinforcement learning, in: International conference on machine learning, PMLR, 2023, pp. 3676–3713

2023
[13]

Valmeekam, M

K. Valmeekam, M. Marquez, A. Olmo, S. Sreedharan, S. Kambhampati, Planbench: An extensi- ble benchmark for evaluating large language models on planning and reasoning about change, Advances in Neural Information Processing Systems 36 (2023) 38975–38987

2023
[14]

Kambhampati, K

S. Kambhampati, K. Valmeekam, L. Guan, M. Verma, K. Stechly, S. Bhambri, L. P. Saldyt, A. B. Murthy, Position: Llms can’t plan, but can help planning in llm-modulo frameworks, in: Forty-first International Conference on Machine Learning, 2024. 10 Juarez Monteiro et al.CEUR Wor...

2024

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.

[1] [1]

Monteiro, N

J. Monteiro, N. Gavenski, G. Zuin, A. Veloso, When to ask: Uncertainty-gated language assistance for reinforcement learning, in: 2026 International Joint Conference on Neural Networks (IJCNN), 2026

2026

[2] [2]

M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al., Do as i can, not as i say: Grounding language in robotic affordances, arXiv preprint arXiv:2204.01691 (2022)

arXiv 2022

[3] [3]

Valmeekam, K

K. Valmeekam, K. Stechly, A. Gundawar, S. Kambhampati, A systematic evaluation of the planning and scheduling abilities of the reasoning model o1, Transactions on Machine Learning Research (2025)

2025

[4] [4]

Chevalier-Boisvert, B

M. Chevalier-Boisvert, B. Dai, M. Towers, R. Perez-Vicente, L. Willems, S. Lahlou, S. Pal, P. S. Castro, J. K. Terry, Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks, Advances in Neural Information Processing Systems 36 (2023) 73383–73394

2023

[5] [5]

Morad, R

S. Morad, R. Kortvelesy, M. Bettini, S. Liwicki, A. Prorok, Popgym: Benchmarking partially observable reinforcement learning, arXiv preprint arXiv:2303.01859 (2023)

arXiv 2023

[6] [6]

L. P. Kaelbling, M. L. Littman, A. R. Cassandra, Planning and acting in partially observable stochastic domains, Artificial intelligence 101 (1998) 99–134

1998

[7] [7]

Y. Gal, Z. Ghahramani, Dropout as a bayesian approximation: Representing model uncertainty in deep learning, in: international conference on machine learning, PMLR, 2016, pp. 1050–1059

2016

[8] [8]

URL: https://qwen.ai/blog?id= qwen3.5

Qwen Team, Qwen3.5: Towards native multimodal agents, 2026. URL: https://qwen.ai/blog?id= qwen3.5

2026