{"id":"d6648fc4-010c-442f-af02-3255408e87d1","arxiv_id":"2608.11922","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A fixed learned sentence inserted into a frozen LLM's prompt steers its uncertainty so that lowest-entropy answer selection avoids misleading passages, raising mean F1 from 0.5148 to 0.5339 across five QA benchmarks.","lead":"This paper trains a short prompt string, LODESTAR, that makes a frozen question-answering model less certain when reading misleading evidence, so that choosing the answer the model is most confident about becomes more reliable. On five QA benchmarks the string raises answer F1 from 0.5148 to 0.5339 while keeping the model itself unchanged.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"First-token entropy was chosen on the same five evaluation pools, so the out-of-domain and highest-mean-F1 claims have no untouched test set; a genuinely held-out pool is needed.","rationale":"The reader's CONDITIONAL verdict is the right call, and I do not move it. The reader identified H1 selection as a secondary load-bearing choice but framed the weakest assumption around OOD transfer with noisy judge labels; my main concern is the estimator selection on the evaluation pools, which directly threatens the OOD and ranking claims. The paper is otherwise careful: three-seed runs, paired tests, disclosed baseline deviations, and a transparent ablation all support the internal comparison. No code or data release is a separate reproducibility limitation that the reader already weighed. The concrete test I propose would settle whether the estimator-selection bias actually changes the headline numbers, so the verdict should remain CONDITIONAL pending that check.","tokens_in":33319,"tokens_out":9070,"duration_ms":99740,"concrete_test":"Before any estimator comparison, hold out one entire pool (e.g., WebQuestions). Re-run the Appendix C H1-versus-all-token comparison using only NQ, SQuAD, TriviaQA, and EntityQuestions, pick the better estimator on those four pools, and then evaluate LODESTAR versus the ablated H1 selector on the held-out WebQuestions pool. If the +0.0191 F1 gain and the misleading-read-rate reduction survive on the untouched pool, the selection-bias concern is resolved; if they shrink or reverse, the headline out-of-domain gain is partly an artifact of test-set-guided estimator choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix C selects H1 (first-token entropy) over the aligned all-token mean by comparing selection F1 on the same five pools that later produce every headline number. Section 4.2 then fixes L=1 for LODESTAR and for the polarizer ablation. Because the same pools are reused to claim out-of-domain transfer and to rank LODESTAR against all fourteen baselines, the reported 0.5339 mean F1 and the 26.0% misleading-read rate are partly the result of a design choice made on the evaluation data. This is not an internal inconsistency: the ablation removing the polarizer keeps H1 fixed, so the +0.0191 gain over plain H1 selection remains meaningful. But the central claim that one string outperforms every tested method, and especially the out-of-domain claim, is weakened: the four 'out-of-domain' datasets were used to justify the estimator, so no tested pool is truly untouched by model selection. Paired significance tests do not fix this, because they compare methods on the same questions after the estimator was already chosen. The paper's own citation of Clark et al. (2025) cautions against exactly this first-token approximation, which makes the need for an independent estimator choice more acute.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LODESTAR, an offline-trained natural-language 'polarizer' string inserted between a retrieved passage and the question in the prompt of a frozen LLM respondent. The claim is that a single fixed string, trained by GRPO to maximize the within-question separation of the respondent's first-token entropy between judged-misleading and gold-matching passages, is sufficient to make lowest-entropy answer selection less susceptible to confidently wrong answers. Across 5,008 questions from five QA benchmarks (NQ-Open plus four OOD sets), with a frozen Llama-3.1-8B respondent and shared bge-m3 top-10 pools, the three-seed mean of LODESTAR reaches 0.5339 mean answer F1 versus 0.5148 for plain first-token entropy selection, 0.4769 for the retriever's top-1, and is reported as paired-significant against fourteen re-purposed published configurations; the polarizer ablation reduces the rate of selecting judged-misleading passages from 30.3% to 26.0%. The manuscript also reports cross-respondent transfer (positive diagonals, four negative off-diagonal facets) and a detailed reproduction of CTRL-RAG in an appendix.","tokens_in":33538,"tokens_out":19563,"duration_ms":191123,"significance":"If the results hold, LODESTAR would be a practically valuable, cheap intervention — no extra model, sampling, or gold answers at inference — and a clean demonstration that a frozen respondent's uncertainty signal can be steered by prompt-level text rather than merely measured. The paper's strengths are its unusually complete empirical discipline: all three training seeds are reported individually, with a same-seed execution-level rerun (Appendix E); per-question paired tests (bootstrap, t, Wilcoxon) are tabulated in full (Tables 7, 9, 10); every baseline deviation from official scripts is disclosed in Appendix J, including the semantic-entropy first-wins tie-break inflation and the SPS task mismatch; and the CTRL-RAG case study derives a mechanism (Eq. 7) for the observed format degeneration. The controlled ablation of Section 6 (removing only the polarizer string) is well designed, and the hand-written vs searched vs GRPO-learned ordering (0.5129 / 0.5131–0.5160 / 0.5339) is a strong comparison.","major_comments":[{"comment":"The choice of the L=1 first-token entropy estimator over the aligned all-token mean was made by comparing selection F1 on the very same five evaluation pools that later produce every headline number in Tables 2 and 3 (0.5148 vs 0.5001 on those pools). The four 'out-of-domain' datasets participated in this choice, so no tested pool is untouched by estimator selection, and the headline claims of 'highest mean F1 of any inference-ready selector' and of out-of-domain transfer (0.5274 to 0.5476) are more fragile than the paper's wording suggests: the untrained entropy signal LODESTAR builds on was selected on the test pools, while the fourteen baselines were not given test-pool-based tuning of their own scoring rules. The Table 3 ablation survives this critique because it keeps H1 fixed, and Figure 2's positive Llama diagonal under the all-token selector suggests the polarizer effect is not H1-specific, but the main tables never connect these dots. I ask the authors to either (a) re-select the estimator on NQ-Open training questions only, or (b) report the key tables under both H1 and the aligned all-token estimator (retraining the polarizer for the latter if needed), or (c) state the limitation in the main text and qualify the 'highest F1' and OOD claims accordingly.","section":"Section 4.2 and Appendix C (Table 6)"},{"comment":"The two headline outcomes are not equally independent of the training labels. The reward in Eq. (4) contrasts passages labeled MISLEADING by a dual-judge consensus (Cohen's kappa = 0.675) against passages labeled SUPPORTING by exact match with gold, and Table 3's judged-misleading rate is then computed with the same Section 3 judge rule extended across all pools. While the paper is transparent about this and about the kappa value, the title-level claim that the polarizer keeps the respondent from being confidently misled by the wrong evidence rests partly on a metric that shares its label source with the training reward, and the independent audit in Appendix G is computed only on the 600 consensus rows, leaving disagreement rows unaudited. The F1 and EM results are label-independent and are the strongest evidence for the paper; I ask the authors to additionally report the misleading-rate ablation using the independent 70B audit judge over the full audit set (including disagreement rows), or to state in the main text that the misleading-rate metric evaluates the training objective in-family rather than serving as an independent audit.","section":"Sections 3 and 6, Eq. (4), Table 3, Appendix G"}],"minor_comments":[{"comment":"The phrasing 'one fixed string, enough by itself to outperform every published method the paper tests' should carry the frozen-respondent, inference-ready qualifier consistently, since the paper's own Table 12 reports CTRL-RAG's best checkpoint at 0.5920 mean F1 against LODESTAR's 0.5339 on identical pools, with the disambiguation residing only in an appendix.","section":"Section 1 and abstract"},{"comment":"The Reference column entry for the LODESTAR row contains an illegible glyph sequence that appears to be a corrupted superscript marker; it should be removed or replaced with a proper reference or star notation.","section":"Table 2 (also Tables 9 and 10)"},{"comment":"All main results use the bge-m3 top-10 pool, whose misleading share (28.9% macro) is the lowest band in Table 1; since the paper motivates the problem by showing that rerankers raise the misleading rate to as much as 35.0%, evaluating LODESTAR on at least one reranked pool would test the mechanism where the problem is most severe.","section":"Section 4.1 and Table 1"},{"comment":"The third-party audit (96% agreement) is computed on the 600 consensus rows, and this scope is stated only in the appendix; the main-text sentence in Section 3 should say explicitly that the audit covers only rows on which the two judges already agree, leaving the disagreement cases unaudited.","section":"Section 3 and Appendix G"},{"comment":"Section 6 could cite the Llama diagonal of Figure 2 as evidence that the polarizer's benefit is not specific to the H1 estimator, since those facets use a mean-normalized all-token selector; this would directly address the estimator-dependence concern raised by Appendix C.","section":"Section 6 and Figure 2"},{"comment":"The manuscript describes LODESTAR as a new method but provides no link to released code or artifacts; given the reliance on exact prompt templates, judge protocols, and cached probes, a statement of where the training and evaluation code will be made available is needed for reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is unusually thorough and honest, but the estimator-selection issue in Appendix C is likely to be the central point of contention in review. I would encourage the editor to ask for the estimator to be re-selected on a development split, or for the main tables to be reproduced under both estimators, before a final decision. The abstract's 'outperform every published method' phrasing will be read as overclaiming by many readers; the authors should add the frozen-respondent qualifier there as well. The appendices are valuable but make the manuscript very long, and the CTRL-RAG reproduction could usefully be flagged as a scoped case study in the main text rather than only in an appendix."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The core idea is new and the execution is unusually careful. A GRPO-trained fixed prompt string, inserted between each passage and the question, shifts a frozen LLM's first-token entropy so that lowest-entropy selection avoids misleading passages; the paper calls this directed entropy. The effect is modest but consistent: +0.019 F1 over plain entropy selection, 5/5 datasets, three seeds, paired tests, and a clean ablation. The authors also run fourteen published selectors under identical pools and a frozen respondent, report per-row deviations, and reproduce CTRL-RAG as a case study, including its format collapse. That kind of disclosure earns real credit.\n\nThe main soft spot is exactly where the stress-test note lands. The H1 estimator was chosen on the same five evaluation pools that produce every headline number (Appendix C). So the 'out-of-domain' claim is not fully out-of-domain, and the paper's strongest sentence — 'one fixed string is enough to outperform every published method' — carries a small but real selection bias. The polarizer ablation vs plain H1 is unaffected, because H1 is fixed in both arms. But the comparison against the all-token mean and the OOD transfer are both weakened. A genuinely held-out pool for the estimator choice would fix this, or an explicit admission that H1 is part of the method, not a neutral measurement.\n\nSecond soft spot: no code or data release. For a paper whose contribution is a trained artifact, that is a meaningful gap. The converged string is printed, which helps, but the training pipeline and the judge labels are not independently checkable.\n\nI disagree mildly with the reader's framing that this is 'conditional' in the sense of a load-bearing flaw. The central effect survives the estimator concern; the gain over plain H1 is the core claim and it is solid. The noise in the judge labels (kappa 0.675) is a real limitation but they audit it and the labels are only used for training, not inference. The paper is honest, internally consistent, and the math and stats look sound.\n\nWho this is for: anyone working on RAG answer selection, entropy-based confidence, or prompt optimization. It deserves a serious referee. I would send it to review, and in the review ask for a separate validation set for the estimator choice and for code/data release. With those, it would be a clean accept; without, it is still a solid, citable paper.","headline":"A genuinely new and unusually honest method for steering a frozen LLM's selection uncertainty; the headline gain is real, but the estimator choice on the eval pools makes the OOD claims softer than stated.","tokens_in":34121,"tokens_out":2621,"would_cite":true,"duration_ms":24841,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single learned string inserted into the prompt lets a frozen LLM avoid confidently wrong answers in retrieval-augmented QA.","keywords":["retrieval-augmented generation","entropy-based selection","confidently wrong","polarizer","frozen LLM","reinforcement learning","question answering","prompt intervention"],"falsifier":"Take the published polarizer and apply it, with the same frozen respondent and candidate pools, to a new benchmark from an unseen domain; if the judged-misleading selection rate does not fall below the pool’s base rate and answer $F_1$ does not rise, the transfer assumption is false.","tokens_in":33097,"feed_emoji":"🧭","tokens_out":8498,"duration_ms":76760,"temperature":0.7,"pith_summary":"Retrieval-augmented question answering suffers from a quietly dangerous failure: a misleading retrieved passage can make a frozen answering model more confident, not less, so the standard “keep the lowest-entropy answer” rule selects exactly the wrong evidence. The paper argues this is not a flaw of entropy itself but of treating the model’s uncertainty as a fixed measurement. It introduces LODESTAR, which learns one short natural-language string offline by reinforcement learning and inserts it between the passage and the question; this “polarizer” steers the responder’s first-token entropy upward on misleading passages and leaves supporting passages near the entropy floor. On 5,008 questions across five QA benchmarks, the same minimum-entropy rule with one string inserted lifts mean answer $F_1$ from 0.5148 to 0.5339, exact match to 0.4136, and lowers the rate of selecting a judged-misleading passage from 30.3% to 26.0%. The paper’s central claim is that entropy should be navigated—changed as an input—rather than merely measured.","feed_headline":"One fixed string lifts QA answer F1 to 0.5339","feed_subtitle":"Inserted between passage and question, it stops a frozen LLM from confidently choosing misleading evidence.","key_machinery":"The central object is the polarizer $\\psi^\\star$, a short fixed natural-language string trained once offline by reinforcement learning and inserted between the retrieved passage and the question, never into the model’s weights. It works by “directed entropy”: the reward is the within-question separation $\\Delta_{Mis-Sup}\\bar H_L(q;\\psi)-\\Delta_{Mis-Sup}\\bar H_L(q)$ between the responder’s mean first-token entropy on judged-misleading and supporting passages, so the string is optimized to raise entropy on misleading candidates while leaving supporting ones near the floor. At inference, selection is still $\\arg\\min_a \\bar H_L(q,p,\\psi^\\star)$, with no extra model, no sampling, and no gold answer; the only artifact carried from training is the string itself.","core_discovery":"LODESTAR establishes that within-question entropy can be repaired by a text intervention. When a misleading passage is read alone, the frozen respondent’s answer-token entropy is lower than on a supporting passage—1.22 nats versus 1.33 pooled across the five pools—so argmin-entropy selection is drawn to the wrong answer. The paper trains a polarizer $\\psi^\\star$ with a policy-gradient objective whose reward is the baseline-corrected within-question separation of the responder’s first-token entropy between judged-misleading and supporting passages. Inserted as $[p; \\psi^\\star; q]$, the learned string raises entropy on the misleading side by roughly an order of magnitude in the worked example, moving the argmin to the supporting passage. Across 5,008 questions and fourteen published selectors re-run under the same frozen respondent and candidate pools, this one fixed string gives the highest mean answer $F_1$ (0.5339), exact match (0.4136), and independent-judge accuracy (0.6435), with the string itself, by ablation, contributing $+0.0191 \\pm 0.0054$ $F_1$ across three training seeds; the gain holds both on Natural Questions and on the four out-of-domain benchmarks.","pith_inferences":["If the mechanism is robust, prompt-level steering of first-token entropy could be a general “epistemic navigation” tool for frozen models—e.g., to make a model less credulous about retrieved evidence in other tasks like fact-checking or medical QA—without fine-tuning.","The polarizer’s converged text is an interpretable instruction to doubt that the passage addresses the exact question; independent training seeds converge on similar wording, suggesting the reward has a stable linguistic optimum that could be used to audit what reinforcement learning actually optimizes.","The reward’s training labels come from two LLM judges with Cohen’s kappa 0.675; improving label agreement, or replacing gold-exact-match definitions of support, would likely sharpen the entropy separation and could yield larger gains than the current +3.7% $F_1$.","A natural stress test would be to train the polarizer on one respondent and evaluate on an unseen model family; the paper’s cross-respondent matrix already shows negative off-diagonal transfers, so a pretrained “universal polarizer” is not implied."],"forward_implications":["Because inference is one forward pass per candidate and reads no gold answer, the polarizer can be dropped into existing retrieval-augmented pipelines at negligible cost.","The ablation shows the string, not the training procedure, accounts for the gain: removing it returns $F_1$ and misleading-rate to the plain entropy baseline, positive on 5/5 datasets under all three seeds.","The polarizer is respondent-specific: on two additional frozen respondents, every diagonal pairing (train and evaluate on the same model) improves, while several off-diagonal transfers hurt, so the string should be trained for the model that will answer.","The learned string outperforms both hand-written cautionary instructions and prompt-search optimizers under the same reward, indicating the RL training finds a better optimum than direct search.","Entropy selection alone reads a misleading passage more often than random draw (30.3% vs. the pool’s 28.9%); LODESTAR is the configuration that consistently stays below that floor."],"supporting_citations":[{"why":"Supplies the GRPO policy-gradient objective used to train the polarizer policy.","marker":"(Shao et al., 2024)"},{"why":"Defines the closest prior entropy-based selector, whose information-gain rule reduces to plain minimum-entropy selection.","marker":"(Song et al., 2026)"},{"why":"Supplies the semantic-entropy baseline and the answer-equivalence judging prompt adopted verbatim.","marker":"(Farquhar et al., 2024)"},{"why":"Provides the shared retrieval model that defines the ten-candidate pools used by every method.","marker":"(Chen et al., 2024b)"},{"why":"Documents the caveat about approximating word entropy with first-token entropy, the estimator LODESTAR tests and adopts.","marker":"(Clark et al., 2025)"},{"why":"Provides the Natural Questions in-domain benchmark and the training pool for the polarizer.","marker":"(Kwiatkowski et al., 2019)"}],"fun_headline_variants":["Entropy navigated: one fixed string lifts QA F1 to 0.5339","Reinforced polarizer lifts QA F1 to 0.5339 with one fixed string","One offline-trained string steers frozen LLM to best QA evidence (F1 0.5339)","LODESTAR: a fixed prompt string beats 14 selectors on QA evidence choice"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire transfer claim rests on the assumption that a short English string trained only on Natural Questions can shift a frozen model’s first-token confidence on misleading passages in the same way on other datasets; if the steering effect is specific to NQ-like passages, the out-of-domain gains disappear.","fun_headline_variants_meta":{"raw":{"variants":["Entropy navigated: one fixed string lifts QA F1 to 0.5339","Reinforced polarizer lifts QA F1 to 0.5339 with one fixed string","One offline-trained string steers frozen LLM to best QA evidence (F1 0.5339)","LODESTAR: a fixed prompt string beats 14 selectors on QA evidence choice"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001633,"raw_usage":{"total_tokens":6625,"prompt_tokens":1208,"completion_tokens":5417,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":824,"completion_tokens_details":{"reasoning_tokens":5318}},"tokens_in":824,"tokens_out":5417,"duration_ms":38523,"temperature":1.0,"reasoning_tokens":5318,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:21:41.967238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the published polarizer and apply it, with the same frozen respondent and candidate pools, to a new benchmark from an unseen domain; if the judged-misleading selection rate does not fall below the pool’s base rate and answer $F_1$ does not rise, the transfer assumption is false.","supporting_citations":[{"cited_title":"(2024) in its multi-reference branch","cited_arxiv_id":null,"evidence_quote":"Supplies the semantic-entropy baseline and the answer-equivalence judging prompt adopted verbatim."}],"review_version":1}