{"id":"d57ff1da-d97a-4af3-89be-ce7b24bc99b3","arxiv_id":"2412.00353","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"ZEUS selects chain-of-thought demonstrations by measuring answer uncertainty under perturbations, outperforming prior zero-shot prompting methods on four reasoning benchmarks.","lead":"This paper proposes ZEUS, a method that picks which examples to show a large language model before it answers a question, using how uncertain the model is about each candidate example. ZEUS improves zero-shot chain-of-thought accuracy on reasoning benchmarks without human-written rationales or access to model internals.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ZEUS(LU) selection signal may be inflated by averaging uncertainty over the same unlabeled questions used to build the demonstrations; excluding those questions could change the chosen strategy and break the reported LU≈HA match.","rationale":"I read the paper as a zero-shot CoT method that uses perturbation-based uncertainty both to select demonstration questions (ZEUS) and, critically, to choose among seven uncertainty-range strategies without labels (ZEUS(LU)). The central evidence is Table 3, where ZEUS(LU) tracks the oracle ZEUS(HA) closely across 20 model-dataset cells. That evidence rests entirely on the inverse correlation in Section 5.4. The reader's concern is that this correlation is only observed on the same four benchmarks; my stress-test sharpens this: within those very benchmarks, the correlation may be an artifact because the unlabeled set used to compute average uncertainty overlaps with the demonstration construction set. Including a question and its answer in the prompt and then scoring that same question yields artificially low uncertainty; the effect is largest for strategies that select hard questions, exactly the strategies favored for advanced models. Since the paper does not state that demonstration questions were excluded, this is a plausible confound. The proposed check (rerunning Section 5.4 on Q \\ D_s) would settle it directly. I credit the paper for a broad evaluation across 5 models and 4 datasets and for the honest limitation section; the method is plausible and the LU≈HA agreement is a useful empirical result, but the selection signal needs this control before the claim is accepted. The reader's conditional verdict is appropriate; the check I propose is a natural condition for acceptance.","tokens_in":12292,"tokens_out":15483,"duration_ms":145966,"concrete_test":"For each model and dataset, recompute the average Temp-Perb uncertainty over Q \\ D_s (the unlabeled set with the demonstration questions removed) for each of the seven strategies; reselect the strategy with the lowest recomputed average, and report test accuracy for that strategy. If ZEUS(LU) no longer matches ZEUS(HA) in more than a few of the 20 model-dataset cells, or if the inverse correlation in Figure 6 flattens or reverses, the selection rule is confounded by in-context copying of demonstration answers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.4 states: \"We calculate the average uncertainty on the unlabelled set Q while keeping the demonstration unchanged\" and uses this to select the strategy with the lowest Temp-Perb uncertainty. But Stage 3 constructs each demonstration D_s from a subset Qs of the same unlabeled set Q. When a question q ∈ D_s is then scored with D_s in the prompt, the demonstration already contains q's rationale and answer, so the model can reproduce that answer with near-zero uncertainty. Strategies such as Hard and Very Hard deliberately select high-uncertainty questions; placing those questions in D_s lowers the average uncertainty of Q by a larger amount than Trivial or Easy strategies, whose selected questions were already low-uncertainty. This creates or inflates the inverse correlation in Figure 6 (and Figures 12–16 in the appendix) even if the demonstration does not improve accuracy on unseen questions. The paper does not report excluding demonstration questions from the average. If the inverse correlation and the ZEUS(LU)≈ZEUS(HA) match in Table 3 are driven by this self-copying artifact rather than by genuine uncertainty reduction on the rest of Q, the central claim that ZEUS(LU) performs label-free optimal strategy selection is not supported by the current experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ZEUS, a three-stage zero-shot CoT prompting method: (1) estimate per-question uncertainty by perturbing decoding temperature, trigger phrases, and question rephrasing; (2) define seven selection strategies that pick questions in different uncertainty bands derived from the mean and standard deviation of the unlabeled set; (3) construct demonstrations by k-means clustering within the chosen band, following Auto-CoT. The method is evaluated on GSM8K, Fallacy, StrategyQA, and EPR with five LLMs. The main reported findings are that ZEUS's uncertainty estimates have higher 'sensitivity' than temperature perturbation alone, that the best-performing strategy varies by model and task, and that the strategy selected by lowest average Temp-Perb uncertainty (ZEUS(LU)) closely matches the oracle best strategy (ZEUS(HA)) without needing labels.","tokens_in":12537,"tokens_out":6573,"duration_ms":57131,"significance":"If the ZEUS(LU) match with ZEUS(HA) were established on a clean signal, the paper would provide a practical label-free way to choose demonstration difficulty for CoT prompting, with the welcome property of requiring no model-internal access. Strengths include the breadth of the evaluation (four datasets, five LLMs), the clearly specified three-stage pipeline, and the release of code and datasets. However, the central selection signal is confounded by a self-copying artifact (Major Comment 1), and the headline comparisons lack variance reporting, so the significance of the core claim is currently not established.","major_comments":[{"comment":"The ZEUS(LU) selection rule is computed by averaging Temp-Perb uncertainty over the same unlabeled set Q from which each demonstration D_s is drawn. Since D_s already contains the question, rationale, and answer for every selected q, any such q in Q will have near-zero uncertainty when scored with D_s in the prompt. Strategies that deliberately select high-uncertainty questions (Hard, Very Hard) therefore remove the largest contributions to the average, mechanically lowering the average uncertainty relative to Trivial/Easy strategies, independent of any effect on unseen questions. This can manufacture the inverse accuracy-uncertainty trend in Figure 6 and the LU≈HA match in Table 3. The paper does not report recomputing the average on Q \\ D_s or on a held-out subset. To support the label-free optimal-selection claim, the authors should either exclude demonstration questions when computing the selection signal or use a separate validation subset, and show LU≈HA under that corrected signal.","section":"§5.4, Figure 6, Table 3"},{"comment":"No variance or significance measures accompany the reported accuracies, despite the paper stating in §4 that performance is averaged over three runs. Several key differences are small (e.g., GSM8K GPT4o: 95.8 vs 94.8; EPR Phi3: 76.2 vs 75.3), and on StrategyQA Manual-CoT beats ZEUS(LU) for four of five models. Without standard deviations, confidence intervals, or paired tests, the claim that ZEUS 'consistently outperforms' baselines is not statistically supported. The authors should report variability across runs and perform paired comparisons against Auto-CoT and Manual-CoT.","section":"§5.5, Table 3"},{"comment":"The abstract states ZEUS 'consistently outperforms existing CoT strategies across four challenging reasoning benchmarks,' but Table 3 shows Manual-CoT has the highest StrategyQA accuracy for GPT4o (tie), Phi3, Mistral, and GPT3.5. The body acknowledges this in §5.5 ('Manual-CoT achieves the highest accuracy for most models'). The abstract and conclusion should be revised to describe the task- and model-dependent results accurately, or the claim should be restricted to the datasets and settings where it holds.","section":"Abstract, §6"}],"minor_comments":[{"comment":"The confidence score in Eq. (1) is written as a sum over n responses, but the total number of generated responses per question is n × t × v; the normalization should be made consistent and the notation clarified.","section":"§3.1, Eq. (1)"},{"comment":"The paper says temperature is set to 0 for deterministic outputs and results are averaged over three runs; please clarify what varies across the three runs (for example, random seed or sampling of the unlabeled set).","section":"§4"},{"comment":"Manual-CoT results for GPT3.5 and GPT3-XL on the Fallacy dataset are marked with dashes; please explain why these baselines are missing.","section":"§4, Table 3"},{"comment":"The rephrasing perturbation is generated with GPT4o for all evaluated models; this makes the method dependent on a strong external model, and for the GPT4o rows the uncertainty estimator and the evaluated model coincide. This dependency should be disclosed and its effect discussed.","section":"§3.1, §4"},{"comment":"The sensitivity slopes are reported without error bars or confidence intervals, so the claim that ZEUS is 'closest to ideal' is not quantified; please add uncertainty estimates to the regression coefficients.","section":"§5.2, Figure 5"}],"recommendation":"major_revision","confidential_remarks":"Dear Editor, the paper is a reasonable empirical contribution, but the central label-free selection claim is currently not trustworthy because of the self-copying artifact in §5.4. I recommend major revision: the authors should rerun the ZEUS(LU) analysis with demonstration questions excluded from the uncertainty average, and add variance and statistical testing. If the LU≈HA match disappears, the paper's contribution reduces to an oracle-style analysis (ZEUS(HA)) plus a set of empirical observations, which may still be publishable but with very different claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the multi-perturbation uncertainty estimator and the strategy-band analysis are genuinely useful, but the label-free selection rule (ZEUS(LU)) has a confound that needs to be addressed before the central claim is credible.\n\nWhat's new: ZEUS combines temperature, trigger-phrase, and rephrasing perturbations into a predictive-entropy uncertainty score, then selects demonstration questions from uncertainty bands and clusters them Auto-CoT style. The sensitivity analysis (confidence-accuracy slope) is a nice diagnostic, and the comparison across four datasets and five LLMs is broad. For a fixed strategy, the method is well-described and reproducible in principle.\n\nSoft spots: The main issue is Section 5.4. To pick the best strategy without labels, the paper computes average Temp-Perb uncertainty on the unlabeled set Q while the demonstration is in the prompt. But the demonstration was built from a subset of Q. For any question q that appears in the demonstration, the model can copy the rationale and answer, so its uncertainty is near zero. Strategies that deliberately select high-uncertainty questions (Hard, Very Hard) therefore get a mechanical reduction in average uncertainty that has nothing to do with generalization to unseen questions. The paper never excludes demonstration questions from that average. If the inverse correlation in Figure 6 and the LU≈HA match in Table 3 are driven by this self-copying artifact, then the label-free selection story is unsupported. The fact that LU and HA often match exactly (95.8/95.8, etc.) makes me suspect the selection rule is not doing much real work on these benchmarks.\n\nThis is fixable: recompute the average uncertainty on Q excluding the demonstration questions, or use a held-out subset of Q for selection. The authors should also report variance or significance over runs, since they average three runs but give no error bars. The abstract overclaims 'consistently outperforms': Manual-CoT wins on StrategyQA for most models, and ZEUS is sometimes second. Minor point: the paper says code and data are uploaded, but there is no link in the text.\n\nWho should read it: People working on zero-shot CoT and demonstration selection. It is a plausible idea with a clear experimental setup, but the headline result needs the confound removed. I would send it to peer review with a request for revision, not desk-reject it.","headline":"Useful uncertainty-based demonstration selection, but the label-free strategy chooser is confounded by including demonstration questions in the uncertainty estimate.","tokens_in":13089,"tokens_out":3001,"would_cite":false,"duration_ms":27925,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Using the model's own uncertainty to choose chain-of-thought examples matches hand-crafted demonstrations without the human effort.","keywords":["zero-shot chain-of-thought","uncertainty estimation","demonstration selection","predictive entropy","perturbation-based uncertainty","large language models","reasoning benchmarks"],"falsifier":"Run all seven ZEUS strategies on a held-out reasoning dataset and a model family beyond the five tested, computing each strategy's average Temp-Perb uncertainty on the unlabeled set and its actual accuracy on the test set. The central claim would be undercut if the lowest-uncertainty strategy is not among the top performing strategies, or if ZEUS's confidence-vs-accuracy slope is not closer to 1 than the temperature-only estimator's slope.","tokens_in":12092,"feed_emoji":"🎯","tokens_out":10293,"duration_ms":82493,"temperature":0.7,"pith_summary":"The paper proposes ZEUS, a method for building chain-of-thought demonstrations for large language models from unlabeled questions alone. ZEUS estimates each question's difficulty by sampling many rationale-answer pairs under three perturbations — temperature variation, different trigger phrases, and rephrased questions — then collapsing them into a predictive-entropy score. It selects questions whose entropy falls in a chosen band, builds demonstrations by clustering the selected questions, and at inference time chooses the band whose average temperature-perturbation uncertainty is lowest. The authors claim this uncertainty-guided selection matches or beats zero-shot, few-shot, Zero-Shot-CoT, Manual-CoT, and Auto-CoT across GSM8K, Logical Fallacy, StrategyQA, and EPR on five language models, without manual rationales or access to model parameters. If true, this removes the main human cost of chain-of-thought prompting while preserving most of its accuracy gain.","feed_headline":"Uncertainty-guided demo selection beats zero-shot CoT baselines","feed_subtitle":"Perturbing temperature, trigger phrases, and wording reveals which questions make the best reasoning examples.","key_machinery":"The load-bearing object is the perturbation-based predictive entropy: a question's uncertainty $u_j$ computed from unique answers generated across temperature sampling, trigger-phrase variation, and question rephrasing. Low entropy means the model converges to the same answer under all perturbations; high entropy means its reasoning is unstable, and the paper treats that instability as a proxy for the question's difficulty or usefulness as a demonstration. The second piece of machinery is the strategy table: seven uncertainty bands (Trivial, Very Easy, Easy, Moderate, Challenging, Hard, Very Hard) defined by thresholds on $\\mu$ and $\\sigma$ of the unlabeled set. The entropy estimate carries the selection argument, while a separate, well-calibrated temperature-perturbation uncertainty is used at the end to choose which band to deploy without labels.","core_discovery":"The central discovery is that the model's own predictive entropy under perturbations is a usable, label-free signal for choosing which examples to put in a chain-of-thought prompt. For each unlabeled question $q_j$, ZEUS generates 15 responses: ten from five trigger phrases at temperature 1 (two per phrase) and five from GPT-4o-rephrased versions of $q_j$ at temperature 0. Unique answers are scored by their consistency across this pool, and the uncertainty is the predictive entropy $u_j=-\\sum_c p(y^c_j|q_j)\\log p(y^c_j|q_j)$. The paper reports that the regression slope of confidence against accuracy for this score is close to the ideal value of 1, while temperature-only perturbation is either under-sensitive or over-sensitive depending on the dataset. Seven selection strategies are then defined as bands on the unlabeled distribution using the mean $\\mu$ and standard deviation $\\sigma$; demonstrations are formed by clustering the selected questions and taking the cluster-centroid representative, following the Auto-CoT pipeline. The recommended variant, ZEUS (LU), picks the strategy with the lowest average Temp-Perb uncertainty and reaches accuracy close to the oracle-best strategy ZEUS (HA).","pith_inferences":["The same selection logic could be applied to other discrete prompt choices, such as demonstration count, clustering granularity, trigger phrase, or answer-format instructions, treating each as a strategy with its own average uncertainty.","The confidence-versus-accuracy slope used as a sensitivity diagnostic could become a general, label-free way to compare uncertainty estimators on new tasks, independent of whether ZEUS itself is used.","The advanced-versus-simple model split is described from five specific models and is likely a capability effect, so as model generations improve, the optimal-strategy mapping should be expected to shift rather than stay fixed."],"forward_implications":["On GSM8K, ZEUS (LU) outperforms all five baselines for every one of the five models tested.","On Logical Fallacy, ZEUS (LU) beats the baselines for four of the five models; the exception is GPT3-XL, where Few-Shot remains higher.","On StrategyQA and EPR, ZEUS (LU) matches or exceeds zero-shot, Zero-Shot-CoT, and Auto-CoT for most models, with Manual-CoT still best on StrategyQA for several models.","The optimal difficulty band is model-dependent: advanced models (GPT-4o, Phi3, GPT-3.5) do best with Hard or Challenging demonstrations, while simpler models (Mistral, GPT3-XL) do best with Trivial or Easy ones.","The recommended ZEUS (LU) requires no human annotations, no rationales, and no model parameter access, only unlabeled questions and the ability to sample from the model."],"supporting_citations":[{"why":"Introduces Zero-Shot-CoT and the trigger phrases that ZEUS uses as both a perturbation and a baseline.","marker":"Kojima et al., 2022"},{"why":"Introduces Manual-CoT with handcrafted demonstrations, the human-effort baseline that ZEUS aims to match without rationales.","marker":"Wei et al., 2022"},{"why":"Auto-CoT; provides the clustering-based demonstration construction pipeline that ZEUS adopts in Stage 3.","marker":"Zhang et al., 2022"},{"why":"Provides the consistency principle, that confident models give the same answer across temperatures, on which the temperature perturbation is based.","marker":"Wang et al., 2022"},{"why":"Shows that selecting uncertain questions for annotation improves CoT prompting, motivating the uncertainty-guided selection at the heart of ZEUS.","marker":"Diao et al., 2024"},{"why":"Supplies the perturbation-based uncertainty family, including Temp-Perb, whose sensitivity ZEUS is compared against.","marker":"Gao et al., 2024"},{"why":"Supports the premise that the choice of demonstration examples significantly influences zero-shot reasoning performance.","marker":"Wan et al., 2023"}],"fun_headline_variants":["Entropy-based demo selection improves zero-shot CoT","Uncertainty-guided examples boost chain-of-thought accuracy","ZEUS: pick few-shot demos via model predictive entropy","Label-free selection of reasoning examples beats CoT baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the average temperature-perturbation uncertainty (Temp-Perb) of the unlabeled set, measured for each candidate demonstration strategy, is inversely correlated with that strategy's test accuracy, so the lowest-uncertainty strategy is the best one; on a new task or model where this correlation weakens, ZEUS (LU) could select a suboptimal strategy.","fun_headline_variants_meta":{"raw":{"variants":["Entropy-based demo selection improves zero-shot CoT","Uncertainty-guided examples boost chain-of-thought accuracy","ZEUS: pick few-shot demos via model predictive entropy","Label-free selection of reasoning examples beats CoT baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000249,"raw_usage":{"total_tokens":1535,"prompt_tokens":915,"completion_tokens":620,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":553}},"tokens_in":531,"tokens_out":620,"duration_ms":6431,"temperature":1.0,"reasoning_tokens":553,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:27:29.581230+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run all seven ZEUS strategies on a held-out reasoning dataset and a model family beyond the five tested, computing each strategy's average Temp-Perb uncertainty on the unlabeled set and its actual accuracy on the test set. The central claim would be undercut if the lowest-uncertainty strategy is not among the top performing strategies, or if ZEUS's confidence-vs-accuracy slope is not closer to 1 than the temperature-only estimator's slope.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that selecting uncertain questions for annotation improves CoT prompting, motivating the uncertainty-guided selection at the heart of ZEUS."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the perturbation-based uncertainty family, including Temp-Perb, whose sensitivity ZEUS is compared against."}],"review_version":1}