{"id":"c1cdefc3-456a-47ec-a58e-a80a14df1b93","arxiv_id":"2501.15030","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"OptiSeq selects the in-context example ordering whose output gets the highest zero-shot log-likelihood, improving few-shot accuracy by up to 10.5 points in tests on API sequencing and classification.","lead":"This paper introduces OptiSeq, a method that picks the best order to list examples in an LLM prompt by scoring each candidate output with the model's own zero-shot log-probabilities, after removing the examples. It reports accuracy gains of 5.5 to 10.5 percentage points across five datasets and five LLMs, which matters because prompt order sensitivity is a practical headache for LLM application developers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ranking premise—zero-shot log-likelihood Phi_k predicting correctness—is asserted from anecdotal figures; no aggregate selection-accuracy metric is reported, so the 5.5–10.5 pp gains could partly reflect random order effects rather than the selection rule.","rationale":"The reader's weakest_assumption is the ranking premise, and I agree that is the most load-bearing point. The paper's empirical headline—consistent gains over Random/Top-K—is suggestive, but it does not isolate the selection mechanism. The method's success requires that the zero-shot log-likelihood of a candidate output is a better indicator of correctness than the few-shot likelihood used by naive ICL. Figures 6 and 9 and the appendix tables show favorable examples, but the paper does not quantify how often the selected order is correct, nor does it stratify by zero-shot confidence. A method that simply picks the order whose output has the highest zero-shot confidence could improve over random even if the premise is only weakly true, because it benefits from examining all orders and choosing a high-confidence candidate; the crucial question is whether the confidence signal is actually correlated with correctness. The proposed diagnostic directly measures this correlation and the selection success rate. The paper's lack of error bars is a secondary issue; the main concern is mechanistic. Given that the reader's verdict is CONDITIONAL and this stress-test reinforces the same condition, the verdict should remain unchanged: the paper should be accepted only if the authors provide the diagnostic evidence or a principled argument for why Phi_k is a reliable correctness proxy.","tokens_in":32316,"tokens_out":8916,"duration_ms":86731,"concrete_test":"Run a focused diagnostic on all 25 model–dataset combinations: for every test instance and each of the 6 orderings, record the generated output ok, its correctness (exact match for API tasks; label accuracy for classification), and Phi_k. Then compute (a) the Spearman rank correlation between Phi_k and correctness across all (instance, order) pairs, per dataset; and (b) the 'top-1 selection accuracy': the fraction of test instances where argmax_k Phi_k generates a correct output, restricted to instances having at least one correct ordering. Compare (b) against the random-order baseline, i.e., the average fraction of correct orderings per instance. If the top-1 selection accuracy is not significantly above the random baseline, or if the correlation is not positive and significant, the ranking premise is unsupported and the accuracy gains cannot be attributed to OptiSeq's selection rule.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that OptiSeq improves accuracy by 5.5–10.5 pp because argmax_k Phi_k reliably selects the ordering that yields a correct output. This depends on the premise that the zero-shot log-likelihood of a generated output is a valid proxy for its correctness, after removing the in-context examples. The paper supports this only with a single illustrative comparison (Figure 6), a global confidence distribution (Figure 9), and cherry-picked appendix tables (Tables 5–13). It never reports the aggregate selection quality: how often the selected order is actually correct, or how well Phi_k correlates with correctness across all instances and permutations. Without this, the reported accuracy gains could arise even if the ranking premise is weak, because OptiSeq always examines multiple orders and the test instances where at least one order is correct may dominate the average. Conversely, if the premise is violated on a substantial fraction of 'confidently wrong' zero-shot instances, OptiSeq would select those wrong orders and could underperform random selection in those regimes. The paper provides no failure-mode analysis or stratified results. This is the load-bearing assumption: if it fails, the method reduces to an unprincipled ensemble, and the claimed 5.5–10.5 pp improvements are not attributable to the stated selection mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OptiSeq, a purely inference-time algorithm for choosing the order of in-context examples in an LLM prompt. For a given test instance and a fixed set of E examples, OptiSeq generates candidate outputs for all E! permutations, removes the in-context examples from each prompt, recomputes the sum of log probabilities of each generated output conditioned only on the task instruction, and selects the ordering whose output has the highest such score. A cheaper variant, EOptiSeq, anchors the most SBERT-similar example in the first position and searches only the remaining (E-1)! permutations. The main empirical claim is that this selection improves accuracy by 5.5-10.5 percentage points over random ordering, Top-K ordering, LocalE, and Influence-score baselines, across two tasks (API sequence generation and text classification), five datasets, and five LLMs ranging from 8B to 70B parameters.","tokens_in":32522,"tokens_out":3746,"duration_ms":37827,"significance":"If the central ranking premise holds, OptiSeq is a simple, dataset-free, and model-agnostic contribution: it requires no training data, no fitted parameters, and only the LLM's own log-probability outputs. The algorithm is fully specified in Algorithm 1, and the evaluation covers a reasonable diversity of models and tasks. These are genuine strengths. However, the paper's central assumption, that zero-shot log-likelihood after removing in-context examples ranks outputs by correctness, is supported only by illustrative examples and global confidence distributions, not by any aggregate selection-quality metric. In addition, all headline numbers are point estimates from single greedy-decoding runs with no variance or significance information, which is insufficient for a journal-level empirical claim. The contribution is potentially publishable, but the current evidence does not yet establish the claimed mechanism or the robustness of the improvements.","major_comments":[{"comment":"The entire method depends on the premise that the zero-shot log-probability sum Phi_k, computed after removing in-context examples, is a valid proxy for whether the generated output is correct. The paper supports this only with anecdotal per-instance examples and a global confidence distribution, and it never reports an aggregate selection-quality metric, such as the fraction of instances where argmax_k Phi_k corresponds to a correct output, or the rank correlation between Phi_k and per-order accuracy across all instances and permutations. Without such a metric, the reported accuracy gains can arise even if Phi_k is a weak signal, because OptiSeq always examines multiple orderings and the instances where at least one order is correct may dominate the average. Please report the selection accuracy and a failure-mode analysis, especially for instances where the model is confidently wrong in zero-shot mode.","section":"Section 3.1, selection step; Figures 6 and 9; Appendix Tables 5-13"},{"comment":"All results are reported as point estimates from greedy decoding, with no standard deviations, confidence intervals, or significance tests. The 'Random' baseline is underspecified: it is unclear whether each test instance is evaluated with a single random order and over how many random seeds the average is taken. Since the 5.5-10.5 percentage point improvements are the headline claim, the authors should report variability across multiple random draws and provide paired significance tests where possible.","section":"Section 5, Table 1"},{"comment":"Phi_k is an unnormalized sum of per-token log probabilities, which systematically penalizes longer outputs. In API sequence generation, the number of APIs in a correct sequence is task-dependent, so selection based on the unnormalized sum could be driven by output length rather than by correctness. No length-normalized variant (e.g., average log-probability per token) is reported, and no analysis disentangling length effects from correctness effects is provided. Please add such an analysis or justify the unnormalized choice.","section":"Section 3.1, Eq. (1)"},{"comment":"The LocalE and Influence-score baselines are described as relying on validation data and label-fairness assumptions, yet they are then compared with OptiSeq in the same inference-time, dataset-free setting. The exact protocol used to instantiate these baselines in this paper's experiments (e.g., what development set was used and how the label-fairness assumptions were satisfied) is not documented. This makes the reported 5.5-6.5 percentage point gains over these baselines difficult to interpret and reproduce. Please specify the baseline configuration in the same detail as OptiSeq.","section":"Section 4.3, 'Comparative Techniques'"}],"minor_comments":[{"comment":"The caption contains a typo: 'outpus' should be 'outputs'.","section":"Figure 2 caption"},{"comment":"The sentence 'OptiSeq achieves an average improvement of 10.5% points over random selection, 9.05% points over Top-K, 6.5% over LocalE and 5.5% over Influence score' reports aggregates, but Table 1 shows per-cell differences that vary substantially; please specify whether these averages are computed over datasets, models, or both, and report the per-cell raw differences in the main text.","section":"Section 5, first paragraph"},{"comment":"Several displayed generated sequences appear malformed, for example 'SearchMovie, Mcosy.credits', 'SearchMovie, MovieCredits, ortrailing', and 'SearchMovie, B' in Table 7. If these are tokenization artifacts or truncated outputs, they should be explained; as presented, they undermine the qualitative evidence for the selection mechanism.","section":"Appendix Tables 5-13"},{"comment":"The claim that 'the model is influenced by example order than by the examples themselves' appears to have a missing comparative word (likely 'more'), and the claim is not supported by a statistical test; please rephrase and provide quantitative support.","section":"Section 2, last paragraph"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is testable and the method is simple enough that the missing validation can be added within the scope of a revision. I would not recommend rejection, but the current evidence is not at the bar for a journal: the selection-quality premise needs a direct aggregate test, and the headline comparisons need error bars and a fully specified random baseline. The authors should also more sharply differentiate OptiSeq from LocalE and Influence score, since those methods already use zero-shot output probabilities to score orderings; the novel aspect is the removal of the in-context examples before scoring, and the paper should make this contrast explicit and quantified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the quick take: OptiSeq is a simple, plausible inference-time method for choosing the order of in-context examples. The selection rule—generate an output for each permutation, strip the examples, and pick the order whose output has the highest zero-shot log-likelihood—is genuinely new as far as I can tell from the cited literature. The paper also does solid work showing that the best order varies by instance and model, which motivates the approach.\n\nWhat's good: the evaluation spans five datasets and five LLMs, and the appendix tables give per-example log-prob numbers that do show the criterion frequently selecting the correct output. The comparisons to LocalE and Influence are relevant, and the paper is honest about the factorial cost and the need for log-prob access.\n\nThe weak spots are real but not fatal. Most importantly, there is no aggregate measure of selection quality: no correlation between Phi_k and correctness across instances, no hit rate for 'did the selected order produce the right answer?' The Figure 9 confidence distribution and the anecdotal appendix examples are suggestive, but they don't establish that the ranking premise holds often enough to explain the 5.5–10.5 point gains. A second issue is the evaluation itself: numbers come from single greedy decoding runs, with no variance or significance testing, and the random baseline is described as \"an order is selected at random for each test instance\" without saying how many random draws were averaged. That makes the headline deltas hard to trust precisely. There's also a possible length bias—summing log-probs over tokens tends to penalize longer outputs—so the criterion might favor short answers rather than correct ones; the paper doesn't control for this. None of this sinks the idea, but it means the current evidence is suggestive, not conclusive.\n\nBottom line: this is a useful paper for anyone building few-shot pipelines who wants a training-free ordering heuristic. It deserves a serious referee, but the revision should report selection accuracy and correlation, add error bars, specify the random baseline, and release code. If those come in, I'd be comfortable citing it.","headline":"A novel and plausible ordering heuristic whose headline gains outrun its evidence; worth a careful review.","tokens_in":33103,"tokens_out":3483,"would_cite":true,"duration_ms":33206,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In in-context learning, the order of examples changes what LLMs output, and OptiSeq selects the best order at inference time by scoring each candidate output without the examples.","keywords":["in-context learning","example ordering","prompt sensitivity","log probabilities","inference-time optimization","API sequence generation","text classification","large language models"],"falsifier":"For a test instance where the model's zero-shot answer is wrong but has higher log probability than any correct answer, OptiSeq will pick the ordering whose candidate output is that wrong answer. Collect a set of such confidently-wrong instances and count how often OptiSeq beats random selection; if it does not beat random selection there, the selection rule fails exactly where confidence and correctness diverge.","tokens_in":32118,"feed_emoji":"🔀","tokens_out":5521,"duration_ms":44964,"temperature":0.7,"pith_summary":"The paper argues that the order of in-context examples is not a minor nuisance: the same examples arranged differently can move accuracy by roughly 12 points on API sequencing and 17 points on classification. It proposes OptiSeq, a purely inference-time, dataset-free method that tries all orderings of the examples, generates each candidate output, then re-scores each output with the examples stripped away, using only the task instruction. The ordering whose output scores highest under that zero-shot log-probability score is the one recommended. Across five datasets and five LLMs from 8B to 70B parameters, OptiSeq improves accuracy by 5.5 to 10.5 percentage points over random order, Top-K order, and two recent ordering baselines. The practical point is that an LLM application can get a more reliable prompt without any training data or offline tuning.","feed_headline":"On-the-fly example ordering beats random and Top-K by up to 10.5 points","feed_subtitle":"OptiSeq scores each candidate answer without the in-context examples, then keeps the ordering that looks most reliable.","key_machinery":"The load-bearing object is the example-free log-probability score $\\Phi_k$. Given an output $o_k$ generated under ordering $k$, the paper discards the examples and recomputes the token log probabilities conditioned only on the task instruction $I$; the ordering with the largest $\\Phi_k$ is chosen. This score separates correct from incorrect outputs more cleanly than the contextual log probabilities from the full few-shot prompt, and it needs no labels, validation set, or corpus-level assumptions.","core_discovery":"The central discovery is that while naive in-context learning gives similar log probabilities to correct and incorrect outputs, the same outputs become separable when the in-context examples are removed and only the task instruction conditions the score. OptiSeq formalizes this with $\\Phi_k = \\sum_i \\log P(x_{ik} \\mid I \\oplus x_{j