{"id":"8a2092ef-666b-45ae-8ec3-2439be71ed72","arxiv_id":"2502.10760","paper_version":2,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Optimal conditioning sequences for sequence predictors are often unintuitive, better explained by pretraining distribution than task similarity, and popular demonstration-based prompts are frequently suboptimal even under exhaustive search.","lead":"This paper models prompting as selecting optimal conditioning sequences for near-optimal sequence predictors and shows via controlled binary experiments that good prompts are often unintuitive and tied to the pretraining distribution. A smart generalist might read it to see why common techniques like task demonstrations frequently underperform and how pretraining statistics shape prompt effectiveness.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Binary predictors on known distributions may not reproduce LLM in-context dynamics","rationale":"The reader's weakest assumption correctly isolates the generalization step. The toy-model results can be internally valid, but the load-bearing link to practical neural predictors and frontier models is the untested transfer of conditioning statistics; the proposed check directly tests whether that transfer holds.","tokens_in":1660,"tokens_out":333,"duration_ms":19107,"concrete_test":"Train a 4-layer transformer on the exact same binary pretraining distribution used in the paper's main experiments; recompute the exhaustive-search optimal conditioning sequences for the same tasks and compare their structure and performance gap versus task demonstrations to the binary-predictor results. If the transformer optima differ in length, composition, or suboptimality of demonstrations by more than 15% relative, the toy-model conclusions do not transfer.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that unintuitive optimal conditioning sequences are explained by the pretraining distribution and that task demonstrations are suboptimal even under exhaustive search—rests on experiments with binary sequence predictors trained to near-optimality on fully known distributions. For this to explain prompting in frontier models, the optimal conditioning behavior must be governed by the same statistical factors. The paper invokes this by framing prompting as 'finding the best conditioning sequence on a near-optimal sequence predictor' and reports similar patterns on frontier models, yet provides no ablation showing that the binary models' attention-free or low-capacity dynamics match the in-context learning of transformers (e.g., no comparison of how context length or token dependencies interact with model scale).","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims that prompting is equivalent to finding the best conditioning sequence on a near-optimal sequence predictor. Through controlled experiments on binary sequence predictors trained to near-optimality on fully known distributions, it shows that optimal conditioning sequences are often unintuitive and explained by the pretraining distribution, that popular methods such as task demonstrations remain suboptimal even under exhaustive search, and that similar patterns hold when the same empirical framework is applied to frontier models.","tokens_in":1788,"tokens_out":348,"duration_ms":20853,"significance":"If the results hold, the work supplies a useful statistical perspective on prompting that complements frontier-model studies by leveraging fully known distributions and exhaustive enumeration to identify suboptimality. The controlled binary-predictor setting and the reported replication of patterns on frontier models are explicit strengths that allow precise, falsifiable observations about pretraining-distribution effects.","major_comments":[{"comment":"The central extension from binary predictors to frontier models rests on the assumption that the conditioning dynamics are governed by the same statistical factors. The manuscript invokes this framing in the abstract and introduction but provides no ablation or quantitative comparison (e.g., of context-length scaling or token-dependency effects) between the attention-free binary models and transformer in-context behavior, leaving the representativeness claim load-bearing yet under-supported.","section":"Abstract and introduction framing; § on frontier-model experiments"}],"minor_comments":[{"comment":"Notation for conditioning sequences and pretraining distributions would benefit from an explicit running example early in the methods to improve readability for readers outside the binary-sequence setting.","section":"Methods"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive feedback and for recognizing the strengths of the controlled binary-predictor experiments. We respond to the major comment below.","responses":[{"response":"We agree that the current framing in the abstract and introduction could more precisely delineate the scope of the frontier-model analysis. The manuscript applies the same empirical procedure (exhaustive or targeted search over conditioning sequences) and reports qualitatively similar patterns, but does not assert that the underlying statistical factors or scaling behaviors are identical across architectures. The binary setting supplies known distributions and exhaustive enumeration; the frontier-model results are presented as an existence check that the observed phenomena are not artifacts of the simplified model class. To address the concern, we will revise the abstract, introduction, and discussion to (i) state explicitly that we observe analogous patterns without claiming mechanistic equivalence, (ii) note the architectural differences (attention-free vs. transformer attention), and (iii) acknowledge the absence of direct ablations on context-length scaling or token dependencies. These changes will clarify that the representativeness claim is limited to the recurrence of the reported qualitative phenomena.","revision_made":"yes","referee_comment":"[Abstract and introduction framing; § on frontier-model experiments] The central extension from binary predictors to frontier models rests on the assumption that the conditioning dynamics are governed by the same statistical factors. The manuscript invokes this framing in the abstract and introduction but provides no ablation or quantitative comparison (e.g., of context-length scaling or token-dependency effects) between the attention-free binary models and transformer in-context behavior, leaving the representativeness claim load-bearing yet under-supported."}],"tokens_in":1219,"tokens_out":347,"duration_ms":35064,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is that optimal conditioning sequences on these binary predictors turn out to be unintuitive and directly traceable to the pretraining distribution, which makes standard task demonstrations suboptimal even after exhaustive search. The paper frames prompting as picking the best conditioning sequence for a near-optimal predictor and then runs controlled experiments on fully known binary distributions to show this. They also check the same setup on frontier models and report matching patterns. That controlled setup is the real contribution. It lets them know the exact distribution, search every possible prompt, and tie the results back to pretraining statistics without the usual black-box issues. This moves the discussion past pure trial-and-error and gives a concrete way to think about why certain conditionings work. The experiments are well-designed for isolating the distribution effect, and the authors are clear that real pretraining distributions are rarely known. The softer part is the link to actual large models. The binary predictors are low-capacity and lack attention, so their conditioning behavior may not capture how transformers handle long context or token dependencies at scale. The abstract says similar patterns appear on frontier models, but without ablations that test whether the same statistical drivers dominate once capacity and architecture change, the explanation for practical prompting stays partly observational. If those dynamics differ, the binary results explain a simplified case more than they explain frontier-model prompting. This is for people working on the statistical side of in-context learning. It deserves peer review because the experimental approach is fresh and the binary results are reproducible on their own terms, even if the generalization step needs more work.","headline":"Binary predictors give a clean statistical account of why some prompts beat task demos, but the step to frontier models stays tentative.","tokens_in":2278,"tokens_out":376,"would_cite":false,"duration_ms":21185,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"Prompt optimization on binary sequence predictors has no overlap with RS forcing from distinction to J-cost/phi/periodicity","alignment":"orthogonal","rationale":"The paper models prompting as conditioning near-optimal sequence predictors on hierarchical generative processes (latent τ drawn from pτ, then tokens from px|τ). Its central results concern how pretraining meta-distributions bias optimal conditioning sequences, why task demonstrations are suboptimal, and unreliability of empirical prompt search. None of this machinery invokes or parallels the RS chain: reality_from_one_distinction, AbsoluteFloorClosure, J(x) = ½(x + x⁻¹) − 1, phi-ladder constants, 8-tick periodicity, or AlexanderDuality forcing of D=3. The domain (in-context conditioning of predictors) lies outside RS theorems on cost convexity, recognition lattices, or parameter-free derivation of physical constants.","tokens_in":63922,"confidence":"high","tokens_out":199,"duration_ms":8173,"cache_read_input_tokens":32896,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Optimal conditioning sequences for binary sequence predictors are often unintuitive and explained better by the pretraining distribution than by the target task.","keywords":["prompting","conditioning sequences","sequence prediction","pretraining distribution","binary predictors","in-context learning","frontier models"],"falsifier":"A binary sequence experiment in which the empirically optimal conditioning sequence is not the one predicted by the pretraining distribution or is consistently beaten by task demonstrations.","tokens_in":2563,"feed_emoji":"","tokens_out":654,"duration_ms":23723,"temperature":0.7,"pith_summary":"The paper treats prompting as the problem of selecting the best conditioning sequence for a near-optimal sequence predictor. In controlled experiments with binary sequence predictors whose pretraining distributions are fully known, optimal prompts frequently appear unrelated to the downstream task and outperform standard choices such as task-specific demonstrations. Even exhaustive search over conditioning sequences fails to make optimal prompt identification reliable. The same empirical setup applied to frontier models yields analogous patterns. The pretraining distribution therefore supplies the key explanatory lens for why certain prompts succeed.","feed_headline":"Optimal prompts often beat task demonstrations on binary predictors","feed_subtitle":"Exhaustive search shows conditioning sequences explained by pretraining data outperform popular methods even when the target task is known.","key_machinery":"The framing of prompting as search for the best conditioning sequence on a near-optimal sequence predictor, tested via binary sequence predictors with known pretraining distributions.","core_discovery":"We view prompting as finding the best conditioning sequence on a near-optimal sequence predictor. On numerous well-controlled experiments, we show that unintuitive optimal conditioning sequences can be better understood given the pretraining distribution, which is not usually available. Even using exhaustive search, reliably identifying optimal prompts for practical neural predictors can be surprisingly difficult. Popular prompting methods, such as using demonstrations from the targeted task, can be surprisingly suboptimal. Using the same empirical framework, we analyze optimal prompts on frontier models, revealing patterns similar to the binary examples and previous findings.","pith_inferences":["If the pretraining distribution governs optimal conditioning in simple cases, prompt design for large models may benefit from explicit modeling of training-data statistics rather than task-specific heuristics.","The difficulty of exhaustive search suggests that future prompt-finding algorithms should incorporate distributional priors instead of treating the search as unstructured.","The binary-predictor setup isolates the effect of conditioning from other model behaviors, offering a test bed for theories of in-context learning that focus on sequence statistics."],"forward_implications":["Popular methods that rely on task demonstrations can remain suboptimal even after exhaustive search over alternatives.","The pretraining distribution supplies a systematic way to predict which conditioning sequences will perform well.","Reliable identification of optimal prompts stays difficult for predictors that are only approximately optimal.","Patterns observed on binary predictors appear again when the same search procedure is run on frontier models."],"fun_headline_variants":["Task demonstrations underperform optimal conditioning sequences","Pretraining distribution explains unintuitive optimal prompts","Exhaustive search fails to reliably identify optimal prompts","Prompting remains difficult for near-optimal sequence predictors"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Binary sequence predictors trained to near-optimality on fully known distributions behave like the conditioning mechanisms inside practical neural predictors and frontier models.","fun_headline_variants_meta":{"raw":{"variants":["Task demonstrations underperform optimal conditioning sequences","Pretraining distribution explains unintuitive optimal prompts","Exhaustive search fails to reliably identify optimal prompts","Prompting remains difficult for near-optimal sequence predictors"]},"model":"grok-4.3","cost_usd":0.008574,"raw_usage":{"total_tokens":3844,"prompt_tokens":614,"num_sources_used":0,"completion_tokens":47,"cost_in_usd_ticks":85737000,"prompt_tokens_details":{"text_tokens":614,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3183,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":614,"tokens_out":47,"duration_ms":38531,"temperature":1.0,"reasoning_tokens":3183,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-23T02:35:11.225171+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A binary sequence experiment in which the empirically optimal conditioning sequence is not the one predicted by the pretraining distribution or is consistently beaten by task demonstrations.","supporting_citations":[],"review_version":1}