{"id":"407613bc-e1cf-49c8-897f-1f0feeb95fca","arxiv_id":"2505.13307","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"RBF++ models reasoning limits as harmonic-mean boundaries and uses them to explain, predict, and improve chain-of-thought performance across 38 models and 13 tasks.","lead":"This paper proposes RBF++, a framework that defines a \"reasoning boundary\" as the hardest problem a model can solve step by step, and combines sub-boundaries with a harmonic-mean formula. It applies this to text and image-based reasoning tasks and introduces MARP and MARP++ prompts that improve accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The harmonic-mean combination law (Eq. 3) is a restatement of the additivity and independence assumptions in Appendix A, and its validation relies on unreported free parameters N_i, b_i, so the central quantitative claim is not actually tested.","rationale":"The paper's central claim is that Eq. (3) quantitatively predicts combined CoT reasoning boundaries from measurable sub-boundaries, and that this law explains when prompting methods help. That claim is load-bearing because §IV-D and §VI-D use it to explain PoT/TU and LtM/CCoT behavior, and §VI uses it with constants to quantify unmeasurable multimodal boundaries. The law rests entirely on Appendix A's Definition 1 (additive difficulty) and Assumption 5 (independence). Neither is derived or empirically justified; they are posited. The proof is a first-order Taylor expansion with constants dismissed on empirical authority and with free N_i,b_i left unreported. Consequently, the matching curves in Figures 3 and 11 are curve fits, not predictions. The constant assumption (Eq. 4) makes the framework even more flexible by converting unmeasurable boundaries into free constants. I agree with the reader's identification of the weakest assumption; my check would force the authors to report parameters and test held-out predictive power against a flexible alternative. The engineering contributions — BigGSM++, MARP, MARP++, and the broad model comparison — are real and may be salvageable, but they do not rescue the quantitative law, which is the advertised core of RBF++. Hence the reader's REJECT remains appropriate, though it might be softened to CONDITIONAL if the authors supply parameter-free or pre-registered validation, since the empirical scaffolding is otherwise substantial.","tokens_in":24152,"tokens_out":5019,"duration_ms":44762,"concrete_test":"Estimate N_i, b_i in Eq. (2) on one half of the BIGGSM difficulty grid in §IV-B.2 and use them to predict the combined boundary B(c,p) on the held-out half. Report RMSE and compare against a power-law combination B = (Σ_i (B_i)^α)^{1/α} with α fit to the training half. If the harmonic mean (α=-1) does not predict held-out data at least as well as the power-law alternative, or if no parameter values are reported, the central law is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is Eq. (3), the harmonic-mean combination law. It does not follow from the stated definitions. Appendix A's proof assumes additive difficulty (Definition 1) and mutual independence (Assumption 5), then Taylor-expands to first order and drops the constants k'_i and k0 on the empirical assertion that they vanish. Even granting that expansion, Eq. (21) has the form B ≈ N0 / Σ_i [N_i1/(t_i-b_i) - N_i1 k_i], which is not Eq. (23) unless the k_i are absorbed into the free parameters b_i and N_i1. With unspecified N_i and b_i, Eq. (3) is a flexible curve rather than a law: any monotone relationship between sub-boundaries and the combined boundary can be fit. The experiments in §IV-B and §VI-A overlay predicted and observed boundaries but report no parameter values, no fitted residual, and no baseline comparison, so the agreement is not a test. The qualitative explanations in §IV-D and §VI-D (why PoT/TU help in text but not multimodal) inherit this problem: they compare limits of Eq. (3) under B(c)→∞, so if the law is wrong the explanations lose quantitative grounding. The framework also introduces the constant assumption (Eq. 4) and division mechanism (Eqs. 12-13) with scenario-specific constants, which can absorb any discrepancy. The paper itself concedes in the Fig. 4 caption that all CIRB samples in the Synthetic-CoT analysis are special-value points and no real CIRB samples were obtained, weakening the self-awareness evidence. The load-bearing premise is therefore not independently secured: additivity and independence of sub-difficulties are asserted, and the empirical checks are fits with free parameters.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RBF++, a framework for quantifying and optimizing the reasoning boundaries (RBs) of LLMs in chain-of-thought reasoning. The main components are: a definition of RB as the maximum task difficulty at which a model maintains a target accuracy; a \"combination law\" stating that the combined RB is approximately the reciprocal of the sum of reciprocals of sub-RBs (Eq. 3); a constant assumption for unmeasurable boundaries in multimodal settings (Eq. 4); a reasoning-boundary division mechanism (Eqs. 5-7, 12-13); and a three-way categorization of boundaries (CFRB, PFRB, CIRB). The paper also contributes the BigGSM and BigGSM++ benchmarks, the MARP and MARP++ prompting methods, and a broad empirical study across 38 models and 13 tasks. The authors claim the framework explains why strategies such as Program-of-Thought, Tool Usage, Complex-CoT, and Least-to-Most succeed or fail in different modalities.","tokens_in":24428,"tokens_out":6800,"duration_ms":70329,"significance":"If the harmonic-mean combination law were a validated quantitative law, the paper would provide a practically useful theory for predicting and optimizing CoT performance, and the MARP/MARP++ methods show concrete accuracy gains. The paper has notable strengths: a large experimental scope, detailed benchmark-construction procedures, clear prompting recipes, and a reproducible code/data release. However, the central quantitative claim is not currently established: the derivation in Appendix A restates the model's assumptions, and the empirical validation relies on unreported free parameters and visual inspection. The significance of the framework is therefore conditional on a serious revision that either reframes the combination law as an explicit, testable assumption or derives it from a more substantive model of accuracy and difficulty.","major_comments":[{"comment":"The harmonic-mean combination law is a restatement of the model's assumptions rather than a derived theorem. Definition 1 stipulates that difficulty is additive, D(t1,t2)=D(t1)+D(t2), and Definition 2 makes RB the reciprocal of difficulty. Consequently, 1/B(t1,...,tn)=Σ 1/B(ti) follows algebraically, and the Taylor expansion in Eqs. (18)-(22) adds no content. The proof also discards k'_i and k0 on the empirical assertion that they vanish, and at Eq. (23) it substitutes B(t_i) for t_i as a \"quantized substitute,\" which reintroduces the reciprocal-of-difficulty identification. As written, Eq. (3) is an identity under the definitions, not a falsifiable prediction. Please either present the combination law as an explicit axiom whose empirical content is stated clearly, or derive it from a model of accuracy as a function of difficulty.","section":"Appendix A, Definitions 1-2, Eq. (23)"},{"comment":"The claimed verification of the combination law is visual only. The manuscript does not report fitted values of N_i, b_i, or z, no confidence intervals, no goodness-of-fit statistics, and no residuals. It also does not compare the harmonic-mean law against alternative combination rules (e.g., arithmetic mean, geometric mean, max/min). Because Eq. (2) contains per-task scaling factors N_i and b_i, overlaying a theoretical line on a scatter of observed samples cannot distinguish the law from an interpolating curve. Please report parameter estimates with uncertainty, perform a quantitative model comparison, and evaluate predictions on held-out data rather than on the same tasks used to estimate parameters.","section":"§IV-B, §VI-A, Figs. 3 and 11"},{"comment":"The constant assumption makes the division-mechanism validation hard to falsify. In Eq. (11), z1 is measured from direct-prompt performance on the same task that is then used to check Eq. (11); in Eq. (13), the multimodal perception RB is absorbed into an unspecified constant z'. A scenario-specific constant fitted to the same data can absorb systematic discrepancies between predicted and observed boundaries. The paper should specify how z is estimated (for example, from a calibration task disjoint from the evaluation task) and test how the conclusions depend on the choice of z.","section":"§VI-A/B, Eqs. (4), (11), (13)"},{"comment":"The claim that LLMs have self-awareness of their own reasoning boundaries is weakened by the paper's own caption: \"all samples in CIRB are special value points, like 25000×1000. In fact, no real CIRB samples are obtained.\" The finding that generated samples concentrate in CFRB is compatible with the model avoiding implausible or extreme numeric values rather than knowing its own reasoning boundary. This limitation should be acknowledged in the main text, and the self-awareness claim should be softened accordingly.","section":"§IV-C.4, Fig. 4 caption"},{"comment":"The explanations of why Tool Usage and Program-of-Thought help in text but not multimodal settings, and why Complex-CoT and Least-to-Most behave as observed, are all obtained by taking limits of Eq. (3). Since Eq. (3) is not independently validated, these explanations do not provide independent support for the framework; they are conditional consequences of the same combination law. The paper should state this explicitly, rather than presenting these analyses as confirmatory evidence.","section":"§IV-D, §VI-D, Eqs. (9), (13)"}],"minor_comments":[{"comment":"The formal definition B^{Acc=K1}(t|m) = sup{d | Acc(t|d,m) ≤ K1} conflicts with the accompanying text, which says the boundary is the greatest difficulty at which accuracy exceeds K1. The inequality sign should be reversed or the definition should be reworded.","section":"Eq. (1)"},{"comment":"Notation is inconsistent: the paper uses B^{Acc=K1}(t|m), BCoT(c,p), BT(c,p), B(p,o,v), and B(o,p,v) without a single defining list. Please collect all boundary symbols and their meanings in one table or definition block.","section":"Notation throughout"},{"comment":"The Taylor expansion writes D(x1,...,xn) as a sum of univariate expansions in each xi and omits constant and cross terms without justification. The notation O(x_i) is also undefined for vector arguments; please clarify the expansion and its remainder terms.","section":"Appendix A, Eq. (18)"},{"comment":"The statement that temperature is chosen from the interval [0,1] with temperature serving as the primary error variable is vague. Please specify the exact sampling protocol, number of runs, and how randomness is otherwise controlled.","section":"§III, Experimental Setup"}],"recommendation":"major_revision","confidential_remarks":"The paper is a substantial expansion of a NeurIPS 2024 paper, and the new material on multimodal constants, division mechanisms, and MARP++ is interesting. My main concern is that the central combination law is presented as a proven and validated result when it is currently a restatement of the paper's own definitions, with the empirical check lacking quantitative rigor. I believe this is fixable within the manuscript's scope if the authors reframe the law as an explicit testable assumption, report all fitted parameters and model comparisons, and temper the self-awareness and explanation claims. If such a revision is not possible, the contribution reduces to a vocabulary and a prompt-tuning recipe rather than a quantitative theory."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the central combination law (Eq. 3) is not tested — it is a consequence of the authors' own definitions. Appendix A defines difficulty as additive and RB as reciprocal difficulty, so the harmonic mean follows by construction. The Taylor expansion that supposedly proves it introduces N_i, b_i, k'_i, k0 and then drops the k terms because \"experimental results\" show they vanish, without showing the numbers. The validation figures overlay predicted and observed boundaries but report no parameter values, no error bars, no residual, and no alternative fit. So the paper's load-bearing quantitative claim is a flexible curve, not a predictive law. The stress-test note is right about that.\n\nBut the paper does contain real, usable new engineering. BigGSM++ is a reasonable stress benchmark for reasoning LLMs. MARP and MARP++ look effective on M3CoT, with the gains in Table II being concrete. The comparison of DeepSeek-R1 versus o3-mini boundary shapes is a genuinely interesting observation, even if it is interpreted through the shaky law. Code and data are promised. The authors are also honest about the weak spots: the Fig. 4 caption admits the Synthetic-CoT CIRB analysis rests on special-value points and no real CIRB samples were obtained. Credit where due.\n\nThe soft spots, in proportion: the constant assumption z and division mechanism z' are scenario-specific values that can absorb discrepancies. The qualitative explanations of PoT/TU/LtM success and failure are all limits of Eq. (3), so they are only as strong as the law. The Appendix's step from Eq. (21) to Eq. (23) requires free parameters to absorb the k_i terms; the paper never explains how N_i and b_i are set. That is the heart of the problem — without those values, the \"agreement\" in Figs. 3 and 11 is not a test.\n\nWho is this for? Practitioners who want a benchmark, a prompt recipe, and a plausible story about why tool use helps in text but not vision. Theorists will not be satisfied. The right move is to send it to review: the artifact value is high, and a referee can demand the parameters be reported and the law be tested against, say, a simple product or sum combination. As published, the central law is not established. If the authors can deliver that test, the framework would be worth taking seriously.","headline":"A useful empirical extension wrapped around a combination law that is a restatement of its own definitions, with validation that is curve-fitting rather than a test.","tokens_in":25080,"tokens_out":2382,"would_cite":false,"duration_ms":23062,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A weighted harmonic-mean law predicts when chain-of-thought reasoning succeeds and fails.","keywords":["reasoning boundary","chain-of-thought","large language models","multimodal reasoning","prompt optimization","harmonic mean","reasoning limits","MARP prompting"],"falsifier":"Construct a task with two measurable sub-boundaries in which the second sub-task's input is the output of the first, so an error in sub-task 1 changes the difficulty of sub-task 2 (e.g., a multi-step calculation where an early arithmetic slip makes later operands much larger). Measure $B(t_1)$ and $B(t_2)$ separately, compute the Eq. (3) prediction for the combined boundary, then measure the actual combined boundary. If the actual boundary falls systematically below the predicted value as the dependence between sub-tasks is strengthened, the additive-independence assumption is false and the combination law is falsified for that regime.","tokens_in":23861,"feed_emoji":"🧠","tokens_out":9504,"duration_ms":84457,"temperature":0.7,"pith_summary":"This paper tries to establish that a large language model's chain-of-thought performance on a combined task is predictable from the model's separate reasoning boundaries on the task's sub-skills: the combined reasoning boundary is approximately the weighted harmonic mean of the sub-boundaries, $B(t_1,\\dots,t_n)\\approx 1/\\sum_i 1/B(t_i)$. A reasoning boundary is defined as the hardest difficulty at which accuracy stays above a chosen threshold, so the law converts the vague idea of a model's reasoning limit into a quantity one can measure and test. The paper reports that this law holds for arithmetic, planning, multi-hop question answering, and multimodal reasoning, and that unmeasurable boundaries can be treated as scenario-specific constants. If the law is right, it explains why program-of-thought and tool use help in text but not multimodal settings, and it justifies prompt designs such as MARP/MARP++ that keep reasoning inside measured feasible regions.","feed_headline":"LLM reasoning limits follow one combination law","feed_subtitle":"From arithmetic to multimodal tasks, the paper predicts when chain-of-thought succeeds and when it fails.","key_machinery":"The load-bearing object is the reasoning boundary, $B^{\\text{Acc}=K}(t|m)=\\sup\\{d:\\text{Acc}(t|d,m)\\ge K\\}$, i.e. the hardest task level a model manages at a fixed accuracy target. The load-bearing identity is the combination law $B(t_1,\\dots,t_n)\\approx 1/\\sum_i 1/B(t_i)$, which the paper derives in Appendix A from additive difficulty and independence. This single reciprocal relation does three jobs: it predicts combined performance from measured sub-boundaries, it lets unmeasurable sub-boundaries be absorbed into a constant $z_i$ in Eq. (4), and it makes boundary division possible because one combined boundary can be split into sub-boundaries that recombine by the same formula. The three boundary categories — completely feasible (accuracy $\\ge 90\\%$), partially feasible ($10\\% <$ accuracy $< 90\\%$), and completely infeasible (accuracy $\\le 10\\%$) — convert the identity into optimization guidance: operate inside the first, use consensus or decomposition in the second, and treat the third as requiring a capability change rather than a prompt change.","core_discovery":"The paper's central claim is that every task has a reasoning boundary $B^{\\text{Acc}=K}(t|m)$, the greatest difficulty level $d$ at which accuracy exceeds threshold $K$, and that when a task combines sub-tasks $t_1,\\dots,t_n$, the combined boundary satisfies $B(t_1,\\dots,t_n)\\approx 1/\\sum_i 1/B(t_i)$ under a normalization. The derivation starts from an assumption that difficulty is additive and sub-boundaries are independent, then uses the reciprocal relation to justify the harmonic-mean formula. The same formula is extended to unmeasurable capabilities: a constant $z_i$ replaces any sub-boundary that cannot be measured directly, and a unified boundary such as $B(o,p,v)$ can be divided into finer boundaries $B(k)$ and $B(m_m)$ that still combine by the same law. Across 38 models and 13 tasks the paper reports that the predicted 90% and 10% boundary curves align with observed accuracy transitions, and that the three induced regions behave consistently: zero-shot rationales are correct mainly inside the completely feasible region, self-consistency voting helps mainly in the partially feasible region, and the completely infeasible region stays near chance even with voting. Based on this, the paper explains the success or failure of ten chain-of-thought strategies and proposes MARP/MARP++ prompting to stay within the feasible boundary.","pith_inferences":["If the harmonic-mean law holds generally, a model's combined reasoning capacity is limited by its weakest sub-skill, so optimization effort should target the smallest sub-boundary first; the paper does not formulate this budgeting rule explicitly.","The constant assumption suggests a direct perception-focused experiment the paper does not run: fix the text reasoning and vary image difficulty systematically, and check whether Eq. (13) predicts the boundary shift with a single fitted constant across all difficulty levels.","The same reciprocal law could be applied to multi-model systems where sub-tasks are executed by different agents or models; the prediction would be that the system boundary is dominated by the weakest component, which would be a testable implication for agent orchestrators.","Because the law assumes independence, real tasks with correlated errors (e.g., cascading mistakes in a long plan) are where the framework should over-predict; finding a parametric correction for correlated sub-boundaries would be a natural extension."],"forward_implications":["Given separately measured sub-boundaries, the combined boundary can be predicted without running the combined task, so a model's likely performance on a new multi-step task can be estimated in advance.","Tool use and program-of-thought improve textual reasoning because they push the calculation sub-boundary toward infinity, making the combined boundary equal to the planning boundary; in multimodal settings the perception constant is not expanded, so the same strategies fail.","Decomposing problems with Complex-CoT or Least-to-Most helps only while the smaller steps stay within the planning boundary; beyond that point added planning pressure erases the gain, which explains the observed performance reversal.","Self-consistency voting is predicted to improve accuracy only inside the partially feasible region, not in the already-mastered or hopeless regions, so sampling more reasoning paths has a region-dependent payoff.","For reasoning-tuned models, a large gain in the completely infeasible region with little gain in the completely feasible region explains why a model like o3-mini can outperform on hard problems yet trail on medium-difficulty benchmarks."],"supporting_citations":[{"why":"The preliminary RBF paper supplies the reasoning-boundary definition, the combination law, and the MARP prompt design that this work extends to multimodal and reasoning-LLM settings.","marker":"[1]"},{"why":"Introduced chain-of-thought prompting, the paradigm whose measurable and unmeasurable boundaries this paper quantifies.","marker":"[11]"},{"why":"Provides the M3CoT multimodal multi-domain benchmark used to validate the constant assumption and the boundary division mechanism.","marker":"[17]"},{"why":"PAL supplies the code-planning setup used to compare natural-language and code planning and to explain why program-of-thought raises the planning boundary.","marker":"[24]"},{"why":"Supports the decomposition of mathematical reasoning into separate step-planning and step-calculation sub-boundaries.","marker":"[25]"},{"why":"HotpotQA is the multi-hop question-answering dataset on which the combination law is verified beyond mathematics.","marker":"[27]"},{"why":"Self-consistency voting is the method used to test the nature of the partially feasible reasoning boundary.","marker":"[29]"},{"why":"Program-of-thought prompting is the strategy whose boundary increase in text and failure in multimodal settings the framework explains.","marker":"[32]"},{"why":"Complex-CoT is the step-fragmentation strategy whose boundary trade-off (lower calculation load, higher planning pressure) is analyzed.","marker":"[35]"},{"why":"Least-to-most prompting is the decomposition strategy analyzed in both text and multimodal settings as trading local for global planning pressure.","marker":"[40]"}],"fun_headline_variants":["Reasoning limits follow a harmonic-mean combination law","One law predicts when chain-of-thought succeeds or fails","RBF++: LLM reasoning boundaries obey a single formula","Measurable and unmeasurable reasoning share one boundary law"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is the additive-difficulty assumption of Definition 1 and the mutual-independence assumption of Assumption 5 in Appendix A: $D(t_1,t_2)=D(t_1)+D(t_2)$ with sub-boundaries independent; if real reasoning difficulty is non-additive — for instance when later steps depend on earlier answers so errors compound superlinearly, or when the model spends one shared effort budget across sub-tasks — then the harmonic-mean law does not follow and the paper's quantitative boundary predictions collapse.","fun_headline_variants_meta":{"raw":{"variants":["Reasoning limits follow a harmonic-mean combination law","One law predicts when chain-of-thought succeeds or fails","RBF++: LLM reasoning boundaries obey a single formula","Measurable and unmeasurable reasoning share one boundary law"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00033,"raw_usage":{"total_tokens":1928,"prompt_tokens":1125,"completion_tokens":803,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":741,"completion_tokens_details":{"reasoning_tokens":736}},"tokens_in":741,"tokens_out":803,"duration_ms":8975,"temperature":1.0,"reasoning_tokens":736,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:16:21.987804+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a task with two measurable sub-boundaries in which the second sub-task's input is the output of the first, so an error in sub-task 1 changes the difficulty of sub-task 2 (e.g., a multi-step calculation where an early arithmetic slip makes later operands much larger). Measure $B(t_1)$ and $B(t_2)$ separately, compute the Eq. (3) prediction for the combined boundary, then measure the actual combined boundary. If the actual boundary falls systematically below the predicted value as the dependence between sub-tasks is strengthened, the additive-independence assumption is false and the combination law is falsified for that regime.","supporting_citations":[{"cited_title":"Unlocking the capabilities of thought: A reasoning boundary framework to quantify and optimize chain-of-thought,","cited_arxiv_id":null,"evidence_quote":"The preliminary RBF paper supplies the reasoning-boundary definition, the combination law, and the MARP prompt design that this work extends to multimodal and reasoning-LLM settings."},{"cited_title":"M 3CoT: A novel benchmark for multi-domain multi-step multi-modal chain-of- thought,","cited_arxiv_id":null,"evidence_quote":"Provides the M3CoT multimodal multi-domain benchmark used to validate the constant assumption and the boundary division mechanism."},{"cited_title":"PAL: Program-aided language models,","cited_arxiv_id":null,"evidence_quote":"PAL supplies the code-planning setup used to compare natural-language and code planning and to explain why program-of-thought raises the planning boundary."},{"cited_title":"Causal abstraction for chain-of-thought reasoning in arithmetic word problems,","cited_arxiv_id":null,"evidence_quote":"Supports the decomposition of mathematical reasoning into separate step-planning and step-calculation sub-boundaries."},{"cited_title":"Hotpotqa: A dataset for diverse, explainable multi-hop question answering,","cited_arxiv_id":null,"evidence_quote":"HotpotQA is the multi-hop question-answering dataset on which the combination law is verified beyond mathematics."},{"cited_title":"Complexity- based prompting for multi-step reasoning,","cited_arxiv_id":null,"evidence_quote":"Complex-CoT is the step-fragmentation strategy whose boundary trade-off (lower calculation load, higher planning pressure) is analyzed."},{"cited_title":"Least-to-most prompting enables complex reasoning in large language models,","cited_arxiv_id":null,"evidence_quote":"Least-to-most prompting is the decomposition strategy analyzed in both text and multimodal settings as trading local for global planning pressure."}],"review_version":1}