{"id":"ca78a0e8-653a-4376-baa8-e5bff0036e66","arxiv_id":"2505.07728","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Factored scaling curves that rank environmental factors by predicted marginal success gain allocate a fixed robot data budget more effectively than equal, greedy, or robust-mixture baselines in simulation and real-world manipulation tasks.","lead":"Robot teams usually guess which variations, like lighting or table texture, to collect more examples for. This paper fits a separate performance curve for each variation and spends the data budget on the one predicted to give the biggest success-rate gain, with an offline option that avoids running the robot during the choice.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Real-world FSC factor rankings are fit from 12-20-trial success rates; evaluation noise can flip the top factor and is not bounded by error bars or bootstrap, so the headline gains may not be reproducible.","rationale":"The reader's weakest_assumption is close to mine, but I would sharpen the emphasis: the binding uncertainty in the real-world experiments is not the power-law form per se. The simulation results with 3000-4800 trials per curve point provide real support that the extrapolation can track held-out points in Appendix A.5, so the parametric form is at least partially validated. The weak link is instead the precision of the real-world points used to select factors, and the absence of any uncertainty quantification around those selections. With 12-20 trials per point, the standard error is roughly 0.11-0.14 at p=0.5. The slopes P^K_ij compare differences of such noisy points, and the Top strategy is a maximum over those noisy slopes, so a one-trial shift can change the recommendation. Figure 4 even shows a distorting outlier at n=60, yet no error bars or bootstrap are provided. Table 7 independently demonstrates that at larger budgets the extrapolation can fail, which should bound the generalized claim. The simulation evidence is broad and credible, so I do not want to reject the paper; the real-world central claim is not fully secured without per-trial data, confidence intervals, or a bootstrap analysis. This matches the reader's CONDITIONAL verdict, so no change to the verdict is needed. If the proposed bootstrap test shows stable rankings, the concern would be resolved and the paper's real-world claims would be substantially stronger.","tokens_in":19593,"tokens_out":9306,"duration_ms":103392,"concrete_test":"Run a bootstrap stability check on the real-world FSC fits: from the per-trial outcomes used to compute each curve point (12-20 trials per policy), resample trials with replacement 10,000 times, refit Eq. (5) for every factor pair, and recompute P^K_ij for K=20 and K=100. Record how often the top-ranked pair chosen by the Top strategy matches the original selection. If the original top pair is not the bootstrap majority in any of the three real-world tasks, the headline real-world gains are not distinguishable from evaluation noise. A stronger variant is to collect 50-60 trials per policy on one of the real-world tasks and check whether the ranking and final success gaps survive; the bootstrap is the cheaper first test and would settle whether noise, rather than the power-law form, is the binding failure mode.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 4.2 is that factored scaling curves can accurately predict how policy performance changes with additional factor data. Operationally, this requires ranking the slopes P^K_ij (Eq. 12) obtained from power-law fits (Eq. 5) using as few as four points. The weakest load-bearing step is the precision of those points in the real-world experiments: Appendix C.4 states each policy is evaluated with roughly 12-20 trials. A 12-18 trial success estimate has standard error approximately 0.11-0.14 at success rate 0.5, which is the same magnitude as the predicted improvements and the FSC-vs-baseline gaps reported in Figure 3. Because the Top strategy commits the entire budget to the single highest-slope factor pair, evaluation noise can change the fitted exponent b and flip the recommended factor. Figure 4 explicitly notes an outlier at n=60 in Fold Towel - Visual that slightly distorts the fit, but no confidence intervals, error bars, or bootstrap analyses are provided to show that the selected ranking is stable. Table 7 is an additional red flag: at K=500 on Peg Insertion - Spatial with a 480-demo initial set, prediction-driven allocation is worse than Equal, so the extrapolation is not generally reliable even in simulation. Since no code or data is shipped, the real-world advantage over baselines rests on point estimates whose sampling noise is comparable to the claimed margins. The simulation evidence is much stronger (3000-4800 trials per point, Appendix B.3) and shows the extrapolation can track held-out points, but it validates the method for scripted-collected diffusion-policy data, not for the real-world fine-tuned pi0 regime where each fitted point is only 12-20 trials.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Factored Scaling Curves (FSC) as a tool for deciding which environmental factors to prioritize when collecting additional demonstrations for imitation-learned robot policies. For each factor or factor pair, the method trains policies on increasing amounts of factor-specific data, evaluates them on a target distribution, fits a power law (Eq. 5), and computes an expected improvement slope (Eqs. 7 and 12). The budget is then allocated by Top, Top-Half, or All strategies. The paper evaluates FSC in ManiSkill3 simulation and on a Franka robot, using diffusion policies and fine-tuned pi0 models, and also presents FSC-Proxy, an offline variant where curves are built from policy embedding similarity instead of hardware success rates. The main claims are that factored scaling curves accurately predict the effect of additional factor data, and that the resulting data-collection strategy outperforms Equal, Greedy, and Re-Mix by roughly 10% on average in simulation and by up to 26% in real-world tasks.","tokens_in":19934,"tokens_out":5046,"duration_ms":51209,"significance":"If validated, FSC would be a practically useful and policy-aware method for allocating limited data-collection effort, a question of direct relevance to robot learning. The simulation evaluation is a genuine strength: each curve point is averaged over 3000-4800 rollouts (Appendix B.3), and Table 1 shows consistent gains over strong baselines across five tasks and two budget sizes. The paper is also admirably transparent about limitations, including the degradation at large budgets in Table 7. The FSC-Proxy idea, which replaces hardware evaluation with embedding similarity, is interesting and potentially high-impact. The main reservation is that the central predictive claim rests on real-world curves estimated from only 12-20 trials per policy, and the paper's own ablations show regimes in which FSC does not beat Equal; the real-world headline numbers therefore need either stronger uncertainty analysis or a more carefully scoped claim.","major_comments":[{"comment":"The claim that factored scaling curves 'can accurately predict how policy performance changes with additional factor data' is not supported by uncertainty quantification in the real-world experiments. Appendix C.4 states that each policy is evaluated on 12-20 trials (Pick Place: 20; Fold Towel: 12; Mouse in Drawer: 18), so the standard error of a single success-rate estimate is roughly 0.11-0.14 at a success rate near 0.5. This is the same order of magnitude as the predicted improvements and as the FSC-vs-baseline gaps reported in Figure 3. Because the Top strategy commits the entire budget to the single highest-slope factor pair, trial-level noise can change the fitted exponent b in Eq. (5) and flip the recommended factor. Please provide bootstrap confidence intervals around the fitted curves and the resulting factor rankings, or explicitly restrict the predictive claim to settings in which ranking stability can be demonstrated.","section":"Section 4.2 and Appendix C.4"},{"comment":"The paper's own ablations show that FSC's advantage is regime-dependent, which contradicts the unqualified predictive claim in Section 4.2. In Peg Insertion - Spatial with an initial set of 480 demos, prediction-driven allocation falls below Equal at K=500 (All 59.4 vs Equal 63.0; Top-Half 48.4), and in Table 6, with a 300-demo initial set on Peg Insertion - Visual, Top achieves 58.4 at K=20 while Equal achieves 64.2. The Limitations section acknowledges this, but the abstract and Section 4.2 state the predictive accuracy without this scope. Please add a quantitative characterization of when extrapolation fails, for example in terms of budget horizon relative to the fitted range or performance saturation, and adjust the central claims accordingly.","section":"Appendix A.4, Tables 6 and 7"},{"comment":"The selection statistic P^K_ij is derived from a power law fit to as few as four observed points, but the paper reports no goodness-of-fit, no model comparison, and no sensitivity analysis for the fitted exponents. Figure 4 itself notes an outlier at n=60 in Fold Towel - Visual that 'slightly distorts the fit,' which illustrates how sensitive the fitted exponent, and hence the factor ranking, can be to individual curve points. Since the entire allocation decision is a function of the ranking of slopes, please report fit residuals and prediction intervals, and show that the ranking is stable under reasonable perturbations of the curve points or under alternative parametric forms.","section":"Section 3.2, Eq. (5) and Eq. (12)"}],"minor_comments":[{"comment":"The statement that FSC-Proxy 'matches or surpasses' FSC is not supported by Table 4: FSC-Proxy is lower than FSC on Peg Insertion at K=100 (53.3 vs 57.9) and on Pull Cube Tool at K=100 (73.4 vs 83.4). Please soften this claim or add significance testing.","section":"Section 4.4, Table 4"},{"comment":"The claim that Group 'retains the full performance advantage' is overstated for Pick Place at K=100, where Group (64.4) is essentially tied with Equal (64.3) while Pairwise (78.8) is much higher. This does not invalidate the cost argument, but the wording should reflect the task-dependence of the comparison.","section":"Section 4.3, Table 2"},{"comment":"Figure 3 reports real-world success rates without error bars or trial counts, even though Appendix C.4 gives only 12-20 trials per policy. Adding per-policy confidence intervals or raw trial-level data would greatly improve the reader's ability to judge the magnitude and reliability of the reported gains.","section":"Appendix C.4 and Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for the journal and the simulation evidence is solid, but the real-world evaluation is underpowered for the strength of the claims. In revision, I would ask the authors to provide trial-level real-world data or a bootstrap script so the ranking stability and the reported margins can be independently assessed; without this, the headline real-world numbers are not verifiable. No code or data is currently shipped, which is an additional reproducibility concern for a paper whose contribution is an empirical methodology."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nQuick take: this paper gives you a clear, budget-aware answer to \"what additional demonstrations should I collect next?\" by fitting per-factor power-law scaling curves and allocating data to the steepest curve. The factored construction is genuinely new—prior work treats scaling at the aggregate dataset level or optimizes static mixtures, not factor-wise marginal gains. The simulation evaluation is broad and convincing: thousands of rollouts per point, held-out extrapolation checks, and consistent gains over Equal, Greedy, and Re-Mix. The offline proxy (embedding similarity instead of hardware success rate) is a nice practical addition, and the paper's own ablations show it tracks the real metric well enough to pick the top factor.\n\nThe soft spots are real but mostly located. The real-world numbers rest on 12–20 trials per policy, as Appendix C.4 admits; at success rates around 0.5, that is a standard error of 0.11–0.14, which is the same magnitude as the claimed improvements. Without confidence intervals or bootstrap, the factor rankings in Figures 3 and 4 could flip under evaluation noise. The paper's own Table 7 is the clearest sign that the power-law extrapolation is not universally reliable: at K=500 on Peg Insertion, prediction-driven allocation falls about 10% behind Equal. Table 6 shows FSC loses its edge once the initial dataset is large enough that performance saturates. The authors do flag the large-K limitation in their conclusion, which I respect, but the main text's claim that curves \"accurately predict\" performance is stronger than the evidence supports for the real-world regime.\n\nAlso, no code or data is shipped. For a method whose value is in practical adoption, that matters.\n\nNone of this kills the paper. The simulation evidence is solid, the method is simple enough to implement, and the offline proxy is a genuine step toward reducing evaluation cost. The paper deserves a serious referee—not a desk reject. The referee should push for error bars or bootstrap on the real-world curves, a pre-registered allocation rule, and public artifacts. If those land, I would be comfortable citing it as the go-to for factor-aware data collection.\n\nFor a reading group, it is worth a slot if people care about data efficiency in manipulation. My own verdict would be conditional with those revisions.","headline":"A practical data-collection method that mostly holds up in simulation; the real-world evidence is thinner than the headline margins suggest, but the idea and offline proxy are worth engaging.","tokens_in":20506,"tokens_out":1757,"would_cite":true,"duration_ms":17266,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Factored scaling curves predict which demonstration data will most improve a robot policy.","keywords":["factored scaling curves","data collection","imitation learning","robot manipulation","scaling laws","power-law extrapolation","policy embedding similarity","budget allocation"],"falsifier":"Take a task and initial dataset, fit curves from four sizes such as $n=0,20,40,60$, predict the top factor at budgets $K=100$ and $K=500$, actually collect that much data for the top-ranked factor and for a lower-ranked factor, and evaluate on held-out factor combinations. The central claim fails if the observed gain from the top-ranked factor is no larger, within statistical error, than the gain from a lower-ranked factor or from an equal split, or if the measured success falls outside the fitted curve's prediction interval.","tokens_in":19405,"feed_emoji":"📈","tokens_out":11764,"duration_ms":103708,"temperature":0.7,"pith_summary":"Factored scaling curves are the paper's answer to a practical problem: when a robot policy performs poorly in new environments, which additional demonstrations should be collected, and how many, under a fixed budget? The paper models each environmental factor, such as lighting, table texture, distractors, camera pose, or object pose, with its own curve mapping the amount of factor-specific demonstration data to policy success rate on the target distribution. A fitted power law is extrapolated to the budget horizon, and the steepest predicted slope identifies the factor or factor pair whose data will improve performance most. Across simulation and real-robot tasks, training from scratch or fine-tuning a pretrained policy, this allocation beats equal, greedy, and data-mixture-weighting baselines, with real-world success-rate advantages up to 26 percent. The method also works with an offline embedding-similarity proxy in place of expensive hardware evaluations.","feed_headline":"Fit a scaling curve to pick the robot data that helps most","feed_subtitle":"A fitted power law ranks environmental factors by expected gain, beating uniform collection by up to 26 percent.","key_machinery":"The load-bearing object is the factored scaling curve: a function $\\Phi_i(n)$ that maps the number $n$ of demonstrations varying factor $f_i$ to the policy's expected success on the target environment distribution, holding other factor data fixed. The paper fits each curve as a power law $\\hat\\Phi_i(n)=1-a(n+|D\\setminus D_i|)^b$, then uses the discrete slope $P_i^K$ over the budget horizon $K$ as the ranking signal. Grouping factors into disjoint pairs reduces the number of curves from $O(N^2)$ to $O(N)$ while retaining most of the predictive accuracy, and a version of the curve built on cosine similarity of policy embeddings gives an offline alternative to hardware evaluation. What makes the machinery work is the extrapolation: from as few as four fitted points, the slope at the budget determines which factor will yield the largest improvement.","core_discovery":"The paper's central claim is that factored scaling curves can accurately predict how a policy's success rate changes when more demonstrations are collected for a particular environmental factor, and that this prediction is what should guide the next round of data collection. Starting from a dataset with demonstrations covering several factors, the method trains policies at a few intermediate dataset sizes for each factor or factor pair, evaluates them on the target environment distribution, and fits $\\hat\\Phi_i(n)=1-a(n+|D\\setminus D_i|)^b$ with $a>0$ and $b<0$ to those points. The extrapolated slope $P_i^K=(\\hat\\Phi_i(|D_i|+K)-\\hat\\Phi_i(|D_i|))/K$ ranks factors by expected gain; allocating the whole budget to the top-ranked factor, or across factors when gains are similar, is the recommended strategy. Across simulation and real-robot experiments the paper reports that this allocation beats equal, greedy, and distributionally weighted baselines, with extrapolated curves matching actual performance in the tested cases and real-world success-rate advantages up to 26 percent.","pith_inferences":["Extension the paper leaves implicit: repeated re-fitting after each collection batch could turn FSC into an adaptive budget scheduler and likely reduce the large-budget degradation the paper reports at $K=500$.","Because the embedding-similarity proxy uses only initial observations, it could screen candidate data-generation setups, such as simulators or synthetic pipelines, before any real-robot runs, which the paper does not test.","The slope ranking could be read as a marginal-value signal for deciding whether to add a brand-new factor to the study, not just how much data to add to existing ones.","A natural stress test: transfer a fitted curve from one task to a related task and check whether the factor ranking remains stable across policy backbones."],"forward_implications":["A practitioner with a fixed demonstration budget can compute a small set of curve points and allocate data to the factor pair with the steepest extrapolated slope, rather than spreading data uniformly.","The fitted curves give a quantitative expected success rate after collecting $K$ demonstrations, so a budget can be chosen by inverting the curve to hit a target success rate.","When the fitted curves show one factor clearly dominant, the Top strategy of allocating everything to it works best; when gains are similar, proportional All allocation is preferable.","FSC-Proxy can guide data collection using an offline embedding-similarity metric without large-scale real-world rollouts for every curve point.","The method transfers between training-from-scratch and fine-tuning settings and across visual and spatial factor types, so the same guide applies to different policy families."],"supporting_citations":[{"why":"Supplies the power-law form used to fit the factored scaling curves.","marker":"[42]"},{"why":"Provides evidence that imitation-learning performance follows data-scaling laws, motivating the power-law fit.","marker":"[36]"},{"why":"Supplies the log-log fitting practice the paper follows for numerical stability.","marker":"[43]"},{"why":"Defines the policy-embedding-similarity metric that the offline FSC-Proxy variant uses in place of hardware success rates.","marker":"[41]"},{"why":"Introduces the L-shape factor-wise data collection procedure that serves as the Equal baseline and as the data-collection skeleton.","marker":"[11]"},{"why":"Provides the data-mixture-weighting baseline that FSC is compared against.","marker":"[14]"},{"why":"Supplies the policy backbone used for from-scratch training and the vision-encoder embeddings used by FSC-Proxy.","marker":"[47]"},{"why":"Supplies the generalist vision-language-action model that is fine-tuned in the real-world experiments.","marker":"[2]"},{"why":"Provides the simulation tasks and factor-varied evaluation environments used for the main experiments.","marker":"[46]"}],"fun_headline_variants":["Scaling curves pick which robot data to collect","Predict data gains with factored scaling curves","Fit curves to rank robot training data","Factor-specific curves guide data collection","Scaling curves boost robot success by 26%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method assumes that a power-law curve fitted through only a handful of noisy measurements keeps describing how success rate grows as more demonstrations are added, even well beyond the measured range.","fun_headline_variants_meta":{"raw":{"variants":["Scaling curves pick which robot data to collect","Predict data gains with factored scaling curves","Fit curves to rank robot training data","Factor-specific curves guide data collection","Scaling curves boost robot success by 26%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000276,"raw_usage":{"total_tokens":1644,"prompt_tokens":943,"completion_tokens":701,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":636}},"tokens_in":559,"tokens_out":701,"duration_ms":6544,"temperature":1.0,"reasoning_tokens":636,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:10:03.108450+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a task and initial dataset, fit curves from four sizes such as $n=0,20,40,60$, predict the top factor at budgets $K=100$ and $K=500$, actually collect that much data for the top-ranked factor and for a lower-ranked factor, and evaluate on held-out factor combinations. The central claim fails if the observed gain from the top-ranked factor is no larger, within statistical error, than the gain from a lower-ranked factor or from an equal split, or if the measured success falls outside the fitted curve's prediction interval.","supporting_citations":[],"review_version":1}