{"id":"51368acc-67e2-4d31-9a2e-1e8512bcce4c","arxiv_id":"2605.12944","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"AutoSelection discovers data recipes from a 90K instruction pool that outperform full-data training and other selectors on reasoning tasks for SFT across multiple models.","lead":"This paper formulates supervised fine-tuning data selection as searching for executable curation recipes over a fixed pool of instructions rather than ranking individual examples. A smart generalist might read it to see how automated search over filtering and mixing steps can produce better training subsets than using all data or simple top-k selection.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Proxy signals and warmup probes may not reliably rank recipes for full SFT performance","rationale":"The reader's weakest assumption directly identifies the proxy-predictive-power gap as load-bearing; the empirical outperformance claim cannot be trusted without evidence that the signals preserve ranking fidelity. No other internal inconsistency (e.g., in operator definitions or pool handling) appears more central given the reported code release and stability checks.","tokens_in":1750,"tokens_out":315,"duration_ms":12521,"concrete_test":"For 30 randomly sampled recipes from the 90K pool, run both the warmup probes and full SFT on all three base models; compute Spearman rank correlation between probe scores and final reasoning averages. If mean correlation falls below 0.65 or varies >0.2 across models, the limited-budget search reliability is undermined.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on AutoSelection locating superior recipes via a two-layer solver that substitutes cached task/data/model signals plus limited warmup probes for exhaustive full SFT runs. If these proxies exhibit low or unstable correlation with actual post-SFT metrics (especially across the three base models and reasoning tasks), the Gaussian-process ranking and local-edit search will optimize the wrong objective, rendering outperformance over random recipe search and single-operator baselines an artifact of the proxy rather than true data quality. The reported search-stability analyses and structural ablations do not directly quantify proxy-to-full correlation or test whether probe rankings preserve the same ordering as full evaluations.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper reframes SFT data selection as fixed-pool recipe search over a library of grounded operators (filtering, mixing, deduplication) applied to a 90K instruction pool. AutoSelection is a two-layer solver that materializes subsets via cached task/data/model signals and limited warmup probes, then uses Gaussian-process ranking, local recipe edits, and stagnation reseeding to locate high-quality recipes under a tight budget of full SFT runs. Experiments report that AutoSelection yields the highest in-distribution reasoning average across three base models, outperforming full-data training, random recipe search, random top-k, and single-operator baselines; additional OOD graph-reasoning results, search-stability analyses, structural ablations, and 1.5B-to-7B transfer checks are presented, with code released.","tokens_in":1859,"tokens_out":511,"duration_ms":29777,"significance":"If the proxy signals and warmup probes are shown to preserve ranking order with full SFT performance, the work meaningfully shifts the paradigm from instance-level ranking to structured recipe search, offering a practical route to better data curation with far fewer full evaluations. The provision of code and the empirical breadth across models and tasks are clear strengths that would support adoption if the proxy-to-full correlation is established.","major_comments":[{"comment":"§5 (Results) and search-stability analyses: no direct quantification is given of the correlation between warmup-probe rankings and full SFT metrics across the three base models or reasoning tasks. Because the two-layer solver substitutes these proxies for exhaustive evaluation, this correlation is load-bearing for the claim that AutoSelection locates genuinely superior recipes rather than artifacts of the proxy objective.","section":"§5 and search-stability analyses"},{"comment":"Table 1 / in-distribution reasoning results: the reported averages lack statistical significance tests, run-to-run variance, or explicit controls for post-hoc selection of the final recipe; without these, the claim of consistent outperformance over random recipe search and single-operator baselines cannot be fully assessed.","section":"Table 1 / in-distribution results"}],"minor_comments":[{"comment":"Method section: the distinction between cached signals, realized subset states, and the exact form of the Gaussian-process surrogate could be illustrated with a short pseudocode block or diagram to improve reproducibility.","section":"Method"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments. We address each major comment below and indicate the revisions we will make to the manuscript.","responses":[{"response":"We acknowledge that a direct quantification of the correlation between warmup-probe rankings and full SFT performance is not provided in the current manuscript. Our search-stability analyses show that the discovered recipes lead to strong performance, but to directly address this concern, we will add in the revised version explicit correlation metrics (such as Spearman rank correlation) computed across the three base models and tasks, using the available probe and full evaluation data from our experiments. This will help confirm that the proxy objective aligns with the true performance.","revision_made":"yes","referee_comment":"[§5 and search-stability analyses] §5 (Results) and search-stability analyses: no direct quantification is given of the correlation between warmup-probe rankings and full SFT metrics across the three base models or reasoning tasks. Because the two-layer solver substitutes these proxies for exhaustive evaluation, this correlation is load-bearing for the claim that AutoSelection locates genuinely superior recipes rather than artifacts of the proxy objective."},{"response":"We agree that including statistical significance and variance would strengthen the results. Due to the high computational cost of full SFT runs, our experiments used single evaluations per recipe within the budget. In the revision, we will report run-to-run variance from additional repeated runs on the top recipes where possible, include p-values from statistical tests comparing AutoSelection to baselines, and add a description of the recipe selection procedure to clarify that it follows the fixed budget and automated process without post-hoc cherry-picking.","revision_made":"yes","referee_comment":"[Table 1 / in-distribution results] Table 1 / in-distribution reasoning results: the reported averages lack statistical significance tests, run-to-run variance, or explicit controls for post-hoc selection of the final recipe; without these, the claim of consistent outperformance over random recipe search and single-operator baselines cannot be fully assessed."}],"tokens_in":1477,"tokens_out":440,"duration_ms":27648,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core shift here is treating curation as recipe search: given a library of operators on a fixed 90k pool, find an executable sequence that produces a good training subset without rewriting data. AutoSelection splits this into a cheap materialization layer using cached task/data/model signals and a search layer that runs limited warmup probes, does local edits, and ranks with Gaussian processes before full SFT evaluation. That framing and the two-layer solver with stagnation reseeding are the concrete additions beyond standard top-k work.","headline":"The paper reframes SFT data selection as searching over operator recipes on a fixed pool rather than ranking instances, and AutoSelection finds stronger reasoning subsets than the listed baselines.","tokens_in":2336,"tokens_out":180,"would_cite":true,"duration_ms":22057,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[{"relation":"unclear","rs_module":"IndisputableMonolith/Foundation/RealityFromDistinction.lean","rs_theorem":"reality_from_one_distinction","paper_passage":"We formulate this problem as fixed-pool data recipe search: given a raw instruction pool and a library of grounded operators, the goal is to discover an executable recipe that constructs a high-quality selected subset under a limited budget of full SFT evaluations"},{"relation":"unclear","rs_module":"IndisputableMonolith/Cost/FunctionalEquation.lean","rs_theorem":"washburn_uniqueness_aczel","paper_passage":"AutoSelection ... using warmup probes, realized subset states, local recipe edits, Gaussian-process-assisted ranking, and stagnation-triggered reseeding"}],"headline":"Fixed-pool recipe search for LLM SFT curation is unrelated to RS forcing chain","alignment":"orthogonal","rationale":"The paper's machinery (operator libraries, state vectors z(r), GP surrogates over recipe encodings, warmup probes, local edits, stagnation reseeding) operates entirely in the domain of budgeted combinatorial search over data-curation programs on a fixed instruction pool. No component invokes recognition cost J(x), golden-ratio identities, 8-tick periodicity, Alexander duality for D=3, or any theorem from the RS chain. The problem formulation and evaluation protocol (SFT on 90K pool, GPQA/GSM8K/BBH/MMLU) lie outside the scope of reality_from_one_distinction and its derived physics constants.","tokens_in":60252,"confidence":"high","tokens_out":354,"duration_ms":9029,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Recipe search over fixed instruction pools finds better supervised fine-tuning data than instance ranking or full-data training.","keywords":["supervised fine-tuning","data selection","recipe search","instruction tuning","operator sequences","model training efficiency"],"falsifier":"Running full SFT evaluations on the top recipes returned by AutoSelection and finding that their actual performance is no better than the performance of recipes found by random search under the same evaluation budget.","tokens_in":2648,"feed_emoji":"🔍","tokens_out":667,"duration_ms":26318,"temperature":0.7,"pith_summary":"The paper reframes supervised fine-tuning data selection as a search over executable sequences of operators such as filtering, mixing, and deduplication applied to a fixed raw pool, rather than scoring and keeping top-k individual examples. This matters because high-performing training subsets are typically produced by ordered curation steps that jointly shape the data distribution, yet evaluating every possible recipe with full training runs is prohibitively expensive. AutoSelection solves the problem with a two-layer solver that first materializes candidate subsets from cached task, data, and model signals, then uses warmup probes, local edits, and Gaussian-process ranking to refine recipes under a tight budget of full evaluations. On a 90K instruction pool the method delivers the highest in-distribution reasoning average across three base models while also showing gains on out-of-distribution graph reasoning and stable transfer from 1.5B to 7B scales.","feed_headline":"Recipe search beats instance ranking for SFT data","feed_subtitle":"AutoSelection locates high-performing curation sequences from cached signals and few full runs, outperforming full data and random baselines","key_machinery":"Two-layer solver that materializes candidate subsets from cached task-data-model signals for rapid ranking and then refines executable operator sequences with local edits and Gaussian-process assistance under a limited budget of full SFT runs.","core_discovery":"AutoSelection, a two-layer solver, decouples fixed-pool materialization based on cached signals from expensive full evaluation by using warmup probes, realized subset states, local recipe edits, Gaussian-process-assisted ranking, and stagnation-triggered reseeding, and thereby discovers recipes whose resulting subsets yield stronger in-distribution reasoning performance than full-data training, random recipe search, random top-k, or single-operator selectors.","pith_inferences":["The cached-signal layer could be reused across multiple downstream tasks or base models without retraining the ranking model from scratch.","Extending the operator library with domain-specific filters would allow the same search machinery to target specialized data distributions such as code or math instruction sets."],"forward_implications":["Recipe structure itself matters for final performance beyond the choice of any single operator, as shown by structural ablations.","The discovered recipes transfer across model scales from 1.5B to 7B parameters while preserving the performance ordering.","The same fixed-pool recipe approach produces measurable gains on out-of-distribution graph-reasoning tasks in addition to the in-distribution average.","High-quality subsets can be obtained without generating, rewriting, or augmenting any new training examples."],"fun_headline_variants":["From instance ranking to fixed pool recipes for SFT data","AutoSelection searches executable recipes on fixed instruction pools","Two layer method finds SFT curation sequences using cached signals","Recipe structure matters for SFT beyond individual operators"],"cache_read_input_tokens":64,"weakest_assumption_plain":"Cached task, data, and model signals plus warmup probes can reliably predict which recipes will perform well when the full supervised fine-tuning run is actually executed.","fun_headline_variants_meta":{"raw":{"variants":["From instance ranking to fixed pool recipes for SFT data","AutoSelection searches executable recipes on fixed instruction pools","Two layer method finds SFT curation sequences using cached signals","Recipe structure matters for SFT beyond individual operators"]},"model":"grok-4.3","cost_usd":0.004658,"raw_usage":{"total_tokens":2241,"prompt_tokens":702,"num_sources_used":0,"completion_tokens":61,"cost_in_usd_ticks":46578000,"prompt_tokens_details":{"text_tokens":702,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1478,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":702,"tokens_out":61,"duration_ms":14060,"temperature":1.0,"reasoning_tokens":1478,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-14T20:35:22.139565+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Running full SFT evaluations on the top recipes returned by AutoSelection and finding that their actual performance is no better than the performance of recipes found by random search under the same evaluation budget.","supporting_citations":[],"review_version":1}