{"id":"59b4d0d4-196d-42db-b5ea-79d4598125d1","arxiv_id":"2507.22186","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SourceSplice selects data sources by repeatedly swapping poorly performing sources with better candidates, finding near-optimal training sets with fewer model evaluations than existing greedy or metaheuristic approaches.","lead":"This paper introduces SourceSplice, an algorithm that chooses a small subset of data sources to train a machine learning model, aiming for high accuracy or fairness while training as few candidate models as possible. It reports that on U.S. Census datasets, SourceSplice finds near-optimal source subsets with far fewer model trainings than exhaustive search or a GRASP-based alternative.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No held-out test set is described; if gains are computed in-sample, the reported 'task utility' is training-set fit and the central claim is unsupported.","rationale":"The reader's stated weakest assumption is the homogeneity of data sources, but their rationale also identifies the missing train/test protocol. I agree that homogeneity is a scope limitation; however, the more load-bearing threat to the central claim is the absence of any described held-out evaluation. If G(S) is computed on training rows, then the exhaustive ground truth and all percentile comparisons reflect training fit rather than task utility, so the empirical evidence for 'high task utility' collapses. The concrete check on the repository will settle whether this concern lands. If the code contains a proper train/test split, the paper remains conditionally acceptable with a documentation fix; if not, the central claim is unsupported. I also note the off-by-one indexing in Algorithm 2's slicing (sorted(rmVal)[1:k_local] and sorted-reverse(addVal)[1:k_local] exclude the highest-priority swap when k_local=1), which is an additional internal inconsistency, but the published code may correct it and it is secondary to the evaluation validity issue. The verdict remains conditional pending the feasibility check.","tokens_in":19200,"tokens_out":9414,"duration_ms":111466,"concrete_test":"Inspect the public repository at https://github.com/am-barish/SourceSplice and locate the evaluation code that computes accuracy/fairness used as the gain G(S). Confirm whether a train/test split or cross-validation is applied. If none exists, rerun the ACSIncome classification experiment with a standard 80/20 split: select sources by SourceSplice using validation-set gain, then evaluate the returned subset on the disjoint test set and compare its test accuracy/fairness against Greedy and Random baselines. If the test-set utility of the SourceSplice subset is not clearly above the baselines, the in-sample evaluation concern lands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1.3 defines accuracy as the proportion of true predictions out of total predictions but never states that those predictions are on a held-out set. Section 5.1.1 describes the ACS datasets only as full tables partitioned by U.S. state; no train/test split, cross-validation, or held-out evaluation is mentioned anywhere in the experimental protocol. If G(S) is measured on the same rows used to train the model, then SourceSplice (and the exhaustive 'ground truth' Naïve baseline) are optimizing training accuracy or training fairness, not generalization. The Table 1 percentiles and Figures 3-5 efficiency comparisons would then show only that SourceSplice finds subsets that fit the training data well, which does not support the abstract's claim of 'high task utility' for a downstream ML task. This concern is more immediate than the schema-homogeneity limitation, because it threatens the validity of every reported experiment even under the paper's own homogeneous ACS setup. The reader's rationale lists the missing train/test protocol, but it is not merely a reproducibility detail; it is the crux of whether the central empirical claim is meaningful.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formalizes a source-selection problem: given a set of data sources S and a supervised learning task, choose a subset S* maximizing profit P(S)=G(S)-C(S), where G(S) is a task-utility gain and C(S) is an acquisition cost. It proposes two heuristics: SourceGrasp, a GRASP-style greedy randomized construction with local search, and SourceSplice, an iterative active/inactive-set swapping procedure inspired by gene splicing. Experiments on ACSIncome, ACSPublicCoverage, and ACSTravelTime, with 15 sources obtained by partitioning the data by U.S. state, plus synthetic benchmarks, compare subset percentile, number of models explored, and runtime against exhaustive Naive, Greedy, Random, and DsDM baselines. The authors report that SourceSplice reaches near-optimal subset percentiles with far fewer model trainings and lower runtime than SourceGrasp and DsDM, and they provide ablations for s_max, k_max, cost functions, and scalability.","tokens_in":19516,"tokens_out":11277,"duration_ms":132374,"significance":"If the empirical claims held, the paper would offer a practically useful heuristic: a data scientist could choose a high-quality training-source subset with a few hundred model trainings instead of enumerating 2^15 combinations. The paper has clear strengths: an exhaustive ground-truth baseline over all subsets, a public code repository, and an evaluation metric (subset percentile) that directly measures performance against exhaustive search on the paper's own objective. However, the current manuscript does not establish that the optimized quantity is held-out task performance. Section 5.1.3 defines accuracy, TPR, and MSE without any reference to a test set, and Section 5.1.1 describes only full ACS tables partitioned by state, with no train/test split or cross-validation mentioned. The stress-test concern therefore lands directly: if G(S) is computed on the same rows used for training, the reported 'ground truth' optimum and all subset percentiles measure training-set fit rather than downstream task utility, which would not support the abstract's central claim.","major_comments":[{"comment":"No held-out evaluation protocol is specified anywhere in the experimental setup. Section 5.1.3 defines accuracy as 'the proportion of true predictions out of total predictions' and defines TPR and MSE without stating that predictions are made on a held-out test set, and Section 5.1.1 describes the ACS datasets only as full tables partitioned by state. If G(S) is evaluated on the same rows used to train the model, then the exhaustive Naive 'ground truth' and every subset percentile in Table 1 and Figures 3-8 are measuring in-sample fit, which is not 'task utility' for a downstream ML model and can be maximized by overfitting. This is load-bearing because the abstract's claim that SourceSplice 'effectively identifies subsets of data sources leading to high task utility' is unsupported without a specified split. Please state explicitly how every G(S) is evaluated, add held-out evaluation for all reported metrics, and use a nested protocol if the selection procedure itself uses validation performance.","section":"Section 5.1.3 and Section 5.1.1"},{"comment":"The pseudocode and the running example disagree on how source valuations are computed, and the pseudocode as written does not implement the described swap-by-valuation mechanism. First, lines 28 and 31 of Algorithm 2 use 'sorted(rmVal)' and 'sorted-reverse(addVal)' on dictionaries; in Python this sorts keys (source identifiers) rather than valuation values, so the algorithm would not actually remove the lowest-rmVal or add the highest-addVal sources. Second, lines 24-27 compute addVal(s) as P(A ∪ s) - P(A) before any removal, while Example 4.1 computes addVal(s5) = P({s1,s2,s3,s5}) - P({s1,s2,s3}) after removing s4, and the paragraph immediately after the example states that 's4 was already removed from A' before inactive-set valuations are computed. These two orderings can lead to different swap decisions. Since SourceSplice is the paper's main contribution, the pseudocode, example, and released code must be aligned, and the intended ordering (valuations before or after removals) must be stated explicitly.","section":"Section 4, Algorithm 2 and Example 4.1"}],"minor_comments":[{"comment":"Table 1 reports averages over 10 executions for SourceGrasp and DsDM without standard deviations or confidence intervals; because several entries differ by less than 0.03 percentile, the claim that SourceSplice is 'comparable or better' is not supported at the reported precision.","section":"Table 1"},{"comment":"The cost model in Definition 2.3 makes acquisition cost an increasing function of individual gain, and Section 5.3.5 only varies the exponent t of this self-defined cost formula; the conclusion that SourceSplice 'becomes more effective' for higher cost associations should be described as a sensitivity study of the chosen cost model rather than evidence about real-world acquisition costs.","section":"Definition 2.3 and Section 5.3.5"},{"comment":"The paper frames the problem for general data discovery, but the method assumes sources share one schema and are combined by row concatenation; please add an explicit limitation in Section 5.1.1 or the conclusions that the algorithms and experiments are evaluated for homogeneous relational partitions.","section":"Abstract and Section 5.1.1"},{"comment":"Example 2.6 contains a typo: the second profit line repeats P({s1,s7}); it should refer to P({s4,s7}).","section":"Example 2.6"},{"comment":"The text says Figure 3 shows total time for 15 sources, but the x-axis of Figure 3 shows 10, 12, and 15 sources; please clarify the intended setup.","section":"Section 5.2.2 and Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The missing held-out protocol is the central barrier to publication; the stress-test concern is valid and should be fixed by re-running the experiments with an explicit split and reporting all subset-quality metrics on held-out data. The Algorithm 2 / Example 4.1 inconsistency is also important for reproducibility. Both issues are fixable within the scope of a revision, so I recommend major revision rather than rejection. Please also ask the authors to check that the released code matches the corrected pseudocode."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Jeremy,\n\nYou should know two things about this one. First, it addresses a real problem: how to pick which data sources to union into a training set when you can't try all subsets. Second, and more importantly, the experiments as written never state that the reported 'task utility' is measured on a held-out set. Section 5.1.3 defines accuracy as 'proportion of true predictions out of total predictions' and the setup describes datasets only as full tables partitioned by U.S. state. No train/test split, cross-validation, or held-out evaluation appears anywhere. If G(S) is computed on the rows used to train the model, the subset percentiles in Table 1 and the efficiency gains in Figures 3-5 show only that SourceSplice finds subsets that fit the training data well. That does not support the abstract's claim of 'high task utility' for a downstream ML task. This is not a minor reproducibility detail; it is the crux of the empirical claim.\n\nThe paper does have real value. The problem formulation—maximizing a profit function over source subsets—is clean and useful. SourceGrasp and SourceSplice are sensible heuristics; SourceSplice is a swap-based local search with a cost model, which is a legitimate adaptation of feature selection ideas to source selection. The comparison to datamodels (DsDM) and to exhaustive search is appropriate, and the code is available. I appreciate that the paper is honest about its limitations, including the assumption that sources are homogeneous partitions of one relation.\n\nThe soft spots beyond the missing test set: Example 4.1 is inconsistent with Algorithm 2 in how addVal is computed—the example removes the low-value source from A before computing addVal for I, while the pseudocode computes addVal on the original A. This matters for reproducibility. The random state is never seeded for SourceGrasp or DsDM, so the averaged results cannot be exactly reproduced. The heterogeneity assumption is a limitation but not a flaw given the experiments.\n\nWho is this for? Someone working on data discovery or automated pipeline construction would find the efficiency comparison interesting. But the paper needs a major revision that specifies the evaluation protocol and re-runs the experiments with a proper train/test split before we can trust the empirical claims.\n\nMy recommendation: send it to peer review, but be clear that the missing test set is a blocking issue. The idea is worth refereeing; the evaluation as it stands is not.","headline":"Plausible heuristics for source subset selection, but the reported experiments never mention a held-out test set, so the central empirical claim of task utility is not yet supported.","tokens_in":19962,"tokens_out":3151,"would_cite":false,"duration_ms":35149,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SourceSplice finds near-optimal training-source subsets with far fewer model trainings than exhaustive enumeration.","keywords":["source selection","subset selection","machine learning utility","gene splicing","greedy randomized adaptive search","data discovery","marginal gain","fairness-aware data"],"falsifier":"Take a small instance with 10 sources where the true profit $P(S)$ is computable for all 1,024 subsets, run SourceSplice across many random seeds, and compare its returned subset's profit to the exhaustive optimum; if a substantial fraction of runs land more than a few percent below the optimum, the claim that SourceSplice effectively identifies high-utility subsets is falsified.","tokens_in":19028,"feed_emoji":"🧬","tokens_out":9865,"duration_ms":95790,"temperature":0.7,"pith_summary":"The paper claims that the best training-data sources for a given ML task can be selected without exhaustively trying all combinations. It proposes SourceSplice, a swapping heuristic inspired by gene splicing that refines an active set of sources by exchanging low-value members for high-value outsiders, with each source valued by its marginal contribution to task profit. On three real-world datasets and synthetic data, the authors report that SourceSplice attains subset utility at or near the exhaustive optimum while exploring only a few percent of the subset space, far fewer than the GRASP-based and datamodel-based baselines. If true, a data scientist facing many candidate sources could choose a near-best training set with a small budget of model trainings, including when the objective is model fairness.","feed_headline":"SourceSplice hits near-optimal subsets with under 3% of trainings","feed_subtitle":"A data scientist can skip exhaustive subset search and still keep near-optimal model utility.","key_machinery":"The load-bearing mechanism is the swap valuation pair: for a source $s$ in the active set $A$, $\\mathrm{rmVal}(s) = P(A) - P(A\\setminus\\{s\\})$, and for $s$ in the inactive set $I$, $\\mathrm{addVal}(s) = P(A\\cup\\{s\\}) - P(A)$, where the profit $P(S) = G(S) - C(S)$ is gain (task utility) minus acquisition cost. SourceSplice sorts active sources by $\\mathrm{rmVal}$ and inactive sources by $\\mathrm{addVal}$, swaps the $k_{\\mathrm{local}}$ lowest-$\\mathrm{rmVal}$ active sources with the $k_{\\mathrm{local}}$ highest-$\\mathrm{addVal}$ inactive sources, increases $k_{\\mathrm{local}}$ from 1 to the current subset size, and repeats the whole splicing loop for each subset size $i=1,\\dots,s_{\\max}$ (via the fixedSupport wrapper) until the active set stops changing. The 'gene splicing' name is a metaphor for these targeted exchanges; the algorithm's effectiveness rests on the assumption that marginal profits give reliable swap signals despite source interdependencies.","core_discovery":"The central discovery is that a gene-splicing-style swap search over data sources finds source subsets whose task utility is effectively indistinguishable from the optimal subset for the classification, fairness, and regression tasks tested. The algorithm partitions sources into an active set and an inactive set, repeatedly removes the active source with the smallest marginal profit and adds the inactive source with the largest marginal profit, then repeats with an increasing number of swaps until the active set stabilizes. Across the ACSIncome, ACSPublicCoverage, and ACSTravelTime datasets, SourceSplice returned the ground-truth optimal subset in three of the five ML tasks and subset percentiles above 99.9 in the remaining ones, while training roughly 2% of the $2^{15}$ possible subsets. Under constrained training budgets it reaches the optimum with 16 to 32 times fewer model trainings than SourceGrasp and DSDM, and on a synthetic 100-source setup, exploring under 0.0003% of subsets yields a 13.5% accuracy improvement over using all sources.","pith_inferences":["Editorial inference: the swap valuations could be reused with a cheap surrogate of gain—for instance, a datamodel that predicts subset profit without retraining—so the same splicing loop could operate in a budget regime where even a few hundred real trainings are too many.","Editorial inference: if task utility is submodular in sources, the marginal-profit swap signals are well-behaved and the procedure resembles a local-search guarantee; the paper's experiments are consistent with such behavior but the authors do not state or test submodularity.","Editorial inference: the flat sensitivity to the maximum swap count suggests the swap heuristic is a robust inner loop for other subset-selection problems—feature selection, table union search, or data-augmentation pipelines—where the same active/inactive partition and valuation equations apply.","Editorial inference: the cost-function experiments show relative profit gains grow as acquisition costs become more complex, implying the method's practical benefit may be largest when data sources are expensive and acquired per source."],"forward_implications":["A data scientist with 15 candidate sources can approximate the exhaustive optimum by training on the order of a few hundred subsets instead of 32,768, making utility-driven source selection practical in real data-discovery workflows.","The same swap procedure can optimize composite objectives like accuracy plus group-fairness (true-positive-rate difference), not only raw accuracy or mean squared error, so source selection can serve fairness-aware dataset construction.","Under fixed compute budgets, SourceSplice converges to the best-possible subset with 16 to 32 times fewer model trainings than SourceGrasp and DSDM in the tested settings.","The method scales to larger source collections: with 100 synthetic sources, exploring under 0.0003% of the subset space still yields a 13.5% accuracy improvement over using all sources.","Because SourceSplice explores all subset sizes up to $s_{\\max}$, it can recover optimal subsets of very small cardinality, such as a single best source, that pure greedy addition would miss."],"supporting_citations":[{"why":"Supplies the three real-world ACS datasets (ACSIncome, ACSPublicCoverage, ACSTravelTime) that are partitioned into state-level sources for the experiments.","marker":"[12]"},{"why":"Defines the GRASP metaheuristic that SourceGrasp is built on and that SourceSplice is compared against as a baseline.","marker":"[17]"},{"why":"Provides DSDM, the datamodel-based baseline that SourceSplice must beat in subset percentile and number of model trainings.","marker":"[16]"},{"why":"Introduces datamodels, the surrogate that DSDM uses to estimate subset utilities, which the paper contrasts with its own gain computations.","marker":"[21]"},{"why":"Original gene-splicing-based best-subset selection for features, the inspiration that SourceSplice adapts to the data-source selection problem.","marker":"[48]"},{"why":"The biology paper establishing gene splicing, the concept behind the algorithm's name and swap mechanism.","marker":"[4]"}],"fun_headline_variants":["SourceSplice finds near-optimal ML data subsets in 2% of trainings","Gene-splicing algorithm picks best data sources with minimal training","Skip exhaustive search: SourceSplice selects top data sources fast","SourceSplice: near-optimal source selection in a fraction of runs","Data source selection: SourceSplice reaches the 99.9th percentile fast"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The methods assume every data source is a row-partition of a single relational table with the same schema, so any subset can be combined by concatenating rows and its utility measured by training a model on that concatenation; if sources have heterogeneous schemas, overlapping columns, or conflicting records, the marginal-profit swap valuations do not transfer.","fun_headline_variants_meta":{"raw":{"variants":["SourceSplice finds near-optimal ML data subsets in 2% of trainings","Gene-splicing algorithm picks best data sources with minimal training","Skip exhaustive search: SourceSplice selects top data sources fast","SourceSplice: near-optimal source selection in a fraction of runs","Data source selection: SourceSplice reaches the 99.9th percentile fast"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000797,"raw_usage":{"total_tokens":3527,"prompt_tokens":984,"completion_tokens":2543,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":2447}},"tokens_in":600,"tokens_out":2543,"duration_ms":21070,"temperature":1.0,"reasoning_tokens":2447,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:59:27.406627+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a small instance with 10 sources where the true profit $P(S)$ is computable for all 1,024 subsets, run SourceSplice across many random seeds, and compare its returned subset's profit to the exhaustive optimum; if a substantial fraction of runs land more than a few percent below the optimum, the claim that SourceSplice effectively identifies high-utility subsets is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the three real-world ACS datasets (ACSIncome, ACSPublicCoverage, ACSTravelTime) that are partitioned into state-level sources for the experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the GRASP metaheuristic that SourceGrasp is built on and that SourceSplice is compared against as a baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides DSDM, the datamodel-based baseline that SourceSplice must beat in subset percentile and number of model trainings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces datamodels, the surrogate that DSDM uses to estimate subset utilities, which the paper contrasts with its own gain computations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Original gene-splicing-based best-subset selection for features, the inspiration that SourceSplice adapts to the data-source selection problem."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The biology paper establishing gene splicing, the concept behind the algorithm's name and swap mechanism."}],"review_version":1}