{"id":"013ad00b-9e79-425c-affb-0c867f2094c1","arxiv_id":"2505.14345","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A distance-based sample weighting scheme, taken from the authors' prior Semi-Cart work, is plugged into a neural network loss and reported to beat a supervised baseline on tabular benchmarks.","lead":"This paper reweights training samples by their distance to the test set and uses those weights in a neural network loss, reporting accuracy gains on twelve tabular benchmarks. The gains are measured against a supervised baseline that never sees the test features, so the comparison does not isolate the proposed weighting.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set features, not weighting, may explain gains; no control baseline with equivalent test-set access is provided.","rationale":"The paper proposes a simple weighting scheme and integrates it into a neural network loss; the idea is coherent and the math is straightforward. The central empirical claim, however, rests entirely on the comparison in Tables II and III, and that comparison is not controlled for the most obvious confound: the weighted model's training weights are computed from test-set feature vectors (Eq. 2). Because the baseline never sees test features, the experimental design cannot distinguish the contribution of the weighting from the contribution of transductive access to the test distribution. The reported gains are largest exactly in the 90% test-size regime where the information advantage of knowing the test set is largest, which is consistent with the confound. The per-dataset hyperparameters in Table I are also reported without a validation protocol, and no existing SSL baseline is included despite the claim of outperforming existing methods. These are not mere presentational issues; they determine whether the central claim is true. I agree with the reader's weakest-assumption analysis. A matched control baseline that receives the same test-set information would settle the question, and without it the paper's central claim should not be accepted.","tokens_in":10483,"tokens_out":5544,"duration_ms":56180,"concrete_test":"For each dataset and test size, train three models on the same split: (A) the proposed weighted model; (B) a control baseline whose input features are the original features concatenated with the average distance (and optionally the full distance vector) to the test set, trained with unweighted loss; (C) the original baseline. Use a validation split held out from both weight computation and hyperparameter selection to choose lambda and distance metric for A and B. If A does not beat B by a margin larger than the run-to-run variance (report the five runs individually), the reported advantage is attributable to test-set feature access rather than to the weighting mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the distance-based weighting scheme consistently outperforms existing methods is not supported because the experimental protocol conflates the weighting with access to test-set features. In Eq. (2), the weight for each training sample is computed as the average exponential distance to all test-set feature vectors. The weighted model therefore uses the test distribution during training, whereas the baseline in Section IV is a standard neural network that never sees test features. At large test sizes (e.g., 90%), this extra information alone could explain the reported gains; the weighted model is effectively transductive. The paper provides no ablation that gives the baseline equivalent access to test features (e.g., a transductive baseline, or test-distance features as inputs), so the claim that the weighting itself is responsible is untested. In addition, Table I reports per-dataset choices of lambda and distance metric without describing any validation procedure; if these were selected by looking at test-set results, the comparison is further biased. The comparison to only the baseline and IDW does not address existing methods as claimed. These issues are load-bearing: if the gains disappear when the baseline receives the same test-set information, the central claim collapses.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a distance-based weighting scheme for deep classifiers: each training sample is weighted by its average exponential distance to all test-set feature vectors (Eq. 2), and these weights multiply the per-sample loss (Eq. 3). The authors compare the weighted model against an unweighted baseline and an inverse-distance weighting (IDW) variant on twelve datasets at test sizes of 10%, 50%, and 90%, reporting precision, recall, F1, AUC, and accuracy. The paper claims that the weighted model consistently outperforms the baseline and, in the abstract, that it outperforms existing methods. The method is essentially a transductive, test-aware reweighting scheme, and the central claim is not supported by the evidence as presented.","tokens_in":10693,"tokens_out":6485,"duration_ms":57950,"significance":"The idea of reweighting training samples by proximity to test inputs is simple and could be practically valuable if the reported gains were real. The paper has identifiable strengths: it gives explicit equations, reports per-dataset hyperparameters in Table I, evaluates on twelve datasets, and includes a second comparator (IDW). However, the significance is currently undercut by three central problems: (i) the weighted model is granted access to test-set feature vectors during training while the baseline is not, so the comparison does not isolate the weighting; (ii) no existing semi-supervised learning method is evaluated, despite the abstract's claim of outperforming existing methods; and (iii) the quantitative tables contain internally inconsistent precision/recall/F1 values. These issues prevent the paper from establishing its main empirical claim.","major_comments":[{"comment":"The weighted model's training weights are computed from the test-set feature vectors in Eq. (2), whereas the baseline in Section IV is a standard network trained without access to test features. At test sizes up to 90%, the additional information available to the weighted model is substantial, and the reported gains are therefore not attributable to the weighting scheme per se. No ablation or control gives the baseline equivalent access to the test distribution (e.g., a transductive baseline, test-distance features as auxiliary inputs, or a uniform-weight variant with the same computation graph), so the claim that the weighting itself is responsible for the improvements is untested. This is load-bearing for the paper's main conclusion.","section":"§III-A, Eq. (2); §IV"},{"comment":"The abstract and Section IV-B claim that the method 'consistently outperforms existing methods,' but the experiments compare only the proposed weighted model, the unweighted baseline, and the IDW variant. No existing semi-supervised learning method (pseudo-labeling, consistency regularization, graph-based SSL, or any published SSL baseline) is evaluated. Moreover, the method is not a standard SSL method: no unlabeled data are used apart from the test set, which is exploited transductively in Eq. (2). The stated claim of outperforming existing methods is therefore unsupported by the experimental design.","section":"§IV-B; Abstract"},{"comment":"Many rows of Tables II and III report precision, recall, and F1 values that are mutually inconsistent under the standard binary F1 definition. For example, Adult at test size 0.9, Base row reports P=0.6520, R=0.9429, F1=0.5525, but the harmonic mean of P and R is approximately 0.771; similar inconsistencies appear in the ILPD, Heart, and Statlog rows. The tables do not state whether metrics are macro-averaged, micro-averaged, or computed for a particular class, so these entries cannot be used to verify the claimed improvements.","section":"Tables II and III"},{"comment":"The prose in Section III-B describes the weighting as penalizing less similar samples with high error more heavily and giving closer samples a lower penalty, but Eq. (3) combined with Eq. (2) does the opposite: closer samples receive larger weights w_i and therefore contribute more to the loss for a given error. This internal inconsistency makes it unclear whether the intended mechanism is to upweight or downweight test-proximal samples, and it needs to be resolved before the method can be evaluated.","section":"§III-B, Eq. (3)"},{"comment":"The manuscript reports per-dataset choices of the decay parameter λ and the distance metric in Table I, but no validation procedure is described. If these choices were made using test-set performance, the comparison is further biased in favor of the weighted model; if a validation split was used, that procedure should be explained and the selected hyperparameters should be held out. As written, the experimental protocol does not rule out selection on the test set.","section":"Table I; §IV"}],"minor_comments":[{"comment":"The text refers to 'test sizes (e.g., 70% or higher)' although the experiments use only 10%, 50%, and 90%; this sentence should be aligned with the actual test sizes.","section":"§IV-A"},{"comment":"The paper states that each configuration was run five times, but Tables II and III report only point estimates; reporting standard deviations or confidence intervals would allow the consistency claim to be assessed.","section":"§IV"},{"comment":"The column labels in Table I are confusing: the column named 'I' takes values 'I' and 'B', and the column named 'N' takes values 'N' and 'Y', which makes the legend ('I indicates whether the dataset is imbalanced, N specifies presence of noise') hard to parse; please use unambiguous labels such as 'Balanced?' and 'Noisy?' with Yes/No values.","section":"Table I"},{"comment":"The paper does not provide the neural network architecture, activation functions, preprocessing steps, or data splitting procedure; these details are needed for reproducibility.","section":"§IV"},{"comment":"Figures 3-6 aggregate results across test sizes and datasets but do not include error bars or numeric values, which makes the visual comparisons difficult to interpret.","section":"Figures 3-6"},{"comment":"The paper claims computational efficiency but does not analyze the O(N·M·d) cost of computing all pairwise distances in Eq. (2), which may be substantial at large test sizes.","section":"§III-A, Eq. (2)"}],"recommendation":"reject","confidential_remarks":"The central empirical claim is not supportable in its current form because Eq. (2) gives the weighted model access to test-set feature vectors during training, no SSL baseline is evaluated, and the metric tables show internal inconsistencies. The stress-test concern about test-set leakage is real and load-bearing. A substantially redesigned evaluation with proper transductive controls and existing SSL baselines would be needed before this could be considered for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this paper's central claim does not survive its own experiment. The weighted model computes each training sample's weight using the test feature vectors (Eq. 2), so it is effectively transductive. The baseline is a plain supervised network that never sees test features. At test sizes of 90%, that extra information alone could explain most of the reported gains. The stress-test note is on target; it doesn't overstate the problem.\n\nWhat is new here: the authors take the distance weighting from their earlier Semi-Cart paper and plug it into a neural network loss. That is a minor extension, not a new mechanism. The empirical work on 12 datasets is the bulk of the paper. To their credit, they include an inverse-distance weighting (IDW) comparison, which is a relevant control for the weighting scheme itself.\n\nThe soft spots are load-bearing. There is no control that gives the baseline equivalent access to test features—as additional input features, for example. Without that, the weighting is not isolated from the transductive advantage. Second, the abstract and conclusion claim the method 'consistently outperforms existing methods,' but no existing SSL method is in the comparison. The baseline is supervised and IDW is an alternative weighting, not an SSL method. Third, λ and the distance metric are chosen per dataset with no validation protocol described; Table I reads like a test-set-tuned configuration. Fourth, the numbers don't always add up: for Adult at test size 0.9, precision 0.652 and recall 0.943 imply an F1 around 0.77, but the table lists 0.5525. That kind of inconsistency makes the whole table hard to trust. And there are no error bars despite five runs.\n\nNone of this proves the method is useless. A transductive weighting scheme might have some value when the test set is known in advance. But the paper doesn't frame it that way, and the evaluation cannot support the general claim. This is not a serious SSL contribution. I would not cite it as evidence, and I would not send it to a referee in its current form.","headline":"Claimed gains rest on an unfair transductive advantage; the weighting itself is never actually isolated.","tokens_in":11187,"tokens_out":4980,"would_cite":false,"duration_ms":44547,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68T10"],"pacs":[],"model":"deepseek-v4-flash","headline":"A distance-based weighting scheme that upweights training samples closest to the test set consistently beats a plain neural network across twelve datasets.","keywords":["semi-supervised learning","distance-based sample weighting","loss reweighting","deep neural network classification","class imbalance","transductive learning","exponential decay weighting","tabular benchmarks"],"falsifier":"Train the identical weighted-loss model but compute each training sample's weight against a set of test-like feature vectors drawn from a different label distribution (or randomly permuted test features). If accuracy against the true test set stays as high as with the real test features, the effect is not caused by proximity to the actual target distribution; if accuracy drops to baseline level, the proximity signal is what carries the claim.","tokens_in":10278,"feed_emoji":"🎯","tokens_out":9878,"duration_ms":83411,"temperature":0.7,"pith_summary":"The paper claims that a simple modification to supervised training—multiplying each training sample's loss by a weight that decays exponentially with its average distance to the test set—consistently improves classification accuracy, precision, recall, and AUC over a plain neural network. The method is presented as semi-supervised because the unlabeled test features are used at training time to decide which training samples matter most. Across twelve tabular benchmarks and three test-split sizes, the weighted model outperforms both the baseline and an inverse-distance weighting variant, with the largest gains when the test set is large (so labeled training data is scarce) and on imbalanced or noisy datasets. The authors argue this is a cheap, scalable way to focus learning on samples representative of the target distribution.","feed_headline":"Weighting by test distance beats plain neural nets","feed_subtitle":"Twelve-dataset study shows the simple reweighted loss holds up when labeled data is scarce.","key_machinery":"The load-bearing object is the distance-based sample weight $w_i = \\frac{1}{M}\\sum_{j=1}^M \\exp(-\\lambda\\, d(x_i, x'_j))$, where $d$ is a per-dataset distance metric (Euclidean, Hamming, Cosine, or Jaccard) and $\\lambda$ is a tuned decay constant. This weight is the multiplier in the weighted loss $L_{\\mathrm{weighted}} = \\frac{1}{N}\\sum_i w_i\\, L(y_i, f(x_i;\\theta))$, so it directly changes which training errors the optimizer corrects: high error on a sample far from the test set is penalized more heavily, while close samples contribute favorably. The choice of $d$ and $\\lambda$ per dataset (Table I) is part of the mechanism, along with the exponential form chosen for numerical stability over inverse-distance weighting.","core_discovery":"On the paper's own terms, the core discovery is that reweighting each training sample by its exponential-decay similarity to the test set, then feeding those weights into the loss function, makes a deep neural network generalize better than the same network trained unweighted. The authors report that the weighted model 'consistently outperforms the traditional neural network model across multiple datasets' (Section IV-B), with notable gains on Haberman's Survival, Sonar, and Statlog (Heart), and that it also beats inverse-distance weighting. The improvement is attributed to the weighting's ability to de-emphasize noisy or class-imbalanced samples while amplifying samples that resemble the target distribution, thereby reducing error propagation relative to hard pseudo-labeling.","pith_inferences":["The method is transductive rather than broadly semi-supervised: it uses the test set's feature vectors at training time. A fairer framing would be that it exploits the target distribution's unlabeled features, and its practical use requires test features to be available during training, which is true in some batch settings but not in true online deployment.","The per-dataset tuning of $\\lambda$ and the distance metric, reported in Table I, is a degree of freedom the plain baseline does not receive; part of the reported gap could reflect hyperparameter search rather than the weighting rule itself. Giving the baseline the same tuning budget would isolate the contribution.","A natural stress test is to compute weights against randomly shuffled test features: if accuracy gains persist, the benefit would come from variance reduction or regularization, not from semantic proximity to the target distribution; if they vanish, the proximity signal is doing the work.","The evidence is entirely tabular. Extending the same weighting to image datasets (where the distance metric would need to be learned, say from a feature embedding) would clarify whether the claim 'consistently outperforming existing methods' survives outside standard tabular benchmarks."],"forward_implications":["When the test split is 90% of the data, the weighted model keeps accuracy high while the baseline declines, indicating the method is most useful precisely when labeled examples are few.","The weighted model raises recall and F1 on imbalanced sets such as Haberman and Sonar, suggesting the weighting counteracts minority-class neglect without explicit resampling.","IDW sits between baseline and weighted on most reported metrics, so the exponential-decay form itself matters, not just the act of weighting.","Because the weighting adds only a distance computation over the test set, the approach is cheap to bolt onto an existing network and its optimizer unchanged.","Gains are smaller on near-saturated datasets like Banknote and Mammographic Mass, so the method's edge is concentrated in harder, less separable problems."],"supporting_citations":[{"why":"Supplies the distance-weighting formula (exponential decay over distances to test data) that the paper adapts to deep network loss.","marker":"[1]"},{"why":"Defines the hard pseudo-labeling approach that the paper's soft weighting claims to improve on by mitigating error propagation.","marker":"[27]"},{"why":"Provides the graph-based semi-supervised learning family whose structure-motivated weighting the paper positions itself against and alongside.","marker":"[36]"}],"fun_headline_variants":["Distance to test data guides sample weights for better semi-supervised learning","Test-proximity weighting sharpens semi-supervised classification","Reweight training samples by test similarity to beat neural nets","Sample weights based on test distance lift semi-supervised performance","Test-proximity weighting improves semi-supervised learning on 12 datasets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the accuracy gains come from the distance-weighting formula itself; the weighted model also sees the test set's feature vectors when computing weights and gets per-dataset tuning of the decay constant and distance metric, so if those extras—not the weighting—drive the improvements, the central claim does not hold.","fun_headline_variants_meta":{"raw":{"variants":["Distance to test data guides sample weights for better semi-supervised learning","Test-proximity weighting sharpens semi-supervised classification","Reweight training samples by test similarity to beat neural nets","Sample weights based on test distance lift semi-supervised performance","Test-proximity weighting improves semi-supervised learning on 12 datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001611,"raw_usage":{"total_tokens":6349,"prompt_tokens":818,"completion_tokens":5531,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":434,"completion_tokens_details":{"reasoning_tokens":5445}},"tokens_in":434,"tokens_out":5531,"duration_ms":31536,"temperature":1.0,"reasoning_tokens":5445,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:35:36.237201+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the identical weighted-loss model but compute each training sample's weight against a set of test-like feature vectors drawn from a different label distribution (or randomly permuted test features). If accuracy against the true test set stays as high as with the real test features, the effect is not caused by proximity to the actual target distribution; if accuracy drops to baseline level, the proximity signal is what carries the claim.","supporting_citations":[{"cited_title":"Building semi-supervised decision trees with semi-cart algorithm","cited_arxiv_id":null,"evidence_quote":"Supplies the distance-weighting formula (exponential decay over distances to test data) that the paper adapts to deep network loss."},{"cited_title":"Daso: Distribution- aware semantics-oriented pseudo-label for imbalanced semi-supervised learning","cited_arxiv_id":null,"evidence_quote":"Defines the hard pseudo-labeling approach that the paper's soft weighting claims to improve on by mitigating error propagation."},{"cited_title":"Graph-based semi- supervised learning","cited_arxiv_id":null,"evidence_quote":"Provides the graph-based semi-supervised learning family whose structure-motivated weighting the paper positions itself against and alongside."}],"review_version":1}