{"id":"b3180797-5bd2-435b-bdd7-d24c8b2a6aaa","arxiv_id":"2412.05466","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"A UCB-based training procedure switches between a diversity/photorealism score and a feature-cohesion score to select synthetic training images, reporting up to 10-point accuracy gains over static metrics.","lead":"This paper proposes a bandit-style training loop that switches between two scoring rules to decide which synthetic images a model trains on. The authors report accuracy gains of up to 10 percentage points for image classifiers fine-tuned on their selected synthetic data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own Table 5 contradicts the central claim: the 4-arm UCB with SSIM/PSNR/IS/FID beats the proposed 2-arm U on AlexNet/SP-Car-2 (86.0 vs 80.0), and no seed variance is reported, so 'U achieves the best results' is unsupported.","rationale":"I focused on the internal counterexample because it is the most direct way to test the load-bearing claim. The paper has a plausible MAB framing and a public code link, but the core result is a set of single-run accuracies. The reader's weakest_assumption about static score transfer and validation-reward overfitting is real, but the Table 5 discrepancy is more decisive: it uses the paper's own numbers and the paper's own UCB procedure, so it does not depend on assumptions about how scores generalize. If the 4-arm traditional-metric row is a legitimate comparison, then U is not consistently the best; if it is not legitimate, the paper needs to state why. The proposed multi-seed re-run would settle whether the 86 vs 80 gap replicates. I kept the verdict unchanged because the reader already rejected the paper, and my concern strengthens rather than redirects that rejection. I did not make the missing DPS formula the primary concern only because the internal contradiction already settles the evaluation of the headline claim without needing to reconstruct the method.","tokens_in":16797,"tokens_out":7079,"duration_ms":73408,"concrete_test":"Re-run the exact SP-Car-2/AlexNet fine-tuning protocol from Section 4.3 and Algorithm 2 — same M, optimizer, learning rate, epochs, validation split, patience, and UCB parameters (ε=1e-5, β=2) — for at least five random seeds, comparing the proposed 2-arm U policy with the 4-arm SSIM/PSNR/IS/FID policy. Report per-seed test accuracies and means. If the 4-arm policy's mean accuracy is not below the 2-arm U mean, the Table 4 claim is contradicted under the paper's own setup; if it is below, the single-run contradiction is likely noise, but the headline still needs error bars before 'best results' can be accepted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, asserted in the Table 4 caption and Section 4.3, is that 'Our metric U with the proposed UCB-based training approach achieves the best results.' The paper's own ablation study already contains a direct counterexample. In Table 5's SP-Car-2 block, the row 'SSIM, PSNR, IS, & FID 4 Arms' — the same UCB training procedure of Algorithm 2, with four traditional image-quality metrics as arms — reports 86.0 for AlexNet, while the proposed 'Our Approach (U) 2 Arms' reports 80.0; the 4-arm row also beats U on VGG (80 vs 77) and on several other cells. Since every Table 4/5 number is a single run with no error bars, the phrase 'significantly outperformed' in Section 4.3 is not justified. The consequence is load-bearing: the observed advantage of U may be an artifact of the UCB switching mechanism, of having a different number of arms, or of single-run noise, rather than of the proposed DPS/FCS metric. The claim of consistent superiority is therefore internally falsified unless the 4-arm result is shown not to replicate or is excluded on principled comparison grounds. This is not a disagreement with an outside consensus; it is a failure of the paper's own evidence to support its headline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a multi-armed bandit (UCB) training procedure for fine-tuning classifiers on synthetic data. The authors introduce a two-component usability score U = [Ψ, Φ], where Ψ (DPS) is intended to capture diversity and photorealism using Inception-V3 features, and Φ (FCS) measures feature cohesion via KL divergence in VGG16 feature space. A UCB algorithm is used to switch between two subsets of synthetic images selected by the two scores, based on validation accuracy. The authors also present an LLM-plus-Stable-Diffusion pipeline for generating attribute-aware synthetic datasets and release six such datasets. Experiments fine-tune six architectures on these datasets and report that the proposed U score with UCB achieves the best classification accuracy compared to SSIM, PSNR, IS, and FID.","tokens_in":17093,"tokens_out":8858,"duration_ms":79093,"significance":"If the proposed dynamic selection approach worked as claimed, it would address a real need in the synthetic-data literature: most existing metrics assess generation quality rather than training usability. The idea of adapting sample selection during training via a bandit is plausible, and the public release of the generated datasets is a useful community resource. However, the paper does not yet establish the claim. The DPS component is never given a closed-form definition, the experiments appear to be single runs without statistical significance, and the comparison is confounded between metric choice and the UCB mechanism. The ablation study also contains configurations where the proposed method is outperformed by UCB with traditional metrics. Thus the significance remains potential rather than demonstrated.","major_comments":[{"comment":"The DPS component Ψ is never given a closed-form definition. The text describes computing means and standard deviations of Inception-V3 features but does not specify how these are combined into Ψ; Figure 1 is a schematic with undefined symbols. Since U = [Ψ, Φ] is the central construct of the paper, the method is not reproducible without the exact formula. Please provide the precise equation for Ψ, including any normalizations and distance functions.","section":"Section 3.3, Figure 1"},{"comment":"The ablation contains a direct counterexample to the central claim. For SP-Car-2/AlexNet, the row 'SSIM, PSNR, IS, & FID 4 Arms' reports 86.0, while 'Our Approach (U) 2 Arms' reports 80.0; VGG shows a similar inversion (80 vs 77). Since all accuracy numbers in Tables 4 and 5 are single integers with no error bars or significance tests, the statements in Section 4.3 that 'Our metric U ... achieves the best results' and 'significantly outperformed' are not supported. Please provide repeated runs with standard errors and statistical tests, or revise the claim to exclude configurations where the method is not superior.","section":"Section 4.5, Table 5"},{"comment":"The comparison is confounded because the proposed method uses UCB-based dynamic switching between two arms, while the baselines (SSIM, PSNR, IS, FID) appear to use static ranking and training on a single selected subset. To attribute the improvement to the U metric rather than to the bandit mechanism, the baselines should be evaluated under the same UCB procedure (e.g., each metric as a single arm, or UCB over the four metrics), or the proposed metric should also be evaluated as a static selector. As presented, the gains in Table 4 could be driven entirely by the adaptive switching.","section":"Section 4.3, Table 4"},{"comment":"The UCB reward is the validation accuracy, but the paper does not state whether the final reported accuracy is computed on a held-out test set that is disjoint from the validation split used for arm selection. If the same split is used for both, the reported numbers can overfit the validation criterion and the evaluation is circular. Please clarify the train/validation/test partition and ensure that Table 4 reports test accuracy on a set never used for reward updates or model selection.","section":"Section 3.4, Algorithm 2"},{"comment":"The hyperparameters M (the number of selected synthetic images), K (the number of real images used to compute the class mean in Eq. (4)), and the patience threshold in Algorithm 2 are never given numerical values. These are essential for reproducibility. Please report the exact values used in the experiments and explain how they were chosen.","section":"Section 4.1, Algorithm 2"}],"minor_comments":[{"comment":"The word 'publically' should be 'publicly'.","section":"Abstract and Data Availability"},{"comment":"The entropy term in Eq. (2) and its use in the joint objective Eq. (3) are not clearly connected; the paper does not specify how α(t) is updated over time, despite the claim that it is temporally adaptive.","section":"Section 3.1, Eq. (3)"},{"comment":"The symbols ψ, φ, τ, υ, γ, δ appear in Figure 1 but are not defined in the text; please add a legend or name them in Section 3.3.","section":"Figure 1"},{"comment":"The AUC values are reported without any variance or error bars, so it is unclear whether the differences between artistic, photorealistic, and real datasets are statistically meaningful.","section":"Table 3"},{"comment":"The fine-tuning setup is under-specified: the number of epochs, learning rate, batch size, optimizer, and data augmentation for each architecture are not provided.","section":"Section 4.1"},{"comment":"The phrase 'improvement of up to 10% in classification accuracy' is ambiguous: it is not clear whether this refers to absolute percentage points (e.g., from 70% to 80%) or relative improvement (e.g., from 70% to 77%).","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be an early preprint; the experimental section lacks the detail expected for a journal submission. The main concern is that the authors do not yet have a formal definition of their own metric, which makes the entire evaluation opaque. I would advise the editor to require a major revision with the metric definition, proper statistical comparison, and a clarification of the evaluation protocol before considering publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a plausible idea with a real empirical footprint, but the paper's headline claim is not supported by its own tables, and the missing definitions make it impossible to reproduce from the text alone. It deserves one serious revision cycle, not acceptance as is.\n\nWhat's actually new is the specific combination of UCB-based switching between two synthetic-image selection scores (DPS/FCS) during fine-tuning, plus an LLM+Stable Diffusion pipeline for attribute-aware generation. I haven't seen that exact package elsewhere. The experimental breadth is decent: six architectures, three tasks, artistic vs photorealistic variants, and an ablation with multiple UCB arm counts. They also promise code, datasets, and additional materials, which is real value if the repo is complete.\n\nThe soft spots are substantial. DPS is never defined. Section 3.3 gives a figure and prose about means and standard deviations, but no closed-form equation for Psi. Equation (3) introduces alpha(t) as a temporally adaptive parameter, then alpha never appears in the implemented two-dimensional score U = [Psi, Phi]. So the 'dynamic metric' is actually static per-image scores; the dynamism is the UCB arm switching. That's not a fatal flaw in itself, but it is a mismatch between the text and the method.\n\nAll numbers in Tables 4 and 5 are single runs with no error bars or significance tests. Some margins are small enough that seed noise could flip them. The stress-test note is right about Table 5: the 4-arm UCB with SSIM/PSNR/IS/FID beats the proposed 2-arm U on AlexNet and VGG in Car Accidents (86 vs 80, 80 vs 77). It's only two cells, not 'several', but that is enough to falsify 'consistently superior' and 'significantly outperformed' as stated. The Table 4 comparison is also confounded because proposed uses UCB while baselines use static rankings. Table 5 is the fairer test, and it doesn't give a clean win.\n\nOne more concern: UCB rewards are computed from validation accuracy, and the paper never says whether the same validation split later selects the final checkpoint. If it does, part of the reported gain could be selection on the validation set. The missing hyperparameters (patience, M, K, normalization details) make the method unreproducible from the manuscript alone.\n\nWhere does that leave us? The method is credible and the artifact is substantial. The problems are evidence problems, not conceptual failures. A serious referee should look at it. The authors need to add metric definitions, multi-seed runs, baselines that also get UCB, and explicit validation-split handling. If those come back clean, this could be a useful paper on synthetic-data selection.","headline":"A plausible UCB-plus-metric idea with public data and code, but the core claim is contradicted by the paper's own ablation and the main metric is undefined; worth a revision cycle, not acceptance.","tokens_in":17667,"tokens_out":4007,"would_cite":true,"duration_ms":40594,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A UCB-based training loop that switches between two per-image synthetic-data quality scores improves fine-tuning accuracy by up to 10 percentage points across six architectures and three classification datasets.","keywords":["synthetic data","multi-armed bandit","UCB","usability metric","image classification","fine-tuning","Stable Diffusion","feature cohesion"],"falsifier":"Run the UCB training procedure with the two arms' identities hidden: keep the same data subsets but label them randomly, or swap Ψ and Φ between arms. If the reported accuracy gains persist under random or swapped labels, the selection criterion itself is not the cause. Alternatively, evaluate the same pipeline on a task whose images are far from ImageNet (e.g., medical scans or satellite imagery); if the gains vanish, the ImageNet-pretrained features are the real load-bearing part.","tokens_in":16563,"feed_emoji":"🎰","tokens_out":7133,"duration_ms":63035,"temperature":0.7,"pith_summary":"This paper tries to establish that a training-time selection rule, not just a static quality score, determines how much synthetic images help a supervised classifier. It proposes a usability metric U with two components: one measuring diversity and photorealism from mid-level features, and one measuring feature cohesion between a synthetic image and its real class mean. A multi-armed bandit then switches between the two score-based data subsets during fine-tuning, treating validation accuracy as reward. The authors report that this dynamic selection beats static metrics such as FID, IS, SSIM, and PSNR, improving fine-tuning accuracy by up to 10 percentage points across six architectures on three classification datasets. If true, the practical lesson is that the same synthetic pool can be used more effectively by letting the model's current state decide which examples to train on.","feed_headline":"Bandit-paced training lifts synthetic-data accuracy by 10 points","feed_subtitle":"Switching between photorealism and cohesion scores during fine-tuning beats static metrics like FID on six architectures.","key_machinery":"The central object is the usability score U = (Ψ, Φ) together with the UCB-based training procedure. Ψ (Diversity and Photorealism Score) is computed from the mean and standard deviation of Inception-V3 feature vectors of the image, its class, and the dataset excluding its class; Φ (Feature Cohesion Score) is computed as 1/D_KL of the normalized VGG16 features of the synthetic image against the normalized mean features of real images in the same class. The UCB procedure treats the two score components as arms, computes rewards from validation accuracy after each epoch, and switches data subsets once validation accuracy is flat for more than a patience threshold, with exploration term β√(log(total counts)/(loader counts+ε)). This mechanism is what makes the selection dynamic rather than a one-time ranking.","core_discovery":"The paper's central claim is that per-image synthetic data usability is best captured by a two-dimensional score, U = (Ψ, Φ), where Ψ (DPS) combines the mean and standard deviation of Inception-V3 features for an image, its class, and the rest of the dataset to measure diversity and photorealism, and Φ (FCS) is the inverse KL divergence between the image's normalized VGG16 features and the normalized mean features of real images in the same class. On top of this score, the paper adds a UCB-based training loop with two arms, namely the top-M synthetic images ranked by Ψ and the top-M ranked by Φ, and switches arms when validation accuracy stops improving. The authors report that this combination achieves the best fine-tuning accuracy in Table 4 for AlexNet, EfficientNet, ViT, SwinTransformer, VGG, and REGNet on photorealistic synthetic versions of Car Accidents, CIFAR-10, and Birds-525, outperforming SSIM, PSNR, IS, and FID-based selection.","pith_inferences":["A natural extension is to apply the same two-arm UCB idea with other paired scoring criteria (for example, sharpness versus diversity, or adversarial robustness versus clean accuracy); the paper does not test this generality.","The static scores are computed once with ImageNet-pretrained extractors, so the approach may transfer poorly to domains far from ImageNet; testing on medical or satellite imagery would clarify whether the claim depends on those features.","Because the bandit reward is validation accuracy, there is a risk of overfitting the validation set when the patience and β hyperparameters are tuned on it; a cross-validated reward would make the improvement claim more defensible.","The generation pipeline couples an LLM with Stable Diffusion, but the paper does not compare this attribute-aware pipeline against random prompting; such a comparison would isolate the contribution of the LLM-generated attributes."],"forward_implications":["Fine-tuning on a fixed top-M selection of synthetic images is suboptimal; allowing the training loop to alternate between two quality criteria can raise accuracy by several points on standard classification benchmarks.","The reported gains hold across very different architectures (convolutional, transformer, and scaled networks), suggesting the dynamic selection helps the training procedure itself rather than one model family.","Traditional generative-quality metrics such as FID and IS, which score a whole dataset, can be outperformed by per-image scores that combine mid-level and high-level features.","Because the method only needs two precomputed feature sets, it can be added to an existing fine-tuning pipeline with modest overhead."],"supporting_citations":[{"why":"Supplies the FID baseline that the proposed metric must beat in the fine-tuning comparisons.","marker":"[36]"},{"why":"Supplies the Inception Score baseline used in the same comparisons.","marker":"[37]"},{"why":"Defines the Inception-V3 feature extractor used to compute the DPS (Ψ) component of the usability score.","marker":"[63]"},{"why":"Defines the KL divergence used in the FCS (Φ) component of the usability score.","marker":"[70]"},{"why":"Defines the Stable Diffusion-V2 model used by the attribute-aware generation pipeline to create the synthetic datasets.","marker":"[71]"}],"fun_headline_variants":["Bandit-trained models pick better synthetic images, up 10% accuracy","UCB arm-switching on synthetic data beats static FID in 6 classifiers","Synthetic data usability score + bandit loop boost accuracy by 10 points","Dynamic bandit ranks synthetic images, improving classifiers up to 10%","LLM+Stable Diffusion bandit pipeline lifts synthetic training accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that two static per-image scores, computed once with ImageNet-pretrained Inception-V3 and VGG16 features, stay the right selection criterion for every downstream classifier at every epoch, and that validation accuracy can safely guide the switch between them.","fun_headline_variants_meta":{"raw":{"variants":["Bandit-trained models pick better synthetic images, up 10% accuracy","UCB arm-switching on synthetic data beats static FID in 6 classifiers","Synthetic data usability score + bandit loop boost accuracy by 10 points","Dynamic bandit ranks synthetic images, improving classifiers up to 10%","LLM+Stable Diffusion bandit pipeline lifts synthetic training accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00024,"raw_usage":{"total_tokens":1544,"prompt_tokens":995,"completion_tokens":549,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":451}},"tokens_in":611,"tokens_out":549,"duration_ms":5841,"temperature":1.0,"reasoning_tokens":451,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:41:51.648368+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the UCB training procedure with the two arms' identities hidden: keep the same data subsets but label them randomly, or swap Ψ and Φ between arms. If the reported accuracy gains persist under random or swapped labels, the selection criterion itself is not the cause. Alternatively, evaluate the same pipeline on a task whose images are far from ImageNet (e.g., medical scans or satellite imagery); if the gains vanish, the ImageNet-pretrained features are the real load-bearing part.","supporting_citations":[{"cited_title":"Advances in neural information processing systems 30 (2017)","cited_arxiv_id":null,"evidence_quote":"Supplies the FID baseline that the proposed metric must beat in the fine-tuning comparisons."},{"cited_title":"Advances in neural information processing systems 29 (2016)","cited_arxiv_id":null,"evidence_quote":"Supplies the Inception Score baseline used in the same comparisons."},{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp","cited_arxiv_id":null,"evidence_quote":"Defines the Stable Diffusion-V2 model used by the attribute-aware generation pipeline to create the synthetic datasets."}],"review_version":1}