{"id":"19c2966d-1b80-4cc9-a626-58803cfbdc76","arxiv_id":"2412.01477","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"SHAP-guided edits to synthetic 3D vehicle models improved infrared vehicle detection mAP50 from 94.6% to 96.1%, though the gain is not shown to be independent of test-set inspection.","lead":"This paper uses AI explanation maps to guide how synthetic 3D vehicle models are modified, so that models trained on fake infrared images get better at recognizing real vehicles. In a small proof of concept, the method added 1.5% accuracy beyond the gain from plain synthetic data, but the evaluation is weakened because the same test images helped choose the edits.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 1.5% XAI-guided gain is confounded by test-set-informed edit selection and lacks any random-edit control; the central causal claim is not yet established.","rationale":"The reader's verdict is REJECT with high correctness risk, and the reader's weakest_assumption identifies exactly the two gaps I find most load-bearing: test-set-informed edit selection and the missing random-modification control. My independent reading of the paper confirms both. The method section says SHAP values are computed on '50 randomly selected test samples'; confusion matrices in Section 5.1 and in the supplementary show test-set predictions; orientation-wise misclassification breakdowns in Supplementary Figs. 23-24 are explicitly over the test dataset; and the final mAP50 in Table 1 is reported on that same Otest set. No validation split is mentioned anywhere. The paper deserves credit for averaging over 4 seeds, constructing an orientation-disjoint real train/test split with PCA evidence of reduced leakage, and controlling for dataset size and ratio, and for consistent gains across YOLOv8n/s/x. Those strengthen the 4.6% claim for adding v0 synthetic data. They do not, however, support the central claim that the additional 1.5% is caused by the XAI-guided modifications specifically. The absence of a random-edit control means even a clean validation split, while necessary, would not by itself isolate the effect of SHAP guidance. These are methodological gaps, not stylistic issues, and they undermine the paper's headline claim as stated. REJECT remains the appropriate verdict: the current evidence does not support the causal claim, although the framework could become valid with a proper validation split and controls.","tokens_in":20548,"tokens_out":1927,"duration_ms":18148,"concrete_test":"Freeze a validation split drawn from the same orientation distributions as Otest (or use orientation-disjoint synthetic data for selection). Compute SHAP attributions and confusion-matrix targets using only this validation split; apply the two mesh edits; then evaluate once on the previously untouched Otest. Additionally, generate at least three control synthetic variants matched in edit count and approximate visual magnitude but with edit locations chosen randomly (e.g., random mesh faces, random material brightness/smoothness changes) and report their mAP50. If any random-edit control matches or exceeds the SHAP-guided 96.1%, the XAI guidance has no demonstrable added value; if Otest-informed selection was used for the current edits, the reported gain is not a valid estimate regardless of direction.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim is that SHAP-guided mesh edits outperform simply adding v0 synthetic data (94.6% to 96.1% mAP50). This depends on the edits being selected from information independent of the final evaluation set. Section 2 states SHAP attributions are computed over '50 randomly selected test samples', and Section 5.1 selects target misclassifications from confusion matrices that are computed on Otest (Figs. 4, 8, 23, 24; Supplementary Figs. 23-24 are explicitly over the test dataset). No separate validation split or protocol is described anywhere in the main text or supplementary. Unless all modifications were frozen before any test-set inspection, the 1.5% is an in-sample, human-in-the-loop optimization score rather than an unbiased estimate of the XAI workflow's value. A second load-bearing gap is the absence of any control for arbitrary mesh edits. The authors include controls for dataset size, real:synthetic ratio, and training-time methods (varifocal loss, OHEM), but no control that changes the mesh in a non-SHAP-guided way while matching the number and magnitude of edits. Without either independence of the selection data or a random-edit baseline, the causal attribution of the 1.5% gain to the SHAP-guided common/unique-feature logic is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a human-in-the-loop framework in which SHAP saliency maps computed on a detection model's failures are used to guide modifications to the 3D mesh models used to generate synthetic training images. The authors distinguish 'unique' features (which should be reinforced) from 'common' features (which should be disrupted), motivated by a simple logistic-regression toy model. On the DSIAC ATR infrared vehicle dataset, they report that adding a base set of 9,000 synthetic images to 9,000 real images raises YOLOv8n mAP50 from 90.0% to 94.6%, and that further SHAP-guided reinforcing and disruptive mesh edits raise it to 96.1% (Table 1). The same edits also improve YOLOv8s, with mixed results for YOLOv8x.","tokens_in":20868,"tokens_out":6091,"duration_ms":49303,"significance":"The idea of using XAI to decide where and how to increase or decrease realism in synthetic data is novel and practically relevant, and the paper contains several methodological strengths: the orientation-based train/test split is designed to avoid the temporal leakage that affects much prior work on this dataset; the authors include controls for dataset size and real:synthetic ratio; results are averaged over multiple seeds; and the toy model gives a clean conceptual vocabulary for the editing rules. If the 96.1% result were obtained from a clean evaluation, the paper would be a useful proof of concept for explainable synthetic-data curation. As it stands, the key empirical claim is not yet supported, because the edits were selected using the same test set that is later used for evaluation.","major_comments":[{"comment":"The SHAP attributions guiding the mesh edits are computed over '50 randomly selected test samples' (Section 2), and the target misclassifications are chosen from confusion matrices that are computed on the test orientation split Otest (Figs. 4 and 8; Supplementary Figs. 23 and 24 are explicitly described as being 'over the test dataset'). The final mAP50 is also reported on Otest, so the 1.5% gain (94.6% to 96.1%) is an in-sample, human-in-the-loop optimization score rather than an unbiased estimate of the XAI workflow's value. The manuscript even refers to these samples as 'validation images' in Section 5.1, which is inconsistent with the 'test samples' wording in Section 2. To validate the claim, the authors must either introduce a separate validation split used for all SHAP-based selection and threshold decisions, or pre-register the modifications before any test-set inspection.","section":"Section 2 and Section 5.1"},{"comment":"No control is reported for arbitrary mesh edits that are not guided by SHAP. The comparisons to varifocal loss and OHEM only rule out two specific training-time alternatives; they do not show that the SHAP-guided unique/common-feature logic is what causes the gain. A control in which the same number and type of mesh modifications are made without SHAP guidance (e.g., randomly, or on a different vehicle region) is needed to attribute the improvement to the XAI procedure rather than to any dataset perturbation or to the operator's prior domain knowledge.","section":"Section 5.1 and Table 1"}],"minor_comments":[{"comment":"There is a typo in the introduction: 'whilst previous works have have either improved' should read 'have either improved'.","section":"Section 1"},{"comment":"It would be helpful to report real-only baselines for YOLOv8s and YOLOv8x; the current columns start with Real+Syn v0, so the reader cannot directly assess whether synthetic data helps or hurts for those architectures (note that for YOLOv8x, v0 gives 89.6%, below the YOLOv8n real-only 90.0%, and no real-only YOLOv8x number is given).","section":"Table 1"},{"comment":"The description of the SHAP masking thresholds (40% contribution threshold and 50% visualization mask threshold) is clear, but the authors should state explicitly whether these thresholds were chosen before or after observing test performance; if after, they are additional in-sample tuning choices.","section":"Section 2 and Figure 5"},{"comment":"The caption says 'our dataset size is also 9000 images' for the ratio experiment, while the main experiments use 9000 real plus 9000 synthetic images; clarifying this distinction in the main text would prevent confusion about the dataset sizes used in the two analyses.","section":"Supplementary Figure 22"}],"recommendation":"major_revision","confidential_remarks":"The protocol problem here is serious and some editors might treat it as grounds for rejection. I am recommending major revision rather than rejection because the core idea and the base synthetic-data result are sound, and the flaw is correctable by re-running the workflow with a properly separated validation set and adding a random-edit control. If the authors cannot or will not provide such a re-run, the paper should not be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful part: this is a real idea. SHAP-guided editing of 3D meshes in a game engine to steer synthetic training data is not in the cited literature, and the reinforce/disrupt framing is a clean way to think about unique versus common features. The toy model in Section 2 is internally consistent, and the orientation-based train/test split is a genuinely careful move that avoids the frame-leakage problem common in this dataset. They also run multiple seeds and compare against varifocal loss and OHEM, which is more than many proof-of-concept papers do.\n\nBut the central empirical claim does not survive contact with the method. Section 2 says SHAP attributions are computed over '50 randomly selected test samples'; Section 5.1 picks target misclassifications from confusion matrices that are computed on the test set (Figs. 4 and 8). There is no separate validation split anywhere. So the 1.5% improvement from 94.6% to 96.1% was obtained after the human operator had already seen the test set, selected which classes to modify, and tailored the mesh edits to those specific failures. That is in-sample optimization, not an unbiased estimate of the workflow's value. The paper's own numbers prove the point: the same modifications used for YOLOv8n give a different pattern on YOLOv8s/y, which is unsurprising if you are tuning to a particular model's failures.\n\nThe second soft spot is the missing control for arbitrary mesh edits. They control for dataset size, real:synthetic ratio, and training-time methods, but there is no baseline that changes the mesh in a non-SHAP-guided way with a similar number and magnitude of edits. Without that, even a proper validation split would not tell you whether SHAP's guidance is what drives the gain, or simply that any deliberate augmentation helps.\n\nMinor: no error bars or standard deviations are reported for the 4-seed averages, which matters for a 1.5% claim.\n\nIs the paper worth a referee? Yes, but with heavy revision. The novelty and the careful split are real assets, and a human-in-the-loop XAI workflow for synthetic data curation is worth testing seriously. But as it stands, the 1.5% result should be treated as a motivated hypothesis, not a demonstrated effect. A revised version needs a validation set for edit selection, a held-out test set, a random-edit control, and ideally code/data release so the protocol can be checked.\n\nI'd bring it to a reading group as a case study in test-set overfitting in human-in-the-loop research, but I wouldn't cite the numerical claim until it's re-run with a clean protocol.","headline":"Novel SHAP-guided mesh editing idea, but the 1.5% gain is tuned on the test set and lacks a random-edit control, so the causal claim is not yet established.","tokens_in":21371,"tokens_out":4568,"would_cite":false,"duration_ms":35649,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that SHAP-guided edits to synthetic 3D mesh models—reinforcing distinctive features and disrupting confusable ones—improve infrared vehicle detection beyond simply adding unmodified synthetic data.","keywords":["synthetic data","explainable AI","SHAP","object detection","infrared imagery","human-in-the-loop","YOLOv8","domain randomization"],"falsifier":"Retrain with the same mesh edits after choosing them from SHAP maps computed on a validation split that is never used for the final evaluation; if mAP50 no longer rises 1.5% over the unmodified synthetic baseline, the reported gain was an in-sample fit. A second check is to apply random mesh edits of comparable visual magnitude in the same regions; if they produce the same gain, XAI guidance is not the cause.","tokens_in":20323,"feed_emoji":"🎯","tokens_out":6150,"duration_ms":51255,"temperature":0.7,"pith_summary":"The paper proposes that when synthetic training images are used to improve object detection, the choice of what to change in the 3D models behind those images can be guided by explainable AI rather than trial and error. Its central claim is that SHAP saliency maps, compared across correct classifications and a target misclassification, let a human operator locate unique features that distinguish two vehicle classes and common features that confuse them, and that editing a game-engine mesh to reinforce or disrupt those features improves detection beyond simply adding unmodified synthetic data. In the infrared vehicle-detection test bed, this workflow raises mean average precision at IoU 0.5 from 90.0% with real data alone to 94.6% with initial synthetic data and then to 96.1% after the XAI-guided edits. The paper also shows that both increasing realism on distinctive regions and deliberately reducing realism on confusable regions can help, and that these targeted data changes outperform training-time fixes such as varifocal loss and online hard example mining.","feed_headline":"XAI-guided 3D model tweaks lift detection to 96.1%","feed_subtitle":"Human-in-the-loop mesh edits add 1.5% over unmodified synthetic data in infrared target detection.","key_machinery":"The carrying object is the distinction between unique and common features, formalized in a toy linear classifier where the optimal class weight vector is $w_k^* \\propto \\mathbb{E}[\\phi(x_k)] - \\mathbb{E}[\\phi(x_{k'})]$, so large weights go to features prominent in class $k$ but not $k'$ and small weights to features prominent in both. The mechanism that identifies these features in practice is a comparison of three SHAP (SHapley Additive exPlanations) saliency maps: the correct classification of class A, the correct classification of class B, and the misclassification of A as B, clustered by vehicle orientation. High saliency at the same location in the misclassification and in the correct classification of B marks a common feature to disrupt; saliency present only in the correct classification of A marks a unique feature to reinforce. The edits are applied to material properties of faces in Unity 3D mesh models via Probuilder, so the change is in the training data, not in the loss function or model weights.","core_discovery":"On the paper's own terms, the discovery is that misclassifications between visually similar classes can be traced, via SHAP attributions, to specific regions of the image where the two classes share a bright or salient pattern, and that editing the synthetic mesh model in exactly those regions steers the trained detector away from the confusion. Reinforcing edits make the unique features of a class more prominent in the rendered images (here, changing the SUV's surface smoothness to remove reflections and match the real rear and side texture), while disruptive edits deliberately reduce realism in a region that is common to two classes (here, darkening the ZSU23's hot engine panel so it no longer resembles the BTR70's hot exhaust). Both edits individually raise mAP50 from 94.6% to 95.7%, and together they raise it to 96.1%, averaged over four random seeds. The same edits also improve YOLOv8s and YOLOv8x, though the size of the gain differs.","pith_inferences":["A testable extension would be to turn the unique/common feature criterion into a quantitative design rule: modify synthetic data where the SHAP correlation between misclassified samples and the confusable class is high and the corresponding image regions are visually similar.","The disruptive modification is essentially targeted domain randomization; a natural extension is to randomize only the SHAP-identified common regions across many synthetic samples and measure whether the gain scales with the number of disrupted regions.","The procedure's dependence on a human reading SHAP heat maps suggests an automation path: map 2D saliency onto the 3D mesh via ray tracing and let a second model propose mesh edits, which the authors mention as future work."],"forward_implications":["If the proof of concept holds, synthetic data for object detection can be curated by letting the model's own explanation point to the exact mesh region to edit, reducing the trial-and-error burden on human operators.","Both increasing realism (reinforcing unique features) and decreasing realism (disrupting common features) can improve performance, so realism is not a one-way axis for synthetic data design.","The XAI-guided edits transfer to larger YOLOv8 variants, with reinforcing giving larger gains for YOLOv8s and YOLOv8x than disruptive; the size of the gain varies with architecture.","In this setting, targeted synthetic-data modification outperforms altering the training procedure with varifocal loss or online hard example mining, which slightly reduced mAP50 relative to unmodified synthetic data."],"supporting_citations":[{"why":"Supplies the real infrared dataset and the vehicle classes whose unseen orientations form the test bed.","marker":"[10]"},{"why":"Supplies the SHAP saliency method used to locate unique and common features from model explanations.","marker":"[28]"},{"why":"Provides the Unity tool used to set material properties of mesh faces in order to implement the reinforcing and disruptive edits.","marker":"[39]"},{"why":"Prior saliency-guided augmentation baseline, contrasted with the mesh-level modification approach of this paper.","marker":"[43]"},{"why":"Online hard example mining baseline against which the synthetic-data modifications are compared.","marker":"[53]"},{"why":"Domain randomization, the conceptual basis for disruptive modifications that reduce realism in synthetic data.","marker":"[57]"},{"why":"Unity game engine used to render synthetic infrared images from the modified 3D models.","marker":"[58]"},{"why":"Defines the YOLOv8 detection architecture on which all reported mAP50 numbers are measured.","marker":"[59]"},{"why":"Varifocal loss baseline against which the synthetic-data modifications are compared.","marker":"[66]"}],"fun_headline_variants":["XAI-guided mesh edits lift infrared detection to 96.1%","Explainable AI probes synthetic data, improving object detection by 1.5%","Synthetic data curated by XAI: realism shifts raise mAP50 to 96.1%","SHAP pinpoints mesh tweaks, boosting vehicle detection to 96.1%","Both realistic and abstract edits, guided by XAI, enhance detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported 1.5% gain assumes the test images used to compute the final mAP50 were not the same images whose SHAP maps and confusion matrices were used to choose the mesh edits, but the paper does not describe a separate validation split.","fun_headline_variants_meta":{"raw":{"variants":["XAI-guided mesh edits lift infrared detection to 96.1%","Explainable AI probes synthetic data, improving object detection by 1.5%","Synthetic data curated by XAI: realism shifts raise mAP50 to 96.1%","SHAP pinpoints mesh tweaks, boosting vehicle detection to 96.1%","Both realistic and abstract edits, guided by XAI, enhance detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000589,"raw_usage":{"total_tokens":2829,"prompt_tokens":1077,"completion_tokens":1752,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":693,"completion_tokens_details":{"reasoning_tokens":1646}},"tokens_in":693,"tokens_out":1752,"duration_ms":13125,"temperature":1.0,"reasoning_tokens":1646,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:17:57.860047+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain with the same mesh edits after choosing them from SHAP maps computed on a validation split that is never used for the final evaluation; if mAP50 no longer rises 1.5% over the unmodified synthetic baseline, the reported gain was an in-sample fit. A second check is to apply random mesh edits of comparable visual magnitude in the same regions; if they produce the same gain, XAI guidance is not the cause.","supporting_citations":[{"cited_title":"https://dsiac.org/databases/atr-algorithm- development-image-database/, 2014","cited_arxiv_id":null,"evidence_quote":"Supplies the real infrared dataset and the vehicle classes whose unseen orientations form the test bed."},{"cited_title":"Lundberg and Su-In Lee","cited_arxiv_id":null,"evidence_quote":"Supplies the SHAP saliency method used to locate unique and common features from model explanations."},{"cited_title":"https://unity.com/features/probuilder, 2018","cited_arxiv_id":null,"evidence_quote":"Provides the Unity tool used to set material properties of mesh faces in order to implement the reinforcing and disruptive edits."},{"cited_title":"Girshick","cited_arxiv_id":null,"evidence_quote":"Online hard example mining baseline against which the synthetic-data modifications are compared."},{"cited_title":"Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization","cited_arxiv_id":null,"evidence_quote":"Domain randomization, the conceptual basis for disruptive modifications that reduce realism in synthetic data."},{"cited_title":"https://unity.com/","cited_arxiv_id":null,"evidence_quote":"Unity game engine used to render synthetic infrared images from the modified 3D models."},{"cited_title":"Yolov8: A novel object detection algorithm with enhanced performance and robust- ness","cited_arxiv_id":null,"evidence_quote":"Defines the YOLOv8 detection architecture on which all reported mAP50 numbers are measured."},{"cited_title":"VarifocalNet: An IoU-aware Dense Object Detec- tor","cited_arxiv_id":null,"evidence_quote":"Varifocal loss baseline against which the synthetic-data modifications are compared."}],"review_version":1}