Pith. sign in

REVIEW 2 major objections 4 minor 77 references

Improving Object Detection by Modifying Synthetic Data with Explainable AI

T0 review · 2 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that SHAP-guided edits to synthetic 3D mesh models—reinforcing distinctive features and disrupting confusable ones—improve infrared vehicle detection beyond simply adding unmodified synthetic data.

desk verdict Novel SHAP-guided mesh editing idea, but the 1.5% gain is tuned on the test set and lacks a random-edit control, so the causal claim is not yet established. read the letter →

arxiv 2412.01477 v3 pith:XD7JTH7F submitted 2024-12-02 cs.CV

Nitish Mital , Simon Malzard , Richard Walters , Celso M. De Melo , Raghuveer Rao , Victoria Nockles This is my paper · ORCID
classification cs.CV
keywords syntheticdataexplainableAISHAPobjectdetectioninfraredimageryhuman-in-the-loopYOLOv8domainrandomization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that when synthetic training images are used to improve object detection, the choice of what to change in the 3D models behind those images can be guided by explainable AI rather than trial and error. Its central claim is that SHAP saliency maps, compared across correct classifications and a target misclassification, let a human operator locate unique features that distinguish two vehicle classes and common features that confuse them, and that editing a game-engine mesh to reinforce or disrupt those features improves detection beyond simply adding unmodified synthetic data. In the infrared vehicle-detection test bed, this workflow raises mean average precision at IoU 0.5 from 90.0% with real data alone to 94.6% with initial synthetic data and then to 96.1% after the XAI-guided edits. The paper also shows that both increasing realism on distinctive regions and deliberately reducing realism on confusable regions can help, and that these targeted data changes outperform training-time fixes such as varifocal loss and online hard example mining.

What carries the argument

The carrying object is the distinction between unique and common features, formalized in a toy linear classifier where the optimal class weight vector is $w_k^* \propto \mathbb{E}[\phi(x_k)] - \mathbb{E}[\phi(x_{k'})]$, so large weights go to features prominent in class $k$ but not $k'$ and small weights to features prominent in both. The mechanism that identifies these features in practice is a comparison of three SHAP (SHapley Additive exPlanations) saliency maps: the correct classification of class A, the correct classification of class B, and the misclassification of A as B, clustered by vehicle orientation. High saliency at the same location in the misclassification and in the correct classification of B marks a common feature to disrupt; saliency present only in the correct classification of A marks a unique feature to reinforce. The edits are applied to material properties of faces in Unity 3D mesh models via Probuilder, so the change is in the training data, not in the loss function or model weights.

What would settle it

Retrain with the same mesh edits after choosing them from SHAP maps computed on a validation split that is never used for the final evaluation; if mAP50 no longer rises 1.5% over the unmodified synthetic baseline, the reported gain was an in-sample fit. A second check is to apply random mesh edits of comparable visual magnitude in the same regions; if they produce the same gain, XAI guidance is not the cause.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that misclassifications between visually similar classes can be traced, via SHAP attributions, to specific regions of the image where the two classes share a bright or salient pattern, and that editing the synthetic mesh model in exactly those regions steers the trained detector away from the confusion. Reinforcing edits make the unique features of a class more prominent in the rendered images (here, changing the SUV's surface smoothness to remove reflections and match the real rear and side texture), while disruptive edits deliberately reduce realism in a region that is common to two classes (here, darkening the ZSU23's hot engine panel so it no longer resembles the BTR70's hot exhaust). Both edits individually raise mAP50 from 94.6% to 95.7%, and together they raise it to 96.1%, averaged over four random seeds. The same edits also improve YOLOv8s and YOLOv8x, though the size of the gain differs.

Load-bearing premise

The reported 1.5% gain assumes the test images used to compute the final mAP50 were not the same images whose SHAP maps and confusion matrices were used to choose the mesh edits, but the paper does not describe a separate validation split.

Editorial extensions

If this is right

  • If the proof of concept holds, synthetic data for object detection can be curated by letting the model's own explanation point to the exact mesh region to edit, reducing the trial-and-error burden on human operators.
  • Both increasing realism (reinforcing unique features) and decreasing realism (disrupting common features) can improve performance, so realism is not a one-way axis for synthetic data design.
  • The XAI-guided edits transfer to larger YOLOv8 variants, with reinforcing giving larger gains for YOLOv8s and YOLOv8x than disruptive; the size of the gain varies with architecture.
  • In this setting, targeted synthetic-data modification outperforms altering the training procedure with varifocal loss or online hard example mining, which slightly reduced mAP50 relative to unmodified synthetic data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would be to turn the unique/common feature criterion into a quantitative design rule: modify synthetic data where the SHAP correlation between misclassified samples and the confusable class is high and the corresponding image regions are visually similar.
  • The disruptive modification is essentially targeted domain randomization; a natural extension is to randomize only the SHAP-identified common regions across many synthetic samples and measure whether the gain scales with the number of disrupted regions.
  • The procedure's dependence on a human reading SHAP heat maps suggests an automation path: map 2D saliency onto the 3D mesh via ray tracing and let a second model propose mesh edits, which the authors mention as future work.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes a human-in-the-loop framework in which SHAP saliency maps computed on a detection model's failures are used to guide modifications to the 3D mesh models used to generate synthetic training images. The authors distinguish 'unique' features (which should be reinforced) from 'common' features (which should be disrupted), motivated by a simple logistic-regression toy model. On the DSIAC ATR infrared vehicle dataset, they report that adding a base set of 9,000 synthetic images to 9,000 real images raises YOLOv8n mAP50 from 90.0% to 94.6%, and that further SHAP-guided reinforcing and disruptive mesh edits raise it to 96.1% (Table 1). The same edits also improve YOLOv8s, with mixed results for YOLOv8x.

Significance. The idea of using XAI to decide where and how to increase or decrease realism in synthetic data is novel and practically relevant, and the paper contains several methodological strengths: the orientation-based train/test split is designed to avoid the temporal leakage that affects much prior work on this dataset; the authors include controls for dataset size and real:synthetic ratio; results are averaged over multiple seeds; and the toy model gives a clean conceptual vocabulary for the editing rules. If the 96.1% result were obtained from a clean evaluation, the paper would be a useful proof of concept for explainable synthetic-data curation. As it stands, the key empirical claim is not yet supported, because the edits were selected using the same test set that is later used for evaluation.

major comments (2)
  1. [Section 2 and Section 5.1] The SHAP attributions guiding the mesh edits are computed over '50 randomly selected test samples' (Section 2), and the target misclassifications are chosen from confusion matrices that are computed on the test orientation split Otest (Figs. 4 and 8; Supplementary Figs. 23 and 24 are explicitly described as being 'over the test dataset'). The final mAP50 is also reported on Otest, so the 1.5% gain (94.6% to 96.1%) is an in-sample, human-in-the-loop optimization score rather than an unbiased estimate of the XAI workflow's value. The manuscript even refers to these samples as 'validation images' in Section 5.1, which is inconsistent with the 'test samples' wording in Section 2. To validate the claim, the authors must either introduce a separate validation split used for all SHAP-based selection and threshold decisions, or pre-register the modifications before any test-set inspection.
  2. [Section 5.1 and Table 1] No control is reported for arbitrary mesh edits that are not guided by SHAP. The comparisons to varifocal loss and OHEM only rule out two specific training-time alternatives; they do not show that the SHAP-guided unique/common-feature logic is what causes the gain. A control in which the same number and type of mesh modifications are made without SHAP guidance (e.g., randomly, or on a different vehicle region) is needed to attribute the improvement to the XAI procedure rather than to any dataset perturbation or to the operator's prior domain knowledge.
minor comments (4)
  1. [Section 1] There is a typo in the introduction: 'whilst previous works have have either improved' should read 'have either improved'.
  2. [Table 1] It would be helpful to report real-only baselines for YOLOv8s and YOLOv8x; the current columns start with Real+Syn v0, so the reader cannot directly assess whether synthetic data helps or hurts for those architectures (note that for YOLOv8x, v0 gives 89.6%, below the YOLOv8n real-only 90.0%, and no real-only YOLOv8x number is given).
  3. [Section 2 and Figure 5] The description of the SHAP masking thresholds (40% contribution threshold and 50% visualization mask threshold) is clear, but the authors should state explicitly whether these thresholds were chosen before or after observing test performance; if after, they are additional in-sample tuning choices.
  4. [Supplementary Figure 22] The caption says 'our dataset size is also 9000 images' for the ratio experiment, while the main experiments use 9000 real plus 9000 synthetic images; clarifying this distinction in the main text would prevent confusion about the dataset sizes used in the two analyses.

Circularity Check

1 steps flagged · score 6.0 of 10

The reported 1.5% XAI-guided gain is selected on the test set: SHAP attributions and target misclassifications come from Otest, and the final mAP50 is measured on the same Otest, making the central empirical claim an in-sample fit.

  1. fitted input called prediction [Section 2 (SHAP computation) and Section 5.1 (target misclassification selection and evaluation); Table 1.]
    "We compute pixel attributions over 50 randomly selected test samples for which the model predicts high confidence for the specified class. ... To understand the increase in mAP50 score from the addition of the v0 synthetic data to the training dataset, we compare the confusion matrices in Fig. 8a,b ... In the following sections we target these misclassifications using our new procedure from Section 2.1, demonstrating the use of both Reinforcing and Disruptive modifications to the synthetic data v0 in order to reduce model confusion and improve performance."

    The Reinforcing and Disruptive mesh edits are selected by inspecting SHAP attributions over test samples and by picking target misclassifications from confusion matrices computed on the same Otest used for all reported mAP50 values. The human operator's edit choices are therefore optimized against the test set, and the resulting mAP50 increase from 94.6% to 96.1% is the value of the very objective (reducing those test misclassifications) that guided the edits. The final score is not an independent prediction of the XAI-guided workflow; it is an in-sample fit of the mesh modifications to the evaluation set. No separate validation split, pre-specified edit protocol, or random-edit control is presented to break this loop.

full rationale

The toy derivation in Section 2 (optimal weights w*_k = E[phi(x_k)] - E[phi(x_k')]) is self-contained and not circular; it merely motivates why reinforcing unique features and disrupting common features could help. The v0-versus-real-only comparison (94.6% vs 90.0%) is also a legitimate external comparison because v0 was not chosen by inspecting test errors. The circularity lies in the XAI-guided step: the SHAP maps that drive the mesh modifications are computed on '50 randomly selected test samples' (Section 2), the target misclassifications are taken from confusion matrices on the same Otest (Section 5.1 and Figs. 4/8, with mAP50 reported on Otest), and the final mAP50 after modification is measured on that same Otest (Table 1). The human-in-the-loop operator is effectively minimizing test-set confusion without any held-out split; the 1.5% improvement is therefore a fitted result selected against the evaluation metric, not an independent estimate of the XAI workflow's causal benefit. The absence of a random-edit control further means the gain cannot be attributed to the specific XAI logic, though that is a confounding/control issue rather than a definitional circularity. Self-citations to the authors' prior work are not load-bearing for the central result. Overall partial circularity: score 6.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central empirical result rests on a chain of choices: a toy linear model is assumed to describe deep classifiers; SHAP maps are assumed to locate class-confusing features; human visual mapping from 2D saliency to 3D mesh edits is assumed reliable; and Unity material settings are assumed to translate into the intended image changes. None of these is validated independently, and the free parameters are set by inspection rather than measurement. No new particles, forces, or physical entities are introduced.

free parameters (7)
  • real:synthetic data ratio = 0.5
    Selected from a performance sweep (Supplementary Fig. 22a) as optimal; used in all reported experiments.
  • SHAP contribution threshold = 40% of per-sample maximum
    User-defined threshold in Section 2 to count a pixel as contributing; affects saliency maps and thus which mesh region is edited.
  • SHAP visualization mask threshold = 50% of maximum score
    User-defined threshold in Section 2 for masking low-contribution pixels in displayed maps.
  • SHAP computation settings = 1000 random masks, 50 test samples
    Choices in Section 2 that shape the stability and localization of attributions.
  • orientation bin size for SHAP comparison = 5 degrees
    Chosen in Section 5.1 and Supplementary Section 11 to group validation samples by vehicle orientation.
  • SUV mesh material smoothness reduction = not quantified
    Human operator edits material parameter in Unity (Section 5.1) to reduce reflections; exact value absent.
  • ZSU23 engine panel darkening = not quantified
    Human operator darkens hot engine panel (Section 5.1); exact change absent and no visual or radiometric measurement.
assumptions (6)
  • domain assumption The toy binary classifier derivation (w*_k proportional to E[phi(x_k)] - E[phi(x_k')]) is assumed to transfer to deep object detectors with nonlinear, entangled features.
    Section 2 states the toy model is for motivation and 'suggest[s] this concept is more generally applicable', without proof for deep networks.
  • ad hoc to paper Kernel SHAP over 1000 random masks localizes the visual features responsible for class confusion.
    Section 2 relies on SHAP attributions to identify unique and common features; no validation that SHAP regions correspond to causally editable mesh parts.
  • ad hoc to paper A human operator can reliably map 2D saliency regions to 3D mesh properties by visual inspection.
    Section 2.1 step 5 and the examples in Section 5.1 require this mapping; no inter-operator agreement or objective metric is reported.
  • ad hoc to paper Modifying material smoothness and emission in Unity produces image changes with the intended reinforcing or disrupting effect and no harmful side effects.
    Section 5.1 assumes the smoothness reduction makes the SUV more uniform and the darkening removes confusion; only aggregate mAP is reported.
  • domain assumption The orientation-based train/test split removes data leakage sufficiently for mAP gains to be interpreted as generalization.
    Supplementary Section 12 uses PCA to argue reduced leakage, but PCA separation does not prove the model cannot exploit remaining correlations.
  • domain assumption Four random seeds (three for YOLOv8x) are enough to treat mAP differences of 1.1-1.5% as reliable.
    Table 1 reports averages without variance; no significance tests are provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Object Detection by Modifying Synthetic Data with Explainable AI." pith.science (2026). https://pith.science/paper/XD7JTH7F

@misc{pith2026241201477,
  author       = {Pith},
  title        = {Pith review of: Improving Object Detection by Modifying Synthetic Data with Explainable AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XD7JTH7F}},
  note         = {Machine review of arXiv:2412.01477}
}
read the original abstract

Limited real-world data severely impacts model performance in many computer vision domains, particularly for samples that are underrepresented in training. Synthetically generated images are a promising solution, but 1) it remains unclear how to design synthetic training data to optimally improve model performance (e.g, whether and where to introduce more realism or more abstraction) and 2) the domain expertise, time and effort required from human operators for this design and optimisation process represents a major practical challenge. Here we propose a novel conceptual approach to improve the efficiency of designing synthetic images, by using robust Explainable AI (XAI) techniques to guide a human-in-the-loop process of modifying 3D mesh models used to generate these images. Importantly, this framework allows both modifications that increase and decrease realism in synthetic data, which can both improve model performance. We illustrate this concept using a real-world example where data are sparse; detection of vehicles in infrared imagery. We fine-tune an initial YOLOv8 model on the ATR DSIAC infrared dataset and synthetic images generated from 3D mesh models in the Unity gaming engine, and then use XAI saliency maps to guide modification of our Unity models. We show that synthetic data can improve detection of vehicles in orientations unseen in training by 4.6% (to mAP50 = 94.6%). We further improve performance by an additional 1.5% (to 96.1%) through our new XAI-guided approach, which reduces misclassifications through both increasing and decreasing the realism of different parts of the synthetic data. Our proof-of-concept results pave the way for fine, XAI-controlled curation of synthetic datasets tailored to improve object detection performance, whilst simultaneously reducing the burden on human operators in designing and optimising these datasets.

Figures

Figures reproduced from arXiv: 2412.01477 by the authors.

Figure 1
Figure 1. Conceptual illustration of our proposed approach for improving the performance of object detection and classification algorithms [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the DSIAC ATR Dataset and Synthetic [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Experimental Setup. (left) Training data for only real data, showing vehicle orientations (blue segments) for all classes in training [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (21 more)
Figure 4
Figure 4. Figure 4: Confusion matrices calculated by taking an average over [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Illustration of the synthetic data modification process for reducing confusion. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Example plots of SHAP contributions of pixels to out [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Confusion matrices calculated by taking an average over [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Confusion matrices for the YOLOv8n model trained on only real data from the dataset (i.e no synthetic data). Panels (a)-(d) [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Confusion matrices for the YOLOv8n model trained on real data and the initial synthetic data (dataset v0). Panels (a)-(d) [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Confusion matrices for the YOLOv8n model trained on real data and synthetic data after the reinforcing modification (dataset [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Confusion matrices for the YOLOv8n model trained on real data and synthetic data after the disruptive modification (dataset [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Confusion matrices for the YOLOv8n model trained on real data and synthetic data after the reinforcing and disruptive modi [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Confusion matrices for the YOLOv8s model trained on real data and initial synthetic data (dataset v0). Panels (a)-(d) correspond [PITH_FULL_IMAGE:figures/full_fig_p017_14.png]
Figure 15
Figure 15. Figure 15: Confusion matrices for the YOLOv8s model trained on real data and synthetic data after reinforcing modification (dataset vR) [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]
Figure 16
Figure 16. Figure 16: Confusion matrices for the YOLOv8s model trained on real data and synthetic data after disruptive modification (dataset vD) [PITH_FULL_IMAGE:figures/full_fig_p018_16.png]
Figure 17
Figure 17. Figure 17: Confusion matrices for the YOLOv8s model trained on real data and synthetic data after reinforcing + disruptive modifications [PITH_FULL_IMAGE:figures/full_fig_p018_17.png]
Figure 18
Figure 18. Figure 18: Confusion matrices for the YOLOv8x model trained on real data and initial synthetic data (dataset v0). Panels (a)-(c) correspond [PITH_FULL_IMAGE:figures/full_fig_p019_18.png]
Figure 19
Figure 19. Figure 19: Confusion matrices for the YOLOv8x model trained on real data and synthetic data after reinforcing modification (dataset vR) [PITH_FULL_IMAGE:figures/full_fig_p020_19.png]
Figure 20
Figure 20. Figure 20: Confusion matrices for the YOLOv8x model trained on real data and synthetic data after disruptive modification (dataset vD) [PITH_FULL_IMAGE:figures/full_fig_p021_20.png]
Figure 21
Figure 21. Figure 21: Confusion matrices for the YOLOv8x model trained on real data and synthetic data after reinforcing + disruptive modifications [PITH_FULL_IMAGE:figures/full_fig_p021_21.png]
Figure 22
Figure 22. Figure 22: (a) Model performance with respect to different ratios [PITH_FULL_IMAGE:figures/full_fig_p022_22.png]
Figure 23
Figure 23. Figure 23: (a) Fraction of the misclassifications of the SUV as BTR70 for each [PITH_FULL_IMAGE:figures/full_fig_p023_23.png]
Figure 24
Figure 24. Figure 24: (a) Fraction of the misclassifications of the ZSU23 as the BTR70 for each [PITH_FULL_IMAGE:figures/full_fig_p023_24.png]
Figure 25
Figure 25. Figure 25: Scatter plot of PCA dimensional reduction to two di [PITH_FULL_IMAGE:figures/full_fig_p024_25.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 68 canonical work pages

  1. [1]

    Generating Synthetic Data in Finance: Opportunities, Chal- lenges and Pitfalls

    Samuel A Assefa, Danial Dervovic, Mahmoud Mahfouz, Robert E Tillman, Prashant Reddy, and Manuela Veloso. Generating Synthetic Data in Finance: Opportunities, Chal- lenges and Pitfalls. In Proceedings of the First ACM Inter- national Conference on AI in Finance, pages 1–8, 2020. 1

    work page 2020

  2. [2]

    Bridging the Domain Gap Between Synthetic and Real-World Data for Autonomous Driving

    Xiangyu Bai, Yedi Luo, Le Jiang, Aniket Gupta, Pushyami Kaveti, Hanumant Singh, and Sarah Ostadabbas. Bridging the Domain Gap Between Synthetic and Real-World Data for Autonomous Driving. Journal on Autonomous Trans- portation Systems, 1(2):1–15, 2024. 1

    work page 2024

  3. [3]

    Narrowing the Semantic Gap Between Real and Synthetic Data

    Radu Beche and Sergiu Nedevschi. Narrowing the Semantic Gap Between Real and Synthetic Data. In 2020 IEEE 16th International Conference on Intelligent Computer Commu- nication and Processing (ICCP), pages 361–367, 2020. 1

    work page 2020

  4. [4]

    Mechanistic Inter- pretability for AI Safety - A Review

    Leonard Bereska and Stratis Gavves. Mechanistic Inter- pretability for AI Safety - A Review. Transactions on Ma- chine Learning Research, 2024. Survey Certification, Expert Certification. 2, 8

    work page 2024

  5. [5]

    Unity Perception: Generate Synthetic DData for Computer Vision

    Steve Borkman, Adam Crespi, Saurav Dhakad, Sujoy Gan- guly, Jonathan Hogins, You-Cyuan Jhang, Mohsen Ka- malzadeh, Bowen Li, Steven Leal, Pete Parisi, et al. Unity Perception: Generate Synthetic DData for Computer Vision. arXiv preprint arXiv:2107.04259, 2021. 1

    arXiv 2021

  6. [6]

    Advanced Automatic Target Recognition (ATR) with Infrared (IR) Sensors

    Hai-Wen Chen, Neal Gross, Ravi Kapadia, Joseph Cheah, and Mo Gharbieh. Advanced Automatic Target Recognition (ATR) with Infrared (IR) Sensors. In 2021 IEEE Aerospace Conference (50100), pages 1–13, 2021. 5, 9

    work page 2021

  7. [7]

    Stargan: Unified Gen- erative Adversarial Networks for Multi-domain Image-to- image Translation

    Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Stargan: Unified Gen- erative Adversarial Networks for Multi-domain Image-to- image Translation. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 8789–8797,

    work page

  8. [8]

    Next- Generation Deep Learning Based on Simulators and Syn- thetic Data

    Celso M de Melo, Antonio Torralba, Leonidas Guibas, James DiCarlo, Rama Chellappa, and Jessica Hodgins. Next- Generation Deep Learning Based on Simulators and Syn- thetic Data. Trends in cognitive sciences , 26(2):174–187,

    work page

Show all 77 references
  1. [9]

    Exploring Synthetic Data for Ar- tificial Intelligence and Autonomous Systems: A Primer,

    H Deng and Harry Deng. Exploring Synthetic Data for Ar- tificial Intelligence and Autonomous Systems: A Primer,

  2. [10]

    https://dsiac.org/databases/atr-algorithm- development-image-database/, 2014

    DSIAC. https://dsiac.org/databases/atr-algorithm- development-image-database/, 2014. retrieved July

    2014

  3. [11]

    Brown, Nicholas Joseph, Sam McCandlish, Christopher Olah, Jared Kaplan, and Jack Clark

    Deep Ganguli, Liane Lovitt, John Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Benjamin Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bow- man, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zacha...

    2022 arXiv

  4. [12]

    Adaptive Testing of Computer Vision Models

    Irena Gao, Gabriel Ilharco, Scott Lundberg, and Marco Tulio Ribeiro. Adaptive Testing of Computer Vision Models . In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3980–3991, Los Alamitos, CA, USA,

    2023

  5. [13]

    Harnessing the Power of Synthetic Data in Healthcare: Innovation, Application, and Privacy

    Mauro Giuffr `e and Dennis L Shung. Harnessing the Power of Synthetic Data in Healthcare: Innovation, Application, and Privacy. NPJ digital medicine, 6(1):186, 2023. 1

    2023

  6. [14]

    HaDR: Applying Domain Randomization for Gen- erating Synthetic Multimodal Dataset for Hand Instance Segmentation in Cluttered Industrial Environments

    Stefan Grushko, Ale ˇs Vysock `y, Jakub Chlebek, and Petr Prokop. HaDR: Applying Domain Randomization for Gen- erating Synthetic Multimodal Dataset for Hand Instance Segmentation in Cluttered Industrial Environments. arXiv preprint arXiv:2304.05826, 2023. 1

    2023

  7. [15]

    Susskind

    Jiatao Gu, Ying Shen, Shuangfei Zhai, Yizhe Zhang, Navdeep Jaitly, and Joshua M. Susskind. Kaleido diffusion: Improving conditional diffusion models with autoregressive latent modeling. ArXiv, abs/2405.21048, 2024. 2

    2024 arXiv

  8. [16]

    Nicholas Hamilton, Adam Webb, Matt Wilder, Ben Hen- drickson, Matt Blanck, Erin Nelson, Wiley Roemer, and Timothy C. Havens. Enhancing Visualization and Explain- ability of Computer Vision Models with Local Interpretable Model-Agnostic Explanations (LIME). In 2022 IEEE Sym- po...

    2022

  9. [17]

    Synthetic Data: Development Status and Prospects for Military Applications

    Jinhui Huang, Junsong Yin, Shuangshuang Wang, and Dezhao Kong. Synthetic Data: Development Status and Prospects for Military Applications. In International Con- ference on Computational & Experimental Engineering and Sciences, pages 979–992. Springer, 2023. 1

    2023

  10. [18]

    Image-to-image Translation with Conditional Adver- sarial Networks

    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image Translation with Conditional Adver- sarial Networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134,

  11. [19]

    Sarthak Jain and Byron C. Wallace. Attention is not Expla- nation. In North American Chapter of the Association for Computational Linguistics, 2019. 4

    2019

  12. [20]

    A Style- based Generator Architecture for Generative Adversarial 9 Networks

    Tero Karras, Samuli Laine, and Timo Aila. A Style- based Generator Architecture for Generative Adversarial 9 Networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410,

  13. [21]

    On Convergence and Stability of GANs.arXiv preprint arXiv:1705.07215, 2017

    Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. On Convergence and Stability of GANs.arXiv preprint arXiv:1705.07215, 2017. 1

    2017 arXiv

  14. [22]

    Closing the Domain Gap: Blended Synthetic imagery for climate object detection

    Caleb Kornfein, Frank Willard, Caroline Tang, Yuxi Long, Saksham Jain, Jordan Malof, Simiao Ren, and Kyle Brad- bury. Closing the Domain Gap: Blended Synthetic imagery for climate object detection. Environmental Data Science, 2: e39, 2023. 1

    2023

  15. [23]

    Mode Collapse in Generative Adversarial Networks: An Overview

    Youssef Kossale, Mohammed Airaj, and Aziz Darouichi. Mode Collapse in Generative Adversarial Networks: An Overview. In 2022 8th International Conference on Opti- mization and Applications (ICOA), pages 1–6, 2022. 1

    2022

  16. [24]

    Applica- tions, challenges, and future directions of human-in-the-loop learning

    Sushant Kumar, Sumit Datta, Vishakha Singh, Deepanwita Datta, Sanjay Kumar Singh, and Ritesh Sharma. Applica- tions, challenges, and future directions of human-in-the-loop learning. IEEE Access, 12:75735–75760, 2024. 2

    2024

  17. [25]

    BSED: Baseline Shapley-Based Explainable Detector

    Michihiro Kuroki and Toshihiko Yamasaki. BSED: Baseline Shapley-Based Explainable Detector. IEEE Access, 2024. 2

    2024

  18. [26]

    Zhang, Federica Sarro, Ying Zhang, and Xuanzhe Liu

    Xinyue Li, Zhenpeng Chen, Jie M. Zhang, Federica Sarro, Ying Zhang, and Xuanzhe Liu. Bias Behind the Wheel: Fair- ness Testing of Autonomous Driving Systems. ACM Trans. Softw. Eng. Methodol., 2024. Just Accepted. 8

    2024

  19. [27]

    GLIGEN: Open-Set Grounded Text-to-Image Generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jian- wei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. GLIGEN: Open-Set Grounded Text-to-Image Generation . In 2023 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 22511–22521, Los Alami- to...

    2023

  20. [28]

    Lundberg and Su-In Lee

    Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, page 4768–4777, Red Hook, NY , USA, 2017. Cur- ran Associates Inc. 2, 4

    2017

  21. [29]

    A Comparison of Target Detection Algorithms Using DSIAC ATR Algo- rithm Development Data Set

    Abhijit Mahalanobis and Bruce McIntosh. A Comparison of Target Detection Algorithms Using DSIAC ATR Algo- rithm Development Data Set. In Automatic Target Recogni- tion XXIX, page 1098808, 2019. 5, 9

    2019

  22. [30]

    The best of Two Worlds: Reprojecting 2D Image Annotations onto 3D Models

    Marin Marcillat, Lo ¨ıc Van Audenhaege, Catherine Borre- mans, Aurelien Arnaubec, and Lenaick Menot. The best of Two Worlds: Reprojecting 2D Image Annotations onto 3D Models. PeerJ, 12:e17557, 2024. 7

    2024

  23. [31]

    The ETS2 Dataset, Synthetic Data from Video Games for Monocular Depth Estimation

    David Mar ´ıa-Arribas, Alfredo Cuesta-Infante, and Juan J Pantrigo. The ETS2 Dataset, Synthetic Data from Video Games for Monocular Depth Estimation. In Iberian Con- ference on Pattern Recognition and Image Analysis , pages 375–386. Springer, 2023. 2

    2023

  24. [32]

    Mitigating the Risk of Artificial Intelligence Bias in Cardiovascular Care

    Ariana Mihan, Ambarish Pandey, and Harriette GC Van Spall. Mitigating the Risk of Artificial Intelligence Bias in Cardiovascular Care. The Lancet Digital Health , 6(10): e749–e754, 2024. 8

    2024

  25. [33]

    Interpretable Machine Learning

    Christoph Molnar. Interpretable Machine Learning. Inde- pendently published, 2 edition, 2022. 4

    2022

  26. [34]

    Yair Movshovitz-Attias, Takeo Kanade, and Yaser Sheikh. How Useful is Photo-realistic Rendering for Visual Learn- ing? In Computer Vision–ECCV 2016 Workshops: Amster- dam, The Netherlands, October 8-10 and 15-16, 2016, Pro- ceedings, Part III 14, pages 202–217. Springer, 2016. 1

    2016

  27. [35]

    Analyzing DSIAC ATR Algorithm Development Database Utilizing Transfer Learning

    Kemal Arda ¨Ozertem. Analyzing DSIAC ATR Algorithm Development Database Utilizing Transfer Learning. In SPIE Future Sensing Technologies 2024, page 1308317, 2024. 5, 9

    2024

  28. [36]

    Rishubh Parihar, V . S. Sachidanand, Sabariswaran Mani, Tejan Karmali, and R. Venkatesh Babu. Precisecontrol: Enhancing text-to-image diffusion models with fine- grained attribute control. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 2...

    2024

  29. [37]

    On Domain Randomization for Object De- tection in Real Industrial Scenarios Using Synthetic Images

    Davide Pasanisi, Emanuele Rota, Michele Ermidoro, and Luca Fasanotti. On Domain Randomization for Object De- tection in Real Industrial Scenarios Using Synthetic Images. Procedia Computer Science, 217:816–825, 2023. 4th Inter- national Conference on Industry 4.0 and Smart Manu...

    2023

  30. [38]

    Priddy and Sastry Dhara

    K. Priddy and Sastry Dhara. Explorations in Transfer Learning and Machine Learning Architectures Utilizing the DSIAC ATR Algorithm Development Data Set. In Defense + Commercial Sensing, 2023. 5, 9

    2023

  31. [39]

    https://unity.com/features/probuilder, 2018

    Probuilder. https://unity.com/features/probuilder, 2018. 4, 5

    2018

  32. [40]

    Unrealcv: Connecting Com- puter Vision to Unreal Engine

    Weichao Qiu and Alan Yuille. Unrealcv: Connecting Com- puter Vision to Unreal Engine. In Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8- 10 and 15-16, 2016, Proceedings, Part III 14 , pages 909–

    2016

  33. [41]

    Unrealcv: Vir- tual Worlds for Computer Vision

    Weichao Qiu, Fangwei Zhong, Yi Zhang, Siyuan Qiao, Zi- hao Xiao, Tae Soo Kim, and Yizhou Wang. Unrealcv: Vir- tual Worlds for Computer Vision. InProceedings of the 25th ACM international conference on Multimedia , pages 1221– 1224, 2017. 1

    2017

  34. [42]

    Zero-shot Text-to-image Generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea V oss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot Text-to-image Generation. In International con- ference on machine learning, pages 8821–8831. Pmlr, 2021. 1

    2021

  35. [43]

    Better Regulariza, A. F. M. Shahab Uddin, Sirazam Monira, Wheemyung Shin, TaeChoong Chung, and Sung-Ho Bae. SaliencyMix: A Saliency Guided Data Augmentation Strat- egy for Better Regularization. ArXiv, abs/2006.01791, 2020. 2

    2006 arXiv

  36. [44]

    Adaptive testing and debugging of NLP models

    Marco Tulio Ribeiro and Scott Lundberg. Adaptive testing and debugging of NLP models. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 3253–3267, Dublin, Ireland, 2022. Association for Computational Lin...

    2022

  37. [45]

    Beyond accuracy: Behavioral testing of nlp models with checklist

    Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of nlp models with checklist. In Association for Computational Linguistics (ACL), 2020. 2

    2020

  38. [46]

    High-resolution Image Synthesis with Latent Diffusion Models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution Image Synthesis with Latent Diffusion Models. In Proceedings of 10 the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1

    2022

  39. [47]

    Synthetic Data Generation with Unity 3D and Unreal Engine for Con- struction Hazard Scenarios: A Comparative Analysis

    Aqsa Sabir, Rahat Hussain, Akeem Pedro, Mehrtash Soltani, Dongmin Lee, Chansik Park, and Jae-Ho Pyeon. Synthetic Data Generation with Unity 3D and Unreal Engine for Con- struction Hazard Scenarios: A Comparative Analysis. In- ternational Conference on Construction Engineering ...

    2024

  40. [48]

    Seman- tic Foggy Scene Understanding with Synthetic Data

    Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Seman- tic Foggy Scene Understanding with Synthetic Data. Inter- national Journal of Computer Vision, 126:973–992, 2018. 1

    2018

  41. [49]

    Nasrabadi, and Raghuveer M

    Shoaib Meraj Sami, Nasser M. Nasrabadi, and Raghuveer M. Rao. Deep Transductive Transfer Learning for Automatic Target Recognition. In Defense + Commercial Sensing ,

  42. [50]

    A Value for N-Person Games

    Lloyd S Shapley. A Value for N-Person Games. Contribu- tion to the Theory of Games, 2, 1953. 4

    1953

  43. [51]

    Automatic Virtual 3D City Generation for Synthetic Data Collection

    Bingyu Shen, Boyang Li, and Walter J Scheirer. Automatic Virtual 3D City Generation for Synthetic Data Collection. In Proceedings of the IEEE/CVF winter conference on applica- tions of computer vision, pages 161–170, 2021. 2

    2021

  44. [52]

    Rareplanes: Synthetic Data Takes Flight

    Jacob Shermeyer, Thomas Hossler, Adam Van Etten, Daniel Hogan, Ryan Lewis, and Daeil Kim. Rareplanes: Synthetic Data Takes Flight. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 207– 217, 2021. 1

    2021

  45. [53]

    Girshick

    Abhinav Shrivastava, Abhinav Kumar Gupta, and Ross B. Girshick. Training region-based object detectors with on- line hard example mining. 2016 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR) , pages 761– 769, 2016. 2, 7

    2016

  46. [54]

    The Curse of Re- cursion: Training on Generated Data Makes Models Forget

    Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The Curse of Re- cursion: Training on Generated Data Makes Models Forget. arXiv preprint arXiv:2305.17493, 2023. 1

    2023 arXiv

  47. [55]

    The Rise of Synthetic Data: Enhancing AI and Machine Learning Model Training to Address Data Scarcity and Mitigate Privacy Risks

    Jaswinder Singh. The Rise of Synthetic Data: Enhancing AI and Machine Learning Model Training to Address Data Scarcity and Mitigate Privacy Risks. Journal of Artificial Intelligence Research and Applications, 1(2):292–332, 2021. 1

    2021

  48. [56]

    Daniel Freeman, Theodore R

    Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lind- sey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunning- ham, Nicholas L Turner, Callum McDougall, Monte Mac- Diarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, J...

    2024

  49. [57]

    Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization

    Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Camer- acci, Shaad Boochoon, and Stan Birchfield. Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization. In Proceedings of the IEEE confe...

    2018

  50. [58]

    https://unity.com/

    Unity Technologies. https://unity.com/. Game development platform. 2

  51. [59]

    Yolov8: A novel object detection algorithm with enhanced performance and robust- ness

    Rejin Varghese and Sambath M. Yolov8: A novel object detection algorithm with enhanced performance and robust- ness. In 2024 International Conference on Advances in Data Engineering and Intelligent Computing Systems (ADICS) , pages 1–6, 2024. 5

    2024

  52. [60]

    Learning from Synthetic Humans

    Gul Varol, Javier Romero, Xavier Martin, Naureen Mah- mood, Michael J Black, Ivan Laptev, and Cordelia Schmid. Learning from Synthetic Humans. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 109–117, 2017. 1

    2017

  53. [61]

    Wang, E.P

    Z. Wang, E.P. Simoncelli, and A.C. Bovik. Multiscale Structural Similarity for Image Quality Assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, pages 1398–1402 V ol.2, 2003. 4

    2003

  54. [62]

    Bovik, H.R

    Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image Quality Assessment: From Error Visibility to Struc- tural Similarity. IEEE Transactions on Image Processing, 13 (4):600–612, 2004. 4

    2004

  55. [63]

    Diffusion-gan: Training GANs with Diffusion

    Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion-gan: Training GANs with Diffusion. arXiv preprint arXiv:2206.02262, 2022. 1

    2022 arXiv

  56. [64]

    Dis- covering bugs in vision models using off-the-shelf image generation and captioning

    Olivia Wiles, Isabela Albuquerque, and Sven Gowal. Dis- covering bugs in vision models using off-the-shelf image generation and captioning. In NeurIPS ML Safety Workshop,

  57. [65]

    A survey of human-in-the-loop for machine learning

    Xingjiao Wu, Luwei Xiao, Yixuan Sun, Junhang Zhang, Tianlong Ma, and Liang He. A survey of human-in-the-loop for machine learning. Future Generation Computer Systems, 135:364–381, 2022. 1

    2022

  58. [66]

    VarifocalNet: An IoU-aware Dense Object Detec- tor

    Haoyang Zhang, Ying Wang, Feras Dayoub, and Niko Sun- derhauf. VarifocalNet: An IoU-aware Dense Object Detec- tor . In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8510–8519, Los Alami- tos, CA, USA, 2021. IEEE Computer Society. 2, 7

    2021

  59. [67]

    Efros, Eli Shecht- man, and Oliver Wang

    Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shecht- man, and Oliver Wang. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 586–595, 2018. 4

    2018

  60. [68]

    Unpaired Image-to-image Translation Using Cycle- Consistent Adversarial Networks

    Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired Image-to-image Translation Using Cycle- Consistent Adversarial Networks. In Proceedings of the IEEE international conference on computer vision , pages 2223–2232, 2017. 1 11 Improving Object Detection by Mo...

    2017

  61. [70]

    Dataset The complete ATR DSIAC dataset [10], captured in 2006, contains 9 vehicle targets plus human targets, ground truth data including vehicle labels and bounding box coordi- nates, and other documentation including photographs of the targets and meteorological data to assi...

    2006

  62. [71]

    Example plots of SHAP contributions of pixels to out- puts of models trained on (a) real data only and (b) real + synthetic v0 (initial) data, for a single test image of a ZSU23

    Background SHAP Figure 7. Example plots of SHAP contributions of pixels to out- puts of models trained on (a) real data only and (b) real + synthetic v0 (initial) data, for a single test image of a ZSU23. In each image, the bounding box is shown in purple, and the average cont...

  63. [72]

    Confusion Matrices In Section 5 in the main paper, we present results of the ATR dataset trained on different synthetic data modifica- tions. The full confusion matrices of the initial confusion when the model is trained on real data only and the real data and synthetic data c...

  64. [73]

    Model sensitivity to real:synthetic data ra- tio and dataset size After adding the initial synthetic data model v0 to the train- ing dataset, we seek to establish the affect on model perfor- mance of two main factors:

  65. [74]

    The ratio of real and synthetic data for fixed dataset size

  66. [75]

    First, we consider the effect of the ratio of real and syn- thetic data for fixed dataset size

    The total volume of images (real and synthetic com- bined) in the training dataset on model performance. First, we consider the effect of the ratio of real and syn- thetic data for fixed dataset size. This tells us the impact of indirect replacement of real images by synthetic...

  67. [76]

    In this dataset we have a nat- ural dimension of variation (orientation of the vehicle), and so we do not need to cluster the samples to explore the di- mension of variation

    Angular breakdown of target misclassifica- tion and SHAP plots At the end of Section 2 in the main paper, we note that the comparison of saliency maps to identify unique and com- mon features between two classes must also account for variation across each class. In this datase...

  68. [77]

    Principal Component Analysis of Train- Test Split On inspection of the dataset we noticed that there is con- siderable similarity between successive video frames. To highlight the potential issue of data leakage between train and test sets we use principal component analysis (...

  69. [2023]

    IEEE Computer Society. 2

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.