Preserve the Hard, Regenerate the Rest: Uncertainty-Guided Synthetic Training Data Augmentation with Diffusion Models

Ahmed H. A. Ibrahim; Julian Glei{\ss}ner; Nikolai R\"ohrich; Silvan Mertes; Tobias Huber

arxiv: 2606.31603 · v1 · pith:PZAUPWHH · submitted 2026-06-30 · cs.CV · cs.AI· cs.LG

Preserve the Hard, Regenerate the Rest: Uncertainty-Guided Synthetic Training Data Augmentation with Diffusion Models

Nikolai R\"ohrich , Julian Glei{\ss}ner , Ahmed H. A. Ibrahim , Silvan Mertes , Tobias Huber This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-01 05:30 UTCgrok-4.3pith:PZAUPWHH record.json open to challenge →

classification cs.CV cs.AIcs.LG

keywords semantic segmentationdata augmentationdiffusion modelsuncertainty estimationinpaintingpredictive entropyCityscapesrare class performance

0 comments

The pith

Uncertainty-guided inpainting regenerates only the easy context around hard semantic regions while preserving original labels and excluding generated pixels from the loss.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that semantic segmentation models can be improved by identifying uncertain regions via predictive entropy and using a diffusion model to inpaint only the surrounding visual context. Training then proceeds with loss computed solely on the original preserved pixels, so the model learns the hard regions in novel surroundings without label misalignment risks. This targets data efficiency on rare or visually diverse classes in datasets like Cityscapes and aerial imagery. A sympathetic reader would care because it avoids wasting capacity on easy pixels and sidesteps the need for external guardrail models.

Core claim

Using a baseline segmenter's predictive entropy to locate uncertain semantic regions, the method inpaints only the complementary visual context with a diffusion model. Fine-tuning computes loss exclusively over the unmodified original pixels, focusing learning on the preserved uncertain areas presented in new contexts. This yields mIoU gains on Cityscapes, UAVID, and BDD100K, with largest improvements on rare classes such as buses, trains, and cars from aerial views.

What carries the argument

Uncertainty-guided selective context inpainting, where predictive entropy masks identify regions to preserve and diffusion models fill only the rest, with loss masked to original pixels only.

If this is right

Substantial mIoU gains appear on complex datasets, concentrated on rare and difficult classes.
The approach works without external guardrail models or coarse heuristics that augment entire backgrounds.
Pixel informativeness per synthetic sample increases because only uncertain regions drive the loss.
Label validity is strictly preserved by never altering original pixels or their annotations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Similar selective regeneration could be tested on instance segmentation or depth estimation where uncertainty maps are available.
The method might extend to video by propagating uncertainty across frames and inpainting temporal context.
If diffusion models improve further, the same preserve-hard principle could reduce the volume of real labeled data needed for deployment.

Load-bearing premise

Excluding inpainted regions from the loss is enough to stop any generated context from subtly changing how difficult or easy the preserved uncertain pixels appear.

What would settle it

Running the fine-tuning procedure on Cityscapes and measuring whether mIoU on rare classes stays flat or drops compared with standard random cropping or full-image diffusion augmentation.

Figures

Figures reproduced from arXiv: 2606.31603 by Ahmed H. A. Ibrahim, Julian Glei{\ss}ner, Nikolai R\"ohrich, Silvan Mertes, Tobias Huber.

**Figure 1.** Figure 1: Uncertainty-Guided Context Synthesis. We augment real samples by computing the current segmenter’s per-pixel predictive entropy and inpaint a fresh visual context around hard-to-classify regions. Our method transcends prior region selection heuristics by allocating synthetic budget on informative regions rather than always choosing background or foreground regions. Abstract Semantic segmentation models str… view at source ↗

**Figure 2.** Figure 2: Uncertainty-based region selection transcends the foreground-background contradiction. We argue that the augmented regions should be based on uncertainty, rather than always selecting either the entire foreground [15] or background [18]. tator) supplies the answer to each round’s query, with gains accumulating until uncertainty saturates. We evaluate our method on semantic segmentation for three benchmark… view at source ↗

**Figure 3.** Figure 3: Pixel-exact paste-back preserves label validity. 1024 × 2048), UAVID [22] (aerial urban scenes, 8 classes, ∼200 training / 70 validation images at 4096 × 2160), and BDD100K [37] (urban driving in diverse weather and times of day, 19 classes, 7000 training / 1000 validation images at 720 × 1280). The three datasets span different viewpoints (street-level vs. aerial), different geographic distributions, and … view at source ↗

**Figure 4.** Figure 4: Uncurated qualitative results across both domains. For each example, the columns show the original image, the baseline segmenter’s per-pixel predictive-entropy map, the preserve mask formed from the most uncertain classes, and the final augmented sample after inpainting and pixel-exact paste-back. Generation uses dataset-level prompts: “photorealistic, ultra-detailed, 4K high-resolution, sharp focus, high … view at source ↗

**Figure 5.** Figure 5: Additional uncurated qualitative samples on [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗

**Figure 6.** Figure 6: Additional uncurated qualitative samples on [PITH_FULL_IMAGE:figures/full_fig_p012_6.png] view at source ↗

read the original abstract

Semantic segmentation models struggle with data sparsity and rare or visually diverse regions, e.g., dense regions or small objects in aerial or autonomous mobility data. While synthetic augmentation is an appealing solution, directly generating new labeled data risks misalignment of labels and generated pixels. Existing solutions to this problem often rely on external models, or employ coarse heuristics such as indiscriminately augmenting all foreground objects or entire backgrounds, which wastes capacity on uninformative pixels. To address this, we propose an uncertainty-guided synthetic context augmentation strategy that strictly preserves label validity and efficiently maximizes pixel informativeness per synthetic sample - no external guardrails required. Using a baseline segmenter's predictive entropy, we identify uncertain semantic regions and inpaint only the complementary visual context. When fine-tuning the segmenter on this synthetic data, we compute the loss only over the original pixels, excluding inpainted regions. This focuses learning on the unmodified, uncertain regions while presenting them in novel contexts. We demonstrate substantial mIoU gains on Cityscapes, UAVID, and BDD100K with the largest gains on rare and difficult classes such as buses, trains, or (from the aerial perspective) cars. Our results demonstrate that uncertainty-guided context augmentation is a highly effective lever to improve segmentation performance on complex datasets, with code provided at https://github.com/XITASO/Preserve-the-Hard-Regenerate-the-Rest.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper's core move—using entropy to keep only uncertain regions, inpaint new context around them with diffusion, and mask the loss to the original pixels—targets tail classes in segmentation but the experiments do not isolate whether gains come from better generalization or from the new contexts simply changing task difficulty.

read the letter

The new piece here is the specific pipeline: run a baseline segmenter, pick high-entropy regions, inpaint only the complementary areas, and train the loss exclusively on the preserved pixels. This avoids label misalignment on generated content while trying to give rare classes more varied surroundings. It is a clean, low-overhead recipe that sits on top of existing diffusion inpainting and uncertainty work rather than replacing it.

The paper does a few things right. It evaluates on three public datasets (Cityscapes, UAVID, BDD100K) and claims the largest lifts on the classes that usually lag, such as buses, trains, and small cars from above. Releasing the code lowers the barrier for anyone who wants to test the idea on their own data.

The main weakness is the missing isolation of the mechanism. The claim that masking the loss to original pixels fully protects against context-induced bias rests on the assumption that the diffusion outputs do not systematically alter how hard or easy those pixels are to classify. No ablation is described that holds the preserved pixels fixed while swapping in different contexts or that checks whether per-pixel error rates on those pixels move independently of the supervision. Without that check, the reported mIoU improvements could partly reflect easier or harder instances rather than genuine robustness gains. The abstract gives no concrete numbers or error bars, so the size of the effect is still unclear.

This is aimed at practitioners who already run segmentation models on driving or aerial imagery and are looking for a targeted augmentation step to lift tail performance. It is not a foundational advance, but the method is concrete enough that a referee could usefully check the experiments and ablations. I would send it to review rather than desk-reject; the idea is practical and the code is available, even if the causal story needs tightening.

Referee Report

2 major / 2 minor

Summary. The paper proposes an uncertainty-guided synthetic context augmentation method for semantic segmentation. A baseline segmenter’s predictive entropy identifies uncertain regions; a diffusion model inpaints only the complementary visual context. During fine-tuning the loss is computed exclusively on the preserved original pixels, focusing learning on uncertain regions presented in novel contexts. The authors report substantial mIoU gains on Cityscapes, UAVID and BDD100K, with the largest improvements on rare classes, and release code.

Significance. If the claimed gains are reproducible and attributable to improved generalization rather than altered task difficulty, the approach would provide a practical, label-preserving augmentation strategy that avoids external guardrails and coarse heuristics while targeting the most informative pixels. Releasing code strengthens reproducibility.

major comments (2)

[Method / Experiments] The central claim that restricting the loss to original pixels isolates learning from context-induced bias is load-bearing for attributing gains to the method. No ablation is described that holds the preserved pixels fixed while varying context realism, nor any measurement of whether per-pixel entropy or error rates on those pixels shift independently of the label supervision (see the method description and experimental claims in the abstract).
[Abstract] The abstract asserts “substantial mIoU gains … with the largest gains on rare … classes” yet supplies no numerical values, baseline comparisons, per-class breakdowns, or error analysis. Without these details the magnitude and reliability of the reported improvements cannot be assessed.

minor comments (2)

[Method] Notation for predictive entropy and the precise masking of the loss could be stated more formally (e.g., with an equation) to aid reproducibility.
The title is evocative but does not convey the core technical mechanism (entropy-guided inpainting with masked loss).

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our uncertainty-guided augmentation approach. We address each major comment below and will revise the manuscript accordingly to strengthen the attribution of gains and improve clarity in the abstract.

read point-by-point responses

Referee: [Method / Experiments] The central claim that restricting the loss to original pixels isolates learning from context-induced bias is load-bearing for attributing gains to the method. No ablation is described that holds the preserved pixels fixed while varying context realism, nor any measurement of whether per-pixel entropy or error rates on those pixels shift independently of the label supervision (see the method description and experimental claims in the abstract).

Authors: We agree this ablation would strengthen the central claim. The loss restriction is designed to ensure supervision occurs only on original pixels (with inpainted regions excluded from the loss), thereby presenting uncertain regions in novel contexts without label misalignment. To directly address the concern, we will add an ablation in the revision that holds preserved pixels fixed, compares loss restriction vs. full-pixel loss on the same synthetic samples, and reports shifts in per-pixel entropy and error rates on the original pixels. This will quantify the isolation effect independently of label supervision. revision: yes
Referee: [Abstract] The abstract asserts “substantial mIoU gains … with the largest gains on rare … classes” yet supplies no numerical values, baseline comparisons, per-class breakdowns, or error analysis. Without these details the magnitude and reliability of the reported improvements cannot be assessed.

Authors: We agree that including quantitative details in the abstract will allow readers to assess the improvements immediately. The body of the paper contains the supporting tables (e.g., overall mIoU deltas on Cityscapes/UAVID/BDD100K and per-class breakdowns showing largest gains on rare classes such as bus/train/car). We will revise the abstract to incorporate specific numerical values, baseline comparisons, and highlights for rare-class gains while remaining within length limits. revision: yes

Circularity Check

0 steps flagged

No circularity: purely empirical method with no derivation chain

full rationale

The paper describes an empirical augmentation pipeline: compute predictive entropy on a baseline segmenter, inpaint complementary context via diffusion, and mask the fine-tuning loss to the original (uncertain) pixels only. No equations, first-principles derivations, fitted parameters renamed as predictions, or uniqueness theorems appear in the provided text. Evaluation occurs on external public benchmarks (Cityscapes, UAVID, BDD100K) with no self-citation load-bearing the central claim. The method is self-contained against those benchmarks; any performance gains are attributed to the described procedure rather than reducing to its own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The approach depends on standard properties of diffusion models and entropy-based uncertainty; no free parameters or new entities are introduced in the abstract description.

axioms (1)

domain assumption Diffusion models can synthesize visual contexts that remain compatible with preserved semantic labels when only complementary regions are inpainted.
The method assumes inpainting does not create label inconsistencies or unintended difficulty shifts for the kept pixels.

pith-pipeline@v0.9.1-grok · 5804 in / 1285 out tokens · 40338 ms · 2026-07-01T05:30:54.545672+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

43 extracted references · 43 canonical work pages

[1]

Blended latent diffusion.ACM transactions on graphics (TOG), 42 (4):1–11, 2023

Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion.ACM transactions on graphics (TOG), 42 (4):1–11, 2023. 5

work page 2023
[2]

Flux.1 fill [dev]

Black Forest Labs. Flux.1 fill [dev]. Hugging Face model card, 2024. Accessed 2026-06-08. 7

work page 2024
[3]

The cityscapes dataset for semantic urban scene understanding

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. InProceed- ings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016. 1, 2, 3, 5, 13

work page 2016
[4]

Class-balanced loss based on effective number of samples

Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9268–9277,

work page
[5]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conference on Learning Representa- tions, 2021. 5

work page 2021
[6]

Active learning by labeling features

Gregory Druck, Burr Settles, and Andrew McCallum. Active learning by labeling features. InProceedings of the 2009 conference on Empirical methods in natural language pro- cessing, pages 81–90, 2009. 2, 3

work page 2009
[7]

Data augmentation for object detec- tion via controllable diffusion models

Haoyang Fang, Boran Han, Shuai Zhang, Su Zhou, Cuixiong Hu, and Wen-Ming Ye. Data augmentation for object detec- tion via controllable diffusion models. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1257–1266, 2024. 3

work page 2024
[8]

Di Feng, Christian Haase-Sch ¨utz, Lars Rosenbaum, Heinz Hertlein, Claudius Glaeser, Fabian Timm, Werner Wies- beck, and Klaus Dietmayer. Deep multi-modal object de- tection and semantic segmentation for autonomous driving: Datasets, methods, and challenges.IEEE Transactions on Intelligent Transportation Systems, 22(3):1341–1360, 2020. 1

work page 2020
[9]

Dropout as a bayesian approximation: Representing model uncertainty in deep learning

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. InProceedings of the 33rd International Confer- ence on Machine Learning, pages 1050–1059. PMLR, 2016. 2

work page 2016
[10]

Deep bayesian active learning with image data

Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. InInternational conference on machine learning, pages 1183–1192. PMLR,

work page
[11]

Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020. 1

work page 2020
[12]

Diffusemix: Label- preserving data augmentation with diffusion models

Khawar Islam, Muhammad Zaigham Zaheer, Arif Mah- mood, and Karthik Nandakumar. Diffusemix: Label- preserving data augmentation with diffusion models. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27621–27630, 2024. 2

work page 2024
[13]

Computer vision for autonomous vehicles: Prob- lems, datasets and state of the art.Foundations and Trends in Computer Graphics and Vision, 12(1-3):1–308, 2020

Joel Janai, Fatma G ¨uney, Aseem Behl, and Andreas Geiger. Computer vision for autonomous vehicles: Prob- lems, datasets and state of the art.Foundations and Trends in Computer Graphics and Vision, 12(1-3):1–308, 2020. 1

work page 2020
[14]

Active learning inspired controlnet guidance for aug- menting semantic segmentation datasets.arXiv preprint arXiv:2503.09221, 2025

Hannah Kniesel, Pedro Hermosilla, and Timo Ropin- ski. Active learning inspired controlnet guidance for aug- menting semantic segmentation datasets.arXiv preprint arXiv:2503.09221, 2025. 3

work page arXiv 2025
[15]

Dataset enhancement with instance-level augmentations

Orest Kupyn and Christian Rupprecht. Dataset enhancement with instance-level augmentations. InEuropean Conference on Computer Vision, pages 384–402. Springer, 2024. 2, 3, 4, 5, 6, 7, 13

work page 2024
[16]

Tracer: Extreme attention guided salient object tracing network (stu- dent abstract)

Min Seok Lee, WooSeok Shin, and Sung Won Han. Tracer: Extreme attention guided salient object tracing network (stu- dent abstract). InProceedings of the AAAI conference on artificial intelligence, pages 12993–12994, 2022. 3

work page 2022
[17]

Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InInterna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 2, 3

work page 2022
[18]

A simple background augmentation method for object detection with diffusion model

Yuhang Li, Xin Dong, Chen Chen, Weiming Zhuang, and Lingjuan Lyu. A simple background augmentation method for object detection with diffusion model. InEuropean Con- ference on Computer Vision, pages 462–479. Springer, 2024. 2, 3, 4, 5, 6, 13

work page 2024
[19]

Focal loss for dense object detection

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll´ar. Focal loss for dense object detection. InPro- ceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017. 1

work page 2017
[20]

Decoupled weight decay regularization

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In7th International Conference on Learning Representations, ICLR, 2019. 5

work page 2019
[21]

Repaint: Inpainting using denoising diffusion probabilistic models

Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. InProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461–11471, 2022. 2, 5

work page 2022
[22]

Uavid: A semantic segmentation dataset for uav imagery.ISPRS journal of photogrammetry and remote sensing, 165:108–119, 2020

Ye Lyu, George V osselman, Gui-Song Xia, Alper Yilmaz, and Michael Ying Yang. Uavid: A semantic segmentation dataset for uav imagery.ISPRS journal of photogrammetry and remote sensing, 165:108–119, 2020. 2, 5

work page 2020
[23]

CEREALS - cost- effective region-based active learning for semantic segmen- 9 tation

Radek Mackowiak, Philip Lenz, Omair Ghori, Ferran Diego, Oliver Lange, and Carsten Rother. CEREALS - cost- effective region-based active learning for semantic segmen- 9 tation. InBritish Machine Vision Conference 2018, BMVC, page 121. BMV A Press, 2018. 2

work page 2018
[24]

Dataset diffusion: Diffusion-based synthetic data generation for pixel-level semantic segmentation.Advances in Neural Information Processing Systems, 36:76872–76892, 2023

Quang Nguyen, Truong Vu, Anh Tran, and Khoi Nguyen. Dataset diffusion: Diffusion-based synthetic data generation for pixel-level semantic segmentation.Advances in Neural Information Processing Systems, 36:76872–76892, 2023. 1, 3

work page 2023
[25]

Uncertainty-aware controlnet: Bridging domain gaps with synthetic image generation

Joshua Niemeijer, Jan Ehrhardt, Heinz Handels, and Hristina Uzunova. Uncertainty-aware controlnet: Bridging domain gaps with synthetic image generation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 4184–4193, 2025. 3

work page 2025
[26]

Maxime Oquab, Timoth ´ee Darcet, Th´eo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Je- gou, Julien Mairal, Patr...

work page 2024
[27]

Scalable diffusion models with transformers

William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF inter- national conference on computer vision, pages 4195–4205,

work page
[28]

Sdxl: Improving latent diffusion models for high-resolution image synthesis

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. InInternational Confer- ence on Learning Representations, pages 1862–1874, 2024. 2, 5

work page 2024
[29]

High-resolution image synthesis with latent diffusion models

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 3, 5

work page 2022
[30]

Training region-based object detectors with online hard ex- ample mining

Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard ex- ample mining. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 761–769,

work page
[31]

Effective data augmentation with diffusion models

Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models. InInternational Conference on Learning Represen- tations, pages 14590–14612, 2024. 3

work page 2024
[32]

Datasetdm: Synthesizing data with perception annota- tions using diffusion models.Advances in Neural Informa- tion Processing Systems, 36:54683–54695, 2023

Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annota- tions using diffusion models.Advances in Neural Informa- tion Processing Systems, 36:54683–54695, 2023. 1, 3

work page 2023
[33]

Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using dif- fusion models

Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using dif- fusion models. InProceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 1206–1217,

work page
[34]

Segformer: Simple and efficient design for semantic segmentation with transform- ers.Advances in neural information processing systems, 34: 12077–12090, 2021

Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transform- ers.Advances in neural information processing systems, 34: 12077–12090, 2021. 7

work page 2021
[35]

Freemask: Synthetic images with dense annotations make stronger segmentation models.Advances in Neural Information Processing Systems, 36:18659–18675,

Lihe Yang, Xiaogang Xu, Bingyi Kang, Yinghuan Shi, and Hengshuang Zhao. Freemask: Synthetic images with dense annotations make stronger segmentation models.Advances in Neural Information Processing Systems, 36:18659–18675,

work page
[36]

Learning loss for ac- tive learning

Donggeun Yoo and In So Kweon. Learning loss for ac- tive learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 93–102,

work page
[37]

Bdd100k: A diverse driving dataset for heterogeneous multitask learning

Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Dar- rell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 2636–2645, 2020. 2, 5

work page 2020
[38]

Adding conditional control to text-to-image diffusion models

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3836–3847, 2023. 2, 3, 4

work page 2023
[39]

X-paste: Revisiting scalable copy-paste for instance segmentation us- ing CLIP and StableDiffusion

Hanqing Zhao, Dianmo Sheng, Jianmin Bao, Dongdong Chen, Dong Chen, Fang Wen, Lu Yuan, Ce Liu, Wenbo Zhou, Qi Chu, Weiming Zhang, and Nenghai Yu. X-paste: Revisiting scalable copy-paste for instance segmentation us- ing CLIP and StableDiffusion. InProceedings of the 40th In- ternational Conference on Machine Learning, pages 42098– 42109. PMLR, 2023. 2

work page 2023
[40]

Recon: Region-controllable data augmentation with rectification and alignment for object detection.Advances in Neural Information Processing Systems, 38:74897–74926,

Haowei Zhu, Tianxiang Pan, Rui Qin, Jun-Hai Yong, and Bin Wang. Recon: Region-controllable data augmentation with rectification and alignment for object detection.Advances in Neural Information Processing Systems, 38:74897–74926,

work page
[41]

The columns show the original image, the predictive-entropy map, the preserved uncertain region, and the final augmented image after inpainting and paste-back

Additional Uncurated Qualitative Samples We show more examples that are sampled without any qual- itative filtering in Figure 5 and Figure 6. The columns show the original image, the predictive-entropy map, the preserved uncertain region, and the final augmented image after inpainting and paste-back. Every preserved pixel in the augmented image is bit-wis...

work page
[42]

photorealistic, ultra-detailed, 4K high- resolution, sharp focus, high quality, in the style of the Cityscapes dataset

Additional Implementation Details This section specifies everything needed to reproduce the method and results in the main paper: the fixed data sub- sets, preprocessing, the training and fine-tuning schedules, generation settings, and the checkpoint-selection rule. 7.1. Data Splits and Reproducibility The training data splits used in our experiments are ...

work page
[43]

To apply this to densely labeled semantic segmentation use-cases like Cityscapes [3], we assign each semantic class to either fore- ground or background

Foreground/Background Split For Baseline Methods The two synthetic augmentation baselines used in our ex- periment rely on spatial heuristics based on foreground ob- jects and the background of each image. To apply this to densely labeled semantic segmentation use-cases like Cityscapes [3], we assign each semantic class to either fore- ground or backgroun...

work page

[1] [1]

Blended latent diffusion.ACM transactions on graphics (TOG), 42 (4):1–11, 2023

Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion.ACM transactions on graphics (TOG), 42 (4):1–11, 2023. 5

work page 2023

[2] [2]

Flux.1 fill [dev]

Black Forest Labs. Flux.1 fill [dev]. Hugging Face model card, 2024. Accessed 2026-06-08. 7

work page 2024

[3] [3]

The cityscapes dataset for semantic urban scene understanding

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. InProceed- ings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016. 1, 2, 3, 5, 13

work page 2016

[4] [4]

Class-balanced loss based on effective number of samples

Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9268–9277,

work page

[5] [5]

An image is worth 16x16 words: Transformers for image recognition at scale

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conference on Learning Representa- tions, 2021. 5

work page 2021

[6] [6]

Active learning by labeling features

Gregory Druck, Burr Settles, and Andrew McCallum. Active learning by labeling features. InProceedings of the 2009 conference on Empirical methods in natural language pro- cessing, pages 81–90, 2009. 2, 3

work page 2009

[7] [7]

Data augmentation for object detec- tion via controllable diffusion models

Haoyang Fang, Boran Han, Shuai Zhang, Su Zhou, Cuixiong Hu, and Wen-Ming Ye. Data augmentation for object detec- tion via controllable diffusion models. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1257–1266, 2024. 3

work page 2024

[8] [8]

Di Feng, Christian Haase-Sch ¨utz, Lars Rosenbaum, Heinz Hertlein, Claudius Glaeser, Fabian Timm, Werner Wies- beck, and Klaus Dietmayer. Deep multi-modal object de- tection and semantic segmentation for autonomous driving: Datasets, methods, and challenges.IEEE Transactions on Intelligent Transportation Systems, 22(3):1341–1360, 2020. 1

work page 2020

[9] [9]

Dropout as a bayesian approximation: Representing model uncertainty in deep learning

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. InProceedings of the 33rd International Confer- ence on Machine Learning, pages 1050–1059. PMLR, 2016. 2

work page 2016

[10] [10]

Deep bayesian active learning with image data

Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. InInternational conference on machine learning, pages 1183–1192. PMLR,

work page

[11] [11]

Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dif- fusion probabilistic models.Advances in neural information processing systems, 33:6840–6851, 2020. 1

work page 2020

[12] [12]

Diffusemix: Label- preserving data augmentation with diffusion models

Khawar Islam, Muhammad Zaigham Zaheer, Arif Mah- mood, and Karthik Nandakumar. Diffusemix: Label- preserving data augmentation with diffusion models. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27621–27630, 2024. 2

work page 2024

[13] [13]

Computer vision for autonomous vehicles: Prob- lems, datasets and state of the art.Foundations and Trends in Computer Graphics and Vision, 12(1-3):1–308, 2020

Joel Janai, Fatma G ¨uney, Aseem Behl, and Andreas Geiger. Computer vision for autonomous vehicles: Prob- lems, datasets and state of the art.Foundations and Trends in Computer Graphics and Vision, 12(1-3):1–308, 2020. 1

work page 2020

[14] [14]

Active learning inspired controlnet guidance for aug- menting semantic segmentation datasets.arXiv preprint arXiv:2503.09221, 2025

Hannah Kniesel, Pedro Hermosilla, and Timo Ropin- ski. Active learning inspired controlnet guidance for aug- menting semantic segmentation datasets.arXiv preprint arXiv:2503.09221, 2025. 3

work page arXiv 2025

[15] [15]

Dataset enhancement with instance-level augmentations

Orest Kupyn and Christian Rupprecht. Dataset enhancement with instance-level augmentations. InEuropean Conference on Computer Vision, pages 384–402. Springer, 2024. 2, 3, 4, 5, 6, 7, 13

work page 2024

[16] [16]

Tracer: Extreme attention guided salient object tracing network (stu- dent abstract)

Min Seok Lee, WooSeok Shin, and Sung Won Han. Tracer: Extreme attention guided salient object tracing network (stu- dent abstract). InProceedings of the AAAI conference on artificial intelligence, pages 12993–12994, 2022. 3

work page 2022

[17] [17]

Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InInterna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 2, 3

work page 2022

[18] [18]

A simple background augmentation method for object detection with diffusion model

Yuhang Li, Xin Dong, Chen Chen, Weiming Zhuang, and Lingjuan Lyu. A simple background augmentation method for object detection with diffusion model. InEuropean Con- ference on Computer Vision, pages 462–479. Springer, 2024. 2, 3, 4, 5, 6, 13

work page 2024

[19] [19]

Focal loss for dense object detection

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll´ar. Focal loss for dense object detection. InPro- ceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017. 1

work page 2017

[20] [20]

Decoupled weight decay regularization

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In7th International Conference on Learning Representations, ICLR, 2019. 5

work page 2019

[21] [21]

Repaint: Inpainting using denoising diffusion probabilistic models

Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. InProceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461–11471, 2022. 2, 5

work page 2022

[22] [22]

Uavid: A semantic segmentation dataset for uav imagery.ISPRS journal of photogrammetry and remote sensing, 165:108–119, 2020

Ye Lyu, George V osselman, Gui-Song Xia, Alper Yilmaz, and Michael Ying Yang. Uavid: A semantic segmentation dataset for uav imagery.ISPRS journal of photogrammetry and remote sensing, 165:108–119, 2020. 2, 5

work page 2020

[23] [23]

CEREALS - cost- effective region-based active learning for semantic segmen- 9 tation

Radek Mackowiak, Philip Lenz, Omair Ghori, Ferran Diego, Oliver Lange, and Carsten Rother. CEREALS - cost- effective region-based active learning for semantic segmen- 9 tation. InBritish Machine Vision Conference 2018, BMVC, page 121. BMV A Press, 2018. 2

work page 2018

[24] [24]

Dataset diffusion: Diffusion-based synthetic data generation for pixel-level semantic segmentation.Advances in Neural Information Processing Systems, 36:76872–76892, 2023

Quang Nguyen, Truong Vu, Anh Tran, and Khoi Nguyen. Dataset diffusion: Diffusion-based synthetic data generation for pixel-level semantic segmentation.Advances in Neural Information Processing Systems, 36:76872–76892, 2023. 1, 3

work page 2023

[25] [25]

Uncertainty-aware controlnet: Bridging domain gaps with synthetic image generation

Joshua Niemeijer, Jan Ehrhardt, Heinz Handels, and Hristina Uzunova. Uncertainty-aware controlnet: Bridging domain gaps with synthetic image generation. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 4184–4193, 2025. 3

work page 2025

[26] [26]

Maxime Oquab, Timoth ´ee Darcet, Th´eo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Je- gou, Julien Mairal, Patr...

work page 2024

[27] [27]

Scalable diffusion models with transformers

William Peebles and Saining Xie. Scalable diffusion models with transformers. InProceedings of the IEEE/CVF inter- national conference on computer vision, pages 4195–4205,

work page

[28] [28]

Sdxl: Improving latent diffusion models for high-resolution image synthesis

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas M ¨uller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. InInternational Confer- ence on Learning Representations, pages 1862–1874, 2024. 2, 5

work page 2024

[29] [29]

High-resolution image synthesis with latent diffusion models

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. 1, 3, 5

work page 2022

[30] [30]

Training region-based object detectors with online hard ex- ample mining

Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard ex- ample mining. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 761–769,

work page

[31] [31]

Effective data augmentation with diffusion models

Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models. InInternational Conference on Learning Represen- tations, pages 14590–14612, 2024. 3

work page 2024

[32] [32]

Datasetdm: Synthesizing data with perception annota- tions using diffusion models.Advances in Neural Informa- tion Processing Systems, 36:54683–54695, 2023

Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annota- tions using diffusion models.Advances in Neural Informa- tion Processing Systems, 36:54683–54695, 2023. 1, 3

work page 2023

[33] [33]

Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using dif- fusion models

Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using dif- fusion models. InProceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 1206–1217,

work page

[34] [34]

Segformer: Simple and efficient design for semantic segmentation with transform- ers.Advances in neural information processing systems, 34: 12077–12090, 2021

Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transform- ers.Advances in neural information processing systems, 34: 12077–12090, 2021. 7

work page 2021

[35] [35]

Freemask: Synthetic images with dense annotations make stronger segmentation models.Advances in Neural Information Processing Systems, 36:18659–18675,

Lihe Yang, Xiaogang Xu, Bingyi Kang, Yinghuan Shi, and Hengshuang Zhao. Freemask: Synthetic images with dense annotations make stronger segmentation models.Advances in Neural Information Processing Systems, 36:18659–18675,

work page

[36] [36]

Learning loss for ac- tive learning

Donggeun Yoo and In So Kweon. Learning loss for ac- tive learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 93–102,

work page

[37] [37]

Bdd100k: A diverse driving dataset for heterogeneous multitask learning

Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Dar- rell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 2636–2645, 2020. 2, 5

work page 2020

[38] [38]

Adding conditional control to text-to-image diffusion models

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3836–3847, 2023. 2, 3, 4

work page 2023

[39] [39]

X-paste: Revisiting scalable copy-paste for instance segmentation us- ing CLIP and StableDiffusion

Hanqing Zhao, Dianmo Sheng, Jianmin Bao, Dongdong Chen, Dong Chen, Fang Wen, Lu Yuan, Ce Liu, Wenbo Zhou, Qi Chu, Weiming Zhang, and Nenghai Yu. X-paste: Revisiting scalable copy-paste for instance segmentation us- ing CLIP and StableDiffusion. InProceedings of the 40th In- ternational Conference on Machine Learning, pages 42098– 42109. PMLR, 2023. 2

work page 2023

[40] [40]

Recon: Region-controllable data augmentation with rectification and alignment for object detection.Advances in Neural Information Processing Systems, 38:74897–74926,

Haowei Zhu, Tianxiang Pan, Rui Qin, Jun-Hai Yong, and Bin Wang. Recon: Region-controllable data augmentation with rectification and alignment for object detection.Advances in Neural Information Processing Systems, 38:74897–74926,

work page

[41] [41]

The columns show the original image, the predictive-entropy map, the preserved uncertain region, and the final augmented image after inpainting and paste-back

Additional Uncurated Qualitative Samples We show more examples that are sampled without any qual- itative filtering in Figure 5 and Figure 6. The columns show the original image, the predictive-entropy map, the preserved uncertain region, and the final augmented image after inpainting and paste-back. Every preserved pixel in the augmented image is bit-wis...

work page

[42] [42]

photorealistic, ultra-detailed, 4K high- resolution, sharp focus, high quality, in the style of the Cityscapes dataset

Additional Implementation Details This section specifies everything needed to reproduce the method and results in the main paper: the fixed data sub- sets, preprocessing, the training and fine-tuning schedules, generation settings, and the checkpoint-selection rule. 7.1. Data Splits and Reproducibility The training data splits used in our experiments are ...

work page

[43] [43]

To apply this to densely labeled semantic segmentation use-cases like Cityscapes [3], we assign each semantic class to either fore- ground or background

Foreground/Background Split For Baseline Methods The two synthetic augmentation baselines used in our ex- periment rely on spatial heuristics based on foreground ob- jects and the background of each image. To apply this to densely labeled semantic segmentation use-cases like Cityscapes [3], we assign each semantic class to either fore- ground or backgroun...

work page