Pith. sign in

REVIEW 3 major objections 7 minor 35 references

CorrFill: Enhancing Faithfulness in Reference-based Inpainting with Correspondence Guidance in Diffusion Models

T0 review · 3 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CorrFill claims a training-free plug-in that uses self-attention-derived correspondences to guide diffusion inpainting, improving faithfulness to reference images and raising PSNR by up to 3.84 dB on tested baselines.

desk verdict A useful training-free plug-in for diffusion inpainting, but the headline gains are inflated by per-dataset hyperparameter tuning and the strongest baseline gains are marginal. read the letter →

arxiv 2501.02355 v1 pith:IOMKNB37 submitted 2025-01-04 cs.CV

Kuan-Hung Liu , Cheng-Kun Yang , Min-Hung Chen , Yu-Lun Liu , Yen-Yu Lin This is my paper · ORCID
classification cs.CV
keywords reference-basedimageinpaintingdiffusionmodelscorrespondenceguidancetraining-freeplug-inself-attentionattentionmaskinglatenttensoroptimizationfaithfulness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that diffusion-based reference-guided inpainting is unfaithful to the reference image because nothing in the pipeline enforces geometric correspondence between reference and damaged target. To fix this, it proposes CorrFill, a training-free plug-in that reads correspondences out of the self-attention maps the inpainting model already produces, then uses those correspondences to guide the remaining denoising steps through attention masking and latent tensor optimization. The authors show that plugging CorrFill into four different diffusion baselines improves PSNR, SSIM, and LPIPS on the RealEstate10K and MegaDepth datasets, with the largest PSNR gain being 3.84 dB on an image-adapter baseline. If the claim is right, CorrFill offers a general way to make existing reference-based inpainting models more faithful without retraining any weights.

What carries the argument

The load-bearing mechanism is the attention-consensus correspondence: the map $P_t$ computed as the argmax of the accumulated target-to-reference self-attention scores $C_t$ (Eq. 1), together with the filtering and smoothing steps that clean it. This correspondence is turned into guidance through attention masks $m_t$ that boost or suppress which reference tokens each target token may attend to, and through an objective $S(\cdot)$ that optimizes the latent $z_t$ so its attention maps match the one-hot correspondence. The two roles feed each other: guidance shapes the denoising step, and the denoising step's fresh attention maps update the correspondence for the next step, which is the cycle that lets correspondence inside damaged regions improve without extra training.

What would settle it

Take image pairs with known ground-truth geometry, for example rendered scenes with exact homographies, mask part of the target, and compare the method's estimated $P_t$ to the true flow; if precision in masked regions is no better than chance while PSNR gains disappear, the correspondence-guidance mechanism is not doing the claimed work.

Watch

Extended reading notes

Core claim

CorrFill's central claim is that a reference-based inpainting diffusion model can become more faithful if its own self-attention scores are converted into explicit geometric constraints between the reference and target images. Because the reference and target are stitched side by side, self-attention acts as a soft correspondence map; aggregating these maps over attention layers and timesteps and taking an argmax gives the correspondence $P_t$ (Eq. 1). The correspondences are then refined by removing dominant-token outliers and smoothing displacements, and are imposed on denoising by masking self-attention (Eqs. 3-4) and by optimizing the input latent $z_t$ with a weighted binary cross-entropy objective against the one-hot correspondence (Eq. 5). Since the updated attention maps feed back into the correspondence estimate, inpainting and correspondence estimation improve cyclically. The paper reports consistent gains over Paint-by-Example, IP-Adapter-Plus, Side-by-side, and LeftRefill, and attributes the largest gains to methods that were not trained specifically for reference-based inpainting.

Load-bearing premise

The load-bearing premise is that the argmax over accumulated self-attention scores identifies true reference-target correspondences, even inside the damaged regions, and that steering later denoising steps toward those correspondences corrects errors instead of amplifying them; if the attention scores are systematically wrong, the guidance reinforces the mistakes.

Editorial extensions

If this is right

  • CorrFill is a drop-in module for any diffusion inpainting model whose self-attention operates over a stitched reference-target latent, requiring no retraining and no new parameters.
  • The reported results indicate that the largest gains occur for baselines not specifically trained for reference-based inpainting, so the module acts as a faithfulness corrector rather than a task-specific architecture.
  • The cyclic design implies that correspondence estimates in masked regions become more accurate as denoising proceeds, a property the supplementary supports by counting correct correspondences against pseudo-ground truth.
  • Correspondence smoothing, though small in average metrics, is credited with fixing specific wrong-geometry cases, so its role is case-dependent rather than uniform.
  • On wide-baseline or repetitive-structure scenes, the paper's own failure cases show that 2D correspondence guidance is not enough, so improvements there are expected to be smaller.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper does not explore is combining CorrFill with an external geometric prior or a pretrained matcher that initializes or verifies the attention-derived correspondences, which could reduce failures on repetitive structures and large viewpoint changes.
  • The same correspondence-guidance cycle could be applied to other stitched multi-image diffusion tasks, such as video frame completion or stereo inpainting, where geometric constraints are naturally stronger.
  • Because the guidance parameters were tuned on subsets of the two benchmark datasets, an independent re-evaluation on fresh pairs or a held-out dataset would show how much of the reported gain is parameter-free.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper presents CorrFill, a training-free module for reference-based image inpainting with diffusion models. The method stitches the reference and target images side by side, aggregates target-to-reference self-attention maps across layers and timesteps of the denoising UNet into a matching map C_t, and extracts a correspondence field P_t via an argmax (Eq. 1). P_t is then used to guide the next denoising step through two mechanisms: attention masking, which boosts or hard-masks attention values near the predicted correspondence while suppressing dominant irrelevant tokens (Eqs. 3-4), and latent-tensor optimization, which applies a weighted BCE loss between per-layer target-to-reference attention maps and a one-hot encoding of P_t (Eq. 5). The process is cyclic: updated attention maps are accumulated back into C_t, refining P_t for the next step. Experiments in Table 1 report PSNR/SSIM/LPIPS gains over four diffusion-based baselines on 500-pair subsets of RealEstate10K and MegaDepth; Table 2 ablates the components; the supplementary reports a correspondence-accuracy comparison and a time analysis; the final section candidly lists failure modes (repetitive structures, large viewpoint changes, large masks).

Significance. The core idea--using the inpainting model's own self-attention as a shared domain for both correspondence estimation and guidance--is attractive and, if validated, would deliver a genuinely useful property: a plug-in that improves reference-target fidelity in reference-based inpainting without retraining or extra learnable parameters. Credit should go to the paper's real strengths: the method is architecture-compatible across several baselines; the ablation in Table 2 and the cyclic-versus-ablated correspondence diagnostics in Supp. B.2 are the right validation tools for the guidance loop; and the limitations (Sec. 5, Supp. C.2, Fig. 6) are reported honestly. However, the current evidence does not yet establish the headline claim: the largest gains are on baselines whose input representation is simultaneously changed, the state-of-the-art baseline (LeftRefill) gains only +0.26 dB PSNR on RealEstate10K and +0.00 dB on MegaDepth, and the reported numbers come from per-dataset hyperparameter selection on the evaluation subsets (Supp. A.4) without significance testing.

major comments (3)
  1. [Supp. A.4 / Supp. Table 1 / Table 1] The headline improvements in Table 1 are computed with hyperparameters selected on subsets of the same benchmark datasets used for the reported means: Supp. A.4 states that 'We selected the parameters by evaluating the subsets of our datasets,' and Supp. Table 1 shows that the six parameters (Stepa, Stepo, Wina, Wins, Stra, Stro) take different values for every baseline and every dataset, with Stepa ranging from 5 to 50 and Stra set to 0 in one configuration. The 500-pair means in Table 1 are therefore best-case numbers from a per-dataset search rather than estimates from a fixed method, and no confidence intervals or significance tests are reported. Consequently, the abstract's claim that CorrFill 'significantly enhances the performance of multiple baseline diffusion-based methods, including state-of-the-art approaches' is not supported as an unbiased statement; in particular, the state-of-the-art baseline LeftRefill improves by only +0.26 dB PSNR on RealEstate10K and +0.00 dB on MegaDepth with essentially no SSIM change. I request a proper validation protocol: fix the parameters once (or tune on a separate validation split drawn from the same sources), report performance on a disjoint test set, and provide confidence intervals or paired tests for the gains.
  2. [Sec. 3.2 and Sec. 3.4, Eqs. (1) and (5)] The guidance loop is self-referential in a way that creates an error-lock-in risk: the correspondences P_{t+1} are computed in Eq. (1) as the argmax of the accumulated attention maps, and Eq. (5) then minimizes a BCE loss that coerces those very attention maps toward a one-hot encoding of P_{t+1}. When the initial argmax is wrong--which occurs in repetitive structures, large viewpoint changes, and large masks, all listed as failure modes in Sec. 5 and Supp. C.2--the loss sharpens attention around the wrong mode instead of correcting it, because the constraint target is derived from the same signal being constrained. Consistent with this risk, the ablation in Table 2 shows that attention masking alone degrades LeftRefill on RealEstate10K (26.71 to 26.45 dB) and only recovers after outlier filtering. The correspondence-accuracy analysis in Supp. B.2 is the right kind of evidence, but it is limited to RealEstate10K, it uses an argmax-based correctness metric that is closely tied to the guidance objective, and it does not diagnose the failure cases in Fig. 6. I suggest two additions: (i) an oracle bound in which P_{t+1} is replaced by the GIM pseudo-ground-truth correspondences, quantifying how much of the observed gain is lost to self-estimation error; and (ii) a plot of correspondence correctness over the 50 denoising steps for success versus failure examples, showing directly whether the cycle improves or locks in initial errors.
  3. [Sec. 4.2, Table 1] For Paint-by-Example and IP-Adapter-Plus, Sec. 4.2 states that the authors 'modify their inputs to match the stitched reference formulation and then apply our CorrFill,' so the reported gains (+1.54 dB and +3.84 dB on RealEstate10K) simultaneously change the input representation and add the guidance module. Because the corresponding baseline numbers come from the unstitched pipelines, the gains over these two baselines cannot be attributed to the correspondence guidance alone; part of the effect may come from the UNet directly seeing the reference through the stitched latent. No control column with stitched input and guidance disabled is provided for these baselines, and the implementation of the input modification for models whose released weights are not fine-tuned for the 9-channel inpainting input is not described in Supp. A.1. Please either provide matched-input baselines for Paint-by-Example and IP-Adapter-Plus or relegate these two rows to a supplementary comparison and base the plug-in claim on the stitched-formulation methods (Side-by-side and LeftRefill), where the comparison is clean.
minor comments (7)
  1. [Sec. 3.4, Eq. (5)] The function Norm(·) and the weighting in the weighted BCE are never defined, and the one-hot encoding E(·) requires a discretization decision for the sub-pixel smoothed correspondences P*_t; please specify these details, since Eq. (5) is the main driver of the large runtime increase reported in Supp. Table 2.
  2. [Sec. 3.2 and Sec. 3.3] Please clarify whether P_t stores token indices or continuous coordinates: Eq. (1) is an argmax over discrete tokens, whereas Sec. 3.3 adds fractional displacement fields D*_t (Eq. 2) and assigns them back to P_t, mixing discrete and continuous representations without comment.
  3. [Supp. B.2] The PSNR values 27.39/27.34/27.25 dB for Ours/No acc./No cyc. differ from the LeftRefill RealEstate10K entry in Table 1 (26.97 dB); please state the configuration and subset used in that analysis so the reader can reconcile the two numbers.
  4. [Supp. Table 1] For LeftRefill on MegaDepth the attention-mask boost value Stra is 0; please state what attention masking reduces to in that configuration (only the negative-infinity suppression, with no positive boost) and how the BCE loss remains active when the boost is zero.
  5. [Sec. 3.3] The dominant-token threshold of four tokens is claimed to be insensitive; support this claim with a one-line sensitivity check (for example, thresholds of 2, 4, and 8), since the threshold interacts with the multi-resolution aggregation across attention layers.
  6. [Sec. 4.1] Pair selection restricted to DreamSim distances between 0.1 and 0.2 on RealEstate10K bounds the task difficulty; please report the distributions of mask ratios and DreamSim distances in the released subset so that readers can gauge how challenging the benchmark is.
  7. [Sec. 4] The paper would benefit from a statement on code release, since the value of a plug-in method lies in the attention-substitution implementation across Diffusers pipelines; the project page currently promises only the dataset.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CorrFill's guidance loop is self-referential in mechanism, but the central claims are tested against external metrics and an external correspondence matcher, so the result is not equivalent to its inputs by construction.

full rationale

We find no circular step that meets the evidentiary standard. CorrFill estimates correspondence P_t as the argmax of accumulated self-attention maps (Eq. 1) and then uses P_{t+1} as the target in attention masking (Eq. 3) and in the BCE objective (Eq. 5). This is a self-consistency feedback loop: the constraint is a function of the attention maps it constrains, and the updated matching map C_t = C_{t+1} + A_t^{tar2ref} reuses attention maps that were coerced toward the previous argmax. However, the paper does not rely on this loop as proof; final inpainting quality is measured against ground truth with PSNR/SSIM/LPIPS (Table 1), and Supp. B.2 measures correspondence correctness against an external matcher (GIM) that defines pseudo-ground-truth correspondences, showing that the 'Ours' configuration outperforms the 'No acc.' and 'No cyc.' variants. The method's failures under repetitive structure, large viewpoint change, and large masks (Fig. 6, Supp. C.2) further demonstrate that the outcome is contingent and falsifiable rather than forced by definition. The only flagged weakness is statistical, not circular: Supp. A.4 states 'We selected the parameters by evaluating the subsets of our datasets,' and Supp. Table 1 gives per-dataset and per-baseline hyperparameter values, so the reported gains may be optimistically selected; this is an evaluation-validity and overfitting concern that should temper the abstract's broad 'significantly enhances' claim, but it does not make the derivation circular. There is no load-bearing self-citation or imported uniqueness theorem. Accordingly, the circularity score is 0.

Assumptions & free parameters 8 free parameters · 5 assumptions · 0 invented entities

The central claim rests on several unproven empirical assumptions about the interpretability and controllability of attention maps in a frozen Stable Diffusion inpainting model, plus a large set of per-dataset tuned hyperparameters. There are no invented physical entities. The method's self-referential correspondence estimation, together with test-set-tuned parameters, is the main burden.

free parameters (8)
  • Step_a (number of denoising steps with attention masking) = Varies per baseline and dataset, e.g., 50 (Paint-by-Example RealEstate10K), 5 (LeftRefill MegaDepth); see Supp. Table 1
    Tuned by evaluating on subsets of the evaluation datasets; directly controls how long correspondence guidance is applied.
  • Step_o (number of steps with latent tensor optimization) = Varies per baseline and dataset, e.g., 50 to 5; see Supp. Table 1
    Tuned per baseline/dataset on evaluation subsets; controls the strength and duration of gradient guidance.
  • Win_a (attention masking neighborhood radius) = Either token count (2 to 5) or ratio (0.3 to 0.4); see Supp. Table 1
    Defines the neighborhood around the matched point that receives boosted attention; tuned on evaluation subsets.
  • Win_s (correspondence smoothing radius) = Ratios from 0.05 to 0.4; see Supp. Table 1
    Radius for the weighted neighborhood average in Eq. 2; tuned per baseline/dataset.
  • Str_a (attention mask boost value v) = 0 or 1; see Supp. Table 1
    Added to affinity matrix for the matched neighborhood; tuned on evaluation subsets.
  • Str_o (latent optimization guidance weight) = 0.5 to 2; see Supp. Table 1
    Weight controlling the gradient step for latent tensor optimization; tuned per baseline/dataset.
  • Dominant token threshold = 4 tokens
    Reference tokens attended by more than 4 target tokens are treated as outliers (Sec. 3.3); chosen empirically and stated to be insensitive, but it is still a hand-set parameter.
  • DreamSim pair selection thresholds = Distance < 0.2 and > 0.1 for RealEstate10K, < 0.2 for MegaDepth
    Filters image pairs used for evaluation; the thresholds are not justified and affect the difficulty of the benchmark (Sec. 4.1).
assumptions (5)
  • domain assumption The self-attention maps of a frozen Stable Diffusion inpainting model on the stitched image provide meaningful approximations of correspondence, even inside damaged regions.
    Sec. 3.2 states that the self-attention scores present primitive approximations of correspondence, but this is an empirical claim with no theoretical justification and is shown to fail on repetitive structures and large masks.
  • ad hoc to paper Aggregating attention maps over layers and timesteps (consensus) filters out individual biases and yields more stable correspondences.
    Sec. 3.2 invokes consensus to eliminate biases, but no formal or systematic study is provided beyond an ablation of temporal accumulation in Supp. B.2.
  • domain assumption Enforcing attention to match the estimated one-hot correspondences via masks and the BCE objective improves inpainting faithfulness.
    Sec. 3.4 adopts this assumption from prior guidance works (HD-Painter, training-free layout control) but does not prove it holds when the correspondences themselves are derived from the same model.
  • domain assumption The pseudo-ground-truth correspondences from the GIM matcher (a learned model) are reliable for evaluating the correctness of estimated correspondences.
    Supp. B.2 uses GIM to generate pseudo-ground truth; GIM has its own biases and is not an independent measurement.
  • domain assumption The pre-trained Stable Diffusion v2 Inpainting model is a sufficient backbone for reference-based inpainting across all tested baselines.
    All experiments use this backbone; the method's effectiveness may not transfer to other diffusion architectures.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CorrFill: Enhancing Faithfulness in Reference-based Inpainting with Correspondence Guidance in Diffusion Models." pith.science (2026). https://pith.science/paper/IOMKNB37

@misc{pith2026250102355,
  author       = {Pith},
  title        = {Pith review of: CorrFill: Enhancing Faithfulness in Reference-based Inpainting with Correspondence Guidance in Diffusion Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IOMKNB37}},
  note         = {Machine review of arXiv:2501.02355}
}
read the original abstract

In the task of reference-based image inpainting, an additional reference image is provided to restore a damaged target image to its original state. The advancement of diffusion models, particularly Stable Diffusion, allows for simple formulations in this task. However, existing diffusion-based methods often lack explicit constraints on the correlation between the reference and damaged images, resulting in lower faithfulness to the reference images in the inpainting results. In this work, we propose CorrFill, a training-free module designed to enhance the awareness of geometric correlations between the reference and target images. This enhancement is achieved by guiding the inpainting process with correspondence constraints estimated during inpainting, utilizing attention masking in self-attention layers and an objective function to update the input tensor according to the constraints. Experimental results demonstrate that CorrFill significantly enhances the performance of multiple baseline diffusion-based methods, including state-of-the-art approaches, by emphasizing faithfulness to the reference images.

Figures

Figures reproduced from arXiv: 2501.02355 by the authors.

Figure 1
Figure 1. Overview. (a) The reference and target images are stitched side by side, serving as inputs to the model. (b) Reference￾based inpainting using an inpainting fine-tuned Stable Diffu￾sion [19] that employs our training-free correspondence guidance. (c) Our method captures more reliable correlations between ref￾erences and targets than previous methods [2], thereby avoiding incorrect geometry and unwanted objects. gener… view at source ↗
Figure 2
Figure 2. Approach overview. CorrFill jointly guides the inpainting and refines the estimated correspondences at each denoising step. The noise tensor N ϵ , the downscaled mask Mϵ and the encoded stitched image ϵ(Iref;tar) are concatenated into input latent tensor zT . For each denoising step, the self-attention scores from the diffusion model are aggregated into a matching map Ct, and the correspondence Pt are computed from … view at source ↗
Figure 3
Figure 3. Correspondences in the early stage. The image on the left highlights the masked regions of the target and their most attended positions in the reference, indicated by colors, at the very first denoising step. The image on the right depicts a few corre￾spondences computed at the first denoising step. spondence estimation and image inpainting. Unlike meth￾ods [27, 28] using nearest neighbor search with diffusion featu… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Correspondence guidance in the diffusion U-Net. At each denoising step t, the denoising process is guided by the corre￾spondences estimated in the previous step, Pt+1, through attention masking with mt and optimizing zt using the objective function S(·). The generated …
Figure 5
Figure 5. Figure 5: Qualitative results. We present the qualitative results with four different baselines and their counterparts integrated with our method on two datasets. We highlight the problematic regions in the results of the baseline methods that our approach can effectively addres…
Figure 6
Figure 6. Figure 6: Failure cases. The image on the left depicts the esti￾mated correspondences. The image on the right shows the inpaint￾ing result of CorrFill integrated on LeftRefill [2]. In the first row, the repetitive structures and complex geometry introduce ambigu￾ity in correspon…
Figure 2
Figure 2. Figure 2: Numbers of correct correspondences. The graph illustrates the numbers of correct correspondences for three ver￾sions of CorrFill. “T” denotes the timesteps of the reverse process, where inpainting progresses from T = 50 to 0. “Ours” represents our proposed method, whic…
Figure 3
Figure 3. Figure 3: Results with large masks. The inpainting and out￾painting results for the baseline method and CorrFill are presented. The first two rows depict the inpainting results, while the last row illustrates the outpainting results. All masks cover 50% of the tar￾get images. Co…
Figure 5
Figure 5. Figure 5: Additional results with LeftRefill. Problematic re￾gions addressed by CorrFill are highlighted within the red boxes. ample: Exemplar-based image editing with diffusion models. In CVPR, 2023. 1 [5] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip￾adapter: Text com…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 29 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

  2. [2]

    ediff-i: Text-to-image diffusion models with ensemble of expert denoisers

    Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu. ediff-i: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.01324 , 2022

    arXiv 2022

  3. [3]

    Leftrefill: Filling right canvas based on left reference through generalized text-to-image diffusion model

    Chenjie Cao, Yunuo Cai, Qiaole Dong, Yikai Wang, and Yanwei Fu. Leftrefill: Filling right canvas based on left reference through generalized text-to-image diffusion model. In CVPR , 2024

    work page 2024

  4. [4]

    Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models

    Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. TOG , 2023

    work page 2023

  5. [5]

    Training-free layout control with cross-attention guidance

    Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In WACV , 2024

    work page 2024

  6. [6]

    Diffusion models beat gans on image synthesis

    Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In NeurIPS , 2021

    work page 2021

  7. [7]

    Diffusion self-guidance for controllable image generation

    Dave Epstein, Allan Jabri, Ben Poole, Alexei Efros, and Aleksander Holynski. Diffusion self-guidance for controllable image generation. In NeurIPS , 2023

    work page 2023

  8. [8]

    Training-free structured diffusion guidance for compositional text-to-image synthesis

    Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. In ICLR , 2023

    work page 2023

Show all 35 references
  1. [9]

    Dreamsim: Learning new dimensions of human visual similarity using synthetic data

    Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. In NeurIPS , 2023

    2023

  2. [10]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS , 2020

    2020

  3. [11]

    Auto-encoding variational bayes

    Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2022

    2022 arXiv

  4. [12]

    Megadepth: Learning single-view depth prediction from internet photos

    Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In CVPR , 2018

    2018

  5. [13]

    Diffusion hyperfeatures: Searching through time and space for semantic correspondence

    Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. In NeurIPS , 2023

    2023

  6. [14]

    Hd-painter: High-resolution and prompt-faithful text-guided image inpainting with diffusion models

    Hayk Manukyan, Andranik Sargsyan, Barsegh Atanyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Hd-painter: High-resolution and prompt-faithful text-guided image inpainting with diffusion models. arXiv preprint arXiv:2312.14091 , 2023

    2023 arXiv

  7. [15]

    T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. AAAI , 2024

    2024

  8. [16]

    Onion-peel networks for deep video completion

    Seoung Wug Oh, Sungho Lee, Joon-Young Lee, and Seon Joo Kim. Onion-peel networks for deep video completion. In ICCV , 2019

    2019

  9. [17]

    Maxime Oquab, Timoth \'e e Darcet, Th \'e o Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Micha...

    2024

  10. [18]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , 2021

    2021

  11. [19]

    Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment

    Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik. Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. In NeurIPS , 2023

    2023

  12. [20]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj\"orn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , 2022

    2022

  13. [21]

    High-fidelity guided image synthesis with latent diffusion models

    Jaskirat Singh, Stephen Gould, and Liang Zheng. High-fidelity guided image synthesis with latent diffusion models. In CVPR , 2023

    2023

  14. [22]

    Denoising diffusion implicit models

    Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR , 2021

    2021

  15. [23]

    Jacobs, Bharath Hariharan, Yael Pritch, Neal Wadhwa, Kfir Aberman, and Michael Rubinstein

    Luming Tang, Nataniel Ruiz, Qinghao Chu, Yuanzhen Li, Aleksander Holynski, David E. Jacobs, Bharath Hariharan, Yael Pritch, Neal Wadhwa, Kfir Aberman, and Michael Rubinstein. Realfill: Reference-driven generation for authentic image completion. TOG , 2024

    2024

  16. [24]

    Quadtree attention for vision transformers

    Shitao Tang, Jiahui Zhang, Siyu Zhu, and Ping Tan. Quadtree attention for vision transformers. In ICLR , 2022

    2022

  17. [25]

    Reference-based painterly inpainting via diffusion: Crossing the wild reference domain gap

    Dejia Xu, Xingqian Xu, Wenyan Cong, Humphrey Shi, and Zhangyang Wang. Reference-based painterly inpainting via diffusion: Crossing the wild reference domain gap. arXiv preprint arxiv:2307.10584 , 2023

    2023 arXiv

  18. [26]

    Paint by example: Exemplar-based image editing with diffusion models

    Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. In CVPR , 2023

    2023

  19. [27]

    Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arxiv:2308.06721 , 2023

    2023 arXiv

  20. [28]

    Telling left from right: Identifying geometry-aware semantic correspondence

    Junyi Zhang, Charles Herrmann, Junhwa Hur, Eric Chen, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. Telling left from right: Identifying geometry-aware semantic correspondence. In CVPR , 2024

    2024

  21. [29]

    A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence

    Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. In NeurIPS , 2023

    2023

  22. [30]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV , 2023

    2023

  23. [31]

    Geofill: Reference-based image inpainting with better geometric understanding

    Yunhan Zhao, Connelly Barnes, Yuqian Zhou, Eli Shechtman, Sohrab Amirghodsi, and Charless Fowlkes. Geofill: Reference-based image inpainting with better geometric understanding. In WACV , 2023

    2023

  24. [32]

    Stereo magnification: Learning view synthesis using multiplane images

    Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. In SIGGRAPH , 2018

    2018

  25. [33]

    Transfill: Reference-guided image inpainting by merging multiple color and spatial transformations

    Yuqian Zhou, Connelly Barnes, Eli Shechtman, and Sohrab Amirghodsi. Transfill: Reference-guided image inpainting by merging multiple color and spatial transformations. In CVPR , 2021

    2021

  26. [34]

    GIM : Learning generalizable image matcher from internet videos

    Xuelun Shen, zhipeng cai, Wei Yin, Matthias M \"u ller, Zijun Li, Kaixuan Wang, Xiaozhi Chen, and Cheng Wang. GIM : Learning generalizable image matcher from internet videos. In ICLR , 2024

    2024

  27. [35]

    Diffusers: State-of-the-art diffusion models, 2022

    Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models, 2022

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.