{"id":"77f61764-029d-4f91-891e-bc6ad185bd83","arxiv_id":"2412.00306","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Refine-by-Align uses diffusion cross-attention maps to locate the reference region matching a masked artifact, then re-inpaints the artifact with that reference detail.","lead":"Refine-by-Align fixes small artifacts in AI-generated images by finding the matching detail in a reference photo and copying it into the damaged region. The paper also introduces GenArtifactBench, a 146-image benchmark with human annotations for this new refinement task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quantitative superiority is not yet substantiated: the alignment hyperparameters (t=0, l=9) are selected using ground-truth correspondence on the same 146-image GenArtifactBench used for final metrics, and no held-out split, error bars, or significance tests are reported.","rationale":"Reading in good faith, the paper contributes a new task, a self-supervised two-stage model with shared weights, a new matching algorithm, and a human-annotated benchmark. It also gives the reference-guided inpainting baselines the accurate reference regions, which is a conservative choice favoring the baselines. The main fragility is not the internal logic of the method but the evaluation protocol used to support the headline comparison. The alignment hyperparameters are selected on the same 146-image benchmark used for the final quantitative and user-study results, so the mIoU-based validation in Sec. 4.4 is partially circular for the reported final numbers. A held-out evaluation would settle whether the fixed t=0,l=9 configuration generalizes to unseen artifact instances and source models. This is exactly the kind of issue that a conditional acceptance with code/data release and an independent evaluation split can address. I therefore agree with the reader's identification of the weakest assumption and see no reason to change the CONDITIONAL verdict, but the concern is real enough that unconditional acceptance would be premature.","tokens_in":15315,"tokens_out":4991,"duration_ms":47451,"concrete_test":"Construct disjoint selection and evaluation partitions of GenArtifactBench, e.g., by generated-image source model and object identity. Use only the selection partition to choose t and l (including the t=24,l=9 grid optimum and the t=0,l=9 speed choice), freeze the configuration, and recompute Table 1 metrics on the evaluation partition for all methods, with paired bootstrap 95% confidence intervals and a Wilcoxon signed-rank test against the strongest baseline. If the DINO-I and CLIP-I margins over AnyDoor/ObjectStitch vanish or fall inside the confidence intervals, the central superiority claim is not supported by the current evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Refine-by-Align consistently outperforms six baselines in identity preservation and realism. That claim rests on Table 1 and Table 2, computed on GenArtifactBench (Sec. 4.1). However, the alignment stage's only free parameters, the diffusion timestep t and the transformer layer l, are chosen on this exact benchmark: Algorithm 1 maximizes mIoU against ground-truth correspondence masks Mgt, and Sec. 4.4 reports a grid search over all t and l on GenArtifactBench (Figs. 4 and 7) before fixing t=0, l=9 for all comparisons. The final numbers in Table 1 are therefore measured on the same data used to select the method's configuration; the baselines receive no equivalent selection step. The choice is also not the grid-search optimum (t=24, l=9 was best), so inference uses a speed-accuracy compromise validated entirely on the test set. Additionally, no error bars or significance tests accompany Table 1, and the user study margins in Table 2 are as small as 55.83% vs. 44.17% over 240 comparisons, which may be within sampling noise. Because the paper's headline advantage over baselines is the main evidence for the method's value, this evaluation leakage is load-bearing: without a held-out split, the reported margins and the claimed generality of the fixed (t,l) configuration are not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Refine-by-Align proposes a two-stage, reference-guided pipeline for repairing localized artifacts in images produced by generative models. Given a generated image, a free-form artifact mask, and a reference image, the alignment stage uses cross-attention maps of a DINOv2-conditioned diffusion model to localize the region of the reference corresponding to the artifact, and the refinement stage inpaints the artifact using the extracted local reference features. The two stages share a single network and are trained self-supervised on Pixabay and a newly collected MVObj dataset. The paper also introduces GenArtifactBench, consisting of 146 annotated artifact/reference pairs from DreamBooth, Zero123++, AnyDoor, and IDM-VTON, and reports comparisons against six baselines using CLIP/DINO metrics and a user study.","tokens_in":15559,"tokens_out":7877,"duration_ms":70311,"significance":"The task is new and practically relevant, and the model-agnostic, no-test-time-optimization design is attractive. The creation of a benchmark with dense human annotations is a useful contribution, as is the self-supervised training scheme that avoids per-instance tuning. The paper also demonstrates qualitative results across several downstream generation tasks. However, the quantitative evidence for the central claim is currently weakened by the use of the evaluation benchmark for alignment-hyperparameter selection, and by the absence of uncertainty estimates. If the reported gains survive a clean held-out evaluation, the work would be a solid contribution; at present the headline superiority is not fully established.","major_comments":[{"comment":"The alignment stage's only free parameters, the diffusion timestep t and transformer layer l, are selected by a grid search on GenArtifactBench using the ground-truth correspondence masks Mgt (Algorithm 1, lines 7-8; Fig. 4; Sec. 4.4), and the same benchmark is used to report the final numbers in Tables 1 and 2. This is evaluation leakage: the configuration is chosen on the test set, the baselines receive no equivalent selection step, and the chosen t=0, l=9 is not even the grid-search optimum (t=24, l=9), so the final configuration is a speed-accuracy tradeoff validated entirely on the benchmark. Please re-run the evaluation with a held-out split or nested cross-validation, and clearly state that the fixed configuration is selected without access to the GenArtifactBench labels.","section":"Sec. 4.4, Algorithm 1, Table 1"},{"comment":"The quantitative claims in Table 1 are reported as point estimates without error bars, confidence intervals, or significance tests. In Table 2, the smallest identity-preference margin is 55.83% vs. 44.17% (against ObjectStitch), and the CLIP-T score in Table 1 is actually below PAL (25.4063 vs. 25.8906); without variance estimates or a preference test the claim of consistently superior performance is not statistically supported. Please report bootstrap confidence intervals for the metrics and exact per-row comparison counts for the user study.","section":"Tables 1 and 2"},{"comment":"The paper states that spatial correspondence 'exists' and is 'proved' in Sec. 3.2, but the argument is an informal intuition about cross-attention aggregation; no formal statement or direct quantitative validation is provided for arbitrary free-form masks, object categories, and pose differences. Since the alignment stage is the key novel component and the output M* is the only mechanism that distinguishes the method from blind inpainting, please provide a direct alignment-accuracy evaluation (e.g., mIoU against human annotations) on data not used for configuration selection, in addition to the qualitative comparison with DIFT/DHF in Fig. 2.","section":"Sec. 3.2 and Sec. 3.3"}],"minor_comments":[{"comment":"Algorithm 1 includes the ground-truth mask Mgt as an input and computes mIoU; this is a validation-time grid-search procedure, not the inference-time algorithm described in Sec. 3.5. Please relabel it accordingly and state that Mgt is unavailable at inference.","section":"Algorithm 1"},{"comment":"The notation is inconsistent: E is R^{256×768} in Algorithm 1 but R^{256×d_phi} in Eq. (3), and the mask Ma is resized to d×d in Algorithm 1 but not in Eq. (5).","section":"Sec. 3.2 and Algorithm 1"},{"comment":"The post-processing pipeline (noise filter, peak detection, clustering) is described only verbally; the threshold and clustering hyperparameters should be reported for reproducibility.","section":"Sec. A.4"},{"comment":"The relationship between the 240 comparisons per question and the 720 total votes is unclear; please state the number of votes per comparison and the number of comparisons per baseline row.","section":"Table 2"},{"comment":"The analysis in Figs. 4 and 7 is called an ablation, but because it is computed on the same benchmark as the final evaluation, it is better described as configuration selection; use a validation set for parameter analysis.","section":"Sec. 4.4"}],"recommendation":"major_revision","confidential_remarks":"The evaluation leakage is the main barrier; the paper otherwise makes a plausible contribution. I would want to see a held-out split and uncertainty estimates before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Fair warning up front: the headline numbers in this paper should not be taken at face value. The alignment hyperparameters (t=0, l=9) are chosen by grid search on GenArtifactBench using ground-truth correspondence masks (Alg. 1, Sec. 4.4), and the same benchmark is then used for the final comparisons in Table 1. That is test-set tuning, and it weakens the main evidence that the method beats six baselines. The baselines don't get an equivalent selection step. Also, the chosen setting is not even the grid-search optimum; it's a speed-accuracy compromise made on the test data.\n\nThat said, the paper is a real attempt at a useful task. Reference-guided artifact refinement—fixing a masked artifact region by transferring the corresponding details from a reference image—is new as far as I know, and GenArtifactBench (146 annotated image groups from four generators) is a genuinely useful resource. The two-stage design (cross-attention alignment plus reference-region-guided inpainting, with shared weights) is a sensible integration of existing components. The self-supervised training scheme with alignment and refinement modes is clever, and the ablations show that the alignment stage and DINOv2 encoder each contribute. The qualitative results look plausible, and the model-agnostic claim is supported across DreamBooth, Zero123++, AnyDoor, and IDM-VTON.\n\nThe soft spots beyond the evaluation leakage: there are no error bars or significance tests anywhere, and the user study has some very thin margins (55.83% vs 44.17%). With 240 comparisons per question, those numbers could easily be noise, especially without a statistical test. The 'proof' of spatial correspondence in cross-attention (Sec. 3.2) is really just an intuitive argument, not a proof; that's a smaller issue. The method's generality rests on the assumption that one forward pass at t=0 and layer 9 reliably aligns arbitrary free-form masks, and that assumption is only checked on the same 146-image benchmark.\n\nWho should read this: anyone working on diffusion-based editing, artifact removal, or personalized generation. The benchmark alone justifies a look. But the quantitative claims need to be verified on a held-out split, with error bars and a proper protocol. I'd send it to review—the task is new and the benchmark is valuable—but I'd ask the authors to redo the evaluation cleanly and release code and data.","headline":"New task and benchmark worth engaging with, but the main quantitative claim is undermined by selecting alignment hyperparameters on the test benchmark itself.","tokens_in":16172,"tokens_out":2791,"would_cite":true,"duration_ms":25636,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Refine-by-Align is a two-stage diffusion pipeline that repairs localized identity artifacts in generated images by matching the masked region to a reference image through cross-attention, with no test-time tuning.","keywords":["reference-guided artifacts refinement","diffusion models","cross-attention alignment","identity preservation","image inpainting","semantic correspondence","GenArtifactBench","generative artifact removal"],"falsifier":"On a held-out artifact/reference set not used to select $t$ and $l$, compare the aggregated cross-attention map at $t=0$, layer 9 with human-annotated reference regions; low mIoU on large pose or shape mismatches, together with no DINO-image gain over feeding the full reference, would refute the alignment claim.","tokens_in":15058,"feed_emoji":"🎨","tokens_out":6970,"duration_ms":60950,"temperature":0.7,"pith_summary":"The paper introduces reference-guided artifacts refinement as a task: given a generated image with a user-marked artifact mask and a reference image, repair the artifact by transferring the corresponding detail from the reference. It argues that the key difficulty is not inpainting but alignment, finding which part of the reference matches the irregularly shaped artifact region. To do this, Refine-by-Align runs the diffusion model once, aggregates the cross-attention maps over the masked area, and takes the resulting correspondence map as the reference crop. The same network then refines the masked area using that crop. The paper claims this outperforms six existing baselines on identity preservation and realism, across customization, composition, view synthesis, and virtual try-on, and releases GenArtifactBench, a benchmark with human-annotated artifact/reference correspondences.","feed_headline":"A single attention pass repairs generated-image artifacts from a reference","feed_subtitle":"Refine-by-Align matches masked artifact regions to reference details and beats six baselines with no test-time tuning.","key_machinery":"The load-bearing object is the cross-attention map $A = \\mathrm{softmax}(qk^T)$, with queries from the noisy generated image and keys and values from 256 DINOv2 reference tokens. The alignment algorithm forms $M^{t,l} = \\sum_{i,j} M_a \\circ A[i,j,:]$, turning the masked artifact region into a 2D score map over reference locations; after noise removal and largest-blob selection this becomes the correspondence map $M^*$. A grid search over timesteps and transformer layers on the benchmark fixes the operating point at $t=0$, layer $l=9$, so inference needs only one denoising step for alignment. The same U-Net then uses the cropped reference $I_r \\otimes M^*$ as guidance, and the two training modes, alignment mode and refinement mode, share weights, which is what lets one model do both localization and repair.","core_discovery":"On its own terms, the paper's central claim is that the cross-attention maps of a diffusion model with a vision-encoder replacement carry enough spatial correspondence to localize a free-form artifact region inside a reference image, and that using only the localized reference region as guidance repairs the artifact while preserving identity. The paper implements this as Refine-by-Align, whose alignment stage aggregates attention maps over the masked pixels to produce a correspondence map, and whose refinement stage uses that map to crop the reference and drive the same weight-shared diffusion model to inpaint the masked area. On GenArtifactBench, the reported CLIP-image and DINO-image similarities are higher than for Paint-by-Example, ObjectStitch, AnyDoor, PAL, Cross-Image Attention, and MimicBrush, and a user study shows higher preference for both identity preservation and realism. Because the pipeline runs one forward pass at inference time, it adds no per-example optimization to the base generative model.","pith_inferences":["A natural next step is to test the same attention-aggregation recipe on other encoder-based diffusion models, since nothing in the algorithm is specific to the particular U-Net beyond the availability of cross-attention maps; if it transfers, alignment could become a generic module for reference-based editing.","The paper selects the inference timestep and layer by grid search on the evaluation benchmark itself; a stronger validation of generality would hold out artifact categories or source models when choosing $t$ and $l$.","Because the correspondence map is built from 16x16 reference tokens, very small logos or fine text may exceed the alignment resolution; using higher-resolution visual tokens or multi-scale attention could remove that ceiling.","The paper does not automate artifact detection, but its benchmark masks could be used to train a detector that proposes masks automatically, turning the interactive pipeline into a fully automatic repair system."],"forward_implications":["Any existing image generation model that produces localized identity artifacts can be paired with Refine-by-Align as a post-processing repair stage, without retraining or fine-tuning the generator.","Users only need to mark the artifact with a free-form mask; the system then chooses the reference region automatically, so no manual correspondence is required.","Because alignment selects a local reference patch instead of the whole object, tiny details such as logos, text, and textures are transferred more faithfully than with full-object guidance.","The shared-weight design means the same checkpoint performs both localization and refinement, keeping the pipeline compact and reducing memory overhead compared with separate matching and inpainting models.","GenArtifactBench supplies human-annotated artifact masks and reference correspondences, giving future artifact-refinement methods a common evaluation set."],"supporting_citations":[{"why":"Supplies the AnyDoor backbone that Refine-by-Align adapts, and is one of the generators whose outputs appear in the benchmark.","marker":"Chen et al. (2023)"},{"why":"DINOv2 provides the vision encoder whose 256 tokens form the keys and values for cross-attention alignment and refinement guidance.","marker":"Oquab et al. (2023)"},{"why":"Establishes the Prompt-to-Prompt cross-attention control result that motivates reading spatial correspondence from attention maps.","marker":"Hertz et al. (2022)"},{"why":"Defines perceptual artifacts, contributes the zoom-in inpainting strategy, and is the PAL baseline.","marker":"Zhang et al. (2023b)"},{"why":"DIFT is the keypoint-matching correspondence baseline that the alignment algorithm is compared against on irregular and repeating patterns.","marker":"Tang et al. (2023)"},{"why":"Diffusion Hyperfeatures is the second keypoint-matching baseline in the Fig. 2 comparison.","marker":"Luo et al. (2023a)"},{"why":"ObjectStitch contributes the Pixabay training data with panoptic labels and is a baseline that is fine-tuned on the same training set.","marker":"Song et al. (2023)"},{"why":"MimicBrush is the concurrent appearance-transfer baseline that the method claims to beat in realism and fidelity.","marker":"Chen et al. (2024)"}],"fun_headline_variants":["Attention map finds reference region to fix artifacts","One attention pass matches artifacts to reference details","Refine-by-Align: semantic alignment fixes image artifacts","No test-time tuning: reference-guided artifact repair","Diffusion model uses reference to inpaint artifact regions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that at a single fixed diffusion timestep and transformer layer, aggregating the cross-attention map over the artifact mask always lands on the corresponding reference region, regardless of artifact shape, object category, or pose difference.","fun_headline_variants_meta":{"raw":{"variants":["Attention map finds reference region to fix artifacts","One attention pass matches artifacts to reference details","Refine-by-Align: semantic alignment fixes image artifacts","No test-time tuning: reference-guided artifact repair","Diffusion model uses reference to inpaint artifact regions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00051,"raw_usage":{"total_tokens":2487,"prompt_tokens":956,"completion_tokens":1531,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":1459}},"tokens_in":572,"tokens_out":1531,"duration_ms":8843,"temperature":1.0,"reasoning_tokens":1459,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:30:48.441446+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out artifact/reference set not used to select $t$ and $l$, compare the aggregated cross-attention map at $t=0$, layer 9 with human-annotated reference regions; low mIoU on large pose or shape mismatches, together with no DINO-image gain over feeding the full reference, would refute the alignment claim.","supporting_citations":[],"review_version":1}