{"id":"7c8bae3a-c29b-420c-a56c-ec8ead51724d","arxiv_id":"2504.17253","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A frozen conditional diffusion model can be inverted via gradient-based discrete optimization, plus a learned layout prior, to perform object detection and faster classification without training a discriminative head.","lead":"This paper converts a pretrained layout-to-image diffusion model into an object detector by optimizing the layout input directly, guided by a separately trained prior over layouts. It also applies the same optimization idea to classification, matching the accuracy of enumeration-based diffusion classifiers with roughly 14x less compute.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (4) treats an unweighted sum of two denoising MSEs as log p(x|y) + log p(y); that identification is asserted, not derived, and if it fails the detection result supports an ad hoc objective rather than Bayes-optimal inversion of a pretrained generative model.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing point: Eq. (4) is not derived from the ELBOs, so the sum of independent denoising MSEs may not correspond to log p(x|y) + log p(y). This is the most important concern because the paper's theoretical framing and its claim of 'more accurate use of Bayes' rule' both rest on that unstated identification. The concern is concrete: the two losses have different reweightings, different noise schedules, and different scales, so the unweighted sum is not guaranteed to be a shared log-density. A ranking audit against calibrated likelihood estimates would settle whether the objective is a faithful posterior energy. I do not recommend changing the verdict because the empirical result and the speedup claim for classification are still informative even if the Bayesian derivation is incomplete. The paper also provides code and reproducible baselines, which gives some independent support. The concern warrants a conditional rather than a rejection: it does not overturn the novelty of using a frozen conditional diffusion model for detection, but it does cast doubt on the theoretical interpretation and on whether the AP numbers are evidence of Bayesian discriminative ability.","tokens_in":18908,"tokens_out":5760,"duration_ms":64209,"concrete_test":"Run a ranking audit on 50 COCO eval images. For each image, compute Eq. (5) values and estimated log-likelihood values log p_theta(x|y) + log p_phi(y) (using a calibrated diffusion likelihood estimator such as annealed importance sampling or the CIM estimator) for three layout sets: DIVE's optimized layout, the ground-truth layout, and 20 layouts sampled from the prior. If the ranking induced by Eq. (5) disagrees with the likelihood ranking on more than a small fraction of images, Eq. (4) is not the Bayes posterior and the central claim should be weakened to 'a heuristic inversion objective performs competitively on a 500-image subset' rather than 'Bayes-optimal inversion of a frozen generative model.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (4) is the hinge of the method: it replaces the Bayes objective argmax_y [log p_theta(x|y) + log p_phi(y)] with an unweighted sum of two denoising-MSE expectations. That substitution is asserted, not derived. In DDPM, the simplified epsilon-prediction loss is a reweighted ELBO, not an exact log-likelihood; the per-timestep weights, variance-schedule constants, and independent noise/timestep samplings are dropped. The two losses also operate in different spaces (latent image noise versus layout-embedding noise) with different scales, so their unweighted sum need not correspond to any common log-density. If Eq. (4) is not a faithful posterior energy, the reported AP is a property of an ad hoc optimization objective rather than evidence that a frozen generative model performs discriminative detection through Bayes' rule. The paper's ablation shows the prior term matters (AP 10.3 vs 6.7 on the 100-image set), but no experiment checks whether the minimized quantity ranks layouts according to p(y|x). This matters because the central contribution is framed as inverting generative models via Bayes' rule; without the identification, the detection results demonstrate an optimization heuristic, not the discriminative ability of the generative model under a Bayesian posterior.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes DIVE, a method for repurposing pretrained conditional diffusion models for discriminative tasks, specifically object detection. Following the Bayes-rule inversion paradigm of prior 'diffusion classifier' work, DIVE treats the detection label (a layout sequence of class and bounding-box tokens) as learnable parameters and optimizes it by gradient descent on the sum of two diffusion denoising losses: one from a frozen layout-to-image model (the likelihood term) and one from an auxiliary prior layout diffusion model (the prior term). To handle the discrete label vocabulary, the method uses a straight-through nearest-neighbor embedding replacement. Experiments on COCO 2017 report AP 7.1 on a 500-image subset, comparable to a 256x256 from-scratch Faster R-CNN R50 (AP 6.8); ablations on 100 images show that both the prior model and the in-vocabulary optimization contribute. On ImageNet classification, DIVE matches the enumeration-based Diffusion Classifier (77.2 vs 77.3) at roughly 14x lower per-image cost. The paper additionally proposes DIVE as an evaluation metric for conditional diffusion models.","tokens_in":19307,"tokens_out":15425,"duration_ms":146485,"significance":"Assuming the central derivation holds, this is the first non-enumeration inversion of a frozen conditional diffusion model for a structured discriminative task with a non-uniform label space, and the learned prior-layout model is a sensible response to the non-uniform layout distribution. The paper has real strengths: code and models are promised; the ablations isolate both proposed components; the classification experiments validate the optimization-based inversion against the enumeration baseline at matched accuracy with a 14x speedup; and the detection baselines are reproduced at matching resolution, which is transparent. The paper also candidly acknowledges its limitations, including compute cost and a concluding caveat about dense-task scalability. The result, however, rests on the asserted identification in Eq. (4) between a sum of two denoising-MSE losses and the log-posterior, and the detection evaluation is thin: a 500-image subset without selection details, no error bars, and a 0.3-AP gap to the 'competitive' Faster R-CNN baseline. These issues are addressable in revision but are load-bearing for the paper's central claims.","major_comments":[{"comment":"The identification of the unweighted sum of two denoising-MSE expectations with log p_theta(x|y) + log p_phi(y) is asserted rather than derived. Eq. (1) itself is only approximate (the simplified epsilon-prediction loss is a reweighted version of the ELBO, with per-timestep coefficients and variance-schedule constants dropped), and Eq. (4) compounds this by summing two losses that live in different spaces (the 32x32x4 latent image space vs. the 100xd layout-embedding space) with independent noise and timestep samplings and different intrinsic scales. The default weight of 1 between the two terms is therefore not justified by any common log-density. This matters because the abstract and Section III-B frame DIVE as making 'more accurate use of the Bayes' rule' and obtaining the posterior maximum. Please either derive the relationship explicitly (e.g., via a weighted ELBO decomposition with the appropriate per-timestep coefficients), or reframe the objective as a heuristic surrogate objective; and add a validity check: on a set of held-out images, compare the objective value at the ground-truth layout against the value at corrupted or random layouts, and sweep the relative weight between the two loss terms to test whether the unit weight is near-optimal. If AP depends strongly on a non-unit weight, the method is not optimizing the stated posterior.","section":"Section III-B2, Eq. (4)"},{"comment":"The detection evaluation is currently too thin to support the headline comparison. The 500-image subset of COCO val is not described (random, seeded, or filtered), no error bars or significance tests are given, and the main claim of being on par with Faster R-CNN R50 rests on AP 7.1 vs 6.8, a 0.3-point gap that is likely within noise at this sample size; on AP50 DIVE is actually lower (11.0 vs 12.7). In addition, the optimization-step hyperparameter K is swept on the same 500 images in Table I before being fixed at 2000, so the reported AP is a test-set-selected result. Please report bootstrap confidence intervals for AP, AP50, and AP75, describe the subset selection procedure explicitly, and either temper the statement about 'competitive performance compared to discriminative methods' or support it with variance-aware comparisons.","section":"Section IV-A1a and Table II"},{"comment":"The manuscript should clarify what 'frozen pretrained image generation model' means in this setup. The layout-to-image LDM-8 (363M) is retrained by the authors on COCO itself because no public model exists (footnote 1), and the prior layout model is trained on the same COCO layout distribution, so both the likelihood and the prior are in-distribution for the test task. This is not a deficiency of the inversion idea, since the Diffuser Classifier line of work has the same property, but the claim in Section IV-A2b ('the first to show that using a frozen pretrained image generation model can successfully tackle the challenging object detection task') should be qualified: the model is frozen at inference, but it is not an externally pretrained model transferring to a new task, and the prior is learned from the target label distribution. The abstract's wording about 'leveraging pretrained diffusion models' should be adjusted to match this setup.","section":"Section IV-A1b and Section IV-A2b"}],"minor_comments":[{"comment":"Please state explicitly how the 500-image subset was selected (for example, a random draw with a fixed seed) so that the reader can judge its representativeness; the current sentence 'we evaluate DIVE on a subset of 500 images' is underspecified.","section":"Section IV-A1a"},{"comment":"In Algorithm 1, the eval function drops none-value and illegal boxes before computing monitor losses; please clarify how the sequence is re-padded to the fixed length of 100 for the loss computation, since both diffusion models expect a fixed-length input.","section":"Algorithm 1 / Section III-B3a"},{"comment":"The timestep set T is shared between the image model and the prior model, but the two models add noise to different spaces (image latents vs. layout embeddings) with different scales; please state the noise schedules used in each model and justify the shared T, or report sensitivity to this choice.","section":"Section III-B3a"},{"comment":"The notation is confusing because v is used both for the learnable continuous parameters and for the output of the frozen embedding mapping v_theta; consider distinguishing the soft and discretized versions (for example, v_soft and v_discrete) to make the straight-through operation in Eq. (6) and the decoding in Eq. (7) easier to follow.","section":"Section III-B2, Eqs. (5)-(7)"},{"comment":"The time-per-image cells are formatted inconsistently ('18min, 45min, 1.5h, 3h'); please unify the units and state in the caption that these AP numbers are measured on the same 500-image subset used in Table II.","section":"Table I"},{"comment":"The concluding caveat about dense tasks, namely that the optimization space becomes large and the per-pixel objective contribution is small 'which may be problematic and needs to be addressed', is an important self-identified limitation; it should be reflected in the introduction or abstract so that the scope of the claimed contribution is clear from the start.","section":"Section V (Conclusion)"}],"recommendation":"major_revision","confidential_remarks":"Given the categorical novelty claim in Section IV-A2b, I suggest the editor ask the authors to document a targeted literature check for prior inversion of layout- or region-conditional diffusion models for detection beyond the related works cited in [88]-[90] and [69]-[70]. The paper fits the journal's scope as an applied contribution, and I found the authors transparent about compute costs and limitations; the requested revision is feasible within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the genuinely new thing here is turning diffusion-model inversion from enumeration into gradient-based optimization over the label embedding, plus a learned prior layout model to keep the optimization in plausible territory. The classification speedup is the most solid result. The detection story is not as strong as the paper claims: Eq. (4) is asserted rather than derived, and the evaluation is too small to support \"competitive with discriminative detectors.\"\n\nWhat's new and good: previous diffusion classifiers enumerate class labels, which doesn't scale to detection. Replacing enumeration with a straight-through discrete optimization over the vocabulary is a natural, sensible recipe, and the authors test it. They also rightly notice detection label priors are non-uniform and train a small diffusion prior over layouts; the ablation shows it helps (AP 10.3 vs 6.7). The classification experiment is clean: 77.2 vs 77.3 accuracy at ~14x lower cost than the enumeration baseline. That is reproducible, useful, and worth having. Code and models are public.\n\nWhere it's soft: Eq. (4) is the load-bearing step and it doesn't hold as written. The denoising loss is an ELBO-type surrogate, not the model's log-likelihood, and the paper drops all weights and constants when adding the two losses. The two terms live in different spaces (image latents vs layout embeddings) with different scales, so their unweighted sum is not obviously a common log-density. The method may still work as an optimization heuristic—the ablations suggest it does—but the Bayes-rule framing overstates what is established. On evaluation: 500 COCO images with no stated selection method and no error bars is thin for a \"competitive with Faster R-CNN\" claim, especially since the baseline detectors were trained from scratch at 256x256 and have AP ~7-9, far below standard COCO numbers. That comparison is not persuasive. The \"first to show competitive performance\" claim should be softened.\n\nThe citation pattern is fine; related work is broad and properly credited. The paper is honest about compute and limitations. Who this is for: anyone working on generative-to-discriminative transfer or using diffusion models as perception engines. It deserves a serious referee—the idea is novel, the classification speedup is a real contribution, and the detection part, while overclaimed, is a reasonable proof of concept. My recommendation: send to peer review with a major-revision request focused on the derivation of Eq. (4) and a more careful evaluation protocol.","headline":"The classification speedup is real and useful, but the detection claim rests on an asserted objective and a thin eval, so treat the headline as provisional.","tokens_in":19698,"tokens_out":3054,"would_cite":true,"duration_ms":29175,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a frozen, pretrained layout-to-image diffusion model can be inverted into an object detector that is competitive with a basic Faster R-CNN baseline on COCO, and that the same inversion speeds up generative…","keywords":["diffusion model","generative modeling","discriminative task","object detection","visual recognition","model inversion","layout-to-image generation","Bayes' rule"],"falsifier":"Estimate the true conditional log-likelihoods of DIVE's predicted layouts with a proper likelihood estimator (for example, annealed importance sampling or an ELBO with many noise samples), and compare the ranking induced by the sum-of-MSE objective with the ranking induced by the estimated likelihoods on a set of correct and incorrect layouts; a mismatch would show the objective is not the posterior. A cheaper check is to re-run detection while scaling the prior loss by a free weight; if AP changes substantially with that weight, the unweighted combination in Eq. (4) is not the unique Bayes objective.","tokens_in":18759,"feed_emoji":"🎯","tokens_out":12012,"duration_ms":94874,"temperature":0.7,"pith_summary":"DIVE studies whether a purely generative model — a frozen diffusion model trained to draw images from a layout description — can do discriminative vision tasks without finetuning or a separately trained head. The paper claims to be the first to show this works for object detection, by reversing the generation direction with Bayes' rule: instead of enumerating all possible labels as prior diffusion classifiers do, DIVE directly optimizes the layout embedding with gradients, and a learned prior over layouts supplies the non-uniform $p(y)$ that detection requires. On a 500-image COCO subset, the inverted generator reaches AP 7.1, slightly above the reproduced Faster R-CNN R50 baseline at 256×256 (6.8 AP). For classification, the same optimization scheme matches the enumeration-based diffusion classifier (77.2 vs 77.3 accuracy) while cutting per-image cost by a factor of about 14.","feed_headline":"Frozen diffusion model detects objects, matching Faster R-CNN","feed_subtitle":"Inverting a layout-to-image generator under Bayes' rule gives AP 7.1 on COCO and a 14x classification speedup.","key_machinery":"The load-bearing identity is Bayes' posterior maximization, $\\arg\\max_y p_\\theta(x|y)p(y)$, converted into an optimization over a learnable layout embedding. The carrying mechanism is gradient-based discrete optimization inside the frozen embedding vocabulary: the current vector $v$ is replaced by its nearest vocabulary embedding via a straight-through gradient copy, so both the layout-to-image denoiser $\\epsilon_\\theta$ and the prior layout denoiser $\\epsilon_\\phi$ backprop into $v$ while the optimized value is forced to decode to a legal box/class sequence. The prior layout diffusion model plays a necessary role: it supplies the non-uniform $p(y)$, since layouts, unlike classes, cannot be assumed uniformly distributed.","core_discovery":"The central discovery, stated on the paper's own terms, is that the discriminative information needed to detect objects is already present in a frozen layout-to-image diffusion model and can be extracted by optimizing the conditional input. Substituting the two diffusion training objectives into Bayes' rule gives a single inversion objective (Eq. 4): minimize the sum of the denoising error of the image model with respect to the layout and the denoising error of a separately trained layout-prior model. Because the layout is represented by discrete vocabulary embeddings, DIVE uses a nearest-embedding replacement with gradient copying, so the optimizer stays in the model's vocabulary and the final prediction is read off as nearest-neighbor indices. With this machinery the paper reports AP 7.1 versus 6.8 for Faster R-CNN R50 at the same low resolution, better AP75 than AP50 (indicating precise but incomplete detections), and a 14x speedup over enumeration for classification with nearly identical accuracy.","pith_inferences":["The objective in Eq. (4) simply adds two unweighted denoising losses; a natural extension is to learn or tune a weighting between the image-conditional and prior terms, and the resulting AP would reveal how much of the detection quality owes to the prior.","The poor small- and medium-object AP is attributed to 8x latent downsampling; testing the same inversion on a pixel-space or higher-resolution latent model would isolate that cause and might raise small-object AP without changing the algorithm.","Because DIVE's advantage over Faster R-CNN appears at AP75, the method may be most useful as a precise proposal generator or verifier rather than a full detector; combining DIVE candidates with a cheap discriminative ranker is a testable hybrid.","The same inversion could be pointed at dense tasks such as semantic segmentation, but the optimization space would explode and per-pixel gradient contributions would be small; coarse-to-fine or grouped optimization would be needed before that becomes feasible."],"forward_implications":["A frozen generative model can act as a zero-training detector: no detector head, no finetuning, and no enumeration of candidate labels are required.","The Bayes-inversion recipe transfers from classification to detection, so any conditional diffusion model with a differentiable condition encoder is a candidate discriminative model.","Detection quality tracks the model's generative prior: the paper's layout prior removes redundant objects and illegal boxes that appear when it is ablated.","DIVE offers a self-contained evaluation metric for conditional diffusion models that does not depend on external classifiers; its rankings agree with FID, with the caveat that diversity rather than realism tracks discriminative ability.","Optimization-based inversion is the main cost bottleneck; early stopping on the monitor loss could roughly halve the average 2000-step budget without, the authors argue, hurting performance."],"supporting_citations":[{"why":"Establishes the enumeration-based diffusion classifier that DIVE extends from classification to detection, and supplies the classification accuracy/speed baseline.","marker":"[1]"},{"why":"Defines the denoising diffusion objective whose per-sample MSE terms form the inversion loss in Eq. (1) and consequently in Eq. (4).","marker":"[4]"},{"why":"Provides the latent-diffusion architecture and layout-conditioning codebase that the paper retrains as the frozen layout-to-image model.","marker":"[7]"},{"why":"Supplies the Bayes-rule conversion $p(y|x) \\propto p(x|y)p(y)$ that turns the generative model into a discriminative predictor.","marker":"[30]"},{"why":"Defines COCO, the 500-image evaluation subset, and the AP metrics used in all detection experiments.","marker":"[31]"},{"why":"The basic Faster R-CNN R50 detector that DIVE is compared against and slightly exceeds at 256×256 resolution.","marker":"[32]"},{"why":"The class-conditional DiT model used in the ImageNet classification comparison, the speed test, and the generative-model evaluation table.","marker":"[65]"}],"fun_headline_variants":["Frozen diffusion model inverts to detect objects","Inverting layout-to-image diffusion for detection","Diffusion inversion matches Faster R-CNN on COCO","DIVE: Inverting diffusion for discriminative tasks","Detect objects by inverting a frozen diffusion model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that adding the conditional image model's denoising loss and the prior layout model's denoising loss equals the log-posterior in Bayes' rule; if that equality is only approximate or mis-weighted, the optimized layout is not a MAP estimate and the reported AP is not evidence of the frozen generator's discriminative power.","fun_headline_variants_meta":{"raw":{"variants":["Frozen diffusion model inverts to detect objects","Inverting layout-to-image diffusion for detection","Diffusion inversion matches Faster R-CNN on COCO","DIVE: Inverting diffusion for discriminative tasks","Detect objects by inverting a frozen diffusion model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1376,"prompt_tokens":895,"completion_tokens":481,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":407}},"tokens_in":511,"tokens_out":481,"duration_ms":4410,"temperature":1.0,"reasoning_tokens":407,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:44:38.285772+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Estimate the true conditional log-likelihoods of DIVE's predicted layouts with a proper likelihood estimator (for example, annealed importance sampling or an ELBO with many noise samples), and compare the ranking induced by the sum-of-MSE objective with the ranking induced by the estimated likelihoods on a set of correct and incorrect layouts; a mismatch would show the objective is not the posterior. A cheaper check is to re-run detection while scaling the prior loss by a free weight; if AP changes substantially with that weight, the unweighted combination in Eq. (4) is not the unique Bayes objective.","supporting_citations":[{"cited_title":"Scalable diffusion models with transformers","cited_arxiv_id":null,"evidence_quote":"The class-conditional DiT model used in the ImageNet classification comparison, the speed test, and the generative-model evaluation table."}],"review_version":1}