Compositional Scene Understanding through Inverse Generative Modeling

Justin Dauwels; Yanbo Wang; Yilun Du

REVIEW 4 major objections 6 minor 1 cited by

Compositional Scene Understanding through Inverse Generative Modeling

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Understanding a scene can be reduced to inverting a generative model: find the concepts whose summed denoisers best predict the image noise, and the search stays correct on scenes more complex than the training set.

desk verdict Solid compositional-inverse-generative-modeling paper with impressive OOD results on interaction-free benchmarks; the independence assumption is real, acknowledged, and bounds the scope of the headline claim. read the letter →

arxiv 2505.21780 v4 pith:2LSMNSTJ submitted 2025-05-27 cs.CV

Yanbo Wang , Justin Dauwels , Yilun Du This is my paper

classification cs.CV

keywords inversegenerativemodelingcompositionalmodelsdiffusionsceneunderstandingobjectdiscoveryzero-shotmulti-objectperceptionout-of-distributiongeneralizationdenoisingerrorinference

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes treating scene understanding as the inverse of image generation. Given an image, the goal is to find the set of visual concepts — object coordinates, facial attributes, object categories — whose composed generative model reproduces the image best, measured by one denoising error: how well the model predicts the noise that was added to the image. The central claim is that this inversion generalizes precisely because the generative model is built compositionally, by summing one separately conditioned denoiser per concept, so the same trained pieces can be recombined for more objects, new shapes, and new attributes than appeared in training. The paper demonstrates the claim on three tasks: locating objects in CLEVR scenes after training on images with only 3-5 objects, predicting three facial attributes after training on female faces only, and naming animal pairs in natural images using a pretrained text-to-image diffusion model with no additional training. A reader should care because the proposal ties perception accuracy to the same objective used to train generative models, so better generators should directly yield better scene understanding.

What carries the argument

The load-bearing object is the composed denoiser $\epsilon^{\mathrm{comb}}_\theta(x_t,t) = \sum_{k=1}^{K} \epsilon_\theta(x_t,t\,|\,c_k)$ (Eq. (4)): the sum of per-concept denoising networks, which approximates the score (the gradient of the log-likelihood) of the product distribution $p(x\,|\,c_1,\dots,c_K) \propto \prod_{k=1}^{K} p(x\,|\,c_k)$ (Eq. (2)). Training minimizes the denoising diffusion objective applied to the sum itself (Eq. (5)), so the composition is jointly calibrated rather than an ad-hoc test-time combination; inference then inverts it by minimizing the same objective over the concepts (Eq. (7)), and the number of concepts $K$ is chosen as the count with the smallest achievable error (Eq. (8)). Continuous concepts are optimized by stochastic gradient descent from many random starting points to escape local minima, while discrete concepts are handled by exhaustive enumeration or by a gradient-relaxed search that scales linearly in the number of concepts.

What would settle it

Render CLEVR-style scenes in which one object partially occludes another while keeping the layout otherwise identical to a non-occluded reference, and run the inference on both. If the coordinate error concentrates on the occluded object, or the inferred object count drifts, while a model trained with explicit pairwise interaction terms stays accurate, then the independence factorization in Eq. (2) is the load-bearing assumption, and the generalization claim is bounded by it.

Watch

Extended reading notes

Core claim

The paper's central claim is that scene understanding is an inverse generative modeling problem: given an image $x$, the correct set of visual concepts $\hat{c}_1,\dots,\hat{c}_K$ is the one minimizing the expected denoising error $\mathbb{E}_{\epsilon,t}\lVert \epsilon - \sum_{k=1}^{K}\epsilon_\theta(x_t, t\,|\,c_k)\rVert^2$ (Eq. (7)), where $\epsilon_\theta(x_t, t\,|\,c_k)$ is the noise prediction of a diffusion model conditioned on concept $c_k$ alone. Each concept's denoiser is a separate factor of a product-form generative model, and the sum is trained jointly so it behaves as one accurate composed denoiser (Eq. (5)) while remaining re-composable at test time with more terms than were ever seen together in training. Because the same inversion objective handles coordinates, attribute sets, and text prompts, and applies both to domain-trained diffusion models and to pretrained text-to-image models, the paper argues that a single framework covers object discovery, global attribute inference, and zero-shot multi-object perception, with the largest margins over baselines appearing exactly on out-of-distribution scenes.

Load-bearing premise

The method rests on the approximation that a scene's likelihood factorizes as the product of per-concept likelihoods, so when objects interact through occlusion, contact, shadows, or correlated attributes, the summed denoiser has no place to represent the interaction and will misassign or miss concepts, and the reported out-of-distribution gains may not transfer to scenes that do not respect the independence.

Editorial extensions

If this is right

Scene understanding needs no separate discriminative model: the same trained generative model, composed of per-concept denoisers, serves as its own detector and attribute predictor through inversion.
A model trained on CLEVR images with 3-5 objects reaches an 85.3% object perception rate on CLEVR scenes with 6-8 objects and 72.4% on CLEVRTex scenes with 6-8 objects, where the best baseline remains below 60%.
Trained on female faces only, the method predicts black hair, eyeglasses, and smiling jointly on male faces at 65.6% accuracy, the highest of all compared methods on that out-of-distribution split.
A pretrained text-to-image diffusion model becomes a zero-shot multi-object perceiver: composing single-object prompts and choosing the pair with the lowest denoising error reaches 87.3% accuracy on two-animal web images, versus 70.4% for the diffusion classifier baseline.
The number of objects in a scene is inferred by the model itself: the count $K$ giving the lowest average denoising error matches the ground-truth object number on CLEVR test images.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable consequence the paper leaves implicit: perception accuracy under this framework should track the quality of the underlying generator, so any improvement to the diffusion model — higher fidelity, more accurate conditioning — should transfer automatically to the inferred concepts, making 'better generator, better perceiver' a measurable prediction.
The concept-count rule is a likelihood-based model-selection criterion that uses the generative model as its own evaluator; the same rule could select relational structure, such as which pairs of objects interact, with the accuracy of the recovered structure as the yardstick.
Because each denoiser is trained on its concept's marginal appearance, the factorization's blind spot predicts a specific failure: a concept that is rare or correlated with another concept in training should be misinferred even when the test scene is in-distribution, a failure a correlated-attribute split on CelebA could expose.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the paper is worth a serious look. It gives a concrete recipe for turning a compositionally trained diffusion model into a multi-concept perception system, and the OOD results on interaction-free benchmarks are strong. The thing to remember is that the compositional factorization in Eq. (2) is the load-bearing assumption, and the authors say so themselves in the Limitations section.

What is actually new: prior work did test-time composition of separately trained denoisers; here they jointly train the summed denoiser (Eq. 5) and then invert it to infer concepts. They also add a likelihood-based criterion for choosing the number of concepts, and a practical continuous optimization scheme with multiple restarts for coordinate inference. The zero-shot multi-object perception demo with pretrained Stable Diffusion is a nice extra.

What works well: the experiments are clean and the margins are large. On CLEVR with 6-8 objects, IGM reaches 85.3% perception rate versus 58.7% for the best baseline; on CLEVRTex it is 72.4% versus 52.9%. The facial attribute gain is smaller (65.6% vs 62.2%) but consistent. Table IV is honest: performance improves steadily with the number of random initializations, which makes clear the method is sensitive to that hyperparameter.

The soft spot is exactly the one the stress-test flags. Eq. (7) is only justified when the joint distribution factors as a product of per-concept conditionals. The benchmarks avoid interactions: CLEVR objects are spatially separated, facial attributes are roughly independent, and the two-animal web images are mostly non-interacting. So the abstract's talk of 'robust generalization to new shapes' really needs the qualifier 'in interaction-free scenes.' The authors acknowledge this, but it leaves open whether the gains survive occlusion, contact, or relational semantics.

Minor issues: no error bars; the zero-shot evaluation is on 71 images; the concept-number criterion has no complexity penalty and is only illustrated on a single example, which is mildly surprising since larger K should overfit. The citation pattern is fine; the self-citations to prior compositional generation are appropriate because the method builds directly on them.

This is a paper for people working on generative perception, object discovery, and compositional generalization. It deserves a serious referee, not a desk reject. A good referee should ask for a stress test with interacting objects and for code and error bars.

In short: recommend engagement, with the independence assumption front and center.

Referee Report

4 major / 6 minor

Summary. The paper proposes an inverse generative modeling (IGM) framework for scene understanding. Given an image x, the method infers a set of visual concepts c_1,...,c_K (object coordinates, facial attributes, or object categories) by minimizing the denoising error against a compositionally trained diffusion model. The composition is built on the product factorization p(x|c_1,...,c_K) ∝ ∏_k p(x|c_k), and the composed denoiser is the sum of per-concept denoisers. At inference, Eq. (7) is optimized to recover the concepts, and the number of concepts K is selected by comparing average denoising errors across K. Experiments cover object discovery on CLEVR/CLEVRTex, facial attribute prediction on CelebA, and zero-shot multi-object perception with Stable Diffusion, reporting strong out-of-distribution improvements over generative and discriminative baselines.

Significance. If the central claim holds, the paper offers a unified and compositional way to turn generative models into scene-understanding engines, with generalization to more objects or novel attribute combinations than seen in training. The empirical contributions are concrete: Table 1 shows a large OOD margin for object discovery (85.3% vs 58.7% for GC on CLEVR with 6-8 objects), Table 2 shows consistent OOD gains on CelebA, and Table 3 shows 87.3% zero-shot multi-object perception accuracy with Stable Diffusion. The paper also provides ablations (Table IV) and appendices with algorithms and dataset details. The main weaknesses are theoretical: the likelihood surrogate in Eq. (7) is not the true ELBO, the concept-count criterion in Eq. (8) lacks a complexity penalty, and the compositional extrapolation to K=6-8 from training on K=3-5 is an assumption that is not analyzed.

major comments (4)

[Sec. 3.2, Eq. (8), Fig. 3] The procedure for selecting the number of concepts K minimizes the average denoising error over K without any complexity penalty. For a generative model with K additional conditioning arguments, the maximum of the (surrogate) likelihood is generally nondecreasing in K, so the claim that the ground-truth K always yields the lowest denoising error requires explicit support. Figure 3 shows one qualitative example and Algorithm 4 is described, but no quantitative accuracy, confusion matrix, or error bars for K selection are reported for the CLEVR/CLEVRTex test sets. Because object discovery requires unknown K, this is load-bearing. Please add a quantitative evaluation of K-selection accuracy, or derive and include a penalty term that justifies the criterion.
[Sec. 3.2, Eq. (7)] The inference objective is presented as an approximation of the variational lower bound on log p(x|c_1,...,c_K), but the diffusion ELBO contains t-dependent weighting terms that are discarded, and the expectation is replaced by a finite number of Monte Carlo samples. Minimizing the unweighted denoising error is not equivalent to maximum likelihood; it is a heuristic (used in prior generative classifiers) rather than the likelihood-based inversion claimed in the paper. Since Eq. (7) is the core inference objective, please either use the properly weighted ELBO or provide an explicit sensitivity analysis (e.g., varying the t-sampling distribution and the number of samples) demonstrating that the approximation does not drive the reported results.
[Sec. 3.1, Eq. (2), and Limitations] The composition rule rests on the conditional-independence factorization p(x|c_1,...,c_K) ∝ ∏_k p(x|c_k), which the Limitations section acknowledges ignores interactions between objects. The abstract's claim of robust generalization to scenes with more objects of new shapes is supported only for settings where objects are largely independent: CLEVR/CLEVRTex objects are spatially separated and the CelebA attributes are near-independent. Under occlusion, object contact, or shared illumination, the true joint distribution does not factor, and the minimizer of Eq. (7) can be biased. Please qualify the generalization claim to interaction-free settings, or add experiments with interacting objects to delineate where the method starts to fail.
[Sec. 3.1, Eq. (5), Table 1] The training objective Eq. (5) fits the composed denoiser only for scenes with K = 3-5 concepts, but the out-of-distribution evaluations use K = 6-8. At test time the same per-concept denoisers are summed for an unseen number of summands. The paper provides no argument that networks trained through Eq. (5) compose correctly outside the training range; this extrapolation is an implicit assumption. The headline OOD results depend on this assumption. Please add an analysis of how the composed denoising error degrades as K increases, or an ablation that varies the training K range and measures the resulting test generalization, so readers can see the boundary of the claimed compositional generalization.

minor comments (6)

[Throughout] Please fix typos: 'Hungrian' (Sec. 4.1), 'determinative' (Fig. 4 caption), 'exlusively' (Fig. 1), 'pre-trained' misspelled as 'pretained' (Impact Statement), 'in stead' (Sec. 3.3), and 'bbject' (Fig. XI).
[Limitations] The beam-search complexity statement reads 'from O(M K) to O(M K)'; this should presumably be 'from O(M^K) to O(MK)'. Please correct the formula.
[Algorithm 2] Line 13, '1/N E[j]' is ambiguous; it should be written as E[j]/N or N should be defined as the number of samples before accumulation.
[Appendix A.1] The choice between Eq. (2) and Eq. (9) is a per-dataset modeling decision ('we experiment with both approaches and selected the more effective one for each dataset'). This is an extra design choice that should be reported as a hyperparameter, with details of how the selection was made (e.g., on a validation split) to avoid test-set peeking.
[Table 3] The zero-shot multi-object perception evaluation uses only 71 manually collected images. Report confidence intervals or per-class accuracies so the 16.9% margin over Diffusion Classifier can be assessed statistically.
[Fig. 3 and Fig. XII] The claim that the ground-truth K consistently yields the lowest denoising error is supported only by visual examples; add quantitative error bars or a histogram for K selection across the test set.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the inference objective Eq. (7) is a variational likelihood approximation over fixed trained denoisers, not a fitted parameter; the compositional factorization is an acknowledged, empirically tested ansatz from self-cited prior work.

full rationale

The derivation chain is self-contained with respect to circularity. Equation (7) minimizes the denoising error over candidate concepts c_k while the model weights are fixed after training; this is the standard variational lower bound on log p(x|c1,...,cK), the same likelihood principle used by generative classifiers, and not a fitted parameter renamed as a prediction. The identity between the training objective Eq. (5) and the inference objective Eq. (7) is by design: the composed denoiser defines the model's likelihood, so selecting concepts with the lowest denoising error is maximum-likelihood inference, and the paper's test labels are never used to fit the model. The main self-citation is the product factorization Eq. (2), adopted from prior work by the same authors (Du & Kaelbling, 2024; Liu et al., 2022; Du et al., 2020); however, the paper explicitly calls it 'a biased approximation' and validates it on external benchmarks (CLEVR/CLEVRTex, CelebA female-to-male, and zero-shot Stable Diffusion), so the citation is not an unverified load-bearing uniqueness claim. The Limitations section itself flags that the full independence approximation 'ignores the interaction between objects, which are crucial in many real-world scenarios'; that is a genuine correctness risk for scenes with occlusion or relational structure, but it is a modeling limitation, not a circular derivation. No step reduces Eq. (7) to Eq. (5) by construction in a way that would make the predictions statistically forced; the score reflects only the minor self-cited ansatz underlying the compositional factorization.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The ledger is small: the paper postulates no new physical entities. It relies on the conditional-independence factorization, the score-model approximation, the unweighted denoising surrogate for likelihood, and the empirical claim that the true concept count minimizes denoising error. The main hand-chosen elements are inference hyperparameters and a per-dataset choice of whether to include the unconditional term p(x).

free parameters (3)

Number of random initializations R for continuous concept inference = 20 (ablation: 1, 5, 10, 15, 20)
Hand-chosen inference hyperparameter; Table IV shows perception rate improves monotonically with R, so the reported 94.7% and 85.3% depend on this choice.
Per-dataset modeling choice: include unconditional term p(x) in composed distribution = CLEVR: include; CelebA and zero-shot: exclude
Appendix A.1 says 'we experiment with both approaches and selected the more effective one for each dataset'. This is post hoc model selection and affects the composed denoiser in Eq. (11).
Inference hyperparameters (learning rate lambda, step count Nstep, evaluation samples Nsample) = Not reported
Required by Algorithms 3-6; without these values the experiments cannot be exactly reproduced from the text.

assumptions (5)

domain assumption Conditional independence of concepts given the image, p(x|c1,...,cK) proportional to product_k p(x|ck).
Eq. (2) and Appendix A.1; explicitly acknowledged in Limitations: 'assumes object concept independence given the input image', ignoring interactions.
standard math The denoising function epsilon_theta(x_t,t|ck) approximates the score gradient of the EBM, so summing denoisers approximates the composed EBM score.
Section 3.1, standard score-based diffusion theory; reasonable but approximate.
domain assumption Unweighted denoising error is a valid surrogate for log-likelihood (ignoring weighting terms in the variational bound).
Section 3.2, following Li et al. 2023a; known approximation with no exactness guarantee for concept inference.
ad hoc to paper Training the sum of per-concept denoisers with the denoising objective Eq. (5) yields a valid composed score for novel concept counts at test time.
Section 3.1; asserted to enable generalization, but no theorem is provided and support is empirical only.
ad hoc to paper The concept count K that minimizes average denoising error is the true object count.
Section 3.2 and Figure 3; no penalty for larger K is derived, yet the paper reports ground truth K gives the lowest error.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Compositional Scene Understanding through Inverse Generative Modeling." pith.science (2026). https://pith.science/paper/2LSMNSTJ

@misc{pith2026250521780,
  author       = {Pith},
  title        = {Pith review of: Compositional Scene Understanding through Inverse Generative Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2LSMNSTJ}},
  note         = {Machine review of arXiv:2505.21780}
}

read the original abstract

Generative models have demonstrated remarkable abilities in generating high-fidelity visual content. In this work, we explore how generative models can further be used not only to synthesize visual content but also to understand the properties of a scene given a natural image. We formulate scene understanding as an inverse generative modeling problem, where we seek to find conditional parameters of a visual generative model to best fit a given natural image. To enable this procedure to infer scene structure from images substantially different than those seen during training, we further propose to build this visual generative model compositionally from smaller models over pieces of a scene. We illustrate how this procedure enables us to infer the set of objects in a scene, enabling robust generalization to new test scenes with an increased number of objects of new shapes. We further illustrate how this enables us to infer global scene factors, likewise enabling robust generalization to new scenes. Finally, we illustrate how this approach can be directly applied to existing pretrained text-to-image generative models for zero-shot multi-object perception. Code and visualizations are at https://energy-based-model.github.io/compositional-inference.

Figures

Figures reproduced from arXiv: 2505.21780 by the authors.

**Figure 1.** Compositional Scene Understanding. Our approach demonstrates strong generalization across various scene understanding tasks. For object location inference (first row), the model is trained on CLEVR images containing 3-5 objects, while the test set is CLEVRTex, which contains 6-8 objects. For multi-facial attribute inference (second row), the model is trained only on female faces from CelebA, and is tested exlusive… view at source ↗

**Figure 2.** Compositional Scene Understanding. Our model achieves scene understanding by identifying the optimal conditioning concepts (e.g., facial attributes or object coordinates) that best interpret the input test image. Its compositional structure allows for simultaneous inference of multiple concepts and enables robust generalization to images that differ substantially from the training data. Algorithm 1 Training Algorith… view at source ↗

**Figure 3.** Concept Number Inference. Illustration of object number inference on CLEVR. Given a test image, our model evaluates each number K ∈ {3, .., 8} respectively by using K objects to fit the image and obtain corresponding denoising errors. Out of the potential options K ∈ {3, , .., 8}, our model determines the one with the lowest denoising error as object number, which turns out to be consistent with the ground truth num… view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: In-distribution Object Discovery. We train our model with CLEVR images containing 3-5 objects. During inference, given an in-distribution image (also containing 3-5 objects), our approach accurately identifies object coordinates. Compared with both determinative and ge…

**Figure 5.** Figure 5: Out-of-distribution Object Discovery. Object perception results on out-of-distribution images: CLEVR images with 6-8 objects (Left) or CLEVRTex images with 6-8 objects (Right). Our model is trained with CLEVR images containing 3-5 objects. During inference time, given …

**Figure 6.** Figure 6: In-Distribution and Out-of-Distribution Facial Feature Prediction. Facial feature prediction results for in-distribution (Left) and out-of-distribution (Right) CelebA images. Our model is trained on female faces from CelebA. During inference, our model can accurately p…

**Figure 7.** Figure 7: Zero-Shot Multi-Object Perception. Our approach can faithfully interpret given real-world images by predicting object categories that are consistent with the ground truth. Models Accuracy ↑ Diffusion Classifer (Li et al., 2023a) 70.4% Diffusion Classifer Variant 73.2% …

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

CARV: A Diagnostic Benchmark for Compositional Analogical Reasoning in Multimodal LLMs
cs.AI 2026-03 conditional novelty 5.0 of 10

A new diagnostic benchmark shows frontier multimodal LLMs fail at composing visual transformation rules from multiple image pairs, with the bottleneck in decomposing visual changes into symbolic rules.

Reference graph

Works this paper leans on

70 extracted references · 32 canonical work pages · cited by 1 Pith paper

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...
[3]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...
[4]

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

work page
[5]

Backpropagation and stochastic gradient descent method

Amari, S.-i. Backpropagation and stochastic gradient descent method. Neurocomputing, 5 0 (4-5): 0 185--196, 1993

work page 1993
[6]

Segdiff: Image segmentation with diffusion probabilistic models

Amit, T., Shaharbany, T., Nachmani, E., and Wolf, L. Segdiff: Image segmentation with diffusion probabilistic models. arXiv preprint arXiv:2112.00390, 2021

arXiv 2021
[7]

Break-a-scene: Extracting multiple concepts from a single image

Avrahami, O., Aberman, K., Fried, O., Cohen-Or, D., and Lischinski, D. Break-a-scene: Extracting multiple concepts from a single image. In SIGGRAPH Asia 2023 Conference Papers, pp.\ 1--12, 2023

work page 2023
[8]

Towards compositional understanding of the world by agent-based deep learning

Bengio, Y. Towards compositional understanding of the world by agent-based deep learning. In NeurIPS’2019 Workshop on Context and Compositionality in Biological and Artificial Neural Networks, 2019

work page 2019

Show all 70 references

[9]

Recognition-by-components: a theory of human image understanding

Biederman, I. Recognition-by-components: a theory of human image understanding. Psychological review, 94 0 (2): 0 115, 1987

1987
[10]

A., Kornblith, S., Chen, T., Parmar, N., Minderer, M., and Norouzi, M

Brempong, E. A., Kornblith, S., Chen, T., Parmar, N., Minderer, M., and Norouzi, M. Denoising pretraining for semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 4175--4186, 2022

2022
[11]

Your diffusion model is secretly a certifiably robust classifier

Chen, H., Dong, Y., Shao, S., Hao, Z., Yang, X., Su, H., and Zhu, J. Your diffusion model is secretly a certifiably robust classifier. arXiv preprint arXiv:2402.02316, 2024

2024 arXiv
[12]

K., Lu, J., Inouye, D

Cho, W., Ravi, H., Harikumar, M., Khuc, V., Singh, K. K., Lu, J., Inouye, D. I., and Kale, A. Towards enhanced controllability of diffusion models, 2023. URL https://arxiv.org/abs/2302.14368

2023 arXiv
[13]

Aspects of the Theory of Syntax

Chomsky, N. Aspects of the Theory of Syntax. The MIT Press, Cambridge, 1965

1965
[14]

and Jaini, P

Clark, K. and Jaini, P. Text-to-image diffusion models are zero shot classifiers. Advances in Neural Information Processing Systems, 36, 2024

2024
[15]

R., Li, L

Cong, Y., Min, M. R., Li, L. E., Rosenhahn, B., and Yang, M. Y. Attribute-centric compositional text-to-image generation. arXiv preprint arXiv:2301.01413, 2023

2023 arXiv
[16]

and Kaelbling, L

Du, Y. and Kaelbling, L. Compositional generative modeling: A single model is not all you need. arXiv preprint arXiv:2402.01103, 2024

2024 arXiv
[17]

and Mordatch, I

Du, Y. and Mordatch, I. Implicit generation and generalization in energy-based models. arXiv preprint arXiv:1903.08689, 2019

1903 arXiv
[18]

Compositional visual generation with energy based models

Du, Y., Li, S., and Mordatch, I. Compositional visual generation with energy based models. In Advances in Neural Information Processing Systems, 2020

2020
[19]

J., and Mordatch, I

Du, Y., Li, S., Sharma, Y., Tenenbaum, B. J., and Mordatch, I. Unsupervised learning of compositional energy concepts. In Advances in Neural Information Processing Systems, 2021

2021
[20]

B., Dieleman, S., Fergus, R., Sohl-Dickstein, J., Doucet, A., and Grathwohl, W

Du, Y., Durkan, C., Strudel, R., Tenenbaum, J. B., Dieleman, S., Fergus, R., Sohl-Dickstein, J., Doucet, A., and Grathwohl, W. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc. arXiv preprint arXiv:2302.11552, 2023

2023 arXiv
[21]

E., and Wang, W

Feng, W., He, X., Fu, T.-J., Jampani, V., Akula, A., Narayana, P., Basu, S., Wang, X. E., and Wang, W. Y. Training-free structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212.05032, 2022

2022 arXiv
[22]

Fodor, J. A. and Lepore, E. The compositionality papers. Oxford University Press, 2002

2002
[23]

Fodor, J. A. and Pylyshyn, Z. W. Connectionism and cognitive architecture: A critical analysis. Cognition, 28 0 (1-2): 0 3--71, 1988

1988
[24]

H., Chechik, G., and Cohen-Or, D

Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-Or, D. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022

2022 arXiv
[25]

H., Chechik, G., and Cohen-Or, D

Gal, R., Arar, M., Atzmon, Y., Bermano, A. H., Chechik, G., and Cohen-Or, D. Encoder-based domain tuning for fast personalization of text-to-image models. ACM Transactions on Graphics (TOG), 42 0 (4): 0 1--13, 2023

2023
[26]

A., and Brendel, W

Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., and Brendel, W. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12231, 2018

2018 arXiv
[27]

Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2 0 (11): 0 665--673, 2020

2020
[28]

On the binding problem in artificial neural networks

Greff, K., Van Steenkiste, S., and Schmidhuber, J. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208, 2020

2012 arXiv
[29]

Deep residual learning for image recognition

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

2016
[30]

and Gimpel, K

Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136, 2016

2016 arXiv
[31]

Hinton, G. E. To recognize shapes, first learn to generate images. Progress in brain research, 165: 0 535--547, 2007

2007
[32]

Denoising diffusion probabilistic models

Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33: 0 6840--6851, 2020

2020
[33]

Composer: Creative and controllable image synthesis with composable conditions

Huang, L., Chen, D., Liu, Y., Shen, Y., Zhao, D., and Zhou, J. Composer: Creative and controllable image synthesis with composable conditions. arXiv preprint arXiv:2302.09778, 2023

2023 arXiv
[34]

Intriguing properties of generative classifiers

Jaini, P., Clark, K., and Geirhos, R. Intriguing properties of generative classifiers. arXiv preprint arXiv:2309.16779, 2023

2023 arXiv
[35]

Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., and Girshick, R. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 29...

2017
[36]

Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation

Karazija, L., Laina, I., and Rupprecht, C. Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation. arXiv preprint arXiv:2111.10265, 2021

2021 arXiv
[37]

Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012

2012
[38]

Kuhn, H. W. The hungarian method for the assignment problem. Naval research logistics quarterly, 2 0 (1-2): 0 83--97, 1955

1955
[39]

C., Prabhudesai, M., Duggal, S., Brown, E., and Pathak, D

Li, A. C., Prabhudesai, M., Duggal, S., Brown, E., and Pathak, D. Your diffusion model is secretly a zero-shot classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 2206--2217, 2023 a

2023
[40]

C., Kumar, A., and Pathak, D

Li, A. C., Kumar, A., and Pathak, D. Generative classifiers avoid shortcut solutions. In ICML 2024 Workshop on Structured Probabilistic Inference \ & \ Generative Modeling , 2024

2024
[41]

Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp.\ 19730--19742. PMLR, 2023 b

2023
[42]

B., Torralba, A., and Mordatch, I

Li, S., Du, Y., Tenenbaum, J. B., Torralba, A., and Mordatch, I. Composing ensembles of pre-trained models via iterative consensus. arXiv preprint arXiv:2210.11522, 2022

2022 arXiv
[43]

Learning to compose visual relations

Liu, N., Li, S., Du, Y., Tenenbaum, J., and Torralba, A. Learning to compose visual relations. Advances in Neural Information Processing Systems, 34: 0 23166--23178, 2021

2021
[44]

Liu, N., Li, S., Du, Y., Torralba, A., and Tenenbaum, J. B. Compositional visual generation with composable diffusion models. arXiv preprint arXiv:2206.01714, 2022

2022 arXiv
[45]

B., and Torralba, A

Liu, N., Du, Y., Li, S., Tenenbaum, J. B., and Torralba, A. Unsupervised compositional concepts discovery with text-to-image generative models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 2085--2095, 2023

2023
[46]

Deep learning face attributes in the wild

Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp.\ 3730--3738, 2015

2015
[47]

Object-centric learning with slot attention, 2020

Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. Object-centric learning with slot attention, 2020

2020
[48]

Compositional risk minimization

Mahajan, D., Pezeshki, M., Mitliagkas, I., Ahuja, K., and Vincent, P. Compositional risk minimization. arXiv preprint arXiv:2410.06303, 2024

2024 arXiv
[49]

Few-shot task learning through inverse generative modeling

Netanyahu, A., Du, Y., Bronars, A., Pari, J., Tenenbaum, J., Shu, T., and Agrawal, P. Few-shot task learning through inverse generative modeling. arXiv preprint arXiv:2411.04987, 2024

2024 arXiv
[50]

and Jordan, M

Ng, A. and Jordan, M. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. Advances in neural information processing systems, 14, 2001

2001
[51]

Controllable and compositional generation with latent-space energy-based models

Nie, W., Vahdat, A., and Anandkumar, A. Controllable and compositional generation with latent-space energy-based models. Advances in Neural Information Processing Systems, 34: 0 13497--13510, 2021

2021
[52]

Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pp.\ 5389--5400

Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pp.\ 5389--5400. PMLR, 2019

2019
[53]

You only look once: Unified, real-time object detection

Redmon, J. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016

2016
[54]

High-resolution image synthesis with latent diffusion models, 2022

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models, 2022

2022
[55]

U-net: Convolutional networks for biomedical image segmentation

Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention--MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, ...

2015
[56]

u gelgen, J., Tr \

Schott, L., Von K \"u gelgen, J., Tr \"a uble, F., Gehler, P., Russell, C., Bethge, M., Sch \"o lkopf, B., Locatello, F., and Brendel, W. Visual representation learning does not generalize strongly within the same domain. arXiv preprint arXiv:2107.08221, 2021

2021 arXiv
[57]

Bridging the gap to real-world object-centric learning

Seitzer, M., Horn, M., Zadaianchuk, A., Zietlow, D., Xiao, T., Simon-Gabriel, C.-J., He, T., Zhang, Z., Sch \"o lkopf, B., Brox, T., et al. Bridging the gap to real-world object-centric learning. arXiv preprint arXiv:2209.14860, 2022

2022 arXiv
[58]

Detecting and recovering sequential deepfake manipulation

Shao, R., Wu, T., and Liu, Z. Detecting and recovering sequential deepfake manipulation. In European Conference on Computer Vision, pp.\ 712--728. Springer, 2022

2022
[59]

Shepard, R. N. and Metzler, J. Mental rotation of three-dimensional objects. Science, 171 0 (3972): 0 701--703, 1971

1971
[60]

Shi, C., Ni, H., Li, K., Han, S., Liang, M., Mishne, G., and Min, M. R. Compositional image generation and manipulation with latent diffusion models. 2023

2023
[61]

Deep unsupervised learning using nonequilibrium thermodynamics

Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pp.\ 2256--2265. PMLR, 2015

2015
[62]

Learning disentangled prompts for compositional image synthesis

Sohn, K., Shaw, A., Hao, Y., Zhang, H., Polania, L., Chang, H., Jiang, L., and Essa, I. Learning disentangled prompts for compositional image synthesis. arXiv preprint arXiv:2306.00763, 2023

2023 arXiv
[63]

B., and Du, Y

Su, J., Liu, N., Wang, Y., Tenenbaum, J. B., and Du, Y. Compositional image decomposition with diffusion models. arXiv preprint arXiv:2406.19298, 2024

2024 arXiv
[64]

Measuring robustness to natural distribution shifts in image classification

Taori, R., Dave, A., Shankar, V., Carlini, N., Recht, B., and Schmidt, L. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33: 0 18583--18599, 2020

2020
[65]

N., Vapnik, V., et al

Vapnik, V. N., Vapnik, V., et al. Statistical learning theory. 1998

1998
[66]

Hierarchical open-vocabulary universal image segmentation

Wang, X., Li, S., Kallidromitis, K., Kato, Y., Kozuka, K., and Darrell, T. Hierarchical open-vocabulary universal image segmentation. Advances in Neural Information Processing Systems, 36, 2024

2024
[67]

Slot-vae: Object-centric scene generation with slot attention

Wang, Y., Liu, L., and Dauwels, J. Slot-vae: Object-centric scene generation with slot attention. In International Conference on Machine Learning, pp.\ 36020--36035. PMLR, 2023

2023
[68]

Compositional generalization from first principles

Wiedemer, T., Mayilvahanan, P., Bethge, M., and Brendel, W. Compositional generalization from first principles. Advances in Neural Information Processing Systems, 36: 0 6941--6960, 2023

2023
[69]

Unleashing text-to-image diffusion models for visual perception

Zhao, W., Rao, Y., Liu, Z., Liu, B., Zhou, J., and Lu, J. Unleashing text-to-image diffusion models for visual perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 5729--5739, 2023

2023
[70]

Robodreamer: Learning compositional world models for robot imagination

Zhou, S., Du, Y., Chen, J., Li, Y., Yeung, D.-Y., and Gan, C. Robodreamer: Learning compositional world models for robot imagination. arXiv preprint arXiv:2404.12377, 2024

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

@esa (Ref

\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

[3] [3]

\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

[4] [4]

@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...

work page

[5] [5]

Backpropagation and stochastic gradient descent method

Amari, S.-i. Backpropagation and stochastic gradient descent method. Neurocomputing, 5 0 (4-5): 0 185--196, 1993

work page 1993

[6] [6]

Segdiff: Image segmentation with diffusion probabilistic models

Amit, T., Shaharbany, T., Nachmani, E., and Wolf, L. Segdiff: Image segmentation with diffusion probabilistic models. arXiv preprint arXiv:2112.00390, 2021

arXiv 2021

[7] [7]

Break-a-scene: Extracting multiple concepts from a single image

Avrahami, O., Aberman, K., Fried, O., Cohen-Or, D., and Lischinski, D. Break-a-scene: Extracting multiple concepts from a single image. In SIGGRAPH Asia 2023 Conference Papers, pp.\ 1--12, 2023

work page 2023

[8] [8]

Towards compositional understanding of the world by agent-based deep learning

Bengio, Y. Towards compositional understanding of the world by agent-based deep learning. In NeurIPS’2019 Workshop on Context and Compositionality in Biological and Artificial Neural Networks, 2019

work page 2019