Pith. sign in

REVIEW 5 major objections 5 minor 23 references

Successes and Limitations of Object-centric Models at Compositional Generalisation

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Object-centric models can recombine familiar object properties into novel combinations that defeat standard disentangled models, and can even reconstruct entirely novel shapes when all local image features were seen during training.

desk verdict A clean existence proof that object-centric models can recombine properties on 3DShapes, with a plausible but under-tested attribution to local feature availability. read the letter →

arxiv 2412.18743 v1 pith:4XWCIUPU submitted 2024-12-25 cs.CV cs.AIcs.LG

Milton L. Montero , Jeffrey S. Bowers , Gaurav Malhotra This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords object-centricmodelsSlotAttentioncompositionalgeneralizationdisentanglementrecombination-to-rangefigure-groundsegmentationpentominodatasetextrapolation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether object-centric generative models, which first segment an image into objects, can generalise to novel combinations of an object's own properties, such as a familiar shape in a colour or rotation that was never seen with it. The authors show that a Slot Attention model succeeds on recombination-to-range tasks for shape and colour in 3DShapes, and that a stripped-down one-slot variant (FgSeg) succeeds on shape-rotation combinations in a new pentomino dataset where standard disentangled autoencoders and a Wasserstein auto-encoder fail badly. They argue the earlier failures of disentangled models arose not from a missing compositional mechanism but from training data missing the local pixel features needed to reconstruct a rotated shape. When those local features are shared across shapes, as in the pentominoes, the model even extrapolates to completely novel shapes. The paper also documents a limitation: the learned representations are not abstract, since a linear probe cannot decode shape and nonlinear probes fail on held-out combinations.

What carries the argument

The load-bearing mechanism is the object-centric bottleneck: Slot Attention and its simplified one-slot version FgSeg, which forces the latent to reconstruct only the foreground figure through a Sigmoid rather than a Softmax slot competition. Position embeddings let the model track how patches move under rotation, and the per-object slot decoder reinforces grouping of patches that should be manipulated together. The pentomino dataset is the control: every shape is built from the same three low-level features (straight lines, convex right angles, concave right angles), so excluded shape-rotation combinations never contain unseen local patches. The Wasserstein auto-encoder serves as the baseline that isolates the contribution of this grouping bias.

What would settle it

Measure the set of local image patches actually present in the pentomino training set and in the test set for excluded rotations: if any excluded rotation contains an oriented edge or corner patch that never appears in training, the paper's claim that 'low-level features are not novel' is false. A complementary test: train FgSeg on a pentomino variant whose rotations introduce curved or diagonal features and check whether generalisation to excluded shape-rotation combinations collapses.

Watch

Extended reading notes

Core claim

The central claim is that object-centric models with figure-ground segmentation can perform compositional generalisation over object properties, a setting where standard disentangled generative models fail. On 3DShapes, Slot Attention reconstructs pills in colours that were never paired with the pill shape during training. On dSprites, the same model partially handles hearts in novel rotations, and the authors attribute the residual failure to novel local features introduced by rotation in image space. To test this diagnosis, they introduce a pentomino dataset whose twelve shapes are all composed of straight lines and right angles, so a novel rotation only rearranges already-seen features. FgSeg, a single-slot attention model, reconstructs held-out shape-rotation combinations and, when up to three of twelve shapes are withheld, reconstructs entirely novel shapes from the shared local features. A Wasserstein auto-encoder control fails on the same conditions, supporting the conclusion that the success is due to the object-centric bottleneck rather than dataset ease.

Load-bearing premise

The load-bearing premise is that rotating a pentomino in rasterised image space never creates novel local pixel features, a claim the paper states informally and never measures; if aliasing or rasterisation produces unseen corners or edge patterns, the pentomino success could be explained by dataset ease rather than by the model's perceptual grouping.

Editorial extensions

If this is right

  • Object-centric inductive biases, rather than stronger symbolic or invariant priors, are sufficient for recombination-to-range generalisation over object properties.
  • Failures of disentangled VAEs at this task are best explained by missing figure-ground segmentation, not by insufficient disentanglement.
  • Ensuring that all local pixel features appear during training can turn a previously unsolvable generalisation condition into a solvable one.
  • Extrapolation to unseen shapes is achievable when a model has learned to recombine a small set of shared local features.
  • The representations learned by these models are not abstract concepts: linear probes fail, and nonlinear probes do not transfer to held-out combinations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the diagnosis is correct, a dataset's local-feature coverage, not the model's inductive bias alone, determines when compositional generalisation is possible; one can test this by computing the novelty of oriented edge patches in any visual dataset before predicting generalisation success.
  • The pentomino result suggests object-centric models could serve as generative part-based synthesizers, but the probe results imply the generated output does not come with an explicit, reusable shape label, so generative competence and conceptual knowledge remain decoupled.
  • A natural extension is to apply the recombination-to-range test to natural-image object crops while controlling for local feature novelty, predicting that Slot Attention succeeds when held-out combinations reuse seen local patches and fails otherwise.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper investigates whether object-centric autoencoders (SlotAttention and a one-slot variant FgSeg) support compositional generalisation over object properties, extending prior work on scene composition. On 3DShapes, SlotAttention reconstructs excluded combinations of pill shape and colour; on dSprites it partially fails for novel heart rotations. To localise the failure, the authors introduce a Pentomino dataset of twelve block shapes and show that FgSeg generalises to novel shape–rotation combinations and even to completely held-out shapes, whereas a Wasserstein autoencoder fails. They attribute success to figure-ground grouping plus the availability of all local pixel features during training, and report probe experiments suggesting that the latent representations are not readily abstract.

Significance. The main contribution is a set of positive existence results plus a new diagnostic dataset. If the results hold, they significantly extend the evidence that object-centric grouping helps combinatorial generalisation beyond scene composition, and they give a concrete hypothesis about the role of local feature availability. The paper is honest about limitations (single seed, uninformative latent visualisations) and includes a baseline control. However, the central explanatory claim—that local-feature availability is the source of the success—is not yet tested, and the quantitative support for the extrapolation claim is statistically thin. The strengths are the clear qualitative demonstrations and the introduction of a controlled dataset; the weak points are the unmeasured rotation-aliasing assumption and the absence of ablations.

major comments (5)
  1. [§2.1, Fig. 2] The claim that rotating pentominoes in rasterised image space does not create novel local pixel features is asserted rather than measured. Anti-aliasing and resampling at arbitrary angles can produce subpixel patterns not present in the training set; this is precisely the quantity on which the paper's explanation rests. Please either quantify the novelty of local patches between training and test rotations or run a control in which rotations are generated by the same rendering style but for curved shapes, to support the attribution.
  2. [Appendix A] All quantitative claims rest on a single seed per experiment, and Tables 1–3 report no variance. The statement in Section 2.3 that there is 'no significant drop' from one to three novel shapes is not statistically supported (Table 3: 1.65 vs 1.29 train; 2.47 vs 2.63 test). Please report multiple seeds and error bars, or explicitly mark the extrapolation result as a single-run qualitative demonstration.
  3. [§2.1, §2.2] The claim that SlotAttention's position embeddings and per-object decoder are the source of its compositional skills is stated as if established ('it is also easy to pinpoint'), but no ablation manipulates these components. FgSeg is itself a simplification; the contribution of each component remains untested. An ablation removing position embeddings or replacing the per-object decoder with a monolithic one would be needed to support the causal attribution.
  4. [§3 vs Appendix D] The Discussion says the probe results 'suggest that the models learn abstract representations,' while Appendix D concludes that because linear and non-linear probes fail on held-out combinations, 'the model's representations are not abstract after all.' This contradiction should be resolved; the current text overstates the probe evidence.
  5. [Main text and appendices] The manuscript does not specify model architectures, hyperparameters, training budgets, or optimisation details for SlotAttention, FgSeg, or the WAE baseline. The Pentomino experiments are said to use 'the same architecture and training configuration' as dSprites, but that configuration is not given anywhere. Without these details the quantitative comparisons cannot be independently assessed.
minor comments (5)
  1. [Table 4] The Scale row lists six values (1.5, 1.8, 2.1, 2.4, 2.7, 3.0) while the text says five values, and the stated total of 960,000 images implies five values; please correct the table.
  2. [Appendix B, Eq. (1)] Equation (1) does not define a normalised sampling distribution: summing the stated probabilities over all training images yields the number of shape classes, not one. Please clarify the reweighting procedure.
  3. [Throughout] There are several typographical errors, including 'Sciecne' in the affiliation, 'this area this area', 'generalsation', 'preeliminary', 'traning', 'angels', and 'dDsprites' in Appendix A.
  4. [§2.1] The text refers to 'Figure 5, panel b' when discussing low-level features, but the relevant figure in the main text is Figure 2; the reference should be updated.
  5. [Appendix D] The phrase 'An even these models cannot achieve significant prediction accuracy' is garbled; also, the appendix reports a two-layer MLP, so the sentence 'we need at least a two-layer MLP' should be checked for consistency.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the headline generalization results are genuine held-out reconstructions benchmarked against a WAE baseline; remaining concerns are unmeasured stimulus assumptions, not definitional circularity.

full rationale

The central claim that FgSeg/SlotAttention can compositionally generalize on 3DShapes and Pentomino is supported by held-out test reconstructions in Figures 1, 3, 6 and 8 and Tables 1-3. It is not manufactured by fitting: the WAE baseline fails on the same conditions, so success is not forced by the training input by construction. The R2R conditions are inherited from the authors' prior work (Montero et al. 2022), but that prior result functions as an external benchmark rather than as a premise that defines the current outcome. The Pentomino dataset was designed so that all low-level geometric primitives (straight lines, convex and concave right angles) are shared across shapes, and the paper explicitly says 'we can be more confident that the low-level features are not novel' rather than proving it. This is an unmeasured empirical assumption about rasterized rotations (aliasing can create novel local pixel patterns), and Appendix A acknowledges the single-seed limitation; these are threats to the source-of-skill attribution but not circular derivation. No parameter is fitted to the target test set and renamed a prediction, no uniqueness theorem is imported from the authors' own work to force the model choice, and no known result is merely relabeled. The only self-citational element is the reuse of the R2R framing, which is not load-bearing in the sense of determining the reported successes, so the paper receives a low score reflecting minor self-citation rather than substantive circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central empirical results rest on the architecture and dataset assumptions listed above. The most consequential is the unmeasured local-feature invariance in pixel space, because the Pentomino experiment is designed to adjudicate that hypothesis. No fitted parameters are used to manufacture the outcome; the free parameters listed are experimental design choices.

free parameters (3)
  • Pentomino generative factor ranges = 12 shapes; stated as 5 scale values but 6 listed (1.5, 1.8, 2.1, 2.4, 2.7, 3.0); 40 rotation values; 20 X and 20 Y…
    Hand-chosen dataset design parameters that define the compositional generalization test. The internal inconsistency in the scale count is a factual issue in the paper.
  • Excluded shape-rotation combinations = [F, P, T, W] with rotation > 180 for the shape-rotation condition; [P, T, W] and [F, P, N, T, V, W] for extrapolation
    The excluded shapes were selected in Appendix C.1 to include both confusable and distinct shapes, which can modulate task difficulty and is a hand choice rather than a neutral random split.
  • One seed per experiment = single seed
    Appendix A discloses that only one seed was run after preliminary tests showed no significant difference. This is an experimental design choice that limits variance estimates and statistical support.
assumptions (4)
  • domain assumption Rotating pentominoes at the 40 chosen angles does not introduce novel low-level pixel features.
    Section 2.1 states 'we can be more confident that the low-level features are not novel', but the property is asserted, not measured; the causal diagnosis depends on it.
  • ad hoc to paper SlotAttention's position embeddings and per-object decoder are the source of generalization.
    Section 2 says 'it is easy to pinpoint to why this can be the case', with no ablation removing position embeddings or changing the decoder to test this attribution.
  • domain assumption FgSeg, a one-slot variant with a Sigmoid decoder, is a faithful proxy for full SlotAttention.
    Section 2.1 introduces FgSeg for analysis, but the paper does not compare FgSeg to full SlotAttention on the Pentomino conditions.
  • domain assumption The ground-truth generative factors in 3DShapes, dSprites, and Pentomino correctly define the compositional structure being tested.
    Standard dataset assumption; the paper relies on these factors to define train/test splits in Appendix B.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Successes and Limitations of Object-centric Models at Compositional Generalisation." pith.science (2026). https://pith.science/paper/4XWCIUPU

@misc{pith2026241218743,
  author       = {Pith},
  title        = {Pith review of: Successes and Limitations of Object-centric Models at Compositional Generalisation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4XWCIUPU}},
  note         = {Machine review of arXiv:2412.18743}
}
read the original abstract

In recent years, it has been shown empirically that standard disentangled latent variable models do not support robust compositional learning in the visual domain. Indeed, in spite of being designed with the goal of factorising datasets into their constituent factors of variations, disentangled models show extremely limited compositional generalisation capabilities. On the other hand, object-centric architectures have shown promising compositional skills, albeit these have 1) not been extensively tested and 2) experiments have been limited to scene composition -- where models must generalise to novel combinations of objects in a visual scene instead of novel combinations of object properties. In this work, we show that these compositional generalisation skills extend to this later setting. Furthermore, we present evidence pointing to the source of these skills and how they can be improved through careful training. Finally, we point to one important limitation that still exists which suggests new directions of research.

Figures

Figures reproduced from arXiv: 2412.18743 by the authors.

Figure 1
Figure 1. Slot Attention generalisation results: Reconstructions for a model when trained on all but some combinations of generative factors. The model is tested on said excluded combinations. Left) Generalisation results when excluding half of the combinations of the colors with the pill shape in 3DShapes. Right) Analogous test on dSprites where we exclude half of the rotations of the heart. This has left the problem of comp… view at source ↗
Figure 2
Figure 2. Pentomino shapes a) The twelve Pentomino shapes and their names2 . We construct the dataset by performing affine transformations of these shapes: 5 values of scale, 40 values of rotation and 20 values of translation along each of the X and Y axis. b) The low-level features that comprise the different shapes. From top to bottom: straight lines, convex right angles and concave right angles. We test a SlotAttention (SA… view at source ↗
Figure 3
Figure 3. Generalization to novel shape and rotation combinations in the Pentomino dataset. Generalization reconstructions for both FgSeg and a WAE control model. The models where trained on 11 of the 12 Pentomino shapes and tested at reconstructing a held out one in different configurations of position, rotation and scale. Softmax to decide which slot is responsible for reconstructing a particular pixel, we use a Sigmoid act… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: New shape extrapolation On the left, Slot Attention reconstructions of a novel shape, in this case the W. Left to right, different values of rotation sampled uniformly over the whole range of values [0, 360) can be seen. On the right, the same results for WAE. It is cl…
Figure 5
Figure 5. Figure 5: Pentomino shapes a) The twelve pentomino shapes and their names4 . We construct the dataset by performing affine transformations of these shapes: 5 values of scale, 40 values of rotation and 20 values of translation along each of the X and Y axis. b) The low-level feat…
Figure 6
Figure 6. Figure 6: Generalization to novel shape and rotation combinations in the Pentomino dataset. Generalization reconstructions for both FgSeg and a WAE control model. Example rotations of the excluded shape are plotted inside their circumference according to their rotation value. Sa…
Figure 7
Figure 7. Figure 7: Extrapolation to three new shapes. Figure-ground Segmentation model reconstructions when three shapes are excluded from training (P, T, W). Every pair of consecutive rows contains images of one of the shapes at 10 different rotation values. These are taken at evenly sp…
Figure 8
Figure 8. Figure 8: Extrapolation to six new shapes. Figure-ground Segmentation model reconstructions when six shapes are excluded from training (F, P, N T, V, W). Every pair of consecutive rows contains images of one of the shapes at 10 different rotation values. These are taken at evenl…
Figure 9
Figure 9. Figure 9: Testing abstract representations. On the left accuracy of three different probing models when prediction the shape of training images using the representations learned by a Slot Attention model: A simple Linear classifier trained with SGD, an MLP with one hidden layer …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 15 canonical work pages

  1. [1]

    Humboldt:'On language': On the diversity of human language construction and its influence on the mental development of the human species

    Wilhelm Von Humboldt, Wilhelm Freiherr von Humboldt, et al. Humboldt:'On language': On the diversity of human language construction and its influence on the mental development of the human species. Cambridge University Press, 1999

    work page 1999

  2. [2]

    Aspects of the Theory of Syntax, volume 11

    Noam Chomsky. Aspects of the Theory of Syntax, volume 11. MIT press, 2014

    work page 2014

  3. [3]

    Connectionism, constituency, and the language of thought

    Paul Smolensky. Connectionism, constituency, and the language of thought. University of Colorado at Boulder, 1988

    work page 1988

  4. [4]

    Infinite use of finite means? evaluating the generalization of center embedding learned from an artificial grammar

    R Thomas McCoy, Jennifer Culbertson, Paul Smolensky, and G \'e raldine Legendre. Infinite use of finite means? evaluating the generalization of center embedding learned from an artificial grammar. 2021

    work page 2021

  5. [5]

    Unsupervised Model Selection for Variational Disentangled Representation Learning

    Sunny Duan, Loic Matthey, Andre Saraiva, Nicholas Watters, Christopher P. Burgess, Alexander Lerchner, and Irina Higgins. Unsupervised Model Selection for Variational Disentangled Representation Learning . arXiv:1905.12614 [cs, stat], February 2020. URL http://arxiv.org/abs/1905.12614. arXiv: 1905.12614

    work page Pith review arXiv 1905

  6. [6]

    \ beta\ - VAE : Learning basic visual concepts with a constrained variational framework

    Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. \ beta\ - VAE : Learning basic visual concepts with a constrained variational framework. page 13

    work page

  7. [7]

    Montero, Casimir J.H

    Milton L. Montero, Casimir J.H. Ludwig, Rui Ponte Costa, Gaurav Malhotra, and Jeffrey S. Bowers. The role of disentanglement in generalisation. In International Conference on Learning Representations, 2020

    work page 2020

  8. [8]

    Montero, Jeffrey S

    Milton L. Montero, Jeffrey S. Bowers, Rui Ponte Costa, Casimir J.H. Ludwig, and Gaurav Malhotra. Lost in latent space: Examining failures of disentangled models at combinatorial generalisation. Advances in Neural Information Processing Systems, 35: 0 10136--10149, 2022

    work page 2022

Show all 23 references
  1. [9]

    u gelgen, Frederik Tr \

    Lukas Schott, Julius von K \"u gelgen, Frederik Tr \"a uble, Peter Gehler, Chris Russell, Matthias Bethge, Bernhard Sch \"o lkopf, Francesco Locatello, and Wieland Brendel. Visual representation learning does not generalize strongly within the same domain. arXiv preprint arXiv...

    2021 arXiv

  2. [10]

    Illiterate DALL-E learns to compose

    Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate DALL-E learns to compose. CoRR, abs/2110.11405, 2021. URL https://arxiv.org/abs/2110.11405

    2021 arXiv

  3. [11]

    Paxon Frady, Spencer Kent, Quinn Tran, Pentti Kanerva, Bruno A

    E. Paxon Frady, Spencer Kent, Quinn Tran, Pentti Kanerva, Bruno A. Olshausen, and Friedrich T. Sommer. Learning and generalization of compositional representations of visual scenes. URL http://arxiv.org/abs/2303.13691

    arXiv

  4. [12]

    Compositional generalization from first principles

    Thaddäus Wiedemer, Prasanna Mayilvahanan, Matthias Bethge, and Wieland Brendel. Compositional generalization from first principles. URL http://arxiv.org/abs/2307.05596

    arXiv

  5. [13]

    Perception viewed as an inverse problem

    Zygmunt Pizlo. Perception viewed as an inverse problem. Vision research, 41 0 (24): 0 3145--3161, 2001

    2001

  6. [14]

    Bronstein, Joan Bruna, Taco Cohen, and Petar Velickovic

    Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Velickovic. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. CoRR, abs/2104.13478, 2021. URL https://arxiv.org/abs/2104.13478

    2021 arXiv

  7. [15]

    Object-centric learning with slot attention

    Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. CoRR, abs/2006.15055, 2020. URL https://arxiv.org/abs/2006.15055

    2006 arXiv

  8. [16]

    Wasserstein auto-encoders

    Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. arXiv preprint arXiv:1711.01558, 2017

    2017 arXiv

  9. [17]

    Neural systematic binder

    Gautam Singh, Yeongbin Kim, and Sungjin Ahn. Neural systematic binder. arXiv preprint arXiv:2211.01177, 2022

    2022 arXiv

  10. [18]

    A century of gestalt psychology in visual perception: Ii

    Johan Wagemans, Jacob Feldman, Sergei Gepshtein, Ruth Kimchi, James R Pomerantz, Peter A Van der Helm, and Cees Van Leeuwen. A century of gestalt psychology in visual perception: Ii. conceptual and theoretical foundations. Psychological bulletin, 138 0 (6): 0 1218, 2012

    2012

  11. [19]

    Visual perception: Essential readings

    Steven Yantis. Visual perception: Essential readings. Psychology Press, 2001

    2001

  12. [20]

    A feature-integration theory of attention

    Anne M Treisman and Garry Gelade. A feature-integration theory of attention. Cognitive psychology, 12 0 (1): 0 97--136, 1980

    1980

  13. [21]

    u gelgen, Francesco Locatello, Peter Gehler, Thomas Brox, Matthias K \

    Matthias Tangemann, Steffen Schneider, Julius Von K \"u gelgen, Francesco Locatello, Peter Gehler, Thomas Brox, Matthias K \"u mmerer, Matthias Bethge, and Bernhard Sch \"o lkopf. Unsupervised object learning via common fate. arXiv preprint arXiv:2110.06562, 2021

    arXiv

  14. [22]

    3d shapes dataset

    Chris Burgess and Hyunjik Kim. 3d shapes dataset. https://github.com/deepmind/3dshapes-dataset/, 2018

    2018

  15. [23]

    dSprites : Disentanglement testing Sprites dataset

    Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dSprites : Disentanglement testing Sprites dataset . 2017. URL https://github.com/deepmind/dsprites-dataset/

    2017

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.