SlotDiT: Object-Centric Representations for Diffusion Transformers

Summary

Text-conditioned latent diffusion models perform strongly in video generation and are promising backbones for robotic applications.

Abstract Summary

Text-conditioned latent diffusion models perform strongly in video generation and are promising backbones for robotic applications. However, existing approaches rely on pixel-level or VAE-based latent representations that lack explicit semantic structure, leaving the impact of the representation space largely unexplored. Slot-based object-centric representations offer a structured alternative by decomposing scenes into object-level latents, or slots. While they have shown success in dynamics modeling and planning, they have not yet been explored for diffusion-based generative modeling. We introduce SlotDiT, a text-guided Diffusion Transformer (DiT) that operates in a slot-based latent space. Given a reference image and a language instruction, SlotDiT decomposes the scene into object-centric slots representing individual entities. Conditioned on the instruction and observed scene context, the model autoregressively denoises future slot trajectories to predict scene dynamics. To systematically investigate latent-space design for diffusion transformers, we compare slot-based representations against VAE-based and semantics-aligned alternatives within a unified DiT framework. Our experiments show that using slots as DiT latents yields competitive video generation quality while consistently improving task-completion rates across four robotic datasets. Furthermore, their compact representation provides a computationally efficient alternative to VAE-based and semantics-aligned latent spaces. Overall, our results demonstrate that object-centric structure is a powerful inductive bias for diffusion-based generative modeling in robotic environments. The project page is available at https://slot-dit.github.io/.

Key Points

Text-conditioned latent diffusion models perform strongly in video generation and are promising backbones for robotic applications.
However, existing approaches rely on pixel-level or VAE-based latent representations that lack explicit semantic structure, leaving the impact of the representation space largely unexplored.
Slot-based object-centric representations offer a structured alternative by decomposing scenes into object-level latents, or slots.
While they have shown success in dynamics modeling and planning, they have not yet been explored for diffusion-based generative modeling.
We introduce SlotDiT, a text-guided Diffusion Transformer (DiT) that operates in a slot-based latent space.

Methodology

We propose SlotDiT , depicted in Figure 2 , a latent Diffusion Transformer (DiT) that learns in a slot-based object-centric latent space. Given a single reference image $\textbf{X}_{1}$ and a language instruction $\mathcal{C}$, SlotDiT first encodes the scene into a set $\textbf{S}_{1}$ of $N_{\mathbf{S}}{}$ object-centric latent vectors, called slots (Sec. 3.1 ).

Then, conditioned on the instruction and observed scene context, the model iteratively denoises future slot trajectories from Gaussian noise to generate $N-1$ future sets of slots.

Methodology — SlotDiT: Object-Centric Representations for Diffusion Transformers — Figure 2: SlotDiT training. (a) Slot representation learning: A frozen DINOv2-ViT extracts visual features $\textbf{h}_{t}$ from each frame $\textbf{X}_{t}$, which are decomposed into object-centric slots $\textbf{S}_{t}$ via Slot Attention. The model is trained by minimizing a feature and image reconstruction objective $\mathcal{L}_{\text{Slot}}$. (b) SlotDiT: With the slot encoder frozen, noisy slot sets $\mathbf{S}_{t}^{(k_{t})}$ and text embeddings $\mathbf{C}$ are processed by a Diffusion Transformer. Per-frame noise levels are injected via adaLN, and the model is trained with the diffusion objective $\mathcal{L}_{\text{SlotDiT}}$.

Experiments

To evaluate SlotDiT , we investigate three key research questions: (i) can diffusion transformers operate effectively in an object-centric latent space? (ii) do slot-based representations improve task-solving and motion-following capabilities in robotic environments compared to established latent spaces? and (iii) do slots offer a more computationally efficient representation?

To this end, we evaluate SlotDiT on two downstream tasks: text-guided video generation and robot control.

Experiments — SlotDiT: Object-Centric Representations for Diffusion Transformers — Figure 3: Qualitative video generation examples on a synthetic ( LanguageTable-Synthetic ) and a real-world ( Bridgev2 ) dataset. Each block shows the ground-truth sequence (top row) and the predictions of SlotDiT , DiT + SD-VAE , and DiT + VideoVAE .

Sources and Demonstrations

Figures are reproduced from Plepi et al. See the full paper for experimental details and the project page, when the paper links one, for demonstrations and videos.