How diffusion models, a kind of generative AI, compose data

What is it about?

Generative AI creates new content by combining data features it has encountered during its training. However, these combinations are not just pieces of data put together but rather coherent combinations in some latent semantic space. How this process works in the latent space remained a mystery. In this work, we used a theoretical model displaying this kind of latent compositionality to show how high-level and low-level features get assembled during the generation process of diffusion models. We discovered that different levels of features undergo different processes: a sharp transition for high-level ones and a smooth evolution for low-level ones. These different behaviors result from compositionality relationships between features at different levels, which cannot be understood using shallow data models. Remarkably, we were able to measure the predicted effects in real data, specifically in state-of-the-art diffusion models on high-resolution images.

Featured Image

Photo by Google DeepMind on Unsplash

Photo by Google DeepMind on Unsplash

Why is it important?

Our findings reveal something counterintuitive about how generative AI models assemble features in the data. Most theoretical results make simplifying assumptions where data features are shallow compositions in data space. Our work, instead, assumes a specific kind of interaction between features that are composed at different hierarchical levels of abstraction and shows that the same low-level features are re-used by generative AI to create data with different high-level concepts. This is one of the first theoretical results of this kind.

Perspectives