DALL-E 2 is an AI system from OpenAI that creates images and art from a text prompt. It can also add or remove elements from an existing image and produce variations of it. Underneath, it is a text-guided diffusion model, and the 2022 paper by Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen (where the system is called unCLIP) explains how the pieces fit together and how it compares with OpenAI’s previous image model, GLIDE.
The two components
The first component is CLIP, or Contrastive Language-Image Pre-training, a neural network that links text to its visual representation. Rather than predicting a caption for an image, CLIP learns how related a given caption is to a given image. That gives it a contrastive objective instead of a predictive one, and it means CLIP can produce an embedding, a vector that captures what an image of a prompt should contain.
The second component is a diffusion model. A diffusion model is trained by gradually adding noise to an image until it is unrecognizable, then learning to reverse the process until a coherent image comes back out. DALL-E 2 uses GLIDE (Guided Language-to-Image Diffusion for Generation and Editing) for this step.
Approach
The full system splits into two stages. A prior converts the text prompt into an image representation, and a decoder converts that representation into an actual image. DALL-E 2 uses CLIP to generate image embeddings from the prompt, then uses GLIDE to generate images from those embeddings. The authors tried two kinds of prior, autoregressive and diffusion.
To evaluate the result, they ran human comparisons between unCLIP and GLIDE on three qualities: photorealism, caption similarity, and sample diversity. They also computed the Fréchet Inception Distance (FID) on the Microsoft Common Objects in Context (MS COCO) dataset, where a lower score means higher image quality.
Findings
At higher guidance scales, human raters found unCLIP produced more diverse images than GLIDE while staying comparable in photorealism and caption similarity. Along the whole guidance scale, unCLIP was preferred on at least one of the three qualities. On MS COCO, unCLIP with either prior had a lower FID than GLIDE at every guidance scale, so it generated better images even as the guidance parameter changed.
The authors describe DALL-E 2 as producing more diverse, accurate, and photorealistic images than its predecessor, and as a step toward understanding how AI systems perceive and interpret the world. It cannot replace artists, but it gives people another way to express an idea visually.
Source: Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. (2022). Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv:2204.06125