Understanding generative AI

Every generative model is trying to do the same thing. Somewhere there is a distribution over real images, or real sentences, or real molecules. Nobody has written it down and nobody can. All you have is a pile of samples from it. The job is to build something you can draw new samples from that a person would accept as belonging to the same pile.

Three families dominate, and the interesting thing is not that they work. It is that each one gives up something different to get there.

Variational autoencoders: compress, then decompress badly on purpose

A VAE squeezes each training example down to a small vector and learns to rebuild it. On its own that is just an autoencoder. The variational part adds a constraint: the squeezed vectors have to be spread out in a shape you can sample from — usually a Gaussian blob. Train it, throw away the encoder, draw a random point from the blob, and the decoder turns it into something plausible.

The bargain: you get a latent space you can move around in, and training that almost never blows up. You pay in sharpness. The loss rewards being close to the average of everything it might have meant, and the average of many faces is a blurry face.

GANs: hire a critic

A GAN skips the reconstruction objective entirely. A generator invents samples, a discriminator tries to tell them from real ones, and each gets better by making the other's job harder. There is no pixel-wise loss telling the generator to hedge, so it does not. GAN outputs are sharp in a way VAE outputs are usually not.

The bargain: two networks chasing each other is not a stable thing. Training can oscillate, or collapse onto a handful of outputs the discriminator happens to be bad at — a model that produces five convincing faces and nothing else scores well on the objective and is useless in practice.

Mode collapse is the honest failure of an adversarial objective: winning the game and modelling the distribution are not the same goal.

Diffusion: destroy it slowly, learn to undo one step

Diffusion models take a training image and add a little noise, then a little more, for hundreds of steps, until nothing is left but static. Then they learn the reverse: given a noisy image, predict the noise that was added. That is a small, well-posed regression problem, and it is stable to train.

To generate, start from pure static and apply the learned denoiser over and over. Each step is a modest correction, and the hard problem of "invent an image" gets amortised across hundreds of easy ones.

# the training step, stripped to its bones
t     = random_timestep()
noise = randn_like(x0)
xt    = sqrt(alpha[t]) * x0 + sqrt(1 - alpha[t]) * noise

loss  = mse(model(xt, t), noise)   # predict what was added

The bargain: quality and coverage, in exchange for compute. One sample costs many forward passes where a GAN costs one. Most of the last few years of diffusion research has been about buying that cost back — fewer steps, smaller latent spaces, distilled samplers.

Choosing between them

• Need a smooth latent space to interpolate or edit in? VAE, or a diffusion model operating in a learned latent space.
• Need one sample, right now, on a tight budget? A GAN is still hard to beat on pure latency.
• Need coverage and fidelity and can afford the sampling? Diffusion.

The framing that stuck with me: these are not three techniques so much as three positions on a triangle of sample quality, distribution coverage, and sampling cost. You cannot have all three, and the architecture you pick is really a statement about which one you are willing to lose.