Generative Models

The fundamental goal of generative modeling is to learn a probability distribution $p_{\text{data}}(x)$ from samples, well enough to draw new samples from it. This sounds simple but is extraordinarily hard: even a modest 64×64 RGB image lives in a space of dimension $64 \times 64 \times 3 = 12{,}288$, and $p_{\text{data}}$ is an incredibly complex, low-dimensional manifold embedded in that space.

We want a model $p_\theta(x)$ parameterized by a neural network such that (1) samples $x \sim p_\theta$ look real, (2) we can evaluate likelihoods, and (3) the model generalizes rather than memorizing training data. Different generative model families make different trade-offs among tractability of sampling, tractability of likelihood evaluation, and quality of fit:

• Explicit density models: directly parameterize $p_\theta(x)$. Autoregressive models (PixelCNN, GPT) are a special case. VAEs maximize a lower bound on $\log p_\theta(x)$.
• Implicit density models: define a sampling procedure without an explicit density formula. GANs are the canonical example — you can sample easily but cannot evaluate $p_\theta(x)$.
• Score-based / diffusion models: model the score $\nabla_x \log p(x)$, then sample via Langevin dynamics or reverse diffusion.

A unifying lens: all generative models are trying to minimize some divergence between $p_\theta$ and $p_{\text{data}}$. The KL divergence $D_{KL}(p_{\text{data}} | p_\theta) = \mathbb{E}{x \sim p{\text{data}}}[-\log p_\theta(x)] + H(p_{\text{data}})$ is the maximum-likelihood objective. GANs implicitly minimize a Jensen-Shannon divergence (in the original paper) or a Wasserstein distance (in WGAN). Diffusion models optimize a denoising score matching objective that is a weighted sum of denoising losses.

# The high-level interface every generative model implements
class GenerativeModel:
    def sample(self, n: int) -> Tensor:
        """Draw n new samples from p_theta(x)."""
        raise NotImplementedError

    def log_prob(self, x: Tensor) -> Tensor:
        """Evaluate log p_theta(x). Not always tractable."""
        raise NotImplementedError

    def train_step(self, x_real: Tensor) -> float:
        """Update theta to better model the data. Returns loss."""
        raise NotImplementedError

An autoencoder is the simplest generative architecture idea: a neural network that compresses data to a low-dimensional latent code $z$ via an encoder $q_\phi(z|x)$, then reconstructs the original from $z$ via a decoder $p_\theta(x|z)$. The bottleneck forces the model to learn the most important structure.

Training minimizes reconstruction loss: $$\mathcal{L} = \mathbb{E}_{x}[|x - \hat{x}|^2]$$ where $\hat{x} = \text{decoder}(\text{encoder}(x))$.

Why autoencoders are not yet generative models: a standard autoencoder maps each training point to a single point in latent space. The latent space may have holes, discontinuities, and poor structure. If you try to sample a random $z$ and decode it, you often get garbage — the decoder has only learned to decode codes it saw during training. To generate, you need the latent space to be dense and smooth.

Autoencoder variants:

• Denoising autoencoder: corrupt $x$ to $\tilde{x}$ (add noise, mask pixels), train to reconstruct $x$. Encourages robust feature learning and is a precursor to diffusion.
• Sparse autoencoder: add $L_1$ penalty on $z$ to encourage few active units. Used in unsupervised feature discovery.
• Contractive autoencoder: penalize the Frobenius norm of the Jacobian $\partial z / \partial x$ to enforce local invariance.

Analogy: an autoencoder is like a data-compression algorithm that learns its own codebook. But unlike JPEG (which has a fixed cosine basis), the encoder learns whatever basis best captures the statistical structure of the specific data distribution.

import torch
import torch.nn as nn

class Autoencoder(nn.Module):
    def __init__(self, input_dim=784, latent_dim=32):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(input_dim, 256), nn.ReLU(),
            nn.Linear(256, latent_dim)
        )
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 256), nn.ReLU(),
            nn.Linear(256, input_dim), nn.Sigmoid()
        )

    def forward(self, x):
        z = self.encoder(x)
        return self.decoder(z)

# MSE reconstruction loss
loss_fn = nn.MSELoss()
model = Autoencoder()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

for x_batch in dataloader:
    x_hat = model(x_batch)
    loss = loss_fn(x_hat, x_batch)
    optimizer.zero_grad(); loss.backward(); optimizer.step()

The Variational Autoencoder (Kingma & Welling, 2013) fixes the fundamental problem of plain autoencoders by placing a prior over the latent space and making the encoder output a distribution rather than a point.

Generative model: $p_\theta(x, z) = p_\theta(x|z),p(z)$, where $p(z) = \mathcal{N}(0, I)$ is the prior. Marginalizing over $z$ gives $p_\theta(x) = \int p_\theta(x|z),p(z),dz$, which is intractable to compute exactly.

Inference model (encoder): $q_\phi(z|x) = \mathcal{N}(z; \mu_\phi(x), \text{diag}(\sigma^2_\phi(x)))$ — the encoder outputs a mean and variance, defining a Gaussian over $z$.

The ELBO (Evidence Lower Bound): since $\log p_\theta(x)$ is intractable, we maximize a lower bound: $$\log p_\theta(x) \geq \underbrace{\mathbb{E}{z \sim q\phi(z|x)}[\log p_\theta(x|z)]}{\text{reconstruction}} - \underbrace{D{KL}(q_\phi(z|x) | p(z))}{\text{regularization}} = \mathcal{L}{\text{ELBO}}$$ The first term says the decoder should reconstruct $x$ from samples of the posterior. The second term says the posterior should not deviate too far from the prior $\mathcal{N}(0,I)$.

Why this gives a good generative model: the KL term forces the encoded distributions to overlap with the prior, filling the latent space densely. At test time, sample $z \sim \mathcal{N}(0,I)$ and decode — since the prior and posteriors overlap, decoded samples are coherent.

The reparameterization trick: backpropagating through $z \sim \mathcal{N}(\mu, \sigma^2)$ is impossible because sampling is stochastic. Instead, write $z = \mu + \sigma \odot \epsilon$, where $\epsilon \sim \mathcal{N}(0,I)$. Now $z$ is a deterministic function of $\epsilon$, so gradients flow through $\mu$ and $\sigma$.

Closed-form KL for Gaussians: $D_{KL}(\mathcal{N}(\mu, \sigma^2) | \mathcal{N}(0,I)) = \frac{1}{2}\sum_j (\mu_j^2 + \sigma_j^2 - \log \sigma_j^2 - 1)$.

Known limitation: VAE samples tend to be blurry. Because the reconstruction loss is pixel-wise MSE (assuming a Gaussian decoder), the model averages over modes of uncertainty, blurring predictions. This is fundamentally different from the perceptual sharpness of GAN or diffusion outputs.

class VAE(nn.Module):
    def __init__(self, input_dim=784, latent_dim=16):
        super().__init__()
        self.enc = nn.Sequential(nn.Linear(input_dim, 256), nn.ReLU())
        self.fc_mu = nn.Linear(256, latent_dim)
        self.fc_log_var = nn.Linear(256, latent_dim)
        self.dec = nn.Sequential(
            nn.Linear(latent_dim, 256), nn.ReLU(),
            nn.Linear(256, input_dim), nn.Sigmoid()
        )

    def encode(self, x):
        h = self.enc(x)
        return self.fc_mu(h), self.fc_log_var(h)

    def reparameterize(self, mu, log_var):
        std = torch.exp(0.5 * log_var)          # sigma
        eps = torch.randn_like(std)              # eps ~ N(0,I)
        return mu + eps * std                    # reparameterization

    def forward(self, x):
        mu, log_var = self.encode(x)
        z = self.reparameterize(mu, log_var)
        x_hat = self.dec(z)
        return x_hat, mu, log_var

def elbo_loss(x, x_hat, mu, log_var, beta=1.0):
    recon = nn.functional.binary_cross_entropy(x_hat, x, reduction='sum')
    kl = -0.5 * torch.sum(1 + log_var - mu.pow(2) - log_var.exp())
    return (recon + beta * kl) / x.size(0)

The GAN (Goodfellow et al., 2014) is arguably the most creative idea in the history of generative modeling. Instead of maximizing likelihood, GANs train two networks adversarially:

• Generator $G_\theta$: takes noise $z \sim p(z)$ (e.g., $\mathcal{N}(0,I)$) and outputs a fake sample $\hat{x} = G_\theta(z)$. Goal: fool the discriminator.
• Discriminator $D_\phi$: takes a sample (real or fake) and outputs $D_\phi(x) \in [0,1]$, the probability that $x$ is real. Goal: correctly distinguish real from fake.

The minimax objective: $$\min_\theta \max_\phi ; \mathbb{E}{x \sim p{\text{data}}}[\log D_\phi(x)] + \mathbb{E}{z \sim p(z)}[\log(1 - D\phi(G_\theta(z)))]$$

At the optimal discriminator $D^\phi$, the generator's objective becomes minimizing the Jensen-Shannon divergence $D{JS}(p_{\text{data}} | p_G)$. The global optimum is $p_G = p_{\text{data}}$ with $D^(x) = 1/2$ everywhere.

Training in practice: alternate gradient steps — update $D$ a few steps, then update $G$. In practice the generator uses the non-saturating loss: $\max_\theta \mathbb{E}z[\log D\phi(G_\theta(z))]$ rather than $\min_\theta \mathbb{E}z[\log(1-D\phi(G_\theta(z)))]$, because the latter saturates (has vanishing gradients) when the discriminator is confident.

Mode collapse: the generator learns to produce only a few highly-convincing samples (modes) rather than the full diversity of the data. One mode fools the discriminator and the generator gets stuck.

Training instability: GAN training is notoriously unstable because the two-player game has no simple convergence criterion. The loss curves are not informative — a decreasing generator loss may just mean the discriminator got worse, not that generation improved.

Key architectural advances: DCGAN introduced convolutional architectures for images. Progressive GAN trained from low to high resolution. StyleGAN separated coarse (pose, shape) from fine (texture, color) control via style injection at multiple scales, producing photorealistic faces at 1024×1024.

# GAN training loop (non-saturating loss)
def train_gan_step(generator, discriminator, real_batch, z_dim, optG, optD):
    B = real_batch.size(0)

    # -- Update discriminator --
    z = torch.randn(B, z_dim)
    fake = generator(z).detach()  # don't backprop into G yet
    d_real = discriminator(real_batch)
    d_fake = discriminator(fake)
    loss_D = -torch.mean(torch.log(d_real + 1e-8)
                         + torch.log(1 - d_fake + 1e-8))
    optD.zero_grad(); loss_D.backward(); optD.step()

    # -- Update generator (non-saturating: maximize log D(G(z))) --
    z = torch.randn(B, z_dim)
    fake = generator(z)
    d_fake2 = discriminator(fake)
    loss_G = -torch.mean(torch.log(d_fake2 + 1e-8))
    optG.zero_grad(); loss_G.backward(); optG.step()

    return loss_D.item(), loss_G.item()

The original GAN minimax objective has a deep flaw: when the generator and data distributions have disjoint supports (which happens when the generator is bad early in training), the JS divergence is a constant $\log 2$, giving zero gradient. Training stalls.

WGAN (Arjovsky et al., 2017) replaces JS divergence with the Wasserstein-1 (Earth Mover's) distance: $$W_1(p_{\text{data}}, p_G) = \sup_{|f|L \leq 1} \mathbb{E}{x \sim p_{\text{data}}}[f(x)] - \mathbb{E}_{x \sim p_G}[f(x)]$$ where the supremum is over all 1-Lipschitz functions $f$ (called the critic rather than discriminator, since it no longer outputs a probability).

Earth mover's intuition: $W_1$ measures the minimum cost to transport mass from $p_G$ to $p_{\text{data}}$, where cost is distance moved. It is defined even when the distributions have disjoint support and varies continuously with the generator's parameters.

WGAN-GP: enforcing the Lipschitz constraint via weight clipping (original WGAN) hurts capacity. WGAN-GP (Gulrajani et al., 2017) instead adds a gradient penalty: for points $\hat{x}$ interpolated between real and fake, penalize $(|\nabla_{\hat{x}} D(\hat{x})|_2 - 1)^2$. This directly enforces the Lipschitz constraint and dramatically stabilizes training.

The WGAN objective provides a meaningful loss metric: a lower critic loss reliably indicates better generation quality, solving one of GAN training's biggest practical headaches.

def gradient_penalty(critic, real, fake, device):
    """WGAN-GP gradient penalty."""
    B = real.size(0)
    alpha = torch.rand(B, 1, 1, 1, device=device)  # random interpolation weight
    interpolated = (alpha * real + (1 - alpha) * fake).requires_grad_(True)
    d_interp = critic(interpolated)
    grads = torch.autograd.grad(
        outputs=d_interp, inputs=interpolated,
        grad_outputs=torch.ones_like(d_interp),
        create_graph=True, retain_graph=True
    )[0]
    grad_norm = grads.view(B, -1).norm(2, dim=1)
    return ((grad_norm - 1) ** 2).mean()

# WGAN-GP critic loss
def wgan_gp_critic_loss(critic, real, fake, lambda_gp=10):
    return (-critic(real).mean() + critic(fake).mean()
            + lambda_gp * gradient_penalty(critic, real, fake, real.device))

How do we know if a generative model is good? This is genuinely hard — there is no single correct answer, and different metrics capture different aspects.

Inception Score (IS) (Salimans et al., 2016): run a pre-trained Inception network on samples. IS rewards samples that (1) look like a specific class (low entropy of $p(y|x)$) and (2) are diverse across classes (high entropy of $p(y) = \int p(y|x)p_G(x)dx$). $\text{IS} = \exp(\mathbb{E}x[D{KL}(p(y|x) | p(y))])$. Higher is better. Limitation: does not compare to real data directly.

Fréchet Inception Distance (FID) (Heusel et al., 2017): extract feature vectors from a pre-trained Inception network for both real and generated images. Model both distributions as Gaussians. FID is the Fréchet distance between them: $$\text{FID} = |\mu_r - \mu_g|^2 + \text{Tr}(\Sigma_r + \Sigma_g - 2(\Sigma_r \Sigma_g)^{1/2})$$ Lower FID = more similar to real data. FID correlates well with human perception and has become the standard metric.

Precision and Recall (Kynkäänniemi et al., 2019): disentangles fidelity (are generated samples realistic?) from diversity (does the generator cover the full data distribution?). Both are important; a model can have high fidelity but low recall (mode collapse).

LPIPS (Learned Perceptual Image Patch Similarity): perceptual similarity between images using deep features, better correlated with human judgment than pixel-wise MSE.

The core tension: quality vs diversity. A model that memorizes one real sample has perfect quality but zero diversity. A model that generates uniform noise has perfect diversity but zero quality. Good models must balance both.

# FID computation sketch (using torchmetrics in practice)
import numpy as np
from scipy.linalg import sqrtm

def compute_fid(real_features, gen_features):
    """real_features, gen_features: (N, D) numpy arrays of Inception features."""
    mu_r, mu_g = real_features.mean(0), gen_features.mean(0)
    sigma_r = np.cov(real_features, rowvar=False)
    sigma_g = np.cov(gen_features, rowvar=False)
    diff = mu_r - mu_g
    # Matrix sqrt of sigma_r @ sigma_g
    covmean = sqrtm(sigma_r @ sigma_g).real
    fid = diff @ diff + np.trace(sigma_r + sigma_g - 2 * covmean)
    return float(fid)

Diffusion models (Ho et al., 2020; Song et al., 2019) are the current state-of-the-art for image generation. The key idea is strikingly elegant: define a process that destroys data by adding noise over many small steps, then train a network to reverse that process.

Forward process (fixed, no parameters): given data $x_0 \sim p_{\text{data}}$, produce a sequence $x_0, x_1, \ldots, x_T$ by adding Gaussian noise: $$q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t}, x_{t-1},; \beta_t I)$$ for a variance schedule $\beta_1, \ldots, \beta_T$ (typically small, e.g. $10^{-4}$ to $0.02$). After $T$ steps ($T \approx 1000$), $x_T \approx \mathcal{N}(0, I)$ — pure noise.

Closed form for any timestep: define $\bar{\alpha}t = \prod{s=1}^t (1-\beta_s)$. Then: $$q(x_t | x_0) = \mathcal{N}(x_t; \sqrt{\bar{\alpha}_t},x_0,; (1-\bar{\alpha}_t)I)$$ So $x_t = \sqrt{\bar{\alpha}_t},x_0 + \sqrt{1-\bar{\alpha}_t},\epsilon$, where $\epsilon \sim \mathcal{N}(0,I)$. This lets us sample any $x_t$ directly without running all $t$ steps.

Reverse process (learned): $p_\theta(x_{t-1}|x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \Sigma_\theta(x_t, t))$. A neural network predicts the parameters of this Gaussian.

Training objective (simplified, Ho et al.): predict the noise $\epsilon$ that was added: $$\mathcal{L}{\text{simple}} = \mathbb{E}{x_0, \epsilon, t}\left[|\epsilon - \epsilon_\theta(\sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}t}\epsilon, t)|^2\right]$$ Sample a random timestep $t$, add the corresponding noise to get $x_t$, and train $\epsilon\theta$ (a U-Net) to predict the noise. This is a denoising loss — the model learns to clean up noisy data.

Sampling: start from $x_T \sim \mathcal{N}(0,I)$, then repeatedly apply: $$x_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}t}}\epsilon\theta(x_t, t)\right) + \sigma_t z$$ where $z \sim \mathcal{N}(0,I)$. After $T$ steps, $x_0$ is a generated sample.

Connection to score matching: $\epsilon_\theta(x_t, t) \propto -\nabla_{x_t} \log q(x_t)$, the score function of the noisy distribution. Diffusion models are learning to follow the score — gradient ascent on the log-probability — to climb toward regions of high density.

import torch
import torch.nn as nn

class DiffusionProcess:
    def __init__(self, T=1000, beta_start=1e-4, beta_end=0.02):
        self.T = T
        betas = torch.linspace(beta_start, beta_end, T)
        alphas = 1 - betas
        self.alpha_bar = torch.cumprod(alphas, dim=0)  # alpha_bar_t

    def q_sample(self, x0, t, noise=None):
        """Sample x_t from q(x_t | x_0) in one shot."""
        if noise is None:
            noise = torch.randn_like(x0)
        sqrt_ab = self.alpha_bar[t].sqrt().view(-1, 1, 1, 1)
        sqrt_1mab = (1 - self.alpha_bar[t]).sqrt().view(-1, 1, 1, 1)
        return sqrt_ab * x0 + sqrt_1mab * noise, noise

    def train_loss(self, model, x0):
        """DDPM simplified training loss."""
        B = x0.size(0)
        t = torch.randint(0, self.T, (B,), device=x0.device)
        x_t, true_noise = self.q_sample(x0, t)
        pred_noise = model(x_t, t)  # U-Net predicts noise
        return nn.functional.mse_loss(pred_noise, true_noise)

The neural network backbone used in most diffusion models is a U-Net — an encoder-decoder with skip connections, originally developed for biomedical image segmentation (Ronneberger et al., 2015).

Why U-Net for denoising? Denoising requires both local detail (sharp edges, fine textures) and global context (what kind of object is this?). The encoder contracts the spatial resolution while building semantic features; the decoder expands back to full resolution using skip connections that preserve fine details from the encoder.

Architecture: a stack of ResNet blocks with downsampling (strided convolutions or pooling), followed by a bottleneck, followed by a mirrored stack of ResNet blocks with upsampling (transposed convolutions or bilinear + conv). Skip connections concatenate encoder feature maps to decoder feature maps at the same resolution.

Conditioning on timestep $t$: the network must know which noise level it is operating at. $t$ is embedded as a sinusoidal positional embedding (like Transformers), then projected and added to the hidden states via FiLM (Feature-wise Linear Modulation): $h \leftarrow \gamma(t) \odot h + \delta(t)$.

Attention layers: modern diffusion U-Nets (as in DDPM and Stable Diffusion) insert self-attention layers at lower spatial resolutions to capture long-range dependencies. Cross-attention allows conditioning on text embeddings (CLIP or T5), which is the mechanism behind text-to-image models like Stable Diffusion.

Latent diffusion: Stable Diffusion (Rombach et al., 2022) runs diffusion not in pixel space but in the latent space of a pretrained VAE — a 4× to 8× downsampled, more compact representation. This reduces computational cost dramatically while retaining image quality, since the VAE handles perceptual quality.

# Minimal U-Net block with timestep conditioning (FiLM)
class ResBlock(nn.Module):
    def __init__(self, in_ch, out_ch, t_emb_dim):
        super().__init__()
        self.conv1 = nn.Sequential(nn.GroupNorm(8, in_ch), nn.SiLU(),
                                   nn.Conv2d(in_ch, out_ch, 3, padding=1))
        self.t_proj = nn.Sequential(nn.SiLU(),
                                    nn.Linear(t_emb_dim, out_ch * 2))  # scale+shift
        self.conv2 = nn.Sequential(nn.GroupNorm(8, out_ch), nn.SiLU(),
                                   nn.Conv2d(out_ch, out_ch, 3, padding=1))
        self.skip = nn.Conv2d(in_ch, out_ch, 1) if in_ch != out_ch else nn.Identity()

    def forward(self, x, t_emb):
        h = self.conv1(x)
        scale, shift = self.t_proj(t_emb)[..., None, None].chunk(2, dim=1)
        h = h * (1 + scale) + shift          # FiLM conditioning
        h = self.conv2(h)
        return h + self.skip(x)              # residual

A major practical limitation of DDPM (Ho et al., 2020) is the need for $T \approx 1000$ sequential network forward passes to generate a single sample — each pass through a large U-Net, making generation slow.

DDIM (Denoising Diffusion Implicit Models, Song et al., 2020) shows that the same trained noise-prediction network $\epsilon_\theta$ can be used with a completely different (non-Markovian) sampling process that takes far fewer steps: $$x_{t-1} = \sqrt{\bar{\alpha}{t-1}}\underbrace{\left(\frac{x_t - \sqrt{1-\bar{\alpha}t},\epsilon\theta(x_t,t)}{\sqrt{\bar{\alpha}t}}\right)}{\text{predicted } x_0} + \sqrt{1-\bar{\alpha}{t-1}-\sigma_t^2},\epsilon_\theta(x_t,t) + \sigma_t\epsilon$$ When $\sigma_t = 0$, the process is deterministic: the same $x_T$ always produces the same $x_0$. This enables latent space interpolation and inversion.

Accelerated sampling: DDIM allows skipping timesteps — instead of $t = 1000, 999, \ldots, 1$, use a subsequence like $t = 1000, 980, 960, \ldots$ with only 50 or 20 steps, with minimal quality loss. This gives 20–50× speedup.

DPM-Solver (Lu et al., 2022) and DPM-Solver++ use higher-order ODE solvers to further reduce the number of steps needed (often 10–15 steps for high quality), by exploiting the analytical structure of the diffusion ODE.

Classifier-free guidance (CFG) (Ho & Salimans, 2021): train the noise network jointly on conditioned and uncondititioned inputs (randomly drop the condition). At sampling, linearly extrapolate: $$\hat{\epsilon} = \epsilon_\theta(x_t, \emptyset) + w(\epsilon_\theta(x_t, c) - \epsilon_\theta(x_t, \emptyset))$$ Guidance scale $w > 1$ trades diversity for fidelity — the standard way to make text-to-image models produce sharper, more prompt-faithful images.

# DDIM deterministic sampling step
@torch.no_grad()
def ddim_step(model, x_t, t, t_prev, alpha_bar):
    """
    model: noise prediction network
    t, t_prev: current and previous timestep indices (t > t_prev)
    alpha_bar: precomputed cumulative products
    """
    ab_t    = alpha_bar[t]
    ab_prev = alpha_bar[t_prev] if t_prev >= 0 else torch.tensor(1.0)

    eps = model(x_t, t)                            # predict noise
    x0_pred = (x_t - (1 - ab_t).sqrt() * eps) / ab_t.sqrt()  # predict clean image
    x0_pred = x0_pred.clamp(-1, 1)

    # deterministic DDIM update (sigma=0)
    x_prev = ab_prev.sqrt() * x0_pred + (1 - ab_prev).sqrt() * eps
    return x_prev

# Classifier-free guidance
def cfg_noise(model, x_t, t, cond, uncond, w=7.5):
    eps_cond   = model(x_t, t, cond)
    eps_uncond = model(x_t, t, uncond)
    return eps_uncond + w * (eps_cond - eps_uncond)

Song et al. (2020) unified DDPM and NCSN (Noise Conditional Score Networks) into a single elegant framework using stochastic differential equations (SDEs).

Forward SDE: generalize the discrete noise-adding process to a continuous-time SDE: $$dx = f(x, t),dt + g(t),dW$$ where $W$ is a Wiener process (Brownian motion). The forward process diffuses data into noise. DDPM corresponds to the VP-SDE (Variance-Preserving); NCSN corresponds to the VE-SDE (Variance-Exploding).

The reverse SDE (Anderson, 1982): for any SDE with known score $\nabla_x \log p_t(x)$, the reverse-time SDE is: $$dx = [f(x,t) - g(t)^2 \nabla_x \log p_t(x)],dt + g(t),d\bar{W}$$ where $\bar{W}$ is a reverse Wiener process. If we learn $s_\theta(x, t) \approx \nabla_x \log p_t(x)$ (the score function), we can integrate the reverse SDE to generate samples.

Score matching objective: $\mathbb{E}{t, x_0, x_t}[\lambda(t)|s\theta(x_t, t) - \nabla_{x_t} \log q(x_t|x_0)|^2]$. Note $\nabla_{x_t} \log q(x_t|x_0) = -(x_t - \sqrt{\bar{\alpha}_t}x_0)/(1-\bar{\alpha}_t)$, which is $-\epsilon/\sqrt{1-\bar{\alpha}_t}$. So predicting the score is equivalent to predicting the noise — DDPM and score matching are the same model.

Probability flow ODE: there is also a deterministic ODE with the same marginals as the SDE: $$\frac{dx}{dt} = f(x,t) - \frac{1}{2}g(t)^2 \nabla_x \log p_t(x)$$ This gives an exact bijection between data and noise, enabling likelihood computation via the instantaneous change-of-variables formula and enabling DDIM-style deterministic generation.

Langevin dynamics connection: at each noise level $t$, $\nabla_x \log p_t(x)$ points toward higher-density regions. Iterating $x \leftarrow x + \eta \nabla_x \log p_t(x) + \sqrt{2\eta},\epsilon$ (Langevin MCMC) explores $p_t$. Annealing $t$ from large to small is a form of simulated annealing.

# Score matching loss — equivalent to DDPM noise prediction
# score = -eps / sqrt(1 - alpha_bar_t)
# So: lambda(t) * ||score_theta - true_score||^2
#   = ||eps_theta - true_eps||^2  (up to a weighting)

def score_matching_loss(score_net, x0, alpha_bar, T):
    B = x0.shape[0]
    t_idx = torch.randint(0, T, (B,))
    noise = torch.randn_like(x0)
    ab = alpha_bar[t_idx].view(B, *([1]*(x0.dim()-1)))
    x_t = ab.sqrt() * x0 + (1 - ab).sqrt() * noise
    # true score: -noise / sqrt(1 - ab)
    true_score = -noise / (1 - ab).sqrt()
    pred_score = score_net(x_t, t_idx)
    # weight by (1-ab) to recover simple eps-prediction loss
    return ((pred_score - true_score)**2 * (1 - ab)).mean()

Raw generative models produce samples from the learned distribution, but the most powerful applications require controlled generation: generate an image matching a text prompt, a layout sketch, a depth map, or a reference pose.

Class conditioning: the simplest form. Embed the class label and add it to the U-Net (via FiLM or cross-attention). Used in class-conditional DDPM and ADM (Dhariwal & Nichol, 2021).

Text-to-image (CLIP + Diffusion): DALL-E 2 uses a CLIP image embedding as the condition, with a prior network that maps CLIP text embeddings to CLIP image embeddings, then a diffusion decoder. Stable Diffusion uses a CLIP text encoder and cross-attention in the U-Net to attend over text token embeddings.

ControlNet (Zhang et al., 2023): a technique to add spatial conditioning (depth, edge, pose, segmentation) to a pretrained diffusion model without retraining it. The idea is to create a trainable copy of the U-Net's encoding layers, connected to the frozen backbone via zero-initialized convolutions ("zero convolutions" — start at zero so the model initially ignores the control signal). Only the copy is trained, on (image, condition) pairs. This preserves the original model's generative quality while adding precise spatial control.

InstructPix2Pix (Brooks et al., 2022): fine-tune a text-conditioned diffusion model to follow editing instructions ("make it look like winter"). Conditions on both the original image and the text instruction.

Classifier guidance (Dhariwal & Nichol, 2021): at each denoising step, shift the score estimate by the gradient of a separate classifier: $\tilde{\epsilon} = \epsilon_\theta(x_t) - s \cdot \sqrt{1-\bar{\alpha}t} \nabla{x_t} \log p_\psi(y|x_t)$. Effective but requires a separately trained noisy-image classifier.

# Cross-attention conditioning in U-Net (text-to-image)
class CrossAttentionBlock(nn.Module):
    def __init__(self, query_dim, context_dim, heads=8):
        super().__init__()
        self.heads = heads
        d = query_dim // heads
        self.to_q = nn.Linear(query_dim, query_dim, bias=False)
        self.to_k = nn.Linear(context_dim, query_dim, bias=False)
        self.to_v = nn.Linear(context_dim, query_dim, bias=False)
        self.out  = nn.Linear(query_dim, query_dim)

    def forward(self, x, context):
        # x: (B, L_img, C)  context: (B, L_text, C_text)
        Q = self.to_q(x)
        K = self.to_k(context)
        V = self.to_v(context)
        # Multi-head attention: each image token attends to all text tokens
        # (standard scaled dot-product, split into heads)
        attn = torch.softmax(Q @ K.transpose(-2,-1) / Q.size(-1)**0.5, dim=-1)
        return self.out(attn @ V) + x  # residual

Each generative model family has distinct strengths, weaknesses, and inductive biases. Understanding these helps you choose the right tool.

VAEs:

• Strengths: fast sampling (single forward pass), exact (approximate) likelihood, smooth and interpretable latent space, stable training.
• Weaknesses: blurry samples due to pixel-wise reconstruction loss; the ELBO is a lower bound so true likelihood is underestimated; latent space not perfectly continuous.
• Best for: representation learning, disentanglement, anomaly detection, when latent interpolation matters, as a compression backbone (Latent Diffusion).

GANs:

• Strengths: highest perceptual quality on specific domains (StyleGAN faces); fastest sampling; no explicit likelihood needed.
• Weaknesses: training instability; mode collapse; no tractable likelihood; hard to evaluate convergence; requires careful tuning.
• Best for: high-fidelity generation of a specific domain (faces, cars), image-to-image translation, real-time generation where speed matters.

Diffusion Models:

• Strengths: state-of-the-art sample quality AND diversity; principled likelihood via probability flow ODE; stable training; excellent conditioning; composable.
• Weaknesses: slow sampling (100s of steps, improved but not fully solved); computationally expensive; no compact latent space natively.
• Best for: text-to-image, general-purpose high-quality generation, scientific applications (protein structure, molecular design), audio synthesis.

Quantitative comparison (approximate, circa 2023–24 on ImageNet 256×256):

• BigGAN FID: ~7
• VQ-VAE-2: ~3.8
• ADM (Diffusion): ~3.9
• ADM+guidance: ~2.97
• DiT-XL/2+guidance: ~2.27

Emerging unifications: Flow Matching (Lipman et al., 2022) trains continuous normalizing flows with a simpler objective than maximum likelihood, achieving diffusion-level quality with fewer steps. Consistency Models (Song et al., 2023) distill diffusion trajectories to enable single-step generation.