What if two numbers could paint a sunrise?
A map where similar sunrises live near each other. Move a little to the right and the sunrise shifts later in the day; move up and the season warms.
The latent space is a low-dimensional manifold learned by the VAE. Each point represents a compressed description of an image. The decoder maps this back: .
Unlike PCA (linear), the VAE learns a nonlinear mapping with a probabilistic prior. Unlike t-SNE and UMAP (which only embed, don't decode), the VAE can generate new images from any point in latent space. The trade-off: VAE embeddings may be less visually separated than t-SNE but support generation and interpolation.
Pick a sunrise to see how it gets compressed to two numbers:
Original
Reconstruction
A compressor that finds the two most important numbers to describe any sunrise.
The encoder is a neural network that maps an input image to a distribution in latent space. It outputs two vectors: a mean and a log-variance .
Instead of optimizing a separate latent code for each image (as in classical variational inference), the encoder amortizes the cost by learning a single function that maps any image to its approximate posterior. This is what makes VAEs fast at test time — one forward pass instead of an optimization loop.
Somewhere inside a neural network lives a tiny world of sunrises. Every point in this two-dimensional space maps to a unique image — a gradient of color, light, and atmosphere.
The horizontal axis captures something like time of day: deep pre-dawn blues on the left, bright midday light on the right.
The vertical axis captures time of year: cool winter tones below, warm summer golds above.
Hover over the canvas to decode sunrises in real time. Click to pin two favorites and interpolate between them.
A painter that only needs two instructions. Give it two numbers and it produces a full sunrise image.
The decoder is , a neural network that maps a latent vector to an image . The final layer uses a Sigmoid activation to keep pixel values in [0, 1].
The decoder uses transposed convolutions (sometimes called "deconvolutions") to progressively upsample from a small spatial representation. Starting from the 2D latent code, a linear layer maps to a 128-channel 4×4 feature map, then three ConvTranspose2d layers upsample to 32×32. Each layer has a receptive field that grows with depth, allowing the network to capture both local texture and global structure.
Latent distribution
Reconstruction quality
Allowing a little randomness makes the generator more creative. Instead of mapping each image to a single point, the encoder maps it to a small cloud — and we sample from that cloud.
We sample using:
This lets gradients flow through the sampling step, since the randomness is isolated in .
Backpropagation requires differentiable operations. Sampling from a distribution is inherently stochastic and non-differentiable. The reparameterization trick rewrites the sample as a deterministic function of the parameters plus external noise, making the loss differentiable with respect to and .
A measure of how far the encoder's guesses are from a simple baseline (a standard normal distribution). The smaller the KL, the more "regular" the latent space.
The VAE optimizes the Evidence Lower Bound:
The reconstruction term wants accurate decoding. The KL term wants the encoder to stay close to the prior.
Information-theoretically, the ELBO can be interpreted via bits-back coding: encoding an image costs log p(x|z) bits, but we "get back" KL(q||p) bits because the decoder expects samples from the prior. The gap between the ELBO and the true log-likelihood is the KL divergence between the approximate and true posterior.
Can you find this sunrise in latent space?
Your sunrise:
z = (0.00, 0.00)