Chapter 29: Pictures from noise
The trick behind AI image generators, small enough to watch.
Before streaming, a TV pulled its picture out of the air with an antenna. Tune it to a channel where nothing was being broadcast and the screen filled with a hissing blizzard of flickering specks. People called it static: a picture made of pure randomness, with no pattern in it at all.
Here is a strange fact about AI image generators. Ask one for a fox in a spacesuit and it does not start with a blank page. It starts with a screen full of static and cleans it, a little at a time, until what is left is a fox in a spacesuit. Nobody drew that fox, and it was not hiding in the static either.
The method is called diffusion. In this chapter you will run a real diffusion model, a network that makes pictures by removing static step by step, and it is small enough that you can watch every step.
Pictures made of dots
A phone photo is millions of numbers, three for every pixel. That is too many to watch. So our pictures are simpler: each one is a cloud of 2,000 dots that draws a shape, and each dot is only two numbers, how far across it sits (x) and how far up (y).
There are five pictures: a smiley, a heart, a spiral, a star and the word HI. Everything a real image generator does to pixels, we are going to do to dots.
Ruining a picture is easy
Take the heart. Give every dot a tiny random kick, a different kick for each dot. Now do it again. And again. With each round the dots drift further from where they began, and the picture forgets a little more of itself.
Scientists call randomness like this noise, even when there is nothing to hear. In this chapter "noise" and "static" mean the same thing. We will do 50 rounds and call the round number the noise level: level 0 is the clean picture and level 50 is as ruined as it gets.
You are about to ruin five different pictures in exactly the same way. At noise level 50, will you be able to tell which one was the heart and which was the star?
By level 50 only 3% of each dot's real position is left; the rest is static. Every picture ends as the same shapeless blob. Hold on to that, because it means one screen of static could become any picture.
Ruin a picture
Drag the slider, or press ruin it. Every step gives each of the 2,000 dots a small random kick. Find the level where you can no longer tell what the picture was. Then leave the slider at 50 and switch pictures.
each dot = 1.00 × its real position + 0.00 × static
loading the trained network…
What just happened
Each step did two small things to every dot: it pulled the dot a tiny bit toward the middle of the screen, then gave it a random kick. The pull is there so the cloud does not swell up. It stays the same size while its shape drains away.
The orange line under the buttons keeps score. At level 0 a dot is 1.00 × its real position and 0.00 × static. At level 50 it is 0.03 × its real position and 1.00 × static, which for all practical purposes is pure static. (The two numbers are not supposed to add up to 1. They are chosen so that the cloud keeps its size.)
That line is also a shortcut. Fifty small random kicks add up to one big random kick, so nobody has to walk through the levels one at a time. Pick any level, look up its two numbers, and mix:
noisy dot = keep × real position + noise × static
Training is about to use that recipe many thousands of times a second.
Un-ruining it is the real skill
Anyone can scribble over a drawing. Getting the drawing back is another matter. Once a picture has dissolved into the blob, the exact spot each dot came from is gone for good. Nothing can recover those particular 2,000 dots.
But you could make a heart again, if you knew what hearts look like. That knowledge is what we are going to train into a network. It will not store the dots. It will learn where the dots of a heart tend to be.
The training game: guess the static
Here is the whole game. It is played one dot at a time.
- Pick a real dot from one of the five pictures.
- Pick a noise level from 1 to 50.
- Make some static and mix it into the dot with the recipe. We made the static ourselves, so we know exactly what it was.
- Show the network the ruined dot, its noise level and which picture it came from. Ask one question: what static did I mix in?
- The network answers with two numbers. The distance between its answer and the real static, squared, is the loss. Backpropagation from chapter 14 works out how to nudge every weight so that the distance shrinks.
Play that game 30,000 times, with 1,024 dots in every turn, and the network becomes a very good noise-spotter.
Why ask for the static and not for the clean dot? Two reasons. First, the right answer is free. We created the static, so every turn comes with a perfect answer key and no person ever has to label anything. A language model gets its answers the same way: the next word is already sitting there in the text (chapter 23).
Second, the two questions are really one. Look at the recipe: if you know the noisy dot and the static, a little arithmetic gives you the clean dot, and the other way around. The researchers who first made this work well tried both and got better pictures when the network was asked for the noise.
The network itself contains nothing you have not met. It has 25 inputs, three hidden layers of 128 ReLU neurons like the ones in chapter 17, and 2 outputs: 36,610 weights in all. The inputs are:
- the dot's x and y;
- the same x and y passed through sine and cosine waves at four zoom levels, 16 more numbers (a common trick that helps a small network draw thin, sharp lines);
- the two numbers from the recipe, which tell it how noisy things are;
- five on/off switches that say which picture the dot belongs to, one-hot like the answers in chapter 10.
Making a picture
Now play the game backwards.
- Start with 2,000 dots of pure static. No picture went into them.
- Tell the network "this is noise level 50" and ask what static is in each dot.
- Take out a little of the static it guessed, just enough to bring the dots down to level 49.
- Ask again at level 49. Then 48. Keep going until you reach 0.
Before you try it: the network's weights never change, and no dice are rolled during the 50 steps. If you run it twice, starting from two different screens of static, what do you get?
Different static in, different dots out. But every dot still lands on the heart, because the heart is the thing the network learned.
Un-ruin it
These 2,000 dots are pure static. Press un-ruin it and the network cleans them, one noise level at a time. When it has finished, drag the slider to scrub back and forth through the steps. Then press again a few times and compare the hearts.
loading the trained network…
What just happened
The network did all of it. While the dots were moving, nothing on this page looked at a drawing of a heart. Each step, every dot was handed to the network on its own, and the network answered "here is the static I think is in you". 2,000 dots times 50 steps is 100,000 runs of the network for one picture.
The picture arrives in stages. Scrub slowly through the steps. For the first ten or so the cloud hardly seems to change. Then it turns lumpy, a hazy heart shows up around step 30, and the last ten steps pull the haze into a sharp line. Rough decisions come first ("this dot belongs on the left side") and fine ones last ("exactly here on the curve").
The variety comes from the static. Press again and you get a heart every time, but never the same heart. The weights were the same on every run and so were the 50 steps. The only thing that changed was the static you started with.
Honesty note: many image generators also stir a pinch of fresh static back in after each step, so they roll dice all the way through. Ours uses a version of the method that does not, which is why here the starting static decides everything.
It learned the pattern, not the dots. Switch on show the 2,000 training dots, then zoom in on the tip. The blue dots are the ones the network was trained on. The orange ones are not copies of them: they are new dots scattered along the same line. Press again while you are zoomed in and the orange dots land somewhere new, while the blue ones never move.
We measured it, too. A generated dot lands, on average, 0.013 away from the nearest training dot, on a heart about 3.4 wide. Brand-new real dots, sprinkled along the heart the same way the training dots were, land 0.014 away. The network's dots are as new as new real dots would be.
Why not one big step?
Fifty steps means running the network fifty times per dot. If the network knows what static is in a dot, why not remove all of it at once?
Suppose we skip the patience: from pure static, ask the network once and remove all the static it guesses in a single giant step. What comes out?
Try it below and see. The explanation follows the widget.
How many steps does it need?
Same network, same starting static every time. The only thing you change is how many steps the trip from level 50 down to level 0 is cut into. Start with 1 and work your way up.
loading the trained network…
What just happened
One step gives a scattered mess. Two steps give a smudge. Five give a blurry heart, ten are close, and fifty are sharp.
Think about what the network is being asked at noise level 50. The dot it is shown is nearly pure static, so that dot could belong almost anywhere on the heart. Forced to give a single answer, the best the network can do is a kind of average over every place the dot might belong, and the average of all the points on a heart's outline is somewhere in the empty middle. A giant step sends every dot toward a bad guess. (In a single step, small mistakes in the guess also get hugely magnified. That is the scatter you saw.)
Small steps avoid the trap. The first guess is vague, so you trust it only a little: remove a sliver of static and stop. Now the dots are slightly less noisy, and the same question has a slightly sharper answer. Fifty slivers later the answers have gone from "somewhere around the middle" to "right here, on the left curve".
The prompt
So far every run made a heart, because we always set the "heart" switch. Time to use the other four.
Same static, different prompt
Pick a prompt and watch what the static turns into. Then pick another one. Every run starts from the same static, the one in the small screen on the right. Only the prompt changes.
loading the trained network…
What just happened
Remember the first widget: every picture ruins into the same blob. So the static cannot be what decides the picture. Here, the same 2,000 specks became a star or a spiral depending on one thing: which of the five switches was on.
Giving a network extra information to steer what it makes is called conditioning. Our network learned what each switch means the ordinary way. During training every ruined dot arrived with a label saying which picture it came from, so the network learned five different answers to "what static is in this dot?", one per label.
And notice that it is one network. A single set of 36,610 weights holds all five pictures.
From dots to pixels
Millions of numbers at once. Our network looked at one dot at a time, and a dot is two numbers. It never needed to know where the other dots were. So each dot was really its own tiny two-number "picture", and the heart is what you see when 2,000 of them are made side by side. A real image is the other way around: it is a single "dot" made of millions of numbers. A 1,024 × 1,024 color picture has 3,145,728 of them, a red, a green and a blue value for every pixel. The network has to guess the noise in all of them together, because pixels depend on each other: the left eye has to match the right one. That takes a far larger network, with hundreds of millions to billions of weights, built from parts you have met: the convolutions of chapter 28 and the attention of chapter 22.
A compressed picture. Guessing the noise in three million numbers, dozens of times over, is slow. So most generators do not work on pixels directly. A separate network first learns to squeeze a picture into a much smaller grid of numbers, called a latent, and to un-squeeze it again. All the ruining and un-ruining happens on the small grid, and the result is un-squeezed into pixels once at the very end. The first Stable Diffusion squeezed a 512 × 512 picture, 786,432 numbers, into 16,384 numbers: 48 times fewer.
A prompt made of words. Our prompt was one of five switches. A real prompt is a sentence, and there is no end to the sentences people can write, so switches will not do. Instead the sentence goes through a language network, a relative of the ones in chapter 23, which turns it into lists of numbers that capture its meaning, like the embeddings in chapter 20. Those numbers are fed to the noise-guessing network at every step, right where our switches went. To learn what they mean, the generator plays the same training game on hundreds of millions of pictures that come with captions: here is a ruined picture, its noise level and its caption, so what static was mixed in?
Honesty note: "diffusion" is now a family of methods, not one recipe. Newer generators mix in the noise differently, take far fewer steps, or use a close cousin called flow matching. A few build a picture piece by piece instead, the way a chatbot writes. Start from noise and clean it up, steered by a prompt, is the idea that set off the boom in AI pictures, and it is still at work in many of today's image and video generators.
Where the idea came from
2015. Jascha Sohl-Dickstein and three colleagues at Stanford and Berkeley published the core idea: destroy data slowly with noise, and train a network to undo the destruction. They borrowed it from physics, where diffusion is what a drop of ink does in a glass of water: it spreads until no trace of the drop's shape is left. The method worked, but its pictures were small and fuzzy, and for a few years it stayed in the background.
2020. Jonathan Ho, Ajay Jain and Pieter Abbeel at Berkeley showed how to make it work well. Their paper trained the network with the plain guess-the-noise game you just read about, and its pictures rivaled the best that any method could make at the time.
2022. Three ingredients came together: the 2020 recipe, steering with text, and the squeeze-it-first trick, published by researchers in Germany at the end of 2021. Within a few months the systems most people have heard of appeared: DALL·E 2 from OpenAI in April, Imagen from Google in May, and Stable Diffusion in August, with its weights released so that anyone with a decent gaming computer could run it at home. Midjourney opened to the public that July as well, though its makers have not published how it works inside.
Two things worth thinking about
Whose pictures did it learn from? Our network learned from 10,000 dots that we made ourselves. The big generators learned from enormous collections of photographs, paintings and drawings made by people, mostly gathered from the internet without asking those people first. (A few companies say they train only on pictures they own or have paid to use.) Like our network, a generator mostly learns patterns instead of storing pictures, though researchers have shown that it can sometimes reproduce a training image almost exactly. Whether artists and photographers should have a say in this, get credit, or be paid is being argued over right now in courtrooms and parliaments. Thoughtful people disagree, and it is worth working out what you think is fair.
Can you still believe a picture? A convincing photo of something that never happened used to take real skill to fake. Now it takes a sentence. That does not mean every picture is fake. It means a picture, on its own, is no longer proof. The habits from chapter 25 apply to pictures too: ask where it came from, who is showing it to you and why, and whether anyone you trust can confirm it.
Train a diffusion model yourself
This is diffusion.py, the script that trained the network on this page. It needs only numpy. Press run and it trains right here, printing its pictures as text: first a training picture, then the same picture being ruined, then pictures it generates from pure static once training is done.
Python inside a web page is slow, so here the script switches itself to a quick setting: two shapes (a heart and a star) instead of five, a smaller network, and 4,000 small training steps. That took 16 seconds on the laptop this chapter was written on, and a slower machine will need longer. Expect rougher pictures than the ones above. On your own computer the script does the full job, 30,000 bigger steps, in a few minutes and writes diffusion-weights.json, the file this page loaded.
python3 diffusion.py (the full run) or python3 diffusion.py --quickTwo things to look for in the output. First, the loss. Guessing "no static at all" every time would score 1.0. The network gets down to about 0.5 quickly, then improves only slowly, and it never gets anywhere near 0. That is not a bug. Static that knocks a dot sideways off its line is easy to spot, but static that slides a dot along the line leaves it sitting on the line, looking innocent. No network can guess that part, so some loss always remains.
Second, after each generated picture the script prints how far its dots are from the nearest training dots. That is the check we quoted earlier: a copy would score 0. The quick run scores higher than the full one because its pictures are fuzzier, not because its dots are newer.
A diffusion model learns one small skill: look at something noisy and say what the noise is. Use that skill over and over on pure static and a picture appears that never existed before. The training data decides what is possible, the prompt decides what you get, and the static decides the details.
Teach it a shape of your own. In the script, find PEN_STROKES and add a line such as "bolt": [[(0.3, 1), (-0.5, 0), (0.2, 0), (-0.3, -1)]], which is a single pen stroke through four corners. Then change ["heart", "star"] near the top to ["heart", "bolt"] and run it. Does a lightning bolt come out of the static? Now design a shape that needs two strokes.
During training, what is the network asked to do?
We ruin real data with noise we made ourselves, then ask the network what that noise was. Since we made the noise, the right answer is always known.
Two people type exactly the same prompt into an image generator and get two different pictures. What is the main reason?
Same weights, same prompt, different starting static. The prompt decides the subject and the static decides the details.
Why does a diffusion model remove the noise in many small steps instead of one big one?
The weights never change while a picture is being made. What changes is the question: with a little less noise each time, the possible answers narrow until only a sharp picture fits.