How AI photo restoration works

Training on damage, on purpose

The training method is simpler than it sounds. Take an enormous set of clean photographs, then deliberately ruin copies of them: blur them, fade them, add scratches, throw away resolution. Now show the model the ruined version and score it on how closely its output matches the clean original it never saw.

Do that a few hundred million times and the model develops a working understanding of what damage does to a picture, and therefore how to undo it. It has essentially studied the difference between before and after across more photographs than a human retoucher could examine in a hundred lifetimes.

Why faces get their own model

Human vision is extraordinarily sensitive to faces. We detect something wrong in an eye at a glance while ignoring an equivalent error in a brick wall, which means a general-purpose restorer that is good enough for landscapes reads as unsettling on portraits.

Dedicated face models solve this by learning facial structure specifically: that an eye contains a pupil, an iris and a catchlight, that eyelashes radiate in a direction, that the boundary between lip and skin behaves differently from the boundary between hair and background. Given a smudge where an eye should be, a face model rebuilds an eye instead of a sharper smudge.

Plausible is not the same as true

This is the part worth understanding properly, because it determines what you can responsibly do with the output.

When the model meets an area where your photo holds no information, it does not stop. It fills that area with what such an area usually contains. Given eight grey squares where a face was, it produces a face. That face is a statistical average of faces which fit the surrounding context, and it may not be the person who was standing there.

For a family album this is a good trade, because a nearly right grandmother is far better than a cracked grey smear. For identification, evidence, or any claim about what somebody looked like, the output carries no authority at all.

What it does well, and what it fumbles

Reliable: fading and contrast recovery, moderate blur, scratches and creases over textured backgrounds, resolution increases up to about 4x, and colorization of skin, foliage and sky.

Unreliable: heavily degraded faces, small text and numbers, patterned fabric, jewellery and other fine repeated detail, and any colour that context cannot imply. A dress that was emerald green renders as the same grey as a navy one, so the model picks whichever it has seen more often.

It has improved faster than most people expect

Restoration output from 2020 has a look to it: waxy skin, symmetrical doll eyes, backgrounds smoothed into paste. The face-specific models that arrived after it fixed most of that, and the gap between a good automated restoration and a mediocre human one has closed considerably.

Which means an old photo you gave up on three years ago is worth another attempt. The file has not changed, but what can be done with it has.