what's denoiser is better? (the benchmark edition)

Hi, guys!

Recently I had the curiosity what denoiser get the better results. I couldn’t find on google any information about that, except that optix faster. Even on the the official documentation. So I decided to check it myself.

How?

There is a software developed by Netflix called Vmaf. Basically, Its use artificial intelligence and machine learning to calculate the perceptual quality of compressed video in comparison to the original source file.

Its not so far from what we need. denoisers often used to replace high sample rate. So if I will compare a high samples render as a source file to a low samples render with a denoiser, I can actually get some numbers. So lets go benchmarking!

(boring details about the procces:

I exported 8bit PNG lossless images of every render the high quality renders made using Sheepit! Render Farm. Then I stitched them together to a lossless x264 video sequence via Kdenlive. Then I created video at the same length, from the high samples render. Then I used FFmetrics to calculate the Vmaf score. I used the 'Vmaf_v0.6.1.json' model)

So what denoiser is the best?

Short answer: Intel Open AI Denoiser.

Long answer:

starting with an easy one, I chose the godette character by pablo dobarro. Its a very cartoony render, and the denoisers should really shine in this benchmark. The source file is a 10240 samples render without denoising.

Here, their is a clear winner. Intel denoiser win in all sample counts.

But what will happened if we insert subsurface scattering to the equation?

I chose the ‘Monster Under The Bed’ file by Metin Seven. The source file is a 10240 samples render without denoising.

There is no much of a difference, Except in the high samples.

In this file and in the last ones you will notice that the renders without denoiser start to challenge the denoised renders, score-wise. So I should remind you; Its not an appealing benchmark. Its just comparing how much details we have, compared to the high quality render.

(in other quality measurement tools, like PSNR and SSIM, the advantage of the denoised images are more consistent. But I chose to use Vmaf, because its considered the latest and greatest in video quality benchmarking. I’m not an expert in video compression, at all. But its look like this is the new standard)

OK, now lets try some highly detailed, photo-realistic images. Like ‘Still Life’ by Erindale Woodford. The source file is a 5120 samples (just because its a very heavy scene) render without denoising.

First thing you will notice, its that the non-denoised images are very close to the optix ones, And they win in high samples. But its not so important (and maybe Vmaf even don’t know how to handle noise like that. Denoised images are more close to the compressed videos situations that Vmaf deal with them regularly.)

what’s important is, that in more complex scenes, the gap between Open AI and Optix just widening.

The lost one, is just another case of complex geometry. Tree, leaves and grass by Simon Thommes.

There it is. In all scenarios, Intel Open AI Denoiser crush it. And its just on blender 2.93, wich means its version 1.3 of the denoiser. In blender 3.0, the denoiser will update to 1.4. and the developers keeps improving it.

Can’t wait.