Screen Space Curvature in Workbench Engine for better surface detail visualization.

WHY WE NEED IT

I'd like to propose adding Screen Space Curvature overlay post-process. It's main purpose is Sculpting Mode where typical shaders aren't enough to properly visualize high-frequency surface details. Most of sculpting apps include such feature. Take a look at this example:

Currently we can achieve similar effect in Clay Engine using Edge and Cavity sliders. It works good but require a lot of computation to produce noise-free image needed for visualizing small details. This isn't optimal for sculpting which already is heavy task for GPU. I propose to add lightweight solution. Mind that current approach from Clay Engine and mine can work in conjunction. The first one will be perfect for large and medium forms (using fewer samples) and the second will bump up high frequency details.

HOW IT WORKS (TECHNICAL)

My solution is based on Prewitt Operator , a common image processing technique used for edge detection.

1. First we need View Normal render pass.
2. We sample it four times, with one pixel offsets in four direction.
3. Red channels from images shifted left and right are subtracted.
4. Green channels from images shifted up and down are subtracted.
5. We add everything together.

Now we have image where flat areas have a value of 0 , concave are -1 and convex are 1 . Before overlaying it onto Viewport it needs to be remapped, flat surfaces should have a value of 0.5 . We can remap it to ( 0, 1 ) range using mix(0.5, 1.0, a) function, however we should increase contrast here to let users amplify curvature effect. To get image in ( -0.5, 1.5 ) range we don't need to use mix() function, we simply add 0.5 to result of Prewitt Operator.

Calculating curvature is almost done. You may notice strong outlines on object silhouettes. This is caused by step changes in normal values. They are distracting so we need get rid of them using 2 methods:

1. Find and exclude object outlines using Object ID render pass and modified Prewitt Operator.
2. Find and tone down extreme concave areas in curvature image.

Together they will generate a mask we can use to mix our curvature and neutral 0.5 value. Finally we composite refined curvature image with Viewport using Overlay blending mode. Blending factor should be exposed to users in shading Pop-over and have default value of 0.5.

This node graph below shows whole curvature shader. It can be implemented with few lines of code and will work super fast. I combined View Normal and Object ID passes into one image to reduce texture samplers. Feel free to download and play with .blend file.

Blend File