(Shaders / NPR / BNPR) Separate Core-Shadow (Self-Shadow) with Cast-Shadow in Eevee

When you're learning how to paint, one of the most important concepts to learn is the types of shadow to allow you to work in phases -- such as Core-Shadow (the side that light doesn't hit), Cast-Shadow (the shadow cast from one object / extension to another), and Ambient Occlusion (you know it).

When you're working with stylized art, it's very important to be able to control all three individually, just as you would control them in digital painting by working in passes / layers.

Unfortunately, while controlling AO is easy enough in Blender, there is no real way to separate Cast-Shadow and Core-Shadow / Self-Shadowing. I've done my best to find a way to do it, but so far it's been impossible to do it on my own. For instance, the Emission shader allows casting shadows, but it doesn't itself accept shadows cast on it while simultaneously ignoring its own core-shadow.

Recently, however, I bumped into a repository where Unity has achieved exactly that.

github.com/unity3d-jp/RaytracedHardShadow

I'm hoping somebody would be willing to add this feature to Blender, because it could be of serious help in many BNPR projects in a whole lot of ways.

EDIT:

Based on feedback provided so far, thought I'd update this post on how all this could be possible from a user-interface perspective.

Also, while the Unity script / plugin has a higher GPU requirement, we may not specifically need it. I'll explain how.

General Functionality

I request this for both Cycles (for precise raytrace shadows) and Eevee (for faster calculations). Cycles can be used for major productions where errors need to be controlled. However, if it is difficult to create such a feature, going Eevee-first is not a bad idea considering this feature will mostly be needed for non-photorealistic rendering (such as toon-shading).

This is quite similar to, but not the same as, the Shadow Catcher function which ignores the Core Shadow in favor of object-based Cast Shadow. The Shadow Catcher, however, is limited in the fact that it does not allow you to self-cast shadow, nor let one shadow-catcher cast a shadow onto another shadow-catcher. These barriers will need to be overcome.

Lights (Global)

The individual Light settings in Eevee already have the ability to separate Core-Shadows with Cast-Shadows, except they don't give you much control. Unchecking the "Shadows" wil just switch off the Cast-Shadows while keeping the Core-Shadows on the object.

What we essentially need is an option to reverse this, where checking / unchecking an option will let you have the Cast-Shadows but not the Core-Shadows.

This could, however, create complications based on how lights and shadows are calculated in Blender, and considering the object IS in shadow by default, and that cast-shadows merely mask out the light. In which case, such a setting in the Lights themselves may not be ideal, lest we overcomplicate things under-the-hood.

In which case, we could instead achieve this via:

Material Settings (Per-Object)

An additional Cast-Shadow shader node that allows one to ignore the core-shadows while still respecting cast-shadows. (Optionally, we may also need a separate "Core-Shadow Only" shader in a similar fashion -- or just a switch that gives us one or the other, or something). Theoretically, the Shadow Catcher should also still be able to catch shadows cast by this shader.

This may come in the form of an emmissive material that still casts shadows onto other objects, while also rejecting its own core-shadow. There can be the following settings for this feature:

1) Receive Cast-Shadow -- If unchecked, this will behave like a standard Emission shader, but checking it will allow this object / material to receive shadows cast upon it by other objects.

2) Self-Cast -- This allows the object to cast shadow upon itself while still ignoring the core-shadow. What this means is that, if you have a single body with a torso, neck and head, and if you place a Butterfly Light, then the chin would cast a shadow on the neck. This wouldn't happen if the option is unchecked, therefore to cast a shadow on the neck you'd need a separate object.

3) Softness -- Be default, the shadows can be hard-edged, like a cartoon. This slider, however, should allow you to control the softness of the shadows if desired.

4) Shadow Only / Shadow Dot -- Think "Normal" node's Dot / Black-and-White output, but for Shadows. Reason being, this can serve as a mask to allow heavy customation of the shadow as needed. This is influenced by all the settings mentioned above, but not the ones below.

5) World Color -- This influences how much the shadows respect the World settings. Leaving it to 0 would always leave the shadows Black (and only influneced by the followinng optional setting) and influenced only by the lights in the scene. If the pushed to 1, the shader respects the world-settings much like a Diffuse shader would, thus the shadows following the world color, etc.

6) (Optional) Shadow Color + Transparency -- If the World Color is left to 0, it's likely because you want control over your shadows. Therefore, this option allows you to select color your shadows need to be, and how transparent you want them to be. (Perhaps possible to just plug in an alternative tinted texture in here, just in Principled's Base Color slot.)

Creating a shader, however, can be an arduous task. So part of this can be another feature which may be easier:

Render Pass

We already have a Shadow pass, and Blender can already distinguish between Core and Cast shadows. It would be great to have a specific Core Shadow and Cast Shadow passes that can eventually be fused with Diffuse pass in order to create cartoon-ish flat renders, or even customized to one's liking.