3D object with Holdout, but with transparent shadows and lights around

That's something I do for ages but using my own piece of code :

1. I put something above a diffuse plane, and render to get Image A

2. I render the diffuse plane with the same lighting, but without the object, and get Image B (the background)

3. I set the diffuse plane to "holdout" and render to get Image C (the mask)

4. Then I use my piece of code to compute a bitmap D = A - B

I use an alpha operation that I invented, that computes the inverse of alpha blending. Suppose x = alphablending (y ; z )

my code compute y from x and z. To do so I use a linear projection p in the color space of z from x onto the boundaries of the color space (the box [0;0;0]->[1;1;1] ) and then I compute the alpha value a= dist(z,x)/dist(p,x) so x= a * (p-z) + z

y= [alpha: a; color:p] (of course if x ==z then a =0) The new image D contains only shadows and light that were added to the background by the object, but unfortunately some pixels from A that should be opaque (solid pixels from the object that are similar colored to the background) are not fully opaque.

5. So I use the alpha of mask C on image A, and I put image D behind.

Now I have a cool effect of reflections and lighting on a plane, but with alpha blending so it can be placed onto any background.