Multi View Mesh Modelling and Refinement by Inverse Rendering
Traditionally, multi-view stereo algorithms are fully automatic. They reconstruct point clouds which suffer from detail vs noise tradeoff.
On the other hand, there are user guided tools like fSpy which are limited to one view and cannot handle organic shapes well.
The paper "User-guided 3D reconstruction using multi-view stereo" combines the advantages of both by guiding a multi-view reconstruction with a quad model made by the user to achieve much cleaner reconstructions.
Via cameras calibrated with structure from motion like in prior work, the user models in the 2D plane of a reference image. The other views allow the user to adjust the depth of the vertices as seen in the reference view.
One could implement the mesh modelling in such system in Blender by modifying the UV map editor.
To use extra information like shading cues to further refine the mesh and acquire textures like albedo and normals, an developer friendly path is to modify EEVEE to estimate gradients with respect to the vertex positions by randomly perturbing them like in arxiv.org/pdf/2404.09758 to do stochastic gradient descent. One can refine the mesh further via differentiable rendering such as in Mitsuba 3.
0 Comments