Not strictly Blender related, however, I thought this may be interesting and possibly relevant in the future. Imagine being able to quickly sketch backdrops to images (possibly even HDRIs) rather than having to rely on scans? Or being able to quickly sketch out a few dozen faces and the AI turning them into assets which can be added as textures to models in a crowd simulation?
tl;dr; The video shows deep AI being used to change a rough sketch into a passably realistic face. It also includes a clip from an Nvidia prototype which will do the same for backgrounds.
p
peter.boos
*
@kidtempo
* If an AI would learn how to sculpt (or alter a mesh) so the shadows and look the same by altering light and mesh.. we're there.
That probaply wont take to much years i think.
1
k
kidtempoAuthor
*
@peter.boos
* I think there is already some degree of sculpting at play here.
For faces, I expect they must be working from some sort of base mesh and then being modified by the AI - as you can see from the video thumbnail, the sizes of the chin etc. varies with the face shape. Also, I expect the examples in the video probably already had default lighting set up. If a tool using the technique were being used to generate faces, you'd probably want it to bake textures without directional lighting information anyway as you'd add it in your scene.
For landscapes, while the AI could be doing simply 2D painting, I suspect that there is a 3D component in there (if only at a primitive level). For backdrops where you just want some hills/mountains, instead of fiddling with the random landscape generator it could be as simple as sketching what you want and the AI creating and texturing the mesh in real-time. It would not surprise me that the Nvidia tool demoed in the video couldn't be used to generate a usable mesh (of course, that's not to say that it
does
do that - it looks like a proof-of-concept so I'm saying that it's something that is probably could do that with relatively little additional effort)
Taking this into the realm of
possible
future implementations; imagine being able to sketch/block out the environment for a scene (especially if the real focus of your scene is the character and the backdrop is just something which you want to be
there
, but not spend any real time working on) and having the AI fill it out the details.
For example: when making a urban backdrop just using the annotation tool (or similar) to sketch in buildings, balconies, aircons, pipes, cables, etc. and the AI being smart enough to not just work out how to add the meshes (assuming a kitbash asset library is available) in the correct place and orientation, but also extrapolate the directions to other buildings in the scene. You sketch only 2, 3, 4... buildings, and the AI mashing up those designs in order to create unique variations for all the other buildings as required.
In the area of kitbashing. AI neural networks are good at mashing up the components they are provided in order to provide lots of variations. The user only has to train the AI by telling it which combinations are acceptable, and which aren't. Starting from a relatively small base of model components, an AI could offer the user a huge library to choose from, and also texture them appropriately.
Something like modelling a aircon unit with a few components (like vents, coupling, power switches, etc.) and then with input from the user (e.g. more/less "urban grunge", "hi tech shiny", "emission lights", "buttons/switches", "gauges/dials", etc.) the AI can generate variations. By manually texturing a few examples, an AI can then replicate that style (again using training data which the user rates).
I can see these types of features/tools being in high demand for development tools like the Unreal engine. Users currently spend a lot of time modelling inconsequential details which nobody notices if it's there, but everyone notices if it isn't. It would be highly beneficial to be able to spend less time modelling inconsequential background objects, and be more like an art director or set designer. Picking/creating a few sample objects similar to what is required and then letting the AI create variations - the artist then makes design decisions like "longer", "more rusty", "less shiny", etc. while the AI takes that training information and offers alternatives.
I reckon we're still a few years away from the type of tools I'm describing, but not
so
far in the future. As the guy in the video says
"What a time to be alive".
3 Comments
Not strictly Blender related, however, I thought this may be interesting and possibly relevant in the future. Imagine being able to quickly sketch backdrops to images (possibly even HDRIs) rather than having to rely on scans? Or being able to quickly sketch out a few dozen faces and the AI turning them into assets which can be added as textures to models in a crowd simulation?
tl;dr; The video shows deep AI being used to change a rough sketch into a passably realistic face. It also includes a clip from an Nvidia prototype which will do the same for backgrounds.
* @kidtempo * If an AI would learn how to sculpt (or alter a mesh) so the shadows and look the same by altering light and mesh.. we're there. That probaply wont take to much years i think.
* @peter.boos * I think there is already some degree of sculpting at play here.
For faces, I expect they must be working from some sort of base mesh and then being modified by the AI - as you can see from the video thumbnail, the sizes of the chin etc. varies with the face shape. Also, I expect the examples in the video probably already had default lighting set up. If a tool using the technique were being used to generate faces, you'd probably want it to bake textures without directional lighting information anyway as you'd add it in your scene.
For landscapes, while the AI could be doing simply 2D painting, I suspect that there is a 3D component in there (if only at a primitive level). For backdrops where you just want some hills/mountains, instead of fiddling with the random landscape generator it could be as simple as sketching what you want and the AI creating and texturing the mesh in real-time. It would not surprise me that the Nvidia tool demoed in the video couldn't be used to generate a usable mesh (of course, that's not to say that it does do that - it looks like a proof-of-concept so I'm saying that it's something that is probably could do that with relatively little additional effort)
Taking this into the realm of possible future implementations; imagine being able to sketch/block out the environment for a scene (especially if the real focus of your scene is the character and the backdrop is just something which you want to be there , but not spend any real time working on) and having the AI fill it out the details.
For example: when making a urban backdrop just using the annotation tool (or similar) to sketch in buildings, balconies, aircons, pipes, cables, etc. and the AI being smart enough to not just work out how to add the meshes (assuming a kitbash asset library is available) in the correct place and orientation, but also extrapolate the directions to other buildings in the scene. You sketch only 2, 3, 4... buildings, and the AI mashing up those designs in order to create unique variations for all the other buildings as required.
In the area of kitbashing. AI neural networks are good at mashing up the components they are provided in order to provide lots of variations. The user only has to train the AI by telling it which combinations are acceptable, and which aren't. Starting from a relatively small base of model components, an AI could offer the user a huge library to choose from, and also texture them appropriately.
Something like modelling a aircon unit with a few components (like vents, coupling, power switches, etc.) and then with input from the user (e.g. more/less "urban grunge", "hi tech shiny", "emission lights", "buttons/switches", "gauges/dials", etc.) the AI can generate variations. By manually texturing a few examples, an AI can then replicate that style (again using training data which the user rates).
I can see these types of features/tools being in high demand for development tools like the Unreal engine. Users currently spend a lot of time modelling inconsequential details which nobody notices if it's there, but everyone notices if it isn't. It would be highly beneficial to be able to spend less time modelling inconsequential background objects, and be more like an art director or set designer. Picking/creating a few sample objects similar to what is required and then letting the AI create variations - the artist then makes design decisions like "longer", "more rusty", "less shiny", etc. while the AI takes that training information and offers alternatives.
I reckon we're still a few years away from the type of tools I'm describing, but not so far in the future. As the guy in the video says "What a time to be alive".