VSE to Texture

Video Sequencer to Texture ... yes I know... I recently took another look at Nuke, which I used to use but stopped being able to justify paying for because a lot of my work uses non-standard resolutions and to export to them I needed to maintain the top tier license which was prohibitively expensive.

Regardless, the whole time I was in Nuke I was thinking about how much of the workflow would theoretically be reproducible in blender if I could construct compositions in the VSE and then directly pipe them to textures for objects within the 3D viewport.

I know, the VSE is the last step, and this is probably currently impossible due to the way that data flows through blender, but man, it would be soooo cool.

I work in live entertainment and immersive media. We use blender A LOT, not just for modeling work, but also to exchange project documents like set designs and to do quick previz. Blender's media framework is honestly better in a lot of ways than nuke's (it certainly supports more asset types). If we were free to integrate the VSE in more novel ways into the workflows we build in Blender...

sigh... it would be glorious.

EDIT: How it might work:

Currently, each Scene container in Blender has its own statically routed Compositor and VSE contexts.

As I see it the principal changes needed would be for VSE Sequences (and perhaps Compositor Graphs) to become their own Objects which can be created, duplicated and deleted.

The output of a VSE Sequence Objects would have an internal time domain that can be sampled,sub-sampled,looped similar to how a movie clip, when evaluated, returns a texture for the given time.

This behavior more or less already exists, however the texture routing is currently statically bound. The crux of this proposal is that these textures could then be used in shader graphs or compositor graphs or downstream VSEs as strips.

I understand that this creates the potential for recursive loops in the way that textures are evaluated, but it would be a powerful way to work.

To support this workflow it would also be great to not have the "Scene" be something that globally applies to all viewports.

Imagine that you have a scene where a character is watching something on a monitor. For fun let's say it's a CRT that casts a glow into the scene.

The material being used to create the CRT has a "VSE Sequence Object" as its texture source. It looks like Suzanne bobbing in place with occasional glitches and static.

The animator 'pins' the current scene view of the character looking at the monitor so they can keep an eye on it and make some tweaks.

1. They open up a VSE with the Glitchy Suzanne Sequence finds the Scene strip with Suzanne in it and opens that up in a 3D editor to make adjustments to the movement.
2. They then make adjustments to the glitch texture footage (ie: making it more or less intense)

These adjustments could be getting made while seeing the result on the final output.

edit2: Also, just to be clear, this isn't some "nuke-centric" workflow. I went looking at Nuke again recently because of its modular texture routing and re-routing graph. Within our current pipeline we actually wind up doing most of this kind of stuff in TouchDesigner with extra shaders and stuff thrown in to do esoteric stuff that we can't do natively. TD is great, but it does not have great 3D viewport tools or a built in timeline (like blender does). We tend to use blender as our defacto standard for moving 3d assets for review and texture processing. Our current modeling stack is primarily Houdini, and our compositing work happens in TD when possible, but we also still wind up in AE more than we would like.