Improve VSE speed with full GPU pipline (optimized multi-frame computation graph)

Problem: Timeline Speed

On my machine (AMD Ryzen 7 5800X/16 Cores) and AMD Radeon RX 6600 XT on Arch Linux (latest drivers), a single unedited 4k30p Video cannot be played back in realtime. After two seconds the pre-computed frame cache is exhausted and from then on, the video continually stutters.

Testing with ffmpeg directly:

ffmpeg -hwaccel $HWACCEL -i input4k30.mp4 -an -f null /dev/null

Gives a speed value of 5-6 for HWACCEL=none/opencl/vulkan.

Repeating the same test with with an actual output reduces the speed value to 2-3:

ffmpeg -hwaccel $HWACCEL -i input4k30.mp4 -an -f null '/tmp/out%03d.bmp'

This means a smooth realtime playback and light editing should in theory be achievable on my PC (even without GPU support in this case).

Idea: General Node/Computation Graph and Optimization

A full GPU pipeline (decoding, processing and rendering/encoding) would probably do wonders for processing speed and usability. In the most common simple scenarios, the full pipeline could be compiled into a single GLSL shader (see Davinci Resolve, comment 1).

In Blender 5 the Compositor Modifier was introduced in the VSE. This makes the editor much more flexible and powerful. But now it seems that the whole input video -> final video pipeline is very fragmented (computationally and considering usability).

For blurring you could use

• e.g. sequencer/intern/effects/vse_effect_blur.cc (not optimized: slow sigma-dependent algorithm, CPU-only)
• or you could use the Compositor node (optimized: fast algorithm, CPU or GPU)

Thus we have a duplication of effort. I assume the former effect will eventually removed in favor of the latter approach?

All other code in sequencer/intern/effects and sequencer/intern/modifiers suffers from the same problem of being CPU-only. Thus even if up to that point all compositor modifier effects is handled on the GPU, it then has to be transferred to the CPU to do the additonal VSE processing.

Although the compositor graph-based approach is flexible, for the VSE new challenges appear, since an an effect like cross fade has two input images from different strips. I'm also unsure whether the current architecture allows for multi-frame effects within the same strip e.g. temporal noise reduction, optical flow estimation or simply averaging multiple adjacent frames (please tell me if I missed it!).

To reusably and efficiently implement such effects I think it makes sense to build a full computation graph over every single input file frames, the desired output frames and all intermediate steps.

For a video strip that is 100 frames long and an output of 100 frames without any other processing we would have 100 "unrelated" computation graphs.

output_frame[0] <- strip_a[0];
output_frame[1] <- strip_a[1];
...
output_frame[99] <- strip_a[99];

If a color transformation and some global exposure adjustment is necessary, the graph would look like this:

output_frame[0] <- exposure(some_value) <- color_space_transform_some_hdr_to_srgb() <- strip_a[0];

output_frame[1] <- exposure(some_value) <- color_space_transform_some_hdr_to_srgb() <- strip_a[1];

...

output_frame[99] <- exposure(some_value) <- color_space_transform_some_hdr_to_srgb() <- strip_a[99];

Note how the full pipeline is reused for all frames in this example and thus the full pipeline could be condensed into a single shader. Having a cross-fade between two clips would look like this:

output_frame[0] <- strip_a[0];
output_frame[1] <- strip_a[1];
...
                        V-------------- strip_a[49];
output_frame[49] <- cross_fade(0.4)
                        ^-------------- strip_b[49];

                        V-------------- strip_a[50];
output_frame[50] <- cross_fade(0.5)
                        ^-------------- strip_b[50];

                        V-------------- strip_a[51];
output_frame[51] <- cross_fade(0.6)
                        ^-------------- strip_b[51];
...
output_frame[98] <- strip_b[98];
output_frame[99] <- strip_b[99];

Note how there are now two different type of pipelines. One that simply copies 'frag_color = input_color' and one that condenses to a:

'frag_color = mix(intermediate_input_x, intermediate_input_y, factor) = mix(texel(strip_a), texel(strip_b))'

If we have the full output frame computation graphs available we could choose heuristically during execution time, whether it is faster to compute the output in multiple steps with intermediate textures or compile everything to a complete shaders (or a mixture of both approaches). This might depend on how often a specific filter sub-graph is used!

With this framework, also multi-frame effects are expressible. E.g. let's build a noise-reduction graph that looks 1 frame into the future and 1 frame into the past:

output_frame[0]   <- temporal_nr(some_parameters) <- depends on graphs NO_IMG,     strip_a[0], strip_a[1]

output_frame[1]   <- temporal_nr(some_parameters) <- depends on graphs strip_a[0], strip_a[1], strip_a[2]

...

output_frame[100] <- temporal_nr(some_parameters) <- depends on graphs strip_a[98], strip_a[99], NO_IMG

Note how (almost) each strip_a[*] is reused three times. If getting strip_a[*] would have been a complex computation (other slow filters used to obtain strip_a), we should cache the frames during rendering in a texture as long as there are other unprocessed nodes that depend on it.

If computing strip_a[*] is cheap (e.g. a simple color matrix added on top of some input file frames), we could also choose heuristically to inline that previous computation (duplicates the work but save the memory).

Within this framework, parameteric masks like vignetting (darken everything according to some parametric shape) could be efficiently compiled into the pipeline shader without creating intermediate textures.

This idea is tangentially related to the 'VSE: Strip Transition Gizmo' GSOC2026 - new transition effects should ideally be efficiently be implemented as shaders.

Is this idea of a full computation graph implemented somewhere and could be reused? Have there been discussions before? I'm very new to Blender and don't have a large overview, so I'm grateful for more context and pointers.

Color Space Handling

The standard workflow in Davinci Resolve is to have one CST at the beginning (camera color space to workspace color space) and one at the end (workspace color space to output-file color space). Thus only the minimal amount of conversions is performed. Preferably everything is mapped into an extremely big linear working color space with sufficient bit depth in the middle.

Blender seems to more intelligently/proactively perform color space conversions (e.g. the 'Linear Modifiers' button). Could this lead to more frequent conversions/reduced performance?

My approach would be to set the color space in menus once per input video file, once for the output file. Then let Blender complain:

• e.g. for cross-fades: when multiple inputs for some computation have different color spaces (cross-fade between HDR and SDR makes no sense - the use has to convert one of these)
• if the color space of the last node in the graph does not match with the color space of output file
• the operation does not make sense in the current space (blurring in non-linear space)

Global conformity can be checked easily with the full end-to-end computation graph (where only CST nodes can change the metadata). This way the user has full control but there is still hand-holding for beginners.

I also like the idea of color-space aware 3D LUTs. Usually creative 3D LUTs are designed for very specific input and output color spaces, but it is easy to get it wrong (in Resolve you have to manually look up and convert to the correct color space before and after).

Background

I'm an embedded software developer (C/C++/Rust) by trade with some background in graphics programming/rendering. I recently started working with Davinci Resolve (Free Version) as a hobbist. While I'm impressed with Resolves's feature set on Windows, stability on my main Linux machine is severly lacking. I wish for an open-source video editor with node-based editing. Since there seems some active development in the VSE space in Blender, I thought I might try to join the efforts.

I'd be interested to look into some challenges if it is doable in a reasonable timeframe for a hobbist and the general direction is approved by the developer community.

[1] When editing footage in Davinci Resolve, I noticed a peculiar behaviour. Whenever I first modified a cerain control knob (contrast, tint, exposure, ...), the video output would freeze for a few seconds. But after that, every change of contrast/tint/exposure/... would reflect on the output view instantly .

My guess is that Davinci Resolve recompiles a (big?) GPU image processing kernel with more options/code whenever a feature is actually used.