Local AI Auto-Dubbing & Subtitles for the Video Sequence Editor (Whisper + XTTS)

Currently, localizing a video (translating, subtitling, and dubbing) requires leaving Blender, using multiple external tools, and painstakingly synchronizing the new audio and text strips manually in the Video Sequence Editor (VSE). This process is slow, tedious, and breaks the editing workflow. The Proposed Solution: 5 I propose adding a native (or officially supported add-on) workflow in the VSE that allows users to select an audio strip and, with a single click, automatically generate synchronized subtitles and voice-cloned dubbing in another language using Local AI.

How it works (The Workflow): 8 1. Select an audio strip in the VSE (even if it's a cut/trimmed strip). 9 2. Click "Process AI Dubbing" in the N-Panel. 10 3. The system automatically extracts the exact selected audio using FFmpeg. 11 4. Uses OpenAI Whisper for high-precision transcription. 12 5. Uses MarianMT to translate the text to the target language. 13 6. Uses Coqui XTTS v2 for zero-shot voice cloning (the dubbed voice sounds like the original speaker). 14 7. Uses FFmpeg (atempo/silenceremove) to automatically time-stretch the generated audio so it matches the exact lip-sync timing of the original video. 15 8. Automatically creates a Text Strip (Subtitle) and a Sound Strip (Dubbed Audio) directly in the VSE timeline, perfectly synced. 16 17 Architecture (Avoiding Python Dependency Hell): 18 To prevent bloating Blender's internal Python environment with heavy AI libraries (like PyTorch or CUDA), I have designed and successfully built a working prototype using a Client-Server architecture : 19 * Server: A lightweight local FastAPI server running in a separate virtual environment that handles the heavy AI models. 20 * Client: A lightweight Blender Python Add-on that sends the audio cuts via curl and creates the VSE strips using context.sequences . 21 22 Why this is huge for Blender: 23 This brings cutting-edge, privacy-first (100% local, no cloud APIs), professional AI dubbing directly into Blender's VSE. It turns Blender into an absolute powerhouse for international content creators. 24 25 I have already coded the complete working prototype (Add-on, FastAPI server, and installation scripts) and I am ready to share the .zip source code with the developers or the community if there is interest in integrating this!