Extensions should provide an option to list required wheels instead of bundling them
I've been rewriting an addon that was working with Blender 4.1 to work with Blender 4.2. Because this addon uses AI, I need to do this so that I can include the necessary wheels. This has turned into a bit of a nightmare because there are so many wheels. scipy and torch are both indirectly included which require large wheel collections.
My original addon was about 20kb and could work on any distribution of Blender 4.1 or 4.0. As part of my script, I included a custom operator which would run pip on the user's system to install the necessary wheels, and it worked well. With 4.2, this no longer worked which is why I started looking into the extension system.
The first issue I ran into was that there is no easy way to download the wheels you need for any platform other than the one you are running on.
pip wheel allosaurus -w ./wheels
only downloaded Windows specific wheels, and to get Mac and Linux wheels I had to find Linux and Mac machines I could use and run the same command there and then copy the wheels to my project folder. I was also only able to get x86 wheels - I don't know how I'd get arm64 unless I found machines with that architecture.
When I finally got all the wheels I needed and built my extensions, they were massive. Instead of one 20kb addon, I now had Windows and Mac extensions which were both 200Mb, and a Linux extension that was 2 Gb. The Linux extension is so big I can't even upload it to my Github page.
Given these wheels are also platform specific, I can foresee having to create a new bundle for each version of Python that Blender uses in the future. As future versions of Blender are released with future versions of Python, I'm going to have to release more and more extensions for each Python platform. Not only is this going to be a hassle on my end, but I think most of my non-technical users will be confused as to which platform they need to download for.
Having to bundle wheels is a huge pain for someone trying to maintain a cross platform extension - especially AI and scientific computing which rely on
scipy
,
torch
,
tensorflow
, etc. All I need is to have
allosaurus
and its dependencies installed for the current machine and version of python that Blender is using.
Could the extension spec please be extended to allow me to simply provide a list of extensions I need and have Blender download the ones compatible with the user's OS/architecture/Python version? As it is I'm going to be maintaining a large set of 200Mb and 2Gb extensions just to cover all my user's different install bases.
4 Comments
You could use pip download which allow you to specify the OS, but it's going to be a lot of work getting all of the dependencies for each operating system and then packaging them up. Particularly if you want to support different versions of Blender, as you'll probably need to get different versions for each python version. I also believe that some different versions of the same OS require different package versions in some cases (according to deepai.com at least).
Even if you manage to do all that, I'm told by lazy dodo that anything with torch will likely be rejected due to licencing issues.
I think the only option is to provide it as an addon and have the addon download the dependencies during or after installation. That way the user is guaranteed the correct versions for their OS without the hassle of creating and maintaining multiple variations of the same addon. You can also specify the package versions in the pip command to ensure all packages remain compatible with each other regardless of newer ones being released.
I understand the problem. Solving it in Blender might introduce unclear security aspects, that users, add-on developers or Blender doesn't want to be dealing with. Distributing wheels makes it clear. The add-on delivers its dependencies and is therefor also responsible for validation etc. Python is known to break packaging.
Distributing wheels is the most proven way how to solve compatibility issues between patch releases. I have seen that minor patches in wheels include different API's or other dependencies and even break starting at all. If downloading wheels is pushed to Blender responsibility an add-on can break based on the time it was installed as the downloaded wheels are different.
You could also consider to automate downloading the dependencies inside the add-on sourced from a file server for compatibility reasons, or use a web approach.
I would also expect from the 2GB of data/code there is a huge overlap that you don't need to store multiple times. You could also try to find them and use logical links on the file system.
How would I automate downloading dependencies from inside the addon? Would I have to set up a special server for this? I'm not an expert python developer - I'm just trying to find a way to include the
allosauruslibrary in my addon. I already tried runningpipfrom within the addon - while this technique worked in Blender 4.1, it no longer works in Blender 4.2.I already have my build script separating the Linux extension from the others. I'm not sure what I would need to exclude or how I would do it. I think most of the 2Gb is coming from a bunch of nVidia wheels that are used for GPU computing. I would expect anyone else using
torchwould run into this same issue. How am I supposed to reduce the size here? Create separate linux distros for each nVidia card?I can understand wanting installation to be consistent, but this is going to be a nightmare of hundreds of different distributions I have to manage. One for each OS, chip architecture, version of Python and now nVidia card too. I also don't have the multiple gigabytes bytes of server space I'm going to need to store all this. I would rather have Blender managing this, even if this means that different users may have wheels with different versions.
I also don't know how I'd figure out each of the dependency lists I'd need for each OS/architecture/Python version/GPU.
piponly downloads what you need for the machine you run it on. I've not found any reliable way to get a list of what I need for any other platform. (pip does provide some flags that make it look like you can select other platforms and architectures, but it doesn't work unless every dependency has a binary version).Perhaps it would be acceptable to approve extensions without all dependencies, providing this is declared on the extension page with a list of the packages the extension will download, and a disclaimer that the Blender foundation is not responsible for checking each package's safety/compatibility.